Chapter 10. Verification: A Process That Secures Acceptance in Three Stages

When fourth-stage narrative writing is finished, all 1,500 LLM blocks contain prose. That alone, however, does not mean the Vault is complete.

At this point the Vault is extensive but unverified. It may still contain broken links, duplicate entries for the same person, or entries assigned to the wrong worldbuilding axis. If the Vault is handed to downstream work before such problems are found, that work inherits the errors unchanged.

This chapter covers the verification stage that finds and handles those latent problems.

The misconception that verification slows creation

People encountering verification for the first time sometimes say, “There are too many checks. If every one of these must pass, won’t creation become slower?” This treats verification as an obstacle to creative work. The history of software development has shown, through long experience, that the view is mistaken.

In the early software industry of the 1960s, teams often wrote code as quickly as possible and fixed it only after a problem appeared. They released without verification and repaired defects when users found them. The result was poor: errors accumulated, fixing one produced another elsewhere, and the cost of repair rose exponentially as the codebase grew. The industry learned that the closer an error is caught to the moment it is introduced, the lower its repair cost. An error caught during first-stage extraction costs one unit to correct; the same error, after passing through stages two and three and spreading across 1,500 Vault files, can cost dozens of times more. Verification does not slow creation. Without verification, creation becomes slower.

Manufacturing teaches the same lesson. One core principle of the Toyota Production System is to stop the line as soon as an abnormality is found. Continuing to make defective products can appear faster, but collecting, discarding, or repairing them later costs far more. Librarying’s three verification stages bring this principle into creative work: each stage catches a different layer of problem at a different cost.

Two cases in which verification protects creation

Wikipedia is a public creative project that operates verification at great scale. It has three core content policies, one of which is verifiability: claims likely to be challenged and their citations must be supported by reliable sources. Edits that violate the principle can be restored to an earlier state through a revert. When edits conflict over the same document, the dispute is handled through processes such as dispute-resolution noticeboards or Requests for Comment (RfC). These are not regulations that prevent creation; they protect it. If unsupported claims accumulate, nobody can distinguish fact from invention and the encyclopedia as a whole becomes untrustworthy. Verifiability and reversion prevent that collapse. Librarying’s automated stage, which catches broken links and duplicate entries, plays a comparable role.

Games provide another example. Large games run regression tests as part of QA before release, checking that a new feature has not broken a part that was already working. After release, patch notes disclose what changed and how. Together they preserve the consistency of a large game world. Librarying is no different: rerunning the automated stage after a third-stage skeleton build or fourth-stage writing is its regression test, and recording verification status in one line in the Vault index is its patch note.

Issue priorities: P0, P1, and P2

Issues found during verification are classified into three levels. The classification prevents time from being wasted by treating every problem as equally urgent.

P0 issues directly affect whether the Vault can be used or damage entry identity, and they are fixed immediately. Examples include one person registered under two entries, a broken link in a hub entry, a file whose frontmatter YAML fails to parse, or a worldbuilding-axis value outside the allowed set. These are failures at the level of the Vault itself. If even one file fails YAML parsing, downstream work that tries to read it produces an error; tracing the source later takes time. A fix that would take three minutes when first discovered can take thirty minutes once it must be traced.

Issues that damage traceability or reference integrity are P1 and must be fixed before handoff. A related link pointing to a nonexistent file, or a mismatch between the manifest’s entry count and the actual file count, belongs here. The Vault still runs, but its traceability is compromised.

P2 covers structural improvements or later reinforcement and may be carried into the next cycle. A leaf entry slightly under the target narrative length, or unfilled %% LLM %% blocks in five percent or fewer of entries, are P2: they do not prevent use of the Vault.

Consciously deciding what must be fixed now and what can wait is the essence of issue prioritization.

Stage 1: the automated stage—let machines catch what machines can catch

The automated stage checks integrity items that scripts and automated tools can inspect. It decides pass or fail without human judgment.

It resembles unit testing in software development. A unit test automatically checks whether a function behaves as expected; when it fails, the problem is immediately localized, making it faster and more accurate than having a person read all the code.

Librarying’s automated stage has checks for every pipeline phase. The stage-one completion check asks whether work-plan checkboxes are 100% complete, source documents are covered evenly, and all nine categories contain at least one item. An unexpectedly empty object category may mean props such as ladles or cauldrons were missed altogether. The stage-two check requires zero duplicate representative keywords, zero missing English names, zero broken wikilinks, “undecided” version status in no more than five percent of entries, “undecided” worldbuilding axes in no more than ten percent, and zero values outside the allowed axis set. The stage-three check compares the number of .md files with the number of rows in the stage-two CSV and requires zero frontmatter YAML parsing failures, filename collisions, and missing worldbuilding_axis values. The stage-four check requires at least 95% completed LLM narratives, at least 60% content relative to formatting, at least four paragraphs for hub entries and one for leaf entries, structured sections by category, zero self-referential links, and agreement between manifest progress and actual file state.

These items can be checked almost immediately through automation. A failure is registered as a P0 or P1 issue and handled at once. The automated stage has two values: it finds problems immediately instead of waiting for a person to read every file, and it can be rerun after a skeleton rebuild or narrative rewrite to reveal newly introduced problems.

Stage 2: the manual sample stage—review for meaning and judgment

The automated stage catches structural integrity. It cannot decide questions that require meaning and judgment, such as whether a group of synonyms truly refers to one person. That is the role of the manual sample stage.

The principle is sampling. A factory need not inspect all 1,000 products to obtain a practical signal about overall quality; it can inspect a random sample carefully. The manual sample stage applies this principle. Rather than having a person read all 1,500 entries, it closely reviews randomly selected samples and looks for risk signals in the overall quality.

Its rule is therefore not complete inspection but direct reading of one or two randomly selected entries from each category. It checks four things.

Appropriateness of consolidation decisions asks whether the alternative names and aliases of five randomly selected entries really point to the same object, and whether different people have been incorrectly merged under one entry. Such errors may descend from AI decisions in second-stage consolidation and can only be found by human reading.

Tone consistency in fourth-stage narratives compares one hub entry with five intermediate entries in the same category. It checks whether the core framing established in the hub operates in the same way elsewhere. When different AI instances worked on different batches, the tone can abruptly change at batch boundaries.

Appropriateness of worldbuilding-axis classification randomly selects five entries from each axis and checks whether they truly belong there. An entry classified under “Heavenly Mandate,” for example, may actually belong more naturally to “Event.”

Accuracy of structured sections checks whether the “constraints” section of a device entry points to actual sources. If an AI has written an unsupported guess as a constraint, it is a P1 issue.

This manual stage takes thirty minutes to an hour. Findings are registered as P1 or P2. If a serious issue appears in a sample, the category containing it is inspected more broadly. A sample that sends a warning signal calls for a wider look.

Stage 3: the integrated-build stage—does it work when it is actually used?

The third stage is verification at the highest level. The automated stage checks the integrity of individual files and the manual sample stage checks the accuracy of meaning; the integrated-build stage examines the experience this Vault gives in real use. In software terms it resembles User Acceptance Testing (UAT): even software that has passed unit and integration tests can reveal unexpected problems in the hands of an actual user.

The integrated-build stage does three things.

Follow one hub entry from beginning to end. Open a hub entry and examine, in order, its frontmatter, definition, detailed explanation, LLM narrative, related links, and backlinks—other entries that point to it. Along that path, check whether the information is sufficient, links work correctly, and the narrative answers the IP’s questions. While following the “Rain Ritual” entry in “Usabu”, a reader may want to know what becomes of the person who performed the ritual later. If no link leads to that information, it is a P2 issue: the Vault does not stop, but a necessary connection is absent.

Simulate writing one hypothetical short story. Try to write a short scene using only the Vault and note where the work becomes blocked. That point is precisely where the source material needs reinforcement. If a scene about a person performing the rain ritual for the first time cannot establish where it takes place because the Vault has no such information, the location entry or the ritual narrative must be supplemented. Register that as P2.

Simulate a new user. Check whether someone seeing the Vault for the first time can move from the index to core entries and then to related entries, and understand the IP’s worldbuilding within thirty minutes. Longtime creators already know the Vault’s contents and unconsciously fill gaps from memory; new users cannot. When a new editor or translator joins, they must be able to understand the IP from the Vault alone. The new-user simulation tests exactly that state of readiness.

This integrated-build stage takes one to two hours. Findings are registered as P2 and become a reinforcement plan for the index, hub entries, and supporting documents.

The handoff acceptance checklist

After the three stages, review the handoff acceptance checklist. Every required item must pass before the Vault is handed to downstream work.

Any failed required item holds the handoff. Required items are: every automated-stage check passes; every one of the five review stages after fourth-stage delegation passes; the manifest displays “ready for handoff”; all P0 issues are fixed immediately; all P1 issues are fixed or explicitly deferred; and the profile, worldbuilding-axis catalog, and supporting-document catalog are attached. Recommended items permit up to one failure: the manual sample stage passes; the integrated-build stage finds the material sufficient on its own; supporting documents are produced as catalogued; and hub entries were written directly by the producer.

When this checklist is complete, record the verification state in one line in the Vault index so that anyone opening the Vault can immediately see its condition.

Why the passing thresholds differ by stage

Each pipeline stage has a different threshold because its purpose differs. Stage one aims to collect as much as possible, so 60% coverage can be enough to move forward. Stage two is consolidation and refinement, so most entries must fill all thirteen columns and reach 95%; the remaining five percent may remain “undecided.” Stage three is decisive automation, so every file must be generated without error: 100%. Stage four applies different length standards to hub and leaf entries but requires at least 95% narrative completion across the operational Vault. Short leaf entries are acceptable; a large number of empty ones is not. The remaining unfinished portion must be five percent or less, explicitly recorded as P2, and placed in the next cycle’s reinforcement plan.

This threshold system makes clear where perfection is necessary and where an appropriate compromise is allowed. Pursuing 100% in stage one wastes time, while reaching only 99% in stage three is a problem. Each stage has a different standard for a reason.

From verification to handoff

A Vault that passes these three stages is ready for handoff to downstream work. But after handoff, does the Vault stop there? It does not. It remains alive with the IP.

Chapter 11 examines the operating cycle: how the Vault is updated when new IP content is released, how new entries flow back upstream, and how the Vault continues to live for as long as the IP lives.