Chapter 13. Data Pipeline

Confirmed things must be stored and carried. This is the last action of the work cycle, registration, and the problem of the documentation system that surrounds it. The guiding question of this chapter is: how does a world become a document? The answer is a data perspective. We view worldview work as a flow of information and design a pipeline that runs from source through the master canon to the user. This is not a romantic chapter, but its content is the substance of the profession itself.

1. Data-Flow Protocol: The Origin and Destination of Information

The pipeline's first task is drawing the plumbing diagram. The data-flow protocol is the document that fixes where information about the world is born, by what route it reaches the master canon, and to whom it is supplied. Three points require regulation. The source. Where is this world's information born? From new installments of the source work, design meetings, the writer's Q&A sessions, to proposals from external partners — once the list of sources is fixed, the list of points where extraction must be applied is set. Information born outside the list — private-conversation agreements and personal notes — is the pipeline's single biggest point of leakage. The route. Once information is born, what stages does it pass through to reach the master canon? The order of the extraction list, definition confirmation, and registration review, and the person responsible for each stage, are settled. The protocol's real function is blocking routes not on the list: the shortcut that runs straight from a meeting to the master canon. Information that enters by shortcut is information that has not been cross-checked, and we have already confirmed that uncross-checked information is where paradoxes are produced. The destination. To whom, and in what form, is the master canon's information supplied? The form of supply for each recipient — internal writers, external partners, marketing, disclosure to fans — is fixed. Only with this plumbing diagram in place does the rest of the process find its footing. The design implication is the size of the protocol. A data-flow protocol fits on one page, and any protocol longer than one page will not be observed. A protocol everyone knows protects the pipeline better than an elaborate one. The success metric for the protocol is not the number of clauses but the drop in shortcut traffic.

2. Decomposition of Existing Works: Why Do It

We deal first with the pipeline's upstream, the processing of the source. Most worldview work begins with the decomposition of an existing work. Entry after completion obviously starts with decomposing the accumulated installments, but so does entry mid-serialization, and even blank-page design often has decomposition of a reference work as its preliminary step. The reason decomposition is unavoidable lies in where verification resides. An existing work is a storehouse of settings that have passed through the market. Evidence of which settings worked on readers already exists in the form of the work and its reception, and this history of verification is an asset that no newly invented setting can possess. Decomposition is the mining that extracts that asset. The mining is complete only when the market's history of verification is extracted along with it. The record of which settings earned a response from the fandom lies outside the body of the work itself. Another reason is where authority resides. In a world built on a source work, the master canon's authority ultimately comes from the source text itself. A setting compilation whose contents have not been surveyed in full is a document with a shaky basis for its authority, and it collapses at the first collision with the source work. Decomposition is the foundation work of authority. Only a designer who has completed a full-inventory survey earns the standing to declare flatly that no such mention exists in the source work. The last reason is measuring the debt. The full extent of a naturally-arising world's contradictions comes to light only through decomposition. Without knowing where a paradox is buried, no plan for settling it can be made either. Repayment of a debt begins only from the moment it is measured. The design implication is an upgrade in decomposition's standing. Decomposition is not chore work done before the real creation starts; it is asset due diligence for the world, and it is a process that must be entered as an independent line item in estimates and schedules.

3. The Method of Decomposition: From Work to Keyword

Carrying out decomposition is a full-inventory version of the extraction procedure. The four stages — fixing the source, reading through and marking, cataloguing, judgment — apply as the same skeleton, but with full-inventory scope and systematicity reinforced. The method has three key points. First, multiplying the reading lens. One read-through cannot fish out everything. Switching lenses across several passes — a pass that mines proper nouns, a pass that mines rules and laws, a pass that mines relationships and factions — is faster and more accurate than aiming to catch everything at once. The map of the eight domains is reused here as the list of lenses. The discipline of wearing only one lens per pass protects both focus and speed at the same time. Second, recording positional coordinates. Every fragment fished out must be tagged with the coordinates of its source: the installment and the scene. Half the value of a decomposition deliverable lies in these coordinates. Only with coordinates does citation of the source's grounds hold up, and only then can the text be checked immediately when a later dispute arises. Third, collecting contradictions separately. Setting conflicts discovered during decomposition are collected, not fixed on the spot. If the person decomposing patches over a contradiction on their own judgment in the field, a setting that does not exist in the source work seeps into the compiled version. Contradictions are registered in the contradiction register along with their coordinates, and handling them is passed on to judgment at the registration stage. Separating discovery from judgment is the discipline of the decomposition process. The design implication is the specification of the decomposition deliverable. A keyword list, coordinate notation, and a contradiction register are the three standard deliverables of the decomposition process, and only in this form can the next process pick up the work.

4. Preprocessing: Cleanup Before Registration

The list that decomposition and extraction pour out is still raw. Preprocessing is the cleanup process before it goes up for registration review. We keep the data-work term deliberately, because the work it names is in fact the same. Preprocessing has four standard operations. Merging duplicates. Cases where the same object was fished out several times under different notations are found and combined into one. Whether the Obsidian Knights, the Obsidian Order, and the Knights are the same organization is determined, and if they are, the entries are merged while every notation found is preserved as an alias. Normalizing notation. A provisional representative notation is assigned to the merged entry, and the notation across the whole list is aligned to it. Since fixing the master canon's notation is the naming procedure's job, this is only a provisional unification. Unifying units and format. Format elements such as the notation format for dates, currency, and distance, and the word order of entry descriptions, are aligned. A list with scattered formatting eats into the efficiency of the cross-checking work. Flagging gaps. Any required information for an entry that is empty is explicitly marked as blank. Writing down what is unknown as unknown is an important output of preprocessing, and this list of blanks becomes the raw material for the undefined queue. The design implication is the possibility of dividing this labor. Much of this process can be carried out by rules alone, without deep understanding of the world, which makes it the ideal entry point for new hires, support staff, and automation tools. The designer's judgment is spent sparingly, only where a ruling is actually required. An organization that burns its judgment on cleanup finds the quality of its actual rulings degraded.

5. The Reason for Registration: The Grounds for Needing a Procedure

The gate through which a cleaned-up candidate rises into the master canon is registration. What practitioners ask most often is why the procedure is needed. Why not simply transcribe a finished setting into the document — why does a review gate have to exist? There are three grounds. First, because registration is the act that produces Canon. Invert the practical definition that Canon is the set of registered entries, and the quality of the registration procedure is exactly the quality of Canon. In a world with no gate, there is no way to answer the question of what is official, and where there is no answer, the loudest memory in the room fills the vacancy. Second, because registration is the device that forces cross-checking. Recall the diagnosis that paradoxes arise because settings go uncross-checked. Registration review forces, as procedure, the cross-checking of a new entry against the entire body of existing entries, and without this force, cross-checking is always skipped on a busy schedule. This is also where the principle that the gate is inspection, not censorship, comes from. The review's question is not whether the setting is good but whether it collides with the existing world. Third, because registration notarizes a point in time. The record of since when something has been official becomes decisive evidence in disputes between derivative works, accusations of plagiarism, and rulings on contract performance. The design implication is calibrating the procedure's weight. Once registration becomes a bottleneck, the field starts routing around the gate. Setting up a dual track of an informal and a formal process by an entry's importance is the realistic compromise that keeps the gate standing. A supporting shop's signboard goes through the informal track; the world's fundamental laws go through the formal one.

6. The Registration Procedure: The Gate Into the World

We now fix the standard procedure for registration. It has five stages. First, motion. A candidate whose definition and name have been confirmed is submitted in the form of a registration application. The substance of the application is a properly specified entry draft. It must be equipped with a definition, a name, a source or the rationale for the decision, and reference keywords to qualify for a motion. Second, conflict check. The new entry is checked against the existing master canon. A full cross-check against everything is impossible, so practice rides the reference network. The keywords the new entry references, and the range of entries that reference those keywords, are the primary scope of the check; cross-checking against global documents such as the timeline and the map is the secondary check. Third, ruling. There are three rulings — pass, conditional pass, and rejection — and a conditional pass carries the required revisions while a rejection carries its grounds. Whether to fix the new entry or revise the existing one when a conflict is found is also decided at this stage, and if revising the existing entry is chosen, a separate revision procedure is set in motion. Fourth, recording. A passed entry is incorporated into the master canon, and its registration date, reviewer, and ruling history are attached to the entry. Fifth, notice. The result of registration is propagated to the pipeline's destinations. A writer unaware that a new entry exists will keep writing in ways that collide with it, so notice is not a decorative flourish on the procedure but its closing step. The design implication is a speed target for the procedure. Fix and publish the standard processing time from motion to notice. Only a predictable gate is a respected gate. A review with no known end date is the best advertisement for a detour.

7. The Index Document: The World's Table of Contents

The entries piled up in the master canon demand structure. The top layer of that structure is the index document. The index is both a table of contents that surveys every entry in the world and the front door of the document system. Its form is a hierarchical list. The eight domains serve as the top-level categories, with subcategories and entry names branching beneath them, and each entry name links to its corresponding document. If the working tool is a wiki, the index is the home page; if it's a spreadsheet, it's the table-of-contents tab. The index's function is to complement search. Search is a tool for someone who already knows the name; the index is a tool for someone who doesn't know what exists. A newly joined writer, an external partner, a future successor all enter the world without knowing what words to search for, and the index is the only thing that shows them the world's full picture. The index also serves as a management dashboard. Skew shows up in the distribution of entry counts by domain, and gaps show up in the distribution of undefined markers. The index is the place where the dashboard functions mentioned earlier are physically implemented. The principle of maintenance is automation. A system in which the index is updated by hand is bound to drift out of step with reality. Either build tooling so that registrations and revisions to entries are automatically reflected in the index, or, at minimum, include an index update in the recording stage of the registration procedure. The design implication is treating the index as a landing page. Route every entry point into the document system through the index, and the index's freshness becomes the freshness of the entire system. A house with a shabby front door loses trust before anyone even looks inside.

8. The Datasheet: Standardizing the Entry

If the index is the aerial view, the datasheet is the macro shot. The datasheet is a form that holds an individual entry's information in standardized fields, the form in which the five conditions of deliverable specification are implemented at entry level. The standard fields are as follows: canonical name and aliases, classification domain, one-line summary, definition, detailed description, source or decision log, reference keywords, registration date and version, disclosure tier. The crux of this form is that half its fields are not the body text but management information. The sheet's effect is homogenization. Even when the authors differ, the shape of each entry stays the same, so the reader finds the same kind of information in the same place no matter which entry they open. A secondary effect is that empty fields automatically expose what is missing. A blank is not evidence of ignorance; it is the coordinates for the next task. In a prose document, missing information is invisible, but in a form, it shows up as a blank. The tangible-and-intangible three-way format connects here. Specification form, clause form, and definition-and-example form are implemented as the three writing modes of the detailed-description field, while the management fields are shared by all three. What needs watching is the form's bloat. Fields are easy to add and hard to remove. As unfilled fields pile up, the whole form turns into a formality, so required fields must be capped under ten and the rest demoted to optional fields. The purpose of a form is not perfect record-keeping but sustained record-keeping. The design implication is version control for the form itself. The datasheet form is itself a document that carries a version and gets revised, and revising the form is carried out together with a migration plan for every existing entry.

9. Version Control: Revising Worldview Documents

Worlds get revised. Settings are updated, names change, contradictions get settled. Treating revision as a process rather than an accident is version control. The principles are the same as software's. First, version notation. A version number is assigned to the whole document and to individual entries, and any update raises the number. That a recipient can confirm for themselves whether the document they hold is the latest is the substance of the traceability condition. Second, a change log. A list of what changed, when, and why is kept for every version. The record of rationale matters especially. A change log without a rationale makes the next person in charge repeat the same mistake. Third, managing retroactivity. Unlike code, revising a worldview carries the problem that already-published deliverables cannot be retroactively corrected. Works made with the old edition's settings remain out in the market. So a category unique to worldviews is added to version control: the old-new version correspondence table, the official record of which setting in the old edition became what in the new one. A reboot and a retcon — that is, a declared retroactive revision — can be described as the event of disclosing this correspondence table to the fandom. A retcon without a correspondence table ends up at war with the fandom's memory. The principle of recording differences for managing derivative versions repeats here along the time axis. The design implication is regularizing revision. Ad hoc revisions wear recipients out, so a rhythm that gathers minor updates and releases them as a periodic version becomes the foundation of trust in the document. The requirements for an emergency revision are defined separately.

10. The Document's Readers: Who Reads This Document

The pipeline terminates in a person. A document exists to be read, and when the reader differs, the same world has to become a different document. The readers of worldview documents sort into four kinds. Internal creators. They access the world's full picture and all its management information, and the form optimized for them is the master-canon system designed up to this point. External partners. They must receive only information within the scope of contract, in their medium's own language. The principle of the per-medium briefing document connects here, and the disclosure-tier field is the valve that controls this supply. Management and evaluators. These are readers who read not the world's details but the scale and scalability of the asset, and their document is a due-diligence summary that condenses entry counts, coverage, and the expansion roadmap. And fans. Part of the master canon is supplied as a product, in the form of an official setting compendium or guidebook, and here the document's standing shifts from a management tool to content. The judgment of what to disclose and what to hold in reserve — that is, the cultivation design of gaps — is the editorial principle behind this supply. The principle that runs through all four kinds is single canon, multiple outputs. Only one master canon is maintained, and every reader-specific document is derived entirely from it. The moment separate master canons are made for separate readers, the pipeline gets a forked headwater, and the world suffers a derivative-version problem at the document level. The design implication, and the conclusion of Part 3, is this: the designer's final deliverable is not the settings but this entire flow — the system that keeps the world alive as documents.