Building a Local Knowledge Wiki Part 3 Source Imports and Retrieval Failures

by admin

The local model finally answered a real question correctly: how many domains are in the AI-SGAP framework?

The next question exposed a different problem. Asked how the risk-floor logic was calculated, the application returned “Not enough evidence.”

The detailed specifications existed. They were linked from the wiki. But their contents had never been imported into the local search corpus.

This is where building a knowledge assistant becomes a data-engineering problem.

A link is not imported evidence

Our first local snapshot contained 23 wiki pages: 17 domain pages and six audit pages. Its index had 946 chunks and 6,180 document/chunk/link metadata triples. Thirty-one linked attachments remained references only.

Every page matched its frozen export byte-for-byte through the activated local API. That established snapshot integrity. It did not establish coverage of every linked document.

The distinction should be visible in both the manifest and UI:

State

What the application can claim

Imported and verified

The indexed content belongs to the recorded source snapshot

Reference only

A link exists; the linked contents are not searchable evidence

Stale or changed

Previously indexed evidence needs reconciliation

Unavailable

The application cannot currently establish the contents

Do not let “indexed sources” become a count of URLs that the system has never read.

Preserve source identity before chunking

For each source, retain its identifier, revision, original-content hash and extraction identity. Chunks should point back to the source and its original location.

A conceptual manifest entry looks like this; field names are illustrative:

{
  "source_id": "demo-spec",
  "revision": "1",
  "original_sha256": "<sha256 of original bytes>",
  "extracted_sha256": "<sha256 of extracted text>",
  "content_state": "imported",
  "source_kind": "document",
  "domain_or_audit": "domain"
}

The original file and extracted text need separate hashes. A document can remain unchanged while extraction behavior changes. A hash of extracted prose alone cannot identify the original package.

Our source corpus was a frozen snapshot. The cloud wiki’s nightly updates did not silently synchronize into the Mac. That avoids presenting an old local answer as if it had used today’s cloud source.

Importing the missing evidence

We imported two versioned specifications and the assessment workbook. The resulting corpus held 26 sources: 20 domain sources and six audit sources. Twenty-nine linked files still remained reference-only.

Audit records were excluded by default. Otherwise an execution log describing an old failed answer can compete with the actual domain source that should answer the question.

The first expanded index contained 5,425 chunks and 28,667 provenance triples. A later section-aware retrieval repair changed these to 5,541 chunks and 29,247 triples. These are index records, not accepted real-world facts.

The source-import phase passed 112 aggregate tests and 21 extractor tests in separate scopes. The original risk-floor question then passed. Two harder questions—RF14/adoption and the current executable engine—failed quality checks and were withheld.

That outcome is more informative than saying “the import passed.” File parsing, retrieval and answer quality need separate results.

A correct source can still yield the wrong excerpt

A later question asked for an application count and timeline. Search omitted the relevant IP page from the top five; its best chunk ranked sixth.

The model-preparation path was slightly different. It had selected an IP passage, but that passage discussed support review rather than the filing record. It also selected an irrelevant specification passage.

We reconstructed the prompt and matched it byte-for-byte to the saved request: 2,877 characters. That let us distinguish a retrieval defect from a hypothesis based on a screenshot.

Two lexical problems contributed:

  • Exact matching missed inflections such as patent/patents and filed/filing.

  • Generic words such as many and each promoted unrelated prose.

A shared project name in a title could also dominate selection while the relevant section remained out of view.

The bounded repair

We added generic inflection matching and respected wiki section boundaries. Collection questions could select a later relevant section instead of repeatedly favoring a page’s opening material.

This was not an answer lookup table. We did not embed application counts or dates in the retrieval code.

After repair, Search ranked the relevant page first. The model received two windows from that source: offsets 0:319 and 319:969. The prepared prompt was 2,239 characters and 2,241 UTF-8 bytes. Prior risk-floor and RF14 windows remained byte-identical.

Keeping both character and byte budgets matters when punctuation or non-ASCII text changes encoding length. Keeping original offsets matters when the same sentence appears more than once.

The prompt still ended with clipped source material. A bounded context is not full-document coverage.

Test retrieval independently from generation

A useful retrieval test records:

question
source snapshot identity
ranked search results
selected source IDs
selected source offsets
exact prepared excerpts
prompt hash
expected evidence present or absent

Then inspect the actual model response separately.

Our corrected retrieval passed 116 tests in staging and, after fixing a packaging omission, 116 tests in deployment. The first deployed test run failed to import a fixture module that had not been copied. That was a packaging failure, not an application assertion failure; both runs remained in the record.

Despite the retrieval repair, the combined model answer still returned unknown. This was a retrieval fix, not an end-to-end answer fix.

Check the browser against the deployed version

An already-open browser tab still showed the older 23-page interface. Reloading revealed the 26-source version. Search then ranked the correct source first and the source dialog displayed the expected evidence.

Backend deployment does not refresh an already-loaded frontend automatically. Record both server verification and a fresh browser check.

Earlier visual verification had been blocked by the locked Mac. That observation stayed in the history; a later successful browser check superseded it for the new checkpoint.

What I would reproduce first

Build a small corpus containing a heading, a relevant later section, a similar but unrelated section, and a reference-only attachment. Test inflections and generic query words. Freeze the corpus so that retrieval changes can be compared against identical bytes.

Then ask two questions: one fully supported and one only partially supported. Verify which evidence reaches the model before changing model size, context limits or prompts.

Next: why exact quotations still failed, and how selecting passage IDs improved partial answers.