After we repaired retrieval, the model still failed a combined count-and-timeline question. It had relevant evidence, yet the application could not return a useful validated answer.

The next failure was surprisingly small: the model removed Markdown bold markers while copying source text.

The rewritten quotation read correctly to a human. It was not an exact quotation of the source. Our validator rejected it.

This installment explains the change that helped: the model selects evidence IDs, and application code retrieves the original text.

Diagnose the bytes before relaxing validation

Our initial hypothesis was that the model had associated a quotation with the wrong source window. Exact comparison disproved that.

The source label was correct. Neither returned string appeared verbatim in either selected window or in the complete source. Removing Markdown delimiters explained the mismatch, but we did not introduce quotation normalization into production.

An experimental same-window helper was preserved for the audit trail and reverted before activation. Keeping a failed hypothesis is useful when it explains why a tempting fix would have solved the wrong problem.

Let the model choose and let code copy

The final contract contained exactly three top-level fields: citations, unknowns and status.

Instead of asking Qwen to reproduce original prose, the backend prepared a deterministic candidate inventory. The model selected IDs from that inventory. The backend then resolved them to original source spans.

Bound	Recorded value
Retrieved excerpts	At most 2
Candidate passages	At most 8
Selected quotations	At most 2
Quotation length	At most 240 characters each
Prompt limits	3,000 characters and 3,600 UTF-8 bytes
Context	2K
Generated output	At most 256 tokens

Each ID binds the page identity, source hash, revision and exact Unicode offsets. The backend recomputes the inventory during validation and rejects changed, stale, duplicated, unlisted or cross-field selections.

The following is conceptual validation order, not a replacement implementation:

def validate_selection(raw_model_json, prepared_request):
    result = parse_strict_json(raw_model_json)
    validate_exact_schema(result)
    current_sources = reread_and_verify_sources(prepared_request)
    inventory = rebuild_candidate_inventory(current_sources, prepared_request)
    verify_prepared_inventory_unchanged(inventory, prepared_request)
    spans = resolve_allowed_unique_ids(result["citations"], inventory)
    verify_source_fields_and_named_subject_scope(spans, prepared_request)
    verify_status_and_unknown_contract(result, spans)
    return render_original_spans_and_fixed_unknowns(spans, result)

Candidate passages must be complete eligible source-field values, complete original lines, or complete sentences within the size cap. A clipped prefix or trailing sentence is not promoted into a complete fact. Generated extraction labels are not evidence.

Python Unicode offsets and UTF-8 byte offsets are different units. Our answer metadata explicitly identifies Unicode codepoints. Do not slice UTF-8 bytes with a character offset.

Partial answers need a contract

The supported statuses were answered, partial, unknown and conflict. A partial result required both citations and unknown fields.

Unknown labels came from a bounded deterministic enumeration: a fixed remaining-details sentinel and selected question clauses. The backend rendered fixed wording for unknowns, rather than allowing the model to invent an explanation.

The important semantics are:

Unknown does not mean false.
Unknown does not mean zero.
A cited partial answer does not imply full question coverage.
A model-proposed conflict still needs interpretation.

We ordered the output fields as citations, unknowns, status so the task emphasized extraction before classification. That is a prompt/schema choice, not a guarantee of model reasoning order.

Exact evidence can still be irrelevant

An early model batch returned an exact quotation from the wrong project. It passed provenance validation.

That exposed the limit of “the quote exists.” The application also needs to ask whether the evidence belongs to the requested subject.

We added bounded lexical checks for named projects and identifiers, with exact boundaries so P3 did not match P30 and RF14 did not match RF140. A missing sole named subject returned no evidence before model loading.

Spreadsheet evidence could use original visible cells in the same row for an identifier. Other rows, hidden cells and generated locator labels could not provide that context.

This improves containment. It is not general entity resolution or semantic entailment. Aliases, ambiguous names and unsupported phrasing remain limitations.

Keep reviewed counts separate from generative answers

General Ask Qwen remained extractive source QA. A separate application-record view used fixed SPARQL queries over explicitly reviewed records.

The flow was PROPOSED to ACCEPTED or REJECTED. Model inference could not accept records. Accepted records with source drift or conflict were retained but excluded from eligible counts.

Our review endpoint required JSON containing exactly:

record_id
decision: ACCEPT or REJECT
revision
source_sha256
confirmed_review: true

It also required an X-Review-Token header and enforced loopback, Host, Origin, fetch-site and JSON boundaries. Duplicate JSON keys, extra fields, boolean revisions, stale revisions and changed evidence failed closed.

Repeating the same decision with the current revision was idempotent. Replaying a stale revision returned a conflict requiring refresh. Opposite decisions on reviewed records were blocked.

Persistence used a separately locked, atomically replaced ledger with hash-linked review events. Those hashes help detect inconsistent changes; they are not credentials, proof of human identity or protection against an owner who can rewrite the entire ledger.

What the latest checkpoint establishes

The deployed Python suite passed 195 tests. Another 62 independently authored checks passed against the backend. These are separate test scopes, not 257 independent model-quality cases.

The final five-case batch used four actual model calls and one missing-project control that did not load the model. A browser compound question also returned a cited partial answer.

The application-record view initially contained three unreviewed proposals. I subsequently accepted all three. A read-only verification at 20:27:49 UTC on 11 November found three accepted, source-current, conflict-free records and three persisted review events.

That gives a count of reviewed source-linked statements. It does not establish a complete portfolio or independently verify official records. Unknown dates and other unresolved fields remained unknown after acceptance.

What remains unfinished

The eight-candidate cap omitted three otherwise eligible passages in a risk-floor coverage probe. Exact-text selection protects provenance but cannot recover evidence that never reached the candidate inventory.

Distinct source cells containing identical text can produce repeated wording. Relevance and completeness remain partly model-selected. The earlier RF14/current-engine compound failures were not retested in this phase.

Our next improvements need evaluation against those gaps: candidate coverage, subject attribution, partial-answer usefulness and freshness after source changes. More passing backend tests cannot substitute for that evaluation.

We are fine-tuning the system’s behavior. No model-weight fine-tuning or training has occurred.

Building a Local Knowledge Wiki Part 4 Exact Evidence Partial Answers and Human Review