A local AI pilot needs more than a successful model download. It needs an identifiable runtime, bounded resource use, observable cleanup, and a rule for what happens to model output.
In Part 1, our first MLX and Q4 attempts stopped under memory pressure. This installment covers the configuration and validation boundaries we used to make the next tests interpretable.
Verify artifacts before installation
We checked vendor release metadata, package provenance and archive contents before extracting the portable Ollama runtime. The CLI archive’s recorded SHA-256 was:
a743717494315da7cd352982b805f6b992d4b4452ca117fd5fbf86c4ee46fca0That fingerprint applies to the specific 0.40.16 archive in our record; it is not a checksum for other releases or platforms.
The CLI’s strict code-signature check passed, but Gatekeeper reported that the valid code did not appear to be an app. We then assessed the official application bundle, which passed as a notarized application signed by Infra Technologies, Inc. The embedded CLI was byte-identical to the CLI we ran. No quarantine removal or security exception was used.
That distinction matters. A tool reporting an unexpected artifact type is not automatically evidence of a malicious binary, and a valid signature is not a complete vulnerability audit.
Nine graph dependencies were pinned and hashed: pyoxigraph 0.5.13, pyshacl 0.40.2, rdflib 7.6.0, owlrl 7.6.2, html5rdf 1.2.1, packaging 26.3, prettytable 3.18.0, pyparsing 3.3.3 and wcwidth 0.9.4. Preserve a hash-locked dependency file for a reproduction; version names alone do not reproduce the original verified install.
Isolate the local runtime
Our Q4 server bound to 127.0.0.1:11435. The UI used 127.0.0.1:8765. We kept the two model stores separate so that the GGUF experiment did not silently replace the original MLX build.
The following is an adapted configuration example, not the original launcher. Set the paths to your verified portable installation. It assumes the runtime and Q4 model are already provisioned.
PILOT_DIR="$PWD/local-wiki-pilot"
mkdir -p "$PILOT_DIR/isolated-home" "$PILOT_DIR/tmp" "$PILOT_DIR/q4-models"
env OLLAMA_HOST=127.0.0.1:11435 \
OLLAMA_NO_CLOUD=1 \
OLLAMA_MODELS="$PILOT_DIR/q4-models" \
OLLAMA_NUM_PARALLEL=1 \
OLLAMA_MAX_LOADED_MODELS=1 \
OLLAMA_MAX_QUEUE=2 \
OLLAMA_CONTEXT_LENGTH=2048 \
OLLAMA_KEEP_ALIVE=0 \
TMPDIR="$PILOT_DIR/tmp" \
"$PILOT_DIR/runtime/ollama" serveOur original launcher also supplied an isolated runtime home through a clean child-process environment. The portable runtime created local key material there on first startup. No key contents belong in a post, source bundle or request log.
Directory separation is not an operating-system sandbox or an air gap. Installation and model downloads still contacted external distribution services.
Bound requests and observe cleanup
Our conservative Q4 comparison used 2,048 context, batch 64, one request at a time, thinking disabled, up to 128 generated tokens and keep_alive=0. Later source-question answering retained 2K context with a 256-token output limit.
These are different phase configurations. Do not present them as one unchanged benchmark.
The Ollama chat API supports a JSON schema through format, and keep_alive controls post-request residency. Unloading after a response reduces idle residency; it does not eliminate the peak needed to load and run the model.
The original comparison required three normal baseline samples, stopped immediately at critical pressure, and stopped after three consecutive warning samples. The successful retry used a stricter first-warning stop. It retained the raw baseline samples that the earlier comparison had omitted.
A reproduction should record at least:
Stage | Evidence to retain |
|---|---|
Preflight | Pressure, swap, runtime identity, existing workloads |
Request | Exact payload, schema and selected model |
Inference | Pressure samples, timeout or stop reason, response |
Cleanup | Empty loaded-model list and owned process/listener checks |
Validation | Schema, source and graph validation results |
Terminate only processes whose identity and ownership you have verified. A PID copied from yesterday’s run is not a safe shutdown instruction.
Why tuning flags did not fix the MLX run
The pinned MLX implementation loaded/materialized tensors and initialized caches and structured-output machinery before becoming ready. Its context limit was a soft reported limit, and its prefill chunking differed from the llama.cpp path.
We did not treat generic batch or KV-cache controls as proven MLX memory controls. Likewise, the request’s CPU-thread setting was not a verified effective MLX cap. A configuration key appearing in a request does not establish that the selected backend honors it.
Validate a deliberately small graph
Before connecting real sources, we tested a fictional graph. Here is a simplified teaching fixture, not an export of the production ontology:
@prefix ex: <urn:wiki-demo:> .
ex:claim1 a ex:Claim ;
ex:subject ex:Ada ;
ex:predicate ex:worksFor ;
ex:object ex:Northstar ;
ex:evidence "Ada works for Northstar." ;
ex:reviewState "PROPOSED" .A minimal teaching shape can require an evidence string and constrain the review state:
@prefix ex: <urn:wiki-demo:> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
ex:ClaimShape a sh:NodeShape ;
sh:targetClass ex:Claim ;
sh:property [
sh:path ex:evidence ;
sh:minCount 1 ; sh:maxCount 1 ;
sh:datatype xsd:string
] ;
sh:property [
sh:path ex:reviewState ;
sh:minCount 1 ; sh:maxCount 1 ;
sh:in ("PROPOSED" "ACCEPTED" "REJECTED")
] .Save those examples as data.ttl and shapes.ttl. With pySHACL installed, the following illustrative call validates their structure:
from pyshacl import validate
conforms, report_graph, report_text = validate(
data_graph="data.ttl",
shacl_graph="shapes.ttl",
data_graph_format="turtle",
shacl_graph_format="turtle",
inference="none",
)
print(report_text)
if not conforms:
raise SystemExit(1)This example has not been executed as part of preparing this article. The original graph tests used the pilot’s fuller implementation and included nine passing synthetic checks.
SHACL validates conformance to supplied shapes. It does not prove that Ada exists, that the evidence is authentic, or that the relationship is true. Exact source matching and human review are separate controls.
Small operational errors worth retaining
A mode-0644 launcher failed when executed directly. Invoking the same script with /bin/sh corrected the launch without changing permissions.
A model-store inventory check also falsely treated a same-store symlink as a cross-store dependency. Correcting the assertion showed two separate stores with no shared content digests. Neither model was deleted, because neither had yet established a usable replacement.
These failures belong in the runbook. They distinguish packaging and test-harness defects from model defects.
Next: import actual sources and debug why the right document can still produce the wrong context.