I wanted a local AI assistant that could answer questions about my work without turning my development machine into a dedicated inference server. I already had a knowledge wiki. The next step was to connect a local model to versioned sources, preserve their relationships, and make answers traceable.
The machine was an Apple M4 with 24 GiB of unified memory. The starting model was Qwen3.5 4B through Ollama. The knowledge layer used RDF/OWL, embedded Oxigraph and SHACL validation.
The first lesson arrived before a single answer: a small model download does not establish that a model will coexist comfortably with your current workload.
This series follows the build as it develops. These results cover 10–11 November 2026. The system is still a pilot; several harder questions remain unresolved.
The architecture I actually built
The knowledge lives in source documents. Qwen does not acquire their contents through training. At inference time, the application retrieves a bounded selection of source text and asks the model to work within it.
There are three distinct kinds of data:
Layer | Contents | Authority |
|---|---|---|
Frozen source snapshot | Original Markdown and imported document evidence, revisions and hashes | Evidence for the local snapshot |
Provenance index | Document, chunk and link metadata in Oxigraph | Location and identity of evidence |
Reviewed records | Explicitly accepted source-linked statements | Narrow reviewed queries, subject to freshness and conflict checks |
A fourth component is a separate fictional graph used for tests. Mixing its demo facts into real answers would be a defect.
This is a retrieval-based assistant with a graph-backed provenance layer and a separate reviewed-record path. Calling the entire system autonomous knowledge-graph reasoning would overstate what it does.
Record the actual runtime
A model name alone is not enough to reproduce a run.
Component | Recorded pilot version or identity |
|---|---|
Hardware | Apple M4, 10 cores, 24 GiB unified memory |
Operating system | macOS 26.6.2, build 25G83 |
Python and environment tool | Python 3.15.0; uv 0.13.1 |
Runtime | Portable Ollama 0.40.16 |
First model | qwen3.5:4b; MLX, safetensors, nvfp4; API parameter size 4.5B |
Comparison model | qwen3.5:4b-q4_K_M; GGUF, llama.cpp runner |
Graph packages | pyoxigraph 0.5.13; pyshacl 0.40.2; rdflib 7.6.0 |
These are recorded build identities, not a claim that every listed version is the current recommended release. A reader should retain the actual resolved artifacts, package hashes and model manifest used for their own run.
The “4B” model label and the API’s “4.5B” parameter-size string both appeared in our record. We kept both rather than silently making them agree.
Measure the starting machine
Our initial inspection found approximately 6,067.88 MiB of swap already in use. That matters: a model experiment begins with the machine’s existing workload, not an empty benchmark machine.
The following individual read-only checks reproduce the kinds of observations we collected on macOS. They are separated here for readability, rather than presented as a verbatim historical shell session.
sw_vers
sysctl -n machdep.cpu.brand_string
sysctl -n hw.memsize
sysctl -n hw.ncpu
sysctl vm.swapusage
/usr/sbin/sysctl -n kern.memorystatus_vm_pressure_level
/usr/bin/memory_pressure -QThe pressure tool’s “memory free percentage” is a pressure-related indicator. We did not equate it with literal free physical RAM. Similarly, Ollama’s reported allocation is not an independently measured process RSS.
Initial process/hardware checks encountered “Operation not permitted.” A narrower read-only retry succeeded. We did not disable operating-system protections to make the inspection work.
What failed
Two MLX attempts stopped under memory pressure before completing inference. The first Q4 comparison also stopped: approximately 3.073 seconds into its first request, the three-consecutive-warning-sample guard fired.
At that point Docker still had 15 running containers. We preserved the development environment and stopped the pilot. There was no completed Q4 answer, no extraction-quality result and no valid generation-speed measurement from that attempt.
The comparison wrapper exited successfully because it completed its stop-and-record procedure. That exit status did not mean the model experiment passed. An HTTP 500 observed during cancellation was also not evidence of an independent model crash.
What subsequently worked
On 11 November I explicitly quit Docker Desktop before a new bounded experiment. Four small synthetic cases completed:
Case | Recorded API wall time | Result |
|---|---|---|
Positive extraction, first call | 5.055 seconds | Expected facts and exact evidence |
Positive extraction, repeat | 3.536 seconds | Expected facts and exact evidence |
Negated statement | 1.478 seconds | No claims |
Unstated fact | 1.652 seconds | No claims |
The positive calls generated at about 31.3 tokens per second. All 26 retained memory samples showed normal pressure and zero swap. Each response was followed by verified unloading.
This does not isolate Docker as the cause of the earlier failures. Swap was already zero before Docker quit, and other workload conditions had changed. It establishes that this particular small workload succeeded under the later recorded conditions.
Separate graph cost from inference cost
A tiny graph-only check validated 26 triples in approximately 0.099 seconds, with the Python process peaking at about 66.1 MiB RSS. That does not predict large-graph performance. It does show that the immediate pressure problem was not explained by this small graph operation.
The next experiment should measure load time, prompt evaluation, generation, validation and cleanup separately. A single “request duration” hides where the resource cost occurs.
For a reproduction, begin with positive, repeated-positive, negated and unknown synthetic cases. Save baseline samples and failed attempts. Do not start by importing an entire private corpus or assuming that a successful model load establishes useful answers.
Next: isolate the runtime, bound each request, and validate a graph before allowing model output into it.