Runtime validation, database integration, model evaluation, and the operational evidence still needed before partner release.
Building AI-SGAP now has 652 subscribers. For those who have followed the journey from the assessment workbook to executable workflows, this update goes deeper into the engineering.
Our last article focused on what AWS staging was teaching us about execution discipline. Since then, the work has expanded across container validation, PostgreSQL integration, model-assisted analysis, customer workflows, and maintaining the evidence needed to explain each result.
The central question has become more precise:
Can we connect a reported result to the exact software, inputs, environment, and execution that produced it?
This article draws from our wiki and build, risk, access, and recovery records through November 14. It describes development evidence—not production readiness or completed partner release.
1. Container validation now involves several independent questions
A container that builds successfully has cleared one hurdle. We still need to establish whether it contains the intended application, has the required runtime dependencies, and satisfies the relevant security checks.
Our work has included:
- Python runtime rebuilds and headless-runtime compatibility checks.
- ELF linkage validation to inspect native executable and shared-library dependencies.
- Source-to-image checks to verify that the intended application files are present in the image.
- SHA-256 manifests to identify and verify retained artifacts.
- Software bills of materials, or SBOMs, to document discovered components.
- Trivy and Grype scans to examine reported vulnerabilities.
These checks address different failure modes.
- Application-file integrity does not establish that native libraries resolve correctly.
- Successful linkage does not establish that the application workflow succeeds.
- A vulnerability scan does not establish authorization behavior.
- A working application does not resolve outstanding vulnerability findings.
We have also learned to treat a rebuilt image as a new artifact. An earlier test or risk exception must be checked against the new bytes and its permitted scope.
The practical improvement: our review increasingly asks which checks apply to the specific image, rather than whether an image with a familiar name passed previously.
2. Database integration has moved beyond simulated storage
Test doubles helped us develop and exercise application behavior before all infrastructure was available. They remain useful, but they cannot answer every database question.
We have progressed into bounded integration checks against disposable native PostgreSQL environments.
That distinction matters because an actual database can expose assumptions about:
- SQL behavior and schema compatibility.
- Transaction boundaries.
- Connection configuration.
- Database permissions.
- Persistence across application operations.
Later local packages recorded 52/52, 57/57, and 76/76 checks, with a separate 9/9 replay suite.
Those figures describe separate packages and test scopes. They are not additive coverage, and they do not establish complete hosted acceptance.
Our reporting now needs to distinguish:
| Validation level | What it helps establish |
|---|---|
| Syntax and import checks | The code can be parsed and loaded in the tested environment |
| Mocked tests | Behavior matches expectations under controlled simulations |
| Native database integration | Tested operations behave against an actual database |
| Browser walkthrough | The user can complete the tested interaction |
| Hosted validation | The assembled workflow behaves under the intended cloud conditions |
The outstanding work includes completing the relevant browser and hosted checks and making separate adoption decisions for the tested candidates.
3. Model evaluation includes several kinds of correctness
Our model strategy remains Sonnet as the baseline, Flash as the challenger, and Bedrock as the intended AWS integration path.
The engineering challenge extends beyond obtaining a response.
We are evaluating several distinct properties:
- Transport: Did the request complete, fail, or leave an uncertain outcome?
- Structure: Does the response conform to the expected format?
- Coverage: Did it address the expected questions?
- Citation: Does the cited material exist and support the quoted passage?
- Semantics: Does the interpretation follow from the evidence?
- Operations: Can we account for usage, failures, and the relevant model identity?
These properties can disagree.
A response can contain valid JSON, cover every requested question, and quote a source exactly—while still drawing an unsupported conclusion.
That is why semantic review remains a separate requirement. Our records retain outstanding semantic dispositions rather than treating technical completion as accepted analysis.
The lesson: automated validation can establish useful properties of an answer without establishing that the answer is correct.
4. Cloud integration introduces boundaries that local tests cannot prove
AWS staging has forced us to examine the operational dependencies around the application.
The work and remaining checks span:
- Identity and access management.
- Database connectivity using the intended identity and TLS configuration.
- Encrypted object storage and key access.
- Application load-balancer identity-token verification.
- Model destination and logging configuration.
- Provider usage and cost attribution.
- The complete hosted customer and assessor workflow.
The status of each must remain explicit.
For example, a successful local database test cannot establish the deployed task’s permissions. A successful identity check cannot establish access to every downstream service. An offline provider-adapter test cannot establish live model availability or output quality.
We are also keeping model fallback and proposed local/cloud deployment options separate from demonstrated capability. They remain paths to qualify.
5. The knowledge wiki has become an engineering reference
As the build expanded, finding the correct source became a problem of its own.
We now maintain a wiki that connects:
- Product and technical specifications.
- Architecture and evidence-engine notes.
- Deployment and governance status.
- Open decisions.
- Build history.
- Risk, access, and recovery records.
The purpose is to make decisions and their supporting evidence easier to locate without turning every retained document into current authority.
That matters when two documents disagree.
A passage may accurately describe an earlier version. Another may describe a later intended change. Neither fact alone establishes what the executable product currently implements.
The wiki therefore needs to retain version, date, scope, and unresolved status alongside the content.
6. A separate local knowledge pilot exposed a useful failure
Alongside the product work, we have been experimenting with a separate local knowledge pilot. It is outside the accepted AI-SGAP product.
Its recorded snapshot contains:
- 26 sources.
- 5,541 section-aware chunks.
- 29,247 metadata and provenance triples.
These counts describe the indexed material and its structural relationships. They do not represent validated security conclusions.
One investigation was particularly instructive:
- A question failed to retrieve the most relevant source within the displayed results.
- We reproduced the retrieval problem.
- A bounded correction brought the relevant source to the top.
- The combined model answer still abstained.
The search repair worked within its tested scope. The answer problem remained unresolved.
That gives us two separate evaluation targets:
- Retrieval quality: Did the system locate the right evidence?
- Answer quality: Did it use that evidence correctly and helpfully?
Improving the first did not automatically solve the second.
7. The build ledger records failures as well as progress
The Command and Build Register now contains 18,399 records.
That number needs context. It is a record count, not a count of successful commands or completed features.
Entries distinguish:
- Proposed work.
- Confirmed execution.
- Failed attempts.
- Superseded instructions or results.
- Work whose execution remains unknown.
The ledger helps answer practical questions:
- Which command was actually executed?
- From which environment?
- Against which artifact?
- What failed?
- Which correction followed?
- What evidence supports the reported result?
Versioned launchers and prerequisite checks are becoming equally important. A handoff needs to carry enough context for another person to execute it without reconstructing assumptions from a long conversation.
We have made mistakes here, including incomplete runners and repeated setup assumptions. Recording those mistakes helps preserve history; prevention still needs verification in the intended environment.
8. Risk and recovery records keep unfinished work visible
Our other ledgers serve different purposes.
- The risk ledger tracks findings, treatment status, exceptions, and unresolved ownership.
- The RBAC ledger records access-related work and the limits of authorization testing.
- The BCDR ledger tracks backup and recovery evidence, dependencies, and open exercises.
They also expose gaps that successful component tests can leave untouched.
At the reviewed checkpoint:
- Full governance remains blocked by historical evidence gaps.
- Complete hosted validation remains outstanding.
- Semantic review still has unresolved work.
- Recovery has not been demonstrated.
- Measured recovery point and recovery time objectives—RPO and RTO—have not been established.
- Separate adoption and release decisions remain open.
Historical snapshot availability is useful evidence. It does not demonstrate a successful restore.
Similarly, a recovery procedure on paper is a plan until an exercise establishes what can actually be recovered.
What the next milestone needs to establish
The next useful milestone must bring these workstreams together.
We need evidence that:
- The intended application and runtime are the ones being tested.
- The database, storage, identity, and application configurations work together.
- Customers and assessors can complete the relevant workflow.
- Model assistance is useful within its reviewed scope.
- Failures and uncertain outcomes remain visible.
- Recovery preserves the necessary application and evidence state.
- Another engineer can reproduce the result from the handoff.
The customer problem remains the reason for doing this work: organizations need a clearer connection between AI security evidence, unresolved gaps, and corrective action.
The engineering beneath that experience must be reviewable too.
For the engineers following this series: which integration boundary has exposed the most assumptions in your own builds?
Building AI-SGAP shares development progress and engineering lessons without disclosing proprietary assessment logic or implementation mechanisms. Development results do not establish customer outcomes, certify compliance, or guarantee production readiness.
Explore the series and beta-partner information at cloudgal42.com/ai-sgap/.