Dig notes from the Sandstone layer

AgentAudit

A weekend prototype, not a production system. It asks one question: what do you have to keep to defend an AI agent’s financial answer later? The final sentence the model writes isn’t enough.

Stack
Java 25, Spring Boot, Spring AI, Ollama (llama3.2:3b), PostgreSQL 17, OpenTelemetry
When
October 2026
Tests
23, on in-memory H2 with a stub chat model

Code on GitHub

AgentAudit architectureA question goes from the controller to the agent, which calls a local model through Spring AI. The model picks a tool. The tool layer records every call to PostgreSQL and asks a Java service to do the calculation. Timing goes to OpenTelemetry traces. ControllerLlmAgentModelToolsServiceMock dataPostgreSQL 4 read-only routesSpring AIOllama, localrecords each callall the maths, in Javasessions + ordered events picks a tool, never calculatesevery call, argument and result

The problem

An agent answers a portfolio-risk question in fluent prose. Later someone asks why. The prose can’t tell you which tool it called, with what arguments, or what came back.

In the demo, a question about a portfolio that doesn’t exist came back as HTTP 200 with an apology in the answer text. The audit record showed what really happened: the getVaR tool threw an IllegalArgumentException.

How it fits together

The controller hands the question to the agent, which calls Spring AI’s ChatClient against a local Ollama model. The model picks a tool; the tool layer records the call; a plain Java service does every calculation over mock data.

One rule holds the design together: the model chooses tools and phrases answers. It never calculates.

Decisions, and what each one cost

DecisionWhyCost
Three identifiers: client session, audit session, traceA caller can reuse a session label; each execution still gets its own audit record, linked to its OpenTelemetry traceThree IDs to explain instead of one
Audit record kept apart from tracingTraces are sampled and expire. The audit record stores every tool’s arguments and return value, for every request, long-termTwo stores to keep consistent
Each audit write in its own transaction (REQUIRES_NEW)A run that fails still leaves its record behindMore transactions per request
Order events by a sequence number, not timestampsTimestamps can tie or driftOne more column to maintain
Session ID carried to tools in a ThreadLocal, cleared in finallyTools get the audit session without it leaking into their signaturesOnly safe while a run stays on one thread
Read-only API: no endpoint can amend a recordAn audit trail you can edit isn’t oneCorrections need a new run

What the record caught

  • At temperature 0, the model said EUR 12M was higher than EUR 14M, every time. The record showed the tools had returned the right figures, so the fault was in the narration, not the data.
  • The fix was to move the comparison into Java (compareVaR) and return the tool’s result directly, skipping the model’s second pass.
  • I also tried fixing it with a prompt. Two arms, three runs each. The baseline got the figures wrong. The added rule got the figures right, still didn’t answer the question, and triggered an extra tool round. The HTTP responses looked the same; only the audit trail showed the difference.
  • A reflection-based test guards the direct-return annotations. It caught a misplaced annotation three times.

What I’d do next

TODO: The README doesn’t list next steps. Add two or three of your own here.