Dig notes from the Sandstone layer
AgentAudit
A weekend prototype, not a production system. It asks one question: what do you have to keep to defend an AI agent’s financial answer later? The final sentence the model writes isn’t enough.
- Stack
- Java 25, Spring Boot, Spring AI, Ollama (llama3.2:3b), PostgreSQL 17, OpenTelemetry
- When
- October 2026
- Tests
- 23, on in-memory H2 with a stub chat model
The problem
An agent answers a portfolio-risk question in fluent prose. Later someone asks why. The prose can’t tell you which tool it called, with what arguments, or what came back.
In the demo, a question about a portfolio that doesn’t exist came back as HTTP 200 with an apology in the answer text. The audit record showed what really happened: the getVaR tool threw an IllegalArgumentException.
How it fits together
The controller hands the question to the agent, which calls Spring AI’s ChatClient against a local Ollama model. The model picks a tool; the tool layer records the call; a plain Java service does every calculation over mock data.
One rule holds the design together: the model chooses tools and phrases answers. It never calculates.
Decisions, and what each one cost
| Decision | Why | Cost |
|---|---|---|
| Three identifiers: client session, audit session, trace | A caller can reuse a session label; each execution still gets its own audit record, linked to its OpenTelemetry trace | Three IDs to explain instead of one |
| Audit record kept apart from tracing | Traces are sampled and expire. The audit record stores every tool’s arguments and return value, for every request, long-term | Two stores to keep consistent |
| Each audit write in its own transaction (REQUIRES_NEW) | A run that fails still leaves its record behind | More transactions per request |
| Order events by a sequence number, not timestamps | Timestamps can tie or drift | One more column to maintain |
| Session ID carried to tools in a ThreadLocal, cleared in finally | Tools get the audit session without it leaking into their signatures | Only safe while a run stays on one thread |
| Read-only API: no endpoint can amend a record | An audit trail you can edit isn’t one | Corrections need a new run |
What the record caught
- At temperature 0, the model said EUR 12M was higher than EUR 14M, every time. The record showed the tools had returned the right figures, so the fault was in the narration, not the data.
- The fix was to move the comparison into Java (compareVaR) and return the tool’s result directly, skipping the model’s second pass.
- I also tried fixing it with a prompt. Two arms, three runs each. The baseline got the figures wrong. The added rule got the figures right, still didn’t answer the question, and triggered an extra tool round. The HTTP responses looked the same; only the audit trail showed the difference.
- A reflection-based test guards the direct-return annotations. It caught a misplaced annotation three times.
What I’d do next
TODO: The README doesn’t list next steps. Add two or three of your own here.