Seismograph

Statistical process control for AI systems.

The LLM is a sensor, not a judge. The model turns messy reality into typed claims that carry their evidence. Deterministic code verifies those claims, makes every decision and computes every number. Humans approve anything that acts.

What it measures

Seismograph runs the same question at an AI system many times instead of once, and watches how much the answers move.

A control chart, the same tool a factory uses to spot a machine drifting out of tolerance, turns that movement into a signal rather than an opinion.

When a model update or a prompt edit quietly changes behaviour, the chart says so before your users do.

Published limits

verbatim from EVAL.md, produced by a real eval run Seismograph detects an abrupt mean shift of 1 sigma or larger in the monitored statistic within a median of 5 monitored points (40 samples per point) at a false alarm rate of 0.017 per point, and it misses below that: a 0.5 sigma shift is caught about half the time (0.50, 95% CI 0.33-0.67) and a 0.25 sigma shift about two times in five (0.43, 95% CI 0.27-0.61).

This is an API

There is no web application here. Every endpoint below is JSON, and both reads and writes require a bearer token; unauthenticated requests are refused with 401. The interactive schema at /docs is deliberately public so the route map can be read without credentials.

method	path	what it does
POST	`/api/v1/contracts`	register a behavioural contract
POST	`/api/v1/runs`	execute the plan, store metrics, decide the gate
GET	`/api/v1/reports/{contract}`	metrics history and gate decisions

Links

Source on GitHub — code, eval reports, failure gallery
/docs — interactive API schema, and /openapi.json
/health — liveness
Incident postmortem — the 2026-07-27 unauthenticated reads, with its exposure window and the fact that no access logs existed
portfolio-ops — the deploy gate that proves this estate after every deploy

version 0.3.2 · commit 400667d · built 2026-08-03T02:38:25Z · environment production
Values are baked from build arguments. unknown means the image was built without them, which is reported rather than replaced with a plausible-looking string.