Principal Engineer and independent researcher in the UK. AI agent security and authorization, evaluation reliability, formal verification, and the engineering behind them.
Background · CV · Contact
I build the checks that say an AI system is behaving. Then I try to break them, including my own.
Some of these are systems that keep an AI agent inside a limit; some are the evaluations that decide whether a model is any good. I build them, then attack the check itself, because a check that passes and a system that works are two different things.
Three checks that reported everything was fine
-
An AI agent was limited to 4 KB. It wrote 100 KB. Every test passed.
I built the permission boundary, then broke it. 66 of 66 tests passed, an independently written replay verifier read the decision ledger back and returned VERIFIED, and a checked invariant confirmed that no tool ran without an authorization decision. The part that failed was underneath all three: the component that described the request to the policy engine measured the payload as a string, and the component that carried it out coerced the same argument and wrote it.
-
A dashboard reported 1,026 actions prevented. All 1,026 had already executed.
Two arms of one testbed consumed the same 10,000 events, with the same evaluator, the same policy and the same injected faults. They differed only in position. The arm that observed alongside the action rather than standing in its path recorded intentions, and a record of intentions reads exactly like a record of outcomes.
-
The same failing evaluation reported a different number of completed samples when its scoring steps were listed in a different order.
Not a model error and not a scoring error: the bookkeeping a finished run keeps about itself disagreed with the run. Reported to the maintainers of the evaluation framework on 23 July 2026 and fixed by them on 29 July 2026.
And one that held.
A spending limit shared between AI agents cannot be exceeded, whatever order the requests arrive in. Proved rather than tested: the proof covers every interleaving, not the ones someone thought to try.
A grade says how far the evidence reaches, not how important the result is. These come from the fourteen projects on this site, each of which reproduces from a public clone, and they were found by one researcher who was looking for them. That is a selection effect: the claim is about mechanism, not measured frequency. The failure taxonomy sets out nine such mechanisms.
Selected findings
-
MEASURED Two AI judges scoring the same 600 items made mistakes that correlated positively in 171 of 171 pairs, at a mean phi of 0.567. The correlation between how often a pair agreed and how accurate it was fell from 0.824 to 0.077 once the part of agreement that is forced by accuracy was removed. A stacked arrangement, one judge reviewing another, caught fewer false passes rather than more.
Agreement still carries information. It just answers a different question from error correlation, and only the second is free to disagree with accuracy.
-
OBSERVED Tools filtered out of the list an AI agent is shown remained callable by name, in a server built to exhibit the failure.
-
MEASURED A training run's result re-derives bit for bit from its recorded inputs on one machine. Across different hardware it stays unknown.
Break one yourself
A sealed decision record. Edit the approval limit and the recomputed hash stops matching the seal, so the edit cannot be hidden. Every hash is computed in your browser; nothing leaves it.
omega-record/2026-05-14/cs-agent-refund-1187442
- Authority
- Sarah Okafor, Senior Customer Operations Manager (badge ops-2218), Tier C escalation. Autonomous limit £2,500.
- Evidence
- Order O-2026-04471 valued £4,820; three photographs; channel count and output gain inconsistent with the order specification; 30-day quality-assurance window applies.
- Expected outcome
- Full £4,820 refund to the original payment method, return collection booked, £120 goodwill credit, settlement in 3-5 business days.
- Confirmation
- Identity verified by SSO plus hardware key; a signed approval token was issued and consumed once at commit.
- Traceability
- Sealed with the record's SHA-256 content hash. Recomputing it shows whether the stored record still matches what was committed.
Stored seal e747c3fdcb29…
Recomputed recomputing…
Recomputing the hash in your browser…
Where the work sits
- Proof about code and protocols. Machine-checked, inside a stated frame.
- Limits on what an agent may do. Authority and resource bounds, and the attacks on them.
- Whether an evaluation measures what it claims. Judges, graders and the numbers they produce.
- Boundary audit of protocols. Where a declared surface and a reachable one differ.
- Evidence someone else can re-derive. Sealed records, and what they do not establish.
Start here to understand the pattern
Go deep for claims, code, proofs and limits