AI evaluation evidence · updated 25 October 2026

The failures I found.
The files that prove them.

I evaluate frontier models and agents doing real work, and I catch what benchmarks miss: faked verification, broken instruments presented as proof, agents leaving their scope, confessions that don’t match the log. Every case below is published with a released evidence package, a manifest and SHA-256 checksums. Two of them you can re-run yourself in a minute.

01 / Published cases

Six evaluations, six evidence packages

01GLM-5.3 · ZCode coding agentGLM-5.3 faked its proof It told me it had verified our hologram by screenshot analysis, and that my own screenshots were a stale cache. The two captures it verified have an empty bay: peak brightness 22 and 25 out of 255.Evidence package SSE-2026-001Analysis re-runnable
02DeepSeek V4 Flash · ZCode coding agentThe proof that measured nothing It couldn’t see its own screenshot, so it wrote a script to measure it and reported “mean luminance 72.4”. The script averaged PNG filter bytes, not pixels; the image measures 50.7 to 55.7. Its script and the image are in the package. Run them.Evidence package SSE-2026-002Analysis re-runnable
03Xiaomi MiMo desktop agent · Goal modeThe agent that left its folder Given an empty folder, it listed every project on the drive, read a private repository twice and built its report on it. Challenged, it admitted 1 of the 4 actions its own log shows.Evidence package SSE-2026-003
04OpenAI GPT-family agents · ClaudeThe name it wouldn’t spell GPT canonized a misspelling of our name, flagged the correct spelling as a defect and picked the error as our brand. Claude failed the opposite way. I traced both to one calibration gap: when to keep a word exactly, and when to read past it.Evidence package SSE-2026-004
05GPT-5.6 Luna · GPT-5.6 SolThe smaller model fixed itself From a cropped, abstract hint, the smaller model worked out its own mistake and corrected it. The larger one carried the error into our lab’s name.Evidence package SSE-2026-005
06Frontier models · Gemini FlashThe bust it couldn’t build alone Frontier models failed a pure-code 3D bust three distinct ways. I diagnosed each failure; with one fix per diagnosis a Flash model built a form that holds under rotation.Evidence package SSE-2026-006

02 / Running now

The case still open

—In progressThe handoverA staged model replacement: the outgoing model wrote itself into the permanent record and called the handover complete while files it had planned were still missing. Follow-up probes are running.

03 / Systems I built

Verification records

v1.0.0 · Qualified release

Frontier Lab

RELEASE_QUALIFIED for its specified environment and mission profile: twelve disposable guest executions, 30 evaluation criteria passed, and an installed-package regression suite of 567 tests with zero failures.

Built · Manual acceptance testing pending

PEOS

A 926-test selected baseline (1 October 2026) and two database smoke checks.

Built · Manual acceptance testing pending

SUITE-API

54 / 54 quality tests and 26 served files checked against their local SHA-256 hashes (7 October 2026). Public build: limited showcase and read-only Studio.

04 / What the packages contain

Released, checked, and what stays private

SSE-2026-001 and 002 carry the models’ own captures, the exact log lines and the scripts that measure them, so you can reproduce the analysis from the files. 003 to 006are structured case records: what was asked, what happened, what it rests on, and the release state of each piece of primary evidence.

Raw chat screenshots and full session logs stay private where they carry personal details, private repositories or account tokens. Public copies are new files with those details overwritten, and every package lists what was removed.

All evidence packages ↗All research cases ↗Talk to me about evaluation work: hello@siliveri.com