ddmin, not log grep
Logs correlate with failure; delta debugging isolates it. Reducing the configuration until only the failing part remains converts a hypothesis into a measured result.
The Fault Finder · room 02 of 11
AgentXRay
Something broke your AI agent. Let's find exactly what.
AgentXRay adapts Andreas Zeller's Delta Debugging to tool-calling AI agents: when an agent fails, it isolates the minimal failure-inducing instruction, tool schema or environment flag — then proves the isolation with counterfactual verification.
It ships as a Python SDK (`agentxray.diagnose`), a POSIX CLI with strict exit codes, and an interactive web product with a client-side ddmin sandbox.
the mechanism — watch a configuration shed parts
instruction: be tersetool schema: web_searchtool schema: read_fileinstruction: always retry onceenvironment: DEBUG=1context: full chat historyinstruction: cite sourcestool schema: run_shell
an illustrative configuration — the SDK runs this reduction against the real one
The actual problem
A failing AI agent is a configuration space problem: the prompt has instructions, the tools have schemas, the environment has flags — and any combination of them can be the one that breaks the run. Reading logs narrows the suspicion list but rarely names the culprit.
Brute-forcing every configuration is expensive: each test costs an evaluation call. The engineering problem is to remove as much of the configuration as possible, as few calls as possible, and still be able to prove that what remains is really the cause.
The exploded machine
Delta Debugging treats the agent's configuration as a set of parts and systematically sheds them: each pass tests a reduced configuration and keeps the halves that still fail.
Prompt-card deck — the configuration space
Instructions, tool schemas, environment flags and context blocks are laid out as removable parts of one configuration.Splitter — ddmin passes
Delta Debugging halves the configuration and tests each reduction; failing halves are kept, passing halves are shed.Test cell — red/green lamps
Every reduced configuration runs against the hermetic testbeds — under 15 ms and $0.00 per call, so passes are cheap.1-minimality proof stamp — counterfactual verification
The surviving part is swapped back in and the failure must return: the isolation is proven, not guessed.Terminal-tape reel — the POSIX CLI
Strict exit codes turn the minimiser into a pipeline component — CI can fail a change the tool proved breaks the agent.HF Spaces kiosk — the live demo
A Hugging Face Space runs the sandbox publicly; the web product carries a client-side ddmin sandbox and a terminal tape.
When reduction converges, the result is put through counterfactual verification — the suspected part is swapped back in and the failure must return — establishing 1-minimality rather than a plausible guess.
Everything runs against 5 hermetic offline testbeds (<15 ms per run, $0.00 spend), so diagnosis never burns API budget, and the CLI reports through strict exit codes so pipelines can branch on the outcome.
Engineering decisions
Counterfactual proof, not confidence
A minimised failing case only counts when re-adding the part reproduces the failure — 1-minimality is verified, never assumed.
Hermetic testbeds
Five offline testbeds run in under 15 milliseconds at zero cost, making thousands of reduction calls practical instead of prohibitive.
CLI first
Strict exit codes make the minimiser a pipeline component — CI can fail a change because AgentXRay proved it breaks the agent.
Proof — verified results
- Test suite
- 640 testsrepo
- Offline testbeds
- 5 hermetic (<15 ms, $0.00 spend)repo
- Fewer evaluation calls vs linear scans
- reported 40–70%resume-sourced
- Guarantee established
- 1-minimality via counterfactual verificationrepo
- Interfaces
- Python SDK · POSIX CLI · interactive web sandbox · HF Spaces demorepo
every number here carries its source — my resume's figures and the synthetic-fixture results are marked for what they are