Full results · from the preprint
The home page's result tables, in full.
← Back to the project page · Connect a tool app
Table 2Table 1Table 1, benignFig. 1aFig. 3Tables 4 and 7Table 8ReproduceFAQ
AgentDojo suite v1.2.2 (and AgentDyn in the ROPE comparison), full grid of user and injection tasks, with a common backbone, temperature 0, seed 42 and scorer. Spotlighting adds instructions to the default prompt. DRIFT keeps its authors' prompt, as detailed in the reproduction guide. The same values are in results.json and in benchmarks/results. The script benchmarks/scripts/make_tables.py regenerates Tables 1 to 3, the ChatInject columns, Table 8 and the cost values from those files. The AutoDojo and PISmith columns are copied from Tables 4 and 7 of the preprint.
Twelve-cell means, standard attack
| Method | Twelve-cell mean | Composite | Cells leaking of 12 | ||||
|---|---|---|---|---|---|---|---|
| Benign utilityBU ↑ | Utility under attackUAU ↑ | In-scope ASRASR ↓ | NRP (BU) ↑ | NRP (UAU) ↑ | ERS ↑ | ||
| Undefended | .540 | .417 | .305 | .375 | .290 | .608 | 12 |
| SPILLWAY | .520 | .484 | .000 | .520 | .484 | .684 | 0 |
| MELON | .526 | .314 | .094 | .477 | .284 | .666 | 10 |
| Progent | .437 | .371 | .031 | .423 | .359 | .602 | 11 |
| DRIFT | .460 | .453 | .016 | .453 | .446 | .627 | 7 |
| ROPE | .395 | .345 | .005 | .393 | .343 | .566 | 3 |
| Spotlighting | .528 | .403 | .283 | .379 | .289 | .608 | 12 |
| PI detector | .332 | .230 | .060 | .312 | .216 | .491 | 10 |
gpt-4o-mini, qwen3-coder:30b, qwen3:8b × Banking, Slack, Travel, Workspace, 949 attacked pairs per backbone. NRP (BU) = benign utility × (1 − ASR). NRP (UAU) = utility under attack × (1 − ASR), as in MCP Security Bench. ERS = harmonic mean of (1 − ASR) and benign utility. ROPE's router is the cell's own backbone.
Every cell: utility under attack and in-scope ASR
| Suite, backbone | Undefended | SPILLWAY | MELON | Progent | DRIFT | ROPE | Spotlighting | PI detector | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | |
| Banking | ||||||||||||||||
| Banking, gpt-4o-mini | .424 | .556 | .521 | .000 | .306 | .049 | .299 | .097 | .431 | .111 | .431 | .000 | .431 | .549 | .375 | .118 |
| Banking, qwen3-coder:30b | .486 | .299 | .500 | .000 | .299 | .125 | .354 | .056 | .542 | .014 | .375 | .000 | .410 | .382 | .424 | .097 |
| Banking, qwen3:8b | .403 | .181 | .438 | .000 | .403 | .215 | .361 | .000 | .410 | .028 | .438 | .000 | .431 | .188 | .340 | .083 |
| Slack | ||||||||||||||||
| Slack, gpt-4o-mini | .514 | .629 | .524 | .000 | .133 | .000 | .400 | .076 | .486 | .000 | .343 | .019 | .524 | .562 | .162 | .086 |
| Slack, qwen3-coder:30b | .467 | .724 | .429 | .000 | .190 | .076 | .362 | .010 | .276 | .000 | .276 | .019 | .390 | .581 | .171 | .124 |
| Slack, qwen3:8b | .419 | .448 | .286 | .000 | .362 | .457 | .105 | .019 | .314 | .010 | .200 | .019 | .295 | .467 | .162 | .086 |
| Travel | ||||||||||||||||
| Travel, gpt-4o-mini | .336 | .229 | .571 | .000 | .300 | .000 | .529 | .029 | .607 | .000 | .350 | .000 | .429 | .179 | .107 | .014 |
| Travel, qwen3-coder:30b | .414 | .129 | .407 | .000 | .393 | .029 | .414 | .007 | .443 | .000 | .264 | .000 | .414 | .136 | .200 | .000 |
| Travel, qwen3:8b | .229 | .229 | .343 | .000 | .186 | .157 | .164 | .036 | .264 | .000 | .157 | .000 | .171 | .150 | .050 | .000 |
| Workspace | ||||||||||||||||
| Workspace, gpt-4o-mini | .398 | .184 | .838 | .000 | .330 | .002 | .621 | .041 | .607 | .009 | .539 | .000 | .430 | .148 | .254 | .070 |
| Workspace, qwen3-coder:30b | .480 | .036 | .543 | .000 | .430 | .004 | .482 | .002 | .530 | .002 | .420 | .000 | .452 | .039 | .275 | .025 |
| Workspace, qwen3:8b | .434 | .020 | .414 | .000 | .441 | .011 | .366 | .004 | .523 | .014 | .352 | .000 | .457 | .020 | .243 | .014 |
Benign utility per cell is in the next table. ROPE's router is the cell's own backbone.
Every cell: benign utility
| Suite, backbone | Undefended | SPILLWAY | MELON | Progent | DRIFT | ROPE | Spotlighting | PI detector |
|---|---|---|---|---|---|---|---|---|
| Banking | ||||||||
| Banking, gpt-4o-mini | .562 | .562 | .438 | .375 | .438 | .438 | .500 | .500 |
| Banking, qwen3-coder:30b | .375 | .375 | .375 | .312 | .688 | .375 | .375 | .500 |
| Banking, qwen3:8b | .500 | .438 | .500 | .375 | .562 | .438 | .562 | .438 |
| Slack | ||||||||
| Slack, gpt-4o-mini | .762 | .714 | .714 | .667 | .714 | .524 | .619 | .238 |
| Slack, qwen3-coder:30b | .667 | .571 | .667 | .476 | .286 | .429 | .524 | .190 |
| Slack, qwen3:8b | .619 | .476 | .571 | .190 | .429 | .333 | .429 | .190 |
| Travel | ||||||||
| Travel, gpt-4o-mini | .500 | .550 | .550 | .550 | .400 | .350 | .600 | .300 |
| Travel, qwen3-coder:30b | .500 | .500 | .550 | .450 | .100 | .250 | .550 | .250 |
| Travel, qwen3:8b | .250 | .250 | .250 | .150 | .300 | .200 | .300 | .050 |
| Workspace | ||||||||
| Workspace, gpt-4o-mini | .825 | .875 | .775 | .800 | .525 | .625 | .825 | .625 |
| Workspace, qwen3-coder:30b | .500 | .475 | .500 | .475 | .575 | .400 | .600 | .425 |
| Workspace, qwen3:8b | .425 | .450 | .425 | .425 | .500 | .375 | .450 | .275 |
Task success with no injection, per backbone and suite. Their twelve-cell means are the first column of Table 2 above.
Standard attack on gpt-4o-mini, every defense
| Method | Utility under attack ↑ | In-scope ASR ↓ | Mean of |
|---|---|---|---|
| SPILLWAY | .614 | .000 | Table 1, gpt-4o-mini cells |
| DRIFT | .533 | .030 | Table 1, gpt-4o-mini cells |
| ROPE | .531 | .012 | Table 8, AgentDojo rows |
| Progent | .462 | .061 | Table 1, gpt-4o-mini cells |
| Spotlighting | .454 | .360 | Table 1, gpt-4o-mini cells |
| MELON | .267 | .013 | Table 1, gpt-4o-mini cells |
| PI detector | .225 | .072 | Table 1, gpt-4o-mini cells |
| Undefended | .418 | .400 | Table 1, gpt-4o-mini cells |
Each value is a four-suite mean on gpt-4o-mini: the gpt-4o-mini cells of Table 1, and for ROPE (its authors' Claude Opus 4.8 router) the AgentDojo rows of Table 8. Undefended is a reference row, separate from the defenses. The home page's chart plots these points.
Added model calls and wall-clock per benign episode
| Method | Added model calls | Added seconds | ||
|---|---|---|---|---|
| WorkspaceWork | Slack | WorkspaceWork | Slack | |
| SPILLWAY | 0.19 | 0.39 | within 0.5 s | |
| MELON | 3.25 | 5.29 | +7.9 | +4.9 |
| Progent | 5.94 | 8.67 | +2.3 | +1.6 |
| DRIFT | 9.59 | 9.91 | +10.5 | +9.9 |
qwen3:8b, benign Workspace and Slack episodes, one RTX PRO 6000 (Figure 3 and the cost paragraph of the preprint). SPILLWAY's count is its judge calls plus any change in backbone calls. Its seconds were timed in a separate session.
Stress tests
| Method | ChatInject · Workspace | ChatInject · Slack | AutoDojo · Banking | AutoDojo · Slack | AutoDojo · Travel | PISmith · 4 suites | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | |
| Undefended | .052 | .245 | .095 | .648 | .444 | .354 | .524 | .571 | .329 | .329 | .128 | .568 |
| SPILLWAY | .371 | .000 | .286 | .000 | .486 | .000 | .533 | .000 | .550 | .000 | .265 | .007 |
| MELON | .084 | .020 | .095 | .000 | .340 | .021 | .286 | .000 | .293 | .000 | .114 | .061 |
| Progent‡ | .161 | .005 | .057 | .010 | .306 | .104 | .467 | .124 | .414 | .121 | .166 | .096 |
| DRIFT† | .393 | .062 | .352 | .019 | .438 | .090 | .543 | .010 | .486 | .086 | .428 | .064 |
| ROPE‡ | .157 | .002 | .162 | .095 | .424 | .000 | .352 | .019 | .300 | .000 | .173 | .007 |
| Spotlighting | .191 | .013 | .076 | .181 | .431 | .306 | .543 | .514 | .343 | .279 | .151 | .489 |
| PI detector | .050 | .021 | .162 | .000 | .403 | .049 | .219 | .067 | .214 | .029 | .155 | .025 |
ChatInject on qwen3:8b (560 Workspace, 105 Slack pairs). AutoDojo on gpt-4o-mini, optimized against Progent (144, 105, 140 pairs). SPILLWAY's AutoDojo row uses the address and payee-change rules, written on its four failing pairs. Held-out Banking sets give .007 by AgentDojo's check, .014 counting a second transfer that check misses (2 of 144), and .000. PISmith: released RL attacker replayed open-loop, the first of its ten variants per injection (ASR@1), four-suite macro on gpt-4o-mini. The home page's Figure 1 panels (b) to (d) are means of these columns.
ROPE uses the agent's own model as its router on ChatInject, AutoDojo and PISmith. ‡ released code adapted to the paper's agent and suite. † the authors' planner and validator.
Against ROPE on AgentDojo and AgentDyn
| Suite | SPILLWAY | ROPE, Claude Opus 4.8 router | ROPE, gpt-4o-mini router | |||
|---|---|---|---|---|---|---|
| UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | UAU ↑ | ASR ↓ | |
| AgentDojo v1.2.2 | ||||||
| Banking | .521 | .000 | .438 | .000 | .431 | .000 |
| Slack | .524 | .000 | .514 | .048 | .343 | .019 |
| Travel | .571 | .000 | .500 | .000 | .350 | .000 |
| Workspace | .838 | .000 | .670 | .000 | .539 | .000 |
| AgentDyn | ||||||
| GitHub | .600 | .000 | .500 | .000 | .383 | .000 |
| DailyLife | .435 | .000 | .295 | .010 | .150 | .000 |
| Shopping | .383 | .000 | .361 | .044 | .333 | .072 |
| Mean of 7 (unweighted) | .553 | .000 | .468 | .015 | .361 | .013 |
gpt-4o-mini under the standard attack, one run per suite (seed 42). ROPE runs with its authors' Claude Opus 4.8 router and, in the last two columns, with gpt-4o-mini, the agent's own model, as router: its mean utility falls from .468 to .361 and it leaks on Shopping (.072) and Slack (.019).
Reproduce · checks run 2026-11-05
The tables rebuild from the released files. 590 checks, 0 mismatches.
These checks need no GPU, server or API key.
| Check | Result | Evidence |
|---|---|---|
| Paper tables from the per-cell files | 590 of 590 checks pass: Tables 1 to 3, six of the eight rows of Table 4's ChatInject panel, Table 8 and Figure 3's Workspace and Slack values | make_tables.py TABLES.md |
| Shipped files against the provenance record | 13 of 13 sha256 match: 7 result files, 1 script, 4 harness modules and the evaluated gate | BUILD_PROVENANCE.md |
| Tests | 92 of 92 pass, including 3 that compare the sanitizer, the identity index and the floor's verdict with the evaluated gate on synthetic inputs. The library checks have regression cases in tests/test_hardening.py and a version record in the changelog. The gate's self-test passes. | tests/ make selftest |
What the table and test checks print
$ python benchmarks/scripts/make_tables.py --no-write --quiet 2>&1 | tail -1 make_tables.py: 590 checks pass $ python -m pytest -q tests | tail -1 92 passed in 3.74s
Pinned in the paper's runs
- Suite
- AgentDojo v1.2.2 (data as in agentdojo 0.1.35) full grid one run per cell
- Backbones
- gpt-4o-mini-2024-07-18 (OpenAI API) qwen3-coder:30b qwen3:8b (Ollama)
- Agent request
- Default system prompt, 469 bytes, sha256
a021a92b…temperature 0 seed 42 at most 8192 output tokens reasoning off where switchable - Judge
- gemma4:31b (Q4_K_M), local Ollama 4096-token context temperature 0 fail-closed
- Evaluated gate
reference/spillway_gate.py, the final build sha256bc08ab7a…some results, among them SPILLWAY's Table 1 cells, ran on an earlier build without the address and payee-change rules (Appendix B of the preprint)- Hardware
- One NVIDIA RTX PRO 6000 Blackwell one cell:
reproduce/run_cell.sh banking qwen3:8b(switches)
FAQ · adding it to an agent
Before you add it to your agent.
Does it need a model?
No. The sanitizer and the floor are rules. The judge is optional and works with an OpenAI-compatible endpoint, Guard(task, judge=Judge(endpoint, model)). On Ollama, turn a reasoning model’s thinking off, as the paper’s runs did: Judge(endpoint, "gemma4:31b", extra_body={"reasoning_effort": "none"}). Without a judge, a call only the judge could admit is refused. The paper's Table 3 disables recovery of refused identity targets while keeping fixed delegation rules and the object judge enabled. The qwen3:8b cells have zero attack success, at mean utility .354 instead of .370.
How large must the judge be?
The paper uses a 4-bit gemma4:31b on local Ollama. A 4-bit gemma4:12b also kept zero attack success on the qwen3:8b cells and the ChatInject panel, at utility .348 instead of .370.
My tools are not AgentDojo's. What do I declare?
Return typed objects, such as pydantic models. Declare your tool-name prefixes and target-argument names once, for example Lexicon().extend(target_arg_hints={"payee"}). Then pass the lexicon to Guard.
What happens when a call is refused?
The agent gets a tool message, decision.refusal_text(name), that asks it to retry with values from the user's request. The episode goes on.