Islands of Stability The Map
Where does an AI agent keep doing its job as it is given more tools and its tools start failing?
Snapshot 1 swept three open-weight models, run on a local workstation, across the number of tools the agent is given (1 to 6) and the share of its tool calls that fail (0 to 50 percent): 100 runs per cell, in two tool orders, 18,720 runs in all. The rules were deposited before the first run.
The answer, for all three models, is a continent with a slope, not a band with an edge. Adding tools between two and six did not by itself lower success. Every step up in the failure rate did. gpt-oss-20b held 92 percent success with no failures and 50 percent when half its tool calls failed. The two Qwen models held far smaller islands and never produced a fully correct record, because they never computed the invoice total right.
Scope. The principle this site studies is meant to hold across the sciences. This instrument measures one domain, AI agents, on one task. Its two tests are narrower than the theory's propositions 1 and 2, and they say nothing on their own about organizations or any other domain.
Snapshot 1
The grid shows one model at a time. Each cell is the share of runs that succeeded, with its 95 percent interval underneath. The one-tool column is out of scope: the task needs two tools.
Loading the snapshot data.
Values from the snapshot file. Tiles marked descriptive decide nothing. The two verdicts follow the pre-registered rules.
The verdicts, as pre-registered
Two tests, each scored on its own, on tool order A. The deposit calls them P1 and P2. This page calls them tests 1 and 2 so they are not mistaken for the theory's propositions, which they bear on but do not decide.
| Model | Test 1: upper tool edge | Test 2: recovery cost | In plain words |
|---|---|---|---|
| gpt-oss-20b | restricted drop 0 points [-7, 7] | survives +0.8 turns [0.6, 1.1] | No upper tool edge: a drop of 15 points or more at six tools is ruled out. A failure costs +0.8 extra turns, but net of re-asking for the failed data it costs nothing. |
| Qwen3-14B | restricted drop 0 points [-13, 13] | survives +0.8 turns [0.6, 1.0] | No upper tool edge: a drop of 15 points or more at six tools is ruled out. A failure costs +0.8 extra turns, but net of re-asking for the failed data it costs nothing. |
| Qwen3-4B | below resolution peak 5.7 tools [4.5, ∞] | survives +0.3 turns [0.1, 0.5] | No edge found: success was still rising at six tools, so there was no peak to fall from. A failure costs +0.3 extra turns, but net of re-asking for the failed data it costs nothing. |
Verdict words: survives means the prediction held, restricted means the prediction failed by more than the smallest effect worth reporting, and below resolution means the data cannot tell. The full rules are in the pre-registration.
What else the data show
These reports were committed in writing on 27 September 2026, while the sweep was running and before any of its results were looked at. They are descriptive: none changes a verdict.
| Model | Success, no failures → half the calls failing | Fully correct | Total right | Gave up after a failure | Recovery cost beyond the retry |
|---|---|---|---|---|---|
| gpt-oss-20b | 92% → 50% 92 · 84 · 77 · 64 · 56 · 50 | 60% | 60% | 33% | -0.8 turns [-1.1, -0.6] |
| Qwen3-14B | 38% → 21% 38 · 35 · 31 · 28 · 25 · 21 | 0% | 0% | 15% | -0.5 turns [-0.7, -0.4] |
| Qwen3-4B | 29% → 18% 29 · 27 · 24 · 22 · 19 · 18 | 0% | 0% | 61% | -0.9 turns [-1.0, -0.8] |
- What "success" means. A run succeeds when its weighted field score reaches 0.8. The total is worth 2 of 10, so a run whose only error is the total still counts as a success. The Qwen models never got a total right, so every Qwen success is a record correct in everything except the total.
- Giving up. After an injected failure, the small Qwen stopped without submitting in most runs. The larger Qwen rarely did. Grit differs more between models than tool count does.
- Recovery. A failure costs extra turns only because the agent asks again for what failed. Beyond that retry, faulted runs took fewer turns than their controls, because some gave up early.
Which tool, not how many
The tools arrive in a fixed order, so a change at one step could be the tool, not the count. Tool order B adds the same tools in a different order to tell the two apart. For gpt-oss-20b, the model that could compute the total, the same tool moved success in both orders:
| Tool added | Mix A: position | Mix A: change in success | Mix B: position | Mix B: change in success |
|---|---|---|---|---|
| web search | tool 3 | −4 points [−9, +1] | tool 4 | +4 points [−1, +10] |
| calculator | tool 4 | +5 points [0, +10] | tool 5 | 0 points [−5, +5] |
| spreadsheet | tool 5 | −6 points [−11, −1] | tool 3 | −10 points [−15, −5] |
| tool 6 | +3 points [−2, +8] | tool 6 | +1 points [−4, +6] |
Change in success when the tool is added, pooled over failure rates, with its 95 percent interval. Bold: the interval excludes zero. For the two Qwen models no step moved success beyond chance in either order. The spreadsheet holds each line's amount and should help, but it hurt in both orders.
Why the spreadsheet hurts. Its numbers are right: all 452 line amounts match the answer key. The trouble is when the model reaches for it. In all 453 runs where gpt-oss-20b opened the spreadsheet, it did so right after the invoice had failed to load. Instead of asking for the invoice again, it switched to the spreadsheet, which has the line items but not the invoice number, date, vendor or currency. In 184 of those runs it never saw the invoice at all, and only 16 percent of the runs that opened the spreadsheet succeeded, against 75 percent of those that did not. So the damage grows with the failure rate: in tool order A, 3 points with no failures and 12 when half the calls fail.
In plain words: a plausible shortcut pulled the agent away from simply trying again. A partial substitute did more harm than the failure it was reaching around. With no failures the model never opened the spreadsheet, and the few points lost there are arithmetic slips of the kind any change to the prompt can shift.
The instrument, checked first
Before the sweep, the noise floor measured how much the instrument itself varies. Each model reran one cell (3 tools, 20 percent failures): 50 times with identical inputs on 20 documents, and 50 times with a new random seed on all 100 documents.
| Model | Identical reruns reproduced | Outcome changes, fixed seeds | Cell success across 50 seeds | phi | Documents whose outcome depends on the seed |
|---|---|---|---|---|---|
| gpt-oss-20b | 100% | 0 of 20 | 71–84% | 0.69 | 70 of 100 |
| Qwen3-14B | 100% | 0 of 20 | 30–33% | 0.16 | 7 of 100 |
| Qwen3-4B | 100% | 0 of 20 | 23–27% | 0.20 | 4 of 100 |
Every identical rerun reproduced its transcript byte for byte, so the setup is exactly repeatable. With new seeds the cell's success moved far less than chance alone would move it (phi below 1), so each cell's interval is wider than it needs to be: the intervals err on the safe side. The Qwen models give nearly the same answer on a document whatever the seed, but gpt-oss-20b does not. The noise floor is reported beside the intervals, never inside them.
How it was run
The tool mix
The task needs two tools: fetch_document and submit_record. Each step above two adds one tool in a fixed order. Tool order A, the one the verdicts use:
- Tool 3:
search_web, a plausible alternative route to the document. - Tool 4:
calculator. At this task's difficulty it can do the arithmetic the reader otherwise does in its head. - Tool 5:
read_spreadsheet, which holds each line's amount, so it too can help. - Tool 6:
send_email, plausible but the wrong action.
Tool order B adds the spreadsheet third, then web search, the calculator and email. Both orders ran in snapshot 1. Only order A moves the verdicts, and order B is reported as descriptive, as the pre-registration says. An earlier version of this page called the calculator irrelevant and the spreadsheet a near-duplicate. At the difficulty chosen by calibration neither is true.
What each dial stands for
| Experimental variable | Represents here | Does not claim to measure |
|---|---|---|
| Tool count | The number of external systems an agent can reach | Organizational size |
| Tool-failure rate | How reliable those systems are | Market or environmental volatility |
| Recovery cost | The extra turns an agent spends after a failure | Organizational recovery |
| Success rate | Getting the job done under this fixed harness | General agent capability |
The mapping from these dials to the theory's terms is an assumption, not a finding.
The two shapes the map could return
Before the sweep, this page showed the two answers the instrument could give. Both grids are illustrative, not data. Snapshot 1 returned the second shape for all three models: a continent with a slope.
Table view
Figure A, illustrative values, percent success. Not data. The one-tool column is out of scope: the task is defined at two tools.
| Fault rate | 1 tool (out of scope) | 2 tools | 3 tools | 4 tools | 5 tools | 6 tools |
|---|---|---|---|---|---|---|
| 0 percent | - | 88 | 93 | 90 | 61 | 33 |
| 10 percent | - | 85 | 90 | 86 | 56 | 30 |
| 20 percent | - | 74 | 81 | 75 | 47 | 25 |
| 30 percent | - | 52 | 58 | 53 | 35 | 20 |
| 40 percent | - | 30 | 33 | 31 | 24 | 15 |
| 50 percent | - | 18 | 19 | 18 | 15 | 11 |
Table view
Figure B, illustrative values, percent success. Not data. The one-tool column is out of scope: the task is defined at two tools.
| Fault rate | 1 tool (out of scope) | 2 tools | 3 tools | 4 tools | 5 tools | 6 tools |
|---|---|---|---|---|---|---|
| 0 percent | - | 90 | 92 | 91 | 89 | 88 |
| 10 percent | - | 84 | 86 | 85 | 83 | 82 |
| 20 percent | - | 72 | 74 | 73 | 71 | 70 |
| 30 percent | - | 55 | 57 | 56 | 54 | 53 |
| 40 percent | - | 36 | 37 | 36 | 35 | 34 |
| 50 percent | - | 21 | 22 | 21 | 20 | 20 |
Pre-registration
The rules, revision r4, were confirmed by the author decision by decision, frozen with every fixed input hashed, and deposited on Zenodo on 26 September 2026 before any snapshot run: doi.org/10.5281/zenodo.22982434. The deposit holds the rules, their prose version, the configuration and its lock, the method note and the calibration and determinism records.
Test 1, upper tool edge (P1 in the deposit). One logistic regression per model on tool order A: success against tools, tools squared and failure rate, clustered by document.
- Survives: the curve bends down, its peak lies inside two to six tools, and success at the peak is above success at six tools by more than chance, with no failures.
- Restricted: no downward bend, a peak outside the range, or a drop at six tools shown to be smaller than 15 points, the smallest effect worth reporting.
- Below resolution: the data cannot tell.
Test 2, recovery cost (P2 in the deposit). Runs that recovered from an injected failure against the same document without failures: survives when they took more turns by more than chance, restricted when any extra cost is shown to be below one turn, and below resolution otherwise.
Decided before the sweep, 27 September 2026. Every run the server rejected as unparseable model output is recorded with its cause and reported both ways. The verdicts keep the deposited rule. The descriptive reports above were committed before any result was looked at.
The full rules, the smallest effects, the bootstrap and the order in which verdicts are assigned are in the deposit and in the method note inside it.
Threats to validity
- One task. Snapshot 1 speaks about invoice extraction under this harness, not agents in general.
- Three small local models. A replicate, not a population. An exploratory trial of larger models through OpenRouter, outside the snapshot, found far bigger islands.
- What success means. The 0.8 threshold admits a record whose total is wrong. The fully correct rate is reported beside it.
- Tool identity. The order of tools decides what a step measures. Tool order B separates the two, descriptively.
- An instruction conflict. The prompt asks for one submission. A failure on the submit tool forces a second. The failure axis partly measures whether a model breaks that instruction to retry.
- Unparseable replies. gpt-oss-20b produced 24 replies the server could not parse. Counting them as failures changes neither verdict.
- Construct mapping. The dials above stand in for the theory's terms by assumption.
The harness
The harness is a pinned Python package: a fixed executable grader, a fault injector, a runner that resumes after any interruption, the statistics and a reviewer's command that recomputes every number from the raw runs. The results bundle, 38,475 files, is deposited as 10.5281/zenodo.23066893 and passes that check exactly. The repository is public at github.com/delimiterbob/islands-harness. The deposited lock identifies the exact code that ran.
The same instrument can map your own agent on your own task before production. That is a service, and it lives at encarnacao.net.
Credited instruments
The Map does not replace any of these. It borrows from most of them.
- METR's task-horizon measurements. How long a task an agent can complete at a given success rate, tracked across model releases.
- Chroma's context rot study. Performance degrading as input length grows, even on simple tasks.
- NoLiMa. Long-context retrieval when the answer shares no words with the question.
- tau-bench pass^k. The chance an agent succeeds on all k attempts at the same task, not just once.
- FormatSpread. How far accuracy moves when only the prompt format changes.
- PromptBench. Robustness to adversarial edits of the prompt.
- HAL's scaffold-by-model grids. The same agent scaffolds run across many models, with cost beside accuracy.
- Sinha et al. on self-conditioning. Models becoming more likely to err after seeing their own earlier errors in context.
What none of them publishes is a two-axis viability grid with its tests deposited before the sweep, and within-run recovery with sampling error and a measured noise floor beside it.
What was cut
- Hysteresis and ghost rate as grid quantities. Stateless inference has no path through the grid, so there is nothing to lag. They may return as within-run quantities.
- Per-release cadence. At most two snapshots a year, each dated, with the models pinned. Tracking the same map across model versions is the plan for islands that move.
- Model size as an axis. In snapshot 1 it is a replicate: the same grid on three local models.
- A hosted headline model and a deposited cliff forecast. Snapshot 1 ran on local models only, at no API cost, so the spend forecast and the cliff forecast built on it were not made.
- A ring-shaped island enclosed on the failure axis. No mechanism, corrected in draft r2.