Every benchmark in this space has the same weakness: the scenarios were written by the team that sells the tool, and the baseline numbers live in a spreadsheet nobody can audit. So instead of asking you to trust our simulated harness, we did something more uncomfortable: cloned fifteen real repositories across 10 programming languages, committed the scenario list before running anything, and measured every single token on real bytes.
-46% to -99.8% input tokens, with zero-loss gates deciding what counts. Here is the full method, the numbers, and the honest failures.
The method: pre-registration and gates
Three rules, all enforced in code (src/bench_scenarios.json):
- Pre-registration. The repos (flask, requests, fastapi, rich, httpie), the targets (file + symbol), and the gate definitions were committed before the first measurement. Nothing was tuned after seeing numbers.
- Measurement, not modeling. Baselines are not "estimated tool tokens" — they are the calibrated BPE count of what the model would literally receive: the whole
fastapi/routing.py(73,721 tokens), the rawgit grepoutput, the full-file rewrite. - Zero-loss gates. Every compressed artifact must prove it kept what matters: the target symbol's signature intact and parseable, every top-level definition still visible, the failing test's name and
file:linepreserved. A failed gate disqualifies the scenario — savings without proof don't get counted.
The numbers
Input axis, one measured session per repository:
| Repo | Scenario | Baseline | TokenSafe | Reduction | Gates |
|---|---|---|---|---|---|
| lodash (JS) | Symbol read (skeleton) | 515,843 | 855 | -99.8% | PASS |
| lodash (JS) | Multi-turn + repeat reads | 1,640,674 | 4,233 | -99.7% | PASS |
| lodash (JS) | Architecture discovery | 802,825 | 178 | -99.98% | PASS |
| fastapi (Py) | Junk-read guard (uv.lock) | 367,129 | 855 | -99.8% | PASS |
| sinatra (Ruby) | Architecture discovery | 28,165 | 199 | -99.3% | PASS |
| composer (PHP) | Junk-read guard (composer.lock) | 33,103 | 1,645 | -95.0% | PASS |
| flask (Py) | Write egress (output axis, $15/M) | 7,106 | 44 | -99.4% | PASS |
| curl (C) | Symbol read (skeleton) | 125,168 | 31,174 | -75.1% | PASS |
| rich (Py) | Multi-turn + repeat reads | 80,036 | 12,329 | -84.6% | PASS |
| flask (Py) | Symbol read (skeleton) | 18,864 | 3,784 | -79.9% | PASS |
| sinatra (Ruby) | KV-cache 10-turn bill | 303,850 | 63,393 | -79.1% | PASS |
| retrofit (Java) | KV-cache 10-turn bill | 3,312,375 | 673,361 | -79.7% | PASS |
| gin (Go) | KV-cache 10-turn bill | 317,638 | 65,036 | -79.5% | PASS |
| zod (TS) | KV-cache 10-turn bill | 1,905,787 | 431,857 | -77.3% | PASS |
| serde (Rust) | KV-cache 10-turn bill | 1,307,460 | 300,472 | -77.0% | PASS |
| curl (C) | KV-cache 10-turn bill | 1,586,875 | 423,360 | -73.3% | PASS |
| newtonsoft (C#) | Symbol read (skeleton) | 56,076 | 17,871 | -68.1% | PASS |
| httpie (Py) | Symbol read (skeleton) | — | — | skipped | — |
| rich (Py) | Verbose test output (noise strip) | 746 | 385 | -48.4% | PASS |
| flask (Py) | Secret guard (cat .env -> deny, 0 leak) | 153 | 82 | -46.4% | PASS |
| composer (PHP) | Prompt compaction (TOCP) | 138 | 46 | -66.7% | FAIL |
| flask (Py) | Full-suite pytest output | 6,402 | 6,191 | -3.3% | PASS |
| fastapi (Py) | Git diff churn -> slice | 416 | 393 | -5.5% | PASS |
Aggregate: ~25.7M tokens saved across one traced pass over all fifteen repositories — measured, labeled, and reproducible with one command. The live table with every row, gate and FAIL reason lives at tokensafe.dev/bench.
What the gates caught
The honesty mechanics are not decorative:
- Noise-only stripping, failures untouched. A clean suite scores ~0-3% (flask: 6,402 → 6,191 — the noise strip only removes true noise). A verbose failing run scores -48.4% (rich: 746 → 385) — progress dots and verbose PASSED lines collapse to a counter, while every failing test name, error type and traceback frame survives the gates. An optimizer that claims savings on clean output is lying.
- One scenario is published as FAIL — with the reason attached. fastapi's test-output: fastapi's strict warning filters escalate a
pytest-asyncioconfiguration deprecation into a pytestINTERNALERROR, so the suite cannot be collected in the bench venv. The gate rejects the output, the row ships with the traceback excerpt on the live page, and no savings are claimed. - The skeletonizer refuses to run when the target symbol is more than ~70% of the file — keeping the target intact is the whole point, so savings are bounded by everything around it. That's why flask shows -79.9% while fastapi's fat
routing.pyshows "only" -46.4% (its API docstrings are contracts and stay), and why httpie'score.py(one giantmain()) skips the scenario entirely.
Read the axes separately
Two very different money streams hide behind "token savings":
- Input axis ($3/M): file reads, search results, history, architecture crawls. This is where the -46% to -99.8% lives, and where KV-cache locking (90%+ prefix hits at 0.1× rate) compounds it on multi-turn sessions.
- Output axis ($15/M): a
write_filere-emits the whole file — 7,106 tokens billed at output prices for flask's hub file. The v3 egress guard steers the model to a surgical Edit: 44 tokens, same change, same file, verified by gates. At 5× the input price, this axis matters more per token.
Reproduce it
pip install -U tokensafe
tokensafe bench --real # all 5 pre-registered repos
tokensafe bench --real --repo=pallets/flask # one repo
tokensafe bench --real --here # your current repository
The harness clones at HEAD, runs the pre-registered scenarios, applies the gates, and writes BENCHMARK_REAL.md + raw JSON next to your data directory. Raw output for this run: bench/bench.json. The complete honesty contract (measured vs. estimated, and why we never blend them) is in our costs post.
If a token-optimization tool won't show you its gates, it's selling you truncation.
Setup and agent wiring: the docs. What you pay per plan: pricing.