All posts

Measured on 15 Real Repos: -36% to -99.8% Input Tokens

Nov 14, 2026 4 min read Oleg Savelev
benchmarks measured token-costs ast

Every benchmark in this space has the same weakness: the scenarios were written by the team that sells the tool, and the baseline numbers live in a spreadsheet nobody can audit. So instead of asking you to trust our simulated harness, we did something more uncomfortable: cloned fifteen real repositories across 10 programming languages, committed the scenario list before running anything, and measured every single token on real bytes.

-46% to -99.8% input tokens, with zero-loss gates deciding what counts. Here is the full method, the numbers, and the honest failures.

The method: pre-registration and gates

Three rules, all enforced in code (src/bench_scenarios.json):

  1. Pre-registration. The repos (flask, requests, fastapi, rich, httpie), the targets (file + symbol), and the gate definitions were committed before the first measurement. Nothing was tuned after seeing numbers.
  2. Measurement, not modeling. Baselines are not "estimated tool tokens" — they are the calibrated BPE count of what the model would literally receive: the whole fastapi/routing.py (73,721 tokens), the raw git grep output, the full-file rewrite.
  3. Zero-loss gates. Every compressed artifact must prove it kept what matters: the target symbol's signature intact and parseable, every top-level definition still visible, the failing test's name and file:line preserved. A failed gate disqualifies the scenario — savings without proof don't get counted.

The numbers

Input axis, one measured session per repository:

Repo Scenario Baseline TokenSafe Reduction Gates
lodash (JS) Symbol read (skeleton) 515,843 855 -99.8% PASS
lodash (JS) Multi-turn + repeat reads 1,640,674 4,233 -99.7% PASS
lodash (JS) Architecture discovery 802,825 178 -99.98% PASS
fastapi (Py) Junk-read guard (uv.lock) 367,129 855 -99.8% PASS
sinatra (Ruby) Architecture discovery 28,165 199 -99.3% PASS
composer (PHP) Junk-read guard (composer.lock) 33,103 1,645 -95.0% PASS
flask (Py) Write egress (output axis, $15/M) 7,106 44 -99.4% PASS
curl (C) Symbol read (skeleton) 125,168 31,174 -75.1% PASS
rich (Py) Multi-turn + repeat reads 80,036 12,329 -84.6% PASS
flask (Py) Symbol read (skeleton) 18,864 3,784 -79.9% PASS
sinatra (Ruby) KV-cache 10-turn bill 303,850 63,393 -79.1% PASS
retrofit (Java) KV-cache 10-turn bill 3,312,375 673,361 -79.7% PASS
gin (Go) KV-cache 10-turn bill 317,638 65,036 -79.5% PASS
zod (TS) KV-cache 10-turn bill 1,905,787 431,857 -77.3% PASS
serde (Rust) KV-cache 10-turn bill 1,307,460 300,472 -77.0% PASS
curl (C) KV-cache 10-turn bill 1,586,875 423,360 -73.3% PASS
newtonsoft (C#) Symbol read (skeleton) 56,076 17,871 -68.1% PASS
httpie (Py) Symbol read (skeleton) — — skipped —
rich (Py) Verbose test output (noise strip) 746 385 -48.4% PASS
flask (Py) Secret guard (cat .env -> deny, 0 leak) 153 82 -46.4% PASS
composer (PHP) Prompt compaction (TOCP) 138 46 -66.7% FAIL
flask (Py) Full-suite pytest output 6,402 6,191 -3.3% PASS
fastapi (Py) Git diff churn -> slice 416 393 -5.5% PASS

Aggregate: ~25.7M tokens saved across one traced pass over all fifteen repositories — measured, labeled, and reproducible with one command. The live table with every row, gate and FAIL reason lives at tokensafe.dev/bench.

What the gates caught

The honesty mechanics are not decorative:

  • Noise-only stripping, failures untouched. A clean suite scores ~0-3% (flask: 6,402 → 6,191 — the noise strip only removes true noise). A verbose failing run scores -48.4% (rich: 746 → 385) — progress dots and verbose PASSED lines collapse to a counter, while every failing test name, error type and traceback frame survives the gates. An optimizer that claims savings on clean output is lying.
  • One scenario is published as FAIL — with the reason attached. fastapi's test-output: fastapi's strict warning filters escalate a pytest-asyncio configuration deprecation into a pytest INTERNALERROR, so the suite cannot be collected in the bench venv. The gate rejects the output, the row ships with the traceback excerpt on the live page, and no savings are claimed.
  • The skeletonizer refuses to run when the target symbol is more than ~70% of the file — keeping the target intact is the whole point, so savings are bounded by everything around it. That's why flask shows -79.9% while fastapi's fat routing.py shows "only" -46.4% (its API docstrings are contracts and stay), and why httpie's core.py (one giant main()) skips the scenario entirely.

Read the axes separately

Two very different money streams hide behind "token savings":

  • Input axis ($3/M): file reads, search results, history, architecture crawls. This is where the -46% to -99.8% lives, and where KV-cache locking (90%+ prefix hits at 0.1× rate) compounds it on multi-turn sessions.
  • Output axis ($15/M): a write_file re-emits the whole file — 7,106 tokens billed at output prices for flask's hub file. The v3 egress guard steers the model to a surgical Edit: 44 tokens, same change, same file, verified by gates. At 5× the input price, this axis matters more per token.

Reproduce it

pip install -U tokensafe
tokensafe bench --real                        # all 5 pre-registered repos
tokensafe bench --real --repo=pallets/flask   # one repo
tokensafe bench --real --here                 # your current repository

The harness clones at HEAD, runs the pre-registered scenarios, applies the gates, and writes BENCHMARK_REAL.md + raw JSON next to your data directory. Raw output for this run: bench/bench.json. The complete honesty contract (measured vs. estimated, and why we never blend them) is in our costs post.

If a token-optimization tool won't show you its gates, it's selling you truncation.

Setup and agent wiring: the docs. What you pay per plan: pricing.