Unit tests only prove TokenSafe works on the code we wrote to test it. They don't show what happens when the tool encounters real, messy repositories in the wild. To see where it broke, we set up an automated pipeline across 506 popular open-source repositories written in 11 programming languages. We logged every unexpected failure, patched the bugs, re-ran the full suite, and discarded the local clones. Here is what failed and what we changed.
How the pipeline works
The test harness runs internally. For each repository, it pulls a shallow clone at depth 50, executes 20 test scenarios, logs any anomalies, and deletes the clone so the process leaves no residual files on disk:
- 12 published benchmark scenarios matching the checks on our public benchmark page: symbol extraction, multi-turn pointer deduplication, test output compaction, architecture graphs, KV-cache stability, and secret-guard interception.
- 8 internal checks covering the rest of our product surface: all 13 MCP tools, the guard decision matrix (
cat .env,git add .,rm -rf, TLS bypass, wholesale writes), the Tool Result Ledger, session tool pruning, schema slicing, JSON self-healing, history compaction, and the secret redactor.
None of this run data is published. Failures land in a local backlog with reproduction details, logs, and repository names so we can triage them into fixes. Each clone is wiped in a finally block; during the first sweep, peak clone storage stayed at 4 GB, and the working directory was empty when the run finished.
Ten defects the unit tests never saw
We reproduced every defect below on a real repository, wrote a regression test, and shipped a fix. All figures come directly from sweep logs rather than estimates.
| # | Defect | Symptom on real repos |
|---|---|---|
| 1 | API surface map exploded | 114M tokens on pdf.js (45.7 MB of markdown on caddy); minified bundle one-liners leaked into signatures at 157 KB per line |
| 2 | Path extraction truncated extensions | App.tsx → App.ts; .cs, .rb, .php, .swift not matched at all |
| 3 | Paths with spaces broken | Splay Tree/SplayTree.swift → Tree/SplayTree.swift |
| 4 | JSON self-healing gave up | A 544 KB search index cut after a completed key ("后") was unrepairable |
| 5 | Unicode crash | A single non-UTF-8 byte killed the whole repository row |
| 6 | git diff gate off-by-two | +++ b/ prefix stripping lost asset-only diffs |
| 7 | bench --real --here crashed |
NameErrors on any repo with a test suite (the command we publish for reproduction) |
| 8 | KV-cache gate measured the wrong thing | Hub-file size arithmetic, not normalizer churn |
| 9 | Diff slicer expanded small diffs | Negative "savings" on 19.6% of repositories |
| 10 | Illusory parallelism | GIL-bound threads: one repository took 40 s solo, 4,260 s in the pool |
The worst issue was the API surface generator (Defect 1). It was built as a lightweight alternative to indexing an entire project. In practice, it traversed every single file and extracted signatures indiscriminately. When it encountered minified production bundles, which pack thousands of lines into one, it treated the whole minified line as a single signature. That dumped 157 KB of compressed JavaScript per line into the prompt. On caddy, that produced a 45.7 MB markdown dump; on pdf.js, it consumed 114M tokens.
We capped the map output at roughly 15,000 tokens, prioritized key architecture hubs, and filtered out vendor directories, test fixtures, and minified paths. Output on caddy dropped from 45.7 MB to 61,631 characters. The tool also used to write a tokensafe-out/ folder into the user's workspace as a side effect; it now leaves the working tree alone unless a file export is explicitly requested.
Results before and after
We ran the first 168 repositories against the original code, applied our fixes to the remaining 338, and re-tested the entire 506-repo batch to compare results on the exact same workload:
| Scenario | Before | After |
|---|---|---|
| API surface map | 67.3% of repos failed | 13.4% |
| JSON self-healing | 27.4% | 0.4% |
| git diff gate | 8.3% | 0% |
| Negative diff savings | 19.6% | 0% |
| KV-cache gate | 11.3% | 0% |
| Wall-clock timeouts | 9 repos | 0 |
Fourteen of the twenty scenarios now pass cleanly across all 506 repositories with zero crashes. The remaining failures come from edge cases we are still working through: a few malformed JSON payloads cut in awkward places, occasional prompt-distillation path corruptions, and dense codebases where API maps hit our token cap.
If a benchmark gate fails on every third repository you test, your gate is not strict. Your assumptions were just wrong. We had to fix the product so the benchmark could report real numbers.
Why Python threads killed our throughput
Early in the run, execution appeared to freeze: one Go repository logged 4,260 seconds of wall time. Profiling with cProfile showed that the core scanner was fast, indexing 5,906 files and 58,381 symbols in 25 seconds, and finishing the full scenario run in 40 seconds standalone.
The problem was our harness architecture. We used a thread pool, but almost all the heavy work (tree-sitter AST traversals, skeletonization, regex compaction) executes in Python. The Global Interpreter Lock (GIL) forced eight worker threads onto a single core. Each repository's recorded wall time was absorbing the cumulative CPU load of all active workers.
Moving from ThreadPoolExecutor to ProcessPoolExecutor gave each worker its own Python process and memory space. The second run swept 338 repositories, including large codebases like moby, in 639 seconds on the same hardware.
What we are testing next
The queue pulls from GitHub search, targeting top-starred, active repositories in our supported languages while prioritizing projects we have not tested yet. The next batch of 500 is running now, expanding framework coverage and surfacing another round of edge cases to fix.
You can run these same checks against your own repository locally:
pip install -U tokensafe
tokensafe bench --real --here
We publish the full scenario suite, raw JSON results, and failure logs at tokensafe.dev/bench. You can read more about our test criteria in our benchmarking post, and the underlying context economics in stop paying for code your agent already read.