This benchmark is independent, and it tests a broader question than the public tool benchmarks.
That is why these results may look different from the public numbers. I am not testing whether these tools can win in their best-case marketing setup. I am testing whether they improve actual coding runs across different repo sizes and task types while preserving output quality.
I ran these tools in two batches a few weeks apart. Between the two, the platform drifted on its own: the exact same baseline config came out about 25% more expensive and a few points lower quality the second time, from changes outside my control (model and platform updates over time).
To keep everything on one honest line, I use a single baseline and calibrate the later batch onto it, task by task. Because the baseline was re-run in both batches, I know exactly how much each task drifted, and I subtract that drift out of the later tools. Caveman, Graphify and Ponytail are from the first batch; rtk and code-review-graph are from the second, calibrated back onto the same baseline.
This is a correction, not a fresh measurement. I am re-running everything in one batch to confirm it, but the calibrated numbers should land very close.
Caveman was a clear win in my benchmark.
Numbers:
The quality gain is modest. This was not a massive jump in engineering quality. It looks more like Caveman pushed a few borderline runs over the pass threshold.
My read is that Caveman works because it attacks a very real source of waste: unnecessary talking. It does not try to make the model smarter by adding more process, more repo context, or more tools. It mostly makes the model say less.
That sounds simple, but I think it has two benefits. First, it directly reduces output cost. Second, it may reduce context rot because the model is not filling the session with its own long explanations and progress updates.
In my benchmark, I did not see Caveman make the agent lazy, brittle, or careless. It looked like baseline, but cheaper and slightly more effective. That matches my own usage experience.
My current view: Caveman is the tool I would use by default.
Ponytail is what I would be most careful with.
Numbers:
That drop is not catastrophic, but it is not nothing. The cost savings came with a visible quality trade-off.
The public pitch is that Ponytail stops agents from over-engineering. That is a real problem, and there are tasks where it makes sense. If the model is about to build a large custom component when a native platform feature would work, Ponytail is useful.
But in my benchmark and my own usage, the savings felt less like "same work, less waste" and more like "less work done." That is the danger.
Older AI models often simplified code in destructive ways. You could give them a 1000-line file and they would return a 200-line version, claiming it worked, while quietly removing important behaviour. Claude became useful partly because it was better at preserving complete edits. Ponytail feels a bit like a regression to that. It says to simplify reasonably, but models often over-follow the vibe of an instruction.
I think Ponytail is best for:
I would be cautious using it on production code, multi-stage implementations, or anything that needs clean extension paths.
My current view: Ponytail can be useful, but I would not use it by default.
Graphify had the biggest gap between the public claim and my practical result.
Numbers:
Graphify gives the agent a repo map and relationship graph. That sounds valuable, especially on large projects. In theory it should help the model find the right files at less cost. In my benchmark, that did not translate into better results.
My criticism of the 70x-style claim is the baseline they chose. If you compare Graphify against an agent that reads hundreds of files blindly, then yes, a graph can look massively cheaper, but that is not how a good agent behaves on a decently structured repo. With a good prompt and a clear file structure, the agent usually only needs to inspect a small number of relevant files.
I think Graphify is probably most useful for:
I do not think it is automatically useful on clean repos, even large ones.
My current view: Graphify is situational. It may help in messy systems, but I would not assume it saves money or improves quality by default.
code-review-graph had the biggest headline number and the biggest real trade-off.
Numbers:
The 82x median claim (up to 528x on fastapi) compares a graph query against reading the entire repo — on fastapi that is 951,000 tokens of "read everything" against about 2,200 for the graph. It is the same naive baseline as Graphify, and their own docs admit it, calling the whole-corpus number "an upper bound no real agent pays" because a competent agent greps for identifiers and reads only the best-matching files. That is a retrieval number for one question, not the cost of finishing a task or whether the finished work is correct.
In my benchmark the graph did cut cost, more convincingly than Graphify, but with the largest quality and pass-rate drop of any tool here. My read is that the agent trusts the graph and reads less of the real code, so it edits confidently on an incomplete picture and gets more wrong. This is n=1 on the tasks that flipped, so I want to re-run those, but the direction was clear.
My current view: a real cost saver, but the quality cost was too high to use by default. Like Graphify, I would keep it for messy or very large repos where reading everything is genuinely expensive — not for work where correctness matters most.
rtk was the clearest case of a claim that is true in a narrow place and irrelevant in the place that matters.
Numbers:
The pitch is 60-90% token reduction, with headlines like "cut your AI bill 80%". That is real, but only for the text of bash command output. rtk compresses what git status, test runs and builds dump into the context. It does not touch what the agent reads with its own tools (Read, Grep, Glob), and on Claude that bash text is a small slice of a cache-dominated bill. rtk's own README is honest about this: it says the bash reduction "is not the same as cutting your bill by 90%".
I also saw a real failure tail. On a few tasks the compressed output seemed to cost the agent information and it wandered — one implementation task went from about 23 turns to 82 and tripled in cost. I re-ran the worst offenders three times each and the direction held: rtk was never cheaper on those tasks, and sometimes much worse.
My current view: rtk does what it says for bash output, but it did not reduce my end-to-end cost on Claude. I would not use it to save money.
1. Caveman. The only one I would use by default. It saved cost and slightly improved pass rate without a clear quality downside. 2. code-review-graph. The biggest genuine cost saving, but only worth it where you accept a real quality drop — messy or very large repos. 3. Graphify. Same idea as code-review-graph but weaker: in my benchmark it did not even save money. 4. Ponytail. A small saving for the clearest quality trade among the cheap tools. 5. rtk. The only tool that cost me more with no upside on Claude.
It is not good to judge these tools only by token savings or lines of code. The real question is whether the useful work is preserved.