These findings cover two sets of results. The first, from early October 2026, ran the 55 short tasks on Claude Sonnet 4.6. The second, from late October 2026, ran on Claude Sonnet 5.5 and added the SWE-bench Pro chains and my own custom long tasks. The leaderboards show the Sonnet 5.5 results. The savings are much smaller on Sonnet 5.5 than they were on Sonnet 4.6. Two things have changed, and neither is a fault in the tools:
Where a tool changes the model’s behaviour, any extra cost that behaviour causes counts against the tool. I count extra sub-agents, extra turns and extra tool calls as costs of the tool, not as noise.
This benchmark tests a broader question than the public tool benchmarks: what does a tool do to the cost and the outcome of a whole coding task, on a mix of small and large repos, bug fixes and implementations, and multi stage work. Most public claims measure something narrower, and most of the projects have since moved toward more careful numbers. The Sonnet 4.6 results are from early October 2026, on the tools’ then-current versions. The quotes below come from the projects' public repositories as of 8 October 2026 (Ponytail, Graphify and its glossary, Caveman, rtk); earlier wording is from their commit histories.
The reason the numbers differ is simple: those claims measure one component in isolation, and a coding task is dozens of turns. rtk really does compress command output, but that is a small slice of a bill dominated by re-reading the whole cached context on every turn, and if the compression ever costs the agent one turn of lost information, that extra turn re-reads everything and cancels the saving. Graphify's big number compares a graph query with reading every file, which no good agent does, so they say little about what happens when a graph answer is incomplete and the agent has to read the real code anyway. A claim only counts once it survives the full coding task, which is what this benchmark measures.
A negative change means cheaper. Brackets show the 95% confidence range for the change in cost. When the range includes 0%, the difference could be noise and is not a confirmed saving or increase.
The custom set is 3 long tasks with 6 runs per condition. Each run is a large body of work, but with few runs the confidence ranges are wide.
How “All tasks” is weighted. The view counts the three sets equally. The cost change is the average of the three set results, and its range comes from resampling tasks within each set. Each set measures a different kind of work, so none is allowed to dominate: short tasks are 55 of the 71 tasks, while the custom tasks are over half of the dollars spent. The view shows no dollar amounts; click a set to see them.
Pass rate (runs passed):
Caveman’s early pitch was a short instruction: be terse, drop the filler. On Sonnet 4.6 that worked: 17.5% cheaper with no measurable quality loss, and it was the tool I would use by default.
The plugin has grown a lot since. The earliest version had 148 files, 7 skills and 2 hook events. The build I tested on Sonnet 4.6 had 26 skills and a set of helper agents. The current one has 29 skills, 4 hook events, a reminder injected on every prompt, and mode and status handling on top of the style rules, and its rules are much more specific about wording.
Sonnet 5.5: the saving is gone. On the short tasks it cost 7.1% more than baseline, a real increase and not noise. On the other sets the difference is within noise. Pass rates are level with baseline.
My read is that the original idea was clean: it told the model one thing, once, and the model said less. As the tool grew it gave the model more instructions and nudges. More instructions leave more room for wording that changes behaviour, and newer models respond to wording less predictably. I have not isolated this cause: I did not run the older plugin on the new model. Part of the lost saving is also simply that Sonnet 5.5 already talks less than 4.6 did.
My current view: no longer a default install. The idea still works in principle, but on Sonnet 5.5 it did not pay.
Ponytail is the only tool that saves money across all three task sets on Sonnet 5.5, and the saving is larger on longer tasks: about 9% on short tasks, about 30% on the chains, about 16% on the custom tasks.
Ponytail had a larger quality penalty in my earlier tests, which was the main reason I held back. The author has since worked on it. On Sonnet 5.5, the equal-weight All-tasks quality score is 88.1 against 93.0 for baseline. That average combines graded short and custom tasks with binary chain outcomes, so it also matters that Ponytail passed every test on 20 of 26 chain runs, against 23 of 26 for baseline. The cost saving should be read alongside those outcomes.
It gives the model a short ruleset and a map of the project’s code at session start. Graphify does something similar, but Ponytail’s map is placed in front of the model, while Graphify’s has to be queried. A model that already has a way to find things (grep, glob, file reads) will often use those and skip an unfamiliar tool that does the same job. A map that is already in context avoids that. I think Ponytail has done what Caveman started to try, and done it better.
Credit to the author as well. The README has moved from large headline numbers to a benchmark with intervals and a stated list of limits, and earlier overclaims were withdrawn in public (see the first section). They have kept working on their own testing.
My current view: the best result on Sonnet 5.5, and worth using where cost matters.
Graphify builds a code graph of the repo before the run and tells the agent how to query it. On the chains, cost fell 11.4%, a confirmed saving, and runs took about 7 minutes against about 9 for baseline. On the short tasks there was no saving. On the custom tasks it cost 10.9% more, but the range is too wide to call that a confirmed difference.
The agent did not always use the graph. It queried it in 48 of the 110 short-task runs, and in none of the 26 chain runs or 6 custom runs. Even though the prompt told it how to query the graph, the agent kept using the search tools it already knows (grep, glob, file reads) and ignored the new one. The chain saving therefore cannot be credited to graph queries. In the short tasks, runs that queried the graph cost less than baseline and runs that did not cost slightly more.
On Sonnet 4.6 Graphify saved 28.1%, largely by helping the agent understand the code faster, which in turn stopped it delegating exploration to sub-agents. Sonnet 5.5 does not delegate that way even without Graphify, so that part of the saving is gone. What remains depends on whether the agent uses the graph.
The graph has to be queried, and the agent still has all its normal search tools. On the long tasks it used those instead and never touched the graph. That is a model with a familiar way of doing something passing over an unfamiliar tool that does the same job. This is what the transcripts show; why the agent chose that is my interpretation.
My current view: a reasonable tool that helps when the agent actually uses it. On the long tasks it was not used, so I would not credit it for the chain saving.
rtk does compress command output; that is real, and I measured it earlier. The problem is that a bill is mostly cached context re-read every turn. If the compression ever costs the agent one extra turn because it did not get what it needed, that turn re-reads the whole context and cancels the saving. The project itself now says its “up to 90%” is about bash output, not about your bill.
Sonnet 5.5: no measurable effect on any set. Every difference is within noise.
My current view: harmless, but I would not install it to save money on Claude.
1. Ponytail. The only tool with a clear saving on Sonnet 5.5. 2. Graphify. Faster on long tasks, with a modest saving on the chains. 3. rtk and Caveman. No saving. Caveman was the clear winner on Sonnet 4.6 and no longer is.
The bigger lesson is that a tool measured on one model does not carry over to the next. Savings that came from fixing a model’s habits disappear when the model improves, and instructions that were safe can start to behave differently. Judge a tool by what it does to the whole task on the model you use, and re-check it when the model changes.