Is it agentic enough? Benchmarking open models on your own tooling
Back to Articles Is it agentic enough? Benchmarking open models on your own tooling Published June 18, 2026 Update on GitHub Upvote 19 +13 Lysandre lysandre Follow Nathan Habib SaylorTwift Follow Pedro Cuenca pcuenq Follow Testing software for agentic-use Not all successes are equal How do we run...
Back to Articles Is it agentic enough? Benchmarking open models on your own tooling Published June 18, 2026 Update on GitHub Upvote 19 +13 Lysandre lysandre Follow Nathan Habib SaylorTwift Follow Pedro Cuenca pcuenq Follow Testing software for agentic-use Not all successes are equal How do we run evaluations? Which models to benchmark against? Large open models: hold the model, vary the revision Small models: hold the revision, vary the model Tweaking the tool: markers and results What's a marker? Is the CLI + Skill commit helping? Trying it yourself Closing Acknowledgements Benchmarking transformers revisions across different metrics This is a human-made, agent-focused blogpost. Coding agents increasingly work with our software instead of us: describe a task, and the agent picks the library, writes the calls, runs them, and debugs its own mistakes. When the library gets in the way, it will happily bypass it and rewrite the logic from scratch. This introduces a new concept in library development: the code should not only be correct and fast, but should be designed so that an agent can drive it effectively. A clunky API or stale docs annoy us developers, but it now also sends the agent down a longer, more expensive path. Most benchmarks just look at the final answer. We wanted the whole process instead: not just whether the agent got it right, but how much work it took to get there, and how that shifts across models, library revisions, and tasks. We measured exactly that, using transformers as our case study. Here, we will introduce a tool specific benchmark focusing on how the answer was found, and provide a simple implementation of one such harness, running entirely on open models driven by the pi coding agent, with the full sweep of models × revisions × tasks fanned out across Hugging Face Jobs so every run sees identical hardware. But, how do you optimize software for agents? We're strong believers in the following two software principles: If it isn't tested, then it doesn't work If it isn't documented, then it doesn't exist This remains the same within the realm of agentic-optimized tooling, and, for once, the two are directly tied to each other. You want your tool to exist for an agent: it needs to be discoverable. The API needs to be clear and the docs need to be extensive. They need to be structured in a way that the agent has rapid access to the useful files and examples. If you want your tool to work for an agent, then you should test it for agentic-use. Testing software for agentic-use We'll use transformers as an example throughout this blogpost: agents using it to solve ML tasks (classifying text, captioning images, transcribing audio), not contributing code to it; though the harness was designed to work with any tool that can be operated from the command line. Our intuition on transformers was that usage could be dramatically simplified with a few changes: a CLI, a Skill, and self-contained, task-specific examples. This is the same recipe recently applied to the hf CLI, redesigned to be agent-optimized, where agents used 1.3–1.8× (and up to 6×) fewer tokens. We wanted to know whether that kind of win generalizes, and whether it could be useful for transformers as well. Intuition is a powerful tool, but we wanted more evidence before we opened PRs that add several thousand lines of code to such a widely used codebase as transformers. We set out to measure what success looks like. Not all successes are equal Two agents can both produce the correct label for a sentiment-classification task, but one: writes a 40-line Python script, imports transformers, debugs a shape error, re-runs twice, and finally prints the answer; while the other types transformers classify --model ... --text "..." and is done in one call. Both reach POSITIVE (0.9999), and here are the two paths an agent actually took on this exact task: # Task: classify the sentiment of "I absolutely loved the movie, it was fantastic!" - # one agent: pipe a script into python and parse the output - python -Source: Hugging Face — Published — Category: Open Source