Benchmarking Agents Before Release
How I compare an updated agent with the version already in production.The PR job compares both agent versions on the same approved benchmark. Failed release checks block deployment. Diagram generated with OpenAI ImageGen.If I change an agent’s prompt, add a tool, or swap the underlying LLM, I can…
How I compare an updated agent with the version already in production.The PR job compares both agent versions on the same approved benchmark. Failed release checks block deployment. Diagram generated with OpenAI ImageGen.If I change an agent’s prompt, add a tool, or swap the underlying LLM, I can usually make myself feel good about the result in about five minutes. I ask it the same questions I used while developing it; the answers look better, and everyone is happy.This article reflects my own thoughts, ideas, and experience, with AI-assisted editing.That five-minute test only shows that the agent can answer questions it saw during development.Before releasing an agent, I want to compare the production version and the candidate on the same approved benchmark. The runner gives both versions the same questions, test data, starting state, and scoring rules. Each uses its own prompt, model, and tool configuration. The result is a score out of 100 for each version.The usual unit tests still check whether the API works, a policy rule fires, or a tool returns the right schema. All required tests must pass. An agent’s answers can vary between runs while code still checks particular facts and actions deterministically.The benchmark includes tasks that require the agent to use several sources and tools to complete the work. I keep the easier cases and add harder ones where the agent still gets the work wrong.Make the benchmark difficultHumanity’s Last Exam has 2,500 questions written by subject-matter experts across dozens of subjects. The authors designed it because models were scoring above 90% on existing academic benchmarks. [1] I would apply that idea to harder tasks drawn from the agent’s work.For a metadata generation agent at a fictional North American sales company, I would start with a few dozen cases. The agent reads transformation code, schemas, policy, metadata, and lineage, then writes descriptions, explains calculations, and identifies the likely impact of a proposed change.I might test whether it can:Find the owner and classification of a reporting column.Explain how net_sales_usd is calculated from gross sales, discounts, returns and exchange rates.Trace a figure someone has questioned in Tableau through the reporting table and data mart to the United States, Canadian and Mexican source systems.Recognise a restricted source column without copying sensitive values into its answer.Identify the dashboards, extracts and downstream applications affected by a SQLX change.Ownership and calculation would be the easier cases. For a harder task, I would ask it to trace a calculation across all three countries and identify the affected reports.The first run establishes the baseline. A score of 52 reflects the points earned on this deliberately difficult mix of tasks. I would assess routine work separately before deciding whether the agent is ready for production.Designing the benchmarkFor this example, the business subject-matter expert (SME) defines the expected meaning and unacceptable answers. The data engineer supplies schemas, lineage, and test data; the agent engineer builds the cases and automated graders. The data product owner approves the benchmark and release criteria.Two SMEs independently score a sample of correct, incomplete, and wrong answers, then resolve their disagreements. The agent engineer uses part of that sample to configure the graders and checks them against the remaining answers. Code checks facts and recorded actions. A separate LLM judge scores explanations against the rubric. That mix follows Anthropic’s guidance on agent evaluations. [2]The grader code, judge model, prompt, and rubric stay fixed for comparisons on that benchmark version. The engineer checks the grader again when adding harder cases. The August 2026 AgentJudgeBench preprint found that judge agreement worsened as tool-calling tasks became more difficult. [3]For a metadata update, the runner checks the rows written to the test metadata tables: the description belongs to the right column, the classification is correct, and an existing SME correction is preserved. It also records attempted changes outside the task. If the source names no owner, the expected answer is an unknown owner with an explanation of what is missing.I would keep these cases in one benchmark. To get the score out of 100, I would add the average points earned across cases, divide by the total available points, and multiply by 100. A larger agent could have separate benchmarks for accuracy and quality, tool use and reliability, performance, security, and appropriateness and safety.The configuration defines how each area becomes a score out of 100 and its weight in the overall result. The report keeps the original measurements too, including completion time and cost per completed task.Cases involving restricted values, invented sources, or actions outside the agent’s permissions are mandatory release gates. A failure in any trial blocks the release, regardless of the overall score.Keep the benchmark in GitThe questions, fixtures, scoring rules and approval belong in version control alongside the agent’s other engineering work:agent-benchmarks/ metadata-agent/ v3/ benchmark.yaml cases/ fixtures/ rubrics/ reference-results/Every comparison on v3 uses the approved questions, fixtures, weights and graders. A new feature, changed business rule, faulty reference answer, or newly discovered production failure can justify a new version. The changes go through a pull request, with previous versions and results retained.In this example, each of the four outcomes earns three points when fully met and zero otherwise:id: lineage-017difficulty: hardpoints: 12task: > Trace net_sales_usd from the executive Tableau dashboard to the contributing regional source columns. Explain the currency conversion and identify the affected reports.grading: points_per_outcome: 3 required_outcomes: - identifies all three regional sources - explains discounts and returns before conversion - identifies the exchange-rate dependency - names the dashboard and extract critical_failure: - exposes a restricted source valueI would reserve some cases for release evaluation and keep them out of prompt development. These are version-controlled too. The runner can read the reserved cases and reference answers; the agent’s workspace exposes only its assigned task, data, and tools. Repeated feedback can make reserved cases familiar, so those need replacing over time.Run it from the pull requestIn GitHub, I would make this automatic for pull requests to main or master. Opening or updating an agent PR starts a job that loads the benchmark and grader from an approved, protected revision, prepares the test environment, and runs both agent versions with the same number of trials. Changes proposed in the agent PR cannot alter the benchmark used to judge that candidate.The runner saves the scores, gate results, and tool logs with the benchmark and grader versions, agent commit, prompt, model snapshot, tool versions, and data snapshot. Both runs use the same time, token, and memory limits, which the runner also records. Infrastructure can affect agent scores, as Anthropic’s February 2026 experiments demonstrated. [4]The job posts the comparison to the PR and returns a required status check. With branch protection applied to all contributors and bypass disabled, a failed check prevents merging. [5] The deployment job accepts only the evaluated artifact with a passing result; rebuilding or changing it requires another evaluation.Give each trial the same starting conditionsThe runner restores each trial to its declared starting state. Most cases begin with empty memory. A case that tests memory gives both versions the same saved history. The runner resets the test database and removes answers, cached retrieval results, tool outputs, and files left by earlier trials.The agent can retain state while completing a case over several turns. For timing and cost comparisons, the runner records whether caches start warm or cold and uses the same conditions for both versions. Provider prompt caching reuses computation for a shared prompt prefix; it does not return a saved answer. [6]In March 2026, Anthropic reported agents identifying BrowseComp and retrieving its answer key during evaluation. [7] For the metadata benchmark, I would keep reference answers outside every source the agent can search or query.I would run each case several times and retain every attempt. The runner averages the trials for each case, then compares the two versions case by case. It uses those paired results to estimate a 95% confidence interval for the overall score difference, keeping repeated trials grouped by case. If the interval includes zero, the report marks the improvement as uncertain. [8]Five attempts at each of 30 cases still cover 30 situations. I would inspect how often each case succeeded as well as its average score, and add more distinct cases when small differences are difficult to measure.The score controls the releaseA result might look like this:Benchmark: metadata-agent-v3Production: 64.7 / 100Candidate: 71.2 / 100Critical cases: 11 / 12 passedDecision: BLOCKEDReason: Restricted-column classification failedThe failed classification blocks this candidate despite its 6.5-point gain. The agent engineer fixes the problem and updates the PR, triggering another run.The release check also applies the approved minimum scores and regression limits for each area. A higher total cannot compensate for a category falling below its agreed requirement. Failed runs remain in the history, and a successful deployment records the benchmark run that passed.Keeping the cases, evaluation settings, and results with the release supplies evidence for the documented testing described in the NIST AI Risk Management Framework. [9]Replacing a saturated benchmarkRepeated scores above 90 would be my cue to review the capability benchmark. I would inspect the remaining failures and whether the cases still help me compare releases.A 2026 study found high saturation in 29 of 60 benchmarks. The researchers compared the gaps between leading models with the uncertainty in their scores to assess whether those benchmarks could still distinguish the models. [10]For v4, the SME and data engineer would add harder cases based on work the agent still struggles to complete. After approval, the runner evaluates both the production agent and candidate on v4 to establish a fresh comparison. Scores from v3 remain attached to that version.Generated scenarios could help find gaps. Bloom, for example, generates evaluation scenarios and simulated interactions. [11] I would use those to propose new cases, with the SME checking the expected outcome before the engineer adds them to an approved version. Routine release comparisons would continue to use fixed cases.If every version scores around 95, I would write harder capability cases. The checks for routine work and restricted data would keep running with every release.ReferencesHumanity’s Last ExamAnthropic: Demystifying evals for AI agentsAgentJudgeBench: Evaluating LLM Judges on Agentic Tool-CallingAnthropic: Quantifying infrastructure noise in agentic coding evalsGitHub Docs: About protected branchesClaude documentation: Prompt cachingAnthropic: Eval awareness in BrowseCompAnthropic: A statistical approach to model evaluationsNIST AI Risk Management Framework CoreWhen AI Benchmarks Plateau: A Systematic Study of Benchmark SaturationBloom: An open-source tool for automated behavioral evaluationsThis story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Benchmarking Agents Before Release was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI