Agent Compute Broker: Stop Using Your Best AI Model for Every Task
The next step for AI coding agents may not be choosing a better model, but using a scheduler that decides when advanced reasoning is worth the cost and when to fall back to cheaper models or deterministic tools.Most sophisticated AI-agent workflows have an inefficient default: once you choose a…
The next step for AI coding agents may not be choosing a better model, but using a scheduler that decides when advanced reasoning is worth the cost and when to fall back to cheaper models or deterministic tools.Most sophisticated AI-agent workflows have an inefficient default: once you choose a model, that model tends to handle almost everything.The same expensive frontier model that makes a difficult API-design decision may also spend tokens listing files, searching for call sites, summarizing logs, transforming data, and formatting JSON.The reverse problem happens too. A cheaper model may receive a genuinely difficult debugging or architecture problem, make several unsuccessful attempts, and only then get escalated to something stronger.Both approaches waste compute.A better architecture is an Agent Compute Broker: a scheduling layer that breaks a workflow into tasks, estimates how much reasoning each task deserves, sends it to the appropriate model or deterministic tool, verifies the result, and escalates only when necessary.Instead of asking:Which model should run this agent?the system asks:Which parts of this job actually require expensive intelligence?That distinction becomes increasingly useful as agent platforms expose more control over reasoning effort, task budgets, subagents, tools, and context.The problem with one-model-for-everything agentsConsider asking an AI coding agent to migrate a 40-module monorepo to a new framework version.The job might contain hundreds of individual operations:inventory packages;identify deprecated dependencies;find every caller of a changed API;edit configuration files;apply repetitive syntax migrations;compile the repository;cluster compiler errors;investigate obscure runtime failures;make compatibility decisionsrerun tests.These tasks do not have equal reasoning requirements.Finding every reference to a symbol is mostly an information-retrieval problem. If the language server can answer it exactly, asking a frontier model to reason through the repository is both slower and less reliable.Changing 70 identical import statements is largely mechanical.Deciding whether a framework’s new request lifecycle breaks an internal abstraction is different. That may require understanding documentation, code history, architectural constraints, backward compatibility, and several competing implementation choices.Yet a conventional agent often treats all three operations as roughly the same kind of model invocation.The result is an expensive form of resource allocation: premium reasoning is spent where deterministic computation would work, while difficult reasoning is not always recognized early enough.What an Agent Compute Broker doesThe broker sits between the user’s objective and the agents that execute it.Before execution, it converts the plan into a task graph, usually a directed acyclic graph, or DAG, where nodes represent pieces of work and edges represent dependencies.For each node, the broker estimates attributes such as:reasoning difficulty — how much judgment or inference is required;failure cost — what happens if the answer is wrong;context requirement — how much repository or conversation state must be understood;verification strength — how easily the result can be checked automatically;expected token cost;expected latency.The scheduler then chooses an execution strategy.A simplified migration might look like this:repository inventory ↓inexpensive agentfind 87 callers of changed API ↓language server / programmatic searchapply mechanical edits ↓mid-tier coding agentdecide compatibility strategy ↓frontier reasoning agentrun build and tests ↓deterministic toolingclassify 23 failures ↓code + inexpensive modelinvestigate 2 unexplained failures ↓frontier reasoning agentThe important point is that an LLM does not need to execute every node.Sometimes the best agent is grep. Sometimes it is a compiler. Sometimes it is a short Python program. And sometimes it is the most capable reasoning model available.Why this is becoming a practical needClaude’s current developer stack contains several pieces that make this architecture more feasible.The Claude Agent SDK provides the agent loop and built-in tools for operations such as reading files, editing code, running shell commands, searching repositories, connecting to MCP servers, and delegating work to subagents.Recent Claude models also expose effort controls that let developers trade additional reasoning depth and token consumption against latency and cost. Anthropic explicitly recommends recalibrating effort levels against your own evaluations instead of simply carrying settings over from an older model.That is important for a compute broker. Model choice no longer has to be the only scheduling variable.A broker could potentially choose:cheap model + low effortmid-tier model + medium effortstrong model + normal effortfrontier model + very high effortdepending on the task.Anthropic has also introduced task budgets for newer agentic workloads. These let a model know approximately how much token budget is available across an agentic task so it can pace its work accordingly. They are different from max_tokens: task budgets guide the agent, while max_tokens remains a hard generation ceiling. Anthropic currently describes task budgets as a beta capability and recommends using them selectively rather than constraining every open-ended task.Together, model tier, effort, and task-budget controls start looking less like ordinary inference parameters and more like resources a scheduler can allocate.Programmatic Tool Calling is the other half of the ideaModel routing alone does not solve the efficiency problem.Suppose an agent needs to:retrieve 500 compiler errors;group them by error code;count occurrences;find affected packages;return only the unusual clusters.One implementation makes hundreds of tool calls and returns every intermediate result to the model.That is expensive because the model repeatedly wakes up, reads results, reasons about them, and decides what to call next.But none of those steps requires deep reasoning.Anthropic’s Programmatic Tool Calling attacks exactly this problem. Claude can write code that invokes tools, loops over results, filters data, performs aggregation, and controls which information ultimately enters the model’s context. Anthropic notes that conventional multi-tool workflows can otherwise require repeated inference passes while filling the model’s context with intermediate data.For a compute broker, this creates three execution categories rather than two:deterministic software ↓programmatic tool orchestration ↓model reasoningThe broker’s job is to push a task as far down that stack as safely possible.If Python can compute it, use Python. If a smaller agent can reliably classify it, use the smaller agent. Reserve expensive reasoning for the remaining uncertainty.Verification makes aggressive routing possibleThere is an obvious risk. What if the broker underestimates a task?A cheap model attempts something difficult, produces a plausible but incorrect patch, and the system silently continues. That is why routing and verification have to be designed together.The broker should ask not only:Can a cheaper model probably perform this task?but also:Can I cheaply detect whether it failed?Consider two tasks.Task A: mechanical refactoringA cheaper coding agent modifies 40 call sites.Verification is strong:compile→ run unit tests→ check static analysis→ verify old API has zero remaining referencesIf all checks pass, there may be little value in paying a frontier model to perform the edit.Task B: choosing an authentication architectureAutomated verification is much weaker. Several designs may compile and pass tests while still differing substantially in security, maintainability, or compatibility. The failure cost is also higher. That task deserves stronger reasoning from the beginning.This leads to a useful scheduling principle:The easier a result is to verify, the more aggressively you can route execution toward cheaper compute.Escalation should be automaticRouting does not need to be perfect on the first attempt. The broker can treat model selection as an escalation ladder.For example:mechanical edit ↓mid-tier agent ↓tests fail ↓mid-tier agent retries with diagnostics ↓tests fail again ↓frontier debugging agentOther escalation signals might include:repeated tool errors;contradictory evidence;low classifier confidence;unusually large code changes;failed tests;unexpected performance regressions;unresolved ambiguity;excessive retries;human rejection.The system could eventually learn that particular task classes almost always escalate.If “database migration planning” reaches the frontier model 92% of the time, there is little reason to route it cheaply first.If “rename generated DTO fields” succeeds with the inexpensive tier 99.8% of the time, the broker can confidently keep it there.Routing becomes an empirical optimization problem rather than a collection of hand-written assumptions.The broker can learn from its own workloadThe first version does not require sophisticated machine learning.A simple classifier might use:task_typefiles_touchedestimated_context_sizepresence_of_test_oraclehistorical_success_rateprevious_attempt_countexpected_failure_costand produce:route = cheap | standard | frontier | deterministicOver time, execution telemetry provides training data.For every task node, record:selected model;effort level;input and output tokens;latency;tool-call count;retries;verification outcome;escalation;human intervention;final success.Now the broker can answer useful questions.Which tasks are consistently over-provisioned?Which tasks fail when routed cheaply?Does higher reasoning effort actually improve success for debugging tasks?At what point does a retry cost more than immediate escalation?Which verification methods allow safe use of smaller models?The optimization objective does not have to be “minimize tokens.”It could be:minimize costsubject to ≥ 95% success rateor:minimize elapsed timesubject to a fixed compute budgetor even:maximize expected task successsubject to $5 total costand < 10 minute latencyThat is why “broker” is a more useful mental model than “router.” A router chooses a model. A broker allocates a constrained resource.A realistic prototypeA first implementation could deliberately stay small.Support four task classes:search;mechanical edit;debugging;architecture.Use only two model tiers. Add deterministic tools for search, builds, and tests.Then define a simple policy:search→ tools or cheap modelmechanical edit→ cheap model→ escalate after verification failuredebugging→ cheap model if strong tests exist→ frontier after two failed attemptsarchitecture→ frontier model immediatelyNext, evaluate it against a baseline:Baseline: run every task with the strongest model.Use 30 historical engineering tasks and measure:final task success;total token cost;wall-clock time;number of model calls;number of retries;number of escalations;human interventions.A compelling result does not require dramatic intelligence gains. If the broker retains roughly 95% of the strongest-model baseline’s task success while substantially reducing compute spend, that is already valuable. Then add more sophistication only where the data justifies it.The biggest challenge is classification, not orchestrationBuilding the scheduler is relatively straightforward. Knowing when not to economize is harder.A five-line change can hide a difficult semantic decision. A 300-file refactor can be almost completely mechanical. Task size is therefore a poor proxy for reasoning difficulty.The broker needs signals about uncertainty, reversibility, and verification. It also needs to avoid spending more compute deciding where to spend compute than the original task would have consumed. The classifier itself should be cheap.For obvious cases, rules may be better than models:symbol-reference search → LSPtest execution → shellJSON transformation → codeOnly ambiguous routing decisions need model-based classification.Model selectors are for users. Compute schedulers are for systems.Most AI coding tools expose model choice as a user preference.Pick the fast model. Pick the smart model. Pick the expensive one when the task looks difficult. That works when a “task” means one chat request. It becomes less sensible when one request expands into 300 heterogeneous operations.The human does not know in advance which four nodes of that workflow will require exceptional reasoning. The agent discovers that during execution. So the natural place to make the decision is inside the runtime.The engineer should be able to specify something closer to:Quality target: highMaximum budget: $8Latency preference: balancedThe broker should decide how to spend that budget. That turns model intelligence into something resembling compute infrastructure.We already schedule CPUs, GPUs, memory, database queries, and distributed jobs according to workload characteristics. Agentic systems may need the same abstraction for reasoning.The important question may no longer be:Which model should power this agent?It may become:Where, inside this workflow, is expensive intelligence actually worth buying?SourcesAnthropic, Claude Agent SDK overview — current SDK capabilities including built-in tools, subagents and MCP.Anthropic, Prompting best practices — current guidance on adaptive thinking, effort calibration and agent behavior.Anthropic, Migration guide for current Claude models — effort levels, task budgets, and cost/latency recalibration guidance.Anthropic Engineering, Introducing advanced tool use on the Claude Developer Platform — Tool Search and Programmatic Tool Calling.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Agent Compute Broker: Stop Using Your Best AI Model for Every Task was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI