Your Agent Setup will Probably Fail in Production. Here is How to Avoid it.
How Pydantic helps with functions in tool calling in both ReAct and graph-based paradigms.image cred: https://unsplash.com/photos/white-and-black-polka-dot-textile-mcCAFvlzZFQThe original agentic framework: ReActThe term ReAct comes from the core steps in the framework, Reason + Act, which…
How Pydantic helps with functions in tool calling in both ReAct and graph-based paradigms.image cred: https://unsplash.com/photos/white-and-black-polka-dot-textile-mcCAFvlzZFQThe original agentic framework: ReActThe term ReAct comes from the core steps in the framework, Reason + Act, which basically means reasoning about a request then acting on it (or calling a tool to do it).Now let’s break down the steps from first principle with this diagram:Plain ReAct pattern created with excalidrawStandard ReAct pattern is very straight forward:Reasoning step is where the LLM decides to call a tool.Act is when the tool call happens, which could be either calling an API or Python function.Observe evaluates if end goal is met. If yes, then proceed. If no, then loop back to reason.Once LLM emits no more tools, the loop is conclude and the model generates the final output to user.However, this pattern is not nearly enough for production settings due to its lack of robustness.Production ReAct is largely the same, with the only difference being an intermediate control step between observe and final output.Production ReAct pattern created with excalidrawThe control step serves as an interceptor to capture critical failure modes that are crucial in production settings.While it is not required for the ReAct pattern to function, it is a non-negotiable in production environments.Here is what the control layer is responsible for:‘max_iteration’ parameter or a budget ceiling to avoid infinite loops.Without this, your API bill will become your worst nightmare.2. Pruning: Truncates, summarizes or filters tool outputs before it reaches the mode context or final output.This allows each loop to start “fresh” or have enough context window to work with in order to reduce cost and avoid context rot (performance starts to degrade after reaching 50% context window.3. Guardrails & EvalsType safety: Pydantic validates field types, regex patterns and required keys in payload. More on Pydantic later.Semantic filtering: detects prompt injection, jailbreaks and malicious usageFactual consistency: Use LLM-as-judge to evaluate if responses are grounded in retrieved info or hallucinated.4. Human-in-the-loop (HITL)Fallback gate when unhandled failures occur. Prompts user for manual intervention/correction rather than crashing or loop indefinitely.Why is it fragile and how Pydantic helpsNow going back on the type checks mentioned earlier, while type checks are not necessary, a lot like the control layer, it is an essential production standard as it serves as a safety net to catch type errors and maintain schema integrity to ensure the interaction between Python (or the program) and LLM is robust.Here is a chart of what a Pydantic workflow looks like:+-------------------------------+| Create Pydantic Schema/Model |+-------------------------------+ | v+-------------------------------+ (Passes standardized JSON format to LLM)| JSON Schema Payload | [ Semantic integrity & sanity checks: ]+-------------------------------+ [ regex, database ID, cross-field rules ] | v+-------------------------------+| Downstream Agentic Tool Call |+-------------------------------+Think of it like a two-step verification: Pydantic declares rules and format that the LLM must follow.After the LLM generates output in respect to those constraints, Pydantic then validates it again on the same rules and format before passing it to downstream tasks.Now you might ask, how is this relevant to ReAct?Where does Pydantic Live in ReAct?Let’s use the regular ReAct pattern for sake of simplicity:Where the red arrows are pointing is where Pydantic is executed.Before we break down what each arrow does, there is an important mental model to keep note of:Outbound: Python → LLMInbound: LLM → PythonOutbound must happen before inboundNow with this mental model in mind, let’s take a closer look.Outbound ContractThe user query lives on the program level (Python) and gets passed to the LLM for reasoning, making this an outbound process.Inbound ValidationAfter reasoning, the LLM reasoning must be translated and validated before passing it to Act for tool calls. This is an inbound from the LLM to the program.Structured OutputThe last part where it happens right before the final output. This is like a self-contained process, where instead of intercepting the output and formatting it into raw text and passing back to the reasoning step, it goes down to the final output, where it summarizes the request with LLM and ultimately passing it back down to the program level for display. This is still consistent with the theme of outbound → inbound.A small detail worth pointing out: even though LLM is what generates the text, it must be passed to the program level on an interface for the user.Now that we know how to construct a production-grade agent framework, let’s see how to design a modern version for more complex workflows.Why ReAct is obsolete and why graph-based orchestration is superiorPure ReAct is simple and intuitive, however, it is replaced by graph-based orchestrations due to its lack of robustness, inability to handle complex, long-running tasks.Here are some reasons why graph is superior:Control Flow: System-driven as opposed to LLM-driven, each state dictates different tasks with valid transitions and retries.Robustness: Guardrails and fallback pathways in the graph edges prevents infinite loops and hallucinated tool calls.Task Complexity: Instead of processing tasks linearly, it is highly capable of parallel subtasks and multi-agent coordination.State Management: Stateful database and memory allows for thread persistence and scope context rather than linear-growing chat log.How does Pydantic look in graph-based framework?The Inbound/Outbound structure remains identical.However, state becomes a Pydantic model to type-check graph’s shared memory, variables and history. On top, node-to-node contracts require sub-schemas. Transition control uses Pydantic state fields instead of hardcoded if/else and while loops.Below is a diagram showing where Pydantic lives in a graph agentic framework:graph agentic frameworkAs you can see with the red arrows, it is not so different from the ReAct pattern; just with more intermediate steps.To boil it all down, Pydantic transitions from a tool parameter validator into a global runtime state coordinator.The problem with standard tool calling and why the industry shifted to programmatic callingAs you can probably guess already, programmatic tool calling is exactly what it sounds like, it writes programs (or scripts) inside a sandbox to perform tasks in order to eliminate intermediate steps and handles multi-step workflows much more efficient.Here is what the difference looks like between standard vs. programmatic, respectively:LLM output formatStructured JSON payload vs. raw executable code blockHow tools are invokedruntime parses JSON and invokes the Python callable via dispatch vs. LLM writes code that imports/calls functions and prints results inside a sandboxA quick rundown on how Pydantic works in Programmatic Tool Calling:[ 1. OUTBOUND CONTRACT ]Python exposes raw Pydantic Class code inside the System Prompt / Context │ ▼[ 2. GENERATION STEP ]LLM writes an entire executable Python script referencing those Pydantic models │ ▼[ 3. SANDBOX INGESTION ]Script is passed into an isolated environment (Docker / REPL / Pyodide) │ ▼[ 4. INBOUND RUNTIME EXECUTION ]• Script instantiates Pydantic models• Invalid fields trigger Python `ValidationError` tracebacks inside the sandbox• Script executes native control flow (loops, data joins, transformations) │ ▼[ 5. OBSERVE / STDOUT CAPTURE ]Standard output, return artifacts, or execution tracebacks are captured and returned to LLMOutboundPydantic model contains the data contracts and class attributes so that the LLM can reference the system prompt and the guidelines in declared in the Pydantic model to generate dynamic Python scripts to represent the classes and perform the required task.Error ResolutionBoth methods re-invoke the LLM when an error occurs. The only difference being: standard calling wraps a single failure in a multi-step chain to prompt a retry, whereas programmatic calling allows an agent to write to its own own “try/except” block and handle exceptions without having to invoke the LLM, except when it encounters an unhandled exception which will be wrapped into a raw “stderr” traceback to the LLM with full execution/error context so the LLM can fix the error in one-shot.Latency & ComplexityStandard → O(K) for K number on steps. Programmatic → O(1) because of code-as-action writes a single script that contains a loop that can execute on the program level.Which is more expensive?While standard calling requires sequentially more LLM API calls, it does not require any infrastructure. On the other hand, programmatic calling can perform multiple tasks with just running a single code block, with the tradeoff of infrastructure overhead (hosting and maintaining snadbox runtimes like Docker).This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Your Agent Setup will Probably Fail in Production. Here is How to Avoid it. was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI