What Building Agents for Physical Systems Taught Me About LLM Design
A practitioner’s notes on the constraints you don’t feel until the AI can break something real.Most of what’s written about LLMs and agentic AI comes from people building for software workflows, email triage, code review, document summarization, research assistants. In that world, when an agent…
A practitioner’s notes on the constraints you don’t feel until the AI can break something real.Most of what’s written about LLMs and agentic AI comes from people building for software workflows, email triage, code review, document summarization, research assistants. In that world, when an agent gets it wrong, someone rewrites the summary, undoes the pull request, or ignores the recommendation. The blast radius is small and mostly cognitive.I’ve spent the last couple of years building AI agents for a very different world. One where the agent’s output influences chemical dosing, temperature setpoints, and the health of equipment that costs millions and takes months to replace. In that world, the blast radius isn’t cognitive. It’s physical.That shift changes a lot about how you think about LLMs. Some of it is obvious, some of it is not. This is my attempt to write down the not-obvious parts, the design instincts I’ve had to rewire since moving from software AI to industrial AI.The mental model shift: an LLM isn’t an oracleThe single most useful reframing I’ve come to is this: an LLM is a probabilistic control system sampling from a learned distribution. It is not an oracle.That sounds like a small semantic move, but it changes almost every downstream design decision.If you treat an LLM as an oracle, you build around its answers. You design UIs that surface the answer, workflows that route on the answer, agents that hand off based on the answer. You spend your time trying to make the answer better, better prompts, better context, better models.If you treat an LLM as a probabilistic control system, you build around its distribution. You ask what the model does when it’s uncertain, what it does when the input is out of distribution, what happens when it’s sampling near the edge of what it knows. You design for the shape of the wrongness, not just the correctness.In a software product, treating an LLM as an oracle usually gets you to a working demo faster. In a physical-consequence product, it gets you to a shipped feature that will eventually recommend an operating change no one can explain, on a day the input data was slightly noisier than usual, in front of an operator who now doesn’t trust anything the system says.Faithfulness is a design constraint, not an aspirationThe second thing I’ve had to rewire is what I want the LLM to actually do in the product.In most software AI products, the LLM is the reasoning engine. It reads context, decides what to say, and generates language that reflects that decision. The generation is the intelligence. That’s the whole point.In an industrial context, I’ve come to want the opposite. I want the reasoning to happen in a structured, inspectable layer, evidence scoring, causal traces, confidence values, and I want the LLM to only narrate what that structured layer produced. Not add to it. Not soften it. Not fill in gaps with plausible-sounding filler. Just say, in plain language, what the structure already says.This is what I’ve started calling the faithfulness principle. If it’s not in the structured object, it doesn’t go in the narrative.The reason is straightforward: when an operator reads an AI recommendation and asks “why?”, the answer they get needs to be the actual why. Not a why the LLM assembled from adjacent plausible reasoning. On a plant floor, the difference between the two is the difference between a system people act on and a system people quietly stop trusting.I don’t think this constraint is unique to industrial AI. I think it’s just felt more sharply there because the cost of drift is higher. Software AI products would probably benefit from this discipline too. They just don’t get punished for skipping it.Errors don’t add, they compoundThe third rewiring came from watching agent pipelines behave in ways I hadn’t fully expected.If you build a five-skill agent, say correlation analysis, then graph reasoning, then hypothesis generation, then evidence linking, then ranking, and each step is 90% reliable, your intuition wants to say the whole pipeline is 90% reliable. It isn’t. It’s closer to 59%. That’s the arithmetic of chained probabilities, and it’s brutal.In a software workflow, this often doesn’t matter, because a human is reading the final output and can catch a bad chain. In an industrial workflow, if that chain terminates in an operating recommendation, the compounding is what kills you. Each step looked reasonable in isolation. The result is confidently wrong.What this taught me is that multi-step agent design is fundamentally different from single-turn LLM design. You can’t just concatenate skills and hope the reliability holds. You have to design checkpoints into the pipeline itself, places where the agent pauses if the input to the next step doesn’t meet a quality bar, or the output of the previous step is too uncertain to build on.I’ve started thinking of these checkpoints not as safety features but as first-class states in the agent’s state machine. The pipeline has to know how to not finish. That’s harder to build than it sounds, because the demo-friendly instinct is always to make the agent complete its run. The industrial-friendly instinct is to make the agent comfortable stopping halfway.Autonomy is a dial, not a switchThe fourth thing I’ve come to believe is that AI autonomy is granted on a ladder, not a switch.Most agentic AI conversations frame autonomy as binary. Either the human is in the loop, or the agent runs autonomously. I don’t think that framing survives contact with a system that has physical consequences.The way I’ve come to think about it, autonomy has three stages. In the first, the agent recommends, and a human approves before anything happens. In the second, the agent executes, but a human reviews the action afterward, with the ability to unwind it. In the third, the agent executes autonomously, but only after the first two stages have generated enough evidence that the reliability is real, and only when a rollback path exists if it isn’t.The mistake I see teams make, including versions of my earlier self, is trying to jump to stage three because it demos well. Stage three is where the money is. Stage three is where the “agentic” story lands. But you can’t get to stage three responsibly without earning it through stages one and two.The teams building reliable AI agents aren’t the ones that removed humans the fastest. They’re the ones that knew, precisely, when it was safe to remove them.What this changes about how I spec a featureAll of this has changed how I write specs for AI features. Some of the changes are small; some are structural.I spend more time defining what the feature does when it doesn’t know. What does the agent do when the data is thin? When two pieces of evidence conflict? When the confidence score sits in the ambiguous middle? These used to be footnotes at the end of a spec. Now they’re often the longest section.I spend more time on the state machine than on the happy path. In a software AI product, the happy path is usually 80% of the design work. In an industrial AI product, the happy path is straightforward, and the edge cases are the design work.I spend more time on how the system shows its work than on how it generates the answer. The generation is table stakes. The provenance, the evidence trail, the ability for an engineer to trace why the agent said what it said, that’s the product.And I’ve stopped treating the LLM as the interesting part of the system. In most of the products I’m building, the LLM is the last mile, the surface that translates structured reasoning into human language. The interesting part is the reasoning that happens before the LLM ever sees the prompt.What I think this means more broadlyI don’t think the lessons above are specific to industrial AI. I think industrial AI just surfaces them earlier and more painfully than most other domains, because the feedback loop from a bad output to a real-world consequence is short and expensive.The software AI world will eventually run into the same walls. As agents move from summarizing documents to executing multi-step actions with real stakes, booking travel, moving money, filing documents, negotiating on behalf of a user, the same constraints will start to bite.Faithfulness, error compounding, autonomy laddering, designing for the moment the agent doesn’t know. These aren’t industrial concerns. They’re the general concerns of any AI system whose outputs affect the world.The industrial teams are just further down that road, because we’ve been building for consequential outputs from the start.If I had to summarize what building for physical systems has taught me about LLM design, it’s this: the interesting engineering isn’t in making the model smarter. It’s in designing the system around the model so that the model’s occasional wrongness doesn’t propagate into the world unchecked.That’s a less glamorous story than “we made the AI smarter.” But I think it’s the one the next generation of agentic AI products will actually live or die on.If you’re building agents in this space and thinking about the same problems, I’d genuinely like to hear how you’re approaching them.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!What Building Agents for Physical Systems Taught Me About LLM Design was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI