Harness Engineering: Build the Runtime That Turns an LLM Into an Agent (Part 1/3)

Part 1: The Core — Agent Loop, Tools, and PermissionsThis is Part 1 of a 3-part series. Part 2: The Safety Net is about sandboxing and context. Part 3: The Memory is about skills, planning, and subagents.📦 All the code lives in one repo: coding-harness — roughly 600 lines of Python, no…

Part 1: The Core — Agent Loop, Tools, and PermissionsThis is Part 1 of a 3-part series. Part 2: The Safety Net is about sandboxing and context. Part 3: The Memory is about skills, planning, and subagents.📦 All the code lives in one repo: coding-harness — roughly 600 lines of Python, no frameworks.When an agent fails, we blame the model. Most of the time, it’s the harness.If you’ve used the OpenAI API, you already know the easy part: send messages, get a completion, maybe throw in function calling. What people don’t talk about enough is the gap between that and an agent that can actually sit in a repo, edit files, run tests, and recover when it screws up.That gap is the harness — the runtime around the model. Tools, permissions, sandboxing, context, memory, error recovery. The model is a brain that can only read text and write text. The harness is the program that loops, runs tools, keeps it safe, and talks to you in the terminal. The model is the engine. The harness is the car.An LLM by itself cannot read your files, run pytest, edit code, remember last Tuesday’s chat, or stop itself from running rm -rf /. It can only produce the next message. Everything else is the harness:Loop — keep calling the model until it stops asking for toolsTools — give it hands (bash, files, edits)Permissions — ask you before dangerous stuff; block the worst of itSandbox — make dangerous stuff physically fail at the OS (Part 2)Context — keep the conversation small enough to send (Part 2)Memory — skills, a to-do list, a session file you can resume (Part 3)Subagents — explore a codebase without stuffing the main chat (Part 3)ChatGPT is one round-trip: you ask, it answers. Claude Code or Cursor’s agent mode is different. You say “refactor this module,” and it reads files, runs tests, edits code, runs the tests again, and patches whatever it broke. That isn’t a smarter model. That’s a program around the model that knows how to loop, call tools, and keep going when something fails.People have started calling this harness engineering, which is a slightly pretentious name for a fairly obvious idea: Claude Code, Cursor, Codex aren’t just better models. They’re better harnesses.Here’s a session, because abstracts don’t help. You type: “Fix the failing test in test_parser.py.”Turn 1: LLM reads the test file → bash("cat test_parser.py")Turn 2: LLM runs the test to see error → bash("pytest test_parser.py")Turn 3: LLM reads the source file → read_file("parser.py")Turn 4: LLM edits the bug → str_replace("parser.py", old, new)Turn 5: LLM re-runs the test → bash("pytest test_parser.py")Turn 6: Tests pass → LLM says "Done! Fixed the off-by-one error in line 42."Six turns. Six decisions. You didn’t click anything after the first prompt. That’s the harness doing its job.You don’t need an ML background for this series. Python (functions, loops, dicts, try/except), a little OpenAI API, and basic shell (ls, grep, cat) is enough. If you’ve ever written openai.chat.completions.create(...), you’re fine.A few words before the code, because they show up constantly:LLM — a program that takes a list of messages and returns the next one. It does not remember previous API calls. If you want it to “remember,” you send the history again.Token — a chunk of text the model counts. Roughly ¾ of an English word. "Hello world" is about 2 tokens. A 100-line Python file is roughly 500.Context window — how many tokens it can see at once. GPT-4-class models around 128K; Claude around 200K. Sounds huge until an agent dumps grep -r into the chat 40 times.Tool / function call — the model does not run your code. It writes a request like {"name": "bash", "arguments": {"command": "ls"}}. Your Python runs ls and sends the output back.Stateless — every call_llm(messages) is independent. The messages list is the memory.I’ll walk through a working coding agent in ~600 lines of Python. No frameworks, four libraries (openai, rich, prompt-toolkit, pyyaml). The snippets are the actual code, not pseudocode. By the end, you should be able to look at Claude Code or Cursor and recognize the pieces.Clone it if you want to follow along:git clone https://github.com/shrinidhi-mahishi/coding-harness.gitcd coding-harnesspip install -e .Any OpenAI-compatible API works (OpenRouter, a local server, OpenAI itself). Create ~/.agents/env:BASE_URL=https://openrouter.ai/api/v1API_KEY=sk-or-...MODEL=deepseek/deepseek-v4-flashReal environment variables win over that file. Then coding-harness — that command is wired in pyproject.toml to coding_harness.agent:main. You type in the terminal; the inner loop runs until the agent has no more tool calls.Across the three parts:The agent loop — the heartbeatTools — giving the model handsPermissions — who gets to do whatSandbox — kernel-enforced safety (Part 2)Context engineering — keeping the window clean (Part 2)Skills, todos, sessions — making the agent remember (Part 3)Subagents — spending context somewhere disposable (Part 3)This post is the first three. Those are the bits that turn a chatbot into an agent.The files that matter here, if you want to peek:coding_harness/agent.py # two nested loopscoding_harness/llm.py # one API call + the system promptcoding_harness/tools.py # tool functions + defensive execute()coding_harness/permissions.py # allow / ask / denycoding_harness/config.py # BASE_URL, API_KEY, MODELThere is no database, no web server, no orchestrator library. The “framework” is two while True loops and a dict of tool functions. Read config.py → llm.py → agent.py → tools.py → permissions.py if you’re going in order.1. The Agent LoopEvery agent I’ve looked at — Claude Code, Cursor, Codex — is the same loop:Send the conversation to the modelIf it wants a tool, run itStick the result on the conversation and go back to step 1If there are no tool calls, you’re done — show the replyA chatbot does one round-trip. An agent keeps going until the job is done. That’s the whole difference. Everything else in this series is about not letting that loop fall over.One thing that trips people up: LLMs are stateless. They don’t remember the last turn. Every API call, you send the entire conversation. Messages look like this:messages = [ {"role": "system", "content": "You are a coding agent..."}, {"role": "user", "content": "Fix the failing test"}, {"role": "assistant", "content": "I'll start by reading the test file."}, {"role": "tool", "content": "... contents of test file ..."},]system — standing instructionsuser — youassistant — the modeltool — whatever a tool just returnedThe model reads the list and writes the next message. That’s its “memory.” There isn’t a second store hiding somewhere. If that list is wrong, the agent is wrong.There are two loops. The inner one is one user turn, with as many model turns as needed. The outer one just waits for you.The inner loop itself (simplified — the real file also shrinks old tool output, appends a fresh “reminder” at the end of the request, deletes temp files, and may compact; that’s Part 2):while True: message, usage = call_llm(messages) messages.append(message.model_dump(exclude_none=True)) if message.content: ui.agent(message.content) # No tool calls? The agent is done thinking. if not message.tool_calls: break # Execute each tool and feed the result back for tool_call in message.tool_calls: args, result = execute(tool_call) ui.tool(tool_call.function.name, args, result) messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": result, })📄 Source: agent.pycall_llm(messages) sends the whole history. The model comes back with text, tool calls, or both.model_dump() is just “turn the API object into a dict so we can keep it.” Nothing fancy.If there are no tool calls, we break. If there are, we run them, show them, append the results, and loop. The model sees what happened and decides whether to keep going.The outer loop — the one that waits for you — is even smaller:while True: user_input = ui.ask() if not user_input: break messages.append({"role": "user", "content": user_input}) # ... inner agent loop runs here ...You type. It goes on the list. The inner loop runs until the agent is done. Then we wait again. No state machine. Two nested while Trues.Same shape as Claude Code or Cursor. The model either talks or asks for tools. Tools feed back into messages. Repeat.2. Tools: Giving the Model HandsWithout tools, the loop is a chatbot with extra steps. Tools are how the model actually does things: read files, run commands, write code, search a repo.Picture asking a very smart person to fix a bug, except they’re in a locked room with no computer. They can think. They can’t touch anything. Tools are the door.And here’s the bit a lot of tutorials skip: the model never runs your tools. It writes a request — “please run this command” — and your code does the work, then sends the output back. The model doesn’t get a shell. It gets a form.A tool is four pieces:A name (bash, read_file)A description, so the model knows when to reach for itA JSON Schema for the arguments (required fields, types — the usual)A Python function that actually does the workbash is the powerful one, and the scary one. From tools.py:def bash(command: str) -> str: """Run a shell command and return its combined stdout and stderr.""" try: result = sandbox.run(command) except subprocess.TimeoutExpired as expired: return ( f"Timed out after {expired.timeout}s and was killed. " "Narrow it down - search inside the working directory rather than /." ) return (result.stdout + result.stderr) or "(no output)"The schema is what the model sees:{ "type": "function", "function": { "name": "bash", "description": "Run a shell command and return its combined stdout and stderr.", "parameters": { "type": "object", "properties": { "command": { "type": "string", "description": "The shell command to run", } }, "required": ["command"], }, },}In practice: the model wants a directory listing, so it emits {"name": "bash", "arguments": {"command": "ls -la src/"}}. We run it. We send the output back. It decides what to do next.There are seven tools. Same shape every time — a function and a schema:bash — run a shell command (through the sandbox, Part 2)read_file — read a filewrite_file — create or overwrite a filestr_replace — surgical edit: replace one string with anotherread_skill — load a SKILL.md runbook (Part 3)write_todos — replace the whole todo list (Part 3)task — spin up a subagent, get only its final answer (Part 3)bash is a thin wrapper: run via sandbox.run, 60-second timeout, return stdout+stderr — or a timeout message telling it to search inside the project, not /.Treat tool calls as untrusted inputThis is the part most “build an agent in 20 lines” posts skip. A tool call is text the model wrote. Same threat model as a form on a website. The name might not exist. The arguments might not be JSON. The types might be wrong. Any of that can kill the session if you let it.Models hallucinate. They invent function names. They send broken JSON. If you crash, the user starts over. If you catch it and send back a readable error, the model often just…retries correctly. They’re weirdly good at that, but only if you give them the error instead of a traceback.execute() is the defensive dispatcher, also in tools.py. It does this in order:Parse JSON — if broken, return "Error: arguments were not valid JSON"Look up the name — if unknown, list the real toolspermissions.check() — deny / ask / allowCall the Python functionCatch TypeError and anything else; return "Error: …", never a tracebackThat’s the dispatcher: route a tool call to a function without crashing the session.def execute(tool_call): name = tool_call.function.name# The model might send broken JSON try: args = json.loads(tool_call.function.arguments) except json.JSONDecodeError as broken: return {}, f"Error: arguments were not valid JSON ({broken})." # The model might hallucinate a tool name if name not in TOOLS: return args, f"Error: no tool named '{name}'. Available: {', '.join(TOOLS)}." # Run through the permission layer, then call the function try: action, reason = check(name, args) if action == "deny": return args, f"Blocked by policy: {reason}" if action == "ask" and not ui.approve(reason): return args, "The user denied this tool call." return args, TOOLS[name](**args) except TypeError as mismatch: return args, f"Error: wrong arguments for {name} ({mismatch})." except Exception as failure: return args, f"Error: {name} failed - {type(failure).__name__}: {failure}"Say it invents search_code. JSON is fine. Name isn’t in TOOLS. We return:"Error: no tool named 'search_code'. Available: bash, read_file, write_file, str_replace, read_skill, write_todos, task."Next turn it uses bash with grep -r "auth" .. Nobody crashed. That’s the whole point: never let a bad tool call take down the session. Errors are input.3. Permissions: Who Gets to Do WhatThe agent can run bash now. That’s useful, and also how you wake up to rm -rf / or your secrets posted somewhere.Let it run anything, and it can delete files, push garbage, or phone home. Ask on every command, and you’ll spend the afternoon approving ls. You need a middle.Three levels, like a traffic light:Level Meaning Examples 🟢 allow Run silently ls, grep, cat, git status, pytest 🟡 ask Pause for you mkdir, python script.py, npm install 🔴 deny Block even if you say yes rm, sudo, curl, wget, git pushRules are glob patterns — * means “anything.” Last match wins. From permissions.py:BASH_RULES = { "*": "ask", # default: ask for everything # read-only: let them through "ls*": "allow", "cat *": "allow", "grep *": "allow", "git status*": "allow", "git diff*": "allow", "pytest*": "allow", # risky: never, even if the user says yes "rm *": "deny", "sudo *": "deny", "curl *": "deny", "git push*": "deny",}* is the default: ask. Then we carve out the boring read-only stuff. Then we hard-block the dangerous ones. Everything else (python script.py, npm install) stays on ask.The model will try to get cute. grep foo | rm -rf / is allowed on the left and catastrophic on the right. So we split on |, ;, and && (respecting quotes) and take the strictest verdict:def decide(command): """Rate every part of a compound command; the strictest verdict wins.""" verdicts = [] for part in split_command(command): action = "ask" for pattern, rule in BASH_RULES.items(): if fnmatch(part, pattern): action = rule verdicts.append(action) for strictest in ("deny", "ask"): if strictest in verdicts: return strictest return "allow"grep foo | rm -rf / → deny (rm is deny)ls && mkdir test → ask (mkdir isn’t allowlisted)cat file.py | grep TODO → allow (both read-only)Permissions and sandboxing are not the same job. Permissions are UX: is this worth interrupting you? They are not security. A clever command can look like ls and not be. The sandbox in Part 2 is security: is this even physically allowed? You want both.What we have so farLoop, defensive tools, permissions. That’s already a coding agent. It can read a repo, run tests, edit files, ask before it does something dumb, and recover when it messes up a tool call.It is not safe — a determined model can still cause damage. It is not efficient — the context window will fill. And it has no memory — every session starts cold.Part 2 is the safety net: an OS sandbox so destructive actions fail at the kernel, and four layers of context management so long sessions don’t fall apart.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Harness Engineering: Build the Runtime That Turns an LLM Into an Agent (Part 1/3) was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Generative AI Pub — Published September 18, 2026 at 11:25 UTC — Category: Image AI