Harness Engineering: Build the Runtime That Turns an LLM Into an Agent (Part 3/3)

Part 3: The Memory — Skills, Todos, Subagents, and the Full PictureThis is Part 3 of a 3-part series. Part 1: The Core was the loop, tools, and permissions. Part 2: The Safety Net was sandboxing and context.📦 All the code lives in one repo: coding-harness — roughly 600 lines of Python, no…

Part 3: The Memory — Skills, Todos, Subagents, and the Full PictureThis is Part 3 of a 3-part series. Part 1: The Core was the loop, tools, and permissions. Part 2: The Safety Net was sandboxing and context.📦 All the code lives in one repo: coding-harness — roughly 600 lines of Python, no frameworks.Part 1 was a working agent. Part 2 made it something you could actually leave running. This part is the difference between “it can code” and “it can stay on a problem.”Without memory, every task starts from zero. No playbook, no plan that survives a long session, no way to pick up after a restart. And if it needs to understand a codebase, it dumps 30,000 tokens of grep into the main window just to write two sentences.Four pieces: skills (instructions that stick around), todos (a real plan), sessions (the diary), subagents (exploration you can throw away).Files for this part:coding_harness/skills.py # SKILL.md discovery + read_skillcoding_harness/todos.py # plan that lives outside the chatcoding_harness/session.py # append-only JSONL + rewindcoding_harness/subagent.py # disposable explorer (the task tool)6. Skills, Todos, and SessionsSame word — memory — three different clocks.Skills — permanent, like a recipe bookTodos — one task, like a checklistSessions — across restarts, like a diarySkills — instructions that don’t live in every promptA skill is a SKILL.md file. Recipe cards: the cook knows the names, and only opens a card when they’re making that dish.YAML at the top (metadata between --- lines), Markdown below. Same pattern as a lot of static sites and docs tools:---name: deploydescription: Deploy the app to production using Railway---## Steps1. Run the test suite first2. Build the Docker image3. Push to Railway with `railway up`4. Verify the deployment URLWe look in two places (skills.py):.agents/skills/ in the current project~/.agents/skills/ for you personally, across projectsSKILL_DIRS = [ Path.cwd() / ".agents" / "skills", # project-level Path.home() / ".agents" / "skills", # user-level (shared across projects)]def find_skills(): skills = {} for directory in SKILL_DIRS: for path in sorted(directory.glob("*/SKILL.md")): _, frontmatter, _ = path.read_text().split("---", 2) meta = yaml.safe_load(frontmatter) skills[meta["name"]] = {"description": meta["description"], "path": path} return skillsAt startup we only read the header — name and description — and put those in the system prompt. The full text loads when the model calls read_skill. That’s progressive disclosure.20 skills × 500 words is 10,000 words sitting in every conversation whether you need them or not. With progressive disclosure, the prompt just lists names. You pay for the runbook when you actually deploy.How it plays out:System prompt: Available skills: deploy ("Deploy the app to production using Railway")You say: “Deploy this to production”Model calls read_skill("deploy")Gets the stepsRuns tests, builds, pushesThose instructions only lived in this conversationCursor and Claude Code do the same thing. You can put anything in a skill: deploy runbooks, review checklists, migration notes, how you like tests written.Todos — the plan should not live in the chatFor anything with more than one step, the model writes a plan. The important choice: the plan is not a message. It’s a global that we re-inject every turn (todos.py).Why? Compaction, from Part 2. If the plan is sitting in turn 3 and we compact at turn 40, that message can disappear. A variable plus late injection survives that.MARKS = {"pending": "[ ]", "in_progress": "[~]", "done": "[x]"}TODOS = []def write_todos(todos): """Replace the whole list. Exactly one task may be in_progress.""" active = [t for t in todos if t["status"] == "in_progress"] if len(active) > 1: return f"Error: {len(active)} tasks are in_progress. Only one may be." TODOS[:] = todos return todos_prompt() or "Todo list cleared."The model sends the whole list every update. One rule: exactly one task in_progress.Without that, models “multitask” — three half-done things, none finished. One-at-a-time is annoying in a good way. Finish A, then B. Same reason you don’t keep six tabs of half-written emails.Three fields:{ "content": "Fix the parser", # imperative: what to do "activeForm": "Fixing the parser", # present continuous: for the spinner "status": "in_progress" # pending | in_progress | done}activeForm is a small thing I like: the spinner says “Fixing the parser…” instead of “thinking…”Mid-task it looks like this:[x] Set up the test fixtures[~] Fix the parser to handle nested brackets[ ] Update the documentation[ ] Run the full test suiteFixtures done, parser in progress, two waiting.Anything that has to survive compaction — todos, file-change tracking, git branch — lives outside the transcript and gets re-injected. Don’t trust the chat log to remember the plan.Sessions — tomorrow morningClose the terminal, come back, continue. Without sessions you’d lose the whole thing. This is the diary.We write append-only JSONL — JSON Lines: one JSON object per line. Append to save, read line by line to load (session.py):SESSION_DIR = Path.home() / ".agents" / "sessions" / PROJECTdef save(messages): """Append what is new. Never rewrite what is already on disk.""" global WRITTEN with path_for(CURRENT).open("a") as f: for message in messages[WRITTEN:]: f.write(json.dumps(message) + "\n") WRITTEN = len(messages)We never rewrite the file. Crash mid-write and you lose at most one line, not the chat. WRITTEN is “how many messages are already on disk,” so messages[WRITTEN:] is only the new ones.Rewind isn’t delete. It’s a marker:def rewind_to(count): with path_for(CURRENT).open("a") as f: f.write(json.dumps({"rewind_to": count}) + "\n")The agent went down a hole. You jump back. Old lines stay on disk; load ignores everything after the marker. Undo, not erase.def load(session_id): messages = [] for line in path_for(session_id).read_text().splitlines(): entry = json.loads(line) if "rewind_to" in entry: del messages[entry["rewind_to"]:] elif "compacted" in entry: messages = list(entry["compacted"]) else: messages.append(entry) return messages"compacted" is how Part 2’s compaction survives a restart: the summarized conversation replaces the old messages on load.One file per chat under ~/.agents/sessions//. No database, no migrations. For a personal tool, that’s the right amount of machinery.On disk, the whole harness is just a few paths:~/.agents/env — BASE_URL, API_KEY, MODEL, optional CONTEXT_WINDOW~/.agents/skills/*/SKILL.md — your personal skills/.agents/skills/*/SKILL.md — project skills~/.agents/sessions// — JSONL transcriptstemp dir — sandbox profile and spill files (short-lived)The git repo of your project is what the agent works in. The harness itself is just the installed Python package.7. Subagents: Spend Context Somewhere DisposableThis is my favorite part of the whole thing.You need to know how auth works. You grep, you read files, you follow imports, you read more files. You now have 30,000 tokens of tool output in context to produce a 200-word summary.Those 30k tokens sit there forever after, making every later call noisier and more expensive. Compaction will clean it up eventually — another LLM call, and you’ve lost the raw detail anyway.So: do the exploration in a different window. Throw that window away. Keep the answer. That’s a subagent.You’re the manager. You don’t go to the library yourself and dump a stack of printouts on your desk. You send someone. They come back with a paragraph. Their notes go in the bin.Four rules, in subagent.py:Empty history — doesn’t know the parent chat, so the question has to stand aloneAlmost every tool — no nested task (no infinite agents), no write_todos (not their plan), no writes (str_replace / write_file). It reads. You edit.Same loop as the main agent — same permissions, same sandboxOnly the last message comes back — cap of 12 turns, then a partial reportMAX_TURNS = 12WITHHELD = {"task", "write_todos", "str_replace", "write"}def task(description: str) -> str: """Run a fresh agent on one question and return only its final answer.""" messages = [ {"role": "system", "content": SUBAGENT_SYSTEM_PROMPT}, {"role": "user", "content": description}, ] report = None for _ in range(MAX_TURNS): message, usage = call_llm(messages, tools=toolset()) messages.append(message.model_dump(exclude_none=True)) report = message.content or report if not message.tool_calls: return report or "(the subagent came back with nothing)" for tool_call in message.tool_calls: args, result = execute(tool_call) messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": result, }) return f"(stopped after {MAX_TURNS} turns. Partial findings:)\n\n{report}"return report is the whole trick. The messages list — greps, file bodies, 30k tokens — goes out of scope. Python collects it. You keep a short string.From the main agent’s point of view:Main agent: "I need to understand how auth works in this codebase." → calls task("How does authentication work in this project? Find the main auth files and explain the flow.") Subagent: (runs grep, reads 8 files, follows imports, 12 turns, 30K tokens) → returns: "Auth uses JWT tokens via middleware in src/auth/middleware.py. Login flow: POST /login → verify_password() in auth/users.py → generate_token() in auth/jwt.py. Token validated on every request by the require_auth decorator (line 45 of middleware.py)."Main agent: (sees only the 50-word summary, context still clean) → continues with the actual taskWe tell the subagent, in the system prompt:You are an exploration subagent. Only your final message crosses back, so it has to stand on its own. Keep it under 150 words. Findings only: file paths with line numbers, names, values. No preamble, no restating the question.Compaction throws context away after you spent it. A subagent spends it somewhere that was always meant to be thrown away. The main transcript never pays.The Full ArchitectureAll three parts, in one ugly-but-honest diagram:User Input │ ▼┌──────────────────────────────────────┐│ agent.py ││ ┌────────────────────────────────┐ ││ │ Inner Loop │ ││ │ │ ││ │ context.py ─► late injection │ ││ │ │ │ ││ │ ▼ │ ││ │ messages ──────► llm.py │ ││ │ ◄──────── call_llm() │ ││ │ │ │ ││ │ tool_calls? ──► tools.py │ ││ │ │ execute() │ ││ │ │ │ │ ││ │ │ ┌────────┼────────┐ │ ││ │ │ │ bash ────────┼──┼──┼──► sandbox.py│ │ │ │ read_file │ │ ││ │ │ │ write_file │ │ ││ │ │ │ str_replace │ │ ││ │ │ │ read_skill ────┼──┼──┼──► skills.py│ │ │ │ write_todos ───┼──┼──┼──► todos.py│ │ │ │ task ──────────┼──┼──┼──► subagent.py│ │ │ └────────┼────────┘ │ │ (own loop)│ │ │ ▼ │ ││ │ │ permissions.py │ ││ │ │ check() │ ││ │ │ │ ││ └───────┼────────────────────────┘ ││ │ ││ history.py ◄── cap / strip / drop ││ compact.py ◄── when context full ││ session.py ◄── save after changes ││ prompt.py / ui.py ── input + display│└──────────────────────────────────────┘ │ ▼Terminal Output (ui.py)One task, all the way throughYou type: “Add input validation to the create_user endpoint.”Turn 1 — plan. write_todos:[ ] Read the current create_user endpoint[~] Identify what validation is missing[ ] Add validation logic[ ] Write tests for the validation[ ] Run the test suiteTurn 2 — explore. task("Find the create_user endpoint and list what parameters it accepts"). Subagent reads around and comes back: “create_user is in src/routes/users.py line 34. Accepts: name (str), email (str), age (int). No validation currently.”Turns 3–5 — edit. Sandboxed bash("cat src/routes/users.py"), then str_replace, then a test file. cat is silent. Writes get an ask.Turn 6 — test. bash("pytest tests/test_users.py") is allowlisted. Tests pass. Todos go to done.In the background: old tool output is already 300-char stubs. The subagent’s 20k tokens are gone. Late injection still has branch and time. Session is on disk.That’s all seven primitives, in ~600 lines.Same thing as a call stack, if the box diagram made your eyes glaze over. This is what happens on every user message:You type │ ▼agent.py (outer loop) │ append user message ▼agent_loop (inner loop) │ ├─ history.strip / fit old tool output → stubs / drops ├─ messages + reminder() time, git, todos, file changes (end only) ├─ llm.call_llm OpenAI-compatible chat + tool schemas │ ├─ no tool calls? break │ ├─ tools.execute │ ├─ parse JSON (don't crash) │ ├─ permissions.check allow / ask / deny │ ├─ TOOLS[name](...) │ │ bash → sandbox.run (Seatbelt / bwrap) │ │ read_skill → skills.py │ │ write_todos → todos.py │ │ task → subagent.py (its own inner loop) │ └─ catch errors, return text │ ├─ history.sweep delete spill files ├─ compact if usage ≥ 85% └─ session.saveui.py and prompt.py are how you see it: markdown, panels, a spinner, an input line with history. Display, not the idea.Checklist, if you’re building or judging oneAgent loop — Does it loop on its own, or do you click between every step?Tool interface — Schemas? Does a bad call crash the session?Permissions — Silent / ask / blocked — can you actually set that?Sandbox — Network? Writes outside the project?Context — What happens when the chat gets long? Cap, strip, compact?Memory — Skills, a plan, resume?Subagents — Can it explore without trashing its own window?Production harnesses all do some version of this. TrueForge uses cloud sandboxes and MCP. Claude Code has its own infra. Cursor has a better UI. The seven ideas don’t change.We’ve got them in ~600 lines and four dependencies: openai (any compatible endpoint), rich (terminal chrome), prompt-toolkit (the input line), pyyaml (skill frontmatter). No LangChain, no agent SDK, no vector database. It won’t replace those products. It will make them less magical.Where to go from hereIf you want to learn: clone coding-harness and read it in this order. Each file is short.config.py — how settings loadllm.py — one API call and the system promptagent.py — both loopstools.py — schemas + executepermissions.py — glob rules + compound commandssandbox.py — the jailhistory.py then compact.py — the four context layerscontext.py — late injectionskills.py, todos.py, session.py — memorysubagent.py — throwaway explorerui.py / prompt.py — last; they are displayIf a file is confusing, it is usually agent.py calling into it. Start from the inner loop and chase one tool call all the way down.Safe first experiments:Add an allow rule for ruff * or mypy * in permissions.pyLower CAP in history.py to 2,000 and watch the agent page spill filesWrite a SKILL.md under .agents/skills/review/SKILL.md and ask it to review a fileRead the compact prompt in compact.py and add a “tests” sectionAsk a question that should use task and watch the main context stay smallThen look at something like TrueForge to see the same patterns at scale.If you want to build: start with the loop and one tool. Add stuff when it hurts:Loop + one toolDefensive execute (don’t crash)PermissionsSome context cap/stripThen sandbox, skills, todos, sessions, subagents, compactionIf you want to ship: TrueForge if you want a real product surface. Or steal these primitives into whatever framework you already have. They’re not framework-specific.The model is the engine. The harness is the car. You know how to build the car now.End of the series. Code: coding-harness. Same primitives, less mystery.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Harness Engineering: Build the Runtime That Turns an LLM Into an Agent (Part 3/3) was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Generative AI Pub — Published September 22, 2026 at 01:34 UTC — Category: Image AI