How I Cut a 14-Day Local LLM Classification Job to 87 Hours

The smarter AI pipeline knows when not to call the AI.Image created by the author using Midjourney63,053 incident tickets reclassified on a laptop, with the LLM as a last resort instead of the default.I needed to reclassify a decade of service desk records: 63,053 incident tickets, remapped to a…

The smarter AI pipeline knows when not to call the AI.Image created by the author using Midjourney63,053 incident tickets reclassified on a laptop, with the LLM as a last resort instead of the default.I needed to reclassify a decade of service desk records: 63,053 incident tickets, remapped to a new root-cause taxonomy, with a hard deadline of five days. The data had to stay on-machine. Cloud APIs were not an option.The naive approach (LLM per record) would have taken around 14 days on a local 7B model. The business window would have closed before the run finished.The implementation the coding agent built instead completed in 87 hours on a MacBook, unattended. Coding agents (Claude Code and Trae) monitored the run through structured telemetry artefacts rather than the raw incident data. Small tests looked fine. The problems only showed up in the telemetry after the run passed 10,000 rows.The interesting part was not better prompting. It was finding a way to reuse earlier decisions fast enough to keep the batch on schedule, then instrumenting the run well enough for the agent to keep improving it without reading the tickets.Why LLM-per-record fails at scaleThe records were text-heavy (tickets, incidents, emails, surveys) and needed to be categorised into a controlled taxonomy for routing, analytics, and compliance. Manual triage did not scale. Rules-based systems got less accurate over time as people described the same problems in different words. So the obvious move was to call a model for every record.That approach broke as soon as the project had both volume and a hard constraint.In this case, the constraint set looked like this:63,053 records spanning roughly a decade of incidentsA new taxonomy (Type → Category → Subcategory) with 105 categories across 9 failure typesNo labelled training data for that taxonomyLocal-only processing, no data egressA fixed business deadlineSupervised ML was not an option. To train a 105-category classifier, the project would have needed labelled examples across the category set. The whole point of the project was that those labels did not exist yet.So the path was LLM-based zero-shot classification. The trick was finishing on time.Even with enough time, LLM-per-record gave me no audit trail by default: no cheap way to answer “which decisions were reused?” or “what changed when the run resumed?” unless I built that layer explicitly.And the usual “just add a semantic cache” answer did not solve it. A semantic cache helped when the same or very similar prompt came back later.In this batch, I only processed each ticket once, even though many tickets described the same underlying incident pattern in different words. I needed reuse based on the pattern, not repeated prompts.One option was to skip generation entirely: embed the 105 category descriptions, embed each ticket, and assign the nearest label. It was fast, tens of milliseconds per record, and it avoided an LLM call entirely.But on a three-level taxonomy with overlapping domain language, nearest-label matching kept confusing categories that used similar words. It also had no reliable way to say “none of these fits.”The implementation still used embeddings, but as a reuse mechanism. The first time a pattern appeared, the LLM made the decision. After that, semantically similar tickets reused it at embedding speed. That led directly to the routing pattern.The pattern: deterministic first, LLM lastThe work started with small samples, first 100 rows, then 1,000. Almost everything still fell through to the LLM. At that rate, the batch would miss the deadline, and the model kept solving the same patterns over and over.That pointed to a routing problem, not a prompt problem. If the run had already seen the pattern, the LLM should not be deciding it again.So the coding agent stopped tuning prompts and changed the routing. Deterministic and similarity checks moved to the front of the pipeline. The LLM moved to the end and handled only genuinely new tickets. That raised the next question: how many reuse layers did the system actually need?Why one cache layer is not enough“Add a cache” sounds like one idea. In practice, there are multiple similarity problems hiding inside each other.Exact-match caching (keyed by raw text) only caught perfectly identical tickets. The dataset was full of near-duplicates: “server is down”, “srv not responding”, “cannot connect to srv, urgent”. These described the same failure, but they did not match byte-for-byte.Fuzzy string matching caught typos and small edits, but it failed when meaning stayed the same, and wording changed completely.Embedding similarity solved that meaning gap, but a single semantic cache layer still missed too much at the start of a batch run when there were no prior decisions to reuse and the cache was empty.In a live endpoint, the same or similar query arrived again next week and the cache paid off. In a one-off reclassification run, every early miss still triggered a full LLM call, and if that miss rate stayed high, the batch missed the deadline.The waterfall worked because it used the cheapest possible signal first and reserved embeddings and generation for the cases that actually needed them.That only worked if the pipeline classified the right text in the first place. Feeding it noisy or contradictory fields made every layer worse, from fingerprinting through to the LLM.The noisy field problem (what text did I actually classify?)My service desk tickets rarely had one clean text field. They had a short description, a long description, a resolution note, and free-text comments. These were often contradictory.Concatenating everything fed the model noise that actively undermined the problem statement. The short description might say “User cannot log in”. The resolution might say “Done”.So I applied a selection policy per run:priority_first_meaningful: walk a ranked list of columns and pick the first that clears a minimum length thresholdconcat_all: concatenate capped fields up to a global limit for datasets where the short description is routinely uselessBefore any downstream layer ran, I cleaned the chosen text (strip HTML, remove boilerplate like “resolved” and “done”, normalise whitespace).In parallel, I produced a second representation that I never sent to the LLM. I used it for matching only: lowercase, strip HTML, replace variable identifiers (emails, URLs, IPs, ticket IDs, long numbers) with typed tokens, then strip non-word characters. That was what made two “different” tickets hash the same way.Once the text stayed consistent, I could start reusing decisions safely.The reuse waterfallI did not design the waterfall upfront. The coding agent built it iteratively from run telemetry. When telemetry showed high LLM fallback on tickets that were structurally similar to ones already processed, the agent added or tightened the next reuse layer and reran, without ever inspecting the incident text.Before the waterfall ran, I applied a deterministic rule layer. Config-declared regex patterns mapped known-certain phrases directly to taxonomy categories (for example, a password reset pattern), bypassing all caches and the LLM.After that, each ticket tried strategies in order, cheapest first, LLM last:Fingerprint cache (sha256 of normalised text, essentially free)Fuzzy string match (Jaro-Winkler, around 1ms)Semantic cache (embedding cosine, around 30ms)Cluster reuse (centroid cosine against accumulated cluster centroids, around 30ms)LLM fallback (qwen2.5:7b via Ollama, around 5 to 20s)Each layer covered a different class of similarity:Fingerprint reuse caught “same incident, different identifiers” once the normalisation layer removed ticket IDs, hostnames, IPs, and dates.Fuzzy matching caught small edits and typos.Semantic matching caught different wording with the same meaning.Cluster reuse was a stabiliser. As the run classified more tickets, it accumulated clusters of similar tickets and reused against the centroid rather than a single prior ticket.I only allowed reuse when guardrails passed:Taxonomy version fingerprint matchedSimilarity score cleared a strict thresholdPrior result was auto-accepted (not flagged for review)confidence_score met the auto-accept thresholdIf any guard failed, the pipeline fell through. There was no silent degradation.The output recorded which strategy produced each classification and, when reuse occurred, the similarity score as match_confidence. A fingerprint hit was 1.0. Fuzzy reuse sat just above the 0.96 threshold. Embedding reuse sat above the 0.975 threshold.One representative reuse output (redacted and sanitised):https://medium.com/media/b8e9d4e6e70e529732dc852956accb7a/hrefWhat only broke in the full runOnce the waterfall existed, the “AI” work was not the hard part anymore. The hard part was keeping the pipeline correct and resumable while it ran for days.1. The Python falsy bug that silenced cache seedingAfter the first restart, I still found the semantic and cluster caches empty on disk. Thousands of rows had been processed. Nothing had crashed. The run just was not seeding them.The cause was a guard that looked fine:https://medium.com/media/79aac3ef25e08db82d4fe68d8e2cfdd3/hrefIn Python, if obj: acted as a truthiness check. For collection-like objects, truthiness depended on __len__. An empty cache had len() == 0, so it evaluated as falsy. The pipeline had initialised the cache correctly, then treated it as "missing" and skipped seeding on every row of every run.The fix was explicit is not None checks.https://medium.com/media/986f8cf8b37ab7fb055312f34e80ddd3/hrefThis was the sort of failure that unit tests rarely caught. Test fixtures tended not to use truly empty cache objects with __len__ implemented. A live run started cold, so it hit the edge case immediately.2. The embedder that disabled itself permanentlyThe original embedder code flipped a _disabled flag after a single exception. One transient Ollama timeout, which was common under contention, could shut off embeddings for the rest of the run.That was a real throughput problem, not a nice-to-have cleanup. Once embeddings disappeared, more rows fell back to the LLM and the batch slowed down without a clean hard failure.The fix was a circuit breaker: a small retry budget, trip only after consecutive failures, and auto-recover on the first success. When the circuit was open, the system stopped competing for resources, and the LLM-only path actually ran faster.More importantly, the downgrade stopped being silent. If embeddings were offline, telemetry showed it, and the run behaved predictably.3. Checkpoint correctness after a crashThe pipeline wrote output as JSONL (append-only) and stored progress in a checkpoint keyed by a deterministic row ID (row_md5). That made planned stop/resume easy.The failure case was an unplanned crash between “write output” and “save checkpoint”. On resume, a naive implementation re-processed and re-appended those rows, creating duplicates.The fix was reconciliation at startup: scan the existing output file for row IDs and backfill them into the checkpoint before processing began.That mattered beyond crash recovery. The run was happening on a laptop, overnight and across working days. When telemetry showed a problem- too much LLM fallback, caches not warming, invalid-output retries I could stop the batch, change a config value, and resume without losing completed work.A small prompt change improved throughputThe bulk prompt schema originally included a reasoning field. That looked useful, but in practice it mostly created failure. The model sometimes wrote paragraphs, hit the token cap, and returned invalid JSON.I removed reasoning and capped bulk_num_predict at 128. That improved throughput on LLM-only rows and reduced invalid-output retries. Shorter outputs broke less often.Long ticket texts still overflowed context occasionally. The implementation used a prompt variant cascade, from the full prompt down to a minimal “ids only” variant, so an overflow fell through to a smaller prompt instead of failing the record.It was not elegant, but it kept the batch moving. A production serving stack with grammar constraints could guarantee schema-valid JSON at decode time and collapse the cascade into a single prompt. With Ollama in this configuration, the cascade was the practical way to keep the run alive.What the run provedThe headlines:Records processed: 63,053Review queue: 204Total elapsed: ~87 hoursAverage throughput: ~0.20 rows/sec (~725 rows/hour)API cost: £0 (100% local inference)Data left the machine: neverNot every record went through the reuse waterfall. Deterministic rules handled some records outside it. The strategy mix below focuses on the resumed segment that entered the reuse chain.The strategy mix came from the 44,803-row resumed segment (the portion with complete comparable per-strategy telemetry after the run was resumed mid-batch):LLM fallback — 56.6%Fingerprint cache — 14.9%Semantic cache — 11.8%Fuzzy match — 10.3%Cluster reuse — 6.4%This was a reclassification run against a new taxonomy. The caches were cold at the start. In a steady-state run against a stable taxonomy, the reuse layers would likely handle a much larger share.One caveat: the operational metrics above described throughput and pipeline mechanics, not classification accuracy. In the resumed segment, an uncalibrated LLM classified over half the rows with no gold-set comparison, and the reuse layers propagated the quality of those decisions.There was no evaluation harness or labelled example set yet, so the output distributions are directional rather than definitive. Cache warming was not linear. The first slice of a cold run was always slowest because almost everything hit the LLM.As the fingerprint cache filled with the most common templates and the semantic cache started to recognise recurring problem descriptions, the hit rate climbed, and the LLM handled a shrinking fraction of rows.The review queue was the other practical signal. In this run it was 204 records (0.3%). Some of those were true low-confidence cases. Others were records that were not really incident descriptions at all, such as IT support emails with human-to-human back-and-forth.The point was not whether 204 was “good” or “bad”. The point was that the pipeline produced a bounded worklist a human could actually review, instead of hiding uncertainty inside the bulk output.What it does not do (yet)The run did not measure per-category accuracy. There was no evaluation harness or gold-set comparison yet.The implementation did not build a successor classifier (distill high-confidence labels into a fast encoder for future runs).confidence_score was categorical rather than calibrated. It worked as a contract field, not as a measurement.The pipeline processed sequentially. Ollama blocked on a single request per model, so this build had no row-level parallelism.The implementation selected one text field rather than voting across fields. A quorum policy existed in config, but the agent did not implement it in this version.The similarity thresholds were tuned for incident ticket language. Other datasets would have needed calibration.Reuse assumed taxonomy stability. Every cached decision stayed valid only while the taxonomy that produced it stayed current.The transferable insightThe reuse waterfall was the visible part. What made it usable in a privacy-constrained environment was everything around it: content-safe telemetry, checkpointing, and failure handling. Those artefacts let the coding agent diagnose and tune the run from evidence rather than from ticket text.That was the operating pattern underneath the implementation. The agent could watch throughput by strategy, cache hit rates, circuit breaker trips, and checkpoint state, then adjust thresholds, selection policy, or output shape without the incident text leaving the machine or entering the agent’s context.Repro notesRun conditions: 2026–04; Apple M3 Pro MacBook, 18GB RAM; classification LLM qwen2.5:7b via Ollama; embedding model nomic-embed-text via Ollama; bulk_num_predict: 128Dataset class: enterprise incident backlog (63,053 records; anonymised; no client identifiers)This article does not include the exact prompt schemas or taxonomy definitionsKey takeawaysThe implementation kept deterministic and similarity paths first and pushed the LLM to the end of the chain.The reuse waterfall turned “LLM-per-record” into “LLM-per-new-pattern”.At this scale, the hard problems were operational: cache seeding, circuit breakers, and crash-safe resume.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!How I Cut a 14-Day Local LLM Classification Job to 87 Hours was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Generative AI Pub — Published August 5, 2026 at 13:18 UTC — Category: Image AI