Design an AI Copilot for 10 Million Developers at Scale

Design an AI Copilot for 10M DevelopersBreaking down the architecture behind real-time code completion, repository-aware retrieval, agentic workflows, and multi-tenant security at scale.1. Clarifying Requirements1.1 Functional RequirementsDevelopers receive inline code completions as they…

Design an AI Copilot for 10M DevelopersBreaking down the architecture behind real-time code completion, repository-aware retrieval, agentic workflows, and multi-tenant security at scale.1. Clarifying Requirements1.1 Functional RequirementsDevelopers receive inline code completions as they type — single-line and multi-line suggestions generated in real time within their IDE (VS Code, JetBrains, Vim, Neovim), grounded in the current file, open tabs, and project contextDevelopers can invoke chat-based assistance — asking questions about their codebase, requesting refactors, debugging help, or generating new code from a natural language description, with the conversation grounded in relevant repository contextThe system performs repository-aware retrieval — understanding the developer’s entire codebase (potentially millions of lines across thousands of files) well enough to answer questions like “where is authentication handled?” or “what calls this function?” without the developer manually providing that contextThe system supports multi-file, multi-step agentic edits — a developer can request a feature or bug fix in natural language, and the copilot plans a sequence of file edits, applies them, runs tests, and iterates based on results, with the developer able to review and approve each stepThe system integrates with version control and CI — understanding git history, blame information, open pull requests, and CI failure logs to answer context-aware questions (“why did this test start failing after my last commit?”)The system supports team and organisation-level context — beyond a single developer’s open files, the copilot can be configured with organisation coding standards, internal library documentation, and architecture decision records that inform its suggestionsDevelopers can provide feedback — accepting or rejecting suggestions, rating chat responses, and flagging incorrect or unsafe generated code; this feedback must improve future suggestions for that developer, their team, and (in aggregate) the broader modelAdmins can enforce policies — blocking suggestions that would introduce known-vulnerable dependencies, prevent secrets from being suggested or committed, restrict which repositories or code the copilot can access based on data classification, and audit all code generation activity1.2 Non-Functional RequirementsCompletion latency: inline completions must appear within 100ms of the keystroke that triggers them (p99) — anything slower breaks the “feels instant” requirement that makes inline completion usable at all; typing-speed-scale latency is the defining constraint of this systemScale: 10M active developers, each with 1–20 active repositories, generating an estimated 500M completion requests per day and 50M chat/agentic requests per dayContext window management: individual repositories range from thousands to tens of millions of lines of code; the system must intelligently select the most relevant subset of this code to fit within the LLM’s context window for any given requestFreshness: code context must reflect the developer’s current, uncommitted working state — a function renamed 30 seconds ago in an open editor tab must be reflected in the very next completion request, not just after the next commit or index rebuildAvailability: 99.95% uptime for completions (developers tolerate brief unavailability more than incorrect/stale suggestions); degraded mode (client-side or cached suggestions) when the service is unreachableSecurity and data isolation: proprietary source code from one organisation must never leak into suggestions for another organisation, whether through shared model fine-tuning, cross-tenant caching, or telemetry aggregationCost efficiency: at 500M completion requests/day, even a small per-request compute cost multiplies into a massive infrastructure bill; the system must aggressively minimise unnecessary LLM inference through caching, small models for common cases, and speculative techniquesOffline/degraded operation: when disconnected from the network (or the backend service is down), the local IDE extension should still provide some baseline completion capability using a lightweight local model or cached results2. Back-of-the-Envelope EstimationCompletion traffic (the dominant, latency-critical workload):- Active developers: Given 10M- Completion requests/developer/day: \~50 (every few keystrokes pause)- Total completions/day: 10M × 50 → 500M/day- Peak QPS (concentrated in working hours, 3× avg): 500M / 86,400 × 3 → \~17,400 QPS- Avg tokens generated per completion: \~15 tokens (short, inline)- Avg context tokens per completion request: \~2,000 tokens (surrounding code + retrieved snippets)Chat / agentic traffic:- Chat/agentic requests/day: Given 50M/day- Peak QPS: 50M / 86,400 × 3 → \~1,740 QPS- Avg context tokens (chat, repo-aware): \~8,000 tokens- Avg output tokens (chat): \~500 tokens- Agentic multi-step requests (subset of chat): \~5M/day, avg 8 LLM calls per session → 40M LLM calls/day from agentic loopsLLM inference sizing (the dominant cost driver):- Completion tokens/sec at peak (17,400 QPS × 2,015 tokens) → \~35M tokens/sec- Chat tokens/sec at peak (1,740 QPS × 8,500 tokens) → \~15M tokens/sec- Combined peak inference load → \~50M tokens/sec- At 3,000 tokens/sec/A100 (7B-13B model, completions): 35M / 3K ≈ \~11,700 A100s for completions- At 3,000 tokens/sec/A100 (larger model, chat): 15M / 3K ≈ \~5,000 A100s for chatThis is the single most important number in the design: \~17,000 GPUs is not affordable if every completion triggers a full LLM call. The entire completion-path architecture must be built around avoiding LLM calls wherever possible — caching, small local models, and speculative pre-computation are not optimisations, they are the core design.Repository index scale:- Active repositories (10M devs × avg 3 repos, deduplicated for shared repos) → \~15M unique repos- Avg repo size: \~50K lines / \~2,000 files- Embedding chunks per repo: \~2,000 files × 3 chunks/file- Total chunks across all repos: 15M × 6,000- Vector storage (6 KB/chunk): 90B × 6 KB3. Core APIsInline completion API (IDE extension → backend, hot path)POST /v1/completionsHeaders: Authorization: Bearer Body: { file_path: "src/auth/login.py", cursor_position: { line: 42, column: 18 }, prefix: "...code before cursor, up to N tokens...", suffix: "...code after cursor, up to N tokens...", open_tabs: [{ file_path, content_hash }], // hashes; full content sent only on cache miss language: "python", repo_id: "repo_abc", trigger: "keystroke" | "explicit_invoke", request_id: "comp_req_001"}Response (must return within 100ms p99): { suggestions: [ { text: "return self._validate_token(token, expiry=3600)", confidence: 0.89, source: "small_model" } ], latency_ms: 62, cache_hit: false}Chat API (repo-aware, streaming)POST /v1/chatBody: { session_id: "sess_abc", message: "Why is the login test failing after my last commit?", repo_context: { repo_id: "repo_abc", branch: "feature/oauth", include_ci_logs: true }, attached_files: ["src/auth/login.py"] // explicitly pinned by developer}Response (SSE stream): { event: "token" data: "Looking at your recent commit..." event: "citation" data: { file_path, line_range, commit_sha } event: "done" data: { message_id, tokens_used }}Agentic task API (multi-step, plan-execute-verify)POST /v1/agent/tasksBody: { repo_id: "repo_abc", branch: "feature/oauth", instruction: "Add rate limiting to the /login endpoint using the existing RateLimiter class", approval_mode: "auto" | "step_by_step"}Response: { task_id: "task_001", status: "PLANNING", plan_url: "/v1/agent/tasks/task_001/plan"}GET /v1/agent/tasks/{task_id}Response: { task_id, status: "PLANNING" | "AWAITING_APPROVAL" | "EXECUTING" | "TESTING" | "DONE" | "FAILED", steps: [ { step_id, description: "Import RateLimiter into login.py", status: "DONE", diff_url: "..." }, { step_id, description: "Add rate limit check before auth", status: "AWAITING_APPROVAL", diff_url: "..." } ], test_results: { passed: 12, failed: 0 }}Repository indexing API (background)POST /v1/repos/{repo_id}/indexBody: { source: "git", clone_url: "...", branch: "main", incremental: true }Response: { index_job_id, status: "queued", estimated_chunks: 6000 }Feedback APIPOST /v1/completions/{request_id}/feedbackBody: { action: "accepted" | "rejected" | "modified", modified_text: "..." }POST /v1/chat/{message_id}/feedbackBody: { rating: "helpful" | "unhelpful", flag_reason: "incorrect" | "unsafe" | "off_topic" }Admin policy APIPOST /v1/orgs/{org_id}/policiesBody: { blocked_dependency_patterns: ["log4j:1.*", "event-stream:3.3.6"], secret_detection: "block_and_alert", repo_access_scope: ["internal-only"], telemetry_opt_out: false}4. High-Level Design4.1 Requirement 1: The Completion Hot Path — Meeting the 100ms Constraint Without Bankrupting on GPUsComponents needed:The 100ms p99 latency requirement combined with the 17,400 QPS peak volume is the defining engineering constraint of this entire system. A full LLM inference call, even on a fast, small model, typically costs 200–800ms for meaningful multi-line completions once network round-trip, context assembly, and generation are included.The completion path must therefore be architected as a cascade of increasingly expensive strategies, most requests resolved by the cheap early stages.Local IDE-side prediction (Stage 0, ~0ms network cost): the IDE extension itself runs a small, quantised local model (a distilled 100M–500M parameter model, running on the developer’s own CPU/GPU via ONNX Runtime or similar) that handles the most common, highly predictable completions: closing brackets, completing obvious variable names already seen in the file, boilerplate patterns. This local model requires no network round-trip at all; it resolves an estimated 30–40% of trigger events with zero backend load. This is also the offline/degraded mode fallback (requirement 1.2) — the same local model serves suggestions when the backend is unreachable.Edge cache lookup (Stage 1, 300ms, a strong signal they may accept the current suggestion or the cursor will move soon), the system speculatively pre-computes completions for the most likely next cursor positions (end of current line, start of next line) and caches them locally in the IDE extension. This converts what would be a live request into a Stage 0/1 cache hit for a meaningful fraction of subsequent keystrokes.4.2 Requirement 2: Repository-Aware Retrieval — Understanding Millions of Lines of CodeComponents needed:A developer’s question (“where is authentication handled?”) or a completion needing project-wide context (calling a function defined in another file) requires the system to search the repository intelligently, not just look at the currently open file. This is architecturally similar to the RAG systems in earlier designs but with code-specific structure that materially changes the retrieval strategy.Multi-granularity code indexing — unlike prose documents, code has a native structural hierarchy (file → class → function → block) that a flat 512-token chunking strategy discards valuable signal from. The Code Indexer parses each file with a language-aware AST parser (Tree-sitter, which supports dozens of languages with a unified API) and produces chunks aligned to semantic boundaries: whole functions when they fit the token budget, function signatures plus docstrings as a separate lightweight index entry (useful for “does a function like X exist” queries without needing the full body), and file-level summaries (an LLM-generated one-paragraph description of each file’s purpose, generated once at index time) for coarse-grained repository navigation.Symbol graph alongside the vector index — code retrieval benefits enormously from structural relationships that pure semantic similarity misses: “what calls this function,” “what implements this interface,” “where is this variable assigned.” The Code Indexer additionally builds a symbol graph (using each language’s native analysis tools where available — the TypeScript compiler API, Python’s ast module, gopls for Go) capturing definitions, references, call graphs, and import relationships. A query like "what calls validate_token?" is answered by a graph traversal, not a vector search — far more precise and instant compared to hoping semantic similarity surfaces the right call sites.Hybrid retrieval for chat and agentic requests — when a developer asks a natural-language question, the Retrieval Orchestrator combines: (a) vector similarity search over the chunk index (semantic matches), (b) BM25 keyword search (exact identifier matches — critical for code, where “the AuthMiddleware class" needs exact string matching more than semantic similarity), (c) symbol graph traversal (for structural questions), and (d) currently-open-file context (always included at highest priority, since it's almost certainly relevant to whatever the developer is doing right now). Results from all four sources are merged and reranked by a code-aware cross-encoder before being assembled into the LLM's context window.Real-time index freshness for uncommitted changes — the freshness requirement (a function renamed 30 seconds ago must be reflected immediately) means the system cannot rely solely on a periodically-rebuilt index from committed code. The IDE extension maintains a local working-state overlay: as the developer edits files, a lightweight diff is streamed to the backend (debounced, batched every few seconds) and applied as an ephemeral, session-scoped patch on top of the last-indexed commit. Retrieval queries check this overlay first — if a file has uncommitted local changes, the overlay’s version is used instead of (or merged with) the indexed version. This overlay is never persisted to the shared repository index; it exists only for that developer’s session.4.3 Requirement 3: Agentic Multi-Step Edits — Plan, Execute, VerifyComponents needed:Beyond single completions and chat answers, developers increasingly want to delegate entire tasks (“add rate limiting to this endpoint”). This requires the system to plan a sequence of actions, execute them against the actual codebase, verify the results, and iterate — all while keeping the developer in control.The Planner — receives the natural-language instruction plus repository context (retrieved via the same Hybrid Retrieval described in 4.2) and produces a structured plan: an ordered list of steps, each describing a file to modify, the nature of the change, and any commands to run (install a dependency, run a specific test file). The Planner is a larger, more capable LLM than the completion-path models — planning quality matters more than latency here, since this is not on the 100ms critical path. The plan is presented to the developer for review before any code is touched (or, in approval_mode: "auto", only pauses for review on steps flagged as higher-risk, like deleting files or modifying CI configuration).The Executor — takes an approved plan step and generates the actual code diff, using the same Hybrid Retrieval to pull in exactly the context needed for that specific file (not the entire repository — keeping each individual edit’s context window focused and cost-efficient). The diff is applied to a sandboxed workspace (a container with the repository checked out at the relevant branch, isolated from the developer’s actual working directory until they explicitly accept the changes) — never directly to the developer’s live files without explicit application.The Verifier — after each step (or after the full plan, depending on configuration), runs the project’s existing test suite, linters, and type checkers inside the sandbox. Test failures are fed back to the Planner/Executor loop: the system attempts to diagnose the failure (using the test output as additional context) and proposes a corrective edit, retrying up to a configurable limit (default 3 attempts) before surfacing the failure to the developer rather than looping indefinitely. This plan→execute→verify→repair loop is the core agentic pattern, and its bounded-retry design prevents runaway resource consumption from a task that’s fundamentally stuck.Diff review and staged application — every agentic edit is presented to the developer as a standard diff (integrated into the IDE’s native diff/review UI) before being applied to their actual files. The developer can accept, reject, or modify each hunk individually — the agentic system never silently commits changes. This is a deliberate product and safety boundary: agentic capability accelerates the developer’s own review-and-apply workflow, it does not bypass it.4.4 Requirement 4: Data Isolation and Multi-TenancyComponents needed:Proprietary source code is among the most sensitive data an organisation has. The isolation guarantees here are stricter than typical multi-tenant SaaS because the failure mode (Company A’s code appearing in a suggestion shown to Company B) is catastrophic and immediately visible to the affected customer.Per-organisation index isolation — following the same pattern as the multi-tenant vector database and OpenAI file search designs from earlier: each organisation’s code index lives in a dedicated Qdrant collection (or a dedicated shard-key partition for smaller organisations sharing infrastructure), with no cross-tenant query path possible even in principle — the retrieval layer’s collection routing is resolved from the authenticated developer’s organisation membership, never from any client-supplied parameter.No cross-tenant fine-tuning — organisations that opt into “improve suggestions using our codebase” have their feedback data used only to fine-tune a per-organisation adapter (a LoRA or similar lightweight fine-tune layered on top of the shared base model), never merged into the shared base model weights that serve other organisations. This is an architectural guarantee, not a policy promise: the training pipeline physically cannot mix data across organisation boundaries because each organisation’s fine-tuning job runs against an isolated data partition with its own access-controlled storage bucket.Aggregate telemetry with differential privacy — even anonymised, aggregate usage statistics (e.g. “which language features are commonly suggested”) carry some re-identification risk for an organisation with unusual, distinctive code patterns. Telemetry aggregation applies differential privacy techniques (adding calibrated noise, requiring minimum cohort sizes before publishing any aggregate statistic) so that even organisation-level patterns cannot be reverse-engineered from published metrics.Secret and credential scanning on both input and output — before any code content (from the developer’s files) is sent to a shared/multi-tenant inference backend, a secret scanner (pattern-based plus a lightweight classifier for less-structured secrets like internal API tokens) redacts detected credentials, replacing them with placeholders before the content leaves the organisation’s isolated boundary. Symmetrically, generated completions and chat responses are scanned before being shown to the developer — preventing the (rare but real) failure mode of a model hallucinating a plausible-looking but real credential pattern.4.5 Requirement 5: Policy Enforcement and AuditabilityComponents needed:Organisations adopting an AI copilot at the scale of 10M developers need controls: preventing known-vulnerable code patterns from being suggested, ensuring generated code doesn’t introduce license-incompatible dependencies, and maintaining an audit trail for compliance and incident investigation.Pre-suggestion policy filter — every completion and chat response passes through a Policy Engine (built on Open Policy Agent, consistent with the pattern used in the enterprise RAG and knowledge assistant designs) before being returned to the developer. Policies include: blocking suggestions that import a dependency matching a known-vulnerable version pattern (checked against a continuously-updated CVE database), flagging suggestions that would introduce code matching known-insecure patterns (SQL string concatenation instead of parameterised queries, disabled certificate validation), and license compliance checks (blocking suggested code fragments that closely match GPL-licensed training data if the organisation’s policy requires permissive-license-only code, using a code-similarity fingerprint check against a database of known open-source snippets).Immutable audit log — every completion shown, every chat message exchanged, every agentic edit applied is logged with: the developer identity, the repository and file context, the exact prompt sent to the model, the response received, and any policy actions taken (blocked, flagged, allowed). This follows the same INSERT-only, hash-chained pattern established in the mission-critical and enterprise assistant designs — necessary here because a security incident investigation (“did the copilot suggest code that introduced this vulnerability?”) requires being able to reconstruct exactly what was suggested and when, with tamper-evident guarantees.Admin dashboard for organisation-wide visibility — aggregated (not per-developer, to respect individual privacy within the org unless specifically required for a security investigation) metrics: completion acceptance rates by team, most-flagged policy violations, dependency suggestions trending across the organisation’s repositories (useful for catching organisation-wide adoption of a newly-discovered vulnerable library before it spreads further).4.6 Putting It All TogetherThe system has four concurrent layers. The completion hot path (local model → edge cache → small GPU model → async escalation) is the highest-volume, most latency-sensitive layer, engineered specifically to avoid full LLM calls wherever possible. The repository indexing pipeline (AST parsing → chunk + symbol graph construction → embedding → per-tenant vector store) runs continuously in the background, kept fresh by both periodic full re-indexing and the real-time working-state overlay for uncommitted changes. The chat and agentic layer (hybrid retrieval → planner → executor → verifier) handles lower-volume, higher-value requests where latency budgets are more generous (seconds, not milliseconds) but correctness and safety matter more. The policy and audit layer wraps every other layer, enforcing organisation controls and maintaining the compliance trail, without being a bottleneck on any of the other three (policy checks run in parallel with generation wherever possible, only blocking the final response).Full architecture diagram: — -5. Database Design5.1 SQL vs NoSQLQdrant (per-organisation code vector store) — following the collection-per-tenant pattern established in the OpenAI File Search design, each organisation gets its own Qdrant collection (or shard-key partition for smaller organisations sharing a physical cluster). At 90B chunks across all organisations globally, this must be distributed across thousands of shard nodes; the per-organisation isolation model is non-negotiable here given the sensitivity of proprietary source code — a shared global collection with tenant filtering, as some cheaper architectures use, is an unacceptable risk for this specific data type.Neo4j or a custom graph store (symbol graph) — the call-graph, reference-graph, and import-graph relationships benefit enormously from a native graph query engine. A query like “find all callers of validate_token, then find all callers of those functions, up to depth 3" is a natural Cypher traversal and would require multiple expensive joins in a relational model. Given the code-specific nature of this graph (nodes are functions/classes/files, edges are calls/references/imports, all scoped per-repository), Neo4j's property graph model maps directly onto the domain without an impedance mismatch.Redis (completion cache + prefix cache + working-state overlay) — three latency-critical uses, all requiring sub-10ms access. The completion cache (completion_cache:{repo_id}:{file_hash}:{prefix_hash}) is a straightforward key-value lookup. The working-state overlay (uncommitted diffs per developer session) uses Redis Hashes keyed by session_id, holding the diff patches as they stream in from the IDE extension, with a TTL matching session inactivity timeout. Redis Cluster distributes both across the fleet, sharded by repo_id for the completion cache (keeping all cache entries for one repository together, useful for bulk invalidation when a file's indexed version changes) and by developer_id for the working-state overlay.PostgreSQL (repository metadata, indexing job state, feedback, audit log) — repository registration, indexing job status, per-developer feedback records, and the policy/audit log are all relational, ACID-requiring workloads at modest volume (compared to the completion hot path). The audit log follows the same INSERT-only, hash-chained pattern used throughout — a service account with no UPDATE/DELETE privilege, ensuring the compliance trail cannot be altered even by an administrator with database access.Object storage / S3 (raw repository snapshots, model checkpoints, LoRA adapters) — full repository clones used for indexing, fine-tuned per-organisation LoRA adapter weights, and periodic index snapshots for fast cold-start recovery are all large, infrequently-accessed binary blobs best served by object storage rather than any database.5.2 Database Schema-- PostgreSQLorganizations ( org_id UUID PRIMARY KEY, name TEXT NOT NULL, telemetry_opt_in BOOLEAN DEFAULT false, fine_tune_opt_in BOOLEAN DEFAULT false, data_residency TEXT, -- 'us'|'eu'|'in' for compliance created_at TIMESTAMPTZ DEFAULT now())repositories ( repo_id TEXT PRIMARY KEY, org_id UUID REFERENCES organizations(org_id), clone_url TEXT NOT NULL, default_branch TEXT DEFAULT 'main', language_breakdown JSONB, -- { "python": 0.6, "typescript": 0.3, ... } index_status TEXT DEFAULT 'PENDING', -- 'PENDING'|'INDEXING'|'ACTIVE'|'STALE' last_indexed_commit TEXT, last_indexed_at TIMESTAMPTZ, chunk_count BIGINT, vector_collection_ref TEXT -- which Qdrant collection/shard_key this maps to)developers ( developer_id UUID PRIMARY KEY, org_id UUID REFERENCES organizations(org_id), email TEXT UNIQUE NOT NULL, local_model_version TEXT, -- which distilled model version their IDE has cached created_at TIMESTAMPTZ DEFAULT now())completion_feedback ( request_id TEXT PRIMARY KEY, developer_id UUID NOT NULL, repo_id TEXT NOT NULL, suggestion_source TEXT, -- 'local'|'edge_cache'|'small_model'|'escalated' action TEXT NOT NULL, -- 'accepted'|'rejected'|'modified' modified_text TEXT, latency_ms INT, created_at TIMESTAMPTZ DEFAULT now())agent_tasks ( task_id UUID PRIMARY KEY, developer_id UUID NOT NULL, repo_id TEXT NOT NULL, instruction TEXT NOT NULL, status TEXT DEFAULT 'PLANNING', approval_mode TEXT DEFAULT 'step_by_step', retry_count INT DEFAULT 0, created_at TIMESTAMPTZ DEFAULT now(), completed_at TIMESTAMPTZ)agent_task_steps ( step_id UUID PRIMARY KEY, task_id UUID REFERENCES agent_tasks(task_id), step_index INT NOT NULL, description TEXT NOT NULL, file_path TEXT, diff_ref TEXT, -- pointer to the generated diff in object storage status TEXT DEFAULT 'PENDING', test_result JSONB)policy_events ( event_id UUID PRIMARY KEY, org_id UUID NOT NULL, developer_id UUID NOT NULL, event_type TEXT NOT NULL, -- 'BLOCKED_CVE'|'BLOCKED_SECRET'|'BLOCKED_LICENSE' policy_rule TEXT NOT NULL, request_context JSONB, created_at TIMESTAMPTZ DEFAULT now())audit_log ( log_id UUID PRIMARY KEY, developer_id UUID NOT NULL, org_id UUID NOT NULL, event_type TEXT NOT NULL, -- 'COMPLETION'|'CHAT'|'AGENT_EDIT' repo_id TEXT, prompt_hash TEXT NOT NULL, response_hash TEXT NOT NULL, policy_actions JSONB, prev_log_hash TEXT NOT NULL, -- hash chain created_at TIMESTAMPTZ NOT NULL DEFAULT now() -- INSERT-only)Qdrant point schema (one point per code chunk):{ "id": "", "vector": [0.021, ...], "payload": { "repo_id": "repo_abc", "file_path": "src/auth/login.py", "chunk_type": "function"|"class"|"file_summary"|"signature", "symbol_name": "validate_token", "start_line": 42, "end_line": 58, "language": "python", "commit_sha": "a1b2c3d", "last_modified_at": "2024-06-01T10:00:00Z" }}Neo4j (symbol graph) schema:(:Function {name, file_path, repo_id, signature})(:Class {name, file_path, repo_id})(:File {path, repo_id})(fn1:Function)-[:CALLS]->(fn2:Function)(fn:Function)-[:DEFINED_IN]->(file:File)(cls1:Class)-[:IMPLEMENTS]->(cls2:Class)(file1:File)-[:IMPORTS]->(file2:File)6. Design Deep Dive6.1 The Small-Model-First Cascade — Quantifying Why It’s NecessarySection 2’s estimation showed that a naive “every completion request hits a full LLM” architecture requires ~17,000 A100s. This deep dive works through why the cascade specifically achieves the needed cost reduction and what makes each stage’s hit rate realistic rather than optimistic.Why Stage 0 (local model) achieves 30–40% resolution: the distribution of completion triggers is heavily skewed toward highly predictable continuations — closing a bracket that was just opened, completing a variable name that appears earlier in the same file, finishing a common language construct (if __name__ == "__main__":). A small model trained specifically to recognise these patterns (distilled from the larger model's outputs on exactly this category of "obvious" completions) achieves high accuracy on this subset without needing broader code understanding. The 30-40% figure is consistent with published data from production code-completion systems that report local/cheap-model resolution rates in this range.Why Stage 1 (edge cache) hit rate depends on cache key design: the naive approach — caching on the full prefix string — has a low hit rate because prefixes are highly variable (different variable names, different surrounding code). The effective cache key instead normalises the prefix: replacing identifier names with positional placeholders (x = .method() rather than the literal variable name), which dramatically increases hit rate for structurally-similar-but-textually-different code (the same error-handling pattern used with different exception types, the same test assertion pattern with different values). This normalisation is what pushes the cache hit rate from a naive single-digit percentage to the 20-30% figure used in the estimation.Why the residual Stage 2 load is still 2,000–3,000 GPUs, not near-zero: even after Stages 0 and 1 absorb roughly 55–65% of requests combined, the remaining 35–45% of 500M daily completions (and their associated ~2,000 context tokens each) still represents a substantial inference load — this residual is precisely the “genuinely novel, context-dependent completion” category that no caching or local heuristic can shortcut, and it is where the actual value of the backend LLM service is delivered. The architecture’s goal is not to eliminate this cost but to ensure the system only pays it for requests that actually need it.6.2 Context Window Assembly for Repository-Scale CodeA 10-million-line repository cannot fit in any LLM’s context window. The Retrieval Orchestrator’s job of selecting what subset of code to include is the single highest-leverage decision affecting completion and chat quality.Priority-ordered context assembly for completions: unlike chat (where the query directly indicates what’s relevant), a completion request has only the cursor position and surrounding code as its “query.” The context assembly for completions follows a strict priority order, filling the available token budget (typically 2,000–4,000 tokens for the completion model) from highest to lowest priority: (1) the immediate prefix and suffix around the cursor (always included, verbatim); (2) other locations in the same file that reference the same symbols currently in scope (found via the symbol graph — if the cursor is inside a function that calls self.config, the config class definition is pulled in); (3) the signatures (not full bodies) of functions imported into the current file; (4) semantically similar code elsewhere in the repository (vector search fallback, lowest priority, used only if budget remains).Import-graph-guided context for cross-file awareness: a critical code-specific optimisation over generic RAG: when completing code that calls SomeImportedClass.method(), the single most valuable context is not the semantically-nearest chunk by embedding similarity — it is the exact definition of SomeImportedClass.method from wherever it was imported from, found via the symbol graph's DEFINED_IN and import edges. This is a case where structural retrieval (graph traversal) strictly dominates semantic retrieval (vector search) for a well-defined subclass of queries, and the Retrieval Orchestrator checks for this case first before falling back to vector search for less structurally-determinable context needs.Token budget allocation for chat (larger budget, different priorities): chat requests have a more generous context budget (~8,000 tokens) and the query itself provides much stronger relevance signal, so the allocation shifts toward the Hybrid Retrieval’s top-ranked results across all four sources (vector, BM25, graph, open-file) rather than the completion path’s structural-context-first approach. Git blame and commit history are included specifically for “why” questions (why does this function work this way, why did this test start failing) — retrieved via a targeted git log -p on the relevant file/line-range rather than being part of the standard vector index.6.3 The Working-State Overlay — Consistency Without Re-IndexingThe freshness requirement (a rename 30 seconds ago must be reflected immediately) cannot be met by re-running the full indexing pipeline on every keystroke — that pipeline (AST parse, chunk, embed, symbol graph update) takes seconds to minutes per file, not milliseconds.Debounced diff streaming: the IDE extension does not stream every keystroke to the backend. It debounces edits (batching changes over a ~2–3 second window of no further edits, or on save) and sends a diff (not the full file) representing what changed. This keeps the overlay update traffic low relative to raw keystroke volume while still meeting the “30 seconds ago” freshness bar comfortably.Overlay application at retrieval time, not index time: rather than re-indexing the changed file (expensive), the overlay is applied as a query-time patch: when the Retrieval Orchestrator is about to return a chunk from file X, it checks whether an overlay exists for file X in the current developer’s session; if so, it applies the diff to the chunk’s text before returning it, and — critically — re-computes whether the chunk’s embedding is still valid or needs an on-the-fly re-embedding. For small edits (a renamed variable), the system uses a cheap heuristic (edit distance below a threshold) to decide the existing embedding is “close enough” and skip re-embedding; for larger edits (a function body substantially rewritten), it triggers a fast single-chunk re-embedding (a single small model call, not a full re-index) before returning that chunk.Overlay garbage collection: overlays are session-scoped and exist only in Redis with a TTL. When the developer commits their changes, the standard indexing pipeline picks up the new commit through its normal incremental indexing path, and the overlay for that file is cleared (since the base index now reflects the change). If a session times out without a commit (developer closes their laptop for the day without committing), the overlay simply expires — the next session starts fresh from the last committed index, which is the correct behaviour since there’s no other durable record of the uncommitted state.6.4 Agentic Loop Safety — Bounding Cost and Blast RadiusAn unbounded agentic loop (plan → execute → verify → repair → execute → verify → repair…) risks both runaway compute cost and, more importantly, a cascade of increasingly-wrong edits if the model gets stuck in a bad local pattern (repeatedly “fixing” a test in a way that breaks something else).Bounded retry with escalating context: the default retry limit (3 attempts) is not just a blunt cap — each retry provides progressively more diagnostic context to the Executor: attempt 1 uses the original plan and the direct test failure output; attempt 2 additionally includes a diff of what changed between the passing and failing test runs (isolating exactly what the previous attempt’s edit affected); attempt 3 includes the full history of all previous attempts in this task, explicitly framed as “these approaches did not work, try a different strategy.” This escalation gives each retry a genuinely better chance of success rather than blindly repeating a similar failed approach, while still guaranteeing termination.Sandboxed execution with resource limits: the Executor’s sandbox (a container per active agentic task) has hard resource limits — CPU, memory, and wall-clock time for any test/build command it runs — preventing a single agentic task from consuming disproportionate infrastructure resources (a common failure mode being an infinite loop or resource leak introduced by a bad generated edit, which without sandboxing could affect the shared build infrastructure).Blast-radius limiting via file-scope declaration: the Planner’s output explicitly declares which files each step will touch, before execution. If the Executor’s actual generated diff touches files outside this declared scope (a sign of the model going off-plan, e.g. “fixing” an unrelated file it noticed while investigating), the Verifier flags this as an anomaly requiring explicit developer approval even in approval_mode: "auto" — auto-approval only applies to edits within the originally-declared scope.6.5 Model Fine-Tuning Isolation — The LoRA Boundary in DetailSection 4.4 established that per-organisation fine-tuning uses isolated LoRA adapters rather than merging into the shared base model. This deep dive addresses the operational mechanics that make this boundary enforceable rather than merely policy.Physical data isolation for training jobs: each organisation’s fine-tuning job (triggered when an organisation opts in and accumulates sufficient feedback signal — accepted/rejected completions, chat ratings) runs against a data partition stored in an organisation-specific, access-controlled storage bucket. The training infrastructure’s job scheduler enforces that a training job for Organization A’s adapter can only mount Organization A’s data bucket — this is an IAM-level guarantee, not an application-level check that could be bypassed by a bug in the training code itself.Inference-time adapter selection: at inference time, the base model weights are loaded once and shared across all requests (this is what keeps the GPU fleet size manageable — you cannot afford a fully separate model per organisation at 10M developers). The organisation-specific LoRA adapter (a small set of low-rank weight deltas, typically

Source: Generative AI Pub — Published October 26, 2026 at 06:02 UTC — Category: Image AI