The Case for Markdown: Maximum Semantic Signal with Minimal Overhead

Markdown is far more than lightweight plain text. It serves as a clean, structured roadmap that Large Language Models (LLMs) parse efficiently.

Token Efficiency

Markdown delivers approximately 95% of the semantic structure found in HTML while adding only about 5% token overhead from formatting. HTML, by contrast, often burdens the context window with 18% or more boilerplate tags, scripts, and styling noise. Real-world benchmarks show Markdown can reduce token consumption by 20–30% compared to HTML equivalents, lowering API costs and enabling denser, more valuable context within the same window.

Attention Pattern Optimization

Transformer models develop stronger, more reliable attention patterns on consistently structured data. Markdown’s predictable hierarchy (# H1, ## H2, lists, code blocks) helps models focus on semantically important elements. Studies and observations indicate models trained on Markdown-heavy datasets achieve roughly 15% better performance on structured reasoning tasks compared to mixed or noisier formats.

Zero-Latency Retrieval for Small-to-Medium Datasets

For knowledge bases under ~100MB — personal notes, project wikis, small documentation sets — direct Markdown ingestion is unbeatable. You skip the entire “retrieval tax”: no vector database queries, no embedding lookups, no reranking. The model receives the full, unfiltered file immediately. This delivers instant context and eliminates variability introduced by chunking or search approximations.

The Case for RAG: Scalability with Consistent Sub-Second Performance

When your data grows into gigabytes or terabytes, stuffing the entire context hits hard limits on latency, cost, and model comprehension.

Consistent Latency

Modern high-performance RAG pipelines achieve vector search latencies around 30–50ms on optimized vector databases, with full retrieval — including hybrid search and reranking — often completing under 130ms on contemporary hardware. This predictability holds even as your archive scales massively.

Accuracy Lift Through Structure-Aware Chunking

Markdown’s natural headers (H1–H6) enable content-aware chunking, where the system retrieves coherent logical sections rather than arbitrary sentence fragments. This approach can improve retrieval accuracy by 40–60% over naive fixed-size splitting, because chunks align with actual concepts and maintain contextual integrity.

Hybrid Search Advantage

Markdown files pair perfectly with both semantic embeddings and classical keyword methods like BM25. Hybrid retrieval — combining dense vector similarity with sparse exact-term matching — delivers significantly more accurate results, often in the 40–60%+ improvement range for relevance in mixed query scenarios. This fusion captures both “meaning” and precise terminology that pure vector search might miss.

Comparison at a Glance

The Hybrid Strategy: Markdown as the Source of Truth

The most robust production architectures treat Markdown as the canonical source while layering RAG on top for scale.

Why This Wins

Debuggability. When the AI hallucinates or errs, you can instantly search your raw Markdown files with tools like grep or ripgrep to audit the underlying data. No opaque vector indices to reverse-engineer.

Version control and auditability. Store your “AI memory” in plain Markdown files under Git. Track every change to knowledge with full history, branches, and diffs — something binary vector stores or databases make cumbersome or impossible.

Seamless transition. Start simple with direct Markdown loading for small sets. As data grows, add indexing and RAG without rewriting your content. Markdown’s structure makes chunking, metadata extraction, and hybrid search far more effective.

In practice, many advanced systems index Markdown files directly for both BM25 keyword search and vector embeddings, preserving the format’s strengths while gaining scalability.

Final Takeaway for AI Architects

Choose Markdown-first when you can. It maximizes semantic density, minimizes latency and cost for moderate scales, and keeps your system transparent and maintainable.

Layer on RAG (especially hybrid) when scale demands it. It provides predictable performance and handles massive, dynamic knowledge without overwhelming context windows.

The winning blueprint isn’t Markdown or RAG — it’s Markdown as the source of truth, with high-quality RAG as the scalable retrieval layer.

This combination delivers the best of both worlds: clean, efficient data representation for LLMs and engineered retrieval that scales without sacrificing clarity or debuggability. Implement this thoughtfully, measure your specific latency/accuracy/cost tradeoffs, and iterate. In AI infrastructure, the format you choose today determines how fast — and how reliably — your systems will run tomorrow.

Most “my agent broke” investigations don’t end at the model. They end in memory design, tool scope, orchestration logic, or missing observability. That’s what this post is about.

Orchestration Patterns: The Architectures That Drive Agent Behavior

Orchestration is not a single technique. It’s a family of patterns — each with distinct trade-offs in reliability, cost, latency, and debuggability. Choosing the right one is a foundational architectural decision.

ReAct (Reason + Act)

ReAct is the most widely deployed orchestration pattern. At each loop iteration, the model generates a thought (explicit reasoning about what to do next), selects an action (a tool call with parameters), observes the result, and updates its context before the next iteration.

┌──────────┐     ┌──────────┐     ┌─────────────┐
│  Thought │────▶│  Action  │────▶│ Observation │
│          │     │          │     │             │
│ Reason   │     │ Tool call│     │ Result +    │
│ about    │     │ with     │     │ context     │
│ next step│     │ params   │     │ update      │
└──────────┘     └──────────┘     └──────┬──────┘
      ▲                                  │
      └──────────────────────────────────┘
              repeat until terminal

The tight thought-action-observation cycle makes the model’s reasoning auditable — you can trace exactly why it chose each tool call. This is its primary advantage for debugging.

The trade-off: ReAct is expensive. Every iteration requires a full model call. Long tasks accumulate latency and token cost linearly. It also assumes the model can plan one step ahead effectively. When tasks require reasoning ten iterations out, single-step reasoning degrades and the agent begins making locally reasonable but globally suboptimal decisions.

Chain-of-Thought (CoT) Planning

CoT separates planning from execution. The model produces an explicit multi-step plan before taking any action. Orchestration then executes that plan sequentially, feeding results back as observations.

┌─────────────────────────────────┐
│         Planning phase          │
│                                 │
│  Model generates full plan      │
│  before any tool is called      │
│                                 │
│  Step 1: fetch customer record  │
│  Step 2: check order history    │
│  Step 3: calculate refund       │
│  Step 4: send confirmation      │
└────────────────┬────────────────┘
                 │ optional: human review here
                 ▼
┌─────────────────────────────────┐
│         Execution phase         │
│                                 │
│  Orchestration executes steps   │
│  sequentially, feeds results    │
│  back as observations           │
└─────────────────────────────────┘

The advantage: planning upfront reduces model calls during execution, lowers cost, and creates a natural checkpoint for human review before any action fires.

The limitation: the upfront plan is static. If early tool calls return unexpected results, a pure CoT agent has no mechanism to revise mid-execution. In dynamic environments — where APIs fail, data is missing, or results differ from expectations — rigid plans break down. The fix is hybrid orchestration: plan upfront, but re-enter a ReAct loop whenever observations deviate significantly from plan assumptions.

Hierarchical Planning

For complex, long-horizon tasks, flat orchestration breaks down. Hierarchical planning introduces two levels: a high-level planner that decomposes the goal into sub-goals, and sub-agents or lower-level orchestrators that execute each sub-goal independently.

┌─────────────────────────────────────────┐
│           High-level planner            │
│                                         │
│  Decomposes goal into sub-goals.        │
│  Operates over abstract objectives.     │
│  Does not track tool-level details.     │
└────────┬──────────────┬─────────────────┘
         │              │
         ▼              ▼
┌────────────┐    ┌────────────┐
│  Sub-agent │    │  Sub-agent │
│     A      │    │     B      │
│            │    │            │
│ Specialized│    │ Specialized│
│ tools and  │    │ tools and  │
│ context    │    │ context    │
└────────────┘    └────────────┘
         │              │
         └──────┬───────┘
                ▼
     Results composed back
     by the high-level planner

This separation keeps context windows lean at both levels — the planner doesn’t need tool-level details, and sub-agents don’t need full task context. The trade-off is coordination complexity. Debugging hierarchical systems requires distributed tracing that crosses agent boundaries. Failures can originate at any level and manifest at another.

Selecting an Orchestration Pattern

┌──────────────────────────────────────────────────────────────────┐
│ Decision guide                                                   │
├──────────────────────────────────────────────────────────────────┤
│ Task has unpredictable step sequence       → ReAct              │
│                                                                  │
│ Task structure is known and stable,        → CoT with           │
│ or you need a human approval checkpoint      plan validation    │
│ before execution begins                                          │
│                                                                  │
│ Task exceeds reliable planning horizon     → Hierarchical       │
│ of a single model, or sub-tasks need         planning           │
│ domain-specific context that shouldn't                          │
│ bleed across a monolithic window                                 │
└──────────────────────────────────────────────────────────────────┘

In practice, most production systems are hybrids. The outer loop uses hierarchical decomposition; individual sub-tasks use ReAct; high-stakes sub-tasks inject a CoT plan-then-confirm step before execution.

Memory Architecture: What the Agent Knows and for How Long

Poor memory design is the leading cause of agent unreliability in production. The failure modes are subtle — the agent appears to work, then silently loses track of context, repeats completed steps, or contradicts earlier decisions. By the time the bug is visible, it’s several iterations into a corrupted state.

The design question isn’t “Do I need memory?” It’s: what must survive across runs, what can be reconstructed cheaply, and what should never be stored?

Short-Term / Working Memory: The Context Window

Working memory is the context window — the content assembled for the current model call. It holds the system prompt, current goal, active tool schemas, prior tool results, reasoning steps, and recent observations. It is fast, always available, and finite.

The architectural risk is context degradation: as the window fills over a long task, early content — including the original goal — competes with noise. The model doesn’t forget in a hard, detectable way. It degrades softly: subtly worse decisions, goal drift, increased hallucination rates. You will not see an error. You’ll see subtly wrong outputs.

Token usage across a session
─────────────────────────────────────────────
Iter   Context tokens   % of window
─────────────────────────────────────────────
1      2,260            14%   ░░░░░░░░░░
2      2,440            16%   ░░░░░░░░░░
3      4,890            31%   ███░░░░░░░
4      7,120            45%   ████░░░░░░
5      9,440            59%   █████░░░░░  ← alert threshold
6      3,200            20%   ██░░░░░░░░  ← summarization fired
7      5,100            32%   ███░░░░░░░
─────────────────────────────────────────────
Alert at 60%. Critical at 80%.
By 90%, degradation is already occurring.

Three mitigations address this:

Periodic summarization compresses accumulated reasoning and observations into a dense summary that replaces the raw content. Summarize what happened and what was decided, but keep the most recent tool results verbatim — they are the most decision-relevant content.

Smart eviction tracks which context elements are still decision-relevant and removes those that aren’t. Tool results from five iterations ago rarely need to be verbatim if a summary captures their outcome. This requires tagging context elements at insertion time.

Chunked execution breaks long tasks into sub-tasks, each with a clean context window. State is persisted externally between chunks and retrieved at the start of each. This is hierarchical orchestration applied to memory management.

Monitor token usage explicitly in your orchestration layer. Alert at 60% of the window limit — not 90%.

Long-Term Memory: External Storage

Long-term memory lives outside the context window and is retrieved on demand. It subdivides into three types that serve distinct purposes.

Episodic memory stores logs of past agent runs — what was attempted, what succeeded, what failed, and why. This is the mechanism by which agents improve over time without retraining. When starting a new task, the agent retrieves relevant past episodes and uses them as few-shot context. Episodic memory is the foundation of self-improving agents and is chronically underimplemented.

Design decision: log goal, plan, key decision points, tool call summaries, and final outcome. Discard raw observation payloads after summarization. Full transcripts are expensive and noisy; final-outcome-only logging loses the reasoning that explains why outcomes occurred.

Semantic memory stores factual knowledge and business data — the ground truth the agent needs to answer questions accurately. In production, this is implemented via RAG: a vector store the agent queries via a tool, returning relevant chunks inserted into the context window on demand.

Naive top-k vector similarity retrieval works for simple factual lookups but degrades on complex queries. More robust approaches use hybrid retrieval (combining vector similarity with BM25 keyword search), query decomposition (breaking a complex query into sub-queries before retrieving), and re-ranking (using a second model to score retrieved chunks for relevance before insertion). Each adds latency; the question is whether retrieval quality justifies it for your use case.

Procedural / Coordination memory is shared state in multi-agent systems — task queues, sub-task status, intermediate results, and cross-agent signals. This is the nervous system of multi-agent coordination.

The Memory Design Principle

For every category of information the agent needs, answer:
─────────────────────────────────────────────────────────
  What is its read/write frequency?
  How long must it survive?
  Who else needs access to it?
  What is the cost of losing it mid-task?
─────────────────────────────────────────────────────────
Wrong answers produce:
  Too much in context  →  high cost, latency, degradation
  Too little persisted →  broken workflow continuity
  Wrong things stored  →  security risk

Tool Design: Where Agent Reliability Is Won or Lost

The model decides which tools to call. The orchestration layer executes them. But the tools themselves determine whether those calls succeed reliably. Poorly designed tools are the second most common production failure mode — behind memory mismanagement — and the most fixable.

The Three Tool Categories

Outbound API calls connect the agent to external systems: Slack, GitHub, internal microservices. Primary failure modes are parameter errors, transient network failures triggering retry logic that can produce duplicate actions, and authentication expiry mid-task.

Custom functions are code you own, control, and test. They are deterministic and predictable. Use them whenever predictability matters more than flexibility — tax calculations, date arithmetic, schema validation, data transformations. Custom functions are the highest-reliability tool category and should be preferred for any computation that doesn’t require external state.

Data retrieval (RAG) grounds the agent in current facts rather than training data. Every domain-specific fact the agent needs to get right should have a retrieval path. If there is no tool that retrieves it, the agent will hallucinate it.

Tool Design Principles

Single responsibility. A tool called get_customer_data that conditionally fetches orders, preferences, or account status depending on parameters is three tools disguised as one. Split it. Single-responsibility tools are easier to test, easier to mock, and dramatically easier to debug when they fail.

Explicit, typed interfaces. Every parameter should have a type, a description, and — where applicable — a constraint on valid values. Design tool interfaces as if a careful junior engineer with no context will be calling them at 3 a.m. That junior engineer is the model.

Structured, predictable output. Tools should return consistent schemas regardless of the internal code path taken. A tool that returns a dict in success cases and a string in failure cases forces the model to handle multiple output shapes — and it will handle them inconsistently. Always return a typed, structured result. Make failure states as information-dense as success states — the model needs failure details to decide what to do next.

Idempotency where possible. Agents retry. If your tool has side effects — writes, sends, charges — idempotency prevents those side effects from compounding on retry. Include an idempotency key in any tool that performs a write operation.

Hard usage limits. Tools that make external writes or have financial consequences should have rate limits enforced in the tool layer, not just in prompting. The model cannot be reliably instructed to self-limit. Enforce limits structurally.

Common Tool Anti-Patterns

Anti-pattern         What goes wrong
──────────────────────────────────────────────────────────────
God tool             One call does everything. No intermediate
                     checkpoint. Failure is unattributable.

Under-described      A tool named "search" with no description
schema               of what it searches, what format queries
                     take, or what the output looks like.
                     The model guesses. Sometimes correctly.

Swallowed errors     Returns {"status": "ok", "result": null}
                     on failure. Model interprets "ok" as
                     success, receives null, produces nonsense.

Missing retrieval    The agent hallucinates because no tool
coverage             covers a data source it needs. Audit
                     coverage before you ship.
──────────────────────────────────────────────────────────────

Multi-Agent Coordination: When One Agent Isn’t Enough

Single-agent architectures hit two ceilings as task complexity grows: context window capacity and specialization. Multi-agent architectures address both by decomposing tasks across specialized agents that each operate with lean, relevant context.

This is not free. Coordination introduces failure modes that don’t exist in single-agent systems.

The Orchestrator-Worker Pattern

┌─────────────────────────────────────────────────┐
│                  Orchestrator                   │
│                                                 │
│  Decomposes goal → delegates sub-tasks →        │
│  synthesizes results → determines next steps    │
└──────────┬──────────────────────┬───────────────┘
           │                      │
           ▼                      ▼
┌──────────────────┐   ┌──────────────────────────┐
│   Worker A       │   │   Worker B               │
│                  │   │                          │
│ Domain-specific  │   │ Domain-specific           │
│ tools + context  │   │ tools + context           │
│                  │   │                          │
│ Executes sub-task│   │ Executes sub-task        │
│ independently    │   │ independently            │
└──────────┬───────┘   └──────────────┬───────────┘
           │                          │
           └────────────┬─────────────┘
                        ▼
              Results composed back
              by the orchestrator

The design decisions that determine whether this works:

Sub-task interface design. The sub-task description must be self-contained — it cannot assume the worker has access to the orchestrator’s broader context. This is the most commonly underestimated challenge. Poorly scoped sub-task descriptions produce workers that misinterpret their assignment and return irrelevant results.

Result schema standardization. Worker agents must return results in a schema the orchestrator can reason about reliably. Define a standard result envelope: status, payload, confidence indicator if applicable, and a brief human-readable summary the orchestrator can use for planning.

Failure propagation. When a worker fails, the orchestrator needs to know whether the sub-task failed completely, partially, or produced a low-confidence result. A binary success/failure signal is insufficient for a planning agent that may have recovery options.

Shared State and Write Ownership

In multi-agent systems, coordination memory is a shared persistent store that all agents can read and write. The critical design decision is write ownership. In a well-designed multi-agent system, each piece of shared state has exactly one agent responsible for writing it. Concurrent writes from multiple agents without coordination produce race conditions that exhibit as intermittent, mysterious failures in production. Use optimistic locking or task-claim mechanisms to enforce write ownership at the state layer — not at the prompt layer.

When Multi-Agent Is the Wrong Choice

Multi-agent architecture is frequently over-applied. It is the right choice when the task genuinely exceeds the planning horizon of a single model, when different sub-tasks require domain-specific context that shouldn’t bleed across a monolithic window, or when parallelism is a hard latency requirement.

It is the wrong choice when coordination overhead exceeds task complexity, when a single agent with good context management would suffice, or when the team doesn’t yet have robust observability across agent boundaries. Many systems that claim to need multi-agent coordination are actually single agents with context management problems. Fix the context management first.

Observability: Tracing the Loop, Not Just the Edges

Standard application monitoring tracks requests and responses. Agent observability has to track the loop — every iteration, every tool call, every reasoning step, and the token budget at each point. A single user request can produce dozens of model calls and tool executions. You need to correlate all of them into a coherent session trace.

What to Emit at Every Loop Iteration

This is the minimum viable trace event. Emit one per iteration, structured, before the next iteration begins:

Iteration trace event
─────────────────────────────────────────────
session_id            Ties all iterations together
iteration_n           Which loop cycle this is
context_tokens        Tokens in context at start
thought               Model's reasoning output
tool_name             Tool selected (or "none")
tool_params           Parameters as constructed
tool_latency_ms       Wall time for tool execution
tool_result_size      Tokens in tool response
context_tokens_after  Tokens in context after update
error                 null if clean; typed if not
timestamp_ms          Unix ms at iteration start

Context tokens before and after is the most important pair. The delta tells you how fast the window is filling. If context_tokens_after is growing faster than context_tokens is shrinking from summarization, you will hit degradation before the task completes.

The Session Trace Structure

A complete session trace is a tree, not a flat log:

session: s_abc123
│
├── iteration: 1
│   ├── model_call: thought generation      (320ms, 1,840 tokens in)
│   ├── tool_call:  search_knowledge_base   (210ms, result: 420 tokens)
│   └── context snapshot: 2,260 tokens
│
├── iteration: 2
│   ├── model_call: thought generation      (290ms, 2,260 tokens in)
│   ├── tool_call:  call_crm_api            (850ms, result: 180 tokens)
│   └── context snapshot: 2,440 tokens
│
├── iteration: 3
│   ├── model_call: thought generation      (340ms, 2,440 tokens in)
│   ├── tool_call:  summarize_context       (-1,100 tokens evicted)
│   └── context snapshot: 1,520 tokens     ← summarization fired
│
└── iteration: 4
    ├── model_call: thought generation      (305ms, 1,520 tokens in)
    ├── tool_call:  send_email              (120ms, result: 40 tokens)
    └── terminal: goal_achieved

The context snapshot after iteration 3 shows summarization working correctly — the window dropped from 2,440 to 1,520 tokens. Without this tree structure, you can’t see that event or attribute subsequent behavior to it.

Latency Attribution

Total session latency breaks down into four buckets. You cannot optimize what you don’t attribute:

Session latency breakdown
─────────────────────────────────────────────────────
                                           % of total
Model inference latency     ████████████     48%
Tool execution latency      ████████████████ 38%
  └─ call_crm_api  ████████ 22%
  └─ search_kb     ████     10%
  └─ send_email    ██        6%
Context assembly latency    ████              9%
Orchestration overhead      █                5%
─────────────────────────────────────────────────────
Total session wall time: 4,820ms

In most production agents, tool latency dominates — not model inference. You find this only with per-tool timing. Without attribution, optimization effort lands on the wrong layer.

Error Classification

Mixing error types in a single metric makes debugging impossible. Classify every error at emission time:

Error taxonomy
──────────────────────────────────────────────────────────────
Class           Type              Recovery action
──────────────────────────────────────────────────────────────
Model errors
  malformed_tool_call   Return structured error to model,
                        allow one retry with correction hint
  goal_drift            Reinject original goal, flag for
                        human review if recurs
  reasoning_loop        Detect via repeated tool calls,
                        terminate and surface to operator

Tool errors
  transient_failure     Exponential backoff, max 3 retries
  permanent_failure     Surface to model as context update,
                        trigger replanning
  parameter_invalid     Return typed error with correction
                        schema, allow model to revise call
  timeout               Log latency, treat as transient,
                        apply retry policy

Orchestration errors
  context_overflow      Emergency summarization before
                        next model call
  termination_failure   Hard stop, checkpoint state, alert
  state_corruption      Halt immediately, do not recover,
                        escalate
──────────────────────────────────────────────────────────────

Alert Rules

Alert rules
──────────────────────────────────────────────────────────────
Signal                  Threshold       Action
──────────────────────────────────────────────────────────────
Context tokens          > 60% limit     Trigger summarization
Context tokens          > 80% limit     Force summarization,
                                        suspend iteration
Tool error rate         > 2/session     Log, notify on-call
                                        if task is high-stakes
Repeated tool call      Same tool ≥ 3   Detect retry spiral,
                        consecutive     terminate session
Session duration        > P99 baseline  Flag for review
Termination condition   Not reached     Hard stop at max
not triggered           by max_iter     iteration limit
State corruption        Any occurrence  Halt immediately
──────────────────────────────────────────────────────────────

The repeated tool call rule catches the most destructive failure mode: a retry spiral where the agent calls the same failing tool indefinitely. This burns tokens, time, and potentially triggers external side effects on every call.

Production Guardrails and Recovery

Guardrails: Constraining the Action Space

Guardrails are structural constraints on what the agent can do. They are enforced in the orchestration and tool layers — not in prompting — and cannot be overridden by model output.

Input guardrails validate the goal and context before the loop begins. Malformed goals, goals that reference unavailable tools, or goals that exceed defined scope should be rejected at intake — not discovered three iterations into an expensive loop.

Tool call validation checks every tool call the model generates before execution. Validate parameter types, check values against allowed ranges, and verify the requested tool is in scope. Reject malformed calls and return a structured error to the model. The model can often recover from a rejected call if the error message is informative.

Output guardrails screen the agent’s final output before it’s returned or acted on. For agents that produce content consumed by users, check for policy violations or hallucinated citations. For agents that take real-world actions, validate that the proposed action is within defined operational bounds before execution.

Rate and scope limits enforce hard ceilings on the real-world impact any single session can have: maximum API calls per tool per session, maximum financial transactions per hour, maximum records modified per run. These live at the infrastructure layer — not in the prompt.

Human-in-the-Loop Checkpoints

Not every action should be auto-executed. High-stakes, irreversible, or high-uncertainty actions should route through a human approval checkpoint before execution. The classification logic — which actions are always auto-approved, always human-approved, and conditionally approved — must be enumerated explicitly in a policy configuration that the orchestration layer enforces. Don’t rely on the model to classify this.

Failure Recovery

Structured error handling at the tool layer means every tool returns a consistent failure schema with a failure type (transient, permanent, recoverable), a retry recommendation, and a human-readable explanation. The orchestration layer uses this to route failures correctly.

Loop termination conditions must be explicitly defined. The loop ends when: the goal is achieved, a stopping condition fires (maximum iterations reached, token budget exhausted, error threshold exceeded), or a terminal failure occurs. Without explicit termination logic, loops run until they hit a hard timeout — wasting resources and potentially taking partial, inconsistent actions along the way.

State checkpointing persists the agent’s current state at defined intervals so that a crash mid-task can be resumed rather than restarted from scratch. The checkpoint includes: current goal, completed steps, tool results obtained, and the current context summary.

Audit trails record every action taken, every tool called, and every model decision made during a session. For agents that modify external state, the audit trail should include enough information to reverse each action — not just that an action was taken. Design the trail as if you’ll need to reconstruct and undo a session in production under time pressure. You will.

The Guiding Principles

Design memory like a thoughtful schema: persist what must survive, reconstruct what can be rebuilt cheaply, never store what shouldn’t be retained.

Design tools like clean API contracts: explicit, single-purpose, hard to misuse.

Design orchestration around failure, not the happy path: termination conditions, error classification, retry bounds, and checkpointing are not edge case concerns — they are the architecture.

Design coordination for correctness before performance: enforce write ownership structurally, enumerate dependencies explicitly, and introduce parallelism only when sequential execution has proven insufficient.

Instrument the loop, not just the edges: the inputs and final outputs of a session tell you almost nothing about why it failed. The per-iteration trace — token trajectory, tool selection, latency attribution, error classification — tells you everything.

The model gets the hype. Memory, tools, orchestration, observability, and operations determine whether your agent becomes a reliable system or a cautionary tale.

Why self-host?

Cloud-hosted LLM APIs are convenient, but they come with trade-offs that matter at enterprise scale: data leaves your network on every inference call, costs scale linearly with volume (and models are getting longer), and you have no control over model versioning, rate limits, or uptime SLAs. Self-hosting on Kubernetes gives you full control over the stack — at the cost of having to build and operate that stack yourself.

This guide covers every layer of a production LLM serving platform using exclusively open-source tools. We’ll go from a raw HTTP request all the way down to GPU silicon, explaining why each component exists and how they fit together.

Architecture overview

The full stack has seven layers, each solving a distinct set of problems:

┌─────────────────────────────────────────────────────────────────┐
│                         CLIENTS                                 │
│          Web apps · Mobile · CLI agents · Internal APIs         │
└───────────────────────────┬─────────────────────────────────────┘
                            │ HTTPS
┌───────────────────────────▼─────────────────────────────────────┐
│                   API GATEWAY LAYER                             │
│   cert-manager (TLS) · Keycloak (OIDC) · Rate limiting         │
│   WAF · LLM Guard · Envoy / Kong / Traefik + Gateway API       │
└───────────────────────────┬─────────────────────────────────────┘
                            │ cache hit? return early
┌───────────────────────────▼─────────────────────────────────────┐
│                    LLM CACHE LAYER                              │
│        Exact-match (Redis) · Semantic (Qdrant) · KV prefix      │
└───────────────────────────┬─────────────────────────────────────┘
                            │ cache miss → forward to inference
┌───────────────────────────▼─────────────────────────────────────┐
│                  INFERENCE LAYER (vLLM)                         │
│   Router · Tensor parallelism · Pipeline parallelism            │
│   Continuous batching · Paged attention · Speculative decoding  │
└───────────────────────────┬─────────────────────────────────────┘
                            │ kernel calls
┌───────────────────────────▼─────────────────────────────────────┐
│                  GPU COMPUTE LAYER                              │
│     NVIDIA device plugin · MIG · NVLink/RoCE · KEDA autoscale  │
└───────────────────────────┬─────────────────────────────────────┘
                            │ model weights loaded from
┌───────────────────────────▼─────────────────────────────────────┐
│               MODEL STORAGE & REGISTRY                          │
│          MinIO (S3) · MLflow · Rook-Ceph · Init containers      │
└─────────────────────────────────────────────────────────────────┘

    Cross-cutting concerns (all layers)
    ├── Observability: Prometheus · Grafana · Jaeger · Loki
    └── Control plane: ArgoCD · Vault · Istio / Cilium · OPA Gatekeeper

Let’s walk through each layer in detail.

Layer 1: The API gateway

The gateway is the single entry point for all LLM traffic. It does a lot of work before a single token gets generated.

Why Kubernetes Gateway API (not legacy Ingress)?

The older Kubernetes Ingress API conflates infrastructure and application concerns. The Kubernetes Gateway API (GA and mature in 2026) separates them cleanly:

• GatewayClass — infrastructure team owns the controller (Envoy Gateway, Kong, Traefik, etc.).
• Gateway — defines listeners, TLS, and ports.
• HTTPRoute — application/ML teams define routing per model endpoint.

This separation is essential in enterprises where platform and ML teams are distinct.

TLS termination

Use cert-manager for automated certificates (Let’s Encrypt for public endpoints or internal CA for private clusters). The gateway terminates TLS; internal east-west traffic uses mTLS via the service mesh.

Authentication and authorization

Use OIDC/OAuth2 with Keycloak (or equivalent). Map scopes to models for cost governance — for example, scope:large-model unlocks higher-tier inference and can gate access to expensive 70B+ parameter models.

Rate limiting

LLMs require dual limits:

• Requests per minute (protect against abuse).
• Tokens per hour/day (prevent cost overruns).

Implement via Envoy rate-limit service (backed by Redis) or native plugins in Kong/Traefik. Charge token usage asynchronously after generation.

WAF, DDoS protection, and prompt guarding

Apply OWASP Core Rule Set (via Envoy Lua/WASM or ModSecurity) and a lightweight LLM Guard sidecar for prompt injection, PII scrubbing, and toxicity scoring.

Audit logging and distributed tracing

Emit OpenTelemetry spans for every request (user identity, model, token counts, cache status, latency breakdown). Ship to Jaeger + Prometheus + Loki.

Gateway implementation choices

All three support the Gateway API spec. Choose based on team expertise. Emerging options like Agentgateway add AI-specific routing (agent-to-agent traffic, tool call routing) that may matter for multi-agent workloads.

Layer 2: The LLM cache

Cache hits eliminate GPU work entirely — this is the highest-leverage optimization in the stack.

Three tiers of caching

1. Exact-match cache (Redis/Valkey)

SHA256 of (model + messages + parameters). Great for FAQs, repeatable batch jobs, and any deterministic prompt patterns. Typical hit rate: 5–20%.

2. Semantic cache (GPTCache + Qdrant, or Redis with vector search)

Embedding-based cosine similarity lookup. Tune the threshold carefully — 0.92–0.95 for factual work, disabled for creative tasks. Typical hit rate: 15–40%. The tradeoff is a small added latency for the embedding lookup and risk of incorrect hits if your threshold is too loose.

3. vLLM prefix KV cache (GPU-resident)

vLLM hashes KV blocks for shared system prompts or RAG contexts and reuses them across requests. This is the most powerful tier: for chatbot or RAG workloads where most requests share the same system prompt, prefix cache hit rates of 60–90% are common, dramatically cutting time-to-first-token (TTFT).

Combine all three tiers. The exact-match check is microseconds; the semantic check adds a few milliseconds; the prefix cache operates within vLLM transparently.

Layer 3: vLLM inference

vLLM remains the highest-throughput open-source inference engine for LLMs. Its core innovations are worth understanding because they directly shape how you size and operate it.

Paged attention

Traditional inference pre-allocates a contiguous KV cache block per sequence. vLLM uses paged attention — non-contiguous physical blocks allocated on demand — which eliminates memory fragmentation and enables much higher concurrent sequence counts on the same GPU memory.

Continuous batching

Rather than processing a fixed batch then starting a new one, vLLM uses continuous batching (also called iteration-level scheduling): new requests join the batch mid-iteration as soon as a slot frees. This eliminates GPU idle time between requests and is a primary driver of throughput improvement over naive serving.

Prefix caching

vLLM computes and caches KV blocks for common prefixes (system prompts, RAG contexts). Subsequent requests that share the prefix skip the prefill computation for those tokens entirely. At scale, this is often the difference between a cluster that fits in budget and one that doesn’t.

Parallelism strategies

For models exceeding single-GPU VRAM:

• Tensor parallelism: Splits weight matrices across GPUs within a node. Uses NVLink/NVSwitch for fast all-reduce. Scales to 8 GPUs per node efficiently.
• Pipeline parallelism: Splits model layers across nodes. Uses RoCE or InfiniBand for inter-node communication. Introduces pipeline bubbles but enables serving models that won’t fit on a single node.

Combine both for the largest models.

Quantization

Production recommendations in approximate priority order:

• FP8 (H100/H200): Near-lossless quality, ~2× memory reduction, supported natively in hardware.
• AWQ: Excellent quality/size tradeoff for A100 and older hardware.
• GPTQ: Widely supported, slightly lower quality than AWQ at equivalent bit-width.

Leave 10–15% GPU memory headroom above your model’s requirements for KV cache and kernel overhead.

Context length as an operational lever

max_model_len — the maximum sequence length vLLM will accept — directly controls KV cache memory consumption per slot. Longer contexts consume proportionally more KV cache memory, which reduces the number of concurrent sequences the engine can hold and increases TTFT.

Set max_model_len deliberately rather than leaving it at the model’s architectural maximum. For most production workloads, a limit of 8k–32k tokens is sufficient and meaningfully improves throughput. Monitor vllm:gpu_cache_usage_perc — if it runs consistently above 85%, reducing max_model_len is often the fastest way to recover headroom before adding hardware.

Speculative decoding

A draft model generates candidate tokens at low cost; the target model verifies them in parallel. Effective for latency-sensitive workloads where output is predictable (code, structured data). Adds complexity — evaluate whether the latency gain justifies the operational overhead.

Advanced: disaggregated inference

For very large-scale deployments (many thousands of requests per day), consider disaggregated inference: separate prefill pods (compute-intensive, large batches) from decode pods (memory-bandwidth-intensive, streaming). The llm-d project implements this pattern and integrates with vLLM. It adds significant operational complexity but can substantially improve hardware utilization at scale.

Layer 4: GPU compute

NVIDIA GPU Operator

Install via Helm. It automates driver installation, device plugin deployment, DCGM exporter setup, and container toolkit configuration. Without it, GPU node management becomes a manual nightmare across OS upgrades and Kubernetes versions.

GPU sharing

• MIG (Multi-Instance GPU): Available on A100/H100+. Creates hardware-isolated partitions with dedicated memory slices and compute engines. The right choice when you need strict isolation between workloads (multi-tenant or mixed-criticality).
• Time-slicing: Software-level sharing configured via GPU Operator. Lower overhead than MIG, no memory isolation. Suitable for dev/test environments or homogeneous workloads.

High-speed interconnects

NVLink/NVSwitch for intra-node GPU communication (tensor parallelism collectives). RoCE v2 or InfiniBand for inter-node communication (pipeline parallelism and distributed training). For inference-only clusters, RoCE with RDMA is typically sufficient and significantly cheaper than InfiniBand.

Autoscaling with KEDA

CPU/GPU utilization is a poor signal for LLM autoscaling — a GPU can be 80% utilized but handling requests efficiently, or 20% utilized but with a growing queue. Use KEDA with a Prometheus trigger on vllm:num_requests_waiting.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      query: sum(vllm:num_requests_waiting{model="llama-3-70b"})
      threshold: "8"

Target approximately 5–10 pending requests per replica as your threshold. Combine with Cluster Autoscaler or Karpenter for node-level scaling (GPU nodes take 3–5 minutes to join; plan for that latency in your scale-out strategy).

Readiness probes and rolling deploys

vLLM takes several minutes to load weights before it can serve requests — 5–10 minutes for a 70B model depending on storage throughput. Without a correctly configured readiness probe, Kubernetes will route traffic to pods that are still loading and immediately return errors.

Configure the readiness probe against vLLM’s /health endpoint with a sufficiently long initialDelaySeconds (or use startupProbe with a high failureThreshold to avoid timing fights). Set terminationGracePeriodSeconds long enough to drain in-flight streaming responses — 120–300 seconds is typical. During a rolling deploy, the old replica keeps serving until the new one passes its readiness check.

Graceful scale-down

KEDA’s default scale-down is aggressive. An in-progress streaming response can be 60+ seconds; a pod termination that fires mid-stream silently drops the connection. Set scaleDown.stabilizationWindowSeconds in the ScaledObject (300–600 seconds is reasonable) and configure a preStop lifecycle hook that waits for active connections to drain before the pod accepts SIGTERM. Pair this with minReplicaCount: 1 to prevent complete scale-to-zero for latency-sensitive endpoints.

Layer 5: Model storage and registry

Getting weights into MinIO

Most teams source weights from Hugging Face Hub. The practical pipeline: download once to a secure jump host or CI runner (huggingface-cli download or hf_transfer for speed), verify the SHA256 checksum against the Hub’s published value, apply your quantization step if needed (AWQ/FP8 conversion offline before serving), then push to MinIO with versioned paths (/models/llama-3-70b/awq-4bit/v1.2/). Never pull directly from Hugging Face into production inference pods — it bypasses your checksum verification, creates an external dependency at pod startup, and is slow.

For air-gapped environments, mirror the weights to an internal registry during the ingestion pipeline and gate on checksum + license validation before the weights are considered promotion-eligible in MLflow.

Object storage (MinIO)

Store model weights in MinIO (S3-compatible). Version with prefixed paths (/models/llama-3-70b/v2.1/). Use init containers to pull weights into a shared volume before the inference pod starts, or mount directly via a CSI driver.

Model registry (MLflow)

Track model lineage, quantization config, evaluation results, and deployment history. Enables A/B testing via weighted routing in the gateway. Integrate with your CI/CD pipeline so model promotions are gated on evaluation thresholds.

Persistent storage (Rook-Ceph)

For RWX access patterns (multiple inference pods reading weights simultaneously), Rook-Ceph provides a self-managed distributed filesystem. Alternatively, MinIO with parallel downloads from multiple replicas works well if you cache locally on the node.

Always verify checksums after weight downloads — a corrupted weight file produces subtle, hard-to-diagnose inference errors.

Layer 6: Observability

LLMs have different failure modes than typical services. Your dashboards need to reflect that.

Key vLLM metrics

GPU metrics (DCGM exporter)

Monitor GPU utilization, memory bandwidth, NVLink throughput, and GPU temperature. Throttling events (SM clock drops) indicate thermal or power issues that degrade throughput without obvious errors.

Tracing and logging

Propagate trace context from the gateway through to vLLM. Log request_id, user_id, model, token counts (prompt + completion), cache tier hit (exact/semantic/prefix), TTFT, TPOT, and any guardrail flags. This data is essential for cost attribution and debugging latency regressions.

Layer 7: The platform control plane

GitOps (ArgoCD / Flux)

Every cluster resource — deployments, ScaledObjects, policies, secrets references — lives in Git. ArgoCD syncs it. No manual kubectl apply in production. This makes rollbacks, audits, and multi-cluster management tractable.

Secrets management (HashiCorp Vault)

Dynamic secrets for database credentials, API keys, and model registry tokens. Use the Vault Agent sidecar or External Secrets Operator to inject secrets as environment variables or files. Avoid Kubernetes Secrets for anything sensitive — they’re base64-encoded in etcd, not encrypted by default.

Service mesh (Istio / Cilium)

mTLS for all east-west traffic. Zero-trust: pods cannot communicate unless explicitly permitted. Cilium (eBPF-based) has lower overhead than Istio’s sidecar model and is the better choice for latency-sensitive inference traffic. Use Istio if you need its traffic management features (circuit breaking, retry policies, mirroring).

Policy (OPA Gatekeeper)

Admission control policies that enforce:

• All GPU workloads must have resource limits set.
• No :latest image tags in production namespaces.
• All pods must carry cost-attribution labels (team, model, environment).
• GPU nodes must have the appropriate taint/toleration pair.

Network policies

Restrict pod communication explicitly. Inference pods should only accept traffic from the gateway and observability scrape jobs. They should only egress to the model registry and observability collectors. Default-deny egress on inference namespaces prevents data exfiltration.

Full request lifecycle

Putting it together, a single inference request flows through:

1. TLS termination at the gateway — client presents token.
2. OIDC validation — Keycloak confirms identity, scope checked against requested model.
3. Rate limit check — token budget and request rate verified in Redis.
4. Prompt guard — LLM Guard sidecar scans for injection and PII.
5. Exact-match cache check — SHA256 lookup in Redis.
6. Semantic cache check — embedding lookup in Qdrant (on cache miss).
7. vLLM routing — request forwarded to least-loaded replica.
8. Prefix cache check — vLLM checks KV block hashes for shared prefix.
9. Prefill — prompt tokens processed, KV cache populated.
10. Decode — tokens generated and streamed back via SSE.
11. Token accounting — usage logged asynchronously for cost attribution.
12. Trace closed — span exported to Jaeger with full latency breakdown.

Latency and cost are controlled at steps 5, 6, and 8. Observability at every step.

When to evolve beyond pure vLLM deployments

The stack described here is ideal for focused, high-performance serving of a small number of models. For larger-scale scenarios, consider layering additional tooling:

KServe (with vLLM runtime or LLMInferenceService) adds a standardized control plane for multi-model governance, canary rollouts, and heterogeneous workloads (LLMs + embeddings + vision models). It keeps vLLM as the inference engine while providing higher-level abstractions for model lifecycle management.

llm-d adds advanced distributed routing and disaggregated prefill/decode separation on top of vLLM. Worth evaluating when you have dedicated hardware for prefill compute and want to maximize utilization of decode capacity separately.

These are additive layers — they don’t replace vLLM, they orchestrate it.

Gaps worth filling for your environment

This guide covers the core serving platform. Depending on your context, you’ll also need to address:

• Multi-tenancy: Dedicated namespaces per team (strong isolation, higher overhead) vs. shared inference pool with metering at the gateway (better utilization, more complex chargeback).
• Fine-tuning: Separate GPU node pools with tools like Axolotl or LitGPT. Never share fine-tuning and inference workloads on the same nodes — memory pressure from training jobs will degrade inference latency unpredictably.
• Multi-region / HA: Global load balancing across clusters + MinIO cross-region replication for weight distribution.
• Cost attribution: Token metering at the gateway + GPU-hour tracking via OpenCost or custom labels. Essential for chargeback and for identifying which teams/models are driving cost.
• A/B testing: Weighted routing in Gateway API HTTPRoute + Grafana dashboards comparing TTFT/TPOT/quality metrics between model versions.
• Security hardening: Image scanning in CI, runtime sandboxes (gVisor or Kata Containers for multi-tenant isolation), and prompt/response logging policies that avoid persisting sensitive data.

Summary: the full open-source stack

This stack delivers full data sovereignty, predictable costs, and high performance with no per-token pricing.

The operational investment is real. You need platform engineers who understand Kubernetes, GPU workloads, and distributed systems. But for teams with strict compliance requirements, high inference volume, or custom/fine-tuned models, self-hosting on Kubernetes is overwhelmingly worthwhile — and the tooling has matured to the point where the gap with managed services has narrowed significantly.

The term gets applied to everything from a ChatGPT thread with a button to systems that autonomously manage deployments and customer operations. That imprecision isn’t just annoying — it directly shapes the architectures you choose, the failure modes you inherit, and whether your “agent” actually ships value or becomes a production liability.

Let’s fix that.

The Operational Definition

An AI agent is an application that places a language model in a continuous loop: it perceives its environment through inputs and context, reasons about the next action, executes that action via tools, observes the result, and iterates — until the goal is achieved or a stopping condition fires.

Shorter: a language model in a loop, augmented with tools, to accomplish an objective.

This think-act-observe cycle is what separates agents from static inference. The loop introduces probabilistic control flow, which is simultaneously the source of power and the root of every 2 a.m. incident. Understanding that trade-off in detail is the entire discipline.

┌───────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
│  Perceive │────▶│  Reason   │────▶│    Act    │────▶│  Observe  │
│           │     │           │     │           │     │           │
│ Context + │     │ Plan next │     │ Call a    │     │ Update    │
│ inputs    │     │ action    │     │ tool      │     │ context   │
└───────────┘     └───────────┘     └───────────┘     └─────┬─────┘
      ▲                                                      │
      └──────────────────────────────────────────────────────┘
                     loops until goal is achieved

The Four Components

Every agent is built from the same four parts. Three get the attention. One determines whether it survives production.

The Model — the reasoning engine. The LLM analyzes context and decides what to do next. It never executes actions directly; it only plans them. Its output quality is entirely bounded by the quality of the context you feed it. Weak context produces confident stupidity.

Tools — the interface to the real world. APIs, functions, databases, external services. Tools let the agent fetch live data, run computations, and take real-world actions. Without them, you have expensive autocomplete, not an agent.

The Orchestration Layer — the nervous system. This owns the loop: planning, state tracking, memory retrieval, tool dispatch, error recovery, and termination logic. Orchestration is what converts isolated model calls into coherent, goal-directed behavior. Most production failures originate here, not in the model.

Runtime and Deployment Services — what separates prototypes from production. Monitoring, logging, security, human-in-the-loop approvals, and observability. Skipping this layer is the most common reason capable demos become unreliable systems.

┌─────────────────────────────────────────────┐
│               Agent session                 │
│                                             │
│   ┌─────────────┐      ┌─────────────────┐  │
│   │    Model    │◀────▶│  Orchestration  │  │
│   │  (reasons)  │      │  (owns the loop)│  │
│   └─────────────┘      └────────┬────────┘  │
│                                 │           │
│   ┌─────────────────────────────▼─────────┐ │
│   │                Tools                  │ │
│   │   APIs · functions · databases · RAG  │ │
│   └───────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
                     │
       ┌─────────────▼─────────────┐
       │    Runtime & deployment   │
       │  observability · guardrails│
       │  logging · human-in-loop  │
       └───────────────────────────┘

Chatbot, Copilot, Agent: Different Architectures, Not a Capability Ladder

These aren’t points on a spectrum — they’re fundamentally different system designs with different failure modes and operational contracts.

┌─────────────┬──────────────┬──────────────┬──────────────────────────────┐
│             │   Chatbot    │   Copilot    │           Agent              │
├─────────────┼──────────────┼──────────────┼──────────────────────────────┤
│ State       │ Stateless    │Session-scoped│ Persistent                   │
│ Tools       │ None         │ Suggests     │ Executes                     │
│ Autonomy    │ Reactive     │ Assistive    │ Goal-directed                │
│ Fails via   │Hallucination │Bad suggestion│ Unexpected cascading action  │
│ Who catches │ You review   │ You act on   │ Nobody, unless you built     │
│ failures    │ output       │ output       │ the guardrails               │
└─────────────┴──────────────┴──────────────┴──────────────────────────────┘

The failure mode column is what matters. Agents introduce a new risk class: autonomous actions with real consequences — financial, legal, reputational. Design your system around its actual failure modes, not the ones you hope it avoids. Audit logs, guardrails, and rollback mechanisms aren’t polish — they’re load-bearing.

The Autonomy Spectrum

Not every agent needs maximum autonomy. Matching autonomy level to the task is one of the most underrated architectural decisions.

Level 0 — Pure reasoning
  Model only. No tools, no state.
  Capable for analytical tasks where all needed context fits in the prompt.
  │
Level 1 — Connected
  Model + tools for data fetching and simple actions.
  Where most production agents actually live today.
  Most should stay here.
  │
Level 2 — Strategic
  Multi-step planning with cross-iteration context maintenance.
  The model reasons about sequences, not just immediate next steps.
  Context degradation becomes a real risk.
  │
Level 3 — Collaborative
  Multiple specialized agents coordinated by an orchestrator.
  Coordination overhead rises sharply.
  Communication failures become the new bottleneck.
  │
Level 4 — Self-evolving
  The system modifies its own tools, prompts, or behavior.
  Still largely research or heavily guarded production.
  Requires sandboxing and mandatory human oversight.

Most teams ship Level 1 while targeting Level 2. That gap is where reliability dies.

The rule: start at the lowest level that solves the problem. Only move up when reliability is proven and the use case genuinely demands it. Higher levels add coordination complexity faster than they add value.

How Your Job Changes as a Builder

Traditional software development is deterministic — you code every path explicitly. Agent development is closer to directing a capable but non-deterministic system: you define the goal, equip it with tools, craft the system prompt, and shape behavior through context at each iteration.

Control flow becomes probabilistic. The instinct is to compensate with endless conditionals and guardrails — but often that means fighting the architecture rather than leveraging it.

Your highest-leverage skill shifts from writing logic to context engineering: the deliberate assembly of what goes into the model’s context window at each iteration. Every token costs latency, money, and signal quality. The model’s reasoning quality is bounded entirely by the quality of this assembly.

Expect to spend more time on observability and failure recovery than on the happy path. The happy path in an agent is a minority of real sessions.

The Questions to Ask Before You Build

Skip “Is this an agent?” Ask: what level of autonomy does this task actually require?

Before writing any orchestration code:

┌─────────────────────────────────────────────────────────────────────┐
│  Pre-build checklist                                                │
├─────────────────────────────────────────────────────────────────────┤
│  What specific decisions or actions will the system make           │
│  autonomously?                                                      │
│                                                                     │
│  What are the failure consequences, and what does recovery         │
│  look like?                                                         │
│                                                                     │
│  What triggers each loop iteration?                                │
│                                                                     │
│  What tools can it call, and what real-world impact can            │
│  those calls have?                                                  │
│                                                                     │
│  Are actions auto-executed or human-approved?                      │
│                                                                     │
│  How is failure detected, logged, and recovered?                   │
│  What audit trail exists?                                           │
│                                                                     │
│  What is the blast radius if a single session goes wrong?          │
└─────────────────────────────────────────────────────────────────────┘

Vague answers mean prototype. Production agents are observable, auditable, and recoverable by design — not as an afterthought.

What’s Next

The definition, components, and autonomy model give you the mental framework. The harder questions are about the plumbing: how memory is designed, how tools are built to fail gracefully, how orchestration patterns differ and when to use each, how multi-agent systems coordinate without producing race conditions, and how observability is instrumented so you can debug a session after the fact.

That’s what Part 2 covers.

This is the setup that makes that possible.

CLAUDE.md — the project brain

CLAUDE.md is the most important file in your Claude Code setup. It is read automatically at session start, before any user message. It is your project’s coding constitution, onboarding doc, and persistent context combined.

Resolution order (lowest to highest priority):

1. Managed org policy (/etc/claude-code/CLAUDE.md on Linux, equivalent on macOS/Windows)
2. User global (~/.claude/CLAUDE.md)
3. Ancestor directories (walked upward from cwd)
4. Project root (./CLAUDE.md or ./.claude/CLAUDE.md)
5. Subdirectory files (loaded on-demand when Claude enters that directory)
6. Personal overrides (./CLAUDE.local.md — .gitignore this)

Use @path/to/file.md imports to break CLAUDE.md into modules. Place path-specific rules in .claude/rules/ with YAML frontmatter. Prefix any message with # to write to CLAUDE.md instantly. Run /init on an existing project to generate a starter file from your repo automatically. Keep each file under 200 lines for best adherence.

# Project: Payments API

## Tech Stack
- Runtime: Node.js 20 + TypeScript 5.4 (strict mode)
- Framework: Fastify 4 (NOT Express)
- ORM: Prisma 5 with PostgreSQL 16
- Testing: Vitest + Supertest

## Architecture Rules
- All business logic lives in `src/services/` — controllers are thin
- All monetary amounts stored in cents (integer), never floats
- Use `Result` pattern (neverthrow) — never throw from service layer
- Every public service method must have a corresponding unit test

## Key Commands
- `npm run dev`         — start dev server (port 3000)
- `npm run test`        — run full test suite
- `npm run db:migrate`  — apply pending Prisma migrations

## PR Conventions
- Branch: `feat/`, `fix/`, `chore/`, `refactor/`
- Commits: Conventional Commits format
- Always run `npm run test && npm run lint` before opening a PR

Memory architecture

Three distinct layers operate at different scopes:

Layer 1 — CLAUDE.md (explicit declarative memory). Human-written, version-controlled, always loaded. Ground truth for project knowledge.

Layer 2 — Auto-memory (~/.claude/projects/<hash>/memory/). Claude writes structured files when it learns something worth persisting across sessions. MEMORY.md is the index, truncated at 200 lines. Disable with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 for full manual control.

Layer 3 — In-session context window. Lost on /clear or exit. Use /compact to summarize and compress before it fills. Use /rewind to return to any earlier checkpoint in the session.

The /memory command opens both CLAUDE.md and the auto-memory folder, shows which files are currently loaded, and lets you toggle auto-memory on or off.

Context management and compaction

The context window fills up. How you handle that determines whether long sessions degrade or stay coherent.

/compact summarizes the current conversation into a compressed handoff and continues from there. It preserves the goal and key decisions while discarding low-signal exchange history. Use it proactively — before the window is full, not after Claude starts losing track.

/clear wipes the context entirely. Use it when starting a genuinely new task rather than letting unrelated history pollute the next one.

/context shows current token usage — input tokens consumed, percentage of window used, and estimated remaining capacity. Check it before starting a long agentic run so you know whether to compact first.

/rewind lets you time-travel to any previous checkpoint in the session. Every tool call creates an implicit checkpoint. If Claude takes a wrong turn, rewind to before it happened rather than trying to undo the effects manually.

For long-running projects, seed auto-memory with a handoff file: a structured memory/MEMORY.md containing a progress log, key decisions made, and a “Next Session: Start Here” section that gives the next session immediate orientation.

Permission system

Permissions control which tools Claude can invoke and when it needs to ask. Defined in .claude/settings.json:

{
  "permissions": {
    "allow": [
      "Read",
      "Glob",
      "Grep",
      "Bash(npm run *)",
      "Bash(git diff*)",
      "Bash(git log*)"
    ],
    "deny": [
      "Bash(rm *)",
      "Bash(git push*)",
      "WebFetch"
    ]
  }
}

allow and deny both support glob patterns. deny takes precedence over allow. Tool names match Claude’s internal tool names: Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch, Agent, etc.

The five permission modes (set with --permission-mode or /permissions):

• default — prompts for each tool call that isn’t pre-approved. Safe for interactive sessions.
• acceptEdits — auto-approves file edits, prompts for Bash and network calls.
• plan — read-only mode. Claude can inspect files and reason but cannot write, run commands, or take action. Use this to review a plan before authorizing execution.
• auto — intelligent classifier approves low-risk actions automatically, prompts on high-risk ones. The right default for most interactive sessions once you trust the project setup.
• bypassPermissions — skips all prompts. Only safe inside isolated containers. Never use on a machine with credentials, live databases, or a production environment.

For CI pipelines, combine --permission-mode plan with an explicit --allowedTools whitelist to give Claude exactly the access the job needs and nothing more.

Plan mode

Plan mode (--permission-mode plan or /plan) is read-only. Claude can read files, search the codebase, reason about the problem, and produce a detailed implementation plan — but it cannot write files, run commands, or take any action.

Use it at the start of any non-trivial task:

claude --permission-mode plan -p "design the migration from REST to GraphQL for the orders service"

Claude will read the relevant code, reason through the approach, identify risks, and produce a step-by-step plan. Review it, adjust it, then re-run without plan to execute. This pattern eliminates the most common source of wasted agentic work: Claude executing a plausible-but-wrong approach for ten steps before you realize the direction was off.

/plan inside an interactive session switches to plan mode for the current turn without changing the session’s permission mode permanently.

Checkpoints

Every tool call creates an implicit checkpoint — a snapshot of the conversation and file state at that point. Checkpoints are how /rewind works.

This matters for agentic runs. If Claude is making a sequence of file changes and takes a wrong turn at step 7, you do not need to manually undo each change. Use /rewind to select the checkpoint before the bad decision, adjust your instruction, and continue from there.

Checkpoints also matter when working with worktrees (--worktree). The worktree isolates Claude’s changes from your working tree entirely — changes live in a separate git branch. You review the diff and merge what you want. Combined with checkpoints, this gives you a full undo tree for agentic work.

For long agentic runs in CI, --max-turns creates a hard ceiling on how many tool calls Claude can make before stopping. Use it as a cost and safety valve: if a run hits the limit, it exits cleanly and you can inspect the partial output rather than watching an uncontrolled loop run up a bill.

Slash commands reference

The commands used most often in practice:

Context and memory

Configuration

Development workflow

Tasks and agents

Diagnostics

Custom commands go in .claude/commands/ as Markdown files. They appear in the / menu automatically.

Hooks — lifecycle automation

Hooks attach shell commands, Haiku-based decisions, or sub-agents to events in Claude’s execution lifecycle. They are the highest-leverage configuration most teams skip.

Available events:

• PreToolUse / PostToolUse / PostToolUseFailure
• SessionStart / InstructionsLoaded / UserPromptSubmit
• SubagentStart / SubagentStop
• TaskCreated / TaskCompleted
• PreCompact / PostCompact / PostSession

Defined in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": { "tool": "Bash" },
        "command": "echo '[AUDIT] ${CLAUDE_TOOL_INPUT}' >> ~/.claude/audit.log"
      },
      {
        "matcher": { "tool": "Bash" },
        "type": "prompt",
        "prompt": "Does this bash command look safe to run? Input: ${CLAUDE_TOOL_INPUT}. Reply ALLOW or BLOCK.",
        "on_block": "abort"
      }
    ],
    "PostToolUse": [
      {
        "matcher": { "tool": "Write" },
        "command": "cd \"$CLAUDE_PROJECT_DIR\" && npm run lint --silent || true"
      }
    ],
    "PostSession": [
      {
        "command": "cd \"$CLAUDE_PROJECT_DIR\" && npm test -- --reporter=dot 2>&1 | tail -5",
        "async": true
      }
    ]
  }
}

Hook types:

• command — run a shell command
• prompt — ask Haiku to make a decision (e.g. approve or block a tool call)
• agent — spawn a full sub-agent
• async: true — non-blocking; hook runs in background without holding up Claude

PreToolUse hooks fire even in --dangerously-skip-permissions mode. They are your last-resort guardrail regardless of permission settings.

Skills — reusable instruction sets

Skills are reusable, parameterizable instruction sets that can be invoked like slash commands. They live in .claude/skills/ (project-level) or ~/.claude/skills/ (global), each in their own folder with a SKILL.md.

.claude/skills/
├── pr-review/
│   └── SKILL.md
├── db-migration/
│   └── SKILL.md
└── api-contract/
    └── SKILL.md

A SKILL.md uses frontmatter to define the trigger, parameters, and description:

---
name: pr-review
description: Full PR review — correctness, tests, security, and style
trigger: /pr-review
params:
  - name: focus
    description: Optional area to focus on (security, performance, tests)
    required: false
---

Review the current git diff thoroughly.

{{#if focus}}Focus especially on: {{focus}}.{{/if}}

Check for:
- Logic errors and edge cases
- Missing or inadequate tests
- Security issues (injection, secrets exposure, insecure defaults)
- Consistency with patterns in CLAUDE.md
- Anything that would fail in production but pass in a test environment

Output a structured review with verdict (approve / request-changes), a summary, and a
bulleted list of specific issues with file paths and line numbers.

Invoke with /pr-review or /pr-review focus=security. Skills compose with hooks — a PostToolUse hook can automatically trigger a skill after every Write call if you want continuous review on file changes.

Sub-agents and parallel execution

Sub-agents are Claude instances with their own context windows, models, tools, personas, and isolation levels. Define them in .claude/agents/:

---
name: security-reviewer
description: Security audits, threat modeling, and OWASP analysis
model: claude-opus-4-6
color: red
isolation: worktree
memory: project
effort: high
background: false
---

You are an expert security engineer. Review code for OWASP Top 10, secrets exposure,
injection vulnerabilities, and insecure data handling. Be specific about file paths,
line numbers, and severity (critical / high / medium / low).

Claude auto-invokes sub-agents based on task matching against their description field. You can also invoke them explicitly: Use the security-reviewer agent to audit src/payments/.

Two built-in sub-agents are always available:

• Explore (Haiku, read-only) — fast codebase searches without touching your context window
• Plan (read-only) — planning and architecture reasoning without execution

Background agents — set background: true in the frontmatter. The agent runs in a separate session while you continue working. Manage background tasks with /tasks. Kill them with Ctrl+F.

Agent Teams coordinate multiple sub-agents across parallel sessions for tasks that exceed a single context window. A lead agent decomposes the work, delegates to teammates, and synthesizes results. Useful for large refactors, cross-service audits, or any task where the codebase is too large to reason about in one pass.

Plugins and MCP servers

MCP (Model Context Protocol) servers extend Claude Code with tools beyond the built-in set — databases, external APIs, internal services, custom retrieval systems. Configure them in .mcp.json at the project root:

{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-postgres"],
      "env": {
        "DATABASE_URL": "${DATABASE_URL}"
      }
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_TOKEN": "${GITHUB_TOKEN}"
      }
    },
    "internal-docs": {
      "command": "node",
      "args": ["./scripts/mcp-docs-server.js"]
    }
  }
}

Once configured, MCP tools appear alongside built-in tools in Claude’s toolset. Claude discovers what each server offers and uses them when relevant — no additional prompting required.

Commit .mcp.json. Env vars reference environment variables rather than hardcoded values, so secrets stay out of the file. For project-specific internal tools (a docs server, a deployment trigger, a staging environment API), a small MCP server is almost always the right abstraction.

Key CLI flags

# Non-interactive: run and exit (essential for CI)
claude -p "run the test suite and summarise failures" --output-format json

# Resume the most recent conversation
claude -c

# Start in plan mode — read-only, no writes until you approve
claude --permission-mode plan

# Isolated git worktree (changes don't affect your working tree)
claude --worktree

# Override model for a session
claude --model claude-opus-4-6

# Set reasoning effort (Opus 4.6 only)
claude --effort high

# Hard cap on spend for an agentic run
claude --max-budget-usd 2.00

# Limit agentic turns (safety valve for headless runs)
claude --max-turns 30

# Force structured JSON output against a schema
claude -p "..." --json-schema '{"type":"object","properties":{"verdict":{"type":"string"}}}' \
  --output-format json

# Enable real browser control via Playwright
claude --chrome

# Add context to the built-in prompt without replacing it
claude --append-system-prompt "You are working on the payments service. Be conservative with schema changes."

--append-system-prompt is preferred over --system-prompt. The latter replaces the entire built-in prompt, which strips important behavioral scaffolding that Claude Code relies on internally.

CI/CD and headless pipelines

- name: Claude review
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    claude -p "Review the diff for correctness, test coverage, and security issues.
               Output JSON: {verdict, summary, issues[]}" \
           --output-format json \
           --permission-mode plan \
           --max-turns 20 \
           --max-budget-usd 1.00 \
    | tee review.json

    jq -r '.summary' review.json
    jq -e '.verdict == "approve"' review.json

Use stream-json output format to process results line-by-line in long-running pipeline steps. Use --max-turns and --max-budget-usd together as a two-layer safety valve on every headless run.

The file structure to commit

your-project/
├── CLAUDE.md                  # shared project brain — commit this
├── CLAUDE.local.md            # personal overrides — .gitignore this
├── .mcp.json                  # MCP server config — commit this
└── .claude/
    ├── settings.json          # shared permissions and hooks — commit this
    ├── settings.local.json    # personal overrides — .gitignore this
    ├── agents/                # sub-agent definitions — commit these
    ├── commands/              # custom slash commands — commit these
    ├── skills/                # reusable instruction sets — commit these
    │   └── pr-review/
    │       └── SKILL.md
    ├── rules/                 # modular CLAUDE.md rules — commit these
    │   ├── api.md
    │   └── frontend/
    └── hooks/                 # optional shell scripts — commit these

Treat .claude/, CLAUDE.md, and .mcp.json like your CI configuration. They define how Claude behaves for everyone on the project. Review changes to them in PRs the same way you would review changes to .github/workflows/.

Run /doctor on any new machine to surface installation issues. Run /memory at the start of any new project to see what Claude has already loaded. Run /init if CLAUDE.md doesn’t exist yet.

Next step

If your team is evaluating how to integrate Claude Code into an engineering workflow — alongside existing CI/CD, code review, and operational tooling — book a discovery call to talk through the specifics.