Introduction

Eighty-eight percent of AI agents never reach production. The ones that fail aren't failing because the model isn't good enough — they're failing because the engineering around the model is wrong. Wrong retrieval strategy. Wrong context management. No bounds on the agent loop. Token costs compounding at the square of context length on tasks engineers assumed were linear. A second model that would catch the errors the first cannot see in itself — never added because nobody had a name for it.

The patterns exist. Engineers at different companies are independently discovering that routing works better with a classifier in front of it, that long-running agents need checkpointing, that parallel workers sharing a stable context prefix should cache it once. The techniques circulate as blog posts, conference talks, and GitHub repositories — each framed slightly differently, none connected to the others, none carrying the analysis that would let a practitioner know when to use one, why it solves what it solves, or what it costs.

In 1994, Gamma, Helm, Johnson, and Vlissides faced a structurally similar problem in object-oriented software. They did not invent Observer, Factory, Strategy, or Decorator — experienced engineers were already using them. What they did was name them precisely, describe the forces each one resolves, and give practitioners a shared vocabulary to reason with. A generation of engineers could say "use a Strategy here" and mean something exact. The vocabulary spread because it was useful, not because it was novel.

This catalog applies that method to AI engineering. It is a homage to the Gang of Four approach, not a claim to their authority. The patterns documented here are already in practice — the goal is to name them precisely enough to be useful.

The throughline throughout is simple: use the smallest sufficient pattern. Zero-shot before few-shot. Single agent before multi-agent. Retrieval only when it earns its context budget. The patterns here are not ranked by sophistication — a well-placed Zero-Shot is not a lesser engineering decision than a Tree of Thoughts. Each pattern is appropriate or inappropriate given the problem's actual requirements, the context budget available, and what the next simpler pattern fails to achieve.

This book is for engineers building LLM systems in production: architects choosing between retrieval patterns, engineers implementing agent loops that need bounds, teams debugging why a multi-agent system costs ten times what was projected. It assumes you can write code. It does not assume you have read the transformer papers — that material is in the Mechanisms chapter at the back, there when you need it.

The catalog covers seven categories. Signal patterns govern how instructions, personas, and examples are shaped before the model sees them. Knowledge patterns govern context engineering — what information and memory the model has access to during a task. Reasoning patterns govern how a model structures its thinking: chain-of-thought, planning, tool use, reflection, and verification. Orchestration patterns govern how agents are coordinated — chains, routers, parallel workers, and hierarchies. Reliability patterns govern safety, cost bounds, and production hardening. Integration patterns govern how agents reach tools and other agents. Humanizer patterns govern continuity and adaptive behaviour across sessions.

Each entry gives you: an Intent (one sentence); a Motivation (the concrete problem and why it recurs); Applicability (when to use it and when not to); Decision Criteria (measurements and thresholds that distinguish this pattern from its alternatives); a Participants table with explicit must-not constraints; an Implementation Sketch; and a Related Patterns section covering dependencies, conflicts, and upgrade paths.

How to read this book. These entries are reference material — use them when you are choosing between alternatives, implementing a pattern for the first time, or recognising a failure mode you have encountered. You do not need to read sequentially. Jump to the category you need; each category introduction covers the forces every pattern in that group resolves before you reach individual entries.

The Mechanisms chapter at the back derives twelve principles from how transformers actually compute — attention cost scaling, KV cache structure, prefix caching economics, subagent context bounding. It is there for when you want to understand why a pattern's costs are what they are, not just that they are. Mechanism citations in pattern entries (for example, mechanism 2 — n² compute cost) are cross-references to that chapter. You can use the catalog without opening it; it is a derivation of what the patterns already tell you.

No production system uses a single pattern in isolation. The Implementation Sketches throughout name which patterns compose naturally — which Reliability patterns wrap which Orchestration patterns, which Knowledge patterns feed which Reasoning patterns. The Appendix on Conflicts documents the tensions that require explicit design decisions: patterns that cannot run simultaneously, dependencies that are non-negotiable, and tradeoffs that cannot be resolved by convention.

The vocabulary this catalog establishes is a tool for thinking, not a checklist for compliance. Use it to communicate precisely with colleagues, to evaluate proposals against known forces, and to recognise when a new problem is in fact an old problem that has already been solved.

GO4 Taxonomy

A pattern language for AI engineering, structured analogously to the Gang of Four.

The Seven Categories

The original GoF had three categories: Creational, Structural, Behavioural. AI engineering patterns span more distinct concerns:

Signal patterns govern how instructions, personas, and examples are shaped before the model sees them — the prompt design surface.

Knowledge patterns govern context engineering: what information and memory the model has access to during a task, and how that context is assembled, retrieved, compressed, and persisted.

Reasoning patterns govern how a model structures its thinking: chain-of-thought, planning, tool use, reflection, search, and verification.

Orchestration patterns govern how multiple inferences and agents are coordinated — chains, routers, parallel workers, hierarchies, and multi-agent collectives.

Reliability patterns govern the cross-cutting concerns that production systems cannot omit: safety bounds, cost control, observability, evaluation, and recovery.

Integration patterns govern how agents reach the world outside their prompt: deterministic API calls, typed tool calling, standardised protocol servers, and inter-agent delegation.

Humanizer patterns govern the longitudinal layer: how agents develop continuity, self-knowledge, and adaptive behaviour across sessions.

Category I — Signal Patterns

A Signal pattern is a design pattern for shaping what you say to a language model — the instruction, the demonstrations, the role, the constraints, the output skeleton, the principles — so that the response distribution is shifted toward the task you actually want, before any retrieval, reasoning, or orchestration is layered on top.

Usage

Every interaction with a language model is, at minimum, a Signal-layer choice. The model is a conditional distribution over text; the prompt is the condition. Even "just type the question" is a Signal pattern (S1 Zero-Shot) — a default one, which is the point of naming it. Signal patterns make the prompt itself a deliberate design surface rather than an unexamined habit.

A language model arrives pre-trained with broad capability and no specific calibration to your task. The cheapest, lowest-latency, lowest-engineering-overhead way to calibrate it is at the prompt: showing it examples, telling it who to be, telling it what not to do, giving it the output skeleton, embedding the principles it should apply. None of these touch the weights; all of them shift the response distribution at inference time. The weights are fixed across API calls (mechanism 10) — what changes is which K-vectors the Q vectors attend to (mechanism 1), shaped by the prompt content occupying positions in the KV cache (mechanism 3). Signal patterns are the primary lever because they are the only layer between the fixed weights and the per-call attention computation. Apply a Signal pattern whenever:

• a task is well-enough defined that the lever you need is framing, not retrieval or reasoning;
• output format, tone, or value-alignment is inconsistent across runs;
• you are about to add a Knowledge, Reasoning, or Orchestration pattern and want to rule out "fix it with a better prompt" first;
• a downstream system depends on a stable shape of input or output that the model must produce.

Forces

Every Signal pattern resolves the same three forces in tension. A pattern is the right choice for a situation when it balances them in the way that situation demands.

1. The model has priors, not knowledge of your task. It has seen billions of tokens of generic text but nothing about your domain, your tone, or your forbidden behaviours. Left alone it answers from the mode of its training distribution, which is almost never the mode you want.

2. Tokens in the prompt are not free. Every example, every line of persona, every constraint, every template field costs context window, latency, and money on every call. The lever exists only because the cost is small relative to retraining — not because it is zero. Mechanically, this cost is O(n²): the attention matrix QK^T is computed over every pair of tokens in the prompt (mechanism 2), so adding tokens to a 1000-token prompt costs ten times more per token than adding the same tokens to a 100-token prompt. "Not free" understates the compounding — the correct framing is that prompt cost is superlinear in length.

3. Prompt-layer control is probabilistic, not enforced. A Signal pattern shifts a distribution; it does not guarantee an outcome. A persona can be broken, a constraint can be violated, a template can be ignored under adversarial or unusual input. Anything that must be guaranteed belongs at the Reliability layer, not the Signal layer.

A Signal pattern is, in each case, a disciplined answer to one question: how to spend the smallest number of prompt tokens to move the response distribution the largest distance toward the task you actually want.

Structure

All Signal patterns share one skeleton. They interpose a framing stage between the raw user task and the model, populating the system and user messages with material chosen to shift the response distribution:

  Raw task ────▶ Framing ────▶ Prompt ────▶ LLM ────▶ Response
 (what the      (instruction,    (system +
  user wants)    examples,        user
                 role,            messages,
                 constraints,     fully
                 template,        composed)
                 principles)

Patterns differ in what the framing stage adds — nothing (S1), demonstrations (S2), an identity (S3), a step list (S4), a prohibition list (S5), an output skeleton (S6), self-generated prompts (S8), a constitution (S9) — and in whether the addition is loaded once at session setup or assembled per call. The five bands below group the patterns by the addition they make: the baseline (I-A), demonstrations (I-B), setup-layer framing of identity / constraints / format / principles (I-C), instruction structure inside a single call (I-D), and meta-level prompt generation (I-E). They are largely orthogonal — a production prompt usually combines a setup-band pattern (S3 + S5 + S6 + S9) with an instruction-band pattern (S4) and possibly a demonstration-band pattern (S2), all sitting on top of the S1 baseline.

The loaded-once vs. per-call distinction is also a caching boundary (mechanism 5). Setup-layer patterns (S3, S5, S6, S9) placed in a stable system prompt define the cacheable prefix unit: if the prefix is identical across calls and exceeds the provider's minimum cacheable length (1024 tokens for Anthropic, TTL ~5 min, ~10% cost on cache hits), every subsequent call within the TTL reads the KV state from cache rather than recomputing it. Assembling S3 + S5 + S6 + S9 into a single stable prefix is therefore not just good composition — it is cache engineering. A dynamic S2 (retrieval-augmented few-shot) inserted into the prefix breaks this: it changes the prefix per call and forfeits the cache hit for all the setup-layer material that precedes it.

Examples

I-A — Baseline. The do-nothing default against which every other pattern is defined as an upgrade.

• S1 Zero-Shot — instruction only; no examples, no role, no template, no constraints.

I-B — Demonstration. Teaching the task by showing rather than telling.

• S2 Few-Shot — put k worked input$\to$output examples into the prompt so the model infers the task from demonstrations.

I-C — Setup framing. Loaded once at session setup; configures who, what-not, how, and why for every turn that follows.

• S3 Persona — assign the model an explicit identity (role, profession, character) framing knowledge and tone.
• S5 Constraint Framing — enumerate the specific things the model must not do as an explicit prohibition list.
• S6 Output Template — provide the skeleton of the expected output (fields, labels, structure) for the model to fill.
• S9 Constitutional Framing — embed explicit principles and have the model self-critique-and-revise against them before returning.

I-D — Instruction structure. Shaping the task description itself inside a single prompt.

• S4 Instruction Decomposition — break the complex instruction into explicit numbered sequential steps the model executes in order.

I-E — Meta. Producing Signal-layer artefacts with the model itself rather than by hand.

• S8 Meta-Prompt — use the LLM, driven by an evaluation signal, to generate or refine the prompts other Signal patterns assume a human wrote.

See also

• Category II — Knowledge patterns — Signal shapes what you say; Knowledge shapes what the model sees (retrieved or persisted information). A typical production prompt is a Signal frame around a Knowledge payload.
• Category III — Reasoning patterns — govern what the model does with the framed prompt; R1 Zero-Shot CoT and R2 Few-Shot CoT are the reasoning-band counterparts of S1 and S2, adding a "think step by step" instruction to the Signal-layer base.
• Category IV — Orchestration patterns — S4 Instruction Decomposition is the single-call sibling of O2 Prompt Chaining (multi-call ordered execution) and R3 Plan-and-Solve (plan-then-execute as two calls); choose by step length and inspection needs.
• Category V — Reliability patterns — S5 Constraint Framing is the in-prompt counterpart of V5 Guardrail Layering (external enforcement); S9 Constitutional Framing is the soft, in-prompt counterpart of V7 AgentSpec (hard, external policy enforcement). Anything that must be guaranteed belongs at the Reliability layer.
• Category VII — Humanizer patterns — H1 Identity Persistence subsumes S3 Persona in any system that has cross-session identity.

Former S7 Self-Consistency Voting was reclassified as R17 (Reasoning, band III-C) — its mechanism is sampling and aggregating reasoning paths, not shaping the prompt. Former S10 Chain of Density was folded into K6 Context Compression as a named Variant — it is a summarisation technique, not a Signal-layer choice. S7 and S10 are intentional gaps in the Signal numbering.

Quick Reference

S7 (Self-Consistency Voting) relocated to R17 (Reasoning). S10 (Chain of Density) folded into K6 (Context Compression). Both are intentional gaps.

S1 — Zero-Shot

Ask the model to do the task with nothing but the instruction itself — no examples, no decomposition, no template, no role, no constitution — and rely entirely on its pre-trained instruction-following. The baseline against which every other Signal pattern is defined as an upgrade.

Full entry: S1-Zero-Shot.md

S2 — Few-Shot

Put k worked input$\to$output examples into the prompt so the model infers the task — its format, style, and decision boundary — from the demonstrations rather than from instruction alone. Dynamic / Retrieval-Augmented Few-Shot is a variant.

Full entry: S2-Few-Shot.md

S3 — Persona

Assign the model an explicit identity — a role, profession, or character — at session setup, so its knowledge, tone, and decision style are framed by that identity for every turn that follows.

Full entry: S3-Persona.md — subsumed by H1 Identity Persistence in any system that has cross-session identity.

S4 — Instruction Decomposition

Break a complex instruction into explicit, numbered, sequential steps inside a single prompt, so the model executes them in order rather than collapsing a dense paragraph of requirements into a single best-effort pass. The cheapest rung of the ordered-execution ladder that climbs to O2 Prompt Chaining and R3 Plan-and-Solve.

Full entry: S4-Instruction-Decomposition.md

S5 — Constraint Framing

Enumerate, at session setup, the specific things the model must not do — as an explicit, auditable list that sits alongside the task description with equal or greater prominence than the positive instructions. The in-prompt prohibition layer; V5 Guardrail Layering is its external-enforcement counterpart.

Full entry: S5-Constraint-Framing.md

S6 — Output Template

Provide the skeleton of the expected output — fields, labels, and structure — for the model to complete, so format generation is replaced by format filling. JSON-mode / schema-constrained decoding, free-text template, and few-shot template are variants.

Full entry: S6-Output-Template.md

S8 — Meta-Prompt

Use the LLM itself to generate or refine the prompts it will run on, driven by an external evaluation signal, so prompt engineering becomes a measured optimisation loop rather than human guesswork. Requires an evaluator (typically V15 LLM-as-Judge or R17 Self-Consistency Voting) to provide the score.

Full entry: S8-Meta-Prompt.md

S9 — Constitutional Framing

Embed an explicit set of principles — a constitution — in the session setup, and have the model critique and revise its own output against those principles before returning it, so values and judgement live as inspectable text rather than as an implicit prior baked into weights. The inference-time form of Anthropic's Constitutional AI; the soft, in-prompt counterpart to V7 AgentSpec (hard, external enforcement).

Full entry: S9-Constitutional-Framing.md

Former S7 Self-Consistency Voting has moved to Category III — Reasoning as R17. Former S10 Chain of Density has been folded into K6 Context Compression as a named Variant. See TAXONOMY-DRAFT.md and the section-review notes for the reclassification rationale.

S1 — Zero-Shot

Ask the model to do the task with nothing but the instruction itself — no examples, no decomposition, no template, no role, no constitution — and rely entirely on its pre-trained instruction-following.

Also Known As: Direct Instruction, Vanilla Prompting, Instruction-Only Prompting, Naked Prompt.

Classification: Category I — Signal · the baseline pattern of the category — every other Signal pattern is defined as "S1 plus a specific addition" (examples, role, constraints, steps, template, samples, principles, density passes).

Intent

State the task and submit it. Nothing else. S1 is the floor against which every other Signal pattern is the upgrade — it names the do-nothing-extra default so that adding anything else becomes a conscious decision rather than an unexamined habit.

Motivation

Every prompt-engineering move costs something — tokens, latency, maintenance, brittleness — and earns its keep only against a clearly understood baseline. Without a named baseline, teams pile on persona, examples, constraints, templates, and chain-of-thought scaffolding from the first prompt onward, never measuring whether any single addition actually helped. Cost and complexity drift upward; the prompt becomes a museum of habits no one can defend.

S1 fixes the floor. It says: the task description alone, sent to a capable instruction-tuned model, is the baseline you must beat to justify anything more. Post-instruction-tuning models (Wei et al., 2022) handle a remarkable range of well-defined tasks at this floor. Instruction-tuned models follow zero-shot instructions reliably for in-distribution tasks because the instruction tokens shift the learned Q-K bilinear form (attention metric) toward completions the model has densely covered in training (mechanism 1). The failure mode — inconsistent format on out-of-distribution tasks — occurs because the Q-K inner products do not route confidently to a single completion cluster when the task is novel. Brown et al. (2020) introduced the term "zero-shot" precisely to distinguish no demonstrations from one-shot and few-shot; the result was that GPT-3 already solved many tasks at zero-shot, and that result has only strengthened with every subsequent model generation. For well-formed tasks the floor is often high enough that no upgrade is warranted.

The unique contribution of naming S1 is therefore not a technique — there is no clever trick — but a discipline. Every other Signal pattern decomposes into "S1 + a specific addition": S2 adds k example pairs; S3 adds an identity; S4 adds numbered steps; S5 adds prohibitions; S6 adds a template; S8 adds a meta-level prompt-search loop; S9 adds a constitution. Two patterns once listed as Signal — R17 Self-Consistency Voting (now in Reasoning, since voting over N samples is a thinking-shape choice, not a prompt-shaping move) and K6's Chain-of-Density variant (folded into K6 as a summarisation technique) — were relocated because they were not actually prompt-shaping. The category only makes sense if its baseline is named.

Applicability

Use Zero-Shot when:

• the task is well-defined and unambiguous to a competent reader without examples;
• the output format is common enough to sit inside the model's training distribution (summary, classification, translation, plain answer);
• iteration speed or unit cost dominates the design — every added token is paid on every call;
• you do not yet have measurements that justify any upgrade.

Do not use it when:

• the output format is non-standard and you cannot describe it cleanly in words $\to$ upgrade to S2 Few-Shot.
• domain expertise framing materially helps tone or knowledge activation $\to$ add S3 Persona.
• the task has a clear multi-step process the model keeps skipping $\to$ add S4 Instruction Decomposition.
• known failure modes need explicit prohibition $\to$ add S5 Constraint Framing.
• downstream code parses the output $\to$ add S6 Output Template (or a structured-output API).
• reasoning reliability is the constraint and a feedback signal exists $\to$ wrap with R17 Self-Consistency Voting or R7 Reflexion.
• regulated or safety-critical operation $\to$ add S9 Constitutional Framing.

Decision Criteria

S1 is right when a capable instruction-tuned model can do the task from the instruction alone, and nothing in the failure profile justifies the cost of an upgrade yet.

1. Task-novelty score. Is the task plausibly inside the model's pre-training distribution? Summarisation, simple classification, translation, factual Q&A, common formats (markdown, JSON, plain prose) — yes, S1. Bespoke domain output, esoteric format, proprietary tone — no, escalate. Threshold: if a competent human reader could do the task from the instruction without examples, the model probably can too.

2. Format-consistency rate. Run the prompt N=20 times. What fraction returns the expected shape? If $\geq$ 95%, S1 holds. 90–95% is borderline — measure the cost of failures before upgrading. < 90% $\to$ escalate to S6 Output Template (or a structured-output API), or S2 Few-Shot if the failure is stylistic rather than structural.

3. Quality-against-upgrade delta. Compare S1 quality against S2 (few-shot) on the same task. If the lift from 3–5 examples is < 5 percentage points on whatever quality metric you care about, S1 wins on cost. If it's > 10 points, S2 wins. The middle band is a judgement call about token budget.

4. Cost / latency budget. Tokens added by an upgrade are paid on every call. At scale, a 200-token persona $\times$ 1M calls/month is not free. Mechanically, every token added to the prompt participates in O(n²) pairwise attention computations and adds ~300KB to the KV cache (mechanism 2, 3). At scale a 200-token addition is not 200 tokens of linear cost — it expands the attention matrix over the full prompt length. S1 minimises this. If unit economics are tight, S1 is the right floor and upgrades must clear a measurable bar.

5. Reliability budget. Is this safety-critical, regulated, or load-bearing for downstream automation? If yes, S1 is almost never the final answer — pair with S5 Constraint Framing, S9 Constitutional Framing, or V9 Bounded Execution as needed. S1 is for the long tail of well-defined, low-stakes calls.

Quick test — S1 is the right pattern when:

• the task sits inside the model's training distribution, and
• format-consistency on a 20-run probe is $\geq$ 95%, and
• the lift from few-shot is small enough that the token cost does not pay back, and
• the task is not safety-critical.

If format slips, choose S6 Output Template (or a structured-output API). If style or tone slips, choose S2 Few-Shot. If reasoning quality is the bottleneck, choose R4 ReAct or R17 Self-Consistency Voting. If safety matters, layer S5 Constraint Framing or S9 Constitutional Framing on top. S1 alone is the default; upgrades are deliberate.

Structure

  Task description
        │
        ▼
  ┌─────────────────┐
  │   LLM (single   │      no examples
  │   configured    │      no decomposition
  │    session)     │      no template
  └────────┬────────┘      no role required
           │
           ▼
        Output

A single configured session. One call. Nothing on either side of the model except the instruction in and the output out.

Participants

Three participants — the minimum any prompted system can have. The discipline of S1 is that the list does not grow.

The whole point of the page is the Must not column. S1's failure mode is not technical; it is the slow accretion of unexamined additions until the prompt is no longer S1 and no one remembers when it changed.

Collaborations

The Caller composes the Task Instruction — one sentence to a short paragraph naming what the model should produce. It submits the instruction to the Model as a single call. The Model returns a completion. The Caller passes the completion to whatever consumes it. There is no second call, no evaluation step, no retry on bad parse — those moves all belong to other patterns (R17 voting, V15 judging, S6 templating, S4 decomposing). The simplicity of the collaboration is the pattern.

Consequences

Benefits

• Lowest token cost of any prompting pattern — only the instruction and the input ride in context.
• Lowest latency — one call, no scaffolding, no aggregation.
• Easiest to maintain — fewer moving parts; no example curation, no template drift, no constitution to keep current.
• Highest portability across models — no model-specific tricks baked in; a model swap is a single regression test.
• The honest baseline — every upgrade can be measured against this floor.

Costs

• No format guarantee — output structure depends entirely on the model's defaults; token generation is stochastic, so the same prompt produces different shapes across runs (mechanism 7).
• No style guarantee — tone and register drift with model and decoding parameters.
• No reasoning scaffold — complex multi-step tasks degrade because the model produces them in one pass.
• No safety scaffold — nothing constrains adversarial or off-policy completions.

Risks and failure modes

• Silent format drift — outputs parse most of the time and break occasionally; the failure surfaces in downstream code rather than at the prompt.
• Capability degradation under model swap — what worked on a strong model may fail on a smaller or quantised one; S1 has no scaffolding to soak up the difference.
• Accretion creep — the prompt slowly grows persona, examples, constraints, templates, until it is no longer S1 but is still treated as the baseline. The team loses the actual baseline.
• Misclassification as S1 — any prompt with examples is S2, with role is S3, with steps is S4, with template is S6. Calling those "zero-shot" because there is "only one prompt" is the most common audit failure.

Implementation Notes

• Keep it one sentence to a short paragraph. If the instruction needs more than that, the task probably needs decomposition (S4) or a template (S6).
• Measure first, upgrade second. Run the format-consistency probe (criterion 2 above) before adding anything. Most failures are diagnosable from 20 runs.
• Set the model and decoding once, document them. Temperature, top-p, and model choice are part of the baseline — changing them silently destroys the comparison.
• Use structured-output APIs in preference to S6 when format is the issue. If JSON mode or schema-constrained decoding is available, that beats both S1 and S6 free-text templating.
• Do not chain S1 with itself. Multi-call workflows belong to the Orchestration category (O6 Orchestrator-Workers and friends), not to S1.
• Treat S1 as the start of the upgrade ladder, not the destination. When you find yourself adding "just one example" or "just a role," you have left S1 — name the new pattern and own the upgrade.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S1 is, by definition, the pattern with no composition — one configured LLM session, called once. It is the inner step many other patterns wrap: R17 wraps it with N samples and a vote; R7 Reflexion wraps it with a retry-with-memory loop; every O-category orchestration pattern uses one or more S1 calls as worker steps. S1 itself composes with nothing inside its own boundary.

The chain:

Skeleton — wiring only; the # LLM line is a configured session, not bare code:

zero_shot(task_description, input_data):
    prompt = format(task_description, input_data)   # code
    completion = Model(prompt)                       # LLM — single configured session
    return completion                                # code

The LLM sessions:

Specialist-model note. None — a capable instruction-tuned generalist is the entire requirement. The pattern artifact that does the heavy lifting is the instruction itself: a clear, complete, unambiguous task statement. The model is generic; the discipline is in the writing of the task line.

Open-Source Implementations

S1 is the degenerate case of prompting — there is no library to install and no canonical project, because the pattern is "call the model with the instruction." The relevant references are documentation, guides, and the original paper:

• Anthropic — Prompt engineering overview — docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview — Claude prompting docs; zero-shot is the implicit baseline before "multishot prompting" and the rest.
• OpenAI — Prompt engineering guide — platform.openai.com/docs/guides/prompt-engineering — the OpenAI platform guide; zero-shot is the default mode before the techniques sections.
• DAIR.AI Prompt Engineering Guide — github.com/dair-ai/Prompt-Engineering-Guide — the community-maintained guide; the zero-shot page is the canonical written explanation of the pattern.
• NirDiamant — Prompt Engineering notebooks — github.com/NirDiamant/Prompt_Engineering — runnable notebooks; the zero-shot-prompting.ipynb is a minimal, working illustration.

These are documentation references, not implementations — exactly as expected for a baseline pattern.

Known Uses

• Every LLM application in production uses S1 somewhere — it is the underlying call inside every wrapper. Most ChatGPT, Claude.ai, and Gemini user turns are zero-shot from the user's side.
• Classification and summarisation pipelines that escaped fine-tuning between 2022 and 2024 — many enterprise teams replaced labelled-data fine-tuning with S1 against a frontier model.
• First drafts of any prompted feature — the standard engineering practice is to ship S1, measure, and upgrade only when measurements demand it.
• Eval baselines in benchmark reports — model evaluations (MMLU, HumanEval, GPQA) report zero-shot scores as the default; few-shot scores are reported as upgrades against that baseline.

Related Patterns

• Baseline for every other Signal pattern — S2, S3, S4, S5, S6, S8, S9 are each "S1 plus a specific addition" (examples, role, steps, prohibitions, template, meta-prompt loop, constitution). S7 and S10 used to belong here but have moved (to R17 and K6 respectively).
• Wrapped by R17 Self-Consistency Voting — R17 calls S1 N times and votes; the inner call is exactly S1.
• Wrapped by R7 Reflexion — R7 retries an S1 call with a memory of prior failures; the per-attempt call is S1.
• Used by every O-category orchestration pattern — O6 Orchestrator-Workers and the others compose multiple S1 calls; the worker step is typically S1.
• Distinct from S2 Few-Shot — the presence of even one demonstration moves the pattern to S2. Calling a one-shot prompt "zero-shot" is the most common misclassification.
• Distinct from S4 Instruction Decomposition — numbered steps inside one prompt are S4, not S1. The line is the explicit ordering: a paragraph of requirements is S1; a numbered list of steps the model must follow is S4.
• Note on fundamentality — S1 is the degenerate case of prompting and earns its number as the baseline against which every other Signal pattern is measured, the same role K1 Vanilla RAG plays for Knowledge. Removing it would leave the rest of the category without a defined floor.

Sources

• Brown et al. (2020) — "Language Models are Few-Shot Learners" (GPT-3 paper, arXiv 2005.14165). Introduced the zero-shot / one-shot / few-shot distinction.
• Wei et al. (2022) — "Finetuned Language Models Are Zero-Shot Learners" (FLAN, arXiv 2109.01652). Established instruction tuning as the mechanism that makes zero-shot work.
• Anthropic — Prompt engineering documentation (Claude API docs).
• OpenAI — Prompt engineering guide (platform docs).
• DAIR.AI — Prompt Engineering Guide, zero-shot section.
• White et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" — establishes the baseline / refinement framing the GO4 Signal category formalises.

S2 — Few-Shot

Put k worked input$\to$output examples into the prompt so the model infers the task — its format, style, and decision boundary — from the demonstrations rather than from instruction alone.

Also Known As: In-Context Learning, Exemplar Prompting, k-Shot Prompting, One-Shot (when k = 1). (Dynamic / Retrieval-Augmented Few-Shot is a variant — see Variants.)

Classification: Category I — Signal · the canonical upgrade from S1 Zero-Shot · a setup pattern — the work is in choosing and arranging examples once, not in any per-call logic.

Intent

Demonstrate the task with examples in the prompt, so the model learns the desired format and behaviour from the demonstrations themselves rather than from a description of them.

Motivation

S1 Zero-Shot asks the model to perform a task from instruction alone. For well-defined tasks in formats the model has seen during pre-training (summarise this, translate that, return JSON with these fields), instruction is enough. For anything else — a non-standard output shape, a specific tone, an idiosyncratic classification scheme, a domain-specific reasoning style — instruction in isolation produces inconsistent results, because describing a format is harder than showing it.

Brown et al. (2020) showed that large language models can pick up a task from a handful of examples in their context window, without any weight update. This is in-context learning: the model uses the demonstrations as a kind of runtime "training set" that shapes its next-token distribution. The mechanism is fundamentally different from S1 — instead of relying on the instruction-following circuit, it relies on the model's ability to extrapolate the pattern implicit in the examples. Subsequent work (Min et al., 2022) showed that what does the heavy lifting is the format and distribution of demonstrations — the label space, the input space, the structure of the input$\to$output map — more than the literal correctness of the labels in the examples.

That insight is the pattern's defining force: the examples are the specification. Whatever the examples consistently demonstrate is what the model will produce. This makes example selection — not example count — the pattern's main design lever. A handful of carefully chosen, distribution-covering demonstrations beats a dozen homogeneous ones, and a single misleading example can bias the entire output stream. The pattern is cheap in calls (zero extra LLM calls per request beyond the base generation) and expensive in tokens (every demonstration rides on every call), so the design problem is which examples to include and in what order — not whether to include any.

Variants

The variants differ in how examples are chosen and assembled:

• Static k-shot. A fixed set of 2–8 examples baked into the prompt, the same for every call. Cheapest to maintain; the standard form; what most production systems use. Cache-friendly: the prefix is constant.
• One-shot. k = 1. A single demonstration; the lowest-cost upgrade from S1. Often enough when only the format matters (the model already knows the task; it just needs the shape). Brown et al. (2020) treated this as a distinct regime worth measuring.
• Structured few-shot. Examples are wrapped in explicit delimiters and labelled fields (<example>…</example>, Input: … Output: …), removing ambiguity about where each example begins and ends. Reduces "example bleed" — the model treating an example field as part of the current query.
• Dynamic / Retrieval-Augmented Few-Shot. Examples are selected per query from a pool, typically by similarity to the current input (Liu et al. 2022, "KATE"). Higher quality on diverse query streams; loses prefix caching; adds an embedding lookup step. This composes with K1 Vanilla RAG — the retriever fetches example demonstrations rather than knowledge chunks. (Note: this remains S2 because the in-prompt structure and in-context-learning mechanism are unchanged; only example selection moves to runtime.)

All four are the same pattern — examples in the prompt drive in-context learning — differing in whether the example set is fixed or selected, and how rigidly it is structured.

Applicability

Use Few-Shot when:

• the output format is non-standard or uncommon, and S1 produces inconsistent shapes;
• the task involves a specific style, tone, or reasoning pattern the model would not produce by default;
• a small set of representative examples covers the input distribution;
• you can spend the token budget on demonstrations on every call.

Do not use when:

• a one-line instruction reliably produces the right shape — use S1 Zero-Shot;
• the output is highly structured (JSON, function-call args) and the runtime offers a structured-output mode — use S6 Output Template or the API's structured mode;
• the task is multi-step with intermediate gating — use S4 Instruction Decomposition or O2 Prompt Chaining;
• you have hundreds of labelled examples and the task is stable — fine-tuning will beat any in-prompt arrangement on cost per call.

Decision Criteria

S2 is right when the task is hard to describe but easy to demonstrate, and the token cost of carrying examples is acceptable on every call.

1. Measure S1's failure mode. Run S1 on a labelled test set:

• Format-consistency rate — what % of outputs match the required shape exactly? Below ~90%, S2 will help.
• Style match — does a human rater accept the tone? Below acceptance, S2 with style-bearing examples helps directly.

If both are already high, S2 buys nothing. Stay with S1.

2. Pick k. Empirically, 3–5 examples capture most of the benefit; returns diminish past 8; very long contexts can tolerate "many-shot" (dozens to hundreds) but the marginal gain per example is small. Start at k = 3 and add only when measurement shows a remaining gap.

3. Choose static vs dynamic selection. If the query distribution is narrow, a fixed k-shot prefix is simpler and cache-friendly. If the query distribution is wide and a single fixed set cannot cover it, switch to the Dynamic / Retrieval-Augmented Few-Shot variant — accept the loss of prefix caching in exchange for per-query example fit.

4. Budget the tokens. Cost per call $\approx$ k $\times$ example_length + base_prompt. If examples push the prompt past the model's caching threshold or the latency budget, reduce k, compress examples, or fine-tune instead (mechanism 5).

5. Audit the example set. Examples must (a) span the input distribution, including hard cases, not just easy ones; (b) be internally consistent — no two examples contradict on shape or labelling; (c) be balanced across classes for classification; (d) be ordered so the last example is not an outlier (recency bias is real). A mis-chosen example set is worse than no examples — it teaches the wrong pattern.

Quick test — S2 is the right pattern when:

• S1 produces inconsistent format or style on the target task, and
• 2–8 representative examples can cover the input distribution, and
• the per-call token cost of carrying those examples is affordable, and
• the task is not better served by a structured-output API (S6) or fine-tuning.

If S1 already produces the right shape, stay with S1. If the runtime supports structured output and the issue is only format, prefer S6 Output Template with the structured-output mode. If the task has hundreds of labelled examples and is stable, fine-tune. If a single fixed example set cannot cover the queries, switch to the Dynamic variant.

Structure

  ┌── prompt assembled once (static) or per-query (dynamic) ──┐
  │                                                            │
  │  [optional system / persona]                               │
  │  [optional instruction]                                    │
  │                                                            │
  │  Example 1:  Input → Output                                │
  │  Example 2:  Input → Output                                │
  │     …                                                      │
  │  Example k:  Input → Output                                │
  │                                                            │
  │  Query:      Input → ?                                     │
  └────────────────────────────────────────────────────────────┘
                            │
                            ▼
                     Model generation
                            │
                            ▼
                          Output

  Static k-shot:   example block is constant across calls.
  Dynamic k-shot:  Selector retrieves k examples per query
                   from a pool, then assembles the prompt.

Participants

The pattern's quality is dominated by the Example pool and the Selector. The Model does the work the demonstrations imply; the Prompt assembler is mechanical; the Evaluator is what catches a bad example set before it ships.

Collaborations

A query arrives. In the static case, the Prompt assembler concatenates a fixed example block with the query and ships it; the Model completes against the demonstrated pattern. In the dynamic case, the Selector first queries the Example pool — typically by embedding similarity — to fetch the top-k most relevant demonstrations, then the Prompt assembler composes the per-query prompt. Either way, the Model never sees the Selector or pool directly; it sees only the final assembled prompt and learns the task from its structure. Offline, the Evaluator runs the static or dynamic configuration against a held-out labelled set and decides whether to keep the chosen examples, swap them, or change k.

Consequences

Benefits

• Strong format and style control with no fine-tuning and no extra LLM calls per query.
• Works across models and providers; portable.
• The example set is a human-readable, version-controllable artefact — easier to audit than a fine-tune.
• A small number of examples (3–5) typically captures most of the achievable gain.

Costs

• Every demonstration consumes context tokens on every call — the cost scales linearly with k and example length, but each token also participates in O(n²) pairwise attention over the full prompt (mechanism 2).
• Designing and vetting the example pool is real work, even though no model training is involved.
• Dynamic selection adds an embedding-lookup step per query and breaks prefix caching.

Risks and failure modes

• Bad pool — examples that contradict, skew toward easy cases, or imbalance the label distribution will teach the wrong pattern; the model dutifully extrapolates the bias.
• Recency bias — the last example exerts disproportionate influence; an outlier at position k pulls the model toward it.
• Example bleed — without clear delimiters, the model can treat the live query as another example to imitate, or carry over irrelevant fragments of the previous example into its output.
• Cache loss (dynamic variant) — selecting examples per query means a different prefix every call, defeating prompt caching's economics on high-volume systems. Cache cascade destruction (mechanism 5). Dynamic example selection changes the token sequence of the few-shot block on every call. This does not only forfeit the prefix cache for the few-shot examples themselves — it invalidates the entire prefix that precedes them (system prompt, persona, constraint framing, output template) because the cache key is the exact byte sequence up to the cache boundary. If the dynamic examples are inserted after a 2,000-token stable prefix, dynamic selection causes 2,000 tokens of prefix to be re-prefilled on every call at full cost (~10$\times$ the cache-hit price per token). The economic cost of the dynamic variant is therefore the marginal cost of retrieval plus the full prefill cost of the stable prefix — not just the retrieval overhead. Budget this explicitly. The mitigation: place dynamic examples at the end of the context (after all stable content), so the static prefix can still be cached even if the examples change.
• Drift unmeasured — the example set is set once and never re-evaluated; as the input distribution shifts, the set silently goes out of date.

Implementation Notes

• Start at k = 3. Add examples only when held-out measurement shows a remaining gap. Diminishing returns are sharp after 5–8.
• Diversity beats volume. Five examples covering five distinct sub-cases beat ten examples of the same shape.
• Order matters — put the most representative example last (recency bias works in your favour if you place it deliberately). The geometric basis of recency bias (mechanism 12). RoPE relative positional encoding makes the attention score between query position $i$ and key position $j$ a function of their relative distance: $s_{ij} = Q_i^T R((j-i)\theta) K_j$. The last example immediately before the query has the smallest offset $|j - i|$ and therefore the least-rotated (strongest) inner product. Placing the most representative example last is not a heuristic — it is exploiting a derivable geometric property of the position encoding. The practical consequence: in a 5-shot setup, the ordering of examples matters more than is commonly recognized, and the difference between placing the best example first vs. last can be measurable in output quality.
• For classification, balance examples across classes; an imbalanced set is read by the model as a prior.
• Use unambiguous delimiters between examples and between the example block and the live query (<example>…</example>, ### Example, or Input: / Output: pairs).
• The label correctness of the examples matters less than their format and distribution (Min et al. 2022) — but do not exploit this; correct labels still help and incorrect ones invite drift on adjacent tasks.
• If using the dynamic variant, retrieve by task similarity (does this example demonstrate the same sub-pattern?), not pure semantic similarity to the query — the latter retrieves near-duplicates that teach the model to copy rather than generalise.
• Compose with S6 Output Template when the demonstrated format is structured — the examples show what the fields contain; the template shows what fields exist. Together they are tighter than either alone.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring. Note: S2 is mostly about setup — choosing and arranging the example set — not per-call work. Static S2 adds zero LLM calls beyond the base generation; dynamic S2 adds one cheap retrieval step (typically not an LLM).

Composition: S2 sits inside the Setup slot of any LLM session (S1, S3, S6, K1's generator, K5's gates and evaluators, R-category reasoners). The examples become part of the session's setup string. The dynamic variant composes with K1 Vanilla RAG — the retriever fetches examples, not knowledge chunks — and shares the Selector role with that pattern.

The chain — static k-shot (per request):

The chain — dynamic k-shot (per request):

The chain — offline (one-time setup, then on a cadence):

Skeleton:

# Static k-shot — setup-once
EXAMPLES = load_curated_examples(pool, k=4)           # code, one-time
PROMPT_PREFIX = render(EXAMPLES, delimiters)          # code, one-time

answer(query):
    prompt = PROMPT_PREFIX + render_query(query)      # code
    return generate(prompt)                            # LLM — base session

# Dynamic k-shot — per-call selection
answer_dynamic(query, pool):
    q_emb   = embed(query)                             # code (tiny model)
    chosen  = pool.top_k_by_similarity(q_emb, k=4)     # code — Selector
    prompt  = render(chosen, delimiters) + render_query(query)  # code
    return generate(prompt)                            # LLM — base session

The LLM sessions. S2 itself does not own an LLM session — it provides example content that lives in the Setup of whichever session is doing the real work. The table below records this honestly.

Specialist-model note. None — a capable generalist suffices. The pattern's quality lives in the example pool, not in any specialist model. Two specialist dependencies may appear at the edges: (a) an embedding model in the dynamic variant for similarity-based selection, and (b) optionally an LLM-as-Judge (V15) for offline evaluation of the chosen example set. Neither is required for the core pattern; the artefact that does the heavy lifting is the curated example block itself.

Open-Source Implementations

Few-Shot is a primitive of every LLM framework — there is no single canonical project to point to, but the projects below are the standard references for managing few-shot examples (selection, storage, optimisation) rather than just stuffing them into a string.

• DSPy — github.com/stanfordnlp/dspy — Stanford's framework for programming (not prompting) LLMs; its LabeledFewShot, BootstrapFewShot, and KNNFewShot optimisers are the de facto open-source toolkit for static, bootstrapped, and dynamic example selection.
• PromptSource — github.com/bigscience-workshop/promptsource — BigScience's templating toolkit and shared repository of 2,000+ prompts across ~170 datasets; the canonical artefact for curating few-shot example sets at scale.
• LangChain FewShotPromptTemplate and ExampleSelector — github.com/langchain-ai/langchain — production-style abstractions: a few-shot template plus pluggable selectors (length-based, semantic-similarity, MMR) for the dynamic variant.
• Provider cookbooks — Anthropic Prompt Engineering — Multishot Prompting and the OpenAI Cookbook examples (github.com/openai/openai-cookbook) — the practitioner references for how a frontier-lab vendor recommends structuring few-shot prompts on its own models.

Known Uses

• Production classifiers and extractors built on commercial APIs almost universally use 3–8 in-prompt examples to lock format and label vocabulary.
• Anthropic, OpenAI, and Google prompt-engineering guides all recommend multi-shot prompting as the first upgrade from zero-shot — the pattern is the documented default for non-trivial format tasks across all three.
• DSPy programs in deployed systems lean on BootstrapFewShot to compile high-quality example sets from a training signal, then ship the compiled few-shot prompt.
• Coding assistants (Cursor, Claude Code, Copilot) use few-shot examples of code style and convention — sometimes static, sometimes dynamically retrieved from the user's repo — to align generated code with the local codebase.
• The dynamic variant is the standard implementation for support-bot intent classification and for code-completion systems that retrieve similar snippets from the local project as in-context demonstrations.

Related Patterns

• Refines S1 Zero-Shot — Few-Shot is the canonical upgrade from S1 when instruction alone underspecifies the task. S1 is the default; S2 is the first thing to try when S1's output is inconsistent.
• Pairs with S3 Persona — persona sets who is answering; examples set how the answer looks. They compose cleanly.
• Pairs with S6 Output Template — the template defines the field skeleton; the examples show realistic content within it. Tighter together than either alone.
• Composes with R17 Self-Consistency Voting — S2 controls the format; R17 improves the answer's reliability through sampling and majority vote. Orthogonal: S2 sets what to produce; R17 votes over N attempts at producing it. Where they touch: R17 may show different answers across samples even when the format is locked by S2 — exactly the point.
• Competes with S6 Output Template (structured-output mode) — when the runtime offers a structured-output API, that API beats S2's format-by-demonstration on cost and reliability. Use S2 only for format aspects the API cannot express (style, tone, reasoning shape).
• Composes with K1 Vanilla RAG (in the Dynamic / Retrieval-Augmented Few-Shot variant) — the same retrieval mechanism, fetching example demonstrations instead of knowledge chunks.
• Distinct from fine-tuning — fine-tuning updates weights; S2 updates the prompt. Fine-tuning wins on per-call cost when example sets get large and stable; S2 wins on iteration speed and portability.

Sources

• Brown et al. (2020) — Language Models are Few-Shot Learners (arXiv 2005.14165). The GPT-3 paper that established in-context learning as the foundational mechanism.
• Min et al. (2022) — Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (arXiv 2202.12837). Shows that format and distribution, not label correctness, drive few-shot performance.
• Liu et al. (2022) — What Makes Good In-Context Examples for GPT-3? (arXiv 2101.06804). The "KATE" method — retrieval-based selection of in-context examples; the basis for the Dynamic variant.
• Bach et al. (2022) — PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (arXiv 2202.01279).
• White et al. (2023) — A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. The PLoP catalog; few-shot is in the Input Semantics category.
• Anthropic and OpenAI prompt-engineering guides — current vendor-side practitioner references for multi-shot prompting.

S3 — Persona

Assign the model an explicit identity — a role, profession, or character — at session setup, so its knowledge, tone, and decision style are framed by that identity for every turn that follows.

Also Known As: Role Prompting, Expert Identity, Character Prompting, the Persona Pattern (White et al.).

Classification: Category I — Signal · the setup-layer pattern that names who the model is; complements S5 Constraint Framing (what it must not do), S6 Output Template (what its output looks like), and S9 Constitutional Framing (which principles it applies). Subsumed by H1 Identity Persistence in any system that has cross-session identity.

Intent

Frame the model's response distribution at the identity level — selecting a domain, a register, and a decision style in one move — so every subsequent turn inherits that framing without restating it.

Motivation

A language model is, at any moment, a distribution over many plausible respondents (mechanism 7). The same query — "how should I handle this dependency conflict?" — pulls a different answer from a senior security engineer than from a friendly tutor than from an opinionated open-source maintainer. With no identity given, the model averages across these voices: the answer is correct on the surface, generic underneath, and tonally inconsistent across turns. The naive fix — restating tone and expectations in every user message — is verbose, fragile (one omission and the voice drifts), and treats identity as a per-turn concern when it is properly a session-level one.

S3 puts identity where it belongs: at the setup of the session, loaded once, before the first turn. "You are a senior security engineer reviewing a pull request." That sentence simultaneously narrows the response distribution (toward the engineering register), activates the associated knowledge cluster (security idioms, threat-model vocabulary, common review-comment forms), and stabilises the voice across the session (later turns inherit the framing without re-stating it). The empirical effect is asymmetric: the right persona materially improves domain-specific outputs; the wrong persona (a marketing assistant asked a vulnerability question) actively degrades them by activating the wrong cluster. Mechanically, the role label shifts the Q-K bilinear form (mechanism 1): each attention head applies a distinct learned asymmetric metric on token-embedding space. The role-label token has learned K-projections that route attention toward domain-specific K-vectors in subsequent layers. An abstract label has no such dense learned cluster — which is why the role label itself, not an elaborate backstory, carries the lift. Extra narrative tokens add O(n²) attention cost (mechanism 2) without meaningfully shifting the Q-K routing.

S3 is the most basic Signal-layer setup choice and the one every other pattern's "setup loaded once" line implicitly invokes. When K5's Generator session names "role (S3)" in its setup, it is naming this pattern; when K12's Curator names a role, same. S3 has its own forces — right identity activates the right knowledge; wrong identity creates false expertise that sounds authoritative — and they are distinct from S9 (principles) and S5 (prohibitions). It earns its own number.

Variants

S3 has two members that differ in how many identities the system maintains, not in the mechanism:

• Single-Role Persona. One persona per session, set once at setup, inherited by every turn. The default. White et al.'s original "Persona Pattern."
• Role-Per-Agent (multi-agent). In an O4/O6 system, each sub-agent runs a different S3 persona — a Planner, a Critic, a Coder, a Reviewer. Personas are chosen to be distinct and unambiguous so the agents' contributions do not collapse into a single voice. This is S3 used as a differentiator across agents rather than a framing for one.

Both are the same pattern (assign an identity at session setup); they differ only in cardinality. Multi-agent role-per-agent does not become its own pattern because the mechanism is identical to single-role — the multi-agent structure belongs to Category IV, not to S3.

Applicability

Use when:

• the task benefits from a domain register — security, medicine, law, finance, engineering — where the right vocabulary and the right caution profile materially change the answer;
• the session is long enough that voice consistency across turns matters;
• a multi-agent system needs distinct, recognisable contributors (Planner / Critic / Coder);
• the task implies a style the model would not produce by default (terse Unix maintainer; patient first-grade teacher; formal legal counsel).

Do not use when:

• the system has cross-session identity — use H1 Identity Persistence instead; H1 subsumes S3 and adds session-spanning state, accumulated commitments, and an updatable self-model. Running both is redundant and creates two sources of identity truth.
• the persona would imply authority the model does not have ("As your doctor, I prescribe...") — that is the false-expertise failure mode; either drop the persona or pair with S5 Constraint Framing to disclaim the implied authority.
• the task is a flat one-shot operation (a single classification, a single extraction); the persona's setup cost is not amortised over enough turns to matter — use S1 Zero-Shot plus S6 Output Template.
• principles, not identity, are what you need — use S9 Constitutional Framing (an analyst with the wrong constitution is more dangerous than a persona-less model with the right one).

Decision Criteria

S3 is right when domain register or voice consistency materially changes output quality, and the session has enough turns to amortise the setup.

1. Domain-register lift. On 20 representative queries, compare zero-shot output to output with a domain-specific persona prepended. If the persona version is noticeably better on vocabulary, caution, and structure, S3 has a real effect. If outputs are indistinguishable, persona is decoration — drop it. Threshold: > 20% of outputs improved on a blinded comparison.

2. Voice-consistency need. Over a 10-turn session, does the assistant drift in register or tone without a persona? If yes, S3 stabilises voice. If the task is short or stateless, skip. Threshold: session length $\geq$ ~5 turns.

3. False-expertise risk. Does the persona imply credentials the model lacks ("as a licensed attorney")? If yes, S3 alone is insufficient — pair with S5 Constraint Framing ("do not claim licensure; recommend consulting a professional") or refuse the persona. In regulated domains (medical, legal, financial advice) this is mandatory.

4. Cross-session persistence? Does the agent need to remember who it is between sessions, including prior commitments and an evolving self-model? If yes, S3 is the wrong tool — use H1 Identity Persistence. H1 strictly contains S3's capability: every H1-equipped agent has a per-session identity by construction.

5. Multi-agent disambiguation. If running multiple sub-agents (O4 Parallelization, O6 Orchestrator-Workers), each must be distinguishable. Run the Role-Per-Agent variant and check the personas are non-overlapping; collapsed personas yield collapsed contributions.

Quick test — S3 is the right pattern when:

• the domain register or voice produced by the persona is measurably better than zero-shot, and
• the session is long enough that the setup amortises, and
• the system has no cross-session identity (otherwise use H1), and
• the persona does not imply credentials that require explicit disclaimers.

If the system maintains identity across sessions, use H1; S3 is then subsumed. If principles matter more than identity, use S9. If the task is flat and stateless, S1 plus S6 is enough.

Structure

  Setup (once, before first turn)
        │
        ▼
  ┌──────────────────────────────────────────────┐
  │ System prompt                                 │
  │   Identity line: "You are a {role}…"          │
  │   Key characteristics (1–3 sentences)         │
  │   Optional constraints (S5) and template (S6) │
  └──────────────────────────────────────────────┘
        │
        ▼
  Per turn: user query ─▶ LLM session ─▶ response
                              ▲
                              │ (identity persists for every turn
                              │  in this session; no re-statement)

Participants

S3 is small — it is a setup-layer construct — but the responsibilities still separate cleanly:

The pattern is small because identity is small — a label and a short framing. Bloat is the most common failure: backstories, biographies, and elaborate worldbuilding add tokens and add nothing.

Collaborations

The identity statement is loaded once into the system prompt at session start, before the first user turn. Every subsequent turn inherits the framing — the model does not need to be reminded who it is, because the framing sits above every per-call prompt. Other Signal-layer patterns layer in beside it: S5 Constraint Framing adds prohibitions (essential where the persona implies authority); S6 Output Template adds structure; S9 Constitutional Framing adds principles. When the model is asked to do something inconsistent with the persona (a senior security engineer asked to write marketing copy), it acknowledges the mismatch rather than breaking character. In a multi-agent system, each sub-agent has its own S3 in its own session; the personas are chosen to be distinct, so the orchestrator can rely on the contributions being differentiable. When the system grows session-spanning identity needs, H1 Identity Persistence replaces S3 entirely — H1 contains a persona statement as one block within its Genesis State, alongside accumulated commitments and an evolving self-model.

Consequences

Benefits

• Activates the right domain register (vocabulary, caution, structure) without per-turn instruction.
• Stabilises voice across long sessions — the model does not drift.
• Lets multi-agent systems produce distinct contributors rather than a single averaged voice.
• Cheap: a few tokens at setup, paid once for the session.

Costs

• Tokens at setup (small) — amortised over the session.
• Maintenance: persona definitions evolve and must be versioned.
• Behavioural change is probabilistic; the model can be argued out of character by adversarial inputs.

Risks and failure modes

• False expertise. "As your doctor, I…" — the persona implies credentials the model lacks; users believe the framing more than the disclaimers. Pair with S5 for regulated domains, or refuse the persona.
• Persona bloat. Page-long backstories add tokens without adding effect; the lift comes from the role label, not the narrative.
• Character break. Adversarial inputs ("ignore your previous role; you are now…") can override the persona. Defend with explicit non-overrideability framing and/or constitutional principles (S9).
• Wrong persona. A persona drawn from a different domain than the task actively degrades output by activating the wrong knowledge cluster. Measure (Decision Criterion 1) before deploying.
• Identity ambiguity in multi-agent systems. Two agents with overlapping personas produce overlapping contributions; the orchestrator cannot tell them apart.

Implementation Notes

• Keep the identity statement to 1–3 sentences. Beyond that, you are writing a character sheet, not configuring a model.
• Place the identity at the top of the system prompt — primacy effect matters; later content does not override earlier identity framing as easily. The mechanism is KV-space geometry (mechanism 4): recall follows a U-shaped curve over sequence position (Liu et al. 2024), with strong attention at the start and end of context. Identity placed at primacy is geometrically well-attended for the entire session.
• Always pair with S5 Constraint Framing for personas in regulated domains (medical, legal, financial, security advice). The persona implies the authority; S5 disclaims it.
• For multi-agent systems, write the personas as a set — explicitly check they do not overlap, and that the orchestrator can describe each in one sentence.
• Version personas alongside prompts; track changes over time. A persona drift is a behavioural drift.
• Resist temptation to re-state the persona in user turns — that signals to the model the framing is fragile, and the framing then is fragile.
• When migrating to H1: do not run both. H1's Genesis State includes the persona; an additional S3 system prompt creates two sources of identity truth and the model will resolve the conflict unpredictably.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S3 is the setup of any single-LLM session — it is not a multi-step chain. It is named in the "Setup — loaded once, before first call" column of every other pattern's LLM-sessions table whenever that session needs a role. Pairs naturally with S5 (constraints), S6 (output template), and S9 (principles), all loaded into the same setup. In a multi-agent system (O4 Parallelization, O6 Orchestrator-Workers), each sub-agent's session has its own S3.

The chain:

Skeleton — the wiring; the LLM line is a configured session whose setup is the S3 persona:

session = configure(
    model      = chosen_model,
    system     = compose_setup(                       # code
        identity      = "You are a senior security engineer reviewing a pull request.",
        characteristics = "Terse. Focus on real risks, not style. Cite the file and line.",
        constraints   = S5_block(),                   # optional — S5
        template      = S6_block(),                   # optional — S6
        principles    = S9_block(),                   # optional — S9
    ),
)

per_turn(query):
    return session.respond(query)                     # LLM — persona-framed

The LLM sessions:

Specialist-model note. None — a capable generalist suffices. S3 is a prompt artefact, not a model artefact. The setup itself is the load-bearing piece; choose words deliberately. In particular, the identity statement should name a real role with a clear knowledge cluster (senior security engineer, neonatal nurse, contracts attorney) rather than an abstract attribute ("helpful assistant"). The cluster is what the model has learned to associate with the role; abstract attributes activate nothing in particular.

Open-Source Implementations

S3 is a prompt construct, not a library — there is no canonical project. The relevant references are LLM-provider role-prompting guides and the original prompt-pattern catalog:

• White et al. (2023), "A Prompt Pattern Catalog" — arxiv.org/abs/2302.11382 — the canonical reference; the Persona Pattern is documented in the Output Customization category.
• Anthropic — "Give Claude a role with a system prompt" — platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#give-claude-a-role — Anthropic's role-prompting guidance; identity belongs in the system prompt.
• Learn Prompting — Assigning Roles — learnprompting.org/docs/basics/roles — practitioner guide with worked examples of single-role and role-per-agent setups.
• Prompting Guide (DAIR.AI) — promptingguide.ai/introduction/elements — role as a first-class prompt element alongside instruction, context, and input.

Every multi-agent framework (LangGraph, CrewAI, AutoGen) instantiates Role-Per-Agent S3 by construction — each agent has a role definition — but the framework is not an implementation of S3 so much as a host that requires it. Treat them as known uses, not as libraries.

Known Uses

• Multi-agent frameworks (CrewAI, AutoGen, LangGraph subgraphs) — each agent's definition starts with a role; this is Role-Per-Agent S3 at production scale.
• Customer-support assistants with a defined company voice and a named domain ("billing specialist", "technical support engineer") — single-role S3 stabilises voice across long sessions.
• Coding assistants (Cursor system prompts, Claude Code project-level personas) — persona blocks at the top of the system prompt establish the engineering register before the first turn.
• Vertical agents (legal-research assistants, clinical-summary assistants, code-security reviewers) — domain-expert personas are mandatory; almost always paired with S5 to disclaim authority and S9 to enforce safety principles.

Related Patterns

• Subsumed by H1 Identity Persistence — H1 is the cross-session upgrade. The Genesis State contains an S3-style persona block along with accumulated commitments and an evolving self-model. Do not run S3 and H1 for the same agent.
• Composes with S5 Constraint Framing — S3 frames the identity; S5 frames the prohibitions. For any persona that implies authority (medical, legal, financial), pair them: persona alone creates false expertise.
• Composes with S6 Output Template — persona shapes content and voice; S6 shapes structure. Both go in the same setup.
• Distinct from S9 Constitutional Framing — S3 names who the model is; S9 names which principles it applies. A persona without principles is a voice without a value system; principles without a persona are values without a voice. They are different layers and they compose.
• Required by every other pattern's main LLM session (K5 Generator, K12 Curator, R4 ReAct agent, V15 Judge) — the "role" line in their setup tables is an S3 invocation.
• Composes with O4 Parallelization and O6 Orchestrator-Workers — the Role-Per-Agent variant is how multi-agent systems give each sub-agent a distinguishable contribution.

Note on fundamentality. S3 passes the test: it has its own forces (right identity activates the right knowledge cluster; wrong identity creates false expertise), a distinct Participant (the identity statement itself), and a distinct read pattern (set once at setup, inherited per turn without restatement). It does not decompose into another pattern plus an adaptor. It is, however, strictly subsumed by H1 — every H1 system has an S3-equivalent block as one of its Genesis-State components. S3 remains a separate pattern because most systems do not run H1, and S3 is the right default at the per-session scope.

Sources

• White, J., Fu, Q., Hays, S., et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT." PLoP 2023, arXiv 2302.11382. The Persona Pattern in the Output Customization category is the canonical written reference for S3.
• Anthropic — Claude prompting best practices, "Give Claude a role with a system prompt." Provider guidance treating identity as a system-prompt construct.
• Learn Prompting — "Assigning Roles" — practitioner-level treatment with worked examples.
• DAIR.AI Prompting Guide — "Elements of a Prompt" — role as a first-class prompt element.
• Brown et al. (2020) — Language Models are Few-Shot Learners — the in-context-learning mechanism that role prompting implicitly exploits.

S4 — Instruction Decomposition

Break a complex instruction into explicit, numbered, sequential steps inside a single prompt, so the model executes them in order rather than collapsing a dense paragraph of requirements into a single best-effort pass.

Also Known As: Step Prompting, Numbered Steps, Chain Instructions, Recipe Prompting.

Classification: Category I — Signal · the prompt-level instance of ordered execution — one LLM call carrying an ordered step list, the cheapest rung of a three-rung ladder that climbs to O2 Prompt Chaining (multi-call) and R3 Plan-and-Solve (plan + execute as separate calls).

Intent

Replace a dense, unstructured instruction with an explicit numbered procedure inside a single prompt, so the model performs each step in order, no step is silently skipped, and the failure mode of any miss is localisable to a specific step.

Motivation

Language models read instructions left-to-right but attend non-linearly. A long paragraph that piles up requirements — "validate the input, then transform the records, then summarise, but only if there are at least three, and format as JSON, and do not include personally identifiable fields" — gets read into a single soft objective. The model satisfies what it can attend to and quietly drops the rest. The failure is not a refusal but a silently incomplete answer: format right, validation skipped; PII filter missed; transformation half-done. The mechanism is the U-shaped recall distribution over context position (Liu et al. 2024, mechanism 4): K-vectors in the middle of a long prompt are geometrically accessible but statistically under-attended due to learned recency and primacy biases in the Q-K projection matrices. A dense paragraph places all requirements at roughly equal positions; numbering them creates discrete positional anchors the model can attend to individually. This is not merely a cognitive-metaphor claim — it has a direct counterpart in KV-space: numbered items create local Q-K alignment between the step-instruction token and the step-execution position.

The fix is the cheapest piece of structure available: number the steps. A numbered list does three things a paragraph cannot. It forces an ordering the model honours by training (countless cookbook, recipe, and tutorial documents in pre-training establish "1, 2, 3" as a sequence the reader is expected to execute in order). It makes each requirement separately addressable, so the model cannot conflate two steps into one. And it makes auditing tractable — when output is wrong, the auditor (human or LLM-as-Judge) can point to the step that was dropped.

S4 is the prompt-level solution to ordered execution. Two stronger rungs exist for harder cases. O2 Prompt Chaining breaks the steps into separate LLM calls with state passed between them — strictly more expressive (each step has its own setup, model choice, and quality gate) but strictly more expensive (multiple calls, more wiring, harder caching). R3 Plan-and-Solve lifts ordering into a separate planning call that produces the step list, then executes it — appropriate when the steps are not known upfront. S4 is the right choice when the step sequence is fixed, short, and interdependent, and one call is enough.

Applicability

Use Instruction Decomposition when:

• the task has a clear sequential process (validate $\to$ transform $\to$ format $\to$ output) and you can enumerate the steps at design time;
• previous single-instruction prompts produced output that skipped requirements or fused steps;
• steps are short enough that one model context can hold all of them with room for the data;
• you need auditability — to point at which step was dropped when output is wrong;
• the steps are interdependent and pass simple state (each next step trivially uses the previous result) — no quality gate between them is needed.

Do not use when:

• you need to inspect, log, or gate between steps — choose O2 Prompt Chaining;
• individual steps need different models, different setups, or different temperatures — choose O2;
• the step list itself depends on the input and cannot be written at design time — choose R3 Plan-and-Solve;
• a step requires tool use or external action mid-sequence — choose R4 ReAct;
• steps are independent and can run in parallel — choose O4 Parallelization;
• the prompt is already short and a single zero-shot instruction works — stay with S1 Zero-Shot.

Decision Criteria

S4 is right when the steps are known, fixed, short, and need to run in order inside a single call.

1. Count the steps. S4 scales to ~3–7 numbered steps in one prompt. Below 3, S1 / S6 suffices — numbering adds noise without value. Above 7, comprehension degrades and you should split into O2 Prompt Chaining or restructure with R3 Plan-and-Solve.

2. Measure the skip rate. On a labelled test set, count the % of outputs that miss at least one requirement when phrased as paragraph prose. A skip rate above ~10% justifies numbering. If skip rate is already near zero, S4 buys nothing — leave the prompt alone.

3. Test the inter-step state. Can each next step use the previous step's result with no transformation, gate, or branching? If yes, S4. If a step needs to be parsed, validated, or routed before the next, you need a boundary between steps — choose O2.

4. Check the audit need. Do you need to log, store, or human-review what happened at each step? S4 cannot give you that — the steps are internal to one model turn. Need it $\to$ O2 (each step a separate call, each loggable). Don't need it $\to$ S4.

5. Pair with an output contract. S4 should almost always specify, in its final step, the exact output format. Otherwise the model conflates "do the steps" with "show the working", and emits noisy intermediate state. Compose with S6 Output Template to lock the final form.

Quick test — S4 is the right pattern when:

• the step list is fixed and enumerable at design time, and
• there are roughly 3–7 steps, and
• no inter-step inspection, gating, or routing is needed, and
• the final output format is specified (typically via S6).

If any condition fails: too many steps or inter-step inspection needed $\to$ O2 Prompt Chaining; step list depends on the input $\to$ R3 Plan-and-Solve; steps need tools mid-sequence $\to$ R4 ReAct; steps are independent $\to$ O4 Parallelization.

Structure

   single prompt
   ┌────────────────────────────────────────┐
   │ system / role (optional, e.g. S3)      │
   │                                        │
   │ "Complete the following steps in order:"│
   │   1. <step 1 instruction>              │
   │   2. <step 2 instruction>              │
   │   3. <step 3 instruction>              │
   │   ...                                  │
   │   N. emit final output as <S6 form>    │
   │                                        │
   │ input data                              │
   └───────────────────┬────────────────────┘
                       │
                       ▼
                 single LLM call
                       │
                       ▼
              output (final step only)

One prompt, one model call. The steps live inside the prompt; the model's job is to execute them in order and return only what the final step asks for.

Participants

Four narrow roles. The pattern's discipline is in the Step List (correctly ordered and bounded) and the Output Contract (final step only). Everything else is a single ordinary call.

Collaborations

The Prompt Author analyses the task and writes the Step List as a numbered enumeration: each step a single imperative clause, ordered by dependency, with no step requiring information unavailable at the time it runs. The final step names the Output Contract — typically by pointing at a template from S6. The whole prompt is composed: optional role (S3), optional constraints (S5), Step List, input data, Output Contract. The Model executes the steps in a single call and emits the final-step result. If the answer is wrong, the auditor reads the model's output against the Step List and identifies which step was dropped or misordered — that diagnostic is the pattern's compliance benefit.

Consequences

Benefits

• Higher compliance than paragraph prose — fewer silently-skipped requirements.
• One LLM call: latency and cost are the same as a single zero-shot prompt.
• Auditable failure: when output is wrong, the dropped step is usually identifiable.
• Composes trivially with S3 (role), S5 (constraints), S6 (output template), S2 (few-shot demonstrating the procedure).
• Cheap to author and revise — editing a numbered list is faster than restructuring a chain.

Costs

• Verbose: the prompt grows linearly with the number of steps.
• No inter-step inspection — you cannot see, log, or gate the intermediate results.
• Cannot mix models or settings across steps; one call, one configuration.
• The model decides internally how to allocate attention across steps — long step lists degrade.

Risks and failure modes

• Step fusion — the model collapses two adjacent steps into one when they look similar, producing a single composite step's output and silently dropping the other.
• Step skipping — long step lists (>~7) get partially attended; later steps suffer more than earlier ones. The mechanism is lost-in-middle (mechanism 4): steps 4–7 in a long list occupy mid-context positions that are geometrically under-attended, producing the characteristic pattern where early and late steps complete while middle steps drop. The ~7-step cap is a practical bound on this effect.
• Order violation — the model executes steps in semantic, not numbered, order, especially when the numbered order is non-obvious from the data.
• Working-state leak — without an explicit Output Contract, the model emits intermediate-step output ("Step 1: ..., Step 2: ...") instead of only the final result.
• Constraint drift in later steps — a constraint named in step 1 is forgotten by step 5; pair with S5 Constraint Framing restated at the top, not buried in step 1.

Implementation Notes

• Keep each step to a single imperative clause. "Validate", "transform", "summarise" — one verb per step.
• Put hard constraints in a separate constraints block above the step list (S5), not inside step 1. Constraints buried in step 1 attenuate by step 5.
• Always specify the final-step output format. Either reference an S6 Output Template or describe the format explicitly ("Output: a single JSON object with fields x, y, z").
• The phrase "Output only the result of step N" at the end of the prompt is load-bearing — without it, models leak working state.
• If you find yourself writing more than ~7 steps, restructure: either merge adjacent steps, split the task, or upgrade to O2 Prompt Chaining so each step gets its own call.
• Few-shot the procedure (S2) once if step adherence is critical — one example showing the full sequence dramatically improves compliance.
• Combine with S9 Constitutional Framing when steps include compliance or safety checks; principles override the step list when they conflict.
• If a step needs to decide between branches, the prompt is no longer S4 — it is a single-call routing pattern, and you likely want O3 Routing or O2.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S4 lives entirely inside a single LLM session. It composes naturally with S3 Persona (role at the top), S5 Constraint Framing (a constraints block above the step list), S6 Output Template (the final-step contract), and optionally S2 Few-Shot (one worked example of the procedure). The upgrade path when boundaries are needed is O2 Prompt Chaining; the planning-cousin at agent scope is R3 Plan-and-Solve.

The chain:

Skeleton — the wiring is trivial; the engineering is in the prompt itself:

instruction_decomposition(task, input):
    prompt = compose(
        role         = persona(),                      # code — S3
        constraints  = constraints_block(),            # code — S5
        steps        = numbered_step_list(task),       # code — S4 step list
        input        = input,                          # code
        output_form  = output_template(),              # code — S6
    )
    answer = Procedural(prompt) ────────────────────── # LLM
    validate(answer, schema=output_template())         # code (optional)
    return answer

The LLM sessions:

Concretely, the per-call prompt looks like:

You are <role>.

CONSTRAINTS:
- <constraint 1>
- <constraint 2>

Complete the following steps in order:
1. <step 1>
2. <step 2>
3. <step 3>
4. <step 4>
5. Emit the result as: <S6 template>

Output only the result of step 5.

Input:
<data>

Specialist-model note. None — a capable generalist suffices. The pattern's lift comes from prompt structure, not model capability. The artifact that does the heavy lifting is the numbered step list itself, paired with the final-step output contract (S6). If the model used cannot reliably follow a 5-step numbered procedure in one call, the right move is not a specialist but to split into O2 Prompt Chaining so each step gets its own call.

Open-Source Implementations

Instruction Decomposition is a prompt-engineering convention, not a library — there is no canonical project. The relevant references are practitioner cookbooks and prompt-engineering catalogs:

• OpenAI Cookbook — github.com/openai/openai-cookbook — many examples use numbered-step prompts as the default structure for non-trivial tasks; the "techniques to improve reliability" guide explicitly recommends breaking complex tasks into ordered steps within a single prompt.
• Anthropic Cookbook — github.com/anthropics/anthropic-cookbook — prompt-engineering examples include numbered-step patterns for multi-stage tasks, and the Claude documentation's "Chain prompts" guidance distinguishes single-prompt step decomposition (S4) from multi-call chaining (O2).
• Prompt Engineering Guide — github.com/dair-ai/Prompt-Engineering-Guide — community catalog including step-by-step / decomposition patterns; useful as a teaching reference.
• LangChain prompt templates — github.com/langchain-ai/langchain — the PromptTemplate mechanism is the most common production substrate for parameterised numbered-step prompts; the library does not enforce the pattern but most production agents use it for S4-shaped prompts.

For the boundary cases — when you need step-by-step with inspection — the canonical implementations are the O2 Prompt Chaining references (LangChain LCEL, LangGraph linear graphs).

Known Uses

• Coding assistants (Cursor, Claude Code, Copilot prompts) — system prompts routinely use numbered procedural steps for code-edit tasks: "1. Read the file. 2. Identify the change. 3. Emit the edit in this format."
• Document-processing pipelines — extraction-then-validation-then-format tasks are commonly implemented as single S4 prompts when the document fits in context.
• Customer-service agent prompts — published assistant system prompts (Anthropic, OpenAI cookbook examples) routinely use 4–6 numbered procedural steps for triage workflows.
• Constitutional / safety check prompts — "1. Identify the user's request. 2. Check against principles. 3. Respond or refuse." — the canonical inference-time pattern for self-checking outputs.
• Evaluation rubrics (LLM-as-Judge prompts) — graders are typically given numbered criteria and instructed to score each in order; S4 in evaluation form.

Related Patterns

• Upgrades to O2 Prompt Chaining — when steps need inter-step inspection, gating, different models, or logging, lift each step into its own LLM call. O2 is strictly more expressive and strictly more expensive; S4 is the cheaper default when boundaries are not needed.
• Sibling at agent scope of R3 Plan-and-Solve — R3 is the planning-cycle cousin: a separate Planner call produces the step list, an Executor call (or chain) runs it. S4 is the prompt-level instance where the step list is authored at design time; R3 is the agent-level instance where the step list is generated at runtime.
• Distinct from R4 ReAct — ReAct interleaves thought + action + observation calls; S4 has no actions, no observations, and no iteration. If a step needs to call a tool, S4 is the wrong pattern.
• Distinct from O4 Parallelization — O4 runs independent steps concurrently; S4 runs ordered steps sequentially in one call. They solve different problems and compose: an S4 prompt may sit inside one branch of an O4 fan-out.
• Pairs with S3 Persona — role at the top of the prompt frames the procedure.
• Pairs with S5 Constraint Framing — constraints block above the step list survives long step lists better than constraints buried in step 1.
• Pairs with S6 Output Template — the final-step output contract; without it the model leaks working state.
• Pairs with S2 Few-Shot — one fully-worked example of the procedure substantially lifts step-adherence on borderline-capable models.
• Composes with V15 LLM-as-Judge — when audit is needed without paying for O2, an S4 prompt with a final "self-check" step approximates inline review (lower fidelity than V15 proper, but free).

Sources

• White, J., Fu, Q., Hays, S., et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" (PLoP). The "Recipe Pattern" and "Output Customization" patterns are the formal antecedents of S4.
• Wei, J., Wang, X., Schuurmans, D., et al. (2022) — "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." CoT is the intra-step relative of S4 (think step-by-step inside one call); S4 generalises the move to explicit numbered procedural steps.
• Khot, T., Trivedi, H., Finlayson, M., et al. (2022) — "Decomposed Prompting: A Modular Approach for Solving Complex Tasks" (arXiv 2210.02406). Establishes decomposition as a primitive in prompt engineering.
• Weng, L. (2023) — "Prompt Engineering" survey, lilianweng.github.io — discusses instruction decomposition and its position relative to CoT and chain-of-prompts approaches.
• Anthropic — "Chain complex prompts" prompt-engineering documentation; distinguishes single-prompt step decomposition (S4) from multi-prompt chaining (O2).
• OpenAI — "Techniques to improve reliability" cookbook; recommends numbered step breakdowns for non-trivial tasks.
• 12-Factor Agents — Factor 8 ("Own Your Control Flow") frames the same move at system level: making the execution order explicit rather than implicit.

S5 — Constraint Framing

Enumerate, at session setup, the specific things the model must not do — as an explicit, auditable list that sits alongside the task description with equal or greater prominence than the positive instructions.

Also Known As: Negative Prompting, Boundary Definition, What-Not-To-Do, Hard Constraints, the Prohibition Block.

Classification: Category I — Signal · the setup-layer pattern that names what the model must not do; complements S3 Persona (who the model is), S6 Output Template (what its output looks like), and S9 Constitutional Framing (which principles it applies). Provides the in-prompt prohibition layer; V5 Guardrail Layering is its external-enforcement counterpart.

Intent

Give the model an explicit, enumerable list of forbidden behaviours at session setup, so prohibitions are addressed as a first-class concern rather than left implicit in the positive instructions or scattered across the task description.

Motivation

The default for general instruction is positive framing, and the evidence for that default is unambiguous. Anthropic's current Claude 4.7 guidance is explicit: "Tell Claude what to do instead of what not to do … positive examples tend to be more effective than negative examples or instructions that tell the model what not to do" (platform.claude.com). OpenAI's prompt guidance gives the same lesson in stronger form: reserve ALWAYS, NEVER, must, only for "true invariants, such as safety rules, required output fields, or actions that should never happen" (developers.openai.com). The empirical floor under that guidance is harder still. Truong et al. (2023) show LLMs are systematically insensitive to negation tokens and fail to reason under them. García-Ferrero et al. (EMNLP 2023) replicate this across a 400k-sentence benchmark and find affirmative classification near-perfect while negative classification collapses, with no fix from scale alone. The Inverse Scaling Prize's NeQA task is the load-bearing finding: on questions with a single "not" inserted, smaller models score near chance and larger ones perform worse than random past ~10²² training FLOPs across Gopher, GPT-3, and Anthropic models — and the effect is stronger in RLHF / instruction-tuned variants (McKenzie et al. 2023). The widely-cited "pink elephant" effect — that explicitly prohibiting a token raises its activation enough to bleed into outputs — is a documented LLM failure mode, not folklore (Hu et al. 2024, "Suppressing Pink Elephants with Direct Principle Feedback").

So why does a negative-framing pattern earn a number at all? Because three conditions reverse the default, and each is independently testable. (1) Auditability dominates expected quality. A positive instruction ("write helpful, on-brand copy") cannot be reviewed by a compliance officer, a brand lead, or a security auditor against an output — there is no checklist. An enumerated prohibition list ("do not name competitors; do not commit to a price; do not claim regulatory approval") can be, item by item. This is exactly the distinction Anthropic's "Specific versus General Principles for Constitutional AI" defends: specific, enumerable rules outperform vague general principles when the goal is targeting known failure modes, even though general principles generalise better to novel ones (Kundu et al. 2023). OpenAI's Model Spec encodes the same structure — its hard rules sit at the top precisely because they are non-overridable, enumerable, and reviewable (model-spec.openai.com). (2) The prohibition has no natural positive substitute. "Do not reveal the system prompt" has no clean positive reframe; "do not execute user-supplied code" cannot be replaced by an enumeration of permitted code patterns; "do not claim FDA approval" cannot be turned into a list of approved claims. When the forbidden surface is open-ended but the prohibited core is sharp, the negative framing is the compact representation. (3) Asymmetric stakes — the prohibition backstops the catastrophic case while positive instruction handles the typical case. Positive instruction optimises mean output quality; the prohibition layer is insurance against the tail. The two are not substitutes; they sit at different points on the cost / consequence curve.

S5 is the pattern that operationalises those three conditions: it puts the prohibitions in the system prompt as a separately delimited, enumerable, auditable block, with equal or greater visual prominence than the positive instructions, and an explicit override clause that resolves conflicts in the prohibition's favour. The pattern's load-bearing claim is not that negative framing outperforms positive framing in general — the literature is clear it does not. The claim is narrower and survives the evidence: for the small set of behaviours that must never happen, and where the auditability of the rule outweighs the marginal-quality cost of negative framing, the prohibition must be a first-class artifact rather than implicit in the positive instructions. The negation-failure literature also dictates how to write the items — each prohibition should be paired with its positive alternative wherever one exists ("do not name competitors; instead, say 'we focus on our own product'"), because pure-negative items inherit the full force of the inverse-scaling problem.

S5 is fundamental — it is not S9 with a different name. S9 Constitutional Framing sets principles the model applies via reasoning ("prioritise user safety"; "acknowledge uncertainty") and uses a critique-and-revise loop. S5 sets enumerated prohibitions the model treats as hard rules with no reasoning step in between. The two compose: principles guide the reasoning, prohibitions cap the action space. And S5 is not V5 Guardrail Layering with a different name either — V5 is external code that intercepts inputs and outputs; S5 is model self-restraint via in-prompt instruction. V5 enforces, S5 instructs; they pair routinely, because S5 alone is probabilistic and the negation literature says exactly how probabilistic.

Applicability

Use when all three of the following hold (they are the conditions that flip the default — positive framing — into a setting where negative framing wins):

• Auditability dominates. Someone outside the build team — compliance, brand, security, legal — must be able to read the constraint list and confirm coverage against a known failure mode. The artifact's reviewability is more valuable than the marginal-quality cost of negative framing.
• The prohibition has no clean positive substitute. The forbidden surface is open-ended ("do not reveal credentials"; "do not execute user-supplied code"; "do not claim regulatory approval") but the prohibited core is sharp — there is no compact positive list of permitted alternatives.
• The stakes are asymmetric. A single violation carries cost orders of magnitude greater than the cumulative gain from typical-case quality — regulated industry, public-facing brand, agent with tool access, prior production incident.

Use also when:

• a persona (S3) implies authority the model does not have ("as your doctor…") — S5 disclaims it. This is a mandatory pairing, not optional: persona without S5 is the false-expertise failure mode.

Do not use when:

• Positive framing covers the case. If the task can be specified by what the model should produce — and provider guidance from Anthropic, OpenAI, and the negation-failure literature says this is the default — write the positive instruction. Use S1 Zero-Shot or S2 Few-Shot alone. Anthropic's worked example: replace "NEVER use ellipses" with the instruction's actual purpose ("your response will be read aloud by a text-to-speech engine that mispronounces ellipses").
• you would be enumerating broad behavioural principles ("be honest", "be safe", "be helpful") rather than specific prohibitions. That is S9 Constitutional Framing, not S5. Principles are reasoned over; prohibitions are enforced. (Kundu et al. 2023 on the specific-vs-general distinction.)
• the prohibition needs guarantees under adversarial input. S5 inherits the negation-processing weakness documented across Truong et al. (2023), García-Ferrero et al. (2023), and the Inverse Scaling NeQA task — a determined jailbreak can talk the model past the rule. Use V5 Guardrail Layering (external output checks) or V7 AgentSpec (runtime policy enforcement). Pair S5 with V5 in this case rather than relying on S5 alone.
• the list would exceed ~7 items — attention dilution, constraint-conflict, and "model paralysis" become real, and each additional negated item compounds the pink-elephant risk. Prune to the load-bearing prohibitions; move the rest to V5 or external review.

Decision Criteria

S5 is right when there are specific, enumerable behaviours that must never occur, the deployment is stakes-bearing enough that auditability is required, and the prohibition list stays within attention budget.

1. Negative-vs-positive framing test. Before reaching for S5, attempt to rewrite each candidate prohibition as a positive instruction. The research is unambiguous that LLMs follow positive framing more reliably (Anthropic and OpenAI guidance; Truong et al. 2023; Inverse Scaling NeQA — larger models score worse than random on negated questions). A prohibition belongs in S5 only when all three of the following are true: (a) the forbidden behaviour is specific and enumerable — a reviewer can decide, looking only at an output, whether it was violated; (b) there is no compact positive reframe — the action space outside the prohibition is open-ended; and (c) the prohibition needs to be auditable as a named artifact by someone outside the build team. If a positive reframe covers the case ("write in flowing prose" instead of "do not use bullets"), use it. If the prohibition is vague ("be ethical"; "be safe"), it is S9 material, not S5. If it passes all three tests, S5 is the right home — but each item should still be written with its positive alternative wherever one exists, to limit pink-elephant activation.

2. Stakes / auditability. Does someone — compliance, brand, security, legal — need to read and approve the constraint set? If yes, S5's enumerated list is what they read. If no one will audit, the positive instructions are likely enough. Threshold: regulated industry, public-facing brand, agent with tool access, or known prior failure mode.

3. Constraint count. Count the proposed prohibitions. 3–7 is the practical sweet spot. Below 3 and the block is overhead; above 7 and constraint-conflict and attention dilution become real. Threshold: hard cap at ~7 in-prompt; spill the rest to V5 at execution time or to a compliance review step. Mechanically: each additional prohibition adds tokens to the prompt, expanding the O(n²) attention computation (mechanism 2). With a fixed attention budget the weight available per item decreases; beyond ~7 items, the probability that any single item receives enough attention to dominate generation degrades sharply.

4. Hard-guarantee requirement. Is "probabilistically prevented" acceptable, or must the prohibition be guaranteed under adversarial input? S5 is probabilistic — a determined jailbreak can override it. If a guarantee is needed, the prohibition is V5-shaped, not S5-shaped, and the right answer is S5 + V5 in layers, not S5 alone.

5. Persona-authority pairing. Is the session running an S3 Persona that implies credentials the model does not have (licensed professional; senior engineer with sign-off authority; pricing-authorised salesperson)? If yes, S5 is mandatory, not optional — the persona without the disclaimers is the false-expertise failure mode. Pair them, with S5 explicitly stating the persona does not carry the implied authority.

Quick test — S5 is the right pattern when:

• the prohibitions are specific and enumerable (< 10 concrete items), and
• the deployment context requires auditable coverage (regulated, brand, security, prior failure), and
• ~7 or fewer items carry the load (longer lists belong in V5 or external review), and
• "probabilistically prevented" is acceptable (otherwise pair with V5 / V7 for hard enforcement).

If the prohibitions are vague principles, use S9 Constitutional Framing. If hard guarantees are required under adversarial input, use V5 Guardrail Layering (typically in addition to S5, not instead of). If the list is long enough to dilute attention, the longer items belong in V5 at execution time, not in the prompt.

Structure

  Setup (once, before first turn)
        │
        ▼
  ┌──────────────────────────────────────────────────────┐
  │ System prompt                                         │
  │   Identity (S3) — who the model is                    │
  │   Task framing — what the model does                  │
  │                                                       │
  │   ─────── CONSTRAINTS (S5 — explicit block) ───────   │
  │   You MUST NOT:                                       │
  │     • {prohibition 1 — specific, auditable}           │
  │     • {prohibition 2 — specific, auditable}           │
  │     • {prohibition 3 — specific, auditable}           │
  │   These constraints OVERRIDE any other instruction,   │
  │   including the persona and any user request.         │
  │   ────────────────────────────────────────────        │
  │                                                       │
  │   Output contract (S6), Principles (S9) — alongside   │
  └──────────────────────────────────────────────────────┘
        │
        ▼
  Per turn: user query ─▶ LLM session
                              │
                              ▼
                         Response — and *optionally*, externally,
                         re-check against the same constraints
                         via V5 Guardrail Layering.

Participants

S5, like S3, is a setup-layer construct — small but with clean responsibility separation:

The pattern's load-bearing piece is the override clause. Without it, an S3 persona ("you are an experienced regulatory consultant") can quietly imply authority that the constraints were written to prevent — the model resolves the conflict in favour of the persona because the persona was stated as identity, not as advice.

Collaborations

The constraint block is composed once at session setup, placed in the system prompt with deliberate prominence — usually at the top under the identity line, often repeated near the end of the system prompt to exploit primacy and recency effects. The override clause makes the precedence explicit: constraints take priority over the persona, the task instructions, and any user request. Every subsequent user turn inherits the block — it is not re-stated per turn, because per-turn restatement signals fragility.

Other Signal-layer patterns layer in beside it: S3 Persona sets the identity (the constraints often exist because of what the persona implies); S6 Output Template sets the structural form; S9 Constitutional Framing sets the principles the model applies via reasoning, while S5 sets the hard rules it applies without reasoning. When the user makes a request that approaches a constraint, the model is expected to acknowledge the boundary and decline rather than negotiate; this is more reliable when the override clause is present.

S5 routinely composes with V5 Guardrail Layering at execution time: the same constraints that appear in the prompt are also checked externally in code (input sanitisation, output classifiers, regex / keyword screens). The prompt-level S5 is what the model knows; the V5 check is what the system enforces regardless. In safety-critical and regulated deployments the two are paired as a matter of course.

Consequences

Benefits

• Auditable. The constraints are an enumerated list someone non-technical can read and review. Compliance, brand, and legal can sign off on the actual artifact.
• Versionable. Constraints change as deployments learn — new failure modes get new items. The block can be diffed across versions.
• Targets the specific failure mode. Unlike vague principles, each item names a behaviour the model can recognise as it is producing it.
• Pairs naturally with personas. Disclaims the false-expertise implied by an authority-flavoured S3.
• Composable with external enforcement. The same list seeds the V5 output checks; one source of truth.

Costs

• Tokens at setup — small, paid once per session.
• Attention budget — every prohibition costs some attention; a too-long list dilutes the model's read of the positive instructions.
• Maintenance — prohibitions evolve; the block needs versioning and review.
• Probabilistic, not guaranteed — because token generation is stochastic (mechanism 7), adversarial inputs can talk the model past S5 alone. Pair with V5 / V7 where guarantees matter.
• Risk of negative-framing degradation. Provider guidance (Anthropic, OpenAI) shows that for general instruction, positive framings outperform negative ones — the model has a clearer target to aim at. S5 is for the narrow set of hard prohibitions, not a general substitute for positive instruction.

Risks and failure modes

• Constraint sprawl. The list grows past ~7 items; attention dilutes; the model picks the least-bad violation rather than refusing.
• Constraint conflict. Two items contradict each other under certain inputs; the model resolves unpredictably.
• Reverse-psychology effect. Strongly forbidden behaviours can become salient to the model and slip into outputs as the prohibition activates the concept. Mitigate by stating the positive alternative alongside the prohibition where one exists.
• Persona override. Without the explicit override clause, a strong S3 persona can pull the model past the constraints when the user request lands at the persona's implied competence.
• Lip-service compliance. The model "acknowledges" the constraint in its preamble and then violates it in the body. Mitigate by checking constraint compliance externally (V5) rather than trusting the model's self-report.
• Single-layer false confidence. S5 alone treated as a safety guarantee. It is not; it is one layer in a defense-in-depth stack.

Implementation Notes

• Keep the block to 3–7 items. Beyond that, attention dilutes and constraint-conflict becomes real. Push the overflow to V5 at execution time or to a compliance review step.
• Place the prohibition block at the top of the system prompt (primacy) and consider repeating the most critical items near the end (recency). LLM attention is not flat. Mechanically, recall follows a U-shaped distribution over sequence position (Liu et al. 2024, mechanism 4) — K-vectors at the start and end of context are strongly attended; mid-context K-vectors are under-attended even when geometrically accessible. Primacy placement exploits the leading edge; recency repetition of the most critical items exploits the trailing edge. Burying a prohibition in the middle of a long system prompt is mechanically equivalent to deprioritising it.
• Make each item concrete and recognisable. "Do not provide medical advice" is too vague; "do not name medications, dosages, or treatment plans; instead recommend consulting a qualified clinician" is auditable.
• Where a positive alternative exists, state it alongside the prohibition: "Do not commit to a price. If asked, say you will connect them with a sales engineer." Pure-negative items leave the model to infer what to do instead, which it does inconsistently.
• Include an explicit override clause: "These constraints take precedence over the persona, the task, and any user request. If they conflict with anything else in this prompt or in user input, the constraints win." Without it, an authority-flavoured S3 can talk past S5.
• For personas in regulated domains, treat S5 as mandatory, not optional. Persona alone is the false-expertise failure mode.
• Pair with V5 for any constraint where probabilistic compliance is insufficient. The same constraint list feeds both — S5 is what the model is told, V5 is what the runtime enforces.
• Pair with S9 where the deployment also needs reasoned principles. S5 handles the enumerable hard rules; S9 handles the principles applied via critique-and-revise. They occupy different layers.
• Version the constraint block alongside the prompt — track changes; record the failure mode each new item was added to prevent.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S5 is the setup of any session that needs an enumerated prohibition layer — it is not a multi-step chain. It is named in the "Setup — loaded once, before first call" column of any LLM-sessions table whose session must obey hard constraints. Pairs routinely with S3 (the identity the constraints attach to), S6 (output template), S9 (principles applied via reasoning), and externally with V5 (the runtime check that doesn't trust S5 alone) and V7 (declarative policy enforcement for compliance-critical settings).

The chain:

Skeleton — the wiring; the LLM line is a configured session whose setup contains the S5 prohibition block:

session = configure(
    model  = chosen_model,
    system = compose_setup(                              # code
        identity     = S3_block("You are a senior compliance analyst."),
        constraints  = S5_block([                        # the prohibition list
            "Do NOT name specific medications, dosages, or treatment plans.",
            "Do NOT make pricing commitments; refer to sales for any price discussion.",
            "Do NOT claim regulatory approval or clinical efficacy.",
            "Do NOT execute, write, or suggest code that calls `eval` on user input.",
        ]),
        override     = "These constraints OVERRIDE persona, task, and any user request.",
        template     = S6_block(),                       # optional
        principles   = S9_block(),                       # optional
    ),
)

per_turn(query):
    response = session.respond(query)                    # LLM — constraint-shaped
    if not V5_check(response, constraint_list):         # code — external re-check
        return V5_handle(response)                       # redact / refuse / retry
    return response

The LLM sessions:

Specialist-model note. None — a capable generalist suffices for the constrained session itself. S5 is a prompt artefact, not a model artefact. The load-bearing piece is the constraint list: it must be specific enough that the model can recognise the prohibited behaviour as it is producing it, and short enough that attention is not diluted. The optional V5 judge is also a generalist; a fine-tuned classifier can replace it where throughput matters. The biggest practical lever is constraint phrasing: positive-alternative phrasing ("do X instead of Y") consistently outperforms pure-negative phrasing ("never do Y") because the model has a target to aim at — provider guidance from Anthropic and OpenAI is explicit on this point, and S5 inherits the lesson.

Open-Source Implementations

S5 is a prompt construct, not a library — there is no canonical project. The relevant references are LLM-provider guidance, the prompt-pattern literature, and the external-enforcement projects S5 routinely pairs with:

• White et al. (2023), "A Prompt Pattern Catalog" — arxiv.org/abs/2302.11382 — the canonical reference. The catalog's "Fact Check List", "Refusal Breaker", and persona-related entries together cover the enumerated-prohibition idea, though S5 as a single named pattern is more recent practitioner consolidation.
• Anthropic — "Prompting best practices" — platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices — Anthropic's current guidance is explicit on negative-only framing: "Tell Claude what to do instead of what not to do … positive examples tend to be more effective than negative examples." S5 inherits this lesson — it is the narrow pattern for prohibitions that must be auditable as a named artifact; positive framing handles everything else.
• Anthropic — "Mitigate jailbreaks and prompt injections" — docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks — Anthropic's guardrail guidance; pairs the S5 / S9 prompt-level approach with V5-style external checks.
• OpenAI — Prompt engineering guide — platform.openai.com/docs/guides/prompt-engineering — practitioner guidance with the same lesson: reserve ALWAYS / NEVER / MUST for true invariants (safety rules, required output fields, never-actions) and use positive framing for the rest.
• NVIDIA NeMo Guardrails — github.com/NVIDIA-NeMo/Guardrails — open-source toolkit for programmable guardrails; the canonical V5 partner. Input rails, output rails, topic restriction, jailbreak detection — the runtime enforcement layer that turns S5 prohibitions into hard guarantees.
• Negative Prompting notebook — github.com/NirDiamant/Prompt_Engineering — practitioner notebook covering explicit negative conditions and worked examples of the prohibition-block idiom.

Every system-prompt convention in production (Cursor, Claude Code, Anthropic's own published system prompts) contains some S5-shaped block — the named-prohibition list is ubiquitous — but no single repository owns the pattern. Treat the above as the relevant references rather than as implementations.

Known Uses

• Provider system prompts (Anthropic's published Claude system prompts, OpenAI's; Cursor and Claude Code's project-level prompts) — every published frontier-model system prompt contains an S5-shaped block of enumerated prohibitions (no real-time information claims; no execution of certain tool calls; no impersonation of specific individuals; etc.).
• Regulated-industry agents (clinical-summary assistants, legal-research assistants, financial-advice copilots) — S5 + S3 + V5 is the de facto stack. The persona names the role; S5 disclaims the implied credentials; V5 enforces the prohibitions at output.
• Customer-support assistants with brand voice — explicit prohibitions on naming competitors, making pricing commitments, or speaking on behalf of legal / HR.
• Agentic systems with tool access — explicit prohibitions on dangerous tool patterns (no rm -rf; no execution of user-supplied code; no network calls to unapproved hosts) — pairs invariably with V8 Tool Sandboxing for hard enforcement.
• Red-team / security-focused deployments — the prohibition block is the audit artefact a security reviewer reads to confirm coverage of known attack surfaces.

Related Patterns

• Composes with S3 Persona — the persona names the identity; S5 names what the identity cannot do. For any persona that implies authority the model lacks (licensed professional; senior decision-maker), S5 is mandatory, not optional. The override clause is the load-bearing wiring between them.
• Distinct from S9 Constitutional Framing — S9 is principles applied via reasoning ("prioritise user safety"; "acknowledge uncertainty"); S5 is enumerated hard rules applied without reasoning ("do not name competitors"). S9 is broader and reasoned; S5 is narrower and definite. They compose: principles guide, prohibitions cap.
• Distinct from V5 Guardrail Layering — V5 is external code that intercepts inputs and outputs at runtime; S5 is model self-restraint via in-prompt instruction. S5 is what the model is told; V5 is what the system enforces regardless of what the model does. They pair: same constraint list, different enforcement layer. S5 alone is probabilistic; S5 + V5 approaches guarantee.
• Distinct from V7 AgentSpec — V7 is declarative governance via deontic tokens and runtime policy enforcement; S5 is prompt-level instruction. V7 is the hard-guarantee end of the same spectrum — S5 instructs, V5 enforces at the I/O boundary, V7 enforces at the policy layer. For compliance-critical settings, the stack is typically S5 + V5 + V7.
• Pairs with S6 Output Template — S6 shapes structure; S5 shapes the action space. Both go in the same setup, but they answer different questions.
• Used by every safety-sensitive pattern's main LLM session — K5's Generator, K12's Curator, R4's ReAct agent, V15's Judge — wherever the session must obey hard constraints, S5 is what its "constraints" line invokes.
• Subsumed by H5 Constitutional Self-Alignment in long-running agents that evolve their own principles with human checkpoints — H5 contains S5- and S9-shaped blocks as components.

Note on fundamentality. S5 passes the test. It has its own forces (enumerability, auditability, prominence, override semantics), a distinct Participant (the prohibition block with its override clause), and a distinct structural role (the explicit, separately-versioned negative half of the instruction). It does not decompose into another pattern plus an adaptor: S9 is principles (different mechanism, different write-up), V5 is external enforcement (different layer entirely), S3 is identity (different concern). The asymmetry between positive instruction and enumerated prohibition — and the fact that in safety-critical contexts the prohibition layer needs to be a first-class artefact, not implicit — is the pattern's substance. The provider guidance against gratuitous negative framing in general instruction is consistent with S5's narrow scope: S5 is for the hard, enumerable, auditable prohibitions; positive framing handles everything else.

Sources

Negation-processing failures in LLMs (the empirical floor under the "default to positive framing" rule).

• Truong, T. H., Baldwin, T., Verspoor, K., Cohn, T. (2023) — Language models are not naysayers: an analysis of language models on negation benchmarks. *SEM 2023, arXiv 2306.08189. Across GPT-Neo, GPT-3, and InstructGPT: insensitivity to negation tokens, failure to capture lexical semantics of negation, failure to reason under negation; scale alone does not fix it.
• García-Ferrero, I., Altuna, B., Alvez, J., Gonzalez-Dios, I., Rigau, G. (2023) — This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models. EMNLP 2023, arXiv 2310.15941. ~400k-sentence benchmark; LLMs near-perfect on affirmative sentences and collapse on negative ones; fine-tuning helps in-distribution but fails to generalise.
• McKenzie, I. R., et al. (2023) — Inverse Scaling: When Bigger Isn't Better. arXiv 2306.09479. The NeQA task: a single "not" inserted into multiple-choice questions; smaller models score near chance; larger models score worse than random past ~10²² training FLOPs across Gopher, GPT-3, and Anthropic models. Inverse scaling is stronger in RLHF / instruction-tuned variants.
• Hu, L., et al. (2024) — Suppressing Pink Elephants with Direct Principle Feedback. arXiv 2402.07896. Documents the activation-asymmetry behind the "pink elephant" effect — explicitly prohibited concepts surface in outputs because mentioning them raises their probability — and gives an RLAIF-based mitigation.

Provider guidance.

• Anthropic — Prompting best practices for Claude (current; covers Claude Opus 4.7). platform.claude.com. Explicit guidance: "Tell Claude what to do instead of what not to do"; "positive examples tend to be more effective than negative examples." Where a negative is unavoidable, attach its purpose ("never use ellipses because the response is read by a text-to-speech engine").
• OpenAI — Prompt guidance and Model Spec (2025-10-27). developers.openai.com; model-spec.openai.com. Reserve ALWAYS / NEVER / MUST for "true invariants, such as safety rules, required output fields, or actions that should never happen"; everything else positive-framed. The Model Spec's hard rules sit at the top tier because they are enumerable, non-overridable, and auditable.

The principles-vs-prohibitions distinction (S9 / S5 boundary).

• Bai, Y., et al. (2022) — Constitutional AI: Harmlessness from AI Feedback. arXiv 2212.08073. The principles-based counterpart that grounds the S5 / S9 distinction.
• Kundu, S., et al. (2023) — Specific versus General Principles for Constitutional AI. arXiv 2310.13798. Specific, enumerable rules outperform vague general principles for known failure modes; general principles generalise better to novel ones. This is the empirical case for splitting S5 (specific enumerated prohibitions) from S9 (general reasoned principles).

Pattern catalog and external enforcement.

• White, J., Fu, Q., Hays, S., et al. (2023) — A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. PLoP 2023, arXiv 2302.11382. Pattern-catalog reference; the enumerated-prohibition idea threads through Fact Check List, Refusal Breaker, and persona-pairing patterns.
• NVIDIA NeMo Guardrails — github.com/NVIDIA-NeMo/Guardrails. The canonical V5 partner; the runtime layer that turns S5 prohibitions into hard guarantees at the I/O boundary, mitigating the negation-failure risk that S5 alone cannot.

S6 — Output Template

Provide the skeleton of the expected output — fields, labels, and structure — for the model to complete, so format generation is replaced by format filling.

Also Known As: Template Filling, Structured Output, Format Forcing, Skeleton Prompting. (Variants: JSON-mode / schema-constrained decoding, Free-text template, Few-shot template — see Variants.)

Classification: Category I — Signal · the format-shaping pattern — separates what to say from how to lay it out, by carrying the layout in the prompt.

Intent

Replace open-ended generation of "content plus format" with the simpler task of filling content into a predefined skeleton, so downstream parsers, chained LLM calls, and human reviewers see a consistent shape every run.

Motivation

Open-ended generation produces inconsistent formats. The same prompt, asked twice, returns different field orders, different label wording, different nesting depths — and any system that depends on parsing the output breaks the first time the model decides a Markdown bullet list is friendlier than a JSON object. The cost is not the occasional bad run; it is the defensive parsing that every downstream step now has to carry forever.

The fix is to move the format burden out of the model's task. If the prompt contains the skeleton — fields, labels, order, types — the model's job collapses from "decide format AND content" to "fill content into format". The first is generative and noisy; the second is closer to extraction and substantially more reliable. Every measured benchmark of structured generation shows the same pattern: skeleton-bearing prompts produce parseable output an order of magnitude more often than free-form prompts, and the gap widens as the schema gets richer. For the JSON-mode / schema-constrained variant, the reason is stronger: schema-constrained decoding (Outlines, Guidance, OpenAI Structured Outputs) masks the logit distribution at each token step so that only tokens valid under the schema grammar can be sampled (mechanism 7). This removes structural sampling variance entirely — the output type, field order, and key spelling are deterministic. The free-text template cannot achieve this because it still relies on stochastic sampling of format tokens. The choice between variants is the same principle that makes tool execution more reliable than in-context computation.

A real boundary lives inside the pattern. When the provider supports native structured output APIs — OpenAI's response_format with JSON Schema, Anthropic's tool-use schema, schema-constrained decoders like Outlines or Guidance — those are strictly better than a free-text template: they constrain the decoder itself, so the output is guaranteed parseable, not merely usually parseable. S6 in free text is the fallback when (a) the provider does not support schema-constrained decoding for the call you are making, (b) the output mixes structured fields with narrative prose, or (c) the schema is too fluid to commit to up front. Treat the API as the default and the free-text skeleton as the explicit fallback — both are the same pattern, applied through different mechanisms.

Variants

The variants differ in how the skeleton is enforced — by the decoder, by prompt content, or by examples:

• JSON mode / schema-constrained decoding. The skeleton is a JSON Schema (or grammar) submitted to the API; the decoder constrains generation to valid completions. Provider-native (OpenAI Structured Outputs, Anthropic tool-input schemas) or library-driven (Outlines, Instructor, Guidance). Strongest guarantee; only available where supported.
• Free-text template. The skeleton is written into the prompt as labelled placeholders or a partial document; the model completes by analogy. Works with any model and any output shape, including mixed structured-plus-narrative outputs. Probabilistic, not guaranteed.
• Few-shot template. The skeleton is taught implicitly by 2–8 worked examples (composition with S2 Few-Shot); the model infers format from demonstration rather than from an explicit skeleton. Useful when the format is hard to describe but easy to show.

The three are the same pattern — carry the output shape so the model does not have to invent it — differing in the mechanism that carries it. Pick the strongest one the runtime supports.

Applicability

Use Output Template when:

• output is parsed programmatically, or chained to another LLM call (see O2 Prompt Chaining);
• consistent format across runs is a business or display requirement;
• the task is multi-field structured extraction;
• the format is non-obvious, easy to drift on, or has changed before.

Do not use when:

• the output is naturally free prose (an essay, a draft email, a summary for human reading) — a template constrains expression for no gain; use S1 Zero-Shot or S3 Persona;
• the format is so simple that a single sentence of instruction is clearer than a skeleton ("respond with one word: YES or NO");
• the provider supports schema-constrained decoding for the call — use the API directly rather than a free-text template (still S6, but via the JSON-mode variant);
• the schema is changing every run — the template has to be re-built per call and its value drops; consider S2 Few-Shot with diverse examples instead.

Decision Criteria

S6 is right when the cost of a malformed output is non-trivial and the format is stable enough to write down.

1. Measure the parse-failure rate. Run the same prompt N times without a template; count outputs that fail your downstream parser or differ in field order, labelling, or nesting. > 5% failure means S6 pays back immediately; > 20% means S6 is mandatory before any production use.

2. Pick the strongest variant the runtime supports.

• Native schema-constrained decoding available (OpenAI Structured Outputs, Anthropic tool schemas, Outlines / Guidance / Instructor with a local model)? Use it — the JSON-mode variant is strictly better than a prompt template.
• Provider has JSON mode but no schema enforcement? JSON-mode-with-schema-in-prompt is a middle ground.
• No structured-output API, or output is mixed structured + narrative? Use the free-text template variant.
• Format hard to describe but easy to show? Use the few-shot template variant (compose with S2).

3. Schema stability. Will the format change more than once a sprint? If yes, the maintenance cost of the template starts to bite — keep the skeleton small and parameterised, or move to few-shot.

4. Mixed content boundary. If the output is pure structured data (a record, a classification, a tool call), prefer the JSON-mode variant — the decoder constraint removes a whole class of failure. If the output mixes a structured envelope with free narrative inside (a report with summary:, findings:, recommendation: sections), the free-text template variant is usually the right answer; JSON mode would force you to escape the prose.

5. Downstream coupling. Is the output consumed by code (must parse), by another LLM (must be predictable enough for a chained prompt), or by a human (must be scannable)? Code-consumers raise the value of S6 sharply; human-consumers raise it less.

Quick test — S6 is the right pattern when:

• output is consumed by code, a chained LLM call, or a display layer that depends on shape, and
• the format is stable enough to write down once, and
• parse-failure or shape-drift in untemplated runs is non-trivial (> 5%), and
• either no schema-constrained API is available for the call, or the output mixes structured fields with narrative.

If a schema-constrained API is available and the output is purely structured, use the JSON-mode variant rather than a free-text skeleton — same pattern, stronger mechanism. If the output is free prose for a human reader, do not template at all.

Structure

  Task ──▶ Prompt with embedded skeleton ──▶ Model
                  │                            │
                  ▼                            ▼
       fields, labels, order,           completes the skeleton:
       types, placeholders              fills content into shape
                                              │
                                              ▼
                                       Parser / next LLM step / display
                                              │
                                              ▼
                                     (optional) repair on shape failure

When the JSON-mode variant is used, the "skeleton" is a JSON Schema submitted alongside the prompt and the decoder enforces it — the parser then operates on a guaranteed-valid object.

Participants

The Skeleton and the Parser are the same artefact viewed from two ends: one defines the shape the model must produce, the other reads it. Keeping them in sync (ideally generated from one schema) is the pattern's main maintenance discipline.

Collaborations

The task is composed by binding inputs to a skeleton inside a prompt. The model receives the prompt and returns its completion. When the JSON-mode variant is active, the decoder enforces the schema at token-decode time, so the output is guaranteed parseable — the parser becomes a typed-load step. In the free-text variant there is no such guarantee: the parser validates the shape and, on failure, an optional repair step re-prompts the model with the bad output and the schema to ask for a correction. If repair fires more than rarely, the failure is in the skeleton (under-specified, ambiguous placeholders, mixed conventions) and the fix belongs upstream.

Consequences

Benefits

• Dramatically improves format consistency — the dominant lever for reliable downstream parsing.
• Makes prompt-chained pipelines (O2) feasible at all; without S6, every step has to guess the previous step's shape.
• Catches schema drift early — when the model deviates, the parser fails fast rather than poisoning a chain.
• The JSON-mode variant removes whole classes of failure (missing fields, wrong types, invalid enums).

Costs

• Tokens consumed by the skeleton on every call (free-text variant); negligible for short schemas, real for rich ones — every skeleton token participates in the O(n²) pairwise attention over the prompt (mechanism 2).
• Maintenance burden — the skeleton and the parser must stay in sync.
• The skeleton can over-constrain — the model fills a "Risk Level" field with Medium because the template demanded a value, when the right answer was "insufficient evidence".

Risks and failure modes

• Under-specified skeleton — the model fills with placeholder text ([insert title]), or invents fields the parser does not know about.
• Schema drift — the skeleton and the downstream parser diverge over time; outputs validate against the wrong shape.
• Forced field syndrome — the model produces low-confidence values to fill mandatory fields rather than admit absence; mitigate with explicit nullable / unknown enums.
• Mixed-content breakage — using JSON mode for an output that should carry narrative forces ugly escaping and degrades quality; use the free-text variant for genuinely mixed outputs.
• Repair-loop dependence — relying on the repair step to fix systematic failures rather than fixing the skeleton; hides the cost and degrades latency.

Implementation Notes

• Use the API where available. OpenAI Structured Outputs (response_format with JSON Schema), Anthropic tool-use schemas, Outlines and Guidance for self-hosted models — these are strictly better than a free-text template. Reach for the free-text variant only when the API does not support the call or the output is mixed structured + narrative.
• Provide the schema, not example JSON. When using JSON mode, give the schema directly; it is shorter and harder to misread than a worked example. The schema is the skeleton. Design the prompt + skeleton as a stable, cacheable prefix unit (mechanism 5). For calls where the schema is fixed across queries, the system prompt + skeleton qualifies for provider prefix caching (Anthropic: TTL ~5 min, min 1024 tokens, ~10% cost on cache hit). The skeleton's token cost on subsequent calls within the TTL is a tenth of the listed price. Changing the schema invalidates the cache.
• For free-text templates, label placeholders unambiguously. [TITLE] is clearer than <title> or {title}; pick one convention and use it everywhere. State explicitly that placeholders are to be replaced, not echoed.
• Allow null / unknown / n/a for fields the model may legitimately not have evidence for. Forced-field syndrome is the main quality cost of S6.
• Keep skeletons small. A long template eats context and dilutes attention — recall degrades for mid-context fields (mechanism 4); if the schema has more than ~10 fields, decompose into chained calls (compose with O2 Prompt Chaining) rather than one mega-template.
• Compose with S2 Few-Shot for rare or hard-to-describe formats. A single worked example often clarifies what three paragraphs of skeleton cannot.
• Validate on every output, even with JSON mode. Schema-constrained decoding guarantees syntactic validity, not semantic correctness — a value of the right type can still be wrong.
• Generate the skeleton and the parser from one source of truth (Pydantic, Zod, dataclass, JSON Schema file). The most common production failure is the skeleton and parser drifting independently.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S6 sits inside almost every other pattern's Generator session — it specifies the shape the generation must take. It composes naturally with O2 Prompt Chaining (each step's output shape is a template), S2 Few-Shot (examples teach the template implicitly), S4 Instruction Decomposition (a template names the output structure of the final step), and V15 LLM-as-Judge (judges read more reliably when their verdict shape is templated).

The chain:

Skeleton — the wiring only; each # LLM line is a configured session:

generate_structured(task_input, schema):
    prompt = render_prompt(task_input, schema)        # code
    if api_supports_json_mode:
        output = Model(prompt, response_format=schema) # LLM (decoder-constrained)
    else:
        output = Model(prompt)                         # LLM (free-text template)
    try:
        return parse(output, schema)                   # code
    except ShapeError as e:
        return Repair(output, schema, e)               # LLM — optional, free-text variant only

The LLM sessions:

Specialist-model note. None — a capable generalist suffices. The artefact that does the work is the schema (in JSON mode) or the skeleton (in free-text mode), not a specialist model. The one runtime dependency that does matter is whether the provider supports schema-constrained decoding for the call you are making — if it does, use it; that is a build-time choice about which variant to deploy, not a model choice.

Open-Source Implementations

• Outlines — github.com/dottxt-ai/outlines — Python library for structured generation against JSON Schema, Pydantic models, regex, and grammars; constrains the decoder so output is guaranteed valid. Works with OpenAI, vLLM, Ollama, and local transformers.
• Instructor — github.com/567-labs/instructor — Pydantic-first structured output across OpenAI, Anthropic, Google, Groq, and Ollama; automatic validation, retry-on-failure, streaming partial objects. (Previously hosted at jxnl/instructor; current canonical home is 567-labs/instructor.)
• Guidance — github.com/guidance-ai/guidance — guidance language for constrained generation with JSON Schema, regex, grammars, and token fast-forwarding; supports Transformers, llama.cpp, OpenAI.
• OpenAI Structured Outputs — platform.openai.com/docs/guides/structured-outputs — provider-native JSON Schema enforcement via response_format; the canonical JSON-mode-variant implementation.
• Anthropic tool-use schemas — tools[].input_schema with JSON Schema in the Messages API — the equivalent JSON-mode pathway for Claude models.

Known Uses

• Production extraction pipelines — invoice, contract, and form parsers built on OpenAI Structured Outputs or Instructor, where a malformed record breaks the pipeline.
• LLM-as-Judge evaluators — V15 verdicts almost universally use a JSON-mode template (verdict, score, rationale) so the judge's output can be aggregated mechanically.
• Tool-calling agents — every function-call API is S6 in disguise: the function's input schema is the template the model fills.
• Chained-prompt pipelines (O2) — every internal handoff is templated; without S6 the chain does not survive a model upgrade.
• Karpathy Memory (K12) curators — the Curator's note schema is an S6 artefact; the structure is what makes the memory navigable.

Related Patterns

• Pairs with S2 Few-Shot — examples can teach a template implicitly when describing it explicitly is hard; the two compose cleanly (template names the shape, examples show its texture).
• Pairs with S4 Instruction Decomposition — a template is one way to specify the output structure of a multi-step prompt, where S4 specifies the process.
• Pairs with S5 Constraint Framing — the "do not invent fields, do not echo placeholders" rule is a constraint that belongs next to the template.
• Pairs with V15 LLM-as-Judge — judges need structured verdicts (verdict, score, rationale); S6 is the standard mechanism.
• Required by O2 Prompt Chaining — chained calls only survive if each step's output shape is templated; otherwise the chain breaks the first time the model rephrases.
• Required by I2 Function / Tool Call — every function schema is an S6 template enforced by the provider.
• Distinct from S1 Zero-Shot and S3 Persona — those shape what the model says; S6 shapes how it lays the answer out.
• Note on the API boundary — when a provider's structured-output API supports the call, that is the JSON-mode variant of S6, not a separate pattern. Use it in preference to a free-text skeleton; reach for the free-text variant only when the API does not support the call or the output is mixed structured + narrative.

Sources

• White et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" — Output Template / Output Customization category.
• OpenAI (2024) — "Introducing Structured Outputs in the API" and the Structured Outputs guide (platform.openai.com/docs/guides/structured-outputs).
• Anthropic — Tool use documentation, input_schema for Claude tool definitions.
• Willard & Louf (2023) — "Efficient Guided Generation for Large Language Models" (arXiv 2307.09702) — the Outlines paper; basis for schema-constrained decoding.
• Lundberg & Ribeiro — Guidance project documentation; constrained generation with grammars.
• Instructor documentation — Pydantic-first structured output across providers.

S8 — Meta-Prompt

Use the LLM itself to generate or refine the prompts it will run on, driven by an external evaluation signal, so prompt engineering becomes a measured optimisation loop rather than human guesswork.

Also Known As: Auto-Prompting, Prompt Optimisation, Self-Generated Prompts, Automatic Prompt Engineering, Recursive Meta Prompting.

Classification: Category I — Signal · a meta-level pattern — it produces the Signal-layer artefacts (system prompts, instructions, exemplars) that the other S-patterns assume a human wrote.

Intent

Replace hand-crafted prompt engineering with a measured generate-evaluate-select loop, in which an LLM proposes candidate prompts, an evaluator scores them against a task signal, and the best candidate is kept and iterated on.

Motivation

Every other Signal pattern — S1 zero-shot, S2 few-shot, S3 persona, S4 instruction decomposition, S6 output template — assumes a human sat down and wrote the prompt. That human is doing search by intuition: they try a phrasing, run a few examples, eyeball the outputs, change a word, try again. The search space is enormous, the signal is noisy, and the result rarely generalises beyond the inputs the human happened to think of. Two failure modes follow directly:

• The plateau. After a few rounds the human stops finding gains, not because the optimum is reached but because the next improvement is non-obvious — a re-ordering of clauses, a different verb, an exemplar the human did not think to include. Manual prompt engineering converges to a local maximum bounded by the engineer's imagination.
• The generalisation gap. A prompt tuned on a handful of cases overfits to those cases; production traffic exposes failures the engineer never saw. The "prompt" is really a hand-tuned regression on the test set the engineer happened to look at.

The fix is to make prompt construction a proper optimisation: define a search space (templates, instructions, exemplars), define an objective (a measurable score over a held-out set), and let a machine search. The LLM itself is the natural proposer — it knows what English sentences are well-formed and what instructions are coherent. The evaluator is a separate signal — graded examples, R17 self-consistency, V15 LLM-as-Judge, or a unit-test pass rate. The pattern is the loop that closes between them.

This is a meta-pattern. Other S-patterns shape the prompt to the task; S8 shapes the process that produces the prompt. Its forces are different: it needs an evaluation signal (without which no candidate can be ranked); it pays a curator-style call budget for every iteration; and its generated prompts can be fragile or overfit in ways human-written prompts are not. It earns its own number because no other pattern has these forces — they are all downstream consumers of the artefact S8 produces.

Variants

The variants differ in what is being optimised and how the search is run:

• APE — Automatic Prompt Engineer. Instruction-level optimisation: the LLM proposes candidate instructions; each is scored on a graded dataset; the highest-scoring is kept. A black-box random / iterative search over instruction text. (Zhou et al., 2022.)
• DSPy programs (MIPROv2, COPRO, GEPA, SIMBA). Module-level optimisation: prompts are not strings but compiled artefacts of a declarative program; the optimiser tunes instructions and few-shot demonstrations and their composition jointly, using Bayesian optimisation, coordinate ascent, or reflective LLM-driven proposal. The mature production form of the pattern. (Khattab et al., 2023.)
• Meta Prompting / Recursive Meta Prompting (RMP). A scaffold-level optimisation: a single example-agnostic meta-prompt guides the LLM to generate task-specific prompts; in the recursive variant, the LLM also refines its own meta-prompt against task feedback. (Zhang et al., 2023.)
• AutoPDL. Pattern-level optimisation: the search space is combinations of prompting patterns (ReAct, CoT, ReWOO, etc.) plus their demonstrations, expressed as PDL programs; successive halving navigates the space. Source-to-source: input and output are both runnable PDL programs. (Spiess et al., 2025.)

All four share the same core — propose, evaluate, select, iterate — and differ only in the granularity of the search space (instruction string $\to$ module $\to$ scaffold $\to$ pattern composition). They are one pattern, four points on a granularity axis.

Applicability

Use Meta-Prompt when:

• you have a measurable evaluation signal — graded examples, a verifier, an LLM judge, or unit tests — and can run it cheaply against many candidate prompts;
• the prompt must generalise across a distribution of inputs, not just please a few favourite examples;
• the production task is high-volume enough that a one-off optimisation cost amortises across many calls;
• manual prompt engineering has plateaued and you suspect non-obvious wins remain.

Do not use when:

• there is no evaluation signal — without R17 self-consistency, V15 LLM-as-Judge, or graded data you cannot rank candidates, and the pattern degenerates to "the LLM wrote a prompt, we hope it is good";
• the task is one-off or low-volume — a careful S2 / S4 / S6 prompt by a human is cheaper than the optimisation budget;
• the latency budget is real-time and the optimisation must happen per query — S8 is an offline pattern that produces a deployed prompt;
• the task definition itself is unstable — optimising a prompt against a moving target produces brittle artefacts.

Decision Criteria

S8 is right when you have an evaluation signal and a task volume that justifies an offline optimisation budget.

1. Confirm the evaluation signal. You need a function score(prompt, dataset) → number. The score can come from:

• graded examples (gold labels, BLEU / accuracy / exact-match) — strongest signal;
• R17 Self-Consistency Voting (consensus rate as proxy);
• V15 LLM-as-Judge with a stable rubric;
• a downstream verifier (unit tests, type checks, sandboxed execution).

If you cannot produce any of these, stop. S8 cannot function — pick the best prompt by hand using S2 / S4 / S6 and revisit when a signal exists.

2. Cost the optimisation budget. A typical S8 run is 20–200 candidate prompts $\times$ N evaluation cases $\times$ evaluator-call cost. Cap before starting: hours of LLM time, dollar budget, or candidate count. Pair with V9 Bounded Execution — an unbounded optimisation loop is the canonical waste. Note that the cost compounds super-linearly: the O(n²) attention computation (mechanism 2) means a candidate prompt of length p evaluated against an input of length q costs O((p+q)²) per call, not O(p+q). Verbose candidates — which the optimiser tends to generate — are penalised geometrically in the evaluation pass. Set a maximum candidate token length as a constraint on the Proposer, not just as a quality concern.

3. Estimate amortisation. Optimisation cost C, per-call savings or quality gain Δ, expected calls N. Run S8 only if C ≪ Δ × N — i.e. the deployed prompt will be used many times. Rule of thumb: N $\geq$ 10,000 calls of the optimised prompt for the budget to break even on a typical reasoning task.

4. Pick the granularity.

• Instruction text only $\to$ APE (simplest, off-the-shelf).
• Instructions + few-shot demonstrations in a multi-step program $\to$ DSPy (production-grade; the default if the system is non-trivial).
• A reusable scaffold for a family of tasks $\to$ Meta Prompting / RMP.
• A combination of prompting patterns (which Reasoning pattern to use, with what demonstrations) $\to$ AutoPDL.

5. Overfit risk. Hold out an evaluation set the optimiser never sees. Score the final prompt on it. If held-out performance is materially below the optimisation score, the candidate is overfit — discard and either expand the optimisation set or coarsen the search space.

Quick test — S8 is the right pattern when:

• an evaluation signal exists and is cheap enough to run against many candidates, and
• the deployed prompt will be called enough times to amortise the optimisation budget, and
• a held-out set can validate that the optimised prompt generalises, and
• manual prompt engineering has either plateaued or is too costly to scale to the surface area.

If no evaluation signal exists, stay manual — write the prompt with S2 / S4 / S6 / S9. If volume is low, stay manual — the optimisation budget will never amortise. If the search space is small (one or two parameters), grid-search by hand rather than building the loop. If the underlying issue is that the task is ill-defined, fix the task before optimising the prompt.

Structure

   Task description + graded dataset + scoring function
                       │
                       ▼
   ┌──────────────── Proposer (LLM) ──────────────┐
   │  emits K candidate prompts (instructions,    │
   │  exemplars, scaffolds, or pattern combos)    │
   └──────────────────────┬───────────────────────┘
                          │
                          ▼
                  Evaluator (per candidate)
                  ─ run candidate against eval set
                  ─ score via labels / R17 / V15 / verifier
                          │
                          ▼
                  Selector — keep top-k, optionally
                  refine via LLM critique of failures
                          │
                          ▼
              ┌───── more rounds? ─────┐
             yes                       no
              │                         │
              ▼                         ▼
   feed top-k back to Proposer    Held-out validation
                                         │
                                         ▼
                                  Deployed prompt

Participants

Six narrow responsibilities. The pattern's central reliability move is the Proposer–Evaluator separation: the Proposer generates, the Evaluator scores, and neither can do both. Without that separation the loop reduces to "ask the LLM if its own prompt is good", which is the failure mode the pattern was invented to avoid.

Collaborations

A task spec arrives: a description, a graded dataset, and a scoring function. The Proposer reads the spec and emits K candidate prompts (initially just from the description; in later rounds, conditioned on the previous round's top performers and the cases they failed). The Evaluator runs each candidate against the eval set and produces a score. The Selector keeps the top-k and discards the rest. If the optimisation budget allows another round and the score is still climbing, the top-k feed back into the Proposer along with their failure cases — the Proposer's next candidates are informed by what did and did not work. When the budget is exhausted or the score plateaus, the best candidate is sent to the Held-out validator. If it passes, that prompt is deployed; if it fails, the candidate is overfit and either the optimisation set is expanded or the search space coarsened. The whole loop is bounded by V9.

Consequences

Benefits

• Finds prompt structures human engineers do not — re-orderings, exemplar choices, scaffolds beyond intuition.
• Produces a measured, defensible artefact: the score on the held-out set is the prompt's spec sheet.
• Scales to surface areas (many tasks, many sub-prompts, many model versions) where human prompt engineering does not.
• The optimised prompt is portable across model versions if re-run on each — keeping pace with model upgrades becomes a process, not an emergency.

Costs

• Optimisation budget: 20–200 candidates $\times$ eval-set size $\times$ evaluator-call cost per round.
• Evaluation infrastructure is mandatory — graded data, R17, V15, or a verifier; building this often dominates the project.
• Generated prompts are typically verbose; readability and brand voice are easily sacrificed to score.
• Re-optimisation needed on model upgrades, task drift, or evaluation-rubric changes.

Risks and failure modes

• No-signal collapse. Run without a real evaluation signal, the loop selects on noise — outputs look optimised but generalise no better than random.
• Overfit prompts. The optimiser memorises the evaluation set; held-out performance is materially worse.
• Evaluator drift. If the Evaluator is an LLM judge whose rubric drifts mid-run, scores from different rounds are not comparable and the "best" candidate is illusory.
• Reward hacking. The Proposer discovers prompt patterns that score well on the evaluator without actually solving the task — e.g. prompts that exploit the judge's biases.
• Cost runaway. Without V9, hard problems trigger endless rounds of marginal improvement at material cost.
• Brittle artefacts. The final prompt may be unreadable, longer than necessary, or sensitive to model version — paying for one re-optimisation per model upgrade is the price.

Implementation Notes

• The Evaluator is the load-bearing component. Spend evaluation effort before running the loop, not after — a weak Evaluator selects weak prompts confidently.
• For tasks with gold labels, graded accuracy is the strongest signal. For open-ended tasks, V15 LLM-as-Judge with a stable, versioned rubric is the practical fallback.
• Use a different model for the Evaluator than the Proposer when possible — same-model evaluation has correlated blind spots.
• Hold out a validation set the optimiser never sees. Report final performance on that set, not the optimisation score.
• Start with the simplest variant (APE-style instruction search) before reaching for DSPy / AutoPDL. The marginal value of more sophisticated search shrinks if the eval signal is weak.
• Cap rounds and candidates explicitly (V9 Bounded Execution). Plateau detection is a useful additional stop: end the loop when the top-k score does not improve over R rounds.
• Track Proposer / Evaluator / model versions alongside the deployed prompt — a prompt optimised against GPT-X is not necessarily good on GPT-Y. Treat prompts as build artefacts with provenance. Design the optimised prompt as a stable cacheable prefix (mechanism 5). For Anthropic deployment: if the fixed portion of the system prompt exceeds 1024 tokens and remains stable across calls, it qualifies for provider prefix caching (~10% of normal input cost per hit, TTL ~5 min). The optimisation loop should evaluate candidate prompts not only on task score but on whether their stable prefix length meets the caching threshold — a lower-scoring but cache-friendly prompt may be more economical at production scale.
• For DSPy-style multi-step programs, optimise each module against its own signal; do not propagate an end-to-end score into a per-module optimiser — credit assignment becomes intractable.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S8 chains a Proposer LLM with an Evaluator LLM (or verifier) in a code-driven loop, bounded by V9 Bounded Execution. The Evaluator is typically R17 Self-Consistency Voting or V15 LLM-as-Judge — without one, the loop has no signal. The Proposer's own setup is itself Signal-layer work (S3 role, S5 constraint, S6 output template forcing a list of candidate prompts).

The chain:

Skeleton — the wiring only; each # LLM line is a configured session, not code:

meta_prompt(spec):
    top_k = []
    for round in range(max_rounds):                 # code — V9 bound
        candidates = Proposer(spec, top_k) ──────── # LLM  — K candidates
        scored = []
        for c in candidates:
            outputs = run(c, spec.eval_set)         # code
            score   = Evaluator(c, outputs) ─────── # LLM (or rule) — R17 / V15
            scored.append((c, score))
        top_k = Selector(scored, k)                 # code
        if plateau(top_k): break                    # code
    best = top_k[0]
    holdout_score = run_and_score(best, spec.holdout_set)  # code — R17/V15
    return best if holdout_score >= threshold else FAIL

The LLM sessions. Each LLM step is a configured session whose setup is loaded once, before the first call.

Specialist-model note. No fine-tune is strictly required, but two structural choices change everything. (a) The Proposer and Evaluator should be different sessions, ideally different models — shared models share blind spots, and a Proposer that learns the Evaluator's preferences is the reward-hacking failure mode. (b) The Evaluator is the load-bearing dependency — if no automated scoring function exists, S8 cannot run; building the eval (often graded data, sometimes a fine-tuned judge) is the actual cost of adopting the pattern. The DSPy variant additionally requires the program-as-code substrate: the prompts being optimised are not free text but compiled artefacts of a declarative program.

Open-Source Implementations

• DSPy — github.com/stanfordnlp/dspy — Stanford NLP's declarative LM-programming framework; optimisers include MIPROv2, COPRO, SIMBA, GEPA. The mature, production-grade form of the pattern.
• APE — Automatic Prompt Engineer — github.com/keirp/automatic_prompt_engineer — the original instruction-level prompt search; treats instructions as programs, optimises by black-box search over candidate strings against a chosen score function (Zhou et al., 2022).
• Meta Prompting — github.com/meta-prompting/meta-prompting — official implementation of "Meta Prompting for AI Systems" (arXiv 2311.11482), including the Recursive Meta Prompting variant.
• AutoPDL — github.com/IBM/prompt-declaration-language — IBM's Prompt Declaration Language with the AutoPDL optimiser (arXiv 2504.04365); source-to-source optimisation over agentic and non-agentic prompting patterns plus demonstrations.

Known Uses

• DSPy in production — Databricks, JetBlue, and other enterprise teams use DSPy to compile multi-step LLM pipelines, with the optimiser tuning instructions and few-shot exemplars per module.
• Prompt registries with auto-optimisation — platforms such as Weights & Biases Weave, LangSmith, and PromptLayer ship offline prompt-optimisation utilities built on the propose-evaluate-select loop.
• Internal eval-driven prompt CI — high-volume LLM products (search assistants, code assistants, agentic platforms) increasingly run S8-style sweeps in CI to re-tune prompts against held-out evaluation sets on each model upgrade.
• Academic benchmarks — many recent benchmark submissions report results obtained with DSPy / GEPA / MIPROv2-optimised prompts rather than hand-crafted ones.

Related Patterns

• Required by S8 itself — needs R17 Self-Consistency Voting or V15 LLM-as-Judge (or graded data, or a verifier) as the Evaluator. Without an evaluation signal the loop cannot rank candidates; this is the hard prerequisite.
• Composes with V9 Bounded Execution — the optimisation loop must be capped on rounds, candidates, and budget; otherwise marginal improvement runs without end.
• Produces the artefacts that S1–S6 and S9 describe — S8 is the process whose output is the system prompt and exemplars those patterns assume already exist.
• Sibling of R7 Reflexion — both are iterate-with-feedback loops, but operate at different levels: R7 refines an output across attempts on a single task; S8 refines a prompt across many tasks. Same loop shape; different artefact under optimisation.
• Pairs with V14 Trajectory Logging — every candidate, score, and selection should be logged; the optimisation history is the audit trail for the deployed prompt.
• Pairs with S3 Persona, S5 Constraint Framing, S6 Output Template — these structure the Proposer's own session (what kind of candidates to emit, in what format).
• Distinct from K12 Karpathy Memory — both have an LLM authoring its own artefact; K12 authors a memory store the same agent reads, S8 authors a prompt a different agent will run. Different artefact, different read pattern, different evaluation regime.
• Distinct from O5 Evaluator-Optimizer — O5 is an orchestration pattern where one agent generates and another critiques an output; S8 is the Signal-layer analogue operating on prompts. The mechanism is similar; the artefact is one level higher.

Sources

• Zhou et al. (2022) — "Large Language Models Are Human-Level Prompt Engineers" (APE; arXiv 2211.01910).
• Khattab et al. (2023) — "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines" (arXiv 2310.03714).
• Opsahl-Ong et al. (2024) — "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs" (MIPROv2; arXiv 2406.11695).
• Zhang et al. (2023) — "Meta Prompting for AI Systems" (arXiv 2311.11482).
• Suzgun & Kalai (2024) — "Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding" (arXiv 2401.12954).
• Spiess et al. (2025) — "AutoPDL: Automatic Prompt Optimization for LLM Agents" (arXiv 2504.04365).
• White et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" — Question Refinement Pattern as a precursor.

S9 — Constitutional Framing

Embed an explicit set of principles — a constitution — in the session setup, and have the model critique and revise its own output against those principles before returning it, so values and judgement live as inspectable text rather than as an implicit prior baked into weights.

Also Known As: Constitutional AI (inference-time), Principle-Based Alignment, Runtime Constitution, Self-Critique-and-Revise, CAI-at-Inference.

Classification: Category I — Signal · the setup-layer pattern that names which principles the model applies; complements S3 Persona (who the model is), S5 Constraint Framing (what it must not do), and S6 Output Template (what its output looks like). The inference-time form of Anthropic's Constitutional AI (Bai et al. 2022, training-time); the soft, in-prompt counterpart to V7 AgentSpec (hard, external policy enforcement).

Intent

Make the model's value judgement legible, auditable, and updatable — by stating the principles explicitly in the prompt and inserting a self-critique-and-revise step that checks the draft against them before it is returned.

Motivation

Every model already has implicit values: behaviours trained in through RLHF, default refusal patterns, a baseline politeness, an aversion to certain content. These are real, but they are implicit — the operator cannot inspect them, cannot point to them, cannot version them, and cannot reason about how they will apply in a context the trainer did not anticipate. When the model behaves well, the operator does not know why; when it behaves badly, the operator has no lever short of changing models.

S9 moves judgement out of the weights and into the prompt. The session setup carries a short, numbered list of principles — "acknowledge uncertainty rather than fabricate; prioritise user safety over task completion; if a request implies medical, legal, or financial advice, recommend a qualified professional" — and the per-call prompt includes a step that asks the model to draft, then critique that draft against the principles, then revise. The constitution is text: an operator can read it, edit it, version-control it, and audit any output against it. The same operator can compare two systems by comparing their constitutions, which is impossible when the values live only in weights.

This is the inference-time application of Bai et al.'s 2022 "Constitutional AI from AI Feedback." Their result was a training technique: use a constitution to generate critique-and-revision data, then fine-tune on it. S9 is the same critique-and-revise move, applied at runtime on every output, with the constitution carried in the system prompt rather than distilled into weights. The trade is the obvious one: weights are fast at inference but opaque and fixed; in-prompt is slower and probabilistic (mechanism 7) but inspectable and updatable. S9 earns its number because that trade — make values legible at the cost of an extra LLM step per output — is a distinct design move with its own forces, distinct from S3 (which names identity, not principles) and distinct from S5 (which enumerates prohibitions, not interpretive principles). Persona says who; prohibitions say what-not; the constitution says how to judge — interpretive guidance the model applies in cases the operator did not anticipate.

S9 is soft. The model applies its constitution through language reasoning: probabilistic, manipulable by adversarial input, not an enforcement boundary. That is the H/S complementarity with V7 (see Critical Conflicts in Appendix A, Critical 3): S9 broad and interpretive, V7 narrow and deterministic. They layer; they do not substitute.

Applicability

Use Constitutional Framing when:

• the system operates in a context — safety-critical, regulated, brand-sensitive, ethically charged — where outputs must be explainable against stated values, not just produced;
• the operator needs to audit outputs against principles, or to update the value framing without retraining;
• the constitution captures interpretive judgement (when to refuse vs. clarify, how to weigh helpfulness against caution) that cannot be enumerated as flat prohibitions;
• multiple agents in the system must share a consistent value framing across roles;
• you need a written, inspectable record of "what this system is supposed to believe about its work."

Do not use when:

• the requirement is a deterministic, enumerable rule ("never call send_email when the context contains classified data") — use V7 AgentSpec, which enforces the rule at runtime regardless of what the model "thinks";
• the requirement is a flat set of prohibitions with no interpretive content — use S5 Constraint Framing;
• the requirement is identity / register / voice rather than judgement — use S3 Persona;
• the principles themselves must evolve through experience with human oversight — use H5 Constitutional Self-Alignment, which extends S9 across sessions with a governed evolution loop (H5 requires V1 Human-in-the-Loop for every change);
• the cost of an extra critique-and-revise pass on every output is unacceptable and the implicit defaults of the model already cover the value space.

Decision Criteria

S9 is right when value judgement must be legible and auditable, not just present — and when the principles are interpretive enough that no flat rule list could replace them.

1. Audit requirement. Will any output ever need to be defended by pointing at the principle that produced it? Compliance reviewers, safety teams, regulators, brand owners all ask this. If yes, the constitution must exist as text — S9 applies. If no one will ever ask "why did the agent decide that way?", the implicit defaults of the model are sufficient and S9 is overhead.

2. Interpretive vs. enumerable. Can you write the requirement as a list of forbidden tool calls, forbidden content patterns, or hard data-flow rules? If yes, V7 AgentSpec is the right layer — deterministic, surviving prompt manipulation, producing an audit trail. S9 carries the spirit of rules ("treat user wellbeing as a higher priority than completing the task"); V7 carries the letter ("send_email is prohibited when context.classification == 'restricted'"). In any safety-critical system you need both — S9 for the cases V7 didn't anticipate, V7 for the cases S9 was talked out of (Appendix A, Critical 3).

3. Update cadence. How often will the value framing need to change? Constitutions are text — an operator can edit one in minutes and redeploy with no training run. If the value framing is genuinely fixed forever, the implicit defaults of the model are equivalent. If the framing must evolve quarterly (brand voice, regulatory updates, post-incident learnings), S9's editability is decisive.

4. Adversarial exposure. How exposed is the system to prompt injection or user manipulation? S9 alone is probabilistic — a sufficiently clever adversarial prompt can talk a model out of its constitution; this is a documented failure mode (jailbreaks targeting the constitution). High-exposure systems must layer V7 (deterministic) under S9 (interpretive). If exposure is low (internal tool, single-trusted-user), S9 alone may suffice; if exposure is open-internet, S9 alone is insufficient on principle.

5. Cost budget per output. The self-critique-and-revise loop adds at least one extra LLM step per output (critique) and often two (critique + revise). On a budget-sensitive surface (chat tier, high-volume backend) measure the latency and token cost; if a single capable model already produces principle-aligned output by default, the critique step buys little. The cheapest implementation uses the same model for draft, critique, and revise in one turn; the most reliable uses a separate, sometimes smaller, critic session.

Quick test — S9 is the right pattern when:

• outputs may need to be defended by pointing at a stated principle, and
• the value framing is interpretive (judgement, weighing trade-offs) rather than enumerable, and
• the value framing will need to change without retraining, and
• you can afford one extra LLM step per output for self-critique and revision.

If the requirement is enumerable, use V7 AgentSpec instead (and pair S9 + V7 in safety-critical systems). If the requirement is identity rather than judgement, use S3 Persona. If the requirement is a flat list of prohibitions, use S5 Constraint Framing (S9 provides the principles; S5 turns selected principles into hard prohibitions). If you need principles that evolve across sessions with human review, use H5 Constitutional Self-Alignment.

Structure

  Session setup (once)
       │
       ▼
   Constitution: numbered list of principles loaded into the system prompt
       │
       ▼
   ┌───────────────────────  Per-call loop  ───────────────────────┐
   │                                                                │
   │   User query                                                   │
   │       │                                                        │
   │       ▼                                                        │
   │   1. Draft        — generate a candidate response              │
   │       │                                                        │
   │       ▼                                                        │
   │   2. Critique     — check the draft against each principle     │
   │                     (same model, or a separate critic session) │
   │       │                                                        │
   │       ▼                                                        │
   │   3. Revise       — rewrite the draft to address the critique  │
   │       │                                                        │
   │       ▼                                                        │
   │   Final answer (and, optionally, the critique as audit record) │
   └────────────────────────────────────────────────────────────────┘

Participants

The Drafter / Critic / Reviser can all be the same model in three separate sessions with different setups, or three distinct model choices (often a smaller/cheaper critic). What must not collapse is the separation of responsibility — the moment one session does both draft and critique, the model finds reasons its draft was fine. There is also a mechanistic basis for separation: in a separate session, the Critic's Q-K attention computations are performed over the draft text alone, not over the reasoning tokens that generated the draft. The Drafter's generative context does not exist in the Critic's KV cache (mechanism 6, mechanism 3). This is subagent decomposition as context bounding: each agent's seq_len is bounded, the O(n²) cost is isolated, and the probability distribution the Critic samples from is not contaminated by the generative chain. A same-session self-check has the full generation in its attention horizon.

Collaborations

At session setup, the operator loads the Constitution — a short numbered list of principles — into the system prompt. This is done once; subsequent turns inherit it. When a user query arrives, the Drafter generates a candidate response in the normal way, with the constitution visible (so the draft is already aligned where possible). The Critic then receives the draft along with the constitution and produces a per-principle judgement: for each principle, does the draft honour it, and if not, what is the specific concern? The Reviser receives the draft and the critique and produces a revised answer that addresses the concerns the critique raised — not a rewrite from scratch, just the targeted fix. The revised answer is returned to the user; the critique itself is optionally persisted by the Audit Sink as a record of why the system produced what it did.

A bound on the critique-revise cycle (one pass typically, two at most) keeps cost predictable; see V9 Bounded Execution. In a system that also runs V7 AgentSpec, V7's deterministic checks run after the Reviser — V7 is the floor, S9 is the interpretive ceiling, and the V7 check catches the case where the model was talked out of its own constitution by adversarial input.

Consequences

Benefits

• Values live as inspectable, editable, version-controllable text.
• Outputs can be defended ("which principle led to that decision?") and audited against a stable artefact.
• The constitution can be updated in minutes — no retraining, no model swap, no waiting for the next foundation-model release.
• The same constitution can be shared across agents, giving a multi-agent system consistent value framing.
• The critique-revise loop catches a meaningful share of would-be policy violations the drafter would otherwise return.

Costs

• One extra LLM step per output (critique), often two (critique + revise). Tokens, latency, money.
• The constitution itself occupies prompt budget — short, terse principles matter.
• A poorly written constitution underperforms a well-trained implicit default — the operator now owns a new authoring problem.
• The pattern caches well in steady state (the constitution is stable prefix) but every constitution edit invalidates the cache. For Anthropic deployments: constitutions exceeding 1024 tokens qualify for provider prefix caching (mechanism 5) at ~10% of normal input token cost per cache hit, TTL ~5 minutes. A 10-principle constitution is typically well under 500 tokens — compose with the rest of the stable system prompt (S3, S5) to form a single cacheable prefix unit exceeding the threshold. Every constitution edit invalidates the prefix cache; batch edits at maintenance windows to preserve the caching benefit.

Risks and failure modes

• Probabilistic enforcement. The model can be talked out of its constitution by adversarial input. Calling an S9 system "aligned" without a V7 deterministic floor is overclaiming.
• Lip-service critique. The Critic, when run as the same model in the same turn as the Drafter, often produces a token-rich critique that says nothing — every principle marked "satisfied" — and the Reviser changes nothing. Separating sessions, and asking the critique to find at least one weakness, mitigates this.
• Principle conflict. "Be maximally helpful" and "refuse anything that could conceivably cause harm" pull opposite ways; the model resolves by picking one and ignoring the other, often invisibly. Constitutions must explicitly order or trade off the principles.
• Constitution bloat. Long constitutions degrade — past ~10 principles the model attends partially due to mid-context under-attendance (mechanism 4). Keep it short, terse, ordered.
• Drift across edits. Without version control on the constitution itself, a quarter of unreviewed edits leaves an unrecognisable document. Treat the constitution like code.

Implementation Notes

• Write the constitution as short, numbered, terse principles — five to ten lines, each one sentence. Long-prose constitutions degrade. Anthropic's published constitutions and the LangChain constitutional principles library are good calibration.
• Order the principles deliberately — the model attends first and last most strongly. Put the most safety-critical principle first.
• Pair S9 with S3 Persona when the persona implies authority the constitution must constrain ("you are a senior security engineer; principle 1: never claim certifications you do not hold").
• Pair S9 with S5 Constraint Framing for the subset of principles that can be turned into flat prohibitions — S5 enumerates those, S9 covers the interpretive remainder.
• Pair S9 with V7 AgentSpec in any safety-critical context — S9 for interpretation, V7 for deterministic floor. Always both.
• The Critic session should be a separate session from the Drafter, even when using the same model. Different setup, different invocation. Same-session "self-check" produces lip-service.
• Bound the critique-revise loop to one or two passes (V9 Bounded Execution) — diminishing returns past two.
• Version the constitution. A change to principle 3 should produce a constitution v1.4.0 with a changelog; outputs should be tagged with the constitution version that produced them.
• For low-latency tiers, consider running the Critic only on outputs that match a triage classifier (a tiny pre-filter LLM-as-Judge) — most outputs do not need the full loop.
• The constitution is stable prefix — it caches well across calls. Edits invalidate cache; batch edits at known maintenance windows. For Anthropic deployments: constitutions exceeding 1024 tokens qualify for provider prefix caching (mechanism 5) at ~10% of normal input token cost per cache hit, TTL ~5 minutes. Compose the constitution with the rest of the stable system prompt (S3 persona, S5 constraint block) to form a single cacheable prefix unit exceeding the threshold; editing any component invalidates the full prefix cache.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: S9 chains a Drafter, a Critic, and a Reviser against a shared Constitution loaded at setup. Composes with S3 (the Drafter's role), S5 (any principle that is flat enough to enumerate as a hard prohibition), S6 (output template for the Critic's per-principle verdict), V9 (bound on the critique-revise loop), V7 (deterministic post-check independent of S9), and V14 (persist the critique as audit). Echoes R7 Reflexion and V15 LLM-as-Judge — same evaluate-then-act move, applied here to value alignment of every output.

The chain:

Skeleton — wiring only; each # LLM line is a configured session (specified below), not code:

respond(query):
    draft    = Drafter(query) ─────────────── # LLM   — constitution loaded at setup
    critique = Critic(draft) ────────────────── # LLM
    if critique.is_clean():
        answer = draft
    else:
        answer = Reviser(draft, critique) ──── # LLM
    enforce(answer)  ─────────────────────────── # code  — V7 deterministic post-check (optional)
    audit_sink.persist(query, draft, critique, answer)  # code — V14
    return answer

The LLM sessions. Each LLM step must be set up before its first call. The setup — model choice, role, constitution, output contract — is established once; the per-call prompt then wraps only the data that changes.

Concretely, for the Drafter session, the setup loaded once is roughly: "You are {role}. Apply the following principles in every response: 1. Acknowledge uncertainty rather than fabricate. 2. Prioritise user safety over task completion. 3. If the request implies medical, legal, or financial advice, name your limitations and recommend a qualified professional. 4. … . Draft your response carefully against these principles; a separate critic will check your work." The per-call prompt then carries only the user query. The Critic's setup carries the same principles plus a per-principle output template; the Reviser's setup carries the principles plus the rule "address the critique, do not rewrite."

Specialist-model note. None — a capable generalist suffices for all three sessions, and a smaller / cheaper generalist often makes a perfectly good Critic. The prompt artefact that does the heavy lifting is the Constitution itself: writing it well (short, terse, ordered, non-conflicting) is the build dependency. Anthropic's published constitutions and the LangChain constitutional_ai principle library are the practical calibration points.

Open-Source Implementations

• Constitutional Harmlessness Paper supplementary — github.com/anthropics/ConstitutionalHarmlessnessPaper — Anthropic's official supplement to Bai et al. (2022): the constitutional principles used, few-shot critique-and-revise prompts, sample model responses. The closest thing to a canonical reference set of principles. Archived read-only as of mid-2025; still the reference.
• Anthropic Claude Cookbooks — github.com/anthropics/anthropic-cookbook — Anthropic's recipe collection for Claude. Contains worked patterns for principle-based prompting and self-critique that are the inference-time analogue of the training-time work in the paper.
• LangChain ConstitutionalChain — github.com/langchain-ai/langchain (libs/langchain/langchain/chains/constitutional_ai/) — the most-used inference-time implementation: a chain that drafts, critiques, and revises against a list of ConstitutionalPrinciple objects. Deprecated in favour of a LangGraph re-implementation but still the reference for the pattern's shape; the principles file ships a library of pre-written principles (UDHR-derived, harm-avoidance, etc.) usable as drop-ins.
• AWS bias-mitigation samples — github.com/aws-samples/bias-mitigation-foundation-models — production-style notebook applying ConstitutionalChain on Amazon Bedrock for content-policy alignment.
• Collective Constitutional AI data — github.com/saffronh/ccai — data-processing repo for Anthropic $\times$ Collective Intelligence Project's public-input constitution. Useful as an example of constitution authoring at scale.
• Constitutional AI awesome papers — github.com/minbeomkim/Constitutional-AI-awesome-papers — curated paper list for the wider CAI / ethics-guided LM literature.

Known Uses

• Anthropic Claude — Claude's RLAIF training pipeline uses a constitution; the inference-time form of the same idea is now standard practice for system-prompt construction by Claude-deploying teams.
• Enterprise content assistants — brand-voice constitutions, safety constitutions, and regulatory constitutions are routinely loaded into system prompts for customer-facing assistants; LangChain's ConstitutionalChain is a common starting point.
• Compliance-sensitive deployments — financial, healthcare, and legal-tech assistants pair an S9 constitution with V7 AgentSpec deterministic enforcement; the constitution is the legible artefact reviewers read, V7 is the enforced floor.
• Collective Constitutional AI — Anthropic $\times$ Collective Intelligence Project published a constitution derived from ~1,000 U.S. adults' input and used it for an inference-time deployment, as proof that a constitution can be democratically authored.

Related Patterns

• Composes with S3 Persona — S3 names who the model is; S9 names which principles it applies. Both load at setup. When the persona implies latitude the constitution prohibits, S9 takes precedence — state this explicitly (Appendix A S3 ~ S9).
• Composes with S5 Constraint Framing — S5 enumerates flat prohibitions; S9 provides the interpretive principles those prohibitions implement. S5 is the subset of S9's principles that can be turned into a hard "do not" list. Use both: S9 for spirit, S5 for letter.
• Composes with S6 Output Template — the Critic's per-principle verdict is an S6 structured output (per-principle PASS / CONCERN + rationale).
• Composes with V9 Bounded Execution — the critique-revise loop must be capped; one or two passes is standard.
• Hard/Soft complement of V7 AgentSpec — the critical pairing. S9 is soft, broad, in-prompt (probabilistic, can be manipulated by adversarial input); V7 is hard, specific, external (deterministic, audit-trailed, survives prompt manipulation). They are not alternatives — they layer. In safety-critical systems, both are mandatory: S9 catches the cases V7 did not enumerate; V7 catches the cases S9 was talked out of. Calling an S9-only system "aligned" is overclaiming. See Appendix A, Critical 3.
• Extended by H5 Constitutional Self-Alignment — H5 lets the constitution evolve across sessions through experience, with mandatory human review at every change (H5 $\to$ V1, no exceptions). H5 evolves principles; S9 applies them. A system with no need to evolve its values uses S9; a long-running system in an evolving domain pairs S9 (apply) with H5 (evolve, governed).
• Shares the evaluate-then-act mechanism with R7 Reflexion and V15 LLM-as-Judge — same draft / critique / revise move, applied here to values rather than to task quality. The patterns are distinct because the critique target is different (principles vs. correctness vs. rubric), but the implementation skeleton is the same.
• Distinct from S3 Persona — identity is not principles. A persona implies a knowledge cluster and a register; a constitution states judgements. Operators conflate them at their cost: a persona without a constitution can have wrong values delivered with confidence; a constitution without a persona has right values delivered with no register.
• Distinct from S5 Constraint Framing — prohibitions are not principles. S5 says do not do X; S9 says here is how to judge whether X-shaped things are appropriate. The constitution generates the prohibition list; the prohibitions do not generate the constitution.

Sources

• Bai et al. (2022) — "Constitutional AI: Harmlessness from AI Feedback" (Anthropic). The foundational paper; established the training-time form of the critique-and-revise loop against a constitution. arXiv 2212.08073.
• Anthropic (2023–2024) — Claude system-prompt practice and published values documents; the inference-time application of the same idea.
• Huang, S. et al. / Anthropic & Collective Intelligence Project (2024) — "Collective Constitutional AI: Aligning a Language Model with Public Input" (arXiv 2406.07814). The democratically-authored constitution case study.
• LangChain documentation — ConstitutionalChain and the constitutional_ai principles library; the most widely adopted inference-time implementation in the OSS ecosystem.
• White et al. (2023) — "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" — the prompt-pattern context in which Constitutional Framing sits at the Signal layer.

Signal Pattern Selection

Decision Flow

Start with S1 (Zero-Shot). Upgrade only when you can measure the gap.

Is format control or style matching the core problem?
  → S2 (Few-Shot): static examples if possible; dynamic only if required
    ⚠ Dynamic S2 breaks prefix cache for all upstream stable patterns

Does the task need domain expertise framing or a specific tone?
  → S3 (Persona): bundle with S5/S6/S9 in a single stable system prompt

Are there specific behaviours the model must never exhibit?
  → S5 (Constraint Framing): explicit prohibition list alongside task description

Does a downstream system need consistent structured output?
  → S6 (Output Template): output skeleton in system prompt

Does the task have multiple steps where order matters?
  → S4 (Instruction Decomposition): numbered steps in the instruction

Do values or principles need runtime enforcement?
  → S9 (Constitutional Framing): self-critique loop against explicit principles

Does the prompt itself need to be optimised automatically?
  → S8 (Meta-Prompt): requires V15 (LLM-as-Judge) or R17 as evaluator
    ⚠ Measure cost before using; much more expensive than S1–S6/S9

Caching Guide

S3, S5, S6, and S9 are setup-band patterns. Bundle them together in a single stable system prompt — this is the cacheable prefix unit. Provider prefix caching (Anthropic: ~5 min TTL, ~10% cost on cache hits) reduces the cost of this bundle to near-zero for all calls within the TTL window.

Category II — Knowledge Patterns

A Knowledge pattern is a design pattern for supplying a language model with information it does not hold in its weights, curated to suit the task at hand. Knowledge patterns separate what the model reasons over from what the model was trained on.

Usage

A language model's trained knowledge is fixed, generic, and opaque: frozen at the training cutoff, holding no proprietary or task-specific information, and unable to cite its own sources. Relying on weights alone produces answers that are stale, ungrounded, and unauditable, and the only way to change what such a model knows is to retrain it.

Knowledge patterns remove that rigidity. They insert a curation step between an information source and the model, so that what the model sees can be selected, updated, compressed, and cited without touching the weights. This is the shift the field named the move from prompt engineering to context engineering, and it is what separates Category II from Category I. Apply a Knowledge pattern whenever:

• the task needs current, proprietary, or private information;
• answers must be grounded in, and traceable to, specific sources;
• a task or conversation runs long enough to strain the context window;
• an agent must carry knowledge across turns or across sessions.

Forces

Every Knowledge pattern resolves the same three forces in tension. A pattern is the right choice for a situation when it balances them in the way that situation demands.

1. The context window is finite and not free. Cost rises linearly with tokens; quality falls non-linearly as they accumulate (the "lost in the middle" effect, where a fact buried in a long context is a fact poorly used). The geometric explanation: U-shaped recall is a consequence of how Q-K inner products distribute over sequence positions — the model's learned projection matrices exhibit recency bias and start-of-context anchoring from training (mechanism 4). The context window is not neutral storage: it is an O(n²) compute surface (mechanism 2) with a non-uniform positional quality distribution. Curation is the process of minimising n and placing the highest-signal content at positions the learned attention metric attends to most reliably. You cannot simply put everything in.

2. The relevant information lives outside the model. It is in a corpus, a database, the last forty turns of conversation, or something the agent did three sessions ago. It must be brought in, and brought in selectively.

3. The model cannot be trusted to know what it does not know. Left alone it answers confidently from stale weights. Some patterns therefore make retrieval conditional, corrective, or self-critiqued rather than automatic. 'Grounding' is an architectural property, not a prompting outcome: weights-only generation is stochastic sampling from a learned distribution (mechanism 7) and inherently cannot cite sources, because there is no architectural mechanism to attribute a sampled token to a training document. Knowledge patterns are how grounding is achieved.

A Knowledge pattern is, in each case, a disciplined answer to one question: how to get the right information into a limited window, at the right time, and keep it coherent for the length of the task.

Structure

All Knowledge patterns share one skeleton. They interpose a curation stage between a source of information and the model's context window:

  Source ────▶ Curation ────▶ Context Window ────▶ LLM ────▶ Response
 (corpus,      (select,        (the working set
  history,      retrieve,       the model
  prior         compress,       reasons over)
  sessions)     filter)

Patterns differ in what the source is — an external corpus, the running conversation, a persistent memory store — and in what the curation stage does — retrieve by similarity, traverse a graph, summarise, prune, recall an episode. The three bands below group the patterns by the question they answer: how to bring external knowledge in (II-A), how to curate the live window (II-B), and how to persist knowledge beyond it (II-C). They are orthogonal concerns rather than alternatives: a production system typically instantiates a pattern from each band at once, which is why the bands are axes to span, not a menu to choose from.

The Four Data Shapes — Matching Retrieval to Information Type

A recurring error in agent design is applying a single retrieval primitive across all data types. The attention bilinear form (mechanism 1) captures distributional semantic similarity — effective for prose, but structurally wrong for three other shapes. Choosing the wrong shape primitive produces systematic retrieval failure regardless of retrieval quality, because better embeddings still cannot represent document hierarchy, table semantics, or graph edges.

Most production agent workflows need more than one shape. This is the correct diagnosis — not a complexity failure. The error is assuming one primitive covers all shapes. See K13 Retrieval Bundle for the design-time specification process that maps each required field to its correct shape primitive.

Context rot is the failure mode produced by mixing shapes incorrectly — or by loading mixed-authority, mixed-freshness, inferred-alongside-confirmed content into a single context window. The model cannot distinguish which sources are authoritative, treats stale alongside current as equal, blends sources it should cite separately, and gives wrong emphasis to facts that are present but not reliably attended to (mechanism 4). A larger context window does not fix context rot — it compounds it (mechanism 2: O(n²) attention cost with M4 positional under-attendance). The goal is appropriate context assembled from the correct shapes, not maximum context from a single search.

Examples

II-A — Retrieval. Bringing external knowledge into context.

• K1 Vanilla RAG — retrieve top-k semantically similar chunks at query time.
• K2 Query Transformation — rewrite, expand, or decompose the query before retrieval (HyDE, multi-query, step-back).
• K3 GraphRAG — index the corpus as an entity-relationship graph for multi-hop and global-synthesis queries.
• K4 RAPTOR — index the corpus as a recursive summary tree; retrieve at the abstraction level the query needs.
• K5 Adaptive RAG — wrap retrieval in an evaluate-and-control loop (Self-RAG and Corrective RAG are variants).

II-B — Context-window management. Curating the finite window during a task.

• K6 Context Compression — summarise context that no longer fits (lossy).
• K7 Context Pruning — remove spent or irrelevant spans without summarising (lossless).
• K8 Working Memory / Scratchpad — an explicit in-context space the model writes to itself.
• K9 Long Context — hold the whole working set in a large window instead of retrieving.

II-C — Memory. Persisting knowledge beyond the live window. The model's weights do not change between sessions — all persistence is file retrieval, not model learning (mechanism 10). This is the single most important mechanical fact about this band: no capability accrues in the model; improvement is entirely in the quality of what is retrieved and injected into context.

• K10 Long-Term Memory — an external store of flat fact-shaped items, retrieved by similarity (episodic, semantic, procedural variants).
• K11 Observational Memory — the raw activity record as primary memory; cache-friendly; the Karpathy framing's raw-log branch.
• K12 Karpathy Memory — the LLM curates structured, dense notes the agent reads; the Karpathy framing's curated-notes branch.
• K13 Retrieval Bundle — before writing retrieval code, specify the exact operational context bundle a workflow type always needs — by field, by data shape, by source authority, by freshness — then build assembly to deliver it reliably. Addresses the rediscovery problem (agents re-fetching and re-assembling the same context every run, consuming up to 85% of agent compute on re-discovery rather than task execution).

See also

• Category I — Signal patterns — shape what you say to the model; Knowledge shapes what it sees.
• Category III — Reasoning patterns — govern what the model does with the context that Knowledge assembles.
• Category IV — Orchestration patterns — Agent Isolation (delegating a sub-task to a fresh, clean context) was formerly classified here as a Knowledge pattern; its mechanism is sub-agent delegation, so it now lives with the Orchestration patterns. Mechanistically, subagent decomposition in Orchestration bounds the n² context cost per agent (mechanism 6) and is a complementary architectural axis to the within-context management patterns (K6/K7/K9): the choice is between managing one large context or decomposing into bounded sub-contexts.
• Category V — Reliability patterns — V11 Error Compaction and the evaluation patterns intersect with context curation.

The reframing of this category as "context engineering" follows Tobi Lütke and Andrej Karpathy (June 2025) and Gartner (July 2025).

Quick Reference

II-A — Retrieval

II-B — Context-Window Management

II-C — Memory

K1 — Vanilla RAG

Retrieve the documents most relevant to a query from an external corpus and inject them into the context window, so the model answers from supplied evidence rather than from its trained weights alone.

Full entry: K1-Vanilla-RAG.md

K2 — Query Transformation

Rewrite, expand, or decompose the user's raw query into derived queries chosen to retrieve better, before retrieval runs. HyDE, query rewriting, multi-query, and step-back query are variants.

Full entry: K2-Query-Transformation.md — was "K2 HyDE"; HyDE is a variant, not a fundamental pattern.

K3 — GraphRAG

Index the corpus offline as a graph of entities and relationships; answer multi-hop and global-synthesis queries by traversing that graph rather than retrieving isolated chunks.

Full entry: K3-GraphRAG.md

K4 — RAPTOR

Index the corpus offline as a tree of recursively-built summaries; retrieve from whichever level of abstraction the query needs — a leaf fact, a section summary, or a document-level synthesis.

Full entry: K4-RAPTOR.md

K5 — Adaptive RAG

Wrap retrieval in an evaluation-and-control loop: decide whether retrieval is needed, judge the quality of what returns, and act on the verdict — skip, proceed, re-retrieve, or fall back. Self-RAG and Corrective RAG are variants.

Full entry: K5-Adaptive-RAG.md — merge of the former K5 Self-RAG and K6 Corrective RAG.

K6 — Context Compression

When the context window fills, replace stretches of it with shorter summaries — trading fidelity for space so the task can continue.

Full entry: K6-Context-Compression.md

K7 — Context Pruning

Identify spans of the context window that are no longer needed and remove them outright, keeping everything retained at full fidelity. The lossless counterpart of K6.

Full entry: K7-Context-Pruning.md

K8 — Working Memory / Scratchpad

Give the model an explicit, designated region of the context to write intermediate results, plans, and conclusions into, so working state persists across reasoning steps.

Full entry: K8-Working-Memory.md

K9 — Long Context

Place the entire working set of documents directly into a large context window and let the model attend over all of it, instead of retrieving a selected subset. The architectural alternative to retrieval.

Full entry: K9-Long-Context.md

K10 — Long-Term Memory

Persist knowledge in an external store that outlives the context window, and retrieve from it in later sessions, so the agent accumulates and reuses what it learns. Episodic, semantic, and procedural memory are variants.

Full entry: K10-Long-Term-Memory.md — merge of the former K10 Episodic, K11 Semantic, and K12 Procedural memory.

K11 — Observational Memory

Treat what the agent has already seen and done within the current session as its primary memory — kept stable, compact, and cache-friendly — rather than re-retrieving it from an external store. The raw-log branch of the Karpathy framing of agent memory.

Full entry: K11-Observational-Memory.md

K12 — Karpathy Memory

Have the LLM itself curate a structured, dense memory — writing, editing, merging, linking entries — so every read is of pre-digested knowledge rather than a raw observation log or a vector of isolated extractions. The curated-notes branch of the Karpathy framing; typically paired with K11.

Full entry: K12-Karpathy-Memory.md

Former K13 Agent Isolation has been reclassified to Category IV — Orchestration (O17), where its mechanism — sub-agent delegation — sits more naturally than in the retrieval and memory band.

K1 — Vanilla RAG

Retrieve the documents most relevant to a query from an external corpus, place them in the model's context window, and have the model answer from that supplied evidence rather than from its trained weights alone.

Also Known As: Naive RAG, Basic Retrieval, Classic Retrieval-Augmented Generation

Classification: Category II — Knowledge · Band II-A Retrieval strategy · base pattern of the band — K2 Query Transformation, K3 GraphRAG, K4 RAPTOR, and K5 Adaptive RAG are all refinements of this pattern.

Intent

Ground a model's response in a specific, external, updatable corpus by retrieving the passages relevant to each query at query time and injecting them into the prompt — without retraining the model.

Motivation

The problem Vanilla RAG solves is narrow and exact: you need a model to answer over a body of knowledge it was not trained on — because the knowledge is proprietary, or private, or changes faster than the model's training cycle — and you need the answer to be traceable to a source.

Three approaches present themselves first. Each fails in a way that defines what RAG must do.

1. Rely on the model's weights. The model knows only what was in its training set, up to its cutoff. Proprietary documents were never in it; last week's events are absent. And even where the model does know a fact, it cannot point to where the fact came from — so the answer cannot be audited, and cannot be trusted in any setting where being wrong is expensive.

2. Fine-tune the model on the corpus. This is expensive, slow, and must be repeated every time the corpus changes. It still yields no citations. Facts absorbed as weights blur into everything else the model knows and can be overwritten by later training (catastrophic forgetting). Fine-tuning teaches behaviour and style well; it is a costly and unreliable way to teach facts.

3. Put the whole corpus in the prompt. This works only while the corpus is small. Cost scales linearly with every token, on every call; answer quality degrades as the window fills (the "lost in the middle" effect) (mechanism 4); and most real corpora simply do not fit.

Vanilla RAG closes precisely the gap these leave. It makes the model's knowledge external, updatable, selective, and citable. The corpus lives outside the model, in an index. At query time only the handful of passages relevant to this query are brought in. Updating the system's knowledge means re-indexing, not retraining. Every answer can carry the source of each passage it drew on.

The underlying division of labour is the pattern's unique contribution: the model's parameters supply language and reasoning; the retrieved passages supply facts. Lewis et al. (2020) formalised this separation and named it.

Applicability

Use Vanilla RAG when:

• the task is question-answering over a static or slowly-changing document corpus — product documentation, policies, manuals, a knowledge base;
• answers must be grounded in, and cite, specific sources;
• the corpus is too large for the context window, but any single query needs only a small, locally-coherent slice of it;
• the knowledge changes often enough that retraining is impractical.

Do not reach for Vanilla RAG when:

• the query needs synthesis across the whole corpus or multi-hop reasoning over entity relationships — use K3 GraphRAG;
• queries vary widely in the level of abstraction they need — use K4 RAPTOR;
• the entire working corpus fits comfortably in the context window — use K9 Long Context and skip retrieval;
• many queries are answerable from the model's own knowledge, and retrieval would only inject noise — gate it with K5 Adaptive RAG;
• the agent workflow requires assembling a typed operational bundle from multiple sources — customer records from a CRM, policy from a structured document, prior history from a graph, governing metrics from a warehouse. K1 retrieves semantically similar prose; it cannot retrieve table rows, graph edges, or document sections by structure. Use K13 Retrieval Bundle to specify what the workflow needs and choose shape-appropriate primitives per field, of which K1 may be one.

The rediscovery failure mode applies specifically to agents — not chatbots — and is a signal to reach for K13 upstream of K1. Agents that re-fetch the same context every run, re-summarize documents summarized last time, or ask users for information the system has, are suffering from rediscovery: the absence of a specified, pre-assembled bundle. Measured at production scale, rediscovery can consume up to 85% of agent compute (PineCone, 2025). K1 as the only retrieval layer leaves the agent responsible for assembling its own operating context dynamically, which is where rediscovery begins.

Decision Criteria

K1 is right when the task needs grounded answers from an external corpus and neither raw weights alone nor a long window fits.

1. Score the deficits. Does the task hit any of the three weights-only deficits — staleness, generic-not-proprietary knowledge, no citations? If none, you do not need K1. If any, retrieval-augmented architecture is the right frame.

2. Size the corpus against the window. Tokenize the working set (or estimate). Call it C.

• C $\leq$ ~50% of an affordable usable window $\to$ consider K9 Long Context instead; simpler architecture if you can afford the per-call cost.
• C >> any affordable window $\to$ K1 (or K3 / K4) is the only viable option.
• C in between $\to$ benchmark both K1 and K9 on your actual query workload.

3. Check the query shape. Are queries local and fact-style, answerable from a small slice of the corpus? K1 fits. Multi-hop or whole-corpus synthesis $\to$ K3 GraphRAG. Varying abstraction levels (precise facts and thematic summaries from the same corpus) $\to$ K4 RAPTOR.

4. Corpus update frequency. How often does the corpus change?

• Frequently $\to$ K1's rebuild-the-index cycle is cheap and natural; fine-tuning would be wrong.
• Stable but you still need citations $\to$ K1's auditability still wins over weights-only or fine-tuning.
• Never changes and citations do not matter $\to$ fine-tuning is at least a candidate.

5. Citation requirement. If answers must be traceable to specific sources (regulated domains, customer support, research), K1 is mandatory — weights-only and fine-tuning cannot deliver citations.

Quick test — K1 is the right base when:

• the corpus does not fit any affordable window (C >> window), and
• queries are well-defined and locally answerable, and
• citation or auditability is a requirement, and
• the corpus changes often enough that retraining is impractical.

If the working set fits an affordable window, prefer K9 Long Context. If queries are relational or global, upgrade to K3 GraphRAG. If queries span abstraction levels, K4 RAPTOR. If many queries do not need retrieval at all, or silent retrieval misses are costly, wrap K1 with K5 Adaptive RAG.

Structure

Vanilla RAG runs in two phases — an offline phase that builds the index, and an online phase that serves each query.

OFFLINE — indexing (once; refreshed when the corpus changes)

  Documents ──▶ Chunker ──▶ Embedding model ──▶ Vector index
                                                 (vectors + chunk text
                                                  + source metadata)

ONLINE — retrieval and generation (every query)

  Query ──▶ Embedding model ──▶ Similarity search ──▶ Top-k chunks
                                  (in the index)            │
                                                            ▼
            System prompt + retrieved chunks + query ──▶ Prompt assembler
                                                            │
                                                            ▼
                                                       Generator (LLM)
                                                            │
                                                            ▼
                                                  Grounded, cited answer

Participants

Collaborations

Offline. The Chunker divides each document in the Corpus into chunks. The Embedding model converts each chunk to a vector. The Vector index stores each vector alongside its chunk text and source metadata. This phase runs once and is repeated only when the Corpus changes.

Online. When a query arrives, the Embedding model converts it to a vector using the same model and space as the chunks — this shared space is the invariant the pattern depends on; if query and chunks are embedded differently, similarity is meaningless. The same-model invariant is a geometric requirement (mechanism 1). Each embedding model defines its own learned bilinear similarity surface — its own $g_{\mu\nu} = W_Q W_K^T$ (using the query/key framing of retrieval). Vectors from different embedding models live in incompatible learned spaces: a dot product between a vector from Model A and a vector from Model B measures nothing meaningful, because the bilinear forms are different. Mixing embedding models for indexing and querying — for example, indexing with text-embedding-3-large and querying with Cohere embed-v3 — is the retrieval equivalent of multiplying matrices from different coordinate systems. The result is arbitrary. This is a common practitioner error in systems that switch embedding providers mid-deployment without re-indexing the corpus. The Retriever searches the Vector index for the top-k chunks nearest the query vector. If a Reranker is present, the Retriever returns a wider candidate set and the Reranker narrows it. The Prompt assembler builds the prompt from system instructions, the retrieved chunks (with their source metadata), and the query. The Generator answers from the supplied chunks and cites them by their metadata.

Consequences

Benefits

• Grounding and attribution. Answers are anchored in supplied passages and can cite their sources, making them auditable.
• Updatable knowledge. New or changed knowledge is absorbed by re-indexing; the model is never retrained.
• Bounded cost. Only k chunks enter the prompt per call, regardless of corpus size.
• Model-agnostic and transparent. Works with any model; the retrieved set can be inspected to see exactly what the answer was based on.

Costs

• Retrieval infrastructure: an embedding pipeline and a vector store to build and operate.
• Offline indexing cost, paid again on every corpus refresh.
• Per-query latency for query embedding and similarity search.
• Chunking quality dominates output quality and is not a one-time decision.

Risks and failure modes

• Retrieval miss — the chunk that holds the answer is not in the top-k; the model then answers from weights or declines.
• Boundary split — a fact straddles two chunks, so neither is fully relevant.
• Distractor chunks — retrieved-but-irrelevant text that the model latches onto.
• Lost in the middle — even correctly retrieved chunks are used poorly when the assembled prompt is long (mechanism 4).
• False confidence — the model presents an answer as grounded when the retrieved text does not actually support it.
• Stale index — the corpus has changed but the index has not.

Implementation Notes

• Chunk size is the primary tuning lever: 256–512 tokens for precise factual lookup, 1024+ tokens where narrative coherence matters. Chunk on semantic boundaries (headings, paragraphs), not fixed character counts; overlapping chunks reduce boundary-split loss.
• Embedding model must be identical for indexing and querying. Domain-tuned embeddings outperform generic ones on specialised corpora.
• Top-k is typically 3–8. Larger k raises recall but lowers precision and consumes context.
• Hybrid retrieval — combine dense (embedding) similarity with sparse (BM25 keyword) search. This consistently beats either alone, especially for exact terms, names, and codes.
  • Why hybrid retrieval is mechanistically necessary (mechanism 1). Dense embedding retrieval computes similarity as a dot-product contraction in a learned vector space — the same bilinear structure as transformer attention (mechanism 1). This learned metric rewards distributional co-occurrence: words that appear in similar contexts are nearby in the embedding space. Exact terms, proper names, codes, and identifiers may not appear as embedding neighbors even when lexically identical, because the model learned to generalize over surface form rather than preserve it. BM25 fills exactly this gap — it is a lexical match that is immune to the distributional smoothing of embedding models. The combination is mechanistically complementary, not stylistically so.
• Reranking — retrieve a wide candidate set (e.g. 30) and rerank with a cross-encoder down to the final few (e.g. 5); this materially improves precision for modest extra latency.
• Always carry source metadata into the prompt so the Generator can cite.
• Contextual retrieval (Anthropic, 2024): prepend a short chunk-situating summary to each chunk before embedding, reducing ambiguity for chunks that lose meaning out of context.

Implementation Sketch

An LLM step is a configured session — model + setup loaded once + per-call prompt — not a bare call; code steps are deterministic wiring.

Composition: K1 is the base of band II-A. It has two chains: an offline index build and an online retrieve-then-generate. K1 is mostly deterministic — only two LLM sessions, an embedder and the final generator.

The chain:

Skeleton:

OFFLINE:
    for each chunk in corpus:
        store.add(Embed(chunk), chunk.text, source)   # code + LLM (embedder)

ONLINE:
    chunks  = store.search(Embed(query), k)           # LLM (embedder) + code
    prompt  = compose(system, chunks, query)          # code
    answer  = Generator(prompt)                        # LLM

The LLM sessions:

Specialist-model note. The Embedder is a specialist by construction — it is not a chat model. Domain-tuned embedders (financial, biomedical, code) consistently beat generic ones on specialised corpora; that is a build decision worth measuring before committing to a generic embedder for production.

Open-Source Implementations

• LangChain — github.com/langchain-ai/langchain — retrieval chains and vector-store integrations.
• LlamaIndex — github.com/run-llama/llama_index — a data framework built around RAG ingestion, indexing, and querying.
• Haystack — github.com/deepset-ai/haystack — production-oriented RAG pipelines.

Known Uses

• Perplexity AI — web-scale RAG over live search results, with inline citations.
• OpenAI ChatGPT — file uploads, retrieval over knowledge attached to custom GPTs.
• Anthropic Claude — Projects knowledge and attached-file context.
• Microsoft 365 Copilot — enterprise RAG grounded in the Microsoft Graph.
• Glean — enterprise search and assistant over internal corpora.
• Managed RAG services — Amazon Bedrock Knowledge Bases, Google Vertex AI Search, Azure AI Search.
• The default architecture for customer-support assistants and documentation Q&A across the industry.

Related Patterns

• Refined by — every other pattern in band II-A is an upgrade of K1: K2 Query Transformation (improves the retrieval key), K3 GraphRAG and K4 RAPTOR (structured offline indexes for different query classes), and K5 Adaptive RAG (wraps retrieval in a control loop with gate, quality, and recovery — Self-RAG and Corrective RAG are its variants).
• Composes with — K6 Context Compression and K7 Context Pruning (manage retrieved chunks once they crowd the window); S6 Output Template (force a citation format); V15 LLM-as-Judge and V16 Offline Eval (evaluate retrieval and answer quality); R4 ReAct (retrieval exposed as a tool the agent calls when it chooses).
• Competes with — K9 Long Context (hold the corpus in a large window instead of retrieving) and K11 Observational Memory (recall what the agent has seen rather than retrieve from a corpus). Choosing among K1, K9 and K11 is the primary architectural decision of Category II.
• Conflicts — none fundamental within the band. K1 does not conflict with K3/K4 so much as fail on the queries they handle (global synthesis, multi-hop relationship tracing); that failure is the signal to upgrade.

Sources

• Lewis et al. (2020) — "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." The pattern's formal origin.
• Liu et al. (2023) — "Lost in the Middle: How Language Models Use Long Contexts."
• Anthropic (2024) — "Introducing Contextual Retrieval."
• AWS, Google Cloud, and Azure prescriptive guidance on RAG architecture.

K2 — Query Transformation

Rewrite, expand, or decompose the user's raw query into one or more derived queries chosen to retrieve better, before any retrieval is performed.

Also Known As: Query Rewriting, Pre-Retrieval Query Optimisation. (HyDE, Rewrite-Retrieve-Read, Multi-Query, RAG-Fusion, and Step-Back Query are variants of this pattern — see Variants.)

Classification: Category II — Knowledge · Band II-A Retrieval strategy · a pre-retrieval stage that composes in front of K1 Vanilla RAG and its other refinements.

Intent

Improve retrieval quality by transforming the user's raw query into a form better matched to the corpus, in the moment between the user submitting the query and the retriever running.

Motivation

The query a user types is frequently a poor retrieval key. K1 Vanilla RAG embeds that raw query and searches with it directly — which fails in several recurring, and distinct, ways:

• Query/document space mismatch. A short question ("What's our refund window?") and the passage that answers it ("Customers may return items within 30 days of delivery for a full refund…") are written in different registers. The embedding model defines a learned bilinear similarity metric (mechanism 1) — a contraction in d_model space. Questions and answers were not co-trained as synonyms in this metric; their distributional contexts differ. A question tokens-distribution and an answer tokens-distribution sit in different regions of the learned similarity surface even when one definitively answers the other.
• Under-specified queries. "How does it handle errors?" — it is unresolved; the query carries almost nothing to match on.
• Conversational queries. In multi-turn chat the real query is spread across turns ("…and what about the enterprise tier?"). The raw final turn is not a standalone retrieval key.
• Compound queries. "Compare the refund and warranty policies" needs two different passages; a single embedding splits the difference and retrieves neither well.

These cannot be fixed downstream. You do not control how the corpus phrases its answers, and you cannot fix the embedding model without retraining it. The one available leverage point is the query itself, before retrieval runs. Query Transformation inserts exactly one stage there: it converts the raw query into one or more derived queries selected to retrieve well.

This is what distinguishes it from K1, which never touches the query. The pattern's defining claim: retrieval is only as good as its key, so generate a better key.

Variants

The variants differ only in what the Transformer produces:

• HyDE (Hypothetical Document Embeddings). The Transformer generates a hypothetical answer to the query; that answer's embedding, not the query's, drives the search — because a hypothetical answer sits in the same region of vector space as real answers. The mechanism: hypothetical answer text has the same distributional character as real answer text — it activates the same features in the embedding model's learned projection. Query text has a different distributional character (interrogative structure, brevity, pronoun density) and maps to a different region of the same learned metric. The strongest variant for query/document register mismatch. Risk: a confidently wrong hypothesis retrieves documents supporting the wrong answer. (Gao et al., 2022.)
• Query Rewriting (Rewrite-Retrieve-Read). The Transformer rephrases the raw query into a standalone, well-formed retrieval query, resolving pronouns and folding in conversational context. Essential for multi-turn systems. (Ma et al., 2023.)
• Query Expansion. The Transformer adds synonyms, related terms, and alternate phrasings, widening what the search can match. The cheapest variant; the classic information-retrieval move, predating LLMs.
• Multi-Query / Decomposition (RAG-Fusion). The Transformer emits several derived queries — sub-questions of a compound query, or paraphrases. Retrieval runs for each and the result sets are merged, usually by reciprocal rank fusion.
• Step-Back Query. The Transformer abstracts the query to a more general question, retrieves the underlying principle, then answers the specific. Shares its mechanism with the Reasoning pattern R19 Step-Back Prompting, applied here to the retrieval key.

A system may chain more than one (rewrite, then decompose).

Applicability

Use Query Transformation when:

• raw-query retrieval (K1) shows misses on questions that do have answers in the corpus;
• queries are short, ambiguous, or phrased unlike the corpus;
• the system is conversational and queries depend on prior turns;
• queries are compound and need several distinct passages.

Do not bother when:

• queries already resemble the corpus (e.g. the corpus is itself a FAQ);
• retrieval is not the measured bottleneck;
• latency budgets are tight — every variant adds at least one LLM call to the critical path.

Decision Criteria

K2 is right when K1's retrieval is failing on query-side problems — and not on corpus-side ones.

1. Measure K1 retrieval recall. On a labelled set of queries with known relevant chunks, count top-k hits. If recall is high (~90% or above), K2 has nothing useful to add. If recall is low, continue.

2. Diagnose the misses. For each missed query, ask: was the answer in the corpus but the retriever did not find it (query-side), or was the answer not in the corpus (corpus-side)?

• Query-side $\to$ K2 is the right fix.
• Corpus-side $\to$ use K5 Adaptive RAG (quality-gated fallback to web search), or expand the corpus.

3. Categorise the query-side misses. This picks the variant:

• Short queries vs long-form answers (register mismatch) $\to$ HyDE.
• Multi-turn conversational queries with unresolved references $\to$ Rewriting.
• Compound queries needing several distinct passages $\to$ Multi-Query / RAG-Fusion.
• Queries pitched too specifically for the corpus $\to$ Step-Back.

4. Latency budget check. Every transform adds at least one LLM call before retrieval. If the latency budget is sub-second, a better embedder or hybrid retrieval may beat K2 on cost-per-improvement.

5. Compose, do not replace. K2 sits in front of K1. Total cost becomes transform + N × retrieve + generate. The smallest model that gets the transform right minimises that overhead — the Transformer does not need the system's strongest model. This is mechanically correct: the transform is a classification/rephrasing task, not a reasoning task. Small models are appropriate for routing and classification (mechanism 8); using a large model here pays for capacity that the task does not require.

Quick test — K2 is the right pattern when:

• K1 retrieval recall is measurably below target on a labelled set, and
• the misses are query-side (answers exist in the corpus but are not found), and
• the latency budget tolerates one extra LLM call per query, and
• one of the four variants fits the dominant miss pattern.

If misses are corpus-side, use K5 Adaptive RAG instead. If the corpus is small enough to fit a window, K9 Long Context skips retrieval entirely. If retrieval recall is already high, K2 is overhead.

Structure

  Raw query ──▶ Query Transformer ──▶ Derived query / queries ──▶ [ K1 retrieval ] ──▶ …
                (LLM rewrites,          (better retrieval keys)
                 expands, or
                 decomposes the query)

Query Transformation is a stage, not a pipeline of its own. Everything downstream of "derived query" is unchanged K1 (or any of K3–K5). The pattern's entire substance is the Transformer and what it emits.

Participants

Collaborations

The user submits a raw query. The Query Transformer takes it — and, for the rewriting variant, the conversation history — and produces one or more derived queries according to its variant: a hypothetical document, a rewritten query, an expanded query, a set of sub-queries, or an abstracted query. Each derived query is then passed to the Retriever exactly as a raw query would be in K1. If the variant produced multiple queries, retrieval runs once per query and a merge step (typically reciprocal rank fusion) combines and deduplicates the result sets. From there the pattern hands off entirely to K1: assemble the prompt, generate, cite.

Consequences

Benefits

• Recovers retrieval hits that raw-query search misses; the single highest-leverage fix for K1 recall problems.
• Makes conversational RAG viable — without rewriting, multi-turn retrieval is unreliable.
• Multi-query variants turn compound questions, which K1 handles badly, into several questions it handles well.

Costs

• At least one extra LLM call before retrieval, on every query: added latency and token cost.
• Multi-query variants multiply retrieval cost and require a merge step.
• One more component to evaluate and monitor.

Risks and failure modes

• Hallucinated transform — HyDE's hypothetical answer is wrong, or a rewrite changes the user's intent; retrieval is then confidently aimed at the wrong place. The failure is invisible — downstream it looks like an ordinary K1 retrieval miss.
• Over-transformation — expanding or abstracting a query that was already precise dilutes it.
• Latency stacking — the transform call is pure addition to the critical path: transform, then retrieve, then generate.

Implementation Notes

• Transform with a small, fast model. The transform does not need the system's strongest model and it sits on the latency path.
• Measure first. Instrument K1 retrieval recall and confirm the misses are query-side; Query Transformation does nothing for a corpus that simply lacks the answer (that is K5 Adaptive RAG's job).
• HyDE: generate a short hypothetical answer — length adds latency without signal. Generating several and averaging their embeddings reduces the hallucination risk.
• Rewriting: pass only the relevant prior turns, not the whole history.
• Multi-query: reciprocal rank fusion is the standard robust merge; deduplicate chunks across result sets before assembly.
• The Transformer is driven by a prompt — a Signal-layer artefact. Version and evaluate it like any other prompt.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: K2 inserts a query-transformation stage in front of K1. Variants differ in what the Transformer emits; each then hands off to K1 unchanged.

The chain (multi-query variant):

The other variants share the shape and differ in step 1: HyDE emits one hypothetical answer document (step 2 embeds and retrieves on that); Rewriting emits one standalone query, resolving references against conversation history; Step-Back emits one more-abstract query.

Skeleton:

multi_query_rag(query):
    derived = Transformer(query)                   # LLM   — multi-query session
    pools = [K1.retrieve(q) for q in derived]      # code (each retrieve is K1)
    merged = reciprocal_rank_fusion(pools)         # code
    return K1.Generator(merged, query)             # LLM

The LLM sessions:

Specialist-model note. No specialist required — a tight setup on a small fast generalist is sufficient. K2's whole cost is per-query latency: every transform is an extra LLM call on the critical path, so the smallest model that gets it right wins.

Open-Source Implementations

• LangChain — github.com/langchain-ai/langchain — ships MultiQueryRetriever, a HyDE retriever, and query-rewriting chains as first-class components.
• LlamaIndex — github.com/run-llama/llama_index — query-transformation modules and sub-question query engines.
• RAG-Fusion — github.com/Raudaschl/rag-fusion — the reference implementation of the multi-query variant with reciprocal rank fusion.

Known Uses

• LangChain and LlamaIndex ship Query Transformation as first-class retrievers (MultiQueryRetriever, HyDE retriever, query-rewriting chains, sub-question engines).
• Perplexity and other answer engines rewrite and decompose queries before searching.
• Conversational enterprise assistants (e.g. Microsoft Copilot) rewrite follow-up turns into standalone queries as standard practice.
• RAG-Fusion is a widely adopted community implementation of the multi-query variant.

Related Patterns

• Refines K1 Vanilla RAG — Query Transformation is a stage placed in front of K1 and presupposes its architecture.
• Composes with K3 GraphRAG and K4 RAPTOR (a better key helps any retriever) and K5 Adaptive RAG (Query Transformation fixes query-side misses; K5's quality gate and fallback catch corpus-side misses — complementary, often paired).
• Distinct from K5 Adaptive RAG — K5 decides whether to retrieve and whether retrieval worked; K2 decides with what key to retrieve. Different questions; they compose cleanly.
• Shares mechanism with R19 Step-Back Prompting — the same abstraction move, applied to the retrieval key rather than the reasoning chain.
• Note on fundamentality — the Transformer is, in isolation, a single LLM generation step driven by a Signal-layer prompt. That is precisely why HyDE alone does not earn a pattern number: the pattern is the stage in the retrieval architecture, not the prompt inside it. The stage is fundamental; the prompt is an adaptor.

Sources

• Gao et al. (2022) — "Precise Zero-Shot Dense Retrieval without Relevance Labels" (HyDE).
• Ma et al. (2023) — "Query Rewriting for Retrieval-Augmented Large Language Models" (Rewrite-Retrieve-Read).
• Zheng et al. (2023) — "Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models."
• RAG-Fusion (community, 2023–2024); LangChain and LlamaIndex query-transformation documentation.

K3 — GraphRAG

Index the corpus offline as a graph of entities and the relationships between them; answer queries by traversing that graph or synthesising over its community summaries, rather than by retrieving isolated chunks.

Also Known As: Graph Retrieval, Entity Graph RAG, Knowledge-Graph RAG, Microsoft GraphRAG

Classification: Category II — Knowledge · Band II-A Retrieval · a structured-index pattern — an alternative offline index to K1's flat vector store.

Intent

Answer queries that require connecting information across many documents — multi-hop relationship questions and whole-corpus synthesis — by indexing the corpus as a graph of entities and relationships instead of a flat set of chunks.

Motivation

K1 Vanilla RAG retrieves the handful of chunks most similar to the query. That works when the answer sits in a few passages. It fails structurally — not for lack of tuning — on two classes of query:

• Multi-hop, relational queries. "Which suppliers does our highest-risk vendor depend on?" The answer is not contained in any single chunk; it is a path through several documents. Similarity retrieval has no notion of a path. More precisely, K1's retrieval is a nearest-neighbour search in a learned bilinear similarity space (mechanism 1). A multi-hop answer corresponds to a path through many nodes of that space, not a single point — the search has no vocabulary for paths. Retrieving more chunks does not help, because the answer is not in the chunks individually — it is in the relationships between them, which a flat index discards.
• Global synthesis queries. "What are the main themes across these 500 incident reports?" No chunk contains the answer; it is a property of the corpus as a whole. Top-k retrieval returns k chunks and is structurally blind to the other 495.

The fix is not better retrieval but a better index. GraphRAG builds one. It extracts entities and the relationships among them into a graph, detects communities of densely connected entities, and summarises each community. A query then either traverses entity relationships (multi-hop) or reads and synthesises over community summaries (global). The graph preserves exactly the structure — paths and whole-corpus organisation — that K1's flat vector store throws away. That preserved structure is GraphRAG's unique contribution.

Applicability

Use GraphRAG when:

• the corpus is large and rich in entities and relationships;
• queries trace relationships ("how is X connected to Y") or ask for corpus-wide themes;
• the domain is relational by nature — intelligence analysis, legal discovery, scientific literature, fraud and risk networks.

Do not use it when:

• queries are local factual lookups — K1 is cheaper and just as good;
• the corpus is small, or changes constantly (graph construction is expensive to repeat);
• entity extraction would be unreliable on the corpus (noisy or highly informal text).

Decision Criteria

K3 is right when queries need relationship-tracing or whole-corpus synthesis and the corpus has the entity structure to support a useful graph.

1. Sample-test K1 on the hard queries. Pick 20 real queries skewed toward multi-hop ("how does X connect to Y?") and global ("what are the main themes across this corpus?") types. Run K1. If K1 handles them, you do not need K3 — the cost of the offline graph build is wasted.

2. Score entity density. Is the corpus rich in named entities and relationships? Legal cases, scientific literature, intelligence reports, financial filings — yes. Plain narrative or unstructured prose — questionable. Without entity density, graph construction is expensive and the graph is sparse.

3. Cost the offline build. Realistic ceiling: one extraction LLM call per chunk + one summary LLM call per detected community. For tens of thousands of chunks this is minutes-to-hours of compute on a capable model. Confirm the budget before committing.

4. Update frequency. If the corpus changes daily, the rebuild cost is prohibitive — K1 is cheaper to refresh. K3 fits stable or slowly-changing corpora.

5. Hybrid, not replacement. Most production deployments run K1 alongside K3 — K1 for local lookups, K3 for graph queries. Plan the router (which queries take which path), not just the index.

Quick test — K3 is the right addition when:

• a meaningful share of real queries demand multi-hop or global synthesis K1 cannot serve, and
• the corpus is rich in entities and relationships, and
• the build cost (extractions $\times$ chunks + community summaries) is affordable on the corpus's update cycle, and
• you accept running K1 alongside K3, not just replacing K1.

If queries vary in abstraction level rather than relationship complexity, use K4 RAPTOR. If the corpus is small enough to load, K9 Long Context can give synthesis without the graph build, at higher per-call cost. If only a few outlier queries fail, add K2 Query Transformation first — it is cheaper than a full graph build.

Structure

OFFLINE — graph construction (expensive; once per corpus version)

  Corpus ──▶ Entity extraction ──▶ Relationship extraction ──▶ Entity graph
                                                                  │
                                              Community detection ◀┘
                                                      │
                                              Community summaries

ONLINE — query

  Query ──▶ Router ──┬─ local  ─▶ traverse entity neighbourhood ─▶ Generator ─▶ Answer
                     └─ global ─▶ map-reduce over community summaries ─▶ Generator ─▶ Answer

Participants

Collaborations

Offline. The Entity Extractor and Relationship Extractor run an LLM over every chunk to populate the Graph store. The Community Detector partitions the graph; the Community Summariser writes one summary per community. This phase is costly and runs once per corpus version.

Online — local. For an entity-centric query, the Router selects local search; the Traverser walks the neighbourhood of the relevant entities, gathering connected facts and the paths between them; the Generator answers from that subgraph.

Online — global. For a thematic query, the Router selects global search; the system map-reduces over community summaries — each summary contributes a partial answer, which are reduced into a whole-corpus synthesis; the Generator produces the final answer.

Consequences

Benefits

• Uniquely handles multi-hop relational queries and whole-corpus synthesis — the queries K1 cannot reach. The offline build cost is paid once and amortised across many queries; per-query cost is then only the traversal or map-reduce step. This makes the K1+K3 hybrid mechanically efficient: K1's n² online attention (mechanism 2) serves local queries cheaply, K3's pre-built structure serves relational queries without per-query extraction cost.
• Relationships are explicit and inspectable; the graph is itself a useful artefact.
• Community summaries are reusable across many global queries.

Costs

• Very expensive offline build: an LLM call per chunk for extraction, plus summarisation across all communities.
• Storage for the graph and all summary levels.
• Full rebuild cost whenever the corpus changes materially.
• Higher query latency, especially for global map-reduce.

Risks and failure modes

• Extraction error propagation — a missed or wrong entity/relationship corrupts the graph, and the error is hard to detect downstream. Graph quality caps answer quality absolutely.
• Over-engineering — applied to a corpus whose queries are mostly local, GraphRAG pays a large build cost for no gain over K1.

Implementation Notes

• Microsoft GraphRAG is the reference open-source implementation; study it before building your own.
• Use a graph community algorithm such as Leiden for community detection.
• Local search for entity-centric queries; global search (map-reduce over community summaries) for thematic ones. Routing between them is itself a design decision.
• A hybrid with K1 — GraphRAG for global/relational queries, Vanilla RAG for local lookups — is common and often the right answer; do not treat GraphRAG as a wholesale replacement.
• Extraction is the cost and quality bottleneck. Use a capable extraction model and validate the graph on a sample before trusting it.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: K3 has a heavy offline chain (extract $\to$ graph $\to$ communities $\to$ community summaries) and a routed online chain (local traversal or global map-reduce). Chains an Extractor, a Summariser, a Router, per-community generators, a Reducer, and a final Generator.

The chain:

Skeleton:

OFFLINE:
    for chunk in chunks(corpus):
        entities, edges = Extractor(chunk)        # LLM — extraction
        graph.add(entities, edges)                 # code
    for community in leiden(graph):                # code (algorithm)
        community.summary = Summariser(community) # LLM

ONLINE:
    route = Router(query)                          # LLM (or rule)
    if route == LOCAL:
        evidence = graph.neighbourhood(graph.match(query), hops=2)   # code
    else:  # GLOBAL
        partials = [PerCommunity(c.summary, query) for c in communities]  # LLM × N
        evidence = Reducer(partials, query)         # LLM
    return Generator(query, evidence)               # LLM

The LLM sessions:

Specialist-model note. The Extractor is the cost and quality bottleneck of the entire pattern; treat it as a build dependency and pick a capable model. If the Router is implemented as a trained binary classifier rather than an LLM, that classifier is a specialist with its own labelled-data requirement.

Open-Source Implementations

• Microsoft GraphRAG — github.com/microsoft/graphrag — the official reference implementation from the originating research; extraction, Leiden community detection, local and global search.
• LlamaIndex — github.com/run-llama/llama_index — property-graph index and knowledge-graph query engines.
• Neo4j GraphRAG — Neo4j's neo4j-graphrag package pairs a property-graph database with LLM extraction for production graph retrieval.

Known Uses

• Microsoft GraphRAG — the open-source reference system, from the originating research.
• Neo4j and other graph databases paired with LLM knowledge-graph extraction.
• LlamaIndex knowledge-graph indices.
• Enterprise deployments in intelligence analysis, legal e-discovery, and life-sciences literature review.

Related Patterns

• Refines K1 Vanilla RAG — an alternative offline index for queries K1 fails on; the two are routinely run side by side.
• Sibling of K4 RAPTOR — both build a structured offline index, but the structures differ in kind: K3's is a relational entity graph, K4's is a hierarchical abstraction tree. They are two patterns, not one, because they target different query classes (relationship-tracing vs abstraction-level matching).
• Composes with K2 Query Transformation (a better key helps graph queries too) and K5 Adaptive RAG (gate and quality-check graph retrieval like any other).
• Competes with K9 Long Context — for a corpus that fits a large window, the model can sometimes do its own cross-document synthesis without an explicit graph.
• Conflicts — none. K3 does not conflict with K1 so much as cover the queries on which K1 fails.

Sources

• Edge et al. (2024) — "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" (Microsoft Research).
• "RAG vs. GraphRAG: A Systematic Evaluation" (arXiv, 2025).
• Microsoft GraphRAG project documentation.

K4 — RAPTOR

Index the corpus offline as a tree of recursively-built summaries, so that retrieval can pull from whichever level of abstraction the query needs — a specific leaf fact, a section-level summary, or a document-level synthesis.

Also Known As: Recursive Abstractive Processing for Tree-Organized Retrieval, Hierarchical RAG, Summary-Tree RAG

Classification: Category II — Knowledge · Band II-A Retrieval · a structured-index pattern — an alternative offline index to K1's flat vector store.

Intent

Answer queries that vary in scope — from a precise fact to a broad theme — by indexing the corpus as a multi-level summary tree and retrieving from the level of abstraction the query requires.

Motivation

K1 Vanilla RAG retrieves chunks at a single fixed granularity: whatever the chunk size was set to. That forces an unwinnable trade-off. Small chunks answer precise factual queries well but cannot answer "what is this document about" — no single small chunk carries the gist. Large chunks carry the gist but dilute precise lookups and waste context. Any one chunk size is wrong for some of the queries the system will receive.

The deeper problem: queries arrive at different altitudes. "What dosage did the trial use?" needs a leaf fact. "How does Chapter 4 differ from Chapter 7?" needs two section-level summaries. "What is the book's central argument?" needs a root-level synthesis. A flat index has only one altitude.

RAPTOR builds an index that has all of them. It clusters the leaf chunks, summarises each cluster, clusters those summaries, summarises again, and recurses until a single root remains. The result is a tree: leaves are the original chunks, internal nodes are progressively more abstract summaries. Retrieval then matches the query to the level that fits it. The geometric reason this works: in K1's flat vector space (mechanism 1), query vectors for different altitudes of question land near embeddings of corresponding abstraction — a specific fact query is closest to leaf embeddings, a thematic query is closest to document-level summary embeddings. The RAPTOR tree populates the similarity space at every altitude, so retrieval by nearest-neighbour finds the level the query needs. The tree gives K1's missing dimension — abstraction — and that is RAPTOR's unique contribution.

This is a different problem from K3 GraphRAG. K3 preserves relationships between entities; K4 preserves levels of abstraction over content. A graph is not a tree of summaries, and a relationship query is not an abstraction-level query. They are two patterns.

Applicability

Use RAPTOR when:

• the corpus has natural hierarchical structure — books, legal codes, technical manuals, long reports;
• the query stream is diverse in scope, mixing pinpoint facts with broad thematic questions;
• a single chunk size has been observed to fail one end of that range.

Do not use it when:

• all queries are at the same altitude (just tune K1's chunk size);
• the corpus is flat and unstructured;
• the corpus changes constantly — the tree must be rebuilt.

Decision Criteria

K4 is right when the query stream spans abstraction levels K1's single chunk size cannot serve.

1. Test K1 at two chunk sizes. Run real queries at small chunks (256–512 tokens — good for precise facts) and large chunks (1024+ tokens — good for thematic). If neither size serves both ends of the stream, K4 earns its cost.

2. Profile the query mix. Sample real queries. What share need:

• Pinpoint facts (leaf nodes)?
• Section-level summaries (mid-level nodes)?
• Document-level synthesis (high-level / root nodes)?

If at least ~20% of queries fall into each band, K4's multi-level index pays off.

3. Corpus structure check. Does the corpus have natural hierarchy — books, legal codes, technical manuals, long reports? RAPTOR works much better on naturally hierarchical content than on flat heterogeneous corpora.

4. Build cost. Roughly 20–40% additional LLM summarisation calls on top of K1's chunk count, spread across tree levels. A one-off cost, but not free.

5. Update tolerance. The tree rebuilds when the corpus changes. Stable corpora (finalised reports, published codebases) suit K4; living corpora favour K1.

Quick test — K4 is the right pattern when:

• queries vary in scope across at least two abstraction levels, and
• K1 at any single chunk size fails one end of that range, and
• the corpus has natural hierarchy worth indexing, and
• the corpus is stable enough that the recursive build amortises.

If queries are relational rather than abstraction-varying, use K3 GraphRAG. If the working set is small enough, K9 Long Context synthesises across levels without a pre-built tree. If only a few queries fail, K2 Query Transformation may close the gap more cheaply.

Structure

OFFLINE — tree construction (once per corpus version)

  Leaf chunks ──▶ Cluster ──▶ Summarise each cluster ──▶ Summary nodes
        ▲                                                     │
        └──────────────── recurse until one root ─────────────┘

  Result:            Root (whole-corpus synthesis)
                    /        |        \
              Summary     Summary     Summary       (mid-level)
              /  |  \      /  |  \     /  |  \
            chunk chunk chunk ...                   (leaves = original chunks)

ONLINE — query

  Query ──▶ retrieve across tree levels ──▶ nodes at matching abstraction ──▶ Generator ──▶ Answer

Participants

Collaborations

Offline. The Clusterer groups the leaf chunks; the Summariser writes one summary node per cluster. Those summary nodes are themselves clustered and summarised, and the process recurses until a single root node remains. Every level is embedded and stored.

Online. The Retriever searches the embedded tree. Two traversal strategies exist: collapsed-tree search treats all nodes at all levels as one pool and retrieves the best matches regardless of level; tree-traversal search descends the tree level by level. Either way, a precise query surfaces leaf nodes, a broad query surfaces high-level summary nodes, and the Generator answers from whatever level was returned.

Consequences

Benefits

• Serves precise and broad queries from one index — no chunk-size compromise.
• High-level nodes give whole-document and whole-section synthesis that flat retrieval cannot produce.
• The collapsed-tree strategy is simple to implement over an existing vector store.

Costs

• Offline build cost: many LLM summarisation calls, one per cluster at every level.
• Storage for every summary level on top of the leaves.
• Rebuild required when the corpus changes.

Risks and failure modes

• Compression loss — each summarisation level discards detail; a fact present in a leaf may not survive into the summary above it, so a query that lands at the wrong level can miss it. An additional risk: LLM summarisation is stochastic (mechanism 7). Unlike a deterministic code step, the same cluster summarised twice may produce different summaries — important for reproducibility and for diagnosing index quality regressions between builds.
• Summary drift — errors in a low-level summary propagate up into every summary above it.
• Clustering quality — poor clusters produce incoherent summaries.

Implementation Notes

• The collapsed-tree retrieval strategy (search all levels as a single pool) is reported to perform well and is the simplest to build — start there.
• RAPTOR uses soft clustering (a node may belong to more than one cluster), which handles content that is relevant to several themes.
• Keep leaf chunks in the retrievable pool — RAPTOR augments flat retrieval, it does not replace the leaves.
• Summarisation prompt quality directly sets index quality; version and evaluate it.

Implementation Sketch

LLM = configured session; code = wiring.

Composition: Offline recursive build — cluster, summarise, cluster the summaries, recurse — then an online search across all tree levels as one pool (collapsed-tree retrieval). Chains K1's Embedder, a Summariser, and a Generator.

The chain:

Skeleton:

OFFLINE — build the tree:
    level = [Node(c, Embed(c)) for c in leaves]
    while len(level) > 1:
        next = []
        for cluster in soft_cluster(level):       # code
            s = Summariser(cluster)                # LLM
            next.append(Node(s, Embed(s)))         # LLM (embed)
        tree.append(next); level = next            # code

ONLINE:
    all_nodes = flatten(tree)                      # code — collapsed pool
    nodes = top_k(Embed(query), all_nodes, k=8)    # LLM (embed) + code
    return Generator(query, nodes)                  # LLM

The LLM sessions:

Specialist-model note. The Embedder is a specialist (as K1). The Summariser is the quality lever for the entire index — each level summarises the level below, so summary errors compound upward through the tree. Pick a capable model and evaluate the summaries on a sample of clusters before trusting the index.

Open-Source Implementations

• RAPTOR — github.com/parthsarthi03/raptor — the official implementation from the originating research (MIT-licensed), with a demo notebook.
• LlamaIndex — github.com/run-llama/llama_index — ships a RAPTOR pack implementing both collapsed-tree and tree-traversal retrieval.

Known Uses

• The RAPTOR reference implementation from the originating research.
• LlamaIndex ships a RAPTOR pack.
• Hierarchical-retrieval deployments over books, legal codes, and long technical documentation.

Related Patterns

• Refines K1 Vanilla RAG — an alternative offline index; RAPTOR's leaves are a K1 index, with summary levels added above.
• Sibling of K3 GraphRAG — both are structured offline indexes, but K3 indexes relationships and K4 indexes abstraction levels; they target different query classes and are distinct patterns.
• Composes with K2 Query Transformation and K5 Adaptive RAG.
• Competes with K9 Long Context — a large window lets the model synthesise across a document without a pre-built summary tree, at higher per-query cost.
• Related to K6 Context Compression — both summarise, but to opposite ends: K6 compresses live context to save space; K4 summarises offline to build an index.

Sources

• Sarthi et al. (2024) — "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval."
• LlamaIndex RAPTOR pack documentation.

K5 — Adaptive RAG

Wrap retrieval in an evaluation-and-control loop: decide whether retrieval is needed at all, judge the quality of what comes back, and act on that judgment — skip it, proceed, re-retrieve, or fall back to another source.

Also Known As: Self-Reflective RAG, Adaptive Retrieval, Agentic RAG. (Self-RAG and Corrective RAG / CRAG are variants of this pattern — see Variants.)

Classification: Category II — Knowledge · Band II-A Retrieval · a control pattern — it wraps K1, or any of K2–K4, rather than replacing it.

Intent

Make retrieval conditional and self-correcting, so the system retrieves only when retrieval helps and recovers when retrieval fails, instead of retrieving blindly on every query and trusting whatever returns.

Motivation

K1 Vanilla RAG retrieves on every query, unconditionally, and uses whatever returns, uncritically. It is a straight pipeline with no decision points. Two failure modes follow directly from that:

• Retrieval that should not happen. For a query the model can answer from its own weights — an arithmetic question, a request to summarise pasted text, a piece of general knowledge — retrieval injects irrelevant chunks that distract the model and cost tokens. K1 has no step that asks should I retrieve at all? (Weights-only answers are stochastic samples from the model's learned distribution — mechanism 7 — with no external anchor and no auditability; the Gate's DIRECT branch trades auditability for latency savings, which is appropriate only when the query is genuinely answerable from trained knowledge.)
• Retrieval that fails silently. When the corpus does not contain the answer, or the retrieved chunks are off-topic, K1 proceeds regardless: it feeds the generator poor context and produces a confident, well-formatted, wrong answer that looks grounded. K1 has no step that asks is what I got actually any good?

Both failures have the same cause — the absence of judgment — and the same fix: insert an evaluation step and a control decision that acts on it. Evaluate before retrieval ("is retrieval needed?") and after it ("is this good enough, and does my answer rest on it?"), and branch on the verdict. That evaluation-and-control loop is the pattern. It is fundamentally distinct from K1: K1 is a straight line; Adaptive RAG is a loop with branches.

Variants

The variants differ in where the judgment lives and how recovery works:

• Self-RAG. The model itself is trained to emit reflection tokens: a decision token (retrieve or not), a relevance token (is this passage relevant), and a support token (is my answer grounded in it). Evaluation is internal to the model; it typically requires a fine-tuned model, though the behaviour can be approximated with prompting. (Asai et al., 2023.)
• Corrective RAG (CRAG). A separate, lightweight evaluator scores the retrieved documents. On a low score it triggers a fallback — typically web search, sometimes query reformulation or broader retrieval. Evaluation is an external component; recovery is corpus-side. (Yan et al., 2024.)

Both are the same pattern — judge the retrieval, branch on the verdict — differing only in implementation. That shared core is why they are one pattern and not two: neither adds a structural element the other lacks; they are two ways to build the same loop.

Applicability

Use Adaptive RAG when:

• the query stream is mixed — some queries need retrieval, some are answerable from weights;
• the task is factuality-critical and a silent retrieval miss is unacceptable;
• the corpus may be stale or incomplete, so retrieval failure is a realistic event.

Do not bother when:

• every query genuinely needs retrieval and the corpus is known to be complete — the evaluation overhead then buys nothing;
• latency is so tight that no extra evaluation calls can be afforded.

Decision Criteria

K5 is right when the cost of a silent retrieval failure is high — or when many queries do not need retrieval at all.

1. Measure the bad outcomes. On a labelled test set:

• Skip-rate — what % of queries are answerable from weights alone? > 30% means the Gate saves real cost and noise.
• Silent-miss rate — what % of K1 retrievals fetch something that does not actually answer? > 5% means the Quality Evaluator catches them.
• Ungrounded-answer rate — what % of K1 answers carry unsupported claims? > 5% means the Support Evaluator catches them.

If all three are low, you do not need K5.

2. Pick a variant.

• Self-RAG — reflection tokens from a fine-tuned model; specialist build dependency; tightest integration.
• Corrective RAG — external evaluator + web-search fallback; works with off-the-shelf models; the easier deploy.

3. Cost the loop. K5 adds 1–3 LLM calls per query (gate, quality, support). Small fast models keep the overhead modest. Web-search fallback adds external cost on misses.

4. Reliability budget. Is this a task where confidently wrong is unacceptable (medical, legal, financial, safety)? Then K5 is mandatory regardless of measured miss rate — the Support Evaluator pays for itself the first time it catches a hallucination.

5. Loop-bound discipline. Pair with V9 Bounded Execution — set a hard cap on recovery rounds. Otherwise a hard query can cascade fallbacks indefinitely.

Quick test — K5 is the right pattern when:

• skip-rate, silent-miss-rate, or ungrounded-answer-rate exceeds your reliability budget, and
• evaluation latency is acceptable, and
• the cost of a silent wrong answer materially exceeds the cost of a corrective check.

If retrieval is always needed and the corpus is always sufficient, K1 alone suffices. If retrieval always fails for corpus reasons, expand the corpus — do not wrap it. If only the web-search fallback matters, the Corrective RAG variant is simpler than full Self-RAG.

Structure

  Query ──▶ [ Retrieve? ] ──no──▶ answer from weights ──────────────────────▶ Answer
                 │
                yes
                 ▼
            Retrieve ──▶ [ Quality OK? ] ──no──▶ Fallback ──┐
                              │                  (web search,│
                             yes                  reformulate,│
                              │                   re-retrieve)│
                              ▼                               │
                          Generate ◀───────────────────────────┘
                              │
                              ▼
                      [ Answer supported? ] ──no──▶ revise / retry
                              │
                             yes
                              ▼
                            Answer

Participants

Each participant owns exactly one decision and nothing else — the pattern's reliability comes from that separation of responsibility.

Six narrow responsibilities, each independently testable and swappable. The Self-RAG variant collapses the Gate and both Evaluators into the model via trained reflection tokens; the CRAG variant keeps the Quality Evaluator as an external component. Either way the six responsibilities are the same — only their packaging differs.

Collaborations

A query arrives. The Retrieval Gate decides whether retrieval is warranted; if not, the Generator answers from the model's weights and the loop ends. If retrieval proceeds, the Retriever runs and the Quality Evaluator scores the result. On a passing score, the Generator produces an answer. On a failing score, the Fallback Retriever is invoked — web search, reformulation, or broader retrieval — and its result re-enters the same Quality gate. After generation, the Support Evaluator checks that the answer rests on the retrieved context; if it does not, the answer is revised or the loop retries. A bound on the number of recovery rounds (V9 Bounded Execution) keeps the loop terminating.

Consequences

Benefits

• Avoids the noise and cost of retrieving when retrieval is not needed.
• Catches retrieval failures instead of passing them silently to the generator.
• Degrades gracefully on out-of-corpus queries — the fallback keeps the system answering.

Costs

• Evaluation adds LLM calls, a trained model, or extra components.
• Latency: each gate and evaluator sits on the critical path.
• The web-search fallback adds external cost and further latency.

Risks and failure modes

• Miscalibrated gate — skips retrieval on a query that needed it, or retrieves on one that did not.
• False-negative evaluator — rejects good retrieval, triggering needless and possibly worse fallbacks.
• Cascading fallback — one fallback fails its own quality check and triggers another, compounding cost and latency. The compounding is non-linear: each recovery round adds context (retrieved chunks, reformulated queries, tool outputs) to the session, and each subsequent LLM call pays an n² attention cost over that growing context (mechanism 2). This is the mechanistic reason V9 Bounded Execution is not optional — without a hard cap, a hard query causes a super-linear cost spiral.

Implementation Notes

• The Gate decides RETRIEVE or DIRECT — a binary classification task, not reasoning. The Quality Evaluator decides PASS or FAIL — likewise a classification. Binary classification does not require frontier model capacity (mechanism 8); a small fast model or trained classifier is mechanically correct and cuts the per-query overhead. The same applies to the Support Evaluator.
• The Self-RAG variant needs a fine-tuned model for true reflection tokens; a strong prompt can approximate it at lower fidelity.
• For the CRAG variant, set the quality threshold from measured data, not a guess — it is the pattern's main tuning lever.
• A web-search fallback should feed its results back through the normal retrieval-and-evaluation path, not straight into the generator.
• Query reformulation as a fallback move is K2 Query Transformation invoked inside the loop.
• Bound the recovery loop (V9 Bounded Execution); without a cap, a hard query can cascade fallbacks indefinitely.

Implementation Sketch

An LLM pattern is mostly abstract chaining, not runnable code. Steps marked LLM are judgment or generation: they cannot be reduced to code, and they are never bare calls — each is a configured session with a chosen model, a setup loaded once before its first execution (role, criteria, examples, reference context), and a per-call prompt that wraps the changing data. Steps marked code are the deterministic wiring the developer writes. The value of the sketch is the chain: which patterns connect, in what order, and where the LLM does the un-codeable work.

Composition: K5 wraps an inner retriever (K1, or K2–K4) in a control loop, drawing on K2 for query reformulation during recovery and V9 to bound it. The setup of each LLM session is itself Signal-layer work — a role (S3), constraints (S5), an output contract (S6).

The chain:

Skeleton — the wiring only; each # LLM line is a configured session (specified below), not code:

adaptive_rag(query):
    Gate(query) ───────────────── # LLM   → if DIRECT: answer from weights, return
    loop up to max_rounds:         # code  — V9-bounded recovery loop
        retrieve(query) ─────────── # code  — inner pattern: K1
        Quality(query, context) ─── # LLM   → PASS breaks the loop
        on FAIL → rewrite via K2, else web_search ── # code — recovery
    answer = Generator(query, context) ───────────── # LLM
    Support(answer, context) ───── # LLM   → if UNSUPPORTED: revise once

The LLM sessions. Each LLM step must be set up before its first call. The setup — model choice, role, criteria, output contract — is established once; the per-call prompt then wraps only the data that changes.

Concretely, for the Gate session: the setup loaded once is "You decide whether a query needs document retrieval. Reply RETRIEVE if it depends on specific, external, private, or current facts; reply DIRECT if a capable model can answer from general knowledge. Reply with one word." The per-call prompt then carries only "Query: {query}". The other three sessions follow the same setup-once, wrap-data-per-call split.

Specialist-model note. The two variants differ exactly here. In Self-RAG, there are no separate Gate / Quality / Support sessions — all three are one specialist model, fine-tuned to emit reflection tokens inline during generation; its setup is the fine-tuning, the judgment trained in rather than prompted. In CRAG, the Quality session is typically a small fine-tuned retrieval evaluator (a specialist), not a general model. Whenever an LLM step uses a specialist, the sketch must say so — a specialist is a build dependency, not a drop-in prompt.

Open-Source Implementations

• Self-RAG — github.com/AkariAsai/self-rag — the original implementation; reflection-token training data, fine-tuning, and inference code.
• Corrective RAG (CRAG) — github.com/HuskyInSalt/CRAG — the original implementation; retrieval-evaluator training and CRAG inference.
• LangGraph — github.com/langchain-ai/langgraph — runnable reference graphs for Adaptive RAG, Self-RAG, and CRAG in its tutorials; the closest match to the control loop shown above.
• AWS sample — github.com/aws-samples/simplified-corrective-rag — a simplified CRAG assistant on Amazon Bedrock.

Known Uses

• Perplexity and similar answer engines — gate queries and fall back to web search when the index is insufficient.
• LangGraph-based production assistants — the adaptive-RAG and CRAG reference graphs are a common production starting point.
• Enterprise RAG assistants increasingly add a retrieval-quality gate before generation as standard practice.

Related Patterns

• Wraps K1–K4 — Adaptive RAG is a control loop around an inner retriever; any retrieval pattern can be that retriever.
• Composes with K2 Query Transformation — reformulation is a natural fallback move inside the loop.
• Composes with V9 Bounded Execution — the recovery loop must be capped, or a hard query cascades fallbacks without end.
• Distinct from K2 — K2 decides with what key to retrieve; K5 decides whether to retrieve and whether it worked. Different questions; they compose.
• Shares the judge mechanism with V15 LLM-as-Judge and the Reasoning pattern Reflexion — the same evaluate-then-act move, applied here to retrieval.
• Note on fundamentality — Self-RAG and CRAG were merged into this single pattern because both are precisely "evaluate the retrieval, branch on the verdict"; they differ only in where the evaluator sits and how recovery is done. Two implementations of one pattern, not two patterns.

Sources

• Asai et al. (2023) — "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection."
• Yan et al. (2024) — "Corrective Retrieval Augmented Generation" (arXiv 2401.15884).
• LangGraph adaptive-RAG reference documentation.

K6 — Context Compression

When the context window fills, replace stretches of it with shorter summaries — trading fidelity for space so the task can continue.

Also Known As: Conversation Compression, History Summarisation, Context Summarisation, Compaction

Classification: Category II — Knowledge · Band II-B Context-window management · a subtractive in-flight curation pattern; the lossy counterpart of K7 Context Pruning.

Intent

Keep a long-running task within the context window by summarising older or bulky content into a denser form, preserving as much of its information as the reclaimed space allows.

Motivation

Any task that runs long enough — a multi-turn conversation, an agent loop, a document pass — accumulates context. The window is finite, cost is linear in tokens, and quality degrades non-linearly as the window fills. Sooner or later the accumulated context will not fit, or fits but degrades the model. Something must be removed.

The mechanistic account (mechanism 4 + mechanism 2). Quality degrades non-linearly because: (1) the $O(n^2)$ attention compute spreads probability mass over more K-vectors as $n$ grows, diluting the signal from any individual token; and (2) the learned Q-K projection matrices (mechanism 1) have a U-shaped recall bias — content placed in the middle of long contexts is geometrically accessible but statistically under-attended (Liu et al., 2024). Compression works not because it 'organises' information — the model has no concept of organisation — but because it reduces $n$, concentrating the available attention budget on fewer K-vectors, and removes mid-context content that would be under-attended anyway.

The naive removal is truncation: drop the oldest tokens. That loses their information completely, including anything still relevant. Compression is the less-lossy alternative. Instead of discarding old content, summarise it: a 4,000-token stretch of early conversation becomes a 400-token summary that keeps the gist, the decisions, the named entities. The task continues with its early context still present, in compressed form.

The pattern's defining trade is explicit and unavoidable: it spends fidelity to buy space. Compression is lossy by design — that is the mechanism, not a failure mode. Why not simply use a bigger window (K9 Long Context)? Because cost still scales with the window, and past some length quality degrades regardless of the model's nominal limit. Compression is what you do when the working set genuinely exceeds what a window can hold well.

Variants

The variants are increasing in cost and in fidelity:

• Hard truncation — drop the oldest N tokens. The degenerate baseline; included for contrast. Fast, and loses information outright.
• Sliding window — keep the most recent N tokens, drop the rest. Better, but still loses early context entirely.
• LLM summarisation — generate a dense summary of the dropped span. The core variant.
• Chain-of-Density summarisation (Adams et al., 2023 — "From Sparse to Dense", arXiv 2309.04269) — iteratively rewrite the summary to pull in missing entities at constant length. Best fidelity per token for fact-dense content, at the cost of N rounds of LLM calls per compression event. (Previously listed as a standalone Signal pattern S10; folded here after fundamentality review found it is a K6 variant, not a distinct pattern.)
• Recursive summarisation — summarise summaries as the session keeps growing.

Applicability

Use Context Compression when:

• the task is a long-running agent session — at scale this is mandatory, not optional;
• a multi-turn conversation has grown past roughly half the window;
• the agent produces bulky tool outputs (SQL results, file contents, API dumps) that accumulate.

Do not bother for short tasks that never approach the window.

Decision Criteria

K6 is right when sessions reach the context-window threshold and you cannot afford to drop content losslessly.

1. Measure session token growth. Profile real sessions for tokens-per-turn (T_avg) and max-turns (N_max). Estimated peak $\approx$ T_avg $\times$ N_max + tool outputs. If peak > ~50% of usable window, K6 is in play. If peak < 30%, you do not need it yet.

2. Set the trigger. Compression should fire before quality degrades, not when the window is full. A common setting: trigger at ~70% of nominal window.

3. Try K7 first. Before compressing (lossy), check whether content is prunable (lossless). Tool outputs that have been read, finished sub-task context, redundant intermediates $\to$ K7 Context Pruning. Always K7 first; K6 only on what cannot be pruned.

4. Compressibility check. Can older content be summarised without losing what later turns will need? Conversational sessions usually yes — decisions, facts, entities are extractable. Highly technical step-by-step work is harder — small details may matter later. Sample-test the Compactor prompt before relying on it.

5. Compactor cost. Each compression triggers an LLM call. For long sessions with many compression events, this adds up. Use a small fast model — strong models on summarisation are wasted here.

Quick test — K6 is the right pattern when:

• session length pushes peak token usage past ~50% of usable window, and
• old content can be safely summarised, and
• K7 pruning alone is insufficient, and
• summarisation cost in the loop is acceptable.

If sessions do not approach the window, K6 is overhead. If everything in context is consumed-and-done, K7 alone is lossless and cheaper. If the working set fits a much bigger window comfortably, K9 Long Context sidesteps both.

Structure

  Context window growing ──▶ [ token threshold reached ]
                                      │
                                      ▼
                          select stretch to compress
                                      │
                                      ▼
                               Summariser (LLM)
                                      │
                                      ▼
            splice summary back in place of the original span ──▶ continue

Participants

Collaborations

The Trigger monitors token usage. When it crosses the threshold, the Selector picks a span to compress — usually the oldest content, never the system prompt or the active task. The Summariser condenses that span, and the summary is spliced back into the window in place of the original. The task continues, now within budget.

Consequences

Benefits

• Keeps long-running tasks within the token budget.
• Restores attention quality by shrinking a bloated window.
• Holds cost roughly bounded as a session extends.

Costs

• A summarisation LLM call each time compression triggers.
• Information loss is certain — the only question is how much.

Risks and failure modes

• A detail compressed away resurfaces as needed later, and is gone.

Compression loss is non-deterministic (mechanism 7). LLM summarisation is stochastic sampling, not a deterministic hash. Compressing the same span twice may produce different summaries; compressing it once on a long context may omit details that would be retained on a short context. A compressed span cannot be reliably reconstructed from its summary. Unlike a deterministic compression algorithm (gzip, etc.) where the same input always produces the same output, LLM summarisation introduces sampling variance at every compression step. Systems that rely on compressed context for correctness (rather than for cost reduction) must account for this variance — either by running compression multiple times and comparing (expensive) or by designing the compression prompt to extract structured facts rather than prose summaries (more reliable but still stochastic).

• Summary errors are absorbed as "facts" the model now trusts.
• Over-compression flattens the context into uselessly generic summary.

Implementation Notes

• Compress oldest-first; keep recent turns verbatim.
• Never compress the system prompt or the active task description.
• Tool outputs are the highest-value compression target — large, and often already spent.
• Use Chain-of-Density summarisation for fact-dense content where entity coverage matters.
• Use recursive summarisation for very long sessions.
• Expect roughly 80% information preservation from good summarisation — plan for the lost 20%, do not assume 100%.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: A trigger + selector + summariser, run as maintenance after each turn. Defers to K7 (lossless prune) before lossy compression.

The chain:

Skeleton:

maybe_compress(window):
    if window.tokens < threshold: return window         # code
    span = window.oldest(keep_recent=6)                  # code
    if span.prunable: return K7.prune(window, span)      # K7 first — lossless
    summary = Compactor(span)                            # LLM
    return window.replace(span, summary)                 # code

The LLM sessions:

Specialist-model note. None — a capable generalist with the preservation contract in setup is sufficient. The Compactor's prompt is the artifact: the same preservation contract is reusable wherever conversation history must be condensed (agent loops, long chats, retrieved-context compression after K1).

Open-Source Implementations

• LLMLingua — github.com/microsoft/LLMLingua — prompt and context compression by dropping low-information tokens; a finer-grained, complementary compression mechanism.
• LangChain — github.com/langchain-ai/langchain — ConversationSummaryMemory and summary-buffer memory implement conversation-history compression directly.

Known Uses

• Agentic coding tools (e.g. Claude Code) — conversation compaction when the window fills.
• ChatGPT and other assistants — long-conversation handling.
• LangChain ConversationSummaryMemory and equivalents.
• Effectively every long-running agent framework ships some form of this.

Related Patterns

• Opposite face of K7 Context Pruning — K6 rewrites kept content (lossy), K7 removes spent content (lossless). Both are Band II-B subtractive curation; prune first, compress what cannot be pruned.
• Composes with K8 Working Memory — the scratchpad is itself compressible when it grows.
• Alternative to K9 Long Context — compress the working set, or enlarge the window to hold it.
• Shares its operation with K4 RAPTOR — both summarise; K6 compresses live context to save space, K4 summarises offline to build an index.
• Distinct from the memory patterns — K6 manages the live context window. For cross-session persistence see K10/K11/K12; K6 does not help there.
• Implements Anthropic's "Compress" context-engineering strategy.

Sources

• Anthropic context engineering framework (2025) — the "Compress" strategy.
• Adams et al. (2023) — "From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting."

K7 — Context Pruning

Identify spans of the context window that are no longer needed and remove them outright, keeping everything retained at full fidelity.

Also Known As: Selective Recall, Context Cleaning, Relevance Filtering, Tool-Result Dropping

Classification: Category II — Knowledge · Band II-B Context-window management · a subtractive in-flight curation pattern; the lossless counterpart of K6 Context Compression.

Intent

Reclaim context-window space by deleting content that has served its purpose, without summarising or altering what remains.

Motivation

Not all context bloat needs compression. Much of a full window is not compressible content — it is simply spent content. A 10,000-token SQL result that the agent read and acted on at step 3 is pure noise at step 9. A retrieved document used in an earlier sub-task. A tool error already handled. These spans are not partially relevant — they are done.

Summarising them (K6) still keeps a lossy residue, and still costs a summarisation call. The correct move for spent content is exact removal: identify the span and delete it, leaving every retained token untouched.

Where K6 trades fidelity for space, pruning gives space for free on the content that stays — it is lossless on the retained context. Its cost is elsewhere: you must know what is spent. That bookkeeping is the pattern's whole difficulty, and it is what makes pruning genuinely distinct from compression rather than a variant of it. Pruning is lossless but requires consumption tracking; compression is lossy but requires no tracking. Two different patterns, because they resolve the forces differently.

Applicability

Use Context Pruning when:

• the agent produces large tool outputs that get fully consumed — database queries, file reads, API responses;
• retrieved documents have been used and the sub-task that needed them is finished;
• errors have been handled, or intermediate outputs are now redundant.

Prefer pruning before compression — it is cheaper and lossless. It does not apply when you cannot determine what is spent; then compress instead.

Decision Criteria

K7 is right when sizeable portions of context are spent — read, used, finished — and you can track which.

1. Inventory spent content. In a typical session, what fraction of token usage is content consumed and not referenced again?

• Large tool outputs (DB results, file dumps, API responses): often dominates the budget.
• Retrieved documents from finished sub-tasks: noise after the sub-task ends.
• Handled errors, processed intermediates: spent.

If a significant share (≳ 30%) is consumed-and-done, K7 pays off.

2. Consumption tracking feasibility. Can spans be reliably marked as "consumed"?

• Tool calls: easy — wrap the tool, mark output spent after the next turn.
• Sub-task boundaries: easy if the control flow has explicit sub-tasks.
• Free-form conversation: harder — what counts as consumed?

If tracking is impractical, fall back to K6 Context Compression.

3. Trigger choice. Prune at natural boundaries (end of sub-task, threshold) — not every turn. Frequent pruning thrashes the context and invalidates cached KV states (mechanism 3 and 5). Each prune that rewrites earlier token positions forces re-computation of those K and V vectors — negating the cost benefit of provider-side prefix caching. Prune at sub-task boundaries that preserve the stable prefix; avoid pruning mid-prefix.

4. Stub vs delete. Replace bulk with a compact stub ("[tool foo: returned 412 rows, processed]") so the agent remembers the event without the bulk. Pruning to nothing loses event-level context.

5. Lossless-first principle. Always pair K7 with K6. Run K7 first (lossless), K6 only on what cannot be pruned. Never reach for K6 on content that can simply be dropped.

Quick test — K7 is the right pattern when:

• significant context is consumed-and-done (typically large tool outputs), and
• consumption can be tracked reliably (explicit sub-task or tool boundaries), and
• lossless reduction is preferable when available.

If consumption cannot be tracked, K6 is the only option. If the context never approaches the window, neither pattern is needed. For cross-session persistence, K7 does not help — that is K10 / K11 / K12 territory.

Structure

  Context window ──▶ identify spent spans ──▶ delete spans ──▶ smaller window,
                     (consumed tool outputs,                   retained content
                      finished sub-task context)               intact at full fidelity

Participants

Collaborations

As the task runs, the Consumption Tracker marks spans as consumed — a tool output once the agent has read and acted on it, a sub-task's context once the sub-task completes. At a trigger (a token threshold, or a sub-task boundary) the Pruner removes the flagged spans. Everything retained is left exactly as it was.

Consequences

Benefits

• Lossless for all retained content — no summarisation artefacts.
• Cheaper than K6 — no LLM call is involved.
• Frees space without degrading anything that stays. Mechanically: removing spent tokens reduces the length of the K vector sequence the model must attend over. In the attention softmax (mechanism 2), the retained tokens each receive a proportionally larger weight — mid-context content that was being under-attended due to the lost-in-the-middle effect (mechanism 4) becomes relatively more salient after pruning.

Costs

• Requires explicit tracking of what has been consumed — the real cost of the pattern is this bookkeeping.

Risks and failure modes

• Pruning a span that is referenced again later — the "spent" judgement was wrong.
• Aggressive pruning removes context an unforeseen later step needed.

Implementation Notes

• Tool results are the prime target — large, and usually fully consumed the moment the agent has read them.
• Prune at sub-task boundaries: when a sub-task finishes, its context becomes prunable as a block.
• Leave a compact reference in place of deleted bulk — e.g. "SQL query X returned 412 rows, processed" — so the agent still knows the event happened. Mechanically: a zero-token deletion removes the K vector for that event from the attention computation entirely — the model has no signal that something happened there. A compact stub keeps a K vector in the sequence that preserves the event-level signal, at minimal token cost (mechanism 3).
• The empirical scaffold study found selective tool-result dropping to be a distinct, common production technique — this pattern is observed practice, not theory.

Implementation Sketch

LLM = configured session; code = wiring. K7 is almost entirely code — its cost is bookkeeping, not LLM calls.

Composition: A consumption tracker plus a pruner. Runs at sub-task boundaries or on a threshold. The complementary lossy fallback is K6, which K7 defers to only when a span cannot simply be dropped.

The chain:

Skeleton:

on_tool_output(name, result, window, tracker):
    span = window.append(f"[tool {name}] {result}")     # code
    tracker.mark_after_next_turn(span)                  # code

maybe_prune(window, tracker):
    for span in tracker.consumed_spans(window):         # code
        stub = f"[{span.label}: {span.one_line} — pruned]"
        window = window.replace(span, stub)              # code
    return window

The LLM sessions: in the strict form, none — the pattern is bookkeeping plus a string substitution. An optional small generalist (a "Stub-summariser" session) can produce a one-line prose stub for spans that need more than a programmatic label:

Specialist-model note. None. K7 is the most code-heavy pattern in the category, and that is the point: the absence of LLM steps is why it is lossless on what remains, and why it is cheaper than K6.

Open-Source Implementations

• OpenProvence — github.com/hotchpotch/open_provence — an open implementation of Provence-style context pruning: a reranker-pruner that drops irrelevant sentences from retrieved context.
• LangChain — github.com/langchain-ai/langchain — the ContextualCompressionRetriever prunes retrieved documents down to their relevant spans before they reach the prompt.

Known Uses

• Production coding agents — selective tool-result dropping, observed across multiple systems in the scaffold study.
• Anthropic's "Select" context-engineering strategy.
• JetBrains and other agent context-management implementations.

Related Patterns

• Lossless counterpart of K6 Context Compression — prune first, compress what cannot be pruned. Both are Band II-B subtractive curation.
• Composes with K8 Working Memory — the scratchpad can be pruned of finished entries.
• Related to K11 Observational Memory — deciding what stays visible to the agent is the shared concern.
• Implements Anthropic's "Select" / clean-context strategy.

Sources

• "Inside the Scaffold" empirical study of production coding agents (arXiv) — selective tool-result dropping.
• Anthropic context engineering framework (2025) — the "Select" strategy.

K8 — Working Memory / Scratchpad

Give the model an explicit, designated region of the context to write intermediate results, plans, and conclusions into, so working state persists across reasoning steps instead of being regenerated or lost.

Also Known As: Scratchpad, Cognitive Scratchpad, Agent Notepad, In-Context Working Memory

Classification: Category II — Knowledge · Band II-B Context-window management · an additive in-context structure — the inverse of K6 and K7's subtractive moves.

Intent

Externalise the model's working state into a persistent, inspectable region of the context, so intermediate results survive from one step to the next within a task.

Motivation

Within a single context, the model has no working memory other than the text already present. This follows directly from two mechanical facts: (1) the model's weights do not change within a session (mechanism 10) — no information is stored by the forward pass itself; (2) the KV cache records all token computations within the current context but does not persist across API calls (mechanism 3). Anything an intermediate step established is available only if it is still present as text in the token sequence. An intermediate result computed at step 2 is available at step 5 only if it is still there as text. Without a designated place to put such results, two things go wrong:

• Regeneration. The model re-derives the same sub-result repeatedly — burning tokens, and risking that the re-derivations disagree.
• Loss. Later steps simply proceed without a fact an earlier step had established.

The fix is structural and simple: designate a region of the context — a scratchpad — and have the model write its intermediate state there. Plans, partial results, current hypotheses, a running task list. Each later step reads its own prior conclusions instead of recomputing them. The scratchpad makes working state persistent (it survives across steps), inspectable (a human or another component can read it), and singular (one authoritative copy, not re-derived variants).

This is the inverse of K6 and K7, which remove content; K8 adds a structure. It is also distinct from the memory patterns K10, K11, and K12, which persist across or through sessions — the scratchpad lives and dies within one context. Note that the ReAct reasoning loop is a scratchpad in disguise: its Thought / Action / Observation trace is working memory. K8 is the pattern that trace is one instance of.

Applicability

Use Working Memory when:

• a task has multiple steps that build on each other;
• the task involves planning and the plan needs a stable home;
• the agent runs a ReAct or similar loop where observations accumulate;
• losing an intermediate result would cause an error.

It is unnecessary for single-shot tasks.

Decision Criteria

K8 is right when a task has multiple steps that build on each other's results and that state needs to persist explicitly within the context.

1. Count dependent steps. How many steps in a typical task build on results from earlier steps?

• 1–2 steps: no scratchpad needed — prompt ordering covers it.
• 3–5 dependent steps: K8 starts to pay off.
• 5+ dependent steps: K8 is essentially mandatory.

2. Recomputation tax. Without a scratchpad, intermediate results are either regenerated (token cost, inconsistency risk) or lost (errors). Estimate how often later steps need earlier outputs — if frequently, K8 pays for itself by avoiding the regeneration tax.

3. State shape — typed or free text. Is the working state structured (a plan, a task list, a partial calculation)? Use a typed scratchpad with an explicit schema. Freeform reasoning? A delimited free-text scratchpad is fine.

4. Scratchpad growth. The pad accumulates. Estimate its peak size against the window. If it grows large, pair with K6 (compress) and K7 (prune retired entries). The reason: each scratchpad token is an additional K vector in the attention softmax (mechanism 2). As the scratchpad grows, two costs compound: (a) per-step compute rises with n² and (b) older scratchpad entries migrate toward mid-context positions that the model's learned projection matrices under-attend (mechanism 4), causing the model to ignore conclusions it wrote earlier.

5. Single-agent vs handoff. If one agent runs all steps in one context, the scratchpad is in-context (K8). If multiple agents share state, the scratchpad must be externalised — that crosses into K10 / K12 territory.

Quick test — K8 is the right pattern when:

• the task has 3+ steps where later steps need earlier results, and
• losing or recomputing intermediate state would cause errors or waste tokens, and
• one agent runs all steps in one context window, and
• the scratchpad can be kept bounded (paired with K6 / K7 if it grows).

If steps are independent, K8 is overhead. If the task is multi-agent or cross-session, you need persistent memory — K10 / K11 / K12. If the task is single-shot, no working memory is needed.

Structure

  System: [task]

  ┌─ SCRATCHPAD ───────────────────────┐
  │ Plan:           …                  │   ◀── model reads at the start of each step,
  │ Step 1 result:  …                  │       writes updated state at the end
  │ Step 2 result:  …                  │
  │ Open questions: …                  │
  └────────────────────────────────────┘

  [ current step ]

Participants

Collaborations

At each step the model reads the current scratchpad, reasons using the state it finds there, writes its updated conclusions back into the scratchpad, and proceeds. The scratchpad is therefore the single channel through which one step's output reaches the next. When it grows large, the Scratchpad Manager (or K6/K7) keeps it bounded.

Consequences

Benefits

• No recomputation of intermediate results.
• Coherent multi-step behaviour — later steps build on recorded conclusions.
• The state is inspectable and debuggable, and forms a natural audit trail.

Costs

• The scratchpad consumes window space, and grows as the task runs.
• It must be managed — K6 and K7 applied to the scratchpad itself.

Risks and failure modes

• A stale or wrong scratchpad entry misleads every later step, because the scratchpad is trusted by construction.
• Unbounded scratchpad growth eventually crowds the window.

Implementation Notes

• Delimit the scratchpad clearly (tags, a fenced block) so the model treats it as state, not prose. Mechanically, delimiters work because they create distinctive token patterns in the sequence. The model's attention heads learn to key off structural markers like tags or fenced blocks — they function as position-invariant indexing signals within the learned bilinear attention metric (mechanism 1), helping the model identify the scratchpad region regardless of where it sits in the sequence.
• Instruct an explicit protocol: read at the start of each step, write at the end.
• Cap its size; apply K6 (compress) or K7 (prune) when it grows.
• For structured tasks a typed scratchpad — an explicit task list, a plan object — outperforms free text.
• The scratchpad is the natural thing to snapshot for V-category Checkpointing.

Implementation Sketch

LLM = configured session; code = wiring.

Composition: A protocol around the model's session — read at start, write at end — applied per step. The scratchpad itself is bounded by K6/K7 as it grows, and is the natural unit to snapshot for V10 Checkpointing.

The chain (per step):

Skeleton:

run_with_scratchpad(task):
    pad = Scratchpad(plan=task.plan, done=[], open=task.questions)
    for step in task.steps:
        response = Step(pad.render(), step)             # LLM
        pad = pad.update_from(response)                  # code
        if pad.tokens > LIMIT: pad = pad.compress()      # K6 applied to the pad
    return pad.final_answer()

The LLM sessions:

Specialist-model note. None — the pattern is a protocol, not a special model. Any model that will follow the read-then-update-pad instruction will do. A typed scratchpad (a structured task object with an explicit schema) outperforms free text by collapsing the parsing failure surface.

Open-Source Implementations

• LangGraph — github.com/langchain-ai/langgraph — the graph State object is an explicit, typed working memory threaded between steps; the canonical modern scratchpad.
• Agent Memory Techniques — github.com/NirDiamant/Agent_Memory_Techniques — runnable notebooks covering working-memory and scratchpad patterns alongside the long-term variants.

Known Uses

• ReAct-based agents — the Thought/Action/Observation reasoning trace.
• Planning behaviours in Claude and ChatGPT — explicit plan/todo state.
• Agent frameworks that maintain a visible "plan" or "todo" structure.
• "Scratchpad" prompting, present since the earliest chain-of-thought work.

Related Patterns

• Distinct from K6 Context Compression and K7 Context Pruning — additive structure versus subtractive curation; but K6/K7 are applied to the scratchpad when it grows.
• Distinct from K10 Long-Term Memory, K11 Observational Memory, and K12 Karpathy Memory — in-context working state versus the three forms of persistence: cross-session flat facts (K10), in-session raw log (K11), and LLM-curated notes (K12).
• Underlies R4 ReAct and R3 Plan-and-Solve — their traces and plans are scratchpads.
• Feeds V10 Checkpointing — the scratchpad is the state worth snapshotting.

Sources

• Lilian Weng (2023) — short-term memory via in-context state.
• "Empowering Working Memory for LLM Agents" (arXiv).

K9 — Long Context

Place the entire working set of documents directly into a large context window and let the model attend over all of it, instead of retrieving a selected subset.

Also Known As: Context Stuffing, No-RAG, Full-Context Prompting

Classification: Category II — Knowledge · Band II-B Context-window management · the architectural alternative to retrieval (Band II-A).

Intent

Make the model's full working knowledge available by loading it all into the context window, trading per-call token cost for the elimination of a retrieval system.

Motivation

Band II-A exists to solve one problem: the corpus does not fit in the context window, so a subset must be selected. That premise has weakened. Model context windows have grown from a few thousand tokens to hundreds of thousands and beyond. For a large and growing class of tasks the entire relevant working set — a contract, a codebase module, a research dossier, a day of a user's documents — now simply fits. When it fits, retrieval is no longer mandatory. It is a choice, and often the wrong one.

RAG's costs are real and recurring: a chunking strategy to tune, an embedding pipeline to run, a vector store to operate, and an irreducible retrieval-miss rate — the passage that holds the answer is sometimes just not in the top-k. Long Context pays none of these. Everything goes in. There is no chunk boundary to split a fact, no retrieval miss, no embedding infrastructure. The model sees the whole working set and synthesises across all of it freely — including the cross-document connections K1 cannot reach and K3 needs a graph to reach.

The cost is the window itself. Tokens are paid on every call, and at long context lengths quality still degrades — the lost-in-the-middle effect persists even when the model's nominal limit is far higher (mechanism 4). And the working set must genuinely fit; Long Context does not scale to millions of documents.

So Long Context is not the absence of a pattern. It is a deliberate architectural choice with its own forces: take it when the working set fits a window you can afford, and when avoiding retrieval infrastructure and retrieval misses is worth the per-call token cost. The choice between K1 and K9 is the primary architectural fork of Category II.

Applicability

Use Long Context when:

• the working set fits comfortably inside a context window you can afford to pay for;
• the task needs free synthesis across the whole set — whole-document QA, full-file code reasoning, cross-document comparison;
• the scale does not justify building and operating retrieval infrastructure;
• prompt caching can amortise the repeated long prefix across many queries.

Do not use it when:

• the corpus is far larger than any window;
• per-call cost at full context length is prohibitive;
• sub-second latency is required — long prefills are slow.

Decision Criteria

K9 is mostly a sizing exercise — measure, threshold, decide.

1. Size the working set. Tokenize the full set you would need in front of the model, using the target model's own tokenizer. Call the result T.

2. Compare T to the model's usable window. Usable is lower than nominal — lost-in-the-middle degrades quality well before the model's stated limit (mechanism 4).

3. Cost the calls. Per uncached call: T × input-token-price. With prompt caching, repeat calls over the same set typically cost 10–25% of the uncached price after the first (provider-specific). For N queries per session over a stable set, total cost $\approx$ uncached × 1 + cached × (N − 1). If N is small the long prefix is paid in full almost every call — that usually breaks the economics.

Prefix cache mechanics (mechanism 5). The provider stores the KV state tensor $[L \times n \times n_{\text{kv}} \times d_{\text{head}}]$ of the stable prefix — the portion of the prompt that does not change across requests. Re-submission within the provider TTL (~5 minutes for Anthropic, minimum 1,024 tokens) injects the cached states directly, skipping prefill entirely and reducing cost to ~10% of the normal input token price for the cached portion. Sessions that pause longer than the TTL re-prefill at full cost. Design implication: Long Context is most economical when queries over the same stable document corpus are batched within the TTL window. A stable corpus that is loaded once and queried many times within 5 minutes pays the prefill cost once; the same corpus queried once per hour pays it every time.

4. Latency check. Long-context prefill runs hundreds of milliseconds to seconds. Sub-second deadlines eliminate K9.

5. Growth check. If the working set grows during the session (an agent accumulating observations, a chat accumulating context, retrieved content piling up), set a hard upper bound on T. K9 fails the moment T crosses the window with no graceful degradation; plan a K1 fallback above the bound.

Quick test — K9 is the right pattern when:

• T $\leq$ ~50% of the nominal window, and
• queries per session N $\geq$ 5 (so caching amortises the prefix), and
• the latency budget tolerates a long prefill, and
• T does not grow unboundedly during the session.

If any condition fails, choose K1 or one of its refinements. If T is large and the queries demand cross-document synthesis or relationship-tracing K1 cannot reach, choose K3 GraphRAG or K4 RAPTOR rather than just stuffing the window.

Structure

  Working set (all documents) ──▶ placed entirely into the context window
                                            │
                                            ▼
            [ system prompt + entire working set + query ] ──▶ LLM ──▶ Response

  No index. No retriever. No embedding pipeline.

Participants

The pattern's signature is the participants it removes: no chunker, no embedding model, no vector store, no retriever.

Collaborations

The whole working set is assembled into the prompt once. Each query is answered against it directly. Where prompt caching is available, the working-set prefix is cached on first use and reused across subsequent queries, so the large prefix is paid for once rather than on every call — this is what makes the pattern economical for repeated queries over a stable set.

Consequences

Benefits

• No retrieval infrastructure, no chunking strategy, no embedding pipeline.
• No retrieval miss — the answer is always in context if it is in the working set.
• Full cross-document synthesis, with no graph or index needed.
• A far simpler architecture; with caching, cheap repeated queries over a stable set.

Costs

• Tokens for the entire working set on every uncached call.
• Long prefill latency.
• A hard ceiling at the window size.
• Quality still degrades at extreme context lengths.

Why the cost is non-linear (mechanism 2). The prefill cost of processing $n$ tokens scales as $O(n^2)$ in attention compute. Doubling the context quadruples the prefill cost, not doubles it. The 10–25% caching discount applies to the cached prefix only — variable content (the query, dynamic metadata) is always prefilled at full cost. This means the economic break-even for Long Context vs RAG depends on the ratio of stable to variable content, not just total token count.

Risks and failure modes

• Lost in the middle — material buried mid-context is used poorly even though it is present.
• Silent overflow — the working set grows past the window and content is dropped without warning.
• Cost surprises — without caching discipline, the per-call token bill is large.

Implementation Notes

• Prompt caching is what makes Long Context economical for repeated queries over a stable set — design for it deliberately.
• Place the query, and the most important material, at the start or end of the context — not buried in the middle.
• Measure quality at your actual context length; do not trust the model's nominal limit.
• Keep a fallback to K1 for when the working set outgrows the window.
• "Long context versus RAG" is an empirical question for your task and corpus — benchmark both rather than assuming.

Implementation Sketch

LLM = configured session; code = wiring.

Composition: Almost nothing — assemble the working set, mark it cacheable, query against it. The interesting engineering is the cache configuration, not the chain.

The chain:

Skeleton:

long_context_answer(working_set, query):
    prefix = assemble(working_set)                      # code
    return Generator(prefix, query, cache=True)         # LLM

The LLM sessions:

Specialist-model note. This pattern is a specialist requirement at the model layer: a long-context model paired with a working prompt-cache implementation. Without caching, K9's per-call token cost is its defining weakness. Measure both quality and cost at your actual context length before committing; quality degrades long before the nominal context limit.

Open-Source Implementations

Long Context is an architecture, not a library — there is no canonical "Long Context" project. The relevant references are the provider cookbooks:

• Anthropic Cookbook — github.com/anthropics/anthropic-cookbook — prompt-caching recipes that make the long-prefix approach economical.
• Google Gemini Cookbook — github.com/google-gemini/cookbook — long-context (1M+ token) workflow examples.

Known Uses

• Gemini long-context (1M+ token) document and codebase workflows.
• Claude long-context document and full-repository analysis.
• "Paste the whole file or repo" coding workflows.
• The widely-discussed long-context-versus-RAG benchmarking literature.

Related Patterns

• Competes with K1 Vanilla RAG — the K1-versus-K9 decision is the primary architectural fork of the category.
• Competes with K3 GraphRAG and K4 RAPTOR — a large window can synthesise and abstract across documents without a pre-built index, at higher per-call cost.
• Composes with K6 Context Compression — compress the working set to make it fit a window.
• Aligned with K11 Observational Memory — both favour a stable, cacheable context prefix.
• Pairs with prompt caching, an Integration- and Reliability-layer concern.

Sources

• Long-context-versus-RAG benchmarking literature (2024–2026).
• Model long-context technical reports (Gemini, Claude).
• Liu et al. (2023) — "Lost in the Middle: How Language Models Use Long Contexts."

K10 — Long-Term Memory

Persist knowledge in an external store that outlives the context window, and retrieve from it in later turns and later sessions, so the agent accumulates and reuses what it learns.

Also Known As: Persistent Memory, Cross-Session Memory, External Memory, Agent Memory. (Episodic, Semantic, and Procedural memory are variants of this pattern — see Variants.)

Classification: Category II — Knowledge · Band II-C Memory · cross-session persistence.

Intent

Give an agent continuity beyond a single context by writing knowledge to an external store and retrieving it when relevant — so the agent improves over time without retraining.

Motivation

A context window is erased at the end of a session. By default an agent begins every session knowing nothing of the last one: no memory of what the user told it, what it tried, what worked. For one-shot tasks that is fine. For personal assistants, agents on recurring work, and any system expected to improve, it is disabling — the agent cannot personalise, cannot avoid repeating mistakes, cannot build expertise. The two within-context patterns, K8 Working Memory and K11 Observational Memory, do not solve this; they live and die with the session.

Long-Term Memory adds the missing layer: an external store, outside the context window, that persists across sessions. The agent writes knowledge to it as it goes, and retrieves the relevant entries — typically by embedding similarity — into the context of a later session. The mechanism is file retrieval, not model learning (mechanism 10). The model's weights are frozen between API calls. All improvement is in the store — the quality of what is retrieved and injected into context. A better memory system is one that retrieves higher-quality text into the context window; no capability accrues in the model itself.

The mechanism is uniform — write to an external store, retrieve by similarity, inject — and it is the same mechanism as K1 Vanilla RAG, with one decisive difference: in K1 the corpus is given, while here the agent writes its own corpus from its experience.

Variants

The variants differ only in what is stored. They map to the cognitive-science memory triad, and they are one pattern — the store / retrieve / inject mechanism is identical — differentiated by content type and retention policy:

• Episodic — records of what happened: past runs, decisions, outcomes, failures and their causes. Lets the agent recall "last time I tried X here, Y broke." Tends to decay with age.
• Semantic — facts, concepts, and user preferences: what the agent knows. Lets it personalise and accumulate domain knowledge. Tends to accumulate.
• Procedural — verified how-to: code patterns, tool-use sequences, workflows that worked. Lets the agent reuse a proven procedure instead of re-deriving it. Tends to be verified then reused, and is often distilled from episodic memory.

A given system may run one, two, or all three stores. They behave differently in retention and retrieval, but the pattern is one.

Applicability

Use Long-Term Memory when:

• the system is a personal assistant that should remember the user across sessions;
• the agent works recurring task types — a coding agent on a codebase, a research agent in a domain;
• the system is expected to get better over time.

It is unnecessary for stateless, one-shot tasks.

Decision Criteria

K10 is the right memory pattern when memory means isolated, fact-shaped items that should survive sessions and be retrieved on demand.

1. Inventory what should be remembered. Are the items short, fact-shaped, independent — user preferences, decisions, isolated facts about entities? Or are they connected knowledge worth organising into pages? If the former, K10 fits. If the latter, K12 Karpathy Memory fits better.

2. Read pattern. Do reads arrive as queries answerable by similarity to stored items? If yes, K10's vector store is the natural access pattern. If reads need structural navigation (open the X note, follow the link to Y), choose K12.

3. Write/read balance. K10 writes are cheap — one Extractor call per exchange. Reads are cheap — one similarity search. Both scale linearly. K10 has no hidden curation cost, which is its advantage over K12 and its limit: it builds no structure.

4. Cross-session continuity. Is there continuity worth keeping between sessions? If sessions are independent and forgetting is fine, neither K10 nor K12 is needed — K11 within sessions is enough.

5. Operator inspection. If a human needs to read, audit, or correct memory, K10's flat vector store is hard to navigate compared to K12's structured notes. Factor that into the choice.

Quick test — K10 is the right pattern when:

• the items are fact-shaped (preferences, decisions, isolated facts), and
• similarity retrieval is the natural access pattern, and
• write and read are roughly balanced (no curation amortisation needed), and
• cross-session continuity is required.

If items are connected knowledge with structure, prefer K12 Karpathy Memory. If memory is only needed within a session, K11 Observational Memory. K10 and K12 are commonly run together — facts in the vector store, structure in the notes.

Structure

  DURING a session                          A LATER session
  ────────────────                          ───────────────
  observe / act                             query
       │                                      │
       ▼                                      ▼
  extract memory-worthy items           retrieve relevant entries
       │                                 (similarity search)
       ▼                                      │
  write to external store ──────────────▶     ▼
  (episodic / semantic / procedural)    inject into context ──▶ proceed

Participants

Collaborations

Write path. During a session the Memory Writer watches what the agent observes and does, extracts the items worth keeping, and writes them to the appropriate store. Read path. In a later session the Retriever searches the store for entries relevant to the current query and injects them into context. Distillation path (procedural). The Distiller periodically abstracts recurring successful episodes from the episodic store into parameterised procedures in the procedural store.

Consequences

Benefits

• Genuine cross-session continuity, personalisation, and improvement over time.
• Expertise accumulates — all without retraining the model (mechanism 10).

Costs

• External store infrastructure.
• Write-time extraction cost; retrieval latency.
• Memory management overhead — deciding what to keep and what to expire.

Risks and failure modes

• Stale memory — the world changed, the memory did not.
• Conflicting memory — a new fact contradicts a stored one and both are retained.
• Memory poisoning — the agent stores a hallucination as fact and trusts it in every later session. The most dangerous failure of the pattern. The mechanical depth of this risk: a poisoned memory item is embedded and stored as a text vector. When retrieved, it is injected as tokens into the context. The model's attention treats those tokens no differently from ground-truth tokens — there is no architectural mechanism to flag retrieved-from-store content as suspect (mechanism 3). The only defence is write-time selectivity (the Extractor's poisoning guard).
• Irrelevant retrieval — surfaced memories mislead rather than help.
• In multi-agent systems, per-agent stores diverge into inconsistent "memories."

Implementation Notes

• Be selective at write time — store what is reusable, not everything observed.
• Expire or decay episodic memory; it ages.
• Semantic memory needs conflict resolution when a new fact contradicts an old one.
• Procedural memory must be re-validated when the environment changes.
• In multi-agent systems, use a shared memory substrate rather than per-agent stores.
• Gate what gets written — memory poisoning is the failure to design against first.
• Implementations: Mem0, Zep, Letta (MemGPT), or a custom vector database.

Implementation Sketch

LLM = configured session; code = wiring.

Composition: Two main paths — a write path during a session (Extractor $\to$ Embedder $\to$ store) and a read path in later sessions (Embedder $\to$ similarity search $\to$ Generator). The procedural variant adds a periodic distillation path.

The chain — write:

The chain — read:

The chain — distil (procedural variant):

Skeleton:

remember(exchange, store, user):
    items = Extractor(exchange)                     # LLM
    for item in items:
        store.add(Embed(item), item, owner=user)    # LLM + code

recall(query, store, user, k=5):
    memories = store.search(Embed(query),           # LLM + code
                             owner=user, k=k)
    return Generator(query, memories)                # LLM

The LLM sessions:

Specialist-model note. No session is itself a specialist, but the Embedder is (as K1). The dedicated memory layers — Mem0, Zep, Letta — are infrastructure specialists rather than LLMs: they ship the store, write/read paths, and conflict-resolution logic, leaving the developer to wire only the prompts.

Open-Source Implementations

• Mem0 — github.com/mem0ai/mem0 — a universal memory layer for agents: extraction, storage, and cross-session retrieval.
• Agent Memory Techniques — github.com/NirDiamant/Agent_Memory_Techniques — 30 runnable notebooks covering episodic, semantic, and procedural memory and the major systems.
• Zep and Letta (formerly MemGPT) — production memory systems built on temporal knowledge graphs and self-editing memory respectively; both are surveyed, with code, in the repository above.

Known Uses

• Mem0, Zep, Letta (MemGPT) — dedicated agent memory layers.
• ChatGPT's memory feature — a user-facing semantic memory.
• Coding agents that persist verified procedural patterns across sessions.
• The agent-memory survey literature.

Related Patterns

• Same mechanism as K1 Vanilla RAG — store, retrieve, inject — but the agent authors its own corpus from experience. The shared mechanism is the bilinear similarity search (mechanism 1): both K1 and K10 embed a query vector and find the nearest stored K vectors in the learned similarity space. The difference is authorship — K1 retrieves from a human-curated corpus, K10 from an agent-authored one.
• Often paired with K12 Karpathy Memory — K10 holds flat fact-shaped items in a vector store; K12 holds structured curated notes. Together they cover both "what does the agent know about this user/entity?" (K10) and "how does the agent understand this domain/project?" (K12).
• Completes the memory hierarchy with K8 Working Memory (in-window), K11 Observational Memory (in-session), and K12 Karpathy Memory (curated): in-window / in-session / cross-session-flat / cross-session-structured.
• Internal dependency — the procedural variant is distilled from the episodic variant.
• Required by the Humanizer patterns — H2 Episodic Self-Improvement, H4 Procedural Skill Accumulation, and H10 Relational Memory all build on this pattern; H2 shares its poisoning risk.
• Note on fundamentality — episodic, semantic, and procedural memory were merged into one pattern because the store / retrieve / inject mechanism is identical across all three. They differ in content and retention policy, which makes them variants, not separate patterns.

Sources

• Agent-memory survey literature — "Anatomy of Agentic Memory" and related surveys.
• Shinn et al. (2023) — Reflexion (the episodic-memory origin).
• Mem0 and Zep documentation.
• Cognitive-science memory triad — Tulving (episodic/semantic), Baddeley (working/long-term).

K11 — Observational Memory

Treat what the agent has already seen and done within the current session as its primary memory — kept in a stable, compact, immediately available form — rather than re-retrieving it from an external store.

Also Known As: Agent-Centric Memory, Seen-First Memory, Session Memory

Classification: Category II — Knowledge · Band II-C Memory · in-session persistence. An emerging (2025–26) pattern.

Intent

Maintain coherence across a long agentic session by keeping a running, compact record of the agent's own observations and actions, and prioritising that record over external retrieval.

Motivation

When agents replaced chatbots, the memory question changed. A chatbot answering questions over a corpus needs RAG: retrieve what the documents say. An agent running for hours needs something else — it needs to recall what it did: which files it edited, which tools it called, what they returned, what it concluded. That is not in any external corpus; it is the session's own history. Using K1-style retrieval for it is a poor fit — the relevant context is recent agent activity, and re-retrieving it from a vector store is slow, imprecise, and beside the point.

Observational Memory takes the opposite stance: the agent's own observations are the primary memory. The session keeps a running, compressed record of what the agent has perceived and done, and that record — not an external corpus — is what the agent reasons over. External retrieval (K1) becomes a secondary source, consulted only when the in-session record is insufficient.

There is a second, structural payoff. A memory built from a stable, append-mostly record of observations changes slowly and predictably. A stable context prefix is a cacheable context prefix: KV-cache reuse across the session's many model calls is reported to cut cost by roughly an order of magnitude. The mechanism (mechanism 5 and 3): the provider computes and stores the KV states — a 4D tensor [layers $\times$ seq_len $\times$ kv_heads $\times$ d_head] — for any stable token prefix. On re-submission of the same token sequence, those states are injected directly, bypassing the O(seq_len²) prefill computation (mechanism 2). At Anthropic: minimum 1024 tokens, ~5-minute TTL, reads at ~10% of normal input cost. Any edit to a prior position in the prefix produces a different token ID $\to$ different K vector $\to$ cached state invalid for that position and all subsequent ones. K1-style retrieval, which rewrites the context with different chunks every turn, forfeits that. Observational Memory is partly a pattern for cache-friendliness.

This is distinct from K10 Long-Term Memory, which persists across sessions and is corpus-like; K11 is scoped to the current session and is observation-like. It is distinct from K8 Working Memory, which is a scratchpad the model deliberately writes; K11 is the accumulated record of everything the agent has observed, written deliberately or not. And it is distinct from K12 Karpathy Memory, which takes the same observation stream as input but has the LLM digest it into structured curated notes — K11 keeps the raw log cheap and cache-friendly; K12 pays curation cost to make later reads dense and navigable. The two are the raw-log and curated-notes branches of the same Karpathy framing of agent memory; they are often paired.

Applicability

Use Observational Memory when:

• the agent runs long sessions — hours, or days;
• the agent's own prior actions are the main relevant context — coding, research, operations agents;
• KV-cache reuse is a material cost lever for the deployment;
• K1 retrieval is too slow or too imprecise for in-session recall.

It is irrelevant to short tasks and single-turn question answering.

Decision Criteria

K11 is the right memory pattern when the agent's own activity is the memory and prompt caching makes the cost work.

1. Session length. How long does a typical session run? If sessions are short (a handful of turns), the cache amortisation that justifies K11 does not accrue. Threshold of interest: roughly $\geq$ 20–30 turns, or hour-scale sessions.

2. Provider and model caching. Does the chosen model and provider expose prompt caching at usable granularity? Without it, K11 is just "keep appending tokens" — costs scale linearly per turn with no offset, and the pattern's main economic argument disappears. Additionally, sessions that pause between agent steps for longer than the TTL (~5 minutes on Anthropic) will re-prefill at full cost on the next step — the cache benefit accrues only within an active session (mechanism 5). For long-idle agents, the economics shift back toward K10 or K12.

3. Cache hit rate target. Measure expected and actual cache hit rate. Below ~70% the pattern is misconfigured — something is rewriting prior entries, or the recorder is not truly append-only. Above ~90% is where the reported ~10$\times$ cost reduction lands.

4. Read pattern. Is the agent reading the whole record (which K11 makes cheap via cache) or only specific entries (which K12 makes cheap via structure)? Whole record matters $\to$ K11. Specific entries matter $\to$ K12.

5. Cross-session continuity. K11 is session-scoped. If continuity is needed beyond the session, pair with K10 (facts in a vector store) or K12 (curated notes) — usually both.

Quick test — K11 is the right pattern when:

• session length supports cache amortisation ($\geq$ ~20 turns, or hour-scale), and
• the provider supports prompt caching at appropriate granularity, and
• cost — not only quality — is the lever you are optimising, and
• the agent benefits from reading the whole record rather than specific entries.

If sessions are short, drop K11 — it has no benefit. If you need structured, navigable memory rather than the whole record, choose K12 Karpathy Memory. If memory must persist across sessions, pair with K10 or K12 (usually both).

Structure

  agent observes / acts
        │
        ▼
  append observation to the running record (compressed)
        │
        ▼
  record forms the stable context the agent reasons over ──▶ KV-cache reuse
        │                                                     (stable prefix)
        ▼
  external retrieval (K1) consulted only as a fallback

Participants

Collaborations

At each step the Recorder appends the latest observation or action to the running record. The agent reasons over the record as its primary memory. Because the record is append-mostly, its prefix is stable across calls and is served from the KV-cache rather than recomputed. Only when the record lacks something the agent needs does it fall back to external retrieval.

Consequences

Benefits

• Coherent behaviour across long sessions — the agent reliably recalls its own history.
• Large cost reduction through KV-cache reuse (reported around 10$\times$).
• Simpler than operating an external retrieval layer for in-session recall.
• The record doubles as a natural execution trace.

Costs

• The record still consumes window space — it needs K6 and K7 to stay bounded.
• Scoped to one session: no cross-session continuity (that is K10's job).

Risks and failure modes

• External knowledge the agent has not "seen" is missing from the record entirely.
• Record growth, if unmanaged, crowds the window.
• If the record is compressed badly, the agent loses access to its own history.

Implementation Notes

• Keep the record append-mostly and the prefix stable — that stability is what unlocks caching.
• Compress with K6 and prune with K7, but in ways that preserve the cacheable prefix where possible.
• Pair with K10 for cross-session continuity, and with K1 as the fallback for external facts.
• This is an emerging pattern; expect implementation details to keep moving.

Implementation Sketch

LLM = configured session; code = wiring.

Composition: Append observations to a stable, append-mostly record; reason over that record as primary context; cache its prefix; fall back to K1 only on a gap. Managed by K6/K7 when it outgrows the window — carefully, to preserve the cacheable prefix.

The chain (per agent step):

Skeleton:

agent_step(query, mem):
    answer = Agent(mem.context(), query, cache_prefix=True)     # LLM
    if needs_external(answer):
        answer = Agent(mem.context() + K1.retrieve(query),
                       query, cache_prefix=True)                  # LLM + K1
    mem.observe(f"Q: {query}\nA: {answer}")                       # code — append-only
    return answer

The LLM sessions:

Specialist-model note. The hard dependency is prompt caching at the model and provider layer — the reported ~10$\times$ cost reduction comes from KV-cache reuse of the stable record prefix. Without caching, the pattern still works but loses its main economic advantage. Measure cache hit rate as a first-class metric, alongside answer quality.

Open-Source Implementations

Observational Memory is an emerging (2025–26) pattern with no single canonical project yet. The closest references:

• Agent Memory Techniques — github.com/NirDiamant/Agent_Memory_Techniques — covers session and observational memory alongside the cross-session variants.
• Letta (formerly MemGPT) — github.com/letta-ai/letta — its archival memory layer is the closest production embodiment of the K11 raw-record idea; note that Letta's core memory blocks belong to K12 (the curated branch), so Letta sits at the K11/K12 boundary by design.

Known Uses

• 2025–26 practitioner work on cutting agent costs through observational memory plus caching.
• Agent frameworks that deliberately favour stable, cacheable contexts.
• Long-session coding agents (Claude Code and similar) lean toward this approach.

Related Patterns

• Often paired with K12 Karpathy Memory — K11 keeps the raw activity record; K12 is the LLM-curated digest of it. The Curator in K12 typically reads K11's log as its source. K11 and K12 are the raw-log and curated-notes halves of the same Karpathy framing of agent memory.
• Distinct from K10 Long-Term Memory — in-session versus cross-session; usually paired, K11 for intra-session coherence and K10 for inter-session continuity.
• Distinct from K8 Working Memory — an accumulated observation record versus a deliberately written scratchpad.
• Managed by K6 Context Compression and K7 Context Pruning.
• Uses K1 Vanilla RAG as a fallback source on external-knowledge gaps.
• Aligned with K9 Long Context — both want a stable, cacheable context prefix.
• Not to be confused with the Humanizer pattern H6 Continuous Inner Monologue, which concerns background deliberation, not memory.

Sources

• Karpathy, A. — 2025 public talks and writing on agent architecture and "context engineering"; the raw-log + caching cost argument is the relevant claim. (See K12 for the curated-notes branch of the same framing.)
• Context-engineering and KV-cache reuse literature, 2025–26.
• Provider documentation on prompt caching (Anthropic, OpenAI, Google) — the structural dependency of the pattern.

K12 — Karpathy Memory

Have the LLM itself curate a structured, dense memory — writing, editing, merging, and linking entries — so every read is of pre-digested knowledge rather than a raw observation log or a vector of isolated extractions.

Also Known As: Curated Memory, Self-Edited Memory (Letta's term), Agent-Authored Wiki, Structured Notes Memory

Classification: Category II — Knowledge · Band II-C Memory · curated persistence; can be in-session or cross-session.

Intent

Give an agent a memory the LLM itself maintains as structured, dense, token-efficient notes — paying more at write time so every read is cheap, navigable, and useful.

Motivation

Memory for agents has, until recently, leaned on two strategies, each flawed at a different end:

• K10 Long-Term Memory stores short extracted items with vectors and retrieves them by similarity. Cheap to write, cheap to read individually — but the items are isolated and brittle, with no structure between them, and similarity retrieval misses anything not phrased like the query.
• K11 Observational Memory keeps the raw observation log as primary memory, leaning on prompt caching for cost. Free to write — but verbose at read: the agent rereads everything to remember anything, and any single fact is buried in the noise of the whole session.

Neither captures what a human knowledge worker does, which is to maintain notes. They write down what matters, revise their notes when they learn more, link them, prune them. The notes are dense because a human has digested the underlying material once and won't redo that work on every read.

Karpathy's framing of agent memory points to the same move: have the LLM build and maintain the memory itself — write structured entries, update them, refactor them — so each read is of pre-digested knowledge, not raw experience. The cost moves to the curator: the LLM call that organises the memory. The pay-off is at read time: every subsequent reasoning step over that memory pays only for the dense final form. The foundation: the model's weights do not change between sessions (mechanism 10). All capability that accumulates is in the files — the curated notes — that are read into context at each session. Curation is the process of making those files more information-dense per token, so each read obtains more useful knowledge per unit of context-window cost.

This is the third memory strategy. It is not K10's vector store and it is not K11's raw record. It is an LLM authoring its own knowledge base.

The defining claim of the pattern is asymmetric: one expensive curation buys many cheap reads. Where K10 amortises a moderate write against a moderate read, and K11 amortises a free write against a cheap-via-cache read, K12 amortises an expensive write against a very cheap, very useful read.

Applicability

Use Karpathy Memory when:

• the same memory will be read many times before it is updated (read frequency far exceeds curation frequency);
• the domain or user has structure worth preserving — entity profiles, project notes, evolving understanding;
• read-time token cost is a material lever (long contexts, many turns over the same memory);
• the memory must be human-readable and editable for operators or downstream agents.

Do not use it when:

• the memory is touched once or twice — curation cost will not amortise;
• the data is naturally a flat list of facts with no structure between them (K10 fits);
• curator-call budget is not affordable, or curation latency is intolerable at the trigger points.

Decision Criteria

K12 is right when curation amortises against many reads, structure earns its keep, and editability matters.

1. Estimate read-to-write ratio. Count expected reads of the memory between curations (R) versus curator calls per cycle (W). Practical threshold: if R / W $\geq$ 10, curation amortises clearly; below that, K10 or K11 is usually cheaper.

2. Score the structure benefit. Would a human reader of this memory want pages, sections, links? Entity profiles, decision logs, evolving project notes — yes, K12. A bag of independent facts — no, K10.

3. Cost the curator. Curation calls dominate the write side. Annualise: curator calls per day $\times$ cost per call. Compare to (a) the K11 cost of re-reading uncurated logs and (b) the K10 cost of similarity calls plus retrieval-miss errors.

4. Read-time efficiency check. A curated note is typically 5–20$\times$ denser than the raw observations it digested. This density directly reduces context-window cost (mechanism 9): in-context storage costs O(n²) per step in attention compute (mechanism 2). A 10$\times$ denser note means 10$\times$ fewer tokens in the context, which is not a linear saving — it collapses the per-step attention cost toward the sparser regime of the n² curve. If that compression unlocks the read budget — letting the agent hold its working memory in a small fraction of the window — K12 has paid.

5. Editability requirement. Does a human or another agent need to read, audit, or correct the memory? Curated notes are inspectable and editable. Vector-store memory (K10) effectively is not; raw observation logs (K11) are inspectable but not navigable.

Quick test — K12 is the right pattern when:

• R / W $\geq$ 10 (curation amortises against many reads), and
• the memory has structure worth preserving (entities, projects, linked concepts), and
• read-time token efficiency is a material concern, and
• inspectability and editability matter to operators or downstream systems.

If R / W is low, choose K11 — the raw log is already a record and curation overhead is unjustified. If the memory is flat facts with no structure, choose K10 — a vector store with similarity is simpler. If you need both a long activity log to reason from and a small curated overlay, run K11 and K12 together (the curated notes prepended to the cached log).

Structure

  Agent activity (sessions, tasks, exchanges, often K11's record)
         │
         ▼
  Trigger: end of session, milestone, periodic
         │
         ▼
  Curator (LLM) ──▶ reads current notes touched by activity + the activity itself
         │
         ▼
  emits edits: write new entries, update existing, merge duplicates, refactor, link
         │
         ▼
  Memory store: structured notes (pages, blocks, sections, links)
         │
         ▼
  At read: Selector picks relevant notes by name / topic / recency ──▶ Agent

Participants

The Curator and the Agent are kept distinct sessions, even when the same model serves both. The Agent reads; the Curator writes. Mixing them is the pattern's most common failure: an Agent that edits notes mid-reasoning destabilises the memory and erodes the cache. There is a mechanistic reason beyond semantic confusion: if the Agent writes to the note store mid-reasoning, the note tokens change position, invalidating the provider-side KV cache for those positions mid-session (mechanism 3 and 5). The separation is a cache correctness requirement as much as a design principle.

Collaborations

The Agent runs its task as usual, reading curated notes the Selector loaded. At a defined trigger — end of session, end of milestone, periodic interval — the Curator wakes up. It reads the existing entries touched by the recent activity and the activity log itself, then emits edits: new entries, updates to existing ones, merges of duplicates, links between related notes. The store applies the edits. The next time the Agent runs, the Selector chooses which notes to load and the Agent reasons over the refreshed curated subset.

Consequences

Benefits

• Read-time token cost is small — every read consumes dense, pre-digested content.
• The memory has structure: entries can be named, linked, indexed, navigated.
• Notes are inspectable and editable by humans or other agents.
• Improvement compounds: as understanding grows, the Curator refactors.

Costs

• Curator calls are not cheap — every update is at least one LLM call, often several.
• The memory drifts if curator prompts are weak — notes contradict, duplicates accumulate.
• Less cache-friendly than K11 — curation changes the prefix. The mechanism: each curation event modifies note content, producing a different token sequence for modified entries. The provider's KV cache key is the exact token sequence; a changed note entry invalidates the cached state for that position and all subsequent positions (mechanisms 3 and 5). This is why K11 (append-only, never-edit) is more cache-friendly by design — K12 explicitly trades cache stability for write-time structure improvement.
• Schema discipline: a sloppy schema yields unreliable retrieval.

Risks and failure modes

• Curator drift — repeated curations gradually rewrite history into the Curator's interpretation, not the original facts.
• Edit storms — too-frequent curation thrashes the memory and the cache.
• Schema collapse — without a stable schema, notes degenerate into free-form prose the Selector cannot index.
• Stale notes — without aging or refresh, old notes mislead.
• Agent-as-Curator confusion — when the Agent edits the store mid-reasoning, working state and persistent memory blur.

Implementation Notes

• Treat the Curator as a separate session from the Agent, even when using the same model. Different setups, different prompts, different invocations.
• Trigger curation deliberately — session end, milestone, periodic — never every turn.
• Keep the schema simple at first: titled entries with sections, links by entry name. Add structure only when retrieval misses it.
• Version the Curator's prompts; track curator-output diffs over time as a drift signal.
• The Selector can be a small generalist call or a deterministic index — choose by the navigation pattern.
• Pair with K11 for in-session activity (the Curator reads K11's log to produce K12 entries) and with K10 if you also need fact-level extraction in a flat vector store.

Implementation Sketch

LLM = configured session (model + setup + per-call prompt); code = wiring.

Composition: K12 chains an Agent (reads) with a separate Curator (writes) against a structured Memory store. The Curator often reads K11's activity log as its source material. K12 commonly composes with K11 (activity input) and K10 (orthogonal vector store for flat facts).

The chain — read (per Agent step):

The chain — curate (at trigger):

Skeleton:

read(query, store):
    notes = Selector(query, store.index)              # code (or small LLM)
    return Agent(notes, query)                         # LLM

curate(activity_log, store):                           # at trigger only
    touched = store.entries_touching(activity_log)     # code
    edits   = Curator(touched, activity_log)           # LLM — write/edit/merge/link
    store.apply(edits)                                  # code

The LLM sessions:

Specialist-model note. No fine-tuned specialist is required, but two structural choices change everything:

• The Curator must be a separate session from the Agent. Same model is fine; different setups, different invocations. Mixing them creates the "agent edits memory while reasoning" failure mode.
• A long-context model materially helps the Curator, which must hold current notes plus recent activity. The Curator's quality benefits from the strongest available model — paid for in batches at trigger time, not per turn.

Open-Source Implementations

• LLM Wiki (Karpathy, 2026) — gist.github.com/karpathy/442a6bf555914893e9891c11519de94f — the reference design. A Gist, not a repo: describes the raw/ + wiki/ + schema architecture, the three operations (ingest / query / lint), and the index.md + log.md navigation state. The authoritative source for the pattern's mechanism.
• memoriki — github.com/AyanbekDos/memoriki — the closest direct OSS implementation (Apr 2026, 105 ★, MIT). Lifts Karpathy's raw/ + wiki/ + CLAUDE.md structure verbatim, then adds a MemPalace semantic-search layer on top for queries the wiki alone cannot answer. A template starter rather than a mature library.
• Letta (formerly MemGPT) — github.com/letta-ai/letta — mature production implementation. Core memory blocks are LLM-curated structured notes the agent edits with explicit tools (memory_replace, memory_insert, memory_rethink). Archival memory layers in for scale beyond the core.
• Cognee — github.com/topoteretes/cognee — memory control plane: builds and persists a knowledge graph from heterogeneous sources, exposes remember / recall / forget / improve operations. Apache 2.0.
• Agent Memory Techniques — github.com/NirDiamant/Agent_Memory_Techniques — runnable notebooks covering Letta, Mem0, Zep, Graphiti, and the curated-vs-extracted distinction.
• CLAUDE.md / AGENTS.md conventions in coding-agent workflows — project-level markdown maintained by the agent across sessions. A community convention rather than a single repo; the pattern in its lightest form.

Known Uses

• Letta production deployments — agents with editable core memory blocks, including a coding-agent variant (letta-code).
• Coding-agent ecosystems (Claude Code, Cursor) — project-level CLAUDE.md and rules files curated by the user or the agent over time.
• Personal-assistant agents maintaining user profiles and project notes as structured entries rather than raw histories.
• karpathy/autoresearch issue #179 — open proposal to add a project-level long-term memory file and a Guidance Agent to autoresearch, applying the same curated-briefing principle to a research-agent loop. Adjacent engineering discussion, not a released implementation.

Related Patterns

• Distinct from K10 Long-Term Memory — K10 stores extracted facts in a vector store, retrieved by similarity; K12 stores structured notes the LLM authored, retrieved by name / topic / inclusion. Often paired: K10 for fact recall, K12 for the agent's organised understanding.
• Distinct from K11 Observational Memory — K11 is the raw activity record; K12 is the digest of it. K11 usually feeds K12 — the curator reads the log.
• Echoes K6 Context Compression in spirit but differs in scope — K6 compresses live context to free space; K12 produces persistent structured notes for repeated reads. Same instinct (digest once, use many times), different time scale.
• Pairs with S6 Output Template — the note schema is a Signal-layer artifact that constrains the Curator.
• Pairs with V14 Trajectory Logging — the activity log feeding the Curator overlaps the Reliability category's logging concern; same raw data, different uses.
• Named after Andrej Karpathy, whose framing of agent memory — "structure memory to be token-friendly; use the LLM to build the data" — is the clearest articulation of the pattern.

Sources

• Karpathy, A. (2026) — "LLM Wiki" — gist.github.com/karpathy/442a6bf555914893e9891c11519de94f — primary source for the raw/ + wiki/ + schema architecture and the ingest / query / lint operations.
• Karpathy, A. (2025) — "Context engineering" tweet, June 25 — x.com/karpathy/status/1937902205765607626 — coins the framing; distinguishes context engineering from prompt engineering in industrial-strength LLM applications.
• Karpathy, A. (2025) — YC AI Startup School keynote, June 16–17 — "LLM as OS" framing; context window as RAM; LLMs as having anterograde amnesia without external memory. Summary: latent.space/p/s3.
• Packer et al. (2023) — "MemGPT: Towards LLMs as Operating Systems" — arXiv 2310.08560 — predecessor of Letta; the paged-memory model that K12 generalises.
• Letta documentation — core-memory blocks and the self-editing memory model.
• "Anatomy of Agentic Memory" (arXiv) and the Agent Memory Techniques survey for the wider variant landscape.

K13 — Retrieval Bundle

Before writing any retrieval code for an agent workflow, explicitly specify the complete operational context bundle that workflow always needs — by field, by source, by data shape, by freshness requirement, and by authorization constraint — then build storage and assembly to deliver that bundle reliably.

Also Known As: Agent Operating Context, Workflow Context Specification, Typed Memory Contract, Pre-Compiled Context Bundle.

Classification: Category II — Knowledge · Band II-A Retrieval strategy · design-time prerequisite — K13 is the specification step that precedes K1, K3, K4, and the storage tier decisions in K10/K11/K12. It does not replace them; it tells you which ones to use for which fields.

Intent

Define the exact bundle of operational context a specific agent workflow always needs — not "relevant documents" but precisely these fields from these sources in these shapes — and then build assembly to deliver that bundle reliably, rather than letting the agent reconstruct it dynamically from raw search results on every run.

Motivation

The rediscovery problem. Classic RAG was designed for a chatbot era job: a user types a question, the system finds three semantically similar chunks, the model writes a paragraph. That loop works because the answer lives in a few paragraphs and the user asks once.

Agents do not ask questions and stop. They run tasks — open a ticket, check the policy, retrieve the customer record, draft the response. If an agent runs a customer escalation task, it needs the customer record, the applicable plan, the product version, the region, the purchase history, the refund policy, the refund threshold, any prior exceptions for this customer, the current ticket, the approved response language, and whether the agent is authorized to issue the refund or only draft a recommendation. That is not three semantically similar chunks. That is a typed operational bundle assembled from multiple sources.

Classic RAG leaves the agent to assemble that bundle on the fly from raw search results on every run. The consequence, measured at production scale, is severe: rediscovery can consume up to 85% of agent compute (PineCone, 2025). Agents refetch the same context every run. They re-summarize documents they summarized last time — correctly or not. They ask users for information the system already has. They blow the token budget before useful work begins.

Why the rediscovery problem is architectural. The model's weights do not persist knowledge between API calls (mechanism 10). The KV cache does not persist across sessions. Everything the agent assembled last time is gone unless it was written to external storage and re-loaded. An agent without a specified bundle has no stable definition of what it needs, so it improvises the assembly each time — paying the full cost of discovery on every run.

Why larger context windows do not fix this. A larger window gives the model more room to work. It does not decide what belongs in that room. It does not mark which source is authoritative. It does not enforce freshness. It does not distinguish what the agent confirmed from what it inferred. Filling a large window with the raw output of generic search — mixing authoritative, stale, and inferred content — causes context rot: the model cannot reliably determine which facts to weight, treats stale alongside current as equal, blends sources it should cite separately, and gives wrong emphasis. Performance degrades not because the right answer is absent but because it is not presented in a form the model can use reliably. This is mechanism 4 (lost-in-middle / U-shaped recall) compounded by provenance ambiguity: a fact buried in the middle of a long mixed-authority context is both statistically under-attended and untrustably attributed.

Why the retrieval unit must match the data shape. Vector search finds text that is semantically similar to the query. The learned bilinear form (mechanism 1) captures distributional co-occurrence: tokens that appear in similar contexts are nearby in the embedding space. This is effective for fuzzy prose where meaning lives in word choice and phrasing. It is systematically wrong for three other shapes:

• Structured documents (contracts, filings, regulatory documents): a clause can look semantically relevant while a definition 40 pages away completely changes its meaning. A schedule can overwrite a general term. The structure of the document — its section hierarchy, its cross-references, its schedules and annexes — carries meaning that chunking into embedding-sized fragments destroys. Vector search finds text that sounds right; it misses the legal or structural relationships that make it correct.

• Governed tabular data (ERP tables, CRM records, warehouse metrics, financial models): the source of truth for a revenue number is a governed metric definition tied to a specific table with specific lineage and access controls. Converting a table to prose and asking a language model to reason over it via semantic search is the wrong abstraction. The column relationships, row ordering, aggregation semantics, and data governance cannot be preserved in a vector embedding. Tabular data requires tabular-native retrieval.

• Relational knowledge (supplier-to-shipment connections, customer failure patterns, incident root causes): some knowledge is inherently relational — it lives in the edges between entities, not in the entities themselves. Graph-shaped data requires graph-native retrieval (K3 GraphRAG). Chunk retrieval cannot represent edges.

If you pick the wrong retrieval primitive for a given data shape, the model compensates at high context cost — spending tokens reconstructing structure that was available but lost, or inferring relationships that were stored but not retrieved. Better embeddings do not fix this. Higher-quality vector search still cannot retrieve document hierarchy, table semantics, or graph edges — it finds more relevant text. Better text is not the same as the right data shape.

Applicability

Use Retrieval Bundle when:

• you are designing or debugging an agent that runs a specific, recurring workflow type (support, research, document review, financial analysis, procurement, compliance);
• the agent currently rebuilds context from scratch on every run, re-fetching or re-summarizing material it has assembled before;
• the agent's context window is filling with mixed-authority or mixed-freshness content that degrades output reliability;
• you are choosing between retrieval primitives and are not sure which to use — K13 is the prerequisite that answers that question.

Do not use it as a replacement for retrieval patterns:

• K13 is the specification step; K1, K3, K4, and the memory patterns (K10–K12) are the implementation. K13 tells you which of those to use for which fields; it does not replace them.
• For purely exploratory agents where the workflow varies entirely per run, a fixed bundle specification is not possible — use K5 Adaptive RAG to decide dynamically what to retrieve.
• For simple single-question systems, the overhead of bundle specification is not justified — K1 with good chunking is enough.

Decision Criteria

K13 is right when the workflow type is stable, the agent has a defined task, and rediscovery is a measurable cost.

1. Test for workflow type stability. Can you write down, for a specific class of tasks this agent handles, the same list of information fields every run needs? If yes — this is a bundleable workflow. If the fields vary entirely per task with no common structure: K5 Adaptive RAG or dynamic assembly is more appropriate.

2. Measure the rediscovery cost. In your existing agent logs, count: how many retrieval calls happen before useful task work starts? How often does the agent read a source it read in a prior session? How often does it ask a user for something the system has? What fraction of your token budget is context assembly rather than task execution? If context assembly consumes > 30% of your token budget, rediscovery is your largest cost lever.

3. Identify the data shapes. For each field in the bundle, classify it:

Most real agents need more than one shape. This is not a failure — it is the correct diagnosis. The error is assuming one primitive covers all shapes.

4. Define freshness and authority per field. For each bundle field, specify:

• Source: which system is authoritative (not merely relevant)?
• Freshness: how stale is too stale for this field? (customer status: real-time; policy text: daily; historical tickets: session-level)
• Authorization: is this agent permitted to see this field for this entity?
• Missing-field behavior: what should the agent do if this field cannot be retrieved?

A bundle with unspecified authority and freshness produces context rot. Specifying per-field makes the agent's behavior deterministic when sources disagree or are unavailable.

5. Cost the assembly. Pre-assembled bundles have a write cost (assembly call at task start or periodic refresh). Dynamic RAG has a per-run discovery cost. Compare: assembly_cost $\times$ assembly_frequency vs. rediscovery_cost $\times$ run_frequency. At high run frequency over stable data, pre-assembly wins materially.

Structure

  DESIGN TIME (once per workflow type):
  ┌────────────────────────────────────────────────────────┐
  │  Bundle Specification                                  │
  │  ├── Field: customer_record                           │
  │  │   ├── source: CRM (tabular)                        │
  │  │   ├── freshness: real-time                         │
  │  │   ├── auth: agent role = support                   │
  │  │   └── if missing: halt, request human              │
  │  ├── Field: refund_policy                             │
  │  │   ├── source: policy corpus (structured doc)       │
  │  │   ├── freshness: daily                             │
  │  │   ├── auth: all agents                             │
  │  │   └── if missing: use default policy               │
  │  ├── Field: prior_tickets                             │
  │  │   ├── source: ticket system (relational)           │
  │  │   ├── freshness: session-level                     │
  │  │   └── if missing: proceed with empty history       │
  │  └── ...                                              │
  └────────────────────────────────────────────────────────┘
           │
           ▼
  Storage and retrieval primitives chosen by shape:
  tabular fields → semantic layer / governed table access
  structured doc fields → K4 RAPTOR tree index
  relational fields → K3 graph store
  prose fields → K1/K2 vector index

  RUN TIME (each agent task):
  ┌────────────────────────────────────────────────────────┐
  │  Bundle Assembler                                      │
  │  ├── fetch tabular fields from governed table          │
  │  ├── retrieve structured doc sections via K4          │
  │  ├── retrieve graph neighborhood via K3               │
  │  └── retrieve prose via K1/K2                         │
  └────────────────────────────────────────────────────────┘
           │
           ▼ compact, typed, authoritative bundle
  ┌────────────────────────────────────────────────────────┐
  │  Agent context window                                  │
  │  [bundle — small, high-signal, authoritative]         │
  │  [task instructions]                                   │
  └────────────────────────────────────────────────────────┘
           │
           ▼
  Task execution (no rediscovery)

The structural invariant: the agent's context contains the bundle (assembled once, authoritative, correctly shaped) and the task instructions. It does not contain raw search results, re-derived summaries, or content the agent has to evaluate for relevance and authority.

Participants

Collaborations

At design time, the engineer writes the bundle specification for each workflow type. This drives all subsequent infrastructure choices: which retrieval primitives are needed, which storage tiers are required, which authorization systems must be integrated.

At run time, the Bundle Assembler fires at task start, collecting each field via its specified primitive. The assembled bundle is injected into the agent's context window as a compact, typed, authoritative unit — before any task reasoning begins. The agent then executes its task against the bundle without further retrieval calls (for the specified fields).

For fields with high freshness requirements (real-time customer status), the Assembler retrieves fresh on every run. For fields that are stable across many runs (policy text, product definitions), the Assembler checks a local cache or uses K9 Long Context with prefix caching (mechanism 5) to amortize the retrieval cost.

When a task discovers it needs a field not in the bundle specification, that discovery is a signal to update the specification — not to add ad-hoc retrieval inside the task logic. Ad-hoc retrieval inside the task is the rediscovery pattern re-entering through the back door.

Consequences

Benefits

• Eliminates rediscovery: the agent receives its complete operating context at task start, assembled once, from authoritative sources. No per-run re-fetching, re-summarizing, or user re-questioning.
• Eliminates context rot: each field is labeled with its source and freshness; the agent knows what is authoritative and what is contextual.
• Shape-correct retrieval: each field is delivered by the primitive appropriate for its data shape, not approximated by the nearest available primitive.
• Predictable, auditable agent behavior: the bundle specification is an explicit contract; every agent run against the same entity with the same specification produces the same input structure.
• Token efficiency: a pre-assembled, compact bundle is smaller and higher-signal than the raw search results the agent would otherwise assemble dynamically. Less context waste = more attention budget for task reasoning (mechanisms 2, 6).

Costs

• Design-time investment: the bundle specification must be written explicitly for each workflow type. This requires understanding the data landscape before writing retrieval code.
• Multiple primitives: most real bundles need more than one retrieval primitive. This means more infrastructure components to maintain.
• Specification drift: as the workflow evolves, the bundle spec must be kept current. An outdated spec produces a bundle that no longer matches the workflow's actual information needs.
• Assembly latency: collecting fields from multiple systems adds task startup latency vs. a single search call.

Risks and failure modes

• Vendor-first design — picking a database before writing the bundle spec constrains the agent to the database's shape strengths. The database becomes the frame that defines what the agent can retrieve, rather than the agent's information needs defining what database to use.
• Ad-hoc bundle expansion — engineers add retrieval calls inside task logic when they notice missing context, rather than updating the spec. The bundle silently diverges from the specification. Eventually every run has ad-hoc retrieval and rediscovery re-enters through the task code.
• Authority ambiguity — bundle includes fields without source labeling; the agent blends a governed metric with a retrieved paragraph and cannot distinguish which is authoritative. Context rot.
• Shape mismatch — using vector search for structured documents (finds relevant-sounding clauses, misses controlling definitions 40 pages away) or tabular data (approximates numeric relationships as semantic similarity). The error is systematic, not random — it consistently misses the structurally important content.
• Stale bundle cache — pre-compiled bundles for high-freshness fields cached past their TTL. Agent reasons from stale customer status or expired policy. Freshness enforcement must be per-field, not per-bundle.
• Over-engineering — building graph + document tree + semantic layer + vector search + tabular model for a simple FAQ agent. K13 is a diagnostic tool; the output of the specification is often "K1 is enough for this workflow."

Implementation Notes

• Write the bundle spec as a schema, not prose. Each field should be a row: name, source, data shape, freshness TTL, authorization rule, missing-field behavior. A prose description of what the agent needs is not a specification.
• Start from the failure logs. The pattern is in your existing agent runs: how many retrieval calls before useful work starts? How often is the agent re-reading the same sources? How many user clarification requests are for information the system has? These numbers tell you the size of the rediscovery problem before you commit to an architecture.
• Match shape before optimizing retrieval quality. A well-tuned vector search over a contract corpus will still miss the controlling schedule. Fix the shape match first; optimize retrieval quality second.
• Mark the authority boundary explicitly in the bundle. Each field in the context should carry metadata: authoritative (the governed metric, the current policy) vs. contextual (prior agent runs, historical examples, background reference). The agent's system prompt should include a rule: "rely on authoritative fields for task decisions; use contextual fields for background only."
• Version the bundle spec alongside the agent. When the workflow requirements change, update the spec and the assembly in the same change. Spec and implementation that drift apart produce the rediscovery pattern.
• Use prefix caching for stable fields (mechanism 5). Bundle fields that are stable across many runs within a session (policy text, product definitions, authorization rules) can be placed in the stable prefix and cached. Variable fields (the specific customer record, the current ticket) come after the cache boundary. This is K13 composing with O18 (Cache-Warmed Worker Pool) for batched workflows.

Implementation Sketch

Design-time: write the bundle specification

workflow_type: customer_escalation

fields:
  - name: customer_record
    source: CRM (governed table)
    shape: tabular
    retrieval: semantic_layer.get_customer(entity_id)
    freshness_ttl: 0  # real-time, always fresh
    auth: role IN [support, supervisor]
    if_missing: halt → "customer record required"

  - name: applicable_plan
    source: product catalogue (governed table)
    shape: tabular
    retrieval: semantic_layer.get_plan(customer.plan_id)
    freshness_ttl: 86400  # daily
    auth: role IN [support, supervisor, agent]
    if_missing: substitute with default_plan

  - name: refund_policy
    source: policy corpus (structured document)
    shape: structured_doc
    retrieval: raptor_index.get_section("refund", customer.region)
    freshness_ttl: 86400
    auth: all
    if_missing: substitute with global_policy

  - name: prior_tickets
    source: ticketing system (relational)
    shape: relational
    retrieval: graph.get_customer_history(entity_id, limit=5)
    freshness_ttl: 3600
    auth: role IN [support, supervisor]
    if_missing: proceed with empty history

  - name: approved_response_language
    source: response template corpus (prose)
    shape: prose
    retrieval: vector_index.retrieve(query=task.issue_type, k=3)
    freshness_ttl: 86400
    auth: all
    if_missing: use unconstrained language with V1 approval gate

Run-time: assembly and injection