The term “context engineering” only entered the AI vocabulary in mid-2025, but by mid-2026 it has eclipsed “prompt engineering” as the dominant framing for how serious teams build with LLMs. Gartner’s July 2025 declaration (“context engineering is in, prompt engineering is out”) proved prescient: at production scale, the work of crafting individual prompts has been automated, abstracted, or platform-ized. What matters now is the surrounding infrastructure — what information the LLM sees, where that information comes from, how it’s selected, how it’s versioned, how it’s measured. Context Engineering 2026 is a 15-chapter playbook for engineering teams operating LLM systems at scale: the architecture, the patterns, the tooling, the operational practices, and the anti-patterns that separate production-grade systems from ad-hoc prompt collections.
Table of Contents
- Why prompt engineering became context engineering
- The context engineering architecture
- Context vs prompts vs retrieval — defining the layers
- Building a prompt library
- Prompt versioning and lifecycle management
- Prompt evaluation and A/B testing
- Context window management and trimming
- Retrieval-augmented context patterns
- Memory and stateful context
- Multi-turn conversation context
- Tool-use context patterns
- Observability for context systems
- Cost engineering for context-heavy workloads
- Vendor landscape — prompt platforms and frameworks
- Anti-patterns and the 90-day context engineering plan
- Frequently Asked Questions
Chapter 1: Why prompt engineering became context engineering
The shift from prompt engineering to context engineering happened in three phases. Phase 1, roughly 2022-2023, was the artisan era — individuals crafted prompts by hand, shared “magic” prompts on social media, treated prompt-writing as an art form. Phase 2, 2024 and into mid-2025, was the early industrialization — teams started versioning prompts, running basic A/B tests, building prompt libraries. Phase 3, mid-2025 forward, is context engineering proper — the recognition that prompts are part of a larger context-construction system, and the operational discipline shifts to managing that system end-to-end.
Three forces drove the shift. First, LLM context windows expanded dramatically — from 4K-32K tokens in early 2023 to 1M-2M tokens by 2026. The question is no longer “how do I cram everything into 8K tokens” but “what should I include from the millions of tokens available.” Selection became the central problem. Second, multi-turn agents and tool-using systems made prompts dynamic — the prompt for turn 5 of an agent’s reasoning loop is different from the prompt for turn 1, computed at runtime based on what happened so far. Static prompt text became insufficient. Third, enterprise deployments at scale revealed that prompt-related production failures were dominating reliability metrics — by 2026, multiple studies showed 60%+ of production AI incidents traced to prompt or context issues rather than model issues.
The economic argument is also clear. A 2025 industry survey found that teams investing in systematic context engineering reduced prompt-related production incidents by 60%+ compared to ad-hoc prompt management. The same teams shipped new AI features faster (mean cycle time roughly halved) and ran AI workloads at 30-50% lower per-call cost (better context selection means fewer tokens means cheaper inference). Context engineering pays for itself within the first major deployment.
What does “context engineering” actually mean in practice? It encompasses: prompt library management (version-controlled, reviewed, deployable); context selection logic (which information to include given the user query, conversation history, retrieved knowledge); context window optimization (fitting the relevant information in the available budget); evaluation infrastructure (continuous quality measurement against golden sets); cost and latency management (per-call budgets, prompt prefix caching, etc.); observability (full traces of what context was assembled for each LLM call). Each is its own discipline; together they form context engineering as a coherent practice.
The terminology shift matters strategically too. “Prompt engineering” framed the work as wordsmithing, which led to it being undervalued by traditional engineering teams. “Context engineering” frames the work as systems engineering, which fits naturally into existing software engineering practices — version control, code review, testing, deployment, observability. The reframing has changed how teams staff this work (more senior engineers, not just prompt enthusiasts) and how organizations invest in tooling (proper platforms, not just text files in a repo).
For organizations starting context engineering work in 2026, the strategic context is favorable. The tooling has matured. The vendor landscape is well-developed. The patterns are documented. The investment case is clear. The remaining work is execution: pick the right tools, build the right team, ship the discipline through the organization. Done well, context engineering becomes invisible infrastructure that powers reliable AI features; done poorly, it remains a constant source of production friction.
The skill profile that succeeds in context engineering combines three competencies. Software engineering fundamentals (version control, testing, code review, observability — the standard production practices applied to prompts as artifacts). LLM intuition (understanding how models respond to different framings, what’s in their training distribution, where they’re likely to fail). Domain knowledge (understanding the business problem well enough to know what context is relevant). The intersection of all three is rare; teams typically split the work across multiple roles with overlapping competencies.
Organizations that have been running context engineering for 12+ months report consistent patterns in what works. The most-impactful single investment is the evaluation harness — without it, every other improvement is guesswork. The second-most-impactful is the prompt library — moving prompts out of code into a managed system. The third is observability tracing — seeing the full request lifecycle including context assembly. These three investments compound; teams that have all three operate qualitatively differently from teams that have none.
Counter-intuitively, the biggest barrier to context engineering adoption isn’t technical. The tools exist. The patterns are documented. The barrier is organizational: convincing leadership that “we should invest engineering time in managing prompts as software” requires demonstrating real cost from the current ad-hoc approach. The easiest path to leadership buy-in: instrument current production AI for two weeks; show the incident rate, the cost, the time spent firefighting prompts; pitch the investment as the path to halving these costs. The numbers usually make the case decisively.
Chapter 2: The context engineering architecture
A production context engineering system has six functional layers. Understanding them clarifies which problems each layer solves and where to invest. The layers: prompt templates (the static text and structure), context inputs (dynamic information injected at runtime), retrieval (information fetched from knowledge bases or memory), selection (logic for choosing what to include), assembly (the final prompt construction), and orchestration (how this all hooks into the actual LLM call lifecycle).
# The six layers of context engineering
# Layer 1: Prompt templates
# Static prompt structures with placeholders
# Example:
# "You are a customer support agent for {COMPANY_NAME}.
# The user is asking: {USER_QUERY}
# Relevant context: {RETRIEVED_DOCS}
# Conversation history: {HISTORY}
# Respond in the company's voice as described in: {STYLE_GUIDE}"
# Layer 2: Context inputs
# Dynamic values injected at runtime
# - User identity, role, preferences
# - Current time, day, locale
# - Session state, conversation history
# - Feature flags, A/B test assignment
# Layer 3: Retrieval
# Fetching relevant information from sources
# - Vector search over knowledge bases (RAG)
# - SQL/NoSQL queries for structured data
# - API calls for live external data
# - Tool definitions for agent capabilities
# Layer 4: Selection
# Logic for choosing what to include given the goal
# - Relevance ranking
# - Token budget allocation
# - Filtering by user permissions
# - Prioritization by recency, importance
# Layer 5: Assembly
# Building the final prompt sent to the LLM
# - Template rendering with selected context
# - Format/structure decisions (XML, markdown, JSON)
# - Order of context (most important first/last)
# - Prefix caching alignment
# Layer 6: Orchestration
# Connecting the layers to the LLM call lifecycle
# - Pre-processing (validation, normalization)
# - Inference call (model, parameters)
# - Post-processing (parsing, validation)
# - Observability (tracing, metrics, audit)The mature context engineering architecture also includes meta-layers that operate across the six functional layers: evaluation infrastructure that runs continuous quality checks; experimentation infrastructure for A/B testing context strategies; cost and latency monitoring; security and access controls.
The interaction between layers is where complexity lives. The selection layer needs to know about the model’s context window (Layer 5 concern), the retrieval results (Layer 3 concern), and the user’s context budget (cost concern). These cross-cutting interactions are why context engineering platforms emerged — to provide a unified abstraction over the layer interactions rather than requiring each AI feature to wire them together manually.
# Cross-layer interaction example
# Question: how do we decide which retrieved documents to include in the final prompt?
# Naive answer: include them all
# Problem: they may exceed context window
# Problem: they may exceed cost budget
# Problem: they may dilute the LLM's attention
# Better answer: selection layer reads from all layers
# - Retrieval results (Layer 3): N candidates with relevance scores
# - Token budget (Layer 5 constraint): "prompt has X tokens remaining"
# - User permissions (Layer 2): "user can see docs in groups [A, B, D]"
# - Importance signals: "this is a high-stakes query; favor depth"
# Selection logic:
# 1. Filter to permitted docs
# 2. Rank by relevance
# 3. Allocate tokens (top doc gets full text; lower-ranked get summaries)
# 4. Stop when budget is reached
# 5. Document the selection for audit and debugging
# The 2026 standard: this logic is in the context engineering platform,
# not duplicated across every AI featureFor new context engineering deployments, the architectural question isn’t “do we need all six layers” — you do — but “where do we get each layer from.” Build vs buy decisions apply: build the layers that encode your business logic (selection rules, retrieval against your data); buy/use existing tooling for the generic layers (templates, evaluation harness, observability). The decision framework mirrors any platform engineering build/buy: build what’s differentiating, buy what’s commoditized.
The layered architecture also matters for team boundaries. Different teams can own different layers as the system grows. Platform team owns templates, orchestration, and observability infrastructure. Feature teams own their specific selection logic and retrieval against their data. The shared abstraction (the platform team’s contract) lets feature teams move quickly without reinventing infrastructure.
Avoid the trap of putting all logic in one layer. A common anti-pattern is putting selection logic inside templates (with massive conditionals in the prompt template), which makes templates hard to read and reason about. A cleaner pattern is keeping templates simple and pushing selection into code that runs before template rendering. Same final result; much easier to maintain and test.
# Anti-pattern: selection logic in template
template = """
{% if user.role == 'admin' %}
You have admin context: {ADMIN_CONTEXT}
{% elif user.role == 'manager' %}
You have manager context: {MANAGER_CONTEXT}
{% else %}
You have basic context: {BASIC_CONTEXT}
{% endif %}
... etc
"""
# Hard to test, version, evaluate
# Better: simple template; logic in code
template = """
You have role-appropriate context:
{CONTEXT}
"""
def assemble(user):
if user.role == 'admin':
return {"CONTEXT": admin_context()}
elif user.role == 'manager':
return {"CONTEXT": manager_context()}
else:
return {"CONTEXT": basic_context()}
# Template stays simple; selection is testable PythonChapter 3: Context vs prompts vs retrieval — defining the layers
Confusion about terminology is rampant in 2026. “Prompt,” “context,” “retrieval,” and “system message” are sometimes used interchangeably, sometimes with subtle distinctions. Clear definitions matter because the right tooling and patterns differ depending on which concept you mean.
Prompt: the complete input sent to the LLM in a single inference call. This includes everything — system message, user message, tool definitions, prior conversation turns, retrieved documents, formatting instructions. In modern usage, “prompt” is the entire payload, not just one piece of it.
System message (or system prompt): the persistent instructions about the LLM’s role and behavior. Examples: “You are a helpful customer support agent.” “Respond in JSON matching this schema.” The system message typically doesn’t change between turns in a conversation.
User message: the actual user input or query for this turn. The shortest, most variable part of the prompt.
Context: any information beyond the immediate user query that informs the LLM’s response. This includes retrieved documents, conversation history, user metadata, tool results, environmental state. Context is what “context engineering” focuses on — the systematic management of this layer.
Retrieval: the specific act of fetching information from a knowledge base or other source to include in context. RAG (retrieval-augmented generation) is the dominant pattern. Retrieval produces context but isn’t the same as context (which includes non-retrieved information).
# Concrete example: a customer support agent
# Prompt structure sent to LLM:
{
"model": "claude-opus-4-7",
"system": "You are a customer support agent for ACME Corp...", # System message
"messages": [
# Conversation history (context)
{"role": "user", "content": "I can't access my account"},
{"role": "assistant", "content": "I can help. What's your account email?"},
{"role": "user", "content": "joe@example.com"},
# Current user message
{"role": "user", "content": "It says password is wrong but I'm sure it's right"},
],
"tools": [...], # Tool definitions (context)
}
# Before this call, the context engineering platform:
# 1. Retrieved customer record for joe@example.com (context)
# 2. Retrieved relevant support articles about password issues (retrieved context)
# 3. Loaded the conversation history (context)
# 4. Selected the most relevant 3 articles by relevance score (selection)
# 5. Constructed the system message with company-specific instructions (template)
# 6. Assembled everything into the final promptThe conceptual distinction matters because different teams own different layers. The prompt template team owns the system message structure. The RAG team owns retrieval. The conversation management team owns history truncation. The selection/orchestration team owns assembly. In immature organizations, all of this is one engineer’s responsibility; in mature ones, it’s distributed with clear interfaces. Either model can work; the discipline is keeping the layers’ responsibilities clear.
One under-appreciated distinction: ephemeral vs persistent context. Ephemeral context exists only for this inference call (retrieved docs that may differ next call). Persistent context lives across calls (user preferences, conversation history). The two require different storage and management — ephemeral context lives in the request payload; persistent context lives in a database or memory store. Designing for both deliberately prevents confusion later.
Another distinction worth knowing: structured vs unstructured context. Structured context (JSON, tables, key-value pairs) is easier to validate and easier for the LLM to use precisely. Unstructured context (free text, long documents) is more flexible but harder to validate and sometimes harder for the LLM to extract specific facts from. The 2026 best practice: prefer structured context when you have structured data; only use unstructured for genuinely unstructured content like documents and code.
The role of metadata is also worth highlighting. Each piece of context can carry metadata — source, timestamp, confidence, freshness, importance. Including this metadata in the prompt (rather than just the raw content) lets the LLM reason about it: “this fact comes from a 2-week-old document with 0.7 confidence” enables different treatment than “this fact is authoritative.” For high-stakes use cases, metadata-aware prompts perform noticeably better than metadata-stripped versions.
# Example: passing metadata to the LLM
prompt = """
You are answering a question using retrieved documents.
Retrieved documents:
[
{
"id": "doc-123",
"title": "Q1 2026 Financial Report",
"source": "internal-finance",
"date": "2026-04-15",
"confidence": 0.95,
"content": "..."
},
{
"id": "doc-456",
"title": "Market analysis blog post",
"source": "external-blog",
"date": "2025-11-02",
"confidence": 0.6,
"content": "..."
}
]
Prefer recent, high-confidence sources. Cite your sources by id.
"""
# The LLM now has context to weight sources appropriately
# Internal sources get more trust than external blogs
# Recent dates get more trust than old onesChapter 4: Building a prompt library
The prompt library is the foundational artifact of context engineering. It’s the version-controlled, reviewed, deployable repository of prompt templates used across the organization’s AI features. Without a prompt library, prompts live as strings in source code, get duplicated across teams, drift from approved versions, and become a maintenance nightmare. With a prompt library, prompts are first-class artifacts with proper governance.
The minimum viable prompt library has five capabilities. First, storage — prompts live in a structured format (JSON, YAML, or markdown with frontmatter) in a known location. Second, versioning — each prompt has version history; old versions are accessible. Third, environment binding — different versions can be deployed to different environments (dev, staging, prod). Fourth, validation — prompt syntax and placeholder usage are checked before deployment. Fifth, observability — usage of each prompt version is tracked.
# Sample prompt library structure
prompts/
customer-support/
triage-v3.yaml # current production version
triage-v2.yaml # previous (kept for rollback)
response-drafter-v5.yaml
sales/
lead-qualification-v2.yaml
knowledge-base/
summarizer-v4.yaml
# Sample prompt file format (YAML)
# customer-support/triage-v3.yaml
name: customer-support-triage
version: 3
description: |
Classifies incoming support tickets into categories
and routes to appropriate handler.
model: claude-haiku-4-5
parameters:
temperature: 0.0
max_tokens: 500
system: |
You are a support ticket triage system for ACME Corp.
Classify the user's request into one of:
- billing
- technical
- account-access
- feature-request
- other
Respond with JSON: {"category": "...", "confidence": 0.0-1.0, "reasoning": "..."}
inputs:
- name: user_message
description: The user's support ticket text
type: string
required: true
tags: [triage, classification, customer-support]
owners:
- support-eng@company.com
deployment:
dev: enabled
staging: enabled
production: enabled
created_at: 2026-04-15
last_modified: 2026-05-10For storage backend, the choice is between Git-based (prompts as files in your repo) and database-based (prompts in a dedicated prompt management service like LangSmith, PromptLayer, or Vellum). Git-based is simpler and integrates with existing engineering workflows; database-based offers richer features like UI editing, A/B testing infrastructure, and analytics. Many organizations start with Git-based and migrate to database-based as they scale; some stay with Git-based indefinitely.
# Git-based prompt library workflow
# 1. Engineer creates a new prompt or modifies existing
git checkout -b improve-triage-prompt
nano prompts/customer-support/triage-v4.yaml
# 2. Run validation locally
prompt-lint prompts/customer-support/triage-v4.yaml
# Checks: YAML syntax, required fields, placeholder consistency
# 3. Run evaluation against the golden set
prompt-eval --prompt prompts/customer-support/triage-v4.yaml \
--eval-set evals/triage-v1.jsonl
# Reports: accuracy on each evaluation case; comparison to baseline
# 4. Open PR; CI runs broader evaluation
# Code review focuses on prompt clarity, schema correctness, eval results
# 5. After approval, merge to main
# Deployment pipeline picks up new version
# Old version remains for rollbackNaming conventions matter at scale. The pattern that works: project/component/purpose-vN.yaml. Versions are integers (not semver); the file name has the version, and version history lives in git. Branching strategies follow your team’s git practices. Some teams keep prior versions as separate files; others rely on git history. Both work; pick one and stay consistent.
For organizations with multiple products and many AI features, prompt library structure becomes its own design problem. Common patterns: by product (product-A/, product-B/), by AI capability (classification/, generation/, extraction/), by team (team-1/, team-2/). Each has trade-offs. Most teams converge on a hierarchical structure: top-level by team/product, then by AI feature within that. The discipline is keeping the hierarchy stable enough that prompts can be found and discovered.
Searchability matters too. With hundreds of prompts, finding the right one (or knowing whether one already exists for a use case) is a real problem. Tags, descriptions, and search functionality in your prompt library platform make this manageable. Git-based libraries can use simple grep; platform-based libraries usually have UI-based search. Either way, document prompts well — purpose, inputs, expected behavior — so search returns useful results.
# Searchable prompt metadata example
# In each prompt file
tags:
- customer-support
- classification
- triage
description: |
Classifies incoming customer support tickets into one of
five categories: billing, technical, account-access,
feature-request, other. Returns category plus confidence.
# Search by tag
prompt-cli search --tag classification
# Search by purpose
prompt-cli search "ticket classification"
# List prompts a team owns
prompt-cli list --owner support-eng@company.com
# Find all prompts using a specific model
prompt-cli search --model claude-haiku-4-5Chapter 5: Prompt versioning and lifecycle management
Prompts have lifecycles like any other software artifact: created, tested, deployed, monitored, updated, deprecated, retired. The lifecycle management discipline ensures prompts move through these stages deliberately rather than accumulating randomly.
The standard lifecycle has six stages. Draft: a new prompt being developed; not yet ready for evaluation. Evaluating: in formal evaluation against the golden set; may iterate. Staged: passed evaluation; deployed to staging environment for real-traffic testing. Production: deployed to production for canary or full rollout. Deprecated: superseded by a newer version; still callable but flagged. Retired: removed from active use; historical record only.
# Prompt lifecycle states and transitions
# Draft → Evaluating (when first version is ready)
prompt-cli promote --prompt triage --version 4 --to evaluating
# Evaluating → Staged (when evaluation passes thresholds)
prompt-cli promote --prompt triage --version 4 --to staged
# Triggers: golden-set accuracy ≥ baseline; no regressions on edge cases
# Staged → Production (canary at first, then full)
prompt-cli promote --prompt triage --version 4 --to production --traffic 5%
# Monitor for 24-48 hours; if metrics good, increase
prompt-cli promote --prompt triage --version 4 --traffic 100%
# Production v3 → Deprecated (when v4 is fully rolled out)
prompt-cli deprecate --prompt triage --version 3
# Deprecated → Retired (after retention period)
prompt-cli retire --prompt triage --version 3Versioning strategies vary. Integer versions (v1, v2, v3) work for simple sequential progression. Semantic-style versions (major.minor.patch) help when changes have different scope — major for breaking changes, minor for additions, patch for tweaks. Some teams use date-based versions (2026-05-19) for clarity. Pick a strategy and apply consistently.
Rollback discipline is essential. When a new prompt version regresses on real traffic, rolling back fast matters. The infrastructure for this is straightforward: keep prior versions deployed; route traffic via configuration; flip configuration to roll back. The discipline is using it — don’t ship forward through a known regression hoping to fix it; roll back first, then fix in a controlled environment.
Canary deployment for prompts mirrors the standard canary pattern for code. Roll the new prompt out to a small percentage of traffic first; monitor metrics; expand if metrics are good; roll back if metrics regress. The infrastructure: feature flags or routing configuration that maps traffic to prompt versions; observability that compares metrics between control and canary; automated alerts on regression.
# Canary deployment workflow
prompt-cli deploy --prompt triage --version 4 --canary 5%
# 5% of traffic now sees v4; 95% sees v3
# Wait 24-48 hours; check metrics
prompt-cli metrics --prompt triage --version 4 --compare-to 3
# Output:
# v4 success rate: 91% (n=2500)
# v3 success rate: 89% (n=47500)
# Statistical significance: p=0.03
# Expand canary
prompt-cli deploy --prompt triage --version 4 --canary 25%
# More traffic; more confidence
# Full rollout once confident
prompt-cli deploy --prompt triage --version 4 --canary 100%
# Or roll back if regression
prompt-cli rollback --prompt triage --to-version 3# Production rollback pattern
# Detected regression in triage-v4
# (accuracy on golden set dropped 5% after deployment)
# Immediate rollback
prompt-cli rollback --prompt triage --to-version 3
# Investigate the regression in staging
prompt-cli get --prompt triage --version 4 > /tmp/v4.yaml
prompt-cli eval --prompt /tmp/v4.yaml --eval-set golden-set-extended.jsonl
# Identify what changed; revise; re-evaluate; re-deploy
prompt-cli create --prompt triage --version 5 --from /tmp/v4-fixed.yaml
prompt-cli promote --prompt triage --version 5 --to evaluating
# This whole cycle should take minutes, not hoursChapter 6: Prompt evaluation and A/B testing
Evaluation infrastructure is what makes context engineering work in practice. Without continuous evaluation, prompt changes are guesses; with rigorous evaluation, prompt changes are experiments with known outcomes. Three types of evaluation: golden-set testing (regression checks against curated examples), LLM-as-judge (model-based quality assessment on broader inputs), and A/B testing (real-traffic comparison).
# Golden-set evaluation pattern
# Maintain a golden set of input/expected-output pairs
# evals/triage-golden.jsonl
{"input": "I forgot my password", "expected": {"category": "account-access"}}
{"input": "Where's my $500 refund?", "expected": {"category": "billing"}}
{"input": "The app crashes on startup", "expected": {"category": "technical"}}
{"input": "Can you add dark mode?", "expected": {"category": "feature-request"}}
# ... 100-500 cases representing real distribution
# Run evaluation
prompt-eval --prompt triage-v4 --eval-set evals/triage-golden.jsonl
# Output:
# Accuracy: 94/100 (94%)
# By category:
# billing: 25/25 (100%)
# technical: 24/25 (96%)
# account-access: 23/25 (92%)
# feature-request: 22/25 (88%)
# Regressions vs v3: 2 cases now failing that were passingLLM-as-judge extends evaluation beyond exact-match scenarios. For tasks where the “right” output isn’t a fixed string (response quality, helpfulness, tone), a judge model rates each output against criteria. Done well, judge evaluations correlate well with human ratings; done poorly, they have systematic biases. Validate your judge against human ratings periodically.
# LLM-as-judge evaluation
# evals/support-response-judge.yaml
judge_prompt: |
Rate the support response on three dimensions:
- Helpfulness (1-5): does it address the user's actual question?
- Accuracy (1-5): are the facts correct?
- Tone (1-5): is it appropriately professional and empathetic?
User question: {USER_QUESTION}
Support response: {RESPONSE}
Respond with JSON: {"helpfulness": N, "accuracy": N, "tone": N, "reasoning": "..."}
judge_model: claude-opus-4-7 # use a strong model for judging
threshold:
helpfulness: 4.0 # mean across eval set must be ≥ 4.0
accuracy: 4.5 # mean must be ≥ 4.5
tone: 4.0 # mean must be ≥ 4.0
# Run
prompt-eval --prompt support-responder-v3 \
--eval-set evals/support-questions.jsonl \
--judge evals/support-response-judge.yamlA/B testing is the gold standard for production validation. Run two prompt versions on real traffic; measure outcomes (resolution rate, user satisfaction, conversion). Decisions based on A/B test results are dramatically more reliable than decisions based on offline evaluation alone. The cost is operational complexity; the benefit is confidence in changes.
Statistical rigor matters in A/B tests. Common mistakes: tests stopped too early (peeking and ending when results favor the desired outcome); too small sample size (results not statistically significant); confused metrics (improving one metric while regressing another). The discipline: pre-register the hypothesis and metrics; pre-calculate required sample size; run for the planned duration; analyze with proper statistics; report all relevant metrics not just the favorable ones.
LLM-as-judge biases worth knowing. Self-preference bias: a model judging its own outputs vs another model’s tends to favor its own. Length bias: judges often favor longer responses regardless of quality. Sycophancy bias: judges sometimes mirror the framing of the question. Mitigations: use a different model as judge than the one being evaluated; explicitly instruct the judge to ignore length; use careful prompt framing. Validate periodically against human ratings.
Domain-specific evaluation is the most-skipped step. Generic eval metrics (accuracy, helpfulness) miss domain-specific quality dimensions. For medical applications: clinical safety, terminology accuracy. For legal: citation accuracy, hedging appropriate. For finance: precision on numbers, regulatory compliance. Build evaluation criteria specific to your domain; generic eval misses the issues that matter most.
One specific evaluation pitfall worth highlighting: distribution drift. Your golden set was constructed at a point in time; the production distribution evolves. Eval accuracy on a stale golden set may not predict production accuracy. Refresh the golden set quarterly by sampling recent production traffic; this keeps the eval set representative of current usage.
# Continuous golden set expansion
# Monthly: sample recent production traffic for new eval cases
def expand_golden_set():
# Sample 100 recent successful interactions
sampled = sample_production_traffic(n=100, days=7)
# Have humans label expected outputs for these
labeled = await human_label(sampled)
# Add to golden set
golden_set.extend(labeled)
# Re-run all existing prompts against expanded set
for prompt in active_prompts:
score = evaluate(prompt, golden_set)
if score < threshold:
alert(f"Prompt {prompt.name} regressing against expanded eval")# A/B test setup
prompt-ab-test create \
--name triage-v3-vs-v4 \
--control triage:3 \
--treatment triage:4 \
--split 50/50 \
--metric resolution_rate \
--metric escalation_rate \
--metric cost_per_interaction \
--duration 7d \
--sample-size 5000
# After test completes
prompt-ab-test analyze --test triage-v3-vs-v4
# Results:
# Control (v3): 87% resolution, 12% escalation, $0.04/interaction
# Treatment (v4): 89% resolution, 10% escalation, $0.04/interaction
# Statistical significance: p < 0.05 for resolution improvement
# Decision: promote v4 to 100%Chapter 7: Context window management and trimming
Modern LLM context windows (1M-2M tokens in 2026 for frontier models) seem vast, but real applications fill them faster than you’d expect. A conversation with extensive history; 50 retrieved documents; many tool definitions; long system prompts — quickly approaches the limit. When the context overflows, requests fail. Context window management prevents this.
Three patterns for staying within context limits. Trimming: remove older conversation turns or less-relevant context to fit. Summarization: collapse older content into summaries that occupy fewer tokens. Selective inclusion: choose which context to include based on relevance scoring rather than including everything available.
# Conversation history trimming
def trim_history(messages, max_tokens):
"""Keep most recent messages within token budget."""
total = 0
kept = []
for msg in reversed(messages):
msg_tokens = count_tokens(msg)
if total + msg_tokens > max_tokens:
break
kept.insert(0, msg)
total += msg_tokens
return kept
# Summarization-based trimming
def summarize_old_history(messages, threshold):
"""Replace old messages with a summary."""
if len(messages) < threshold:
return messages
old = messages[:-threshold]
recent = messages[-threshold:]
summary = llm_summarize(old)
return [{"role": "system", "content": f"Earlier conversation summary: {summary}"}] + recent
# Relevance-based selection
def select_retrieved_docs(docs, query, token_budget):
"""Rank by relevance; include top docs that fit the budget."""
ranked = sorted(docs, key=lambda d: d['relevance_score'], reverse=True)
selected = []
used = 0
for doc in ranked:
if used + doc['tokens'] <= token_budget:
selected.append(doc)
used += doc['tokens']
return selectedPrefix caching has become a meaningful optimization in 2026. Most LLM providers support caching the prefix of a prompt (system message, tool definitions, static context) so subsequent calls with the same prefix pay reduced cost and lower latency. Designing your prompt structure to maximize prefix-cacheable content saves substantial cost on high-volume workloads.
# Prompt structure for maximum prefix caching
# Cacheable prefix (same for every call within a session)
# System message
# Tool definitions
# Long static context (style guide, brand voice, etc.)
# Variable suffix (different per call)
# Conversation history (changes each turn)
# Retrieved docs (depends on query)
# Current user message
# Cache for static portion
# - Anthropic: ephemeral cache, 5 minute TTL
# - OpenAI: automatic; varies by usage
# - Self-hosted: vLLM with prefix caching enabled
# Cost savings example
# Without caching: 5000 input tokens per call at full price
# With caching: 4000 cached tokens at 10% price + 1000 fresh tokens at full price
# Net: ~75% reduction on input token costContext budget allocation is its own discipline. For a given total context budget (e.g., 100K tokens), how do you split between system message (essential, small), conversation history (medium, variable), retrieved context (largest, most variable)? A common allocation: 5% system, 15% history, 70% retrieval, 10% buffer. Adjust based on your specific use case; instrument actual usage and re-tune periodically.
The position of context within the prompt affects LLM attention. The “lost in the middle” phenomenon — content placed in the middle of a long context gets less attention than content at the beginning or end — has been studied extensively. Patterns: put the most critical content (the user’s actual question) near the end, just before the response is generated; put long retrieved context in the middle where it’s available but doesn’t dominate; put structural framing (system message) at the start. For very long contexts, repeating critical instructions near the end is sometimes worth the token cost.
Modern frontier models in 2026 have improved on the “lost in the middle” problem but haven’t eliminated it. Claude Opus 4.7 and GPT-5.5 both show meaningfully better attention across long contexts than their predecessors, but still benefit from thoughtful structuring. The general guidance: don’t bury critical information in the middle of a 100K-token context unless necessary; structure the prompt so important content is at the start or end.
Newer techniques like long-context retrieval (retrieving over a large context window vs over a vector database) and hierarchical attention (model architectures that attend differently to different parts of long contexts) are improving the situation but aren’t universally available yet. Design for current model capabilities; revisit when model behavior shifts.
For applications with truly massive context needs (analyzing 500-page documents, processing long meeting transcripts), the architectural pattern shifts. Rather than one giant prompt, use map-reduce: split the input into chunks; process each chunk separately to extract or summarize; combine the chunk-level outputs into the final answer. This pattern scales arbitrarily and produces more reliable results than stuffing everything into one giant context.
# Map-reduce pattern for very long documents
async def process_long_document(doc):
chunks = chunk_document(doc, chunk_size=8000)
# Map: process each chunk independently
chunk_outputs = await asyncio.gather(*[
process_chunk(chunk) for chunk in chunks
])
# Reduce: combine chunk outputs into final
final = await combine_outputs(chunk_outputs)
return final
async def process_chunk(chunk):
# Each chunk gets the same prompt structure
# Returns a summary or extracted information
return await llm.complete(prompt_template.render(chunk=chunk))
async def combine_outputs(outputs):
# Combine the per-chunk outputs
# Often another LLM call summarizing the summaries
combined_text = "\n".join(outputs)
return await llm.complete(combine_template.render(parts=combined_text))Chapter 8: Retrieval-augmented context patterns
RAG (retrieval-augmented generation) is the dominant pattern for grounding LLM outputs in your data. The basic flow: vectorize a query; search a vector database; retrieve top-K relevant documents; include them in the LLM prompt; LLM generates an answer using the retrieved context. The patterns in 2026 have matured significantly from the early RAG implementations of 2023-2024.
Quality retrieval is the foundation. Common improvements over basic RAG: hybrid search (combine vector similarity with keyword/BM25), reranking (use a cross-encoder to refine the top-N), query rewriting (have an LLM rephrase the user’s query before search), metadata filtering (restrict candidates by user permissions, recency, source).
# Modern RAG pipeline
def rag_retrieve(query, user_context):
# Step 1: optional query rewriting
rewritten = llm_rewrite_query(query, user_context)
# Step 2: hybrid search
vector_results = vector_db.search(embed(rewritten), top_k=50)
keyword_results = keyword_index.search(rewritten, top_k=50)
combined = reciprocal_rank_fusion(vector_results, keyword_results)
# Step 3: metadata filtering
permitted = filter_by_user_permissions(combined, user_context.user_id)
fresh = filter_by_recency(permitted, max_age_days=180)
# Step 4: reranking with cross-encoder
reranked = rerank_model(rewritten, fresh, top_k=20)
# Step 5: selection within budget
selected = select_within_budget(reranked, token_budget=8000)
return selectedChunking strategy matters meaningfully. Documents need to be split into chunks for vector search; chunk size and overlap affect retrieval quality. Too small: fragments lose context. Too large: less precise matching, wasted tokens. The 2026 baseline: 256-512 token chunks with 50-100 token overlap; semantic chunking (split on natural boundaries like paragraphs and sections) outperforms fixed-size chunking for most use cases.
Chunk metadata matters too. Each chunk should carry: source document, position within the document, surrounding context (titles, section headings), creation date, source authority level. When the retrieval returns chunks, this metadata helps the LLM contextualize the content — knowing this fact comes from page 5 of a Q1 2026 internal financial report is more useful than knowing this fact comes from “some document somewhere.”
The query side of RAG deserves attention too. Users phrase queries casually; the literal text isn’t always the best embedding query. Query rewriting — using an LLM to rephrase the user’s query into a more searchable form — meaningfully improves retrieval quality. For multi-step questions, query decomposition (splitting one complex query into multiple simpler sub-queries) further improves results.
# Query rewriting pattern
async def smart_rag_retrieve(user_query, conversation_context):
# Use a small fast model to rewrite the query
rewritten = await llm_complete(
model="claude-haiku-4-5",
prompt=f"""Rewrite this user query for vector search.
Make it more specific and searchable. Preserve all key terms.
Conversation context: {conversation_context}
User query: {user_query}
Rewritten search query:""",
max_tokens=100,
)
# Search with the rewritten query
results = await vector_db.search(embed(rewritten), top_k=20)
return results# Chunking strategies
# Fixed-size with overlap
def chunk_fixed(text, chunk_size=512, overlap=64):
tokens = tokenize(text)
chunks = []
for i in range(0, len(tokens), chunk_size - overlap):
chunks.append(tokens[i:i+chunk_size])
return chunks
# Semantic chunking (better quality)
def chunk_semantic(text, target_size=512):
# Split on paragraph or section boundaries
paragraphs = split_paragraphs(text)
chunks = []
current = []
current_size = 0
for para in paragraphs:
para_size = count_tokens(para)
if current_size + para_size > target_size and current:
chunks.append(' '.join(current))
current = []
current_size = 0
current.append(para)
current_size += para_size
if current:
chunks.append(' '.join(current))
return chunks
# Hierarchical chunking (for long documents)
# Index at multiple granularities: full doc, section, paragraph
# Retrieval first finds relevant doc, then drills into sections, then paragraphsEmbedding model choice matters too. Text-embedding-3-large from OpenAI, voyage-3 from Voyage AI, Cohere’s embed-english-v3, and Anthropic’s embeddings are all credible 2026 options. Each has trade-offs on cost, dimensionality, and domain performance. For specific domains (legal, medical, code), specialized embeddings sometimes outperform general-purpose ones. Benchmark on your real data before committing.
Re-embedding when you switch models is the under-considered cost. Embeddings are model-specific; switching from one embedding model to another means re-embedding your entire corpus. For large corpora, this is meaningful infrastructure work and meaningful cost. Plan embedding-model decisions with this in mind; pick one that’s likely to work for years, not one optimal for today by 2% margin.
Evaluation for RAG specifically uses different metrics than general LLM evaluation. Precision@K (what fraction of top-K retrieved docs are relevant), Recall@K (what fraction of relevant docs are in top-K), MRR (mean reciprocal rank), and end-to-end metrics like faithfulness (does the final answer cite the retrieved sources accurately) and answer relevance (does it actually address the question). Mature RAG systems track all of these; immature ones only check end-to-end and miss specific retrieval issues.
# RAG-specific evaluation metrics
class RagEvalCase:
query: str
relevant_doc_ids: list[str] # ground truth
expected_answer: str
def evaluate_retrieval(rag_system, eval_cases):
metrics = {"precision@5": [], "recall@5": [], "mrr": []}
for case in eval_cases:
retrieved = rag_system.retrieve(case.query, top_k=10)
retrieved_ids = [r['id'] for r in retrieved]
relevant = set(case.relevant_doc_ids)
retrieved_top5 = set(retrieved_ids[:5])
precision = len(relevant & retrieved_top5) / 5
recall = len(relevant & retrieved_top5) / len(relevant)
# MRR: 1/rank of first relevant
mrr = 0.0
for i, rid in enumerate(retrieved_ids):
if rid in relevant:
mrr = 1.0 / (i + 1)
break
metrics["precision@5"].append(precision)
metrics["recall@5"].append(recall)
metrics["mrr"].append(mrr)
return {k: sum(v)/len(v) for k, v in metrics.items()}Chapter 9: Memory and stateful context
Stateful context — information that persists across LLM calls — is what makes agents and long-running conversations work. Three main types: short-term memory (within a session, used across multiple turns), long-term memory (across sessions, persisted to a database), and learned memory (model fine-tuning or LoRA adapters that bake in patterns).
Short-term memory implementation. Within a session, append-only message history is the default. As history grows, apply trimming or summarization (Chapter 7). For longer sessions, hierarchical summaries — summarize older portions in batches, keep recent turns verbatim — preserve relevant detail while controlling token usage.
# Short-term memory pattern
class SessionMemory:
def __init__(self, max_recent=20, summary_threshold=50):
self.recent = []
self.summary = ""
self.max_recent = max_recent
self.summary_threshold = summary_threshold
def add_turn(self, role, content):
self.recent.append({"role": role, "content": content})
if len(self.recent) > self.summary_threshold:
old = self.recent[:self.max_recent // 2]
self.summary = self._update_summary(old)
self.recent = self.recent[self.max_recent // 2:]
def _update_summary(self, old_messages):
combined = self.summary + "\n" + format_messages(old_messages)
return llm_summarize(combined, max_tokens=500)
def get_context(self):
msgs = []
if self.summary:
msgs.append({"role": "system", "content": f"Earlier context: {self.summary}"})
msgs.extend(self.recent)
return msgsLong-term memory uses persistent storage. User preferences, prior conversation summaries, accumulated facts about the user — these live in a database (Postgres, Redis, dedicated vector store). On each new session, relevant memories are retrieved and included in context. The retrieval logic mirrors RAG but over a narrower personal corpus.
# Long-term memory schema (Postgres example)
CREATE TABLE user_memories (
id UUID PRIMARY KEY,
user_id TEXT NOT NULL,
memory_type TEXT NOT NULL, -- 'preference', 'fact', 'history-summary'
content TEXT NOT NULL,
embedding VECTOR(1536), -- pgvector for semantic search
created_at TIMESTAMP DEFAULT NOW(),
last_used_at TIMESTAMP,
importance FLOAT DEFAULT 0.5,
metadata JSONB
);
CREATE INDEX user_memories_embed_idx ON user_memories
USING hnsw (embedding vector_cosine_ops);
# On new session, retrieve relevant memories
def load_user_memories(user_id, current_query):
query_embedding = embed(current_query)
memories = db.query("""
SELECT content, importance
FROM user_memories
WHERE user_id = %s
ORDER BY embedding <=> %s
LIMIT 10
""", [user_id, query_embedding])
return memoriesMemory hygiene matters at scale. Memories accumulate over time; old or contradictory memories degrade context quality. Patterns: importance scoring (rate each memory’s significance), age decay (older memories fade unless reinforced), explicit removal (let users see and delete their memories), de-duplication (merge similar memories rather than accumulating duplicates).
Memory write logic deserves design attention. What triggers writing a new memory? Common patterns: explicit user statements (“remember that I prefer X”); inferred preferences from behavior (after many interactions, the system notes a pattern); critical facts during conversation (the user mentions a constraint that matters for future interactions). Over-eager memory creation produces noise; under-eager creation misses opportunities. Aim for high-precision memory writes — only commit things you’re confident about.
Memory contradiction handling is the subtle case. When new information contradicts existing memory (the user previously said X; now says Y), the system must resolve it. Patterns: recent overrides old by default (with timestamp tracking); ask the user to confirm if the contradiction is significant; flag for review if automated resolution is uncertain. Don’t silently keep both — that produces inconsistent agent behavior.
# Memory contradiction resolution
async def add_memory(user_id, content, memory_type):
existing = await find_similar_memories(user_id, content, threshold=0.85)
if existing:
# Possible contradiction; investigate
is_contradiction = await llm_check_contradiction(content, existing[0])
if is_contradiction:
# Newer information; mark older as superseded
await mark_superseded(existing[0]['id'])
# Or prompt user to confirm
return {"status": "needs_confirmation", "existing": existing[0]}
# Save the new memory
await store_memory(user_id, content, memory_type)
return {"status": "saved"}Memory privacy and user control are non-negotiable in 2026. Users should be able to see what’s stored about them; delete specific memories or all memories; opt out of memory features entirely. The combination of explicit user agency plus thoughtful default behavior is what’s needed for memory features to be trustworthy at scale. Regulatory frameworks (GDPR, CCPA) increasingly require this user agency anyway.
# Memory user controls
# Endpoint: list user's memories
GET /api/memories
[
{"id": "...", "content": "Prefers concise responses",
"created": "2026-05-01", "type": "preference"},
{"id": "...", "content": "Works at ACME Corp",
"created": "2026-04-15", "type": "fact"}
]
# Endpoint: delete a specific memory
DELETE /api/memories/<id>
# Endpoint: delete all memories
DELETE /api/memories
# Endpoint: opt out of memory feature
POST /api/memories/disable
# UI for the user to manage:
# - Settings page showing all memories
# - "Forget this" button per memory
# - Global "Disable memory" toggleChapter 10: Multi-turn conversation context
Multi-turn conversations are where context engineering complexity peaks. Each turn needs: the current user message; appropriate conversation history; relevant retrieved context; any tool results from prior turns; and updated state. Managing all this consistently across many concurrent sessions is the architectural challenge.
The standard pattern: a session object that holds all turn-spanning state. Each call assembles the prompt from the session object. The session is persisted between calls (Redis, database). Concurrency control ensures one session isn’t being modified by two requests simultaneously.
# Multi-turn conversation handling
import asyncio
class ConversationSession:
def __init__(self, session_id, user_id):
self.session_id = session_id
self.user_id = user_id
self.messages = []
self.retrieved_context = {}
self.tool_state = {}
self.lock = asyncio.Lock()
async def handle_turn(self, user_message):
async with self.lock: # prevent concurrent modification
# 1. Append user message
self.messages.append({"role": "user", "content": user_message})
# 2. Determine if we need fresh retrieval for this turn
if self._needs_retrieval(user_message):
self.retrieved_context = retrieve_relevant(user_message, self.user_id)
# 3. Assemble prompt
prompt = self._assemble_prompt()
# 4. Call LLM
response = await llm.chat(**prompt)
# 5. Handle tool calls if any
while response.tool_calls:
tool_results = await self._execute_tools(response.tool_calls)
self.messages.append({"role": "assistant", "content": None, "tool_calls": response.tool_calls})
self.messages.append({"role": "tool", "content": tool_results})
response = await llm.chat(**self._assemble_prompt())
# 6. Append final response
self.messages.append({"role": "assistant", "content": response.text})
# 7. Persist session
await self._save()
return response.textContext expiration is the underused subtlety. Some information is only relevant for a few turns (a piece of retrieved context for the immediate question), some is relevant for the whole session (user identity), some is permanent (long-term memories). Tracking expiration explicitly prevents stale context from polluting later turns.
Conversation summarization is a specific multi-turn pattern. For long conversations (more than 20-30 turns), retaining the full history hits context window limits. Periodic summarization — replace older portions with a concise summary, retain recent turns verbatim — keeps context manageable while preserving essential information. Done well, the user doesn’t notice; the agent maintains coherence across very long conversations.
# Hierarchical conversation summarization
class HierarchicalConversation:
def __init__(self, max_recent_turns=20, summary_chunk_size=10):
self.recent_turns = []
self.older_summaries = [] # list of summaries, each from N old turns
self.global_summary = "" # summary of summaries, when needed
def add_turn(self, role, content):
self.recent_turns.append({"role": role, "content": content})
if len(self.recent_turns) > self.max_recent_turns:
# Move oldest 10 to summary
to_summarize = self.recent_turns[:10]
self.recent_turns = self.recent_turns[10:]
summary = self._summarize_chunk(to_summarize)
self.older_summaries.append(summary)
# If too many summaries, consolidate
if len(self.older_summaries) > 5:
self.global_summary = self._summarize_summaries(self.older_summaries)
self.older_summaries = []# Context with expiration
class ContextItem:
def __init__(self, content, importance, expires_after_turns=None):
self.content = content
self.importance = importance
self.expires_after = expires_after_turns
self.turns_remaining = expires_after_turns
def tick(self):
if self.turns_remaining is not None:
self.turns_remaining -= 1
def expired(self):
return self.turns_remaining is not None and self.turns_remaining <= 0
# In session
def add_ephemeral_context(self, content, lifetime_turns):
self.context_items.append(ContextItem(content, 0.5, lifetime_turns))
def next_turn(self):
for item in self.context_items:
item.tick()
self.context_items = [i for i in self.context_items if not i.expired()]Chapter 11: Tool-use context patterns
Tool-using agents are now the dominant LLM application pattern in 2026. The agent reads a user query, decides which tools to call, executes them, observes results, decides next steps. Each tool result becomes additional context for subsequent LLM reasoning. Managing tool-related context is its own discipline within context engineering.
Tool definitions in the prompt. Every tool the agent can call must be described in the prompt — name, purpose, parameters, expected outputs. As the tool catalog grows (10, 50, 100 tools), the prompt overhead grows. Selective tool exposure — only including tools relevant to the current task — keeps prompt size manageable.
# Tool selection by relevance
def select_tools(user_query, all_tools, max_count=10):
"""Choose which tools to include in the prompt for this query."""
# Approach A: keyword/embedding matching
relevant = []
for tool in all_tools:
if relevance(tool.description, user_query) > 0.5:
relevant.append(tool)
# Approach B: LLM-based selection (slower but more accurate)
if len(all_tools) > 20:
relevant = llm_select_tools(user_query, all_tools, max_count)
return relevant[:max_count]
# In the prompt assembly
selected_tools = select_tools(user_query, available_tools)
prompt = {
"system": system_message,
"messages": history,
"tools": [t.to_dict() for t in selected_tools],
}Tool results as context. After a tool executes, the result becomes context for the LLM’s next call. Tool results can be large (a database query returning 1000 rows; a document search returning 50 articles). Truncation and summarization patterns apply: summarize large results before adding them as context; let the agent ask for more detail if needed.
# Tool result handling
async def execute_tool(tool_name, arguments):
raw_result = await tools[tool_name].execute(arguments)
# Truncate or summarize large results
if estimated_tokens(raw_result) > 2000:
# Option 1: truncate
truncated = raw_result[:2000] + f"\n[Result truncated, total {len(raw_result)} chars]"
return truncated
# Option 2: summarize
# summary = llm_summarize(raw_result, max_tokens=1000)
# return f"Summary of result: {summary}\n[Full result available via paginate_tool]"
return raw_resultMulti-tool sequences require careful state management. Tool A’s output becomes input to Tool B; the agent’s reasoning chains across multiple calls. Capture the full chain in context but don’t let it explode. Patterns: keep the most recent N tool calls in detail; summarize older ones; allow the agent to re-fetch specific results if needed.
Schema design for tool outputs is the under-emphasized lever. Tools that return well-structured, named-field outputs are dramatically more useful to LLMs than tools returning raw blobs. Structured outputs let the LLM reference specific fields (“the email field from the customer record”); blob outputs require parsing. Design tool schemas as carefully as you’d design APIs for human consumption.
Documentation in tool descriptions is what makes tools usable by LLMs. The description field of each tool is read by the LLM to decide when to call it. Vague descriptions (“Get information about something”) produce wrong tool selection; precise descriptions (“Get a customer’s record by their unique customer ID; returns name, email, account status, and creation date”) produce correct selection. Treat tool descriptions with the same care you’d give to API documentation that humans read.
Error handling in tool sequences deserves explicit design. When a tool call fails, what should the agent do? Patterns: retry once with the same args; retry with modified args based on error; try a different tool that achieves the same goal; give up gracefully and inform the user. The error response from the tool should guide the agent’s choice — specific error messages enable specific recovery; generic errors force generic responses.
# Tool error response design
# Bad: generic error
{"isError": True, "content": [{"type": "text", "text": "Error"}]}
# Good: actionable error
{"isError": True, "errorType": "not_found",
"content": [{"type": "text",
"text": "Customer with ID 'C-42' not found. Verify the ID is correct, " +
"or use search_customers if you don't have the exact ID."}]}
# The good version lets the LLM:
# - Recognize this is a recoverable error (errorType: not_found)
# - Take the suggested action (try search_customers)
# - Or escalate to user ("I couldn't find that customer; can you double-check?")Chapter 12: Observability for context systems
Observability for context engineering goes beyond standard application observability. You need to see: what prompt template was used; what context was assembled; what the LLM did with it; what the user perceived. The full chain from input to output, captured in detail.
# Context engineering tracing schema
# Per-request trace
{
"trace_id": "trc_abc123",
"timestamp": "2026-05-19T14:32:01Z",
"user_id": "user_42",
"session_id": "sess_xyz",
"request": {
"user_message": "...",
"metadata": {...}
},
"context_assembly": {
"prompt_template": "support-triage-v4",
"system_message_tokens": 245,
"history_messages": 8,
"history_tokens": 1247,
"retrieved_docs": [
{"doc_id": "kb_123", "score": 0.87, "tokens": 412},
{"doc_id": "kb_456", "score": 0.71, "tokens": 389}
],
"selection_rationale": "top 2 of 50 candidates by relevance",
"total_input_tokens": 3500
},
"llm_call": {
"model": "claude-opus-4-7",
"temperature": 0.0,
"max_tokens": 1024,
"latency_ms": 1247,
"input_tokens": 3500,
"output_tokens": 234,
"cost_usd": 0.029
},
"response": {
"text": "...",
"tool_calls": [...]
},
"user_feedback": {
"thumbs": "up",
"edit_distance": 0
}
}Dashboards on this data answer key operational questions: which prompt versions are performing well; which retrieval queries are returning low-relevance results; which sessions are hitting context limits; where cost is concentrated. The dashboards drive prioritization for context engineering improvements.
Continuous evaluation runs on production traffic. Sample N% of real interactions (with appropriate privacy controls); run them through the evaluation harness; compare to historical baselines. Alerts fire when accuracy drifts; investigation starts immediately. Without this, regressions accumulate silently between formal evaluations.
Three additional metrics worth tracking specifically for context engineering. Context utilization rate (what percent of included context the LLM actually uses in its response, measured via citation patterns); context-to-output ratio (input tokens divided by output tokens, often surprisingly high for poorly-tuned RAG); selection precision (of the candidates that selection rejected, how many would have improved the answer if included). These metrics expose subtle quality issues that overall accuracy metrics miss.
For agent-based workflows specifically, additional metrics matter. Step count per session (how many LLM calls before completion); tool call success rate (fraction of tool calls that returned useful results); reasoning depth (how often does the agent go beyond surface-level analysis); termination rate (how often does the agent complete the task vs give up). Each metric exposes a different failure mode; together they characterize agent quality comprehensively.
Cost-aware observability ties together cost engineering and observability. Every traced request has a cost; aggregate cost per feature, per user, per prompt version. The dashboard answers: which features are expensive; which users drive disproportionate cost; which prompt versions are cost-efficient. These insights drive optimization priorities far better than gut feel about where cost lives.
Debugging individual production cases is the other observability use case. When a user reports “the AI gave me a wrong answer,” investigation should answer: what was the user’s actual query; what context was retrieved; what context was selected; what was the LLM’s exact response; where in the chain did the wrong answer originate. Without traceable context engineering, this investigation is guesswork; with it, root cause is usually clear in minutes.
# Per-request debug view
def debug_trace(trace_id):
trace = load_trace(trace_id)
print(f"User query: {trace['request']['user_message']}")
print(f"\nRetrieved {len(trace['retrieved_docs'])} docs:")
for doc in trace['retrieved_docs']:
print(f" {doc['id']} (score: {doc['score']:.2f}, tokens: {doc['tokens']})")
print(f"\nSelected {len(trace['selected_docs'])} (selection rationale: {trace['selection_rationale']})")
print(f"\nFull prompt sent to LLM:")
print(trace['final_prompt'])
print(f"\nLLM response:")
print(trace['llm_response'])
# This view answers most "why did the AI say that" questions in seconds# Production sampling for continuous evaluation
async def handle_request(request):
response = await main_handler(request)
# Sample 1% of requests for eval (with PII handling)
if random.random() < 0.01:
await enqueue_for_eval({
"trace_id": response.trace_id,
"prompt_version": response.prompt_version,
"request": redact_pii(request),
"response": response,
"timestamp": datetime.now()
})
return response
# Background worker processes the queue
async def eval_worker():
while True:
item = await eval_queue.get()
judge_result = await llm_judge(item)
await store_eval_result(judge_result)
if judge_result['score'] < threshold:
await alert("Quality regression detected", judge_result)Chapter 13: Cost engineering for context-heavy workloads
Context engineering and cost engineering are tightly linked. More context means more tokens means more cost. Production AI systems with poor context discipline run at 3-10x the cost of well-engineered systems doing the same job. The cost optimization patterns are well-understood; applying them systematically is what separates expensive and cheap deployments.
The cost optimization stack. Layer 1: model selection — route to cheaper models when capability permits. Layer 2: prompt prefix caching — keep static portions of prompts cached for 90% discount on repeated content. Layer 3: context selection — include only what’s necessary, not everything available. Layer 4: response caching — for identical queries, return cached responses without LLM calls. Layer 5: batch processing — use batch APIs (50% discount) for non-urgent workloads.
# Cost optimization stack example
# Without optimization: $0.08 per request
# - Frontier model: $0.06
# - Long prompt: contributes most of input cost
# - No caching: every call full price
# - Synchronous: no batch discount
# With full optimization:
# - Routing: 70% of requests to small model ($0.005)
# Average becomes: 0.7 * $0.005 + 0.3 * $0.06 = $0.022 (72% reduction)
# - Prefix caching: 75% of input cost cached
# Frontier-model calls: $0.06 → $0.025 (60% reduction on those calls)
# Combined: ~$0.013 (84% total reduction)
# - Context selection: 30% reduction in average input tokens
# $0.013 → $0.009 (88% reduction)
# - Response caching: 20% of requests served from cache
# Effective: ~$0.007 (91% reduction)
# - Batch (where applicable): 50% off batch-eligible requests
# Effective for batch-friendly workload: ~$0.005 (94% reduction)Per-call cost budgeting is the discipline that prevents cost surprises. Each request type has an expected cost; the system rejects or routes elsewhere requests that would exceed the budget. This protects against runaway agents (loop calling expensive tools), prompt injection attacks (trying to inflate context), and silent regressions (a deploy that accidentally lengthened the prompt).
Cost attribution per user, team, and feature is necessary for chargebacks and optimization. Aggregate cost dashboards hide which use cases are expensive vs cheap. With per-feature attribution, you can identify the 20% of features driving 80% of cost; focus optimization effort there. Most prompt engineering platforms surface this attribution natively; self-hosted setups need to instrument it explicitly.
The cost-quality trade-off is the operating tension in production context engineering. Cheaper models, less context, less retrieval, fewer reasoning steps — all reduce cost but may reduce quality. The discipline is finding the right operating point per use case: high-stakes use cases tolerate higher cost for higher quality; low-stakes use cases optimize for cost. Don’t treat one operating point as universally right; calibrate per use case based on the cost of errors vs the cost of inference.
One under-emphasized cost optimization: prompt length itself. Sometimes prompts grow organically — added examples, clarifications, edge-case instructions — until they’re 3-5x longer than necessary. Periodically audit prompts for length; ask “what could we remove and still get the same accuracy on eval?” Often substantial sections can be removed; the eval set confirms quality holds. Shorter prompts mean lower cost on every call, lower latency, and easier maintenance.
The cost-engineering work is genuinely high-leverage in 2026. A team optimizing their context engineering deliberately can cut their AI inference bill by 60-85% over 6 months. For organizations running AI at scale, the savings are meaningful — six- or seven-figure annual reductions. The work is well-understood; the patterns documented above are not novel. What’s needed is application.
# Use-case-specific cost-quality tuning
# High-stakes: legal contract analysis
# - Use frontier model (Claude Opus, GPT-5.5)
# - Include extensive retrieved context
# - Run with chain-of-thought reasoning
# - Cost per call: $0.50-$2.00
# - Justified by stakes of errors
# Medium-stakes: customer support ticket triage
# - Use mid-tier model (Claude Sonnet, GPT-5)
# - Moderate context (top 5 similar tickets)
# - Cost per call: $0.05-$0.20
# Low-stakes: classification or routing
# - Use small model (Claude Haiku, GPT-5-mini)
# - Minimal context (just the input)
# - Cost per call: $0.005-$0.02
# - Errors recoverable; cost-optimized
# Each use case has its own cost-quality operating point
# Right operating point is determined by error cost, not by absolute cost# Per-call cost ceiling
@trace
async def handle_with_budget(request, max_cost_usd=0.50):
estimated = estimate_request_cost(request)
if estimated > max_cost_usd:
# Either reject or route to cheaper path
return await handle_with_cheap_model(request)
response = await main_handler(request)
actual_cost = response.cost_usd
# If actual exceeded estimate substantially, alert
if actual_cost > max_cost_usd * 1.5:
await alert(f"Cost overrun: estimated ${estimated}, actual ${actual_cost}")
return responseChapter 14: Vendor landscape — prompt platforms and frameworks
The prompt platform / context engineering tooling market has matured significantly in 2026. Several categories of vendors compete for different parts of the stack. Understanding the categories helps with build-vs-buy decisions.
| Category | Examples | Strength | When to Use |
|---|---|---|---|
| Full-stack platforms | LangSmith, Vellum, PromptLayer, Pezzo | End-to-end: library + eval + observability | Teams wanting one tool for everything |
| Evaluation specialists | Braintrust, Confident AI, Maxim | Strong eval and A/B testing | Teams with eval as the bottleneck |
| Observability specialists | Helicone, Langfuse, Arize Phoenix | Tracing and operational insight | Teams needing strong observability |
| Frameworks (open source) | LangChain/LangGraph, LlamaIndex, DSPy | Programmatic prompt composition | Code-first teams |
| Optimization tools | DSPy, MIPRO, AutoPrompt | Automatic prompt search/improvement | Advanced teams optimizing at scale |
| RAG-specialized | LlamaIndex, Haystack, Cognita | RAG pipeline tooling | Teams heavy on retrieval |
| Self-hosted/in-house | Build on top of open primitives | Full control, no vendor lock-in | Specific compliance or scale needs |
For most teams, the build-vs-buy decision favors buy on the platform layer. Building a prompt management platform from scratch is months of engineering for capability already available. Build only the parts that are differentiating (your specific selection logic, your business-specific retrieval) and use vendor platforms for the generic layers.
Open-source frameworks deserve specific attention. LangChain (and its successor LangGraph for orchestration) remains the most-used framework for AI applications, with broad integration support. DSPy takes a fundamentally different approach — programmatic prompt composition with built-in optimization. LlamaIndex specializes in RAG patterns. Each has trade-offs; the framework decision depends on team preferences (declarative vs imperative; orchestration-first vs prompt-first) and integration needs.
For self-hosted prompt platforms, several open-source options have matured. Langfuse provides observability and evaluation. Helicone offers tracing and analytics. Promptfoo specializes in eval. Combining these gives you a credible self-hosted stack without paying SaaS pricing. The trade-off is operational burden — you maintain the infrastructure yourself. For organizations with mature platform engineering, the self-hosted route is viable; for smaller teams, SaaS is usually faster.
One specific architectural decision: how tightly to couple to a vendor’s platform. Loose coupling (use the platform for storage and observability; keep your code agnostic) preserves portability but limits how much the platform helps. Tight coupling (use the platform’s runtime, prompt composition, and orchestration) is more productive but harder to migrate later. Most teams start loose and tighten as confidence in the chosen vendor grows.
Chapter 15: Anti-patterns and the 90-day context engineering plan
The patterns above describe what to do. This chapter covers what not to do — the anti-patterns that derail context engineering programs — and a 90-day plan for moving from ad-hoc prompts to systematic context engineering.
Anti-pattern 1: Prompts as strings in code. Putting prompt text directly in source files works for prototypes; breaks at scale. Different teams duplicate similar prompts; changes require code deploys; there’s no version history. Fix: prompt library from day one, even small projects.
Anti-pattern 2: No golden-set evaluation. “We’ll add evals when we have problems” — but by then you have a problem you can’t measure. Build a 50-100 case golden set on day one of any production AI feature; expand it over time.
Anti-pattern 3: Optimizing in production. Tweaking prompts based on individual production failures, without systematic eval, produces a random walk. The fix: every change goes through eval; production observation informs eval set expansion, not direct prompt changes.
Anti-pattern 4: Including everything. “More context can’t hurt” — but it can. Includes irrelevant context dilutes the LLM’s attention, increases cost, and sometimes degrades quality. Selection is a discipline; include what helps, not everything available.
Anti-pattern 5: No cost ceiling. Without per-request cost budgets, occasional pathological cases can cost orders of magnitude more than typical requests. Set ceilings; alert on overruns; investigate root causes.
# 90-day context engineering plan
# Days 1-30: Foundation
# - Pick a prompt management approach (Git-based or platform)
# - Create initial prompt library for highest-traffic feature
# - Build golden-set evaluation (50-100 cases)
# - Set up basic observability (traces of every LLM call)
# - Establish cost tracking per feature
# Days 31-60: Production discipline
# - Define prompt lifecycle (draft → eval → staged → prod)
# - Implement A/B testing for at least one prompt change
# - Add continuous eval on sampled production traffic
# - Build dashboards: accuracy, cost, latency, by feature
# - First retrospective on what's working
# Days 61-90: Optimization
# - Implement prompt prefix caching
# - Add model routing for cost (cheaper for simple cases)
# - Tune retrieval (chunking, embedding model, reranking)
# - Codify selection rules (token budget allocation)
# - Document patterns for new AI features
# Day 90+: Operate and scale
# - New AI features follow the discipline by default
# - Monthly metric reviews
# - Quarterly platform / vendor reviews
# - Continuous expansion of eval coverageThe 90-day plan is intentionally focused. Pick one high-value AI feature; apply the discipline there; harvest the platform components for subsequent features. Skipping this focus by trying to apply context engineering to everything at once produces shallow discipline applied broadly rather than deep discipline applied to important things.
Two additional anti-patterns worth flagging. First, the “prompt of doom” — a massive, multi-thousand-token prompt accumulated over months that nobody dares to refactor because nobody fully understands it. The fix is bite-the-bullet refactoring: methodically break the prompt into modular sections, write tests for each, gradually replace the monolith. The cost is engineering time; the benefit is a maintainable system going forward.
Second, the “shadow prompt library” — informal prompts shared in Slack messages or saved in personal notes that never make it into the official library. Over time, the shadow library diverges from official, and team members use different prompts for similar problems. The fix is making the official library so much better than the shadow alternative (faster to discover, easier to evaluate, properly versioned) that the shadow library naturally fades. Don’t try to ban shadow prompts by policy; out-compete them with quality.
For teams transitioning from ad-hoc prompts to context engineering, the cultural shift matters as much as the technical work. Engineers who got used to “I’ll just tweak the prompt” now need to follow library procedures, write evals, do code review. The initial friction can produce pushback; the discipline pays back over time but requires leadership patience through the transition. Frame the investment as professional maturation, not bureaucratic overhead.
Chapter 16: Frequently Asked Questions
Is prompt engineering still a useful term?
Yes for individual interactions and rapid prototyping. The skill of crafting an effective single prompt remains valuable. But for production systems at scale, context engineering is the larger frame; prompt engineering is one part of it.
How do I introduce context engineering practice to a team that’s never done it?
Lead with a concrete pain point — a recent production incident that better discipline would have prevented; a feature that took too long to ship because of prompt debugging; a cost spike from inefficient context. Use the pain to motivate the investment. Then implement the smallest version first (prompts in version control) and demonstrate the win. Each subsequent investment is easier once the first one is paid back.
How big should my prompt library be?
A small team starting with one AI feature might have 5-10 prompts. A mature platform serving many features has 50-200 prompts. Quality matters more than quantity; well-managed 50 prompts is better than chaotic 500.
How do I prevent prompt library bloat over time?
Retirement discipline. Each quarter, audit prompts by usage; retire ones not invoked in 90 days. Track ownership; un-owned prompts are candidates for retirement. The library size should reflect actual production use, not accumulated history. Old versions are accessible via git history if needed; they don’t need to be live deployments.
How do I structure prompts for high-stakes regulated outputs?
Structured outputs with strict validation. Force the model to output JSON matching a schema; validate every field before downstream use; flag any output that fails validation for human review. Include explicit safety instructions in the system message. Maintain a separate, more thorough evaluation set specifically for safety-critical aspects.
How do I decide between using a platform like LangSmith vs building in-house?
For most teams, use a platform. Building prompt management infrastructure from scratch is months of work for table-stakes capability. Build only if you have specific requirements (regulatory compliance requiring on-prem, extreme scale, integration with proprietary internal systems) that platforms can’t meet.
Should context engineering teams be embedded or central?
Depends on AI portfolio scope. For organizations with one or two AI features, embed context engineering competence within the feature team. For organizations with five or more AI features across teams, a central platform team that builds shared infrastructure is appropriate; feature teams use the platform with their domain-specific knowledge. The hybrid model (central platform + feature-team adopters) is most-common in practice.
How does context engineering evolve as models get better?
Three vectors. First, less prompt-engineering wordsmithing needed — better models follow simpler instructions more reliably. Second, more context can be productively used as context windows expand. Third, the operational discipline matters more, not less — better models exacerbate the consequences of context quality issues because everything else is dialed in. The skills shift from prompt-craft to systems-engineering as models improve.
What’s the right amount of context to include?
Enough to answer well; no more. The temptation is to include everything that might be relevant. Resist; the irrelevant context dilutes the signal and costs money. Measure: include only the top-K most relevant chunks; if quality degrades, increase K; if quality stays the same and cost rises, decrease K.
How should I think about prompt engineering as a career skill?
Still valuable but evolving. Pure prompt-writing skills are commoditizing as platforms automate the work. The high-value version is systems-engineering thinking applied to LLM applications — context engineering, evaluation infrastructure, observability, cost management. Teams hire for this combined skill set increasingly. Pure “prompt engineer” job titles are declining; “AI engineer” or “ML platform engineer” with context engineering competence is rising.
How do I handle prompts that need to update with new product features?
Build the prompt to reference live capabilities rather than hard-coding feature lists. A “current features” section in the prompt can be auto-populated from a feature registry; new features are added to the registry; the prompt picks them up automatically. This pattern decouples prompt updates from feature releases.
What’s the right granularity for prompt versioning?
Each meaningful logical change is a new version. Tweaking a single word: usually not worth a new version (just modify in place if not yet deployed; revert via git if deployed). Significant structural change: new version. Adding a new instruction: new version. As a rule, if you’d want to compare the old and new versions in evaluation, that’s a new version.
How do I balance prompt complexity vs maintainability?
Complex prompts often outperform simple ones in specific cases but degrade over time as complexity accumulates. The pragmatic guidance: start simple; add complexity only when evaluation shows it’s needed; refactor periodically to remove complexity that’s no longer load-bearing. The metric to optimize is “prompt clarity per unit of capability” — a simpler prompt achieving the same accuracy is strictly better than a complex one achieving the same accuracy.
How do I handle prompt deployments in a multi-region setup?
Same as code deployments — version-controlled, deployed via your CD pipeline, replicated across regions. Some prompts may have region-specific variants (different languages, locale-specific instructions); manage these as separate prompts rather than embedding region logic in a single prompt. Keep deployment lag between regions small (minutes, not hours) to avoid version inconsistencies.
How do I document prompts for new team members?
Each prompt should have: a one-paragraph description of its purpose; example inputs and outputs; known edge cases and how they’re handled; the evaluation set used to validate it; the owner team. New team members can read these docs to understand what each prompt does without reverse-engineering from the prompt text.
What about prompt injection attacks?
Prompt injection is the attempt to manipulate the LLM by including malicious instructions in user-provided content (e.g., a document the agent reads contains “ignore previous instructions and…”). Defenses: separate trusted system instructions from untrusted user content explicitly; use structured prompts (XML-like tags) to mark boundaries; validate model outputs before acting on them; constrain tool permissions narrowly. No defense is perfect; layered defense is the operational approach.
How do I A/B test prompt changes safely?
Sample 5-10% of traffic to the new prompt initially; compare key metrics over 24-48 hours; expand if metrics are good; roll back if metrics regress. Have a kill switch (config flag) to immediately route 100% back to the control. Don’t run A/B tests on tiny traffic; statistical significance requires enough sample size.
What’s the role of fine-tuning in context engineering?
Smaller than expected. Modern frontier models are good enough with good context that fine-tuning rarely makes sense for prompt-style work. Fine-tuning matters for: domain-specific style/tone consistency at high volume; cost optimization (fine-tune a smaller model to replace frontier API calls); legacy data formats not in pre-training. For most teams, fine-tuning is a Phase 3 optimization, not Phase 1.
How do I get started with context engineering on a small team?
Start with the basics: move prompts out of code into version-controlled files; build a small golden set (50-100 cases); set up basic tracing of LLM calls; establish a simple prompt review process. Small teams don’t need full platforms; Git-based workflows plus simple Python scripts can cover the foundational discipline. Scale up tooling as the team and AI portfolio grow.
How do I keep my context engineering current as model behavior changes?
Periodically re-run your full eval set when you switch models or when the model provider releases updates. Subtle behavior shifts can degrade your context engineering quality even on a model with the same name (vendors update models in place). Treat the model version as a tracked dependency; bump it deliberately rather than implicitly.
How does context engineering differ for B2B vs B2C applications?
The principles are the same; the operational concerns differ. B2B applications often have higher accuracy requirements, more complex permission models (which users see which context), and longer-tail use cases requiring more domain-specific evaluation. B2C applications have higher volume requiring cost optimization, more emphasis on consumer-facing latency, and broader inputs requiring more robust prompts. Tune the operating points accordingly.
What’s the difference between RAG and context engineering?
RAG is a specific technique for retrieving information to include in context. Context engineering is the broader discipline of managing all context sent to the LLM. RAG is one component (Layer 3 in our architecture); context engineering encompasses prompt templates, selection logic, history management, observability, and more. Many teams confuse the two and end up over-investing in RAG while under-investing in other layers.
How do I keep prompts in sync across multiple environments?
Deploy prompts as configuration, not code. Same prompt versions deployed to dev, staging, prod; environment flags control which version is active where. Promotion from dev → staging → prod follows the same pipeline as code (PR, review, automated tests). This makes prompt deployment as routine as code deployment.
What’s the relationship between context engineering and RAG?
RAG is one technique within context engineering. RAG handles the retrieval portion — fetching relevant information from external sources. Context engineering is the broader discipline of managing all the information sent to the LLM, including but not limited to RAG. A complete system uses RAG for retrieved context plus conversation history plus user metadata plus tool definitions, all assembled by the context engineering layer.
How do I evaluate context engineering quality, not just prompt quality?
Measure outcomes that depend on context, not just outputs. For RAG: precision@K and recall@K on retrieval; accuracy of the final answer; faithfulness (does the answer cite the retrieved sources accurately). For conversation: turn-level satisfaction; task completion rate. For agents: success rate on full multi-step tasks. Different layers need different metrics.
How do I handle multilingual context?
Modern frontier models handle major business languages well. For multilingual applications: include language information in the prompt; ensure retrieval works across languages (multilingual embeddings or translation at retrieval time); evaluate per-language separately (don’t aggregate metrics that hide per-language regressions); maintain language-specific golden sets.
How does context engineering change for streaming responses?
The input side is the same — context assembly happens before the LLM call regardless. The output side is where streaming changes things: incremental rendering for UX, but also enabling early termination if quality signals indicate the response is going off-track. Most context engineering platforms support streaming natively; nothing special needed on the context side.
What’s the simplest first step for a team that’s never done context engineering?
Move your prompts out of code strings into version-controlled YAML files. That single step — taking maybe a few hours — unlocks all subsequent discipline. From there, build a small golden set and add evaluation. From there, add observability. The discipline compounds; the first step is enough to start.
What if my model provider changes pricing or capabilities?
Build for portability where you can. Abstract the model provider behind a thin layer so swapping models is a config change, not a rewrite. Keep prompts as much as possible model-agnostic (avoid heavy reliance on provider-specific features unless necessary). The frontier model landscape evolves quarterly; portability matters.
How does context engineering interact with agent frameworks like LangGraph?
Complementary. Agent frameworks orchestrate multi-step workflows (decide when to call tools, manage state machines, handle retries). Context engineering manages what information goes into each LLM call within those workflows. A mature production agent uses both: a framework for orchestration; context engineering for the per-call content.
How do I think about caching strategies for context engineering?
Three layers of caching. Prompt prefix caching (provider-side; cache static parts of prompts for repeated calls). Response caching (your side; cache LLM responses for identical queries). Retrieval caching (your side; cache embedding lookups for common queries). Each saves cost differently; combine all three for maximum impact. Watch cache invalidation carefully — stale data is worse than no cache.
How do I deal with prompt content that needs to be confidential?
Some prompts contain proprietary business logic, security-sensitive instructions, or competitive intelligence. Treat the prompt library with appropriate access controls — not every engineer needs to see every prompt. For highly-sensitive prompts, encryption at rest plus access logging is standard. Document who can read, modify, and approve changes to each prompt.
How do I handle prompts in regulated industries?
Add compliance-specific controls: audit logging of every prompt and response; PII redaction in logs; access controls on the prompt library (who can edit production prompts); change approval workflows; periodic compliance reviews of prompts in use. The underlying patterns are the same; the governance overhead is higher.
What’s the relationship between context engineering and DSPy?
DSPy takes a programmatic approach — you write Python code describing what the prompt should do (signatures, modules), and DSPy compiles this into actual prompt text optimized for the target model. The DSPy approach automates parts of prompt engineering that other approaches treat as manual work. For teams comfortable with declarative programming styles, DSPy can be highly productive; it’s less popular than LangChain but has strong technical foundations.
Closing thoughts
Context Engineering 2026 is a mature discipline with proven patterns. The shift from artisan prompt-writing to systematic context engineering is essentially complete for serious teams. The patterns documented in this guide — prompt libraries, evaluation infrastructure, context selection, observability, cost engineering — are now table stakes for production AI work.
The opportunity for engineering teams in 2026 is execution rather than invention. The methodology is documented; the tooling is available; the case studies exist. What remains is doing the work: building the library, establishing the discipline, instrumenting the systems, training the team. Teams that invest in context engineering reduce production incidents, ship features faster, and run AI workloads at meaningfully lower cost. Teams that don’t continue paying the tax of ad-hoc prompt management indefinitely.
For organizations starting their context engineering journey in 2026, the path forward is clear. Pick one high-traffic AI feature; apply the 90-day plan; harvest the patterns and platform for subsequent features. Within 6-12 months, the discipline spreads across the AI portfolio and context engineering becomes invisible infrastructure rather than a struggling initiative. The technology is ready; the practices are documented; the rest is organizational will to invest.
One reflection on the broader trajectory of LLM application engineering. The progression from “interesting demo” (2022) to “early production deployments” (2023-2024) to “systematic engineering discipline” (2025-2026) mirrors how other software disciplines matured. Web development went through the same arc, as did data engineering, as did ML platform engineering. Each discipline initially had artisans doing remarkable individual work; then platforms emerged that codified the work; then organizations adopted the platforms and the work became infrastructure. Context engineering is currently in the platform-adoption phase. The teams investing now are positioned for the next phase.
Looking ahead to 2027 and beyond. Context engineering will continue to subsume more of what was previously prompt engineering. Automated context optimization (the system learning what context works best for what query type) will become more capable. Context windows will continue to grow but selection will remain important — bigger doesn’t mean unlimited. New abstractions will emerge for managing multi-agent contexts where each agent’s context affects others. The fundamentals documented in this guide will remain relevant; new techniques will be added on top.
For engineering teams investing in context engineering today, the work compounds. The platform you build, the evaluations you author, the patterns you internalize — these don’t depreciate as models evolve. The specific prompts may need updating; the discipline persists. Teams that establish context engineering as core competence in 2026 are positioned for whatever the AI landscape looks like in 2028 and beyond. Good luck with your context engineering deployment going forward.