Context Engineering in Production: Treating the Agent Context Window as a Resource to Schedule

The Core Problem: Agent Failures Often Stem from Context Chaos, Not Model Weakness

When building LLM agents, many teams first focus on model selection, tool calling, and prompt templates. But in real business scenarios, the hardest problem is usually not “can the model call a tool,” but what exactly the model sees at each step.

An agent’s context typically includes more than just user input. It can contain:

• System prompts and developer instructions
• Multi-turn message history
• Available tool lists and descriptions
• Tool call results
• RAG retrieval chunks
• User preferences, project rules, and permission info
• Runtime state, plans, to-do items, and error logs

If this information is crudely concatenated, it might work for a demo in the short term, but in the long run, it leads to four types of problems: context pollution, attention dilution, cost explosion, and unexplainable behavior. This is why Context Engineering is evolving from a “prompt optimization trick” into core infrastructure for agent engineering.

Anthropic’s 2025 engineering article called context a “critical but finite resource” for agents and defined Context Engineering as the strategy for selecting and maintaining the optimal set of tokens at inference time. LangChain’s documentation also lists “not passing the correct context to the LLM” as one of the primary reasons for agent unreliability. This article treats Context Engineering not as a conceptual slogan, but as a design problem for a production system that can be implemented.

Core Principles: The Context Window as a Production Resource

1. More Context is Not Always Better

An LLM’s context window looks like a capacity limit, but it’s more like an attention budget. As input grows longer, the model must establish relationships across more tokens. Noise, duplicate content, and stale intermediate states all compete for attention.

In production agents, common mistakes include:

• Keeping the entire chat history forever
• Appending every tool call result verbatim
• Stuffing in Top-K retrieval results directly
• Exposing all tools to the model
• Mixing long-term memory, short-term state, and runtime dependencies

These practices make the model “seem to know a lot,” but when it needs to make a decision, the critical signals get drowned out.

2. The Goal of Context Engineering is High Signal Density

Think of context building before each LLM call as a function:

type ContextInput = {
  userRequest: string;
  systemPolicy: string;
  shortTermState: AgentState;
  longTermMemory: MemoryStore;
  retrievedDocs: RetrievedChunk[];
  toolResults: ToolResult[];
  runtimePermissions: PermissionScope;
};

type ModelContext = {
  instructions: string;
  messages: Message[];
  tools: ToolSpec[];
  evidence: EvidenceBlock[];
  workingNotes: string;
  outputContract: JsonSchema;
};

function buildModelContext(input: ContextInput): ModelContext {
  const evidence = selectHighSignalEvidence(input.retrievedDocs);
  const compactedHistory = compactConversation(input.shortTermState.messages);
  const toolSummary = summarizeToolResults(input.toolResults);
  const tools = selectAllowedTools(input.runtimePermissions, input.userRequest);
  return {
    instructions: renderInstructions(input.systemPolicy, input.runtimePermissions),
    messages: compactedHistory,
    tools,
    evidence,
    workingNotes: input.shortTermState.currentPlan,
    outputContract: buildOutputSchema(input.userRequest),
  };
}

The goal of this function is not “put as much in as possible,” but to give the model the minimum but sufficient set of information for the current step.

3. Context Should Be Split into at Least Three Categories

From an engineering perspective, it’s recommended to split context into three layers, rather than concatenating everything into one big prompt:

• Model Context: What the model sees in this call, including instructions, messages, tools, output format, and retrieval evidence.
• Tool Context: Local objects, user identity, permissions, database connections, and runtime dependencies accessible during tool execution. The OpenAI Agents SDK documentation specifically emphasizes that the runtime context object itself is not sent to the LLM; it’s the application context used by tools, lifecycle hooks, and local code.
• Lifecycle Context: Processing logic between model calls and tool calls, including summarization, trimming, approval, logging, retries, error compression, and security checks.

If these three types of context are not separated, two problems commonly arise: first, putting sensitive information that the model shouldn’t see into the prompt; second, mistaking dependencies that belong only to local execution for model knowledge.

Engineering Implementation: From “Prompt Concatenation” to a “Context Scheduler”

1. Establish a Context Builder for Each Model Call

In production systems, business code should not manually concatenate prompts everywhere. Instead, establish a unified Context Builder. It is responsible for five things:

1. Selecting system instructions based on task type
2. Selecting message history based on current state
3. Selecting tools based on permissions and task
4. Selecting evidence and memory based on the question
5. Compressing and degrading based on token budget

A simple context budget can be allocated like this:

Context Part	Suggested Use	Risk
System / Developer Instructions	Behavioral boundaries, role, output rules	Too long becomes hard-to-maintain implicit code
User Request	Current goal and constraints	Can be polluted by old goals in multi-turn conversations
Short-Term History	Recent decisions and conversation continuity	Keeping it verbatim causes rapid expansion
Retrieved Evidence	External knowledge and project data	Top-K noise can interfere with judgment
Tool Results	External execution feedback	Raw logs usually have low token density
Long-Term Memory	Preferences, stable facts, project rules	Stale memory leads to incorrect personalization

2. More Tools is Not Always Better

Tool descriptions themselves consume context. More importantly, if tools have overlapping functionality, the model’s tool selection behavior becomes unstable.

A more reliable approach is:

• Only expose tools needed for the current task
• Use verb + object naming, e.g., search_project_docs, create_gmail_draft
• Write clear usage boundaries and failure conditions in tool descriptions
• Return structured summaries from tool outputs, not large raw logs
• Add approval or policy checks for high-risk tools

This aligns with the 12-Factor Agents principle of “own your context window” and “tools are just structured outputs”: an agent is not an infinitely free loop, but a set of model decision points wrapped by ordinary software control flow.

3. Long Tasks Need Compression, Notes, and Sub-Task Isolation

Long tasks are most prone to context rot: tool outputs, error retries, and intermediate discussions from earlier rounds accumulate, and the model later loses focus on the main point.

Three common handling methods:

• Compaction: When the message history approaches a threshold, compress the history into a high-fidelity summary, retaining the goal, constraints, completed actions, unresolved issues, and key evidence.
• Structured Notes: Have the agent write long-term valid information into structured notes, e.g., plan.md, decisions.md, open_issues.md.
• Sub-agent Isolation: For complex research or code migration tasks, let a sub-agent explore independently and only return the compressed result, preventing the main agent’s context from being polluted by excessive detail.

Compression is not simple summarization; it requires a clear strategy for what to keep and discard:

compaction_policy:
  keep:
    - user_goal
    - hard_constraints
    - accepted_decisions
    - unresolved_errors
    - source_links
    - next_actions
  discard:
    - duplicate_tool_logs
    - obsolete_intermediate_outputs
    - failed_attempt_details_after_summary
    - irrelevant_small_talk

4. RAG Should Evolve from “Top-K Retrieval” to “Evidence Scheduling”

In agent scenarios, RAG shouldn’t just be vector retrieval for Top-K. A better approach is to categorize evidence into different types:

• Factual Evidence: Official documentation, papers, specifications
• Project Evidence: Code, configuration, API descriptions
• User Evidence: Current requirements, preferences, historical decisions
• Runtime Evidence: Logs, metrics, tool return values

Each type of evidence should have a source, timestamp, confidence level, and scope of applicability. The model doesn’t necessarily need to see the original text of all evidence in its final answer, but the Context Builder must know why it selected this evidence.

Applicable Scenarios

Context Engineering is particularly suitable for the following scenarios:

• Coding Agent: Needs to understand project structure, test commands, interface constraints, and historical modification decisions
• Enterprise Knowledge Base Q&A: Needs to handle permissions, timeliness, citation sources, and output format simultaneously
• Customer Service Agent: Needs to integrate user profiles, order status, historical tickets, and business rules
• Data Analysis Agent: Needs to maintain task continuity across large tables, query results, and intermediate charts
• Multi-Agent Research Systems: Needs to isolate exploration contexts and compress results for the main agent

For one-off classification, summarization, or simple Q&A, a full Context Engineering platform might be overkill. But as soon as a task involves multiple turns, tool calls, long histories, permission control, and traceable delivery, context management should be designed upfront.

Common Misconceptions

Misconception 1: Treating the Context Window as a Database

The context window is not a database; it’s the working memory for the model’s current decision. Long-term facts should be stored in databases, files, vector stores, or memory systems, and retrieved on demand.

Misconception 2: Believing Long-Context Models Eliminate the Need for Retrieval and Compression

Long-context models reduce engineering complexity, but that doesn’t mean context quality can be ignored. The larger the window, the more you need to handle noise, duplication, and stale information.

Misconception 3: Exposing All Tools to the Model at Once

The more tools there are, the larger the selection space, and the higher the risk of incorrect calls and prompt injection. Production systems should select tools based on task, permissions, and phase.

Misconception 4: Summarization Only Aims for Brevity

The goal of summarization is not brevity, but actionability. A good compression result should let the agent know the current goal, completed work, key evidence, risks, and next steps.

Misconception 5: Ignoring Context Security Sources

When agents read web pages, emails, documents, code comments, and tool return values, they encounter untrusted content. 2026 research on context-aware prompt injection also emphasizes that attackers can use dynamic context to influence agent decisions. Therefore, context systems need to record sources and trust boundaries.

Go-Live Checklist and Monitoring Metrics

Before going live, check at least the following eight items:

• Context Budget: Does each context part have a token limit and degradation strategy?
• Evidence Selection: Do retrieval results have a source, timestamp, confidence level, and deduplication?
• Tool Trimming: Are only the tools needed for the current task exposed?
• Permission Isolation: Is the local runtime context separated from the LLM-visible context?
• Compression Strategy: Do long tasks have compaction, notes, and recovery mechanisms?
• Observability: Is the context summary, tool list, and token usage logged for each model call?
• Security Boundaries: Is untrusted context flagged, and is its impact on high-risk tool calls limited?
• Regression Evaluation: Is there a fixed set of tasks to evaluate success rate, cost, and latency before and after context trimming?

Key metrics to monitor:

Metric	Meaning
context_tokens_per_call	Number of context tokens per model call
evidence_utilization_rate	Percentage of evidence cited or influencing the answer
tool_result_compression_ratio	Compression ratio of tool results
stale_memory_hit_rate	Percentage of hits on stale memory
context_rebuild_latency	Time taken to build the context
agent_recovery_success_rate	Task success rate after compression or interruption recovery

References

1. Anthropic Engineering, “Effective context engineering for AI agents”, 2025-09-29: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
2. LangChain Docs, “Context engineering in agents”: https://docs.langchain.com/oss/python/langchain/context-engineering
3. OpenAI Agents SDK, “Sessions”: https://openai.github.io/openai-agents-python/sessions/
4. OpenAI Agents SDK, “Context management”: https://openai.github.io/openai-agents-python/context/
5. HumanLayer, “12-Factor Agents”: https://github.com/humanlayer/12-factor-agents
6. Seyedmoein Mohsenimofidi et al., “Context Engineering for AI Agents in Open-Source Software”, arXiv 2025: https://arxiv.org/abs/2510.21413
7. Shihao Weng et al., “ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection”, arXiv 2026: https://arxiv.org/abs/2605.03378
8. Shijie Xia et al., “Diagnosing and Mitigating Context Rot in Long-horizon Search”, arXiv 2026: https://arxiv.org/abs/2606.29718