Context windows, memory and why AI agents forget

An AI agent that was sharp for the first ten steps starts repeating itself, ignores an instruction you gave at the start, or redoes work it already finished. It looks like the agent forgot. It did, in a very literal way: the model has no memory between calls, only the context window you send it each time. Once you understand how context windows and memory work for AI agents, most of these failures become engineering problems with ordinary fixes.

The model has no memory, only context

A model API is stateless. Every call sends the full input: system instructions, tool definitions, the conversation so far and any tool results. The model reads all of it, produces the next output, and keeps nothing. The next call starts from zero and only "remembers" what you send again.

The context window is the maximum amount of text, measured in tokens, that fits in one call. Windows have grown large, but two problems remain even when everything fits:

• Attention is uneven. Models tend to follow the start and end of a long context better than the middle. An instruction buried under thirty tool results gets less weight than the same instruction near the end.
• Cost and latency scale with input. Every turn resends the whole history. A long agent run pays for the early turns again and again.

So the question to design around is "what does the model need to see right now to take the next step well?"

Where the context budget goes

In a typical agent loop, like the one in building a tool-using AI agent in TypeScript, the budget splits into a few parts:

1. System instructions. Fixed per run. Keep them tight.
2. Tool definitions. Every registered tool's name, description and schema, on every call. Fifty tools cost a lot of context before any work starts.
3. Conversation history. Grows every turn.
4. Tool results. Usually the biggest part, and the fastest-growing.

Most "the agent forgot" bugs come from parts 3 and 4 crowding out part 1.

Trim at the source

The cheapest fix is to put less in. Design tools to return what the model needs and nothing more:

• Return the matching lines of a file, with line numbers, instead of the whole file.
• Return "12 tests passed, 2 failed" plus the two failures, instead of the full test log.
• Paginate search results and let the model ask for more.
• Cap every tool result at a fixed size, and say in the result when it was truncated, so the model knows there's more.

A tool that returns a compact, structured result helps on every turn after it, because that result is resent every time.

Summarize old turns

For long runs, keep the recent turns verbatim and replace older ones with a summary. A separate, cheaper model call can write it: what the goal is, what's been tried, what was found, what's left. Then the context for each call is system instructions, the summary and the recent turns, fitted to a budget:

type Message = { role: "system" | "user" | "assistant" | "tool"; content: string };

// Rough heuristic; use your provider's token counts when you need exact numbers.
const approxTokens = (m: Message) => Math.ceil(m.content.length / 4);

export function buildContext(
  system: Message,
  history: Message[],
  summary: string | null,
  budget: number,
): Message[] {
  const head: Message[] = [system];
  if (summary) head.push({ role: "user", content: `Summary of earlier work:\n${summary}` });

  let used = head.reduce((n, m) => n + approxTokens(m), 0);
  const recent: Message[] = [];
  for (let i = history.length - 1; i >= 0; i--) {
    const cost = approxTokens(history[i]);
    if (used + cost > budget) break;
    recent.unshift(history[i]);
    used += cost;
  }
  return [...head, ...recent];
}

One gotcha: many tool-calling APIs reject a history where a tool result appears without the assistant message that requested it, or the other way round. When you cut the history, cut on a turn boundary so each tool call stays paired with its result.

Summaries lose detail, so summarize the things the model can look up again (file contents, search results) more aggressively than decisions and constraints (what the user asked for, what's been ruled out).

Retrieval memory for what doesn't fit

Some knowledge is too big to carry in context but needs to be available: past conversations, project documentation, earlier runs. Store it outside the model and retrieve the relevant pieces when needed. That's the same machinery as retrieval-augmented generation: chunk, embed, store, and search by similarity. If you're already on Postgres, vector search with pgvector is the simplest place to keep it.

Retrieval can be automatic, where your code fetches related memories before each call, or a tool, where the model calls search_memory when it decides it needs something. My default is a tool for agents and automatic retrieval for chat, where the user's message is the natural query.

Keep state outside the model

The most reliable memory is plain application state. Anything the agent must not lose belongs in a database or file, with the model reading it back when needed:

• The task plan. A checklist of steps with their status. The agent marks items done through a tool, and the current list goes into every call.
• Decisions and constraints. "Store this in the existing Postgres database" stored once, injected every turn, and never at risk of being summarized away.
• Artifacts. Files written, records created, ids returned. The agent can query them instead of remembering them.

This also makes runs resumable. If a run crashes at step 14, a new run can read the plan and pick up where it stopped, instead of starting over with an empty context.

The short version

Agents forget because each call only sees what you send. Keep instructions short, make tools return less, summarize old turns on clean boundaries, retrieve long-term knowledge on demand, and put anything that must survive into real application state. When an agent goes off track, read the exact context it saw on that turn first. The cause is almost always visible there. For the safety side of long runs, see guardrails for AI agents in production.