External Memory, Hierarchical Recall & KV Cache Optimization

When an autonomous agent finishes a task and terminates its process, every token in its active context window is discarded. If the same agent is launched five minutes later to fix a related bug in the same codebase, starting from a blank context window forces it to re-read files, re-discover environment settings, and repeat previous diagnostic dead ends.

Building persistent autonomous systems requires bridging in-memory working context with persistent external storage. However, naively dumping previous chat transcripts or thousands of database rows directly into the prompt inflates token consumption, poisons attention, and evicts prompt cache lines.

Production context engineering solves this with a Hierarchical Memory Architecture: an orderly pipeline spanning in-memory KV caches, transactional episodic logs, and indexed semantic stores.

Imagine a detective solving cases in a busy precinct.

Their working memory is their immediate desk notepad: it holds active phone numbers, suspects' names, and notes from today's interviews. It is super fast to glance at, but it runs out of space by dinner.
Their episodic memory is the precinct case file in the filing cabinet: stamped folders labeled with dates, case numbers, and exact transcripts of every past interrogation.
Their semantic memory is the city-wide criminal database: indexed by fingerprints, vehicle types, and modus operandi.

When a new case arrives, the detective does not dump 40 heavy filing cabinets onto their small desk. They run a quick query in the database, pull out the two relevant index cards from previous cases, clip them to today's notepad, and get to work.

1. The Three-Tier Memory Hierarchy

Modern computing systems achieve high throughput and massive storage through memory hierarchies: L1/L2/L3 CPU caches, main DRAM, and NVMe SSDs. Agent memory architectures adopt an identical structure:

+--------------------------------------------------------------------------+
|  Tier 1: L1 Working Memory (In-Context KV Cache)                         |
|  - Storage: GPU VRAM / Host RAM                                          |
|  - Capacity: 8k - 128k tokens                                            |
|  - Latency: < 50ms per token generation step                             |
|  - Retention: Single session execution lifecycle                        |
+------------------------------------+-------------------------------------+
                                     ^
                                     | Recall (Top-K) / Commit (Summary)
                                     v
+--------------------------------------------------------------------------+
|  Tier 2: L2 Episodic Memory (Transactional Store)                        |
|  - Storage: Local SQLite / PostgreSQL WAL                                |
|  - Capacity: 100k - 10M tokens                                           |
|  - Latency: 5ms - 50ms lookup                                            |
|  - Retention: Multi-session history, audit logs, raw execution records  |
+------------------------------------+-------------------------------------+
                                     ^
                                     | Async Extraction / Consolidation
                                     v
+--------------------------------------------------------------------------+
|  Tier 3: L3 Semantic Memory (Vector & Graph Index)                       |
|  - Storage: Qdrant / LanceDB / pgvector + Knowledge Graph                |
|  - Capacity: 100M+ tokens                                                |
|  - Latency: 50ms - 250ms similarity search + rerank                      |
|  - Retention: Permanent organizational knowledge, invariants, heuristics |
+--------------------------------------------------------------------------+

Tier Specifications

L1 Working Memory: The tokens physically present in the LLM's active request payload ( $W_{\text{active}}$ ). Governed by the token budget manager. Access is instantaneous during inference because tokens are mapped to GPU KV cache pages.
L2 Episodic Memory: A structured, transactional relational store (typically SQLite or PostgreSQL). It records full execution sessions: task IDs, exact timestamps, tool calls, tool responses, and exit codes. It acts as the write-ahead log (WAL) of agent execution.
L3 Semantic Memory: A vector database coupled with a relational or graph index. Asynchronous background jobs periodically scan L2 episodic logs, extract persistent facts (e.g. "Repository uses Node 20.10 with pnpm 8"), embed them, and index them.

2. KV Cache Preservation: The Memory Injection Trap

Where should external memories be inserted into the prompt? This architectural decision directly impacts prompt cache efficiency and inference costs.

Predict First

If an agent retrieves 3 relevant historical facts (250 tokens) from external memory at turn 4, where should those tokens be injected in the API payload to maintain maximum KV cache hit rates?

The Cache Line Invalidation Mechanics

Recall how modern inference engines evaluate prefix caches. The KV cache matches an incoming sequence $T = [t_1, t_2, \dots, t_N]$ against stored GPU memory blocks by hashing prefixes:

Static Prefix (Cached):
[t1: System Prompt][t2: Format Rules][t3: Tool Schemas] ----> KV Cache Hit! (100%)

Case A: Injecting Memory into System Prompt (Anti-Pattern):
[t1: System Prompt][t_MEM: Recalled Facts][t2: Format Rules] -> Cache Miss at t_MEM!
Result: Entire prompt must be recomputed from scratch on every turn.

Case B: Injecting Memory into Dynamic User Block (Production):
[t1: System Prompt][t2: Format Rules][t3: Tool Schemas] ----> KV Cache Hit! (100%)
                   \---------------------------------/
                               Cached Prefix
[t4: Pinned Goal][t_MEM: Recalled Facts][t5: Turn History] -> Only tail computed

By enforcing that external memories are injected strictly downstream of the immutable prefix, production agents maintain cache hit rates exceeding 90% across long-running multi-turn operations.

3. Two-Stage Retrieval: Vector Search + Cross-Encoder Reranking

When querying semantic memory, naive vector search often retrieves irrelevant chunks that happen to share superficial keywords or stylistic embeddings. Injecting 10 noisy vector chunks into context wastes 3,000 tokens and confuses the model's reasoning.

Production architectures employ a Two-Stage Retrieval Pipeline:

+-------------------------------------------------------------------------+
|                              Query Intent                               |
|        "Fix JWT expiration drift in auth/session.py with pytest"         |
+-----------------------------------+-------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                  Stage 1: Bi-Encoder Vector Search                      |
|  - Fast approximate nearest neighbor search (HNSW index)               |
|  - Scans 50,000 embedded memory chunks                                  |
|  - Latency: 12ms                                                        |
|  - Output: Top 25 candidate chunks (6,250 tokens)                      |
+-----------------------------------+-------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                Stage 2: Cross-Encoder Reranking (BGE-Reranker)          |
|  - Computes all-to-all cross-attention between Query and Candidate      |
|  - Eliminates semantic false positives                                  |
|  - Latency: 35ms                                                        |
|  - Output: Top 2 highly factual memories (480 tokens)                   |
+-----------------------------------+-------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                      Active Context Injection                           |
|       Injected into dynamic turn block with 92% token savings           |
+-------------------------------------------------------------------------+

By filtering 25 candidates down to the top 2 verified memories, context consumption drops from 6,250 tokens to 480 tokens (a 92.3% reduction) while increasing signal precision.

4. Asynchronous Memory Consolidation

How do raw execution turns transform into persistent semantic knowledge? In human cognition, consolidation occurs during sleep: daily episodic experiences are transferred from the hippocampus to the neocortex as generalized principles.

In autonomous agents, Memory Consolidation runs as an out-of-band asynchronous worker:

[Agent Turn Execution] ---> Writes to SQLite Episodic WAL (L2)
                                       |
                                       v
                        [Trigger: Session Complete or Cron]
                                       |
                                       v
                        [Consolidation Worker (Haiku / Mini)]
                        Extracts:
                        1. Architecture rules ("Uses pnpm, not npm")
                        2. Verified bug root causes
                        3. Flaky test workarounds
                                       |
                                       v
                        [Upsert into LanceDB / Qdrant (L3)]

The consolidation worker strips all ephemeral output—shell progress bars, compile warnings, and HTTP headers—and stores only compact, declarative statements.

5. Implementation: Hierarchical Memory Engine with SQLite & Recall

Below is an executable Python implementation of a hierarchical memory manager. It writes session events to an L2 SQLite transactional store, extracts persistent invariants, executes query-based retrieval, and formats memory injection blocks that preserve prefix caching boundaries.

Run it yourself

import json
import sqlite3
import math
from dataclasses import dataclass
from typing import List, Dict, Any, Optional

@dataclass
class MemoryRecord:
  memory_id: str
  task_id: str
  category: str
  content: str
  relevance_score: float = 0.0

class HierarchicalMemoryManager:
  """
  Manages multi-tier agent memory:
  - L1: Working Context (Token bounded)
  - L2: Episodic Store (SQLite transactional memory)
  - L3: Semantic Store (Retrieval & Reranking)
  """
  def __init__(self, db_path: str = ":memory:"):
      self.conn = sqlite3.connect(db_path)
      self._setup_schema()

def _setup_schema(self) -> None:
      with self.conn:
          self.conn.execute("""
              CREATE TABLE IF NOT EXISTS episodic_store (
                  id TEXT PRIMARY KEY,
                  task_id TEXT,
                  category TEXT,
                  content TEXT,
                  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
              )
          """)

def record_episodic_event(self, mem_id: str, task_id: str, category: str, content: str) -> None:
      """Writes execution events directly to L2 transactional store."""
      with self.conn:
          self.conn.execute(
              "INSERT INTO episodic_store (id, task_id, category, content) VALUES (?, ?, ?, ?)",
              (mem_id, task_id, category, content)
          )

def retrieve_semantic_memories(self, query: str, top_k: int = 2) -> List[MemoryRecord]:
      """
      Stage 1 & 2 retrieval simulation:
      Scans L2 episodic records and scores lexical-semantic relevance against query.
      """
      cursor = self.conn.cursor()
      cursor.execute("SELECT id, task_id, category, content FROM episodic_store")
      rows = cursor.fetchall()

query_tokens = set(query.lower().replace('/', ' ').replace('.', ' ').split())
      scored: List[MemoryRecord] = []

for mem_id, task_id, category, content in rows:
          content_tokens = set(content.lower().replace('/', ' ').replace('.', ' ').split())
          intersection = query_tokens.intersection(content_tokens)
          
          # Cosine-style overlap score
          score = len(intersection) / (math.sqrt(len(query_tokens)) * math.sqrt(len(content_tokens)) + 1e-5)
          if score > 0.1:
              scored.append(MemoryRecord(mem_id, task_id, category, content, score))

scored.sort(key=lambda m: m.relevance_score, reverse=True)
      return scored[:top_k]

def format_memory_injection(self, memories: List[MemoryRecord]) -> Dict[str, Any]:
      """
      Formats retrieved memories into a structured dynamic turn block.
      Mandatory Invariant: Never inject into system prompt to protect KV prefix cache.
      """
      records = [
          {
              "memory_id": m.memory_id,
              "category": m.category,
              "fact": m.content,
              "confidence": round(m.relevance_score, 2)
          }
          for m in memories
      ]
      payload = (
          "<retrieved_memory_context>
"
          + json.dumps(records, indent=2)
          + "
</retrieved_memory_context>"
      )
      return {
          "role": "user",
          "content": payload,
          "token_estimate": len(payload.split()) * 2
      }

# Verification & Test Harness
mem = HierarchicalMemoryManager()

# Populate L2 episodic memory with historical outcomes
mem.record_episodic_event(
  "mem_101", "task_881", "bug_fix",
  "auth/session.py: Fixed UTC clock drift by migrating datetime.utcnow() to datetime.now(timezone.utc)."
)
mem.record_episodic_event(
  "mem_102", "task_882", "devops",
  "Dockerfile: Base container pinned to debian:bookworm-slim to avoid glibc compatibility faults."
)
mem.record_episodic_event(
  "mem_103", "task_883", "performance",
  "db/pool.py: Set max_connections to 25 to prevent connection starvation in worker pods."
)

# New Agent Task
current_query = "Fix timezone expiration issue in auth/session.py"
print(f"Incoming Task Query: '{current_query}'")

# Execute Retrieval
retrieved = mem.retrieve_semantic_memories(current_query, top_k=1)
print(f"Retrieved {len(retrieved)} relevant memory record(s):")
for r in retrieved:
  print(f"  ID: {r.memory_id} | Score: {r.relevance_score:.3f} | Content: {r.content}")

# Format Injection Block
injection = mem.format_memory_injection(retrieved)
print(f"
Formatted Injection Role: '{injection['role']}'")
print(f"Estimated Token Cost: {injection['token_estimate']} tokens")

# Assertions
assert len(retrieved) == 1
assert retrieved[0].memory_id == "mem_101"
assert injection["role"] == "user"  # Verifies system prompt is NOT modified
print("
Hierarchical memory invariants verified successfully.")

6. Real Production Configuration: SQLite WAL & pgvector Storage

In production harnesses, L2 episodic logs and L3 vector storage are configured with Write-Ahead Logging (WAL) and memory-mapped I/O for concurrent access without database locks:

-- Production Agent Memory Schema (PostgreSQL + pgvector / SQLite)
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA temp_store = MEMORY;
PRAGMA mmap_size = 30000000000;

CREATE TABLE IF NOT EXISTS agent_episodes (
    episode_id TEXT PRIMARY KEY,
    session_id TEXT NOT NULL,
    turn_index INTEGER NOT NULL,
    role TEXT NOT NULL,
    content TEXT NOT NULL,
    token_count INTEGER NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE IF NOT EXISTS agent_semantic_facts (
    fact_id TEXT PRIMARY KEY,
    domain TEXT NOT NULL,
    fact_summary TEXT NOT NULL,
    source_episode_id TEXT REFERENCES agent_episodes(episode_id),
    embedding BLOB, -- 1536-dim vector embedding
    access_count INTEGER DEFAULT 0,
    last_accessed_at TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_episodes_session ON agent_episodes(session_id, turn_index);
CREATE INDEX IF NOT EXISTS idx_facts_domain ON agent_semantic_facts(domain);

7. When to Use It and When Not to Use It

Scenario	Recommended Strategy	Why?	Anti-Pattern
Multi-Day Engineering Projects	Full 3-Tier Hierarchy (L1 + L2 + L3)	Persists architectural lessons, past test fixes, and project heuristics across developer sessions.	Treating each agent command as a cold boot that re-scans the repository from scratch.
Ephemeral Data Scraping Tasks	L1 Only (No Persistence)	Scraped web pages and ephemeral DOM trees have zero reuse value across future runs.	Indexing 50,000 raw HTML pages into a permanent vector database.
Strict Compliance / Multi-Tenant SaaS	Tenant-Isolated SQLite Databases	Ensures that memories from Customer A cannot leak into Customer B through shared vector space.	Storing all multi-tenant embeddings in an unpartitioned shared vector collection.
Low-Latency Chat Applications	Asynchronous Offline Consolidation	Running synchronous consolidation and vector queries in the chat path adds 300ms to user perceived latency.	Blocking user response generation while generating semantic summaries.

8. Core Invariants to Remember

Concept	Invariant Rule	Operational Consequence
Prefix Cache Placement	Never inject dynamic external memories into the system prompt prefix.	Prepending dynamic memories invalidates prompt caches across all turns, multiplying API billing and latency.
Top-K Strictness	Limit retrieved memory injections to $K \le 3$ items (under 600 tokens total).	Dumping 15 memory snippets floods working context and triggers attention dilution.
Two-Stage Filtering	Always rerank vector candidates with a cross-encoder before prompt injection.	Bi-encoders frequently surface false positive keyword matches with irrelevant technical context.
Transactional WAL	Write episodic traces immediately to disk via SQLite WAL before issuing API calls.	If the agent container crashes mid-execution, the episodic record survives for recovery.

Sources

Packer, C., Fang, V., Shlapentokh-Rothman, S. G., Menon, K., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint. https://arxiv.org/abs/2310.08560
Anthropic Engineering. (2024). Prompt Caching: Performance and Cost Optimization. Anthropic Documentation. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). Efficient Streaming Language Models with Attention Sinks. arXiv preprint. https://arxiv.org/abs/2309.17453
SQLite Development Team. (2024). Write-Ahead Logging (WAL) in SQLite. SQLite Documentation. https://www.sqlite.org/wal.html