AI agent memory, explained: what your agent remembers and where it lives
An agent's memory is files, indexes, and transcripts on a disk somewhere, not something inside the model. The three layers, the versioned-git pattern, and why the disk's location matters.
An AI agent’s memory isn’t inside the model: models forget everything between calls. Memory is what the agent’s harness saves to disk and feeds back into context: identity and notes files, conversation transcripts, and a search index over the notes. Which means agent memory is really a files-on-a-computer question: where that disk lives, how it’s backed up, who can read it, and whether changes can be reviewed and rolled back. The pattern practitioners keep landing on: treat memory as plain files under version control, on a machine that’s always on and belongs to the agent.
The three layers, in any serious harness
Strip the vendor vocabulary and agent memory has three layers. Working context: whatever fits in the model’s context window this session; fast, and gone when the session resets. Durable files: documents the harness injects at session start (who the agent is, who you are, what it has learned) plus running notes it writes as it works. Recall: search over everything that no longer fits: keyword and/or vector search across notes, sometimes transcripts. OpenClaw is a clean worked example because it’s radically legible: the docs state “the model only remembers what gets saved to disk; there is no hidden state.” Identity and knowledge live in markdown files you can open and edit (injected each session under explicit budgets: 20,000 characters per file, 60,000 total by default), daily notes land in a memory/ folder, transcripts sit in per-agent SQLite, and semantic search runs on an embedding index over the notes.
Why the disk’s location is the real decision
Memory is the most personal artifact an agent produces: months of your conversations, preferences, contacts, and credential-adjacent context distilled into files. Three properties follow. It needs persistence: an agent that lives on a laptop that sleeps, or an ephemeral cloud sandbox that resets per task, keeps losing the thing that makes it useful. It needs backup you’ve actually tested: the files are the agent; lose them and you’ve met a stranger. And it deserves privacy engineering: wherever those files sit, whoever operates that machine can read them, a fact worth choosing deliberately, not discovering later.
The versioned-memory pattern
Because durable memory is plain files, the strongest management tool turns out to be thirty years old: git. Commit the memory directory and every change the agent makes to its own instructions or knowledge becomes reviewable history: you can diff what it learned this week, and roll back the day it wrote something stupid about you into its own rules. OpenClaw leans into this: new workspaces auto-initialize as git repos, and the docs’ recommended backup is a private remote (workspace only, never the state directory, which holds credentials). Practitioners extend the pattern to fleets: a self-hosted git server (Forgejo is the usual pick) as the private remote every agent reads and writes, big changes via pull request, one versioned brain with rollback. The privacy logic is the same one that makes people self-host at all: an agent’s memory is not the thing you want parked in somebody else’s cloud account.
The failure modes worth knowing
- Silent recall loss. Semantic search depends on an embedding provider; if it breaks, memory can degrade quietly. (A real incident: subscription-only model auth doesn’t cover embedding APIs, and vector search pauses. Local embeddings fix it.) Whatever your harness: verify recall actually works, don’t assume.
- Context-window laundering. When a session compacts, whatever wasn’t written to durable files gets summarized or dropped. Good harnesses nudge the agent to save first (OpenClaw runs a silent memory-flush turn before compaction); good operators spot-check that it happens.
- Memory as attack surface. Anything that writes into files later injected as instructions is a persistence mechanism: a prompt injection that gets itself written into memory outlives the session that planted it. Versioned memory is the audit trail that makes this findable.
Where this lands
Give the agent a computer that’s always on, keep its memory as files there, version them to a private remote, and back up the rest. That’s achievable DIY, or it’s the default shape of a managed OpenClaw pod, where the persistent machine, the backups, and the privacy boundary are the product and the memory is, pointedly, yours.