Conversation Memory for an LLM Bot — Whose Context Is It?

Give a chat bot a memory and you've made three decisions at once, usually without noticing. Scope — is the conversation per user, per channel, or per thread — decides whose messages end up in whose context. Length decides what every reply costs, because history is re-sent on every turn. And storage is decided for you: anything held in process memory is gone the moment your container is replaced, which happens on every deploy and every configuration change. Get the scope wrong and your bot quotes one user's message while answering another. Get the length wrong and your bill grows with the length of the conversation rather than the number of them.

Scope is a product decision, not a technical one#

Three common choices, each with a different answer to "who can see this":

For a bot in a shared server, per thread usually matches expectations best: a conversation has a visible boundary, and everyone can see what's in it.

Per channel is the one to think hardest about. It means when the bot answers you, someone else's earlier message may be in the context — so the bot can quote, summarise, or act on something a different person said. In a busy public channel that's both a privacy surprise and an injection path, because anyone in the room can put text into a conversation they're not part of.

Whatever you choose, say so in your bot's description. "Remembers the last few messages in this channel" is a one-line disclosure that prevents most of the unpleasant surprises.

Memory is not free — it's re-sent every turn#

The mechanic people underestimate: models are stateless, so "memory" means resending the conversation with every request. A twenty-turn conversation sends the previous nineteen turns again.

That means cost grows with conversation length, not just message count. A handful of long-running conversations can cost more than many short ones, which is the opposite of the intuition most people start with.

Two controls, and use both:

• Cap the window. Keep the last N turns or last N tokens, whichever your library makes easy. Simple, predictable, and rarely noticed by users.
• Summarise the overflow. When the window fills, ask the model once for a short summary of what fell out and carry that instead. Costs one call, preserves continuity, and keeps the tail bounded.

Both belong behind your existing token budgeting — a long conversation is exactly the shape that turns "twenty requests" into an unexpectedly large number.

Storage: a volume, not a dictionary#

A dictionary keyed by channel ID is the natural first implementation and the wrong one here. The container is recreated on every deploy and every environment, port, volume, or preset change, so in-memory conversations don't survive an ordinary Tuesday.

Put them on a volume. For a single-process bot, SQLite is the right default — one process owns the data, and it's on local disk where SQLite is happy.

A schema that covers most bots:

CREATE TABLE messages (
  id         INTEGER PRIMARY KEY,
  scope_id   TEXT NOT NULL,   -- channel, thread, or user id
  role       TEXT NOT NULL,   -- 'user' or 'assistant'
  content    TEXT NOT NULL,
  created_at INTEGER NOT NULL
);
CREATE INDEX idx_scope_time ON messages (scope_id, created_at);

Reading the last N rows for a scope is then one indexed query, and the scope column is where your per-user / per-channel / per-thread decision physically lives — which makes it cheap to change your mind later.

Prune, or the disk decides for you#

Conversation history grows forever unless something removes it, and disk here has a consequence beyond cost: if your account's total volume size exceeds its limit, every app you run stops and can't restart until you free space.

A bot that stores every message in every channel it can see is a plausible way to get there — especially if privacy mode is off or Message Content is enabled, because then it sees everything.

Cheap policies that work:

• Delete rows older than N days. One scheduled query.
• Keep only the last N per scope. Bounded by conversations rather than time.
• Store only what you'd resend. If you cap the context window at ten turns, keeping a thousand serves no purpose unless you want the archive deliberately.

Run pruning as part of your existing loop rather than as a separate service.

Give people a reset#

A /reset or /forget command is a small feature with outsized value: it lets a user clear a conversation that's gone sideways, and it's your fastest way to reproduce a fresh-context bug during development.

Make it scope-aware — resetting should clear exactly what the bot would have sent, no more. And if you store per-channel, consider who's allowed to reset: in a shared channel, one user wiping a shared context can be surprising.

Don't store what you don't need#

Two habits worth adopting early, because retrofitting them is tedious:

• Store message text, not whole event payloads. You need role, content, and time. Everything else is bulk you'll never read.
• Keep it out of your logs. Message content in a capped log stream pushes out errors, and it's user text arriving somewhere you might later point an AI assistant.

FAQ#

Should bot memory be per user or per channel?#

Per thread suits shared servers best. Per channel means one person's messages appear in the context of another's reply, which is both a privacy and an injection consideration.

Why does a longer conversation cost more per message?#

Models are stateless, so the whole history is resent each turn. Cap the window, and summarise what falls out of it.

Where should conversation history live?#

On a volume — SQLite is fine for a single-process bot. In-memory history is lost every time the container is recreated, which is every deploy.

How much history should I keep?#

Only what you'd actually resend, plus any retention you want deliberately. Prune by age or by count per conversation.

Can two apps share one bot's memory?#

No. A volume belongs to a single application, so a second app needs a real database.

Do users need a way to clear it?#

Yes. A reset command prevents a derailed conversation from persisting, and it's the fastest way to test fresh-context behaviour.

Wrapping up#

"Add memory" sounds like a feature and behaves like three decisions with visible consequences: whose text goes into whose prompt, how quickly cost scales with engagement, and whether your storage grows until it takes your other apps down with it. Pick the scope deliberately and say it out loud, bound the window before you need to, and put the rows somewhere a redeploy can't reach. None of that is much work in advance, and all of it is awkward to retrofit once real conversations exist.