Can You Do RAG on a Small Container? Yes, Within Limits

Retrieval-augmented generation splits neatly into three parts, and only one of them is demanding. Embedding is an API call, storage is a file, and retrieval is arithmetic — so a bot that answers questions from your own documents runs comfortably on a small container, provided you keep two rules. Don't run the embedding model locally, because that's hundreds of megabytes before your code starts, and don't rebuild the index at startup, because your container is recreated on every deploy and every configuration change — which means re-embedding your entire corpus, and paying for it, each time. Both are easy to get right up front and annoying to retrofit.

Where each part lands#

The only row with a ceiling is retrieval, and the ceiling is your memory allocation.

Do the sizing arithmetic first#

You can work out whether your corpus fits before writing anything. An embedding is a list of numbers, so:

bytes ≈ number_of_chunks × dimensions × 4     (float32)

Plug in your model's dimension count and your chunk count. A few thousand chunks lands in the tens of megabytes — comfortable. Tens of thousands starts competing with your app for a small allocation. Hundreds of thousands is a different architecture, not a bigger preset.

Two ways to buy headroom without changing anything structural: store as float32 rather than float64, and keep the text on disk rather than in the same structure as the vectors — you only need the chunk text for the handful of results you actually return.

If your corpus is genuinely large, that's the point to reach for a hosted vector database rather than fighting the memory limit.

Don't run the embedding model locally#

The tempting optimisation is a local embedding model — no per-call cost, no network. It's the wrong trade here.

A sentence-embedding model plus its runtime is hundreds of megabytes resident before your own code allocates anything, against an allocation measured in fractions of a gigabyte. It also makes every startup slow, and startups are frequent because the container is replaced routinely.

Embedding through an API keeps your process small and idle — the shape this platform suits. The per-call cost is real but concentrated at indexing time, not per query.

The expensive mistake: rebuilding at startup#

Here's the one that specifically bites on a container platform.

A natural first implementation reads your documents, embeds them, and holds the vectors in a list in memory. It works locally, where you start the process occasionally. Deployed, every restart re-embeds everything — and restarts happen on every deploy, every environment variable change, every port or volume change, and every crash-loop iteration.

So a config tweak becomes an invoice. Worse, the cost scales with your corpus, so it grows exactly as the project becomes more useful.

Persist the index. SQLite on a volume is enough:

CREATE TABLE chunks (
  id        INTEGER PRIMARY KEY,
  doc_id    TEXT NOT NULL,
  content   TEXT NOT NULL,
  embedding BLOB NOT NULL,   -- packed float32
  model     TEXT NOT NULL,   -- which model produced it
  created_at INTEGER NOT NULL
);

On startup, load what's there. Embed only chunks you don't already have. Then a deploy costs nothing and a document update costs one document.

Store the model name with each vector. Vectors from different models aren't comparable, so when you change model you need to know what to re-embed — and you will change model.

Retrieval without a vector database#

For a few thousand chunks you don't need dedicated infrastructure. Load the vectors into a matrix at startup and compute cosine similarity against the query — with normalised vectors that's a dot product, and a single matrix multiply handles the whole corpus fast enough that the model call dominates your latency completely.

If you'd rather keep it in the database, SQLite extensions for vector search exist and work well at this size. Either is fine; the deciding factor is usually which one you'll enjoy maintaining.

What matters far more than the search algorithm at this scale:

• Chunk size and overlap. Too large and retrieval returns mostly irrelevant text you pay to include in the prompt; too small and you lose context.
• How many results you include. Every retrieved chunk is prompt tokens, so top-k is a cost setting as much as a quality one.
• Whether you tell the model what it retrieved. Marking retrieved content clearly as data helps, and it's also the boundary that matters for prompt injection — retrieved documents are untrusted input if anyone else can write them.

Keeping it fresh#

Re-indexing is the operational half people skip:

• Index on change, not on a schedule, if you can detect changes. A webhook or a file timestamp beats re-reading everything hourly.
• Delete by document ID, so an updated document replaces its chunks rather than duplicating them. That's what the doc_id column is for.
• Watch the disk. Vectors and text accumulate, and total volume size has an account-wide consequence if it runs away.

FAQ#

Can I run RAG on a small container?#

Yes, if you embed through an API and your index fits in memory. A few thousand chunks is comfortable.

Should I run the embedding model locally?#

No. Model weights and runtime need hundreds of megabytes resident and slow every startup. Use an API.

How much memory will my index need?#

Roughly chunks × dimensions × 4 bytes for float32. Do that arithmetic before choosing a preset.

Do I need a vector database?#

Not at this size. SQLite on a volume plus in-process similarity is enough until your corpus is large enough to need real infrastructure.

Why is my embedding bill higher than expected?#

Probably re-embedding at startup. Persist vectors on a volume and only embed what's new.

What happens when I change embedding model?#

Old vectors become incomparable. Store the model name alongside each vector so you know what needs re-embedding.

Wrapping up#

RAG has a reputation for needing infrastructure, and most of that reputation comes from designs built for corpora far larger than a side project's. At a few thousand chunks it's a table with a BLOB column and a matrix multiply. The two decisions that matter are both about where things live: embeddings come from someone else's hardware, and the index lives on a volume rather than in a variable — because on a platform where restarts are routine, an in-memory index is a bill you pay every time you change a setting.