Hosting an AI Agent 24/7 — Calling a Model Is Cheap, Running One Isn't

An AI agent that calls a model API is one of the best-fitting workloads you can put on a small container: it spends nearly all of its life idle, waiting on someone else's GPU, using almost no CPU of its own. An agent that runs a model locally is the opposite — the weights alone exceed what a small container has before your code does anything. That single distinction decides whether this works. Assuming you're calling an API, the difficulties aren't resources at all: they're long request times, restarts landing mid-task, and where conversation state lives.

One thing matters before the architecture: unattended 24/7 operation needs credit on your balance. A new account's first app gets a short trial run so you can confirm the agent works; after that it stays stopped until you add credit. Schedules, public bots, webhooks and any loop that must remain available all depend on that.

Why the API-calling kind fits so well#

Think about what your process is actually doing during an agent turn. It builds a prompt (microseconds), makes an HTTPS request, and then waits — for one second, or thirty, or a minute on a long reasoning call. Then it parses a response and maybe calls a tool.

Almost none of that is CPU work. It's an open socket and a callback. That's the same shape as a Discord gateway bot, which is why the same small allocations work: a fractional CPU is a time budget, and an agent barely touches it.

Your real cost centre is the model provider's bill, not the hosting. That's worth internalising before you optimise the wrong thing.

Local inference is a different story. A quantised small model still wants gigabytes of RAM, loads slowly on every restart, and wants a GPU you don't have here. If your plan involves llama.cpp or a local transformers pipeline, this isn't the right home for it. Calling an API and hosting the agent is.

Slow calls are the real constraint#

A model call that takes 45 seconds breaks assumptions all over a normal web stack.

If your agent is a bot, you cannot answer in-line. Discord gives you three seconds to acknowledge an interaction, and no model call reliably fits in that. The pattern is: acknowledge immediately (defer), do the work, then edit the message with the result. Telegram is more forgiving but the same shape applies — send a "thinking" message, edit it when you have an answer.

If your agent is an HTTP service, decide what happens to the connection. A request that's open for a minute is fine to serve, but the client on the other end may not agree — browsers, proxies, and impatient users all give up. No per-app timeout or buffering middleware is configured on your route, so the constraint is your own client's patience and the defaults of whatever sits between you. Either stream tokens as they arrive, or return a job ID immediately and let the client poll.

Set explicit timeouts on your own calls. Most SDK defaults are generous or absent. An agent blocked forever on a hung connection looks exactly like a hung agent — and nothing here probes your app to notice.

Restarts land mid-task#

This is the failure mode specific to running on a container platform, and it's worth designing for on day one.

When your app is redeployed or restarted, it gets a stop signal and roughly one second before it's killed. A 30-second model call in flight does not finish. Neither does the tool call after it, or the database write that was going to record the result.

So an agent that holds a multi-step task purely in memory will lose it, and the user gets silence. Two habits fix this:

• Persist the task, not the progress. Write "user X asked Y" to storage before you start, and mark it done when you've replied. On startup, look for anything unfinished. That turns a restart into a retry instead of a dropped request.
• Make retries safe. If a restart means the same prompt might run twice, make sure that's harmless — deduplicate on a request ID, and don't take irreversible tool actions without a guard.

The design-for-restarts patterns apply directly; agents just make the window more visible because their tasks are unusually long.

Where conversation state goes#

Agents accumulate state — message history, summaries, tool results, a vector index if you've built one. The rule is the same as everywhere else: anything outside a volume disappears when the container is recreated.

For most agents, SQLite on a volume is the right answer: one process owns the data, there's nothing to expose, and backups are a file you can download. Reach for Postgres when a second app needs the same data.

Two agent-specific notes:

• Cap your history. An unbounded conversation log grows your prompt, your bill, and your disk at the same time. Summarise or truncate on a policy you choose deliberately.
• A small vector store is fine; a large one isn't. A few thousand embeddings in SQLite works. A serious vector database is another service with its own memory appetite — check it fits before committing.

Keys, costs, and blast radius#

Your model API key is the most valuable thing your container holds — it's directly billable by anyone who gets it.

• Environment variable, never in the image. Same discipline as a bot token: baked into a layer means readable by anyone who can pull it.
• Set a spend limit at the provider. This is the backstop that a loop in your agent can't defeat. An agent that retries a failing call aggressively can spend real money before you notice, and no amount of container configuration prevents that.
• Log token usage per request. It's the cheapest observability you'll ever add, and it turns "why is the bill high" into a query rather than a guess.
• Rate-limit yourself. If your agent is user-facing, one enthusiastic user shouldn't be able to drain your credit. A per-user cap is a few lines and saves a bad week.

Deploying it#

Nothing exotic. An agent is usually a worker — no inbound port, so nothing to route and nothing to expose. If it's a bot, that's the whole deployment. If it serves HTTP, bind one port as HTTPS and you get a certificate automatically.

Since agents are frequently the thing you built with an AI, two paths suit them:

• Zip upload, where the build agent works out the runtime and dependencies for you, and pauses to ask for your API key rather than reading it out of your source.
• Git push, if you have a Dockerfile and want deploys to be a git push. Your coding agent can drive this over MCP — it can fetch the deploy URL, set environment variables, read logs, and restart, all from your editor.

FAQ#

Can I host an AI agent on a small container?#

Yes, if it calls a model API. Agents spend nearly all their time waiting on the provider, so CPU and RAM demands are low.

Can I run a local LLM?#

No, realistically. Model weights need gigabytes of RAM and ideally a GPU. Call an API and host the agent instead.

How do I handle model calls that take 30+ seconds in a bot?#

Acknowledge the interaction immediately, then edit the message once the model responds. Discord's three-second deadline is for the acknowledgement, not the answer.

What happens to an in-flight model call when my app restarts?#

It's lost — there's about one second between the stop signal and the kill. Persist the task before starting it so a restart becomes a retry.

Where should conversation history live?#

On a volume, usually SQLite. Anything in the container's filesystem outside a volume is discarded on redeploy.

How do I stop an agent from running up a huge model bill?#

Set a spend limit with the provider, log token usage per request, and rate-limit per user. Container settings can't cap external API spend.

Wrapping up#

The instinct that AI workloads need big machines comes from training and inference, and it doesn't transfer to agents. An agent is a small program that waits a lot — the least demanding kind of long-running process there is. Spend your attention on the things that actually break: acknowledge fast when a model is slow, assume a restart will land in the middle of a task, keep state on a volume, and put a hard spend limit where a runaway loop can't reach it.