Rate-Limiting an LLM Endpoint — Requests Aren't the Unit, Tokens Are

If your app exposes an endpoint that calls a model API, every request spends money, and no hosting setting can stop it — the bill is between you and your model provider, and a container quota has no visibility into it. That changes what rate limiting is for. It's no longer about protecting your CPU, which can absorb far more traffic than your budget can; it's about protecting your balance. And the usual unit is wrong: requests aren't comparable, tokens are. One user sending a 50-word question and another pasting a novel are the same "request" and can differ in cost by a factor of a hundred.

Three layers, in order of reliability#

Build outward from the thing that can't be bypassed:

1. A spend limit at your provider. The only control that holds when your code is wrong. Set it before you deploy anything public — it's the difference between a bad afternoon and a bad month.
2. Per-user accounting in your app. The real control, because it's the only layer that knows who is spending.
3. Cheap rejection before the model call. Free to run, and it removes most abuse before it costs anything.

Note what isn't on the list: there's no web application firewall or rate limiting in front of your app — the edge terminates TLS and routes by hostname. Whatever you don't build, doesn't exist.

Reject cheaply, before you call anything#

The cheapest request is the one you refuse. Do this first, because it's a few lines and catches the bluntest abuse:

• Cap input length in characters before tokenising, and reject with a clear 400. An unbounded prompt field is an unbounded bill.
• Require authentication on anything that costs money. An anonymous LLM endpoint on a public URL will be found — and unlike most abuse, this kind pays for itself for the attacker.
• Validate the shape of the request before doing any work.

That ordering matters: validation before authentication wastes work on requests you'll reject anyway, and authentication before model calls is what turns "someone is abusing my endpoint" into "someone is abusing my endpoint on their own account".

Count tokens, not requests#

A limit of "20 requests per hour" sounds protective and isn't. Twenty maximum-context requests can cost more than a thousand short ones.

Track a token budget per user per window instead:

• Estimate input tokens before the call, and refuse if the user's remaining budget can't cover it.
• Record actual usage after the call — providers return it in the response — and deduct the real number.
• Count output tokens too. A max_tokens cap per request bounds the worst case; without one, a single response can run for a long time.

Estimating before and reconciling after means an over-budget user gets a clean rejection rather than an unexpected charge you discover later.

Your rate limiter resets on every deploy#

Here's the part that catches people on a container platform, and it's specific enough to be worth stating plainly.

A limiter that keeps counters in memory — a dictionary of user IDs to request counts — loses everything when the container is recreated. That happens on every deploy, every environment-variable change, every port or volume change, and every iteration of a crash loop.

So your own deploys reset everyone's quota. Someone at their hourly limit gets a fresh allowance because you shipped a fix, and a crash-looping app effectively has no rate limiting at all — the worst possible moment for it.

The fix is ordinary persistence: keep counters in SQLite on a volume, which is the right default for a single-process app and survives restarts. A tiny table of user_id, window_start, tokens_used is enough. Write it as you go rather than flushing periodically, so a restart loses at most one request's worth of accounting.

Don't pay for requests nobody is waiting for#

Two ways to spend money on output nobody sees.

Abandoned streams. A user closes the tab mid-response. Your connection to the provider keeps going and you keep paying unless you notice. Most frameworks expose client disconnection — check for it and cancel the upstream request.

This matters more here than on platforms with connection draining, because every deploy drops all connections at once. Without cancellation, a deploy during busy traffic means a batch of in-flight model calls that complete into nothing.

Retries that aren't idempotent. A generic retry wrapper around a failed model call can double or triple the cost of a transient error. Retry on connection failures, not on timeouts where the request may well have been processed.

Cache what repeats#

Often the highest-leverage change, and easy to skip because it isn't a limit.

If your app answers similar questions repeatedly — a support bot, a summariser pointed at the same documents, a "explain this page" button — hash the normalised input and cache the response. Repeated identical requests then cost nothing.

Cache on a volume rather than in memory, for the same reason as the counters: an in-memory cache is empty after every deploy, exactly when you've just restarted and traffic resumes.

Make the spending visible#

You can't manage what you can't see, and this is cheap:

• Log tokens per request, with the user ID. That turns "why is the bill high" into a query.
• Use a separate API key per app. Provider dashboards break down by key, so attribution is free — and revoking one doesn't take down everything else.
• Alert on a daily threshold. A push to a self-hosted ntfy instance when spend crosses a line means you find out on the day rather than at the end of the month.

FAQ#

Can hosting limits stop my model API bill?#

No. CPU and RAM quotas govern your container. Spend happens at your provider, so the cap has to be set there and in your code.

Should I rate limit by requests or tokens?#

Tokens. Requests vary enormously in cost, so a request limit protects you only against the cheapest kind of abuse.

Why does my rate limiter stop working sometimes?#

If counters are in memory, they're lost whenever the container is recreated — every deploy and every crash-loop restart. Persist them on a volume.

Do I still pay if the user closes the tab?#

Yes, unless you cancel the upstream call on client disconnect. Otherwise the provider finishes generating and bills you.

Is authentication necessary on an LLM endpoint?#

Effectively, yes. Anonymous access means abuse can't be attributed or limited per user, and public endpoints get found.

What's the single most important step?#

A spend limit at your provider. It's the only control that still works when your code has a bug.

Wrapping up#

Rate limiting an LLM endpoint is a cost-control problem wearing a performance-tuning costume. Your container can serve far more traffic than your budget can afford, so the limits that matter are per-user and denominated in tokens, backed by a provider-side cap that no bug can defeat. And keep the counters somewhere that survives a restart — otherwise the one thing standing between an abusive user and your balance is reset every time you deploy.