Your App Restarts Every Few Days — That's a Memory Leak

·5 min read·Updated
memoryoomworkerstroubleshootingperformance

If your bot or worker restarts on a rough schedule — every two days, every twelve hours, always after roughly the same uptime — you almost certainly have a memory leak meeting a hard limit. Your app's RAM allocation isn't a target, it's a ceiling: cross it and the kernel kills the process immediately, the platform restarts it, and the cycle begins again. There's no error, no stack trace, and nothing in your logs explaining it, because your code didn't fail — it was stopped mid-instruction. The signature is exit code 137 and a suspiciously regular interval, and the fix is usually one unbounded collection.

How to recognise it

Three signs, and the third is the one that distinguishes a leak from an app that's simply too small:

  • Exit code 137 in the event history. That's SIGKILL, and for a container it almost always means out-of-memory.
  • No error before the restart. A crash leaves a traceback; an OOM kill leaves the log ending mid-sentence.
  • A consistent time-to-death. An undersized app dies under load, unpredictably. A leak dies after roughly the same uptime regardless of load, because the growth is per-event and the ceiling is fixed.

If your app dies after twenty minutes under traffic and runs for days when quiet, that's sizing. If it dies every 36 hours whatever happens, that's a leak.

Watch it grow

You don't have to guess. From the shell, read the container's own memory accounting:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max

That's cgroup v2; on v1 the paths are /sys/fs/cgroup/memory/memory.usage_in_bytes and memory.limit_in_bytes. Take a reading now and another in an hour — a leak shows as a line that only goes up, while a healthy app rises and plateaus.

Better still, log it yourself once per cycle. One number per iteration turns "it restarts sometimes" into a graph you can reason about, and it survives the restart in a way an interactive check doesn't. Keep it to one short line, since logs rotate.

The usual culprits

In long-running Python and Node processes, the same few patterns account for most leaks:

  • A dictionary keyed by user or channel that nothing removes from. Cooldowns, "last seen" timestamps, per-user state. Every new person is permanent. This is the single most common one in bots.
  • A list you append to and never trim. Recent messages, processed IDs, a running log kept "just in case".
  • Conversation history without a cap. Bounded windows exist for cost reasons too, and unbounded ones are a leak with a bill attached.
  • A new HTTP client per request. Creating a session or connection pool inside a loop rather than once at startup leaves sockets and buffers behind. Build the client once and reuse it.
  • Handlers registered per connection. If you attach an event listener on every reconnect and never remove it, each reconnection multiplies the work and the retained objects.

Notice they're all the same mistake in different clothes: something grows with usage and nothing bounds it.

Bound things instead of hunting

You can spend a long time with a memory profiler. Usually you don't need to, because the fix is structural rather than diagnostic:

  • Use a bounded cache. An LRU with a maximum size can't leak — Python's functools.lru_cache(maxsize=...), or any dictionary you prune on insert.
  • Cap every collection you keep. Decide the maximum length when you create it, not when it becomes a problem.
  • Move anything that must persist onto a volume, where it lives on disk rather than in the process — SQLite is the easy answer, and disk is far cheaper than RAM.
  • Create clients once. One HTTP session, one database connection pool, at startup.

If you've bounded everything you can find and it still climbs, then reach for a profiler — with a much shorter list of suspects.

The pragmatic option nobody admits to

Here's the part that's specific to running on a container platform: a periodic restart is a legitimate mitigation.

Restarts here are cheap and expected. The process is replaced routinely anyway — every deploy, every configuration change — so an app designed to survive restarts loses nothing by having one more. If your worker leaks slowly and you can't chase it this week, exiting deliberately every few hours is a reasonable stopgap:

if time.monotonic() - started > MAX_UPTIME_SECONDS:
    print("scheduled restart to reclaim memory", flush=True)
    sys.exit(0)

Two honest caveats. It hides the problem rather than fixing it, so keep the growth bounded as well — a leak fast enough to need hourly restarts will eventually outrun any schedule. And it only works if restarts are genuinely harmless for you: state on a volume, idempotent cycles, no long-running work that a restart would truncate.

Done deliberately, on your schedule, with a log line saying why — that's operations. Done accidentally by the kernel at 3am, mid-request — that's an incident.

When it's actually sizing

Worth ruling out before you go leak-hunting. Some workloads have a legitimately higher floor than the preset you chose: a JVM, a framework with a large baseline, an image being processed, a big JSON document parsed into memory all at once.

If usage rises quickly to a plateau just above your limit and dies under load rather than after a duration, step the preset up and see whether it settles. That's a two-minute experiment that saves an afternoon of profiling.

FAQ

Why does my app restart every few days with no error?

It's being killed for exceeding its memory limit. There's no traceback because the process is stopped rather than failing.

What does exit code 137 mean?

SIGKILL — in a container, almost always the out-of-memory killer. It's the confirmation you're looking for.

How do I tell a leak from an app that's too small?

A leak dies after a consistent uptime regardless of load. An undersized app dies under load and survives when quiet.

How do I watch memory usage?

Read memory.current from the container's cgroup in the shell, or log the process's memory once per cycle so you have history.

What usually causes it?

An unbounded collection — a dictionary keyed by user, a list that's only appended to, uncapped conversation history, or a client created per request.

Is restarting on a timer a real fix?

It's a legitimate mitigation on a platform where restarts are routine, but it hides the growth. Bound your collections as well.

Wrapping up

A hard memory cap is unforgiving in a useful way: it converts a slow leak, which is easy to ignore, into a repeating event with a timestamp. Read the exit code, confirm the shape by watching memory.current for an hour, and look for the collection that grows with usage — it's nearly always there and nearly always obvious once you're looking. And if you need to buy time, restart on your own schedule rather than the kernel's. Just don't call that the fix.