Sending Webhooks Reliably When Your Container Restarts

Plenty has been written about receiving webhooks. Sending them is the other half, and it's where the interesting failures live: you are now responsible for delivery, and the thing doing the delivering gets replaced regularly. The standard list — sign the payload, set timeouts, retry with backoff, include an idempotency key — is correct and incomplete. On a container platform there's a fifth item that decides whether the other four matter: your pending deliveries must live somewhere that survives a restart, because every deploy and every configuration change replaces the process holding them.

Sign what you send#

Your receiver has the same problem you'd have: anyone can POST to a public URL. Signing is how they know it's you.

The standard shape:

1. Compute an HMAC over the raw request body using a shared secret.
2. Include a timestamp in the signed material and send it alongside.
3. Put both in headers, and document how to verify.

The timestamp matters — without it, a captured request can be replayed forever. Receivers should reject anything older than a small window, and you should say so in your docs so they know to.

Two practical notes. Sign the exact bytes you send, not a re-serialised version — JSON key order isn't guaranteed to survive a round trip, and a receiver that re-encodes before hashing will fail intermittently in a way that's miserable to debug. And give each receiver its own secret, so rotating one doesn't disturb everyone else.

Set timeouts, and don't send inside a request#

A receiver that accepts your connection and then stalls will hold your request open for as long as your client allows. Default timeouts in HTTP libraries are often generous or absent, so set both connect and read timeouts explicitly, and keep them short — a webhook receiver should acknowledge quickly.

More importantly, don't deliver from inside your own request handler. If a user action triggers a webhook and you send it synchronously, someone else's slow endpoint becomes your latency, on a CPU allocation that's a shared time budget. Record the event, return, and deliver from a background loop.

Retry what can succeed, stop on what can't#

The retry policy matters less than the classification behind it:

• 5xx and network errors — retry with exponential backoff and jitter.
• 429 — retry, honouring Retry-After if present.
• Other 4xx — don't retry. A 400 or 404 means the request is wrong or the endpoint is gone, and repeating it forever helps nobody.

That's the same principle that keeps you clear of trouble with rate limits generally: don't repeat what can't succeed. Cap total attempts, and after the cap, mark the delivery failed and move on rather than retrying indefinitely.

Jitter isn't optional. If a receiver goes down and comes back, every pending delivery retrying on the same schedule arrives simultaneously — the same stampede that follows a redeploy.

You will deliver twice, so make that safe#

Any at-least-once system delivers duplicates: a response lost in transit, a timeout after the receiver committed, a retry that races a slow success.

Send a unique event ID with every delivery and keep it stable across retries. That single field lets a receiver deduplicate, and it turns "did that arrive twice?" into something they can answer without asking you.

Related: don't promise ordering. Retries mean a delivery from Tuesday can land after one from Wednesday. Include a sequence number or a timestamp and let the receiver decide what to do with out-of-order events — promising ordering you can't guarantee is worse than not promising it.

The queue has to outlive the container#

Here's the part specific to running this way.

A retry queue in memory — a list of pending deliveries, a scheduled task holding a payload — is lost when your container is recreated. That happens on every deploy, every environment variable change, every port or volume change, and every crash-loop iteration. A deploy in the middle of a retry window silently drops everything waiting.

The fix is the outbox pattern, and at this scale it's one table on a volume:

CREATE TABLE deliveries (
  id           INTEGER PRIMARY KEY,
  event_id     TEXT NOT NULL UNIQUE,
  endpoint_id  INTEGER NOT NULL,
  payload      TEXT NOT NULL,
  attempts     INTEGER NOT NULL DEFAULT 0,
  next_try_at  INTEGER NOT NULL,
  status       TEXT NOT NULL DEFAULT 'pending'
);

Write the row in the same transaction as the change that caused it, then let a loop pick up whatever is due. A restart costs you nothing but a pause — which is exactly the property you want from a system that gets restarted routinely, and the same shape as any other durable background work.

Don't let one broken receiver hurt you#

If someone's endpoint has been failing for a week, continuing to hammer it wastes your resources and fills your logs with noise you've learned nothing from.

• Circuit-break per endpoint. After N consecutive failures, back off hard or pause that endpoint entirely.
• Tell the owner, if you have a way to. A disabled webhook the customer doesn't know about becomes a support ticket eventually.
• Keep the failures visible to you, ideally as a count rather than a log line per attempt — logs rotate, and a broken endpoint can generate a lot of them.

Log one line per attempt, with the event ID#

The question you'll be asked is "did you send event X?", and the answer should be a lookup rather than an investigation. One line per attempt with the event ID, endpoint, status code, and attempt number covers essentially every support question you'll get.

Don't log the payload by default — it's large, it's often sensitive, and it fills the retained window.

FAQ#

How do I let receivers verify a webhook came from me?#

HMAC the raw body with a per-receiver secret, include a timestamp in the signed material, and document the verification steps.

Should I retry failed webhook deliveries?#

Retry 5xx, network errors and 429 with backoff and jitter. Don't retry other 4xx — those won't start working.

How do receivers handle duplicates?#

Send a stable unique event ID with every delivery, including retries, so they can deduplicate.

What happens to pending retries when I deploy?#

Anything held in memory is lost. Store deliveries in a table on a volume and reconcile on startup.

Can I guarantee delivery order?#

No, not with retries. Include a sequence number and let receivers decide how to handle out-of-order events.

Should I send webhooks from my request handler?#

No. Record the event, respond, and deliver from a background loop so a slow receiver isn't your latency.

Wrapping up#

Sending webhooks is a small distributed system hiding behind an HTTP call, and most of its difficulty is about time — timeouts, backoff, replay windows, deliveries that outlive the process that queued them. That last one is the one to design for first here, because it isn't a rare failure: your container is replaced whenever you ship. Put the queue on disk, make each delivery safe to repeat, and the rest of the list is just diligence.