Webhook Retries: Exponential Backoff & Jitter

To retry webhooks properly, wait longer after each failure instead of firing at a fixed interval. Use exponential backoff with jitter to spread attempts, retry only 5xx and 429 responses, honor Retry-After, and open a circuit breaker when a destination keeps failing. Cap the attempts, then move dead deliveries to a queue for replay.

We have spent a fair amount of time watching webhook traffic misbehave under load, and almost every painful incident traces back to the same root cause: the retry logic was written as an afterthought. A webhook that fails once is normal. A retry policy that turns one failure into a self-inflicted outage is a design bug. This article walks through the pieces of a retry strategy that holds up when a destination goes down: exponential backoff, jitter, conditional retries by status code, and a per-destination circuit breaker. If the request-and-acknowledge exchange underneath all of it is unfamiliar, here is one webhook delivery in full. These live at the gateway-to-destination layer, which is a different concern from whatever retry logic your consumer runs internally. Keep the two separate in your head as you read.

Why naive retries make things worse

The most common webhook retry policy is also the worst one: on failure, wait a fixed number of seconds and try again, forever. It looks harmless in a test with one event. It becomes a weapon during a real incident.

Picture a destination that goes down at 14:00. Every webhook you tried to deliver in that window failed. If your retry interval is a flat 30 seconds, all of those failed deliveries now retry together, every 30 seconds, in lockstep. The destination comes back up, still cold, still reconnecting to its own database, and gets hit by the entire backlog at once. That synchronized wall of retries is the thundering herd, and it frequently knocks the destination straight back down. Your retry logic has now extended the outage it was supposed to survive.

Fixed-interval retries fail for two reasons at once. They do not give a struggling destination room to recover, and they align every retrying client onto the same clock, so the load arrives in spikes rather than a smooth trickle. Fixing the first problem needs backoff. Fixing the second needs jitter. You want both.

Exponential backoff

Exponential backoff means the wait between attempts grows multiplicatively instead of staying flat. Each retry roughly doubles the delay: 1 second, then 2, then 4, then 8, and so on. A destination that is briefly overloaded gets a short pause. A destination that is genuinely down gets progressively longer gaps, so you stop pounding on it while it recovers.

The formula is simple. The delay for a given attempt is the base interval multiplied by two raised to the attempt number, clamped to a maximum:

delay = min(base * 2 ^ attempt, cap)

Three parameters shape the curve, and all three depend on your traffic rather than on any universal rule:

• Base interval — the first wait. Something in the range of one to a few seconds suits most HTTP destinations.
• Cap — the ceiling a single wait will not exceed, so delays do not grow to hours. Many systems settle somewhere between 30 seconds and a few minutes.
• Max attempts — how many tries before you give up and stop retrying inline.

The table below shows one plausible backoff schedule, not a prescription. Treat it as a starting point you tune against your own destinations.

Table: example exponential backoff schedule with a base of 1 second and a cap of 60 seconds, showing how the delay doubles each attempt until it hits the ceiling.

There is no single correct curve here. A payment provider that needs fast confirmation wants a tight base and a low cap. A batch sync that nobody is watching in real time can afford long gaps and many attempts. Pick the shape from what the destination actually needs, and resist the urge to treat one curve as gospel.

Jitter

Backoff on its own fixes the recovery-room problem but leaves the synchronization problem half-solved. Imagine a thousand deliveries that all failed at the same instant. With pure exponential backoff, every one of them waits exactly one second, then exactly two, then exactly four. They double in unison and re-collide at every step. You have pushed the spikes further apart, but they are still spikes.

Jitter breaks the lockstep by adding randomness to each wait. Instead of every client retrying at exactly the four-second mark, each picks a random point in a window, so the load smears out into something the destination can absorb. The AWS Architecture Blog’s write-up on exponential backoff and jitter is the canonical reference, and its conclusion is blunt: adding jitter measurably reduces contention and completes the work faster.

Two variants are worth knowing:

• Full jitter — pick a random delay between zero and the current backoff ceiling. On the attempt that would otherwise wait four seconds, you wait a random amount between zero and four. Simple, and it spreads load well.
• Decorrelated jitter — base each new delay on the previous actual delay rather than the attempt number, which keeps waits growing while still scattering them. It is slightly more involved but handles long outages gracefully.

For most webhook systems, full jitter is plenty. The point is not which flavor you pick; it is that any reasonable jitter beats none.

Conditional retries

Not every failure deserves a retry. Retrying blindly wastes work in the best case and causes damage in the worst. The status code the destination returns tells you whether trying again could plausibly help.

• 5xx (server errors) — retry. The destination is having a bad moment; a later attempt may well succeed.
• 429 (too many requests) — retry, but back off harder. The destination is explicitly telling you it is overwhelmed. If the response carries a Retry-After header, honor it instead of your default schedule; the server is naming the delay it wants.
• 4xx other than 429 — do not retry. A 400 or 422 means the request itself is malformed, and a 401 or 403 means it is not authorized. Sending the identical payload again will fail identically. Retrying a 4xx is noise that never resolves.
• Network timeouts and connection failures — retry. These are usually transient.

The mistake that trips people up is treating all failures as equal. A destination that returns 422 for a bad payload does not want to see that payload seven more times. Route those to your dead letter queue immediately rather than burning attempts on a request that can never succeed.

Circuit breakers

Backoff and jitter smooth out retries, but they still assume the destination will eventually answer. When a destination has been failing for minutes, continuing to send it traffic, even politely spaced traffic, is wasteful for you and unkind to it. A circuit breaker is the switch that stops the bleeding.

The pattern borrows three states from electrical circuits. Closed is normal: traffic flows. When failures to a destination cross a threshold, the breaker trips to open, and further deliveries to that destination are held back immediately instead of being attempted and failing. After a cooldown, the breaker moves to half-open and lets a small number of probe requests through. If those succeed, it closes and normal delivery resumes. If they fail, it opens again and waits longer.

The value of a per-destination breaker is isolation. One destination melting down does not consume the workers, connections, and retry budget that your healthy destinations depend on. Without a breaker, a single dead endpoint can starve an entire delivery pipeline as workers pile up waiting on timeouts. With one, the failure stays contained. Isolation at the queue layer matters just as much: competing workers have to claim deliveries without blocking each other, which is what FOR UPDATE SKIP LOCKED buys you.

This is how Webhooker handles it: smart retries with exponential backoff and a circuit breaker are built into delivery, so a flaky destination gets backed off and isolated automatically rather than dragging the rest of your traffic down with it.

Retries imply duplicates

Here is the tradeoff nobody escapes: the moment you retry, you accept the chance of delivering the same event twice. Webhooker’s delivery queue is at-least-once by design, and at-least-once is not a weakness to apologize for. The alternative, at-most-once, means dropping events whenever a delivery is uncertain, and silently losing webhooks is far worse than occasionally sending a duplicate.

Retries have a second consequence that is easier to miss: any backoff curve is also a reordering function. An event that fails and finally lands eight seconds later arrives behind events created after it, which is why webhooks arrive out of order even when the provider sent them in sequence.

Consider the classic case: your gateway delivers the webhook, the destination processes it, but the acknowledgement is lost on the way back. From the gateway’s side the delivery looks failed, so it retries. The destination now sees the same event a second time. No amount of clever backoff prevents this, because the ambiguity is fundamental to networks.

The fix does not live in the retry layer at all. It lives in the consumer, which must be idempotent: processing the same event twice must produce the same result as processing it once. The standard mechanism is a stable event identifier that the consumer records and checks before acting. We cover the implementation in depth in make retried deliveries safe with idempotency keys. If you take one thing from this section, take this: a retry strategy is only safe when it is paired with an idempotent consumer.

When to give up

Retries should not run forever. Past a certain number of attempts, a failing delivery is telling you something a machine cannot fix on its own: a destination is misconfigured, an endpoint moved, a payload trips a validation bug. Continuing to retry at that point just buries the problem under log noise.

When a delivery exhausts its attempts, it should stop retrying and move somewhere durable and visible rather than vanishing. That somewhere is the dead letter queue. In Webhooker, a dead delivery keeps its full per-attempt history and response logs, so you can see exactly what each destination returned before you decide what to do. Once the underlying issue is fixed, you replay, one delivery or in bulk. We go deeper into where deliveries go after the last retry fails in its own guide.

The reason to keep failed deliveries instead of discarding them is trust. A pipeline that drops events after the last retry loses data quietly, and quiet data loss is the hardest kind of bug to catch. This durable, retrying delivery is included on every plan, because it is the part of a webhook gateway you cannot bolt on later. It is also worth checking whether your vendor bills each retry attempt, since that turns someone else’s outage into your invoice. Whether it is worth writing yourself is a separate question, and we put real numbers against it in build vs buy for webhook infrastructure.

Frequently asked questions