Dead Letter Queues for Webhooks

A webhook dead letter queue (DLQ) is where a failed delivery lands after its last retry fails, instead of being silently dropped. It holds the full event and its delivery history so you can inspect why the destination rejected it, then replay it once the problem is fixed. Think of it as a safety net under your retries.

What a dead letter queue actually is

A DLQ sits at the end of your delivery path. When a webhook arrives, your gateway accepts it, queues it, and a worker tries to deliver it to the destination. If that first attempt fails, the retry schedule kicks in and tries again on a backoff. But retries are finite. After the last one fails, the delivery has to go somewhere. Without a DLQ, that “somewhere” is nowhere: the event is gone, and nobody finds out until a customer asks why their data never showed up.

A dead letter queue, or webhook DLQ, is the alternative. It’s a durable, queryable store that catches every delivery whose retries ran out. The event isn’t lost. It’s parked, labeled with why it failed, and waiting for you to do something about it. In Webhooker, this is exactly what happens: once the retry schedule that runs before the DLQ is exhausted, the delivery lands in the DLQ rather than disappearing.

Why you need one

Webhook destinations fail for boring, ordinary reasons. In our experience running webhook infrastructure, the culprit is almost never exotic. Someone ships a bad deploy and the receiving endpoint returns 500 for twenty minutes. A database migration locks a table. An SSL certificate expires over the weekend. A rate limiter starts rejecting traffic under load. None of these are unusual. All of them will happen to you eventually.

The question is what happens to the webhooks that arrive during that window. If your pipeline drops them after the final retry, that outage becomes permanent data loss. The upstream service already got its 200 OK on ingest and moved on; it won’t send those events again. You can’t ask for a redelivery you never tracked.

A DLQ turns a permanent loss into a temporary one. The events that failed during the outage sit safely in the queue. Once you fix the destination, you replay them. The outage becomes an inconvenience instead of an incident report, and you’re not writing apologetic emails about missing orders.

What to capture

A DLQ is only as useful as the context it stores. Catching the event but throwing away the reason it failed leaves you guessing. At minimum, a good DLQ entry holds four things.

The full original event. The exact payload and headers as they arrived, so a replay sends the real thing, not a reconstruction. Worth noting that this is also the longest-lived copy of the payload you hold, so if the events carry PII, your DLQ retention window is a GDPR question, not just an ops one.

The per-attempt history. Every delivery attempt, when it ran, and how long the gateway waited between them. This shows you whether the destination was flapping or flat-out down.

The response codes and bodies. What the destination actually returned on each attempt. A 401 tells a very different story than a 503, and the response body often carries the real error message.

The failure reason. A short, captured summary of why the delivery ended up here: connection refused, timeout, HTTP 500, circuit open. Repeated connection refusals against one destination often mean an allowlist rather than an outage, which is where a static outbound IP stops the problem recurring.

Webhooker records all of this. Each DLQ entry carries the full per-attempt history and the response logs, so when you open a failed delivery you see the whole timeline rather than a single opaque “failed” flag.

Inspecting the DLQ

When something breaks, the first job is scoping the damage. A pile of failures is noise until you can slice it. That is where a queryable DLQ earns its keep.

You want to filter by destination, because outages are usually per-destination: one endpoint is down while the rest are fine. You want to filter by source, to see whether a particular integration is sending malformed payloads. And you want to filter by time, to draw a box around the outage window and see exactly what fell in.

With those filters you move from “some webhooks are failing” to “everything to the billing endpoint between 14:00 and 14:20 got connection-refused.” That is an answer you can act on. It tells you what broke, roughly when, and how many events are waiting to be recovered. The response bodies attached to each entry usually tell you why, which points you at the fix.

Replaying failed deliveries

Inspection tells you what happened. Replay is how you recover. Once you have fixed the destination, you push the parked deliveries back through.

For a one-off, a single failed delivery gets a one-click resend. You verify the fix, hit resend, and watch it go through. For an outage that caught hundreds of events, clicking one at a time isn’t realistic. That’s where bulk replay comes in: select the window you scoped during inspection and replay all of it at once. Webhooker gives you both, a dead letter queue with one-click and bulk replay, so recovery scales with the size of the incident.

One caveat that is not optional: replay only safely when your consumers are idempotent. Some of the events you are about to replay may have partially succeeded before the destination died, and a replay will deliver them again — a deliberate case of the at-least-once contract every gateway actually offers. If your handler treats a repeated event as a brand-new one, you double-charge a card or send a duplicate email. This is why safe replay needs idempotent consumers: dedupe on an event ID so a redelivery is a no-op. Get that right and you can replay an entire outage window without a second thought.

A DLQ beats grepping logs

You might argue you already have logs. When a delivery fails, you log it, and you can go read the logs later. Why add a queue?

Because a log line is a record, not a recovery tool. Logs tell you something failed. They don’t hand you the original payload in a form you can resend, and they don’t give you a button to resend it. To recover from logs you’d parse them, reconstruct the request from scratch, and fire it manually with a script, hoping you rebuilt the headers correctly. That’s archaeology, not operations.

A DLQ is different because it’s two things a log is not: queryable and replayable. It’s a live store of the actual failed deliveries, filterable by source, destination, and time, with a replay action attached to every entry. You don’t reconstruct anything; the real event is already there, ready to go. Logs are for reading. A DLQ is for fixing.

Frequently asked questions