At-Least-Once vs Exactly-Once Delivery in Webhooks

Exactly-once delivery is not something a webhook provider can hand you over HTTP. A delivery can succeed and its acknowledgement can still be lost on the way back, and the sender cannot tell that apart from a failed request. What you can get is at-least-once transport combined with a consumer that ignores repeats, which produces an exactly-once effect. That combination is usually called effectively-once, and it is the target to design for.

The at-least-once vs exactly-once question matters because “exactly-once” appears on a lot of product pages, and developers reasonably read it as “I do not have to think about duplicates.” You still do. Below: the six points where an acknowledgement can be lost, the four questions that show what a vendor’s exactly-once claim actually covers, and what an at-least-once contract should promise in exchange.

The three delivery guarantees a system can offer

Every messaging system picks one of three contracts, and each one accepts a different failure. No contract avoids failure entirely.

Table: the three delivery semantics, the failure each one tolerates, and where each one fits.

At-most-once is easy to build and rarely what you want: if a request times out, you move on. That is acceptable for a gauge that gets resampled in fifteen seconds and unacceptable for a payment_succeeded event, where the failure mode is a customer who paid and a system that never noticed. At-least-once flips the tradeoff. You retry until the destination confirms, so nothing is lost, and the destination sometimes sees the same event twice. A duplicate can be handled in code; a lost event usually cannot be recovered.

Why exactly-once breaks over HTTP

Skip the formal proof and watch one delivery fail.

A worker claims delivery 88213 and POSTs the event to your endpoint. Your handler validates the signature, opens a transaction, writes a refunds row, commits, and starts building the 200 OK response. At that instant the process is killed: an OOM kill, a node eviction, a deploy that did not drain. The response never leaves the machine.

From the gateway’s side, all that happened is a request with no response before the timeout expired. The refund row exists, but the gateway cannot learn that: the only channel that could have told it is the channel that just broke. Two options remain, and both are wrong some of the time:

• Retry. If the side effect already happened, you have now created a second refund.
• Do not retry. If the request never reached your handler at all, the event is gone.

There is no third option. This is the two generals problem in its most practical form: two parties on an unreliable channel cannot both reach certainty about a shared outcome, because the last message in any exchange can always be the one that is lost. Acknowledging the acknowledgement just moves the uncertainty one hop out.

Where the acknowledgement gets lost

The refund crash is one of six moments a delivery can fail. Line them up and the problem is visible in one column: the sender sees nearly the same thing whether or not your side effect happened.

Table: failure points in a single webhook delivery, whether the side effect ran, what the sender observes, and what a retry does.

Rows two to five look identical from the sending side: a request went out and no usable answer came back. Retry all four and three of them become duplicates. Retry none and the event in row two is lost. Only the first row is safe to decide on, because a request that never connected cannot have run. That table is the whole argument against exactly-once over HTTP.

The last row is the one people forget. The receiver did everything right, the acknowledgement arrived, and the duplicate still happens, because the sender’s own bookkeeping sits on the far side of another crash.

Where duplicates actually come from

In practice they arrive from four directions, and only one of them is under the webhook gateway’s control.

• Retry after a timeout. The rows above. The retry schedule that creates duplicates decides when the duplicate lands, not whether it lands.
• Provider redelivery. Before anything reaches your gateway, the source may resend. Stripe’s webhook documentation states that an event can be sent more than once and that endpoints should tolerate it. No gateway can deduplicate what the provider treats as two separate delivery attempts.
• Network partitions. A partition heals and in-flight requests complete after the sender already gave up on them.
• A crashed worker mid-flight. A worker that dies after the POST leaves the socket, but before it marks the delivery succeeded, leaves a row that looks unattempted. Another worker claims it and sends again.

A slow consumer manufactures duplicates on its own. If your handler routinely takes longer than the gateway’s timeout, those deliveries get retried while the first attempt is still running. Acknowledging fast on your side cuts duplicates as well as latency.

When a vendor claims exactly-once

Treat it as a question rather than a red flag. Exactly-once semantics are achievable inside a closed system, where one vendor controls the producer, the log, and the consumer, and can tie offset commits to the same transaction as the write. That is hard to build, and it is a different problem from delivering an HTTP POST to a server the vendor has never seen. So check what the claim covers:

1. Does it apply end to end, including the final HTTP hop to your endpoint, or only to internal handoffs before that hop? Most claims describe the second.
2. Over what window does deduplication apply? Five minutes says nothing about a redelivery six hours later, after you fixed a bug.
3. What happens when your endpoint processes an event and then times out? That is row three of the table. If the answer is “we retry”, the system is at-least-once with dedupe on top. That is fine, but it is not exactly-once at your boundary.
4. Does it survive a replay from a dead-letter queue, which is a deliberate re-send of something already attempted?

Webhooker provides at-least-once delivery on a durable queue and expects consumers to be idempotent, which puts the consumer’s half of the work in the contract. Ask the same four questions of any vendor you are evaluating. How the main gateways answer them is a shorter list than their feature tables suggest.

What at-least-once actually promises

At-least-once is a checkable contract, and stronger than it sounds. It says: once we have accepted an event we will not lose it, and we will keep trying to deliver it until the destination confirms or the retry budget runs out. Two mechanics make that credible.

The first is persist-before-acknowledge. When a provider POSTs to a Webhooker ingest URL, the event is verified and written to durable storage before the 200 OK goes back, and 99% of events are acknowledged in under 10 ms, measured server-side. The order of those two steps is what makes the guarantee hold. Crash a microsecond after acknowledging and the event is already on disk, so a restarted worker picks it up. A system that acknowledges first and enqueues afterwards has a window where an accepted event exists only in memory, and events die in that window.

The second is out-of-band delivery, and it is the half you inherit the moment you stop polling an API and let events be pushed to you. Workers claim deliveries from a durable queue, and delivery never sits on the provider’s request path, so a destination that takes eight seconds to respond does not slow the acknowledgement Stripe or GitHub is waiting for. On PostgreSQL that queue is a table claimed with FOR UPDATE SKIP LOCKED; the delivery queue design covers the schema, lock timeouts and the reaper that turns a crashed worker into a retry.

What the contract deliberately does not promise is uniqueness. Duplicates are the stated cost of never dropping an event.

Effectively-once: the consumer’s half

Since the transport cannot guarantee uniqueness, the consumer has to:

at-least-once transport + idempotent consumer = exactly-once effect

An idempotent consumer reaches the same end state whether it processes an event once or five times. In practice that means recording the provider’s event id under a unique constraint, in the same transaction as the side effect, so the database rather than your code decides which of two concurrent copies wins. The implementation, the choice of key, and how long to keep keys are in the guide to idempotency keys for webhook consumers.

Ordering is a separate guarantee

At-least-once says nothing about sequence, and the two get conflated constantly. Retries reorder traffic by construction: event A fails on its first attempt and lands eight seconds later, while event B, sent afterwards, succeeds immediately. Your handler sees B then A.

Ordered delivery is its own guarantee with its own cost: serializing a destination’s traffic, and letting one stuck event hold up everything behind it. In Webhooker it is available on the Team plan. It does not remove duplicates, since a strictly ordered stream can still deliver the same event twice in a row. Two cheaper patterns cover most cases, so before you reach for it, read why webhooks arrive out of order and when you need FIFO delivery.

Designing for the contract

Accept at-least-once as the ground truth and a handful of decisions follow.

• Persist before you acknowledge, on both sides. Write the event down, return 200 quickly, do the slow work asynchronously. A handler that runs a 30-second reconciliation before responding is asking to be retried.
• Put a unique constraint on the event id, and make the side effect and the dedupe marker atomic. One transaction, both writes.
• Monitor your duplicate rate. A low steady rate is normal. A spike usually means your endpoint got slower; check handler latency before blaming the sender.
• Know where exhausted deliveries go. Retries end eventually, and what happens after the last retry fails decides whether a bad hour costs you data or a few replay clicks.
• Keep the evidence. When someone asks whether order 4471 was processed twice, you want per-attempt records with response codes and bodies. In Webhooker these are in the delivery history, which also shows attempts that timed out after your handler had already committed.

Of these, the unique constraint and the shared transaction do most of the work. Without them, a duplicate that slips in during a slow deploy can double-charge someone; with them, the second insert hits the constraint, the handler returns 200, and nothing else happens.

Frequently asked questions