At-Least-Once vs Exactly-Once Delivery in Webhooks

Exactly-once delivery is not something a webhook provider can hand you over HTTP. A delivery can succeed and its acknowledgement can still be lost on the way back, and the sender cannot tell that apart from a genuine failure. What you can get is at-least-once transport combined with a consumer that ignores repeats, which produces an exactly-once effect. That combination is usually called effectively-once, and it is the target worth designing for.

This matters because “exactly-once” appears on a lot of product pages, and developers reasonably read it as “I do not have to think about duplicates.” You do. So it is worth reading webhook delivery guarantees the way an engineer reads them: as a statement about which failure the system has chosen to accept.

The three delivery guarantees a system can offer

Every messaging system picks one of three contracts. The choice is not about quality; it is about which failure you are willing to accept, because you have to accept one.

Table: the three delivery semantics, the failure each one tolerates, and where each shows up in practice.

At-most-once is easy to build and rarely what you want: if a request times out, you shrug and move on. Defensible for a gauge that gets resampled in fifteen seconds, indefensible for a payment_succeeded event, where the failure mode is a customer who paid and a system that never noticed. At-least-once flips the tradeoff. You retry until the destination confirms, so nothing is lost, and the price is that the destination sometimes sees the same event twice. That price is payable. Loss usually is not.

Why exactly-once breaks over HTTP

Skip the formal proof and watch one delivery fail.

A worker claims delivery 88213 and POSTs the event to your endpoint. Your handler validates the signature, opens a transaction, writes a refunds row, commits, and starts building the 200 OK response. At that instant the process is killed: an OOM kill, a node eviction, a deploy that did not drain. The response never leaves the machine.

From the gateway’s side, all that happened is a request with no response before the timeout expired. The refund row exists, but the gateway cannot learn that: the only channel that could have told it is the channel that just broke. Two options remain, and both are wrong some of the time:

• Retry. If the side effect already happened, you have now created a second refund.
• Do not retry. If the request never reached your handler at all, the event is gone.

There is no third option, and no amount of engineering produces one. This is the two generals problem in its most practical form: two parties on an unreliable channel cannot both reach certainty about a shared outcome, because the last message in any exchange can always be the one that is lost. Acknowledging the acknowledgement just moves the uncertainty one hop out.

What at-least-once actually promises

At-least-once is a checkable contract, and stronger than it sounds. It says: once we have accepted an event we will not lose it, and we will keep trying to deliver it until the destination confirms or the retry budget runs out. Two mechanics make that credible.

The first is persist-before-acknowledge. When a provider POSTs to a Webhooker ingest URL, the event is verified and written to a PostgreSQL deliveries table before the 200 OK goes back, with a p99 target under 10 ms server-side. The ordering is the whole point. Crash a microsecond after acknowledging and the event is already on disk, so a restarted worker picks it up. A system that acknowledges first and enqueues afterwards has a window where an accepted event exists only in memory, and events die in that window.

The second is out-of-band delivery, and it is the half you inherit the moment you stop polling an API and let events be pushed to you. Workers claim rows with SELECT ... FOR UPDATE SKIP LOCKED, which lets many workers pull from one table without blocking each other (PostgreSQL row locking documents the semantics, and how Postgres works as a job queue shows the full schema and claim query). Delivery never sits on the provider’s request path, so a destination that takes eight seconds to respond does not slow the acknowledgement Stripe or GitHub is waiting for.

What the contract deliberately does not promise is uniqueness. Duplicates are not a defect here; they are the stated cost of never dropping an event.

Where duplicates actually come from

In practice they arrive from four directions, and only one of them is under the webhook gateway’s control.

• Retry after a timeout. The case above. The retry schedule that creates duplicates decides when the duplicate lands, not whether it lands.
• Provider redelivery. Before anything reaches your gateway, the source may resend. Stripe’s webhook documentation states that an event can be sent more than once and that endpoints should tolerate it. No gateway can deduplicate what the provider treats as two separate delivery attempts.
• Network partitions. A partition heals and in-flight requests complete after the sender already gave up on them.
• A crashed worker mid-flight. A worker that dies after the POST leaves the socket, but before it marks the delivery succeeded, leaves a row that looks unattempted. Another worker claims it and sends again.

A slow consumer manufactures duplicates on its own. If your handler routinely takes longer than the gateway’s timeout, those deliveries get retried while the first attempt is still running. Fast acknowledgement on your side is a duplicate-reduction strategy, not only a latency one.

Effectively-once: the practical target

Since the transport cannot guarantee uniqueness, the consumer has to. That is the whole trick:

at-least-once transport + idempotent consumer = exactly-once effect

An idempotent consumer reaches the same end state whether it processes an event once or five times. The usual implementation records the identifier the provider assigned to the event, in a table with a unique constraint:

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;

If that insert affects zero rows, you have already handled the event and can return 200 immediately. Two details decide whether this holds under load:

1. Uniqueness must be enforced by the database, not by a SELECT followed by an INSERT. Two workers handling the same duplicate concurrently will both see “not processed yet” and both proceed. Only a unique constraint arbitrates that race.
2. The dedupe record and the side effect belong in the same transaction. Insert the marker, commit, then create the refund, and a crash in between leaves an event marked done that was never done.

We go through the failure modes and key-selection tradeoffs in the guide on how to make handlers idempotent with a stable event id. The short version: use the provider’s own event id, never a hash of the payload alone, and never a timestamp.

When a vendor claims exactly-once

Treat it as a question rather than a red flag. Exactly-once semantics are achievable inside a closed system, where one vendor controls the producer, the log, and the consumer, and can tie offset commits to the same transaction as the write. That is a real engineering achievement, and a different problem from delivering an HTTP POST to a server the vendor has never seen. So check what the claim covers:

• Does it apply end to end, including the final HTTP hop to your endpoint, or only to internal handoffs before that hop?
• Over what window does deduplication apply? Five minutes says nothing about a redelivery six hours later, after you fixed a bug.
• What happens when your endpoint processes an event and then times out? If the answer is “we retry”, the system is at-least-once with dedupe on top. That is fine, but it is not exactly-once at your boundary.
• Does it survive a replay from a dead-letter queue, which is a deliberate re-send of something already attempted?

Our own position is on the record: Webhooker is at-least-once delivery on a PostgreSQL-backed queue, and we tell you to keep your consumers idempotent. That is not a smaller promise. It is the same promise with the consumer’s half of the work named out loud. Ask the same four questions of any vendor you are evaluating — how the main gateways answer them is a shorter list than their feature tables suggest.

Ordering is a separate guarantee

At-least-once says nothing about sequence, and the two get conflated constantly. Retries reorder traffic by construction: event A fails on its first attempt and lands eight seconds later, while event B, sent afterwards, succeeds immediately. Your handler sees B then A.

Ordered delivery is its own guarantee with its own cost: serializing a destination’s traffic, and letting one stuck event hold up everything behind it. In Webhooker it is available on the Team plan. It does not remove duplicates, since a strictly ordered stream can still deliver the same event twice in a row. Before you reach for it, it is worth knowing why webhooks arrive out of order and when you actually need FIFO delivery — two cheaper patterns cover most cases.

Designing for the contract

Accept at-least-once as the ground truth and a handful of decisions follow.

• Persist before you acknowledge, on both sides. Write the event down, return 200 quickly, do the slow work asynchronously. A handler that runs a 30-second reconciliation before responding is asking to be retried.
• Put a unique constraint on the event id, and make the side effect and the dedupe marker atomic. One transaction, both writes.
• Monitor your duplicate rate. A low steady rate is normal. A spike usually means your endpoint got slower, not that the sender changed behaviour.
• Know where exhausted deliveries go. Retries end eventually, and what happens after the last retry fails decides whether a bad hour costs you data or a few replay clicks.
• Keep the evidence. When someone asks whether order 4471 was processed twice, you want per-attempt records with response codes and bodies. On any Webhooker plan you can see every attempt in the delivery history, including attempts that timed out after your handler had already committed.

That list is the difference between a pipeline that occasionally double-charges someone and one where a duplicate is a non-event.

Frequently asked questions