Webhook Ordering: When You Need FIFO Delivery

Webhook ordering is not something the HTTP delivery model gives you. Webhooks arrive out of order because parallel workers, per-destination retries, and provider-side concurrency all break the original sequence. Ordered delivery fixes that by holding one in-flight delivery per ordering key, so the next event waits for the previous acknowledgement. The cost is head-of-line blocking: one slow destination stalls everything queued behind it. Ordering also does not remove duplicates, and it orders the requests you receive rather than the work you do after acknowledging them, so idempotent consumers stay mandatory.

Most teams meet this problem the hard way. An event that logically came second lands first, some piece of state gets overwritten with an older value, and the bug resists reproduction because it depends on timing nobody controls. The instinct is to reach for strict ordering. Sometimes that is right. More often a cheaper pattern solves the same problem without the throughput penalty, and it is worth knowing which situation you are in before you turn FIFO on.

Why webhooks arrive out of order

Nothing in the webhook model promises order. Each event is an independent HTTP POST. The sender fires them, the network reorders them, and the receiver has no built-in notion of sequence. Three separate layers each add their own shuffle.

The provider sends concurrently. When one action generates several events, the provider dispatches them from a pool rather than one at a time. Stripe says so plainly in its webhooks documentation: event delivery order is not guaranteed. Most large providers take the same position, because guaranteeing global order across millions of destinations would mean guaranteeing that no destination is ever slow.

Retries move events through time. A delivery that fails at 14:00:01 and finally succeeds on its fourth attempt at 14:00:15 lands after events created ten seconds later. This is not a defect in the retry policy; it is what backoff does. We wrote separately about how retries reshuffle the delivery order, and the short version is that any backoff curve is also a reordering function.

Delivery is parallel by design. In Webhooker, workers claim rows from the PostgreSQL deliveries table with SELECT ... FOR UPDATE SKIP LOCKED and forward them out of band. Ten workers pulling from the same queue will finish in whatever order the destinations respond. That parallelism is exactly what keeps a queue drained during a spike. It is also what destroys ordering.

The bug this causes

The canonical failure looks like this. A customer cancels. Your billing provider emits customer.subscription.updated and then customer.subscription.deleted a few hundred milliseconds apart. The deleted event reaches your endpoint first because it happened to get a warm worker. The updated event, carrying the pre-cancellation state, arrives a second later and writes status: active back over the row you just tombstoned.

The account is live again and nobody is billed for it. It surfaces weeks later when someone asks why a cancelled customer still has API access. Nothing in your logs looks wrong: both events were delivered, both returned 200, both were processed exactly as written. The ordering assumption was never stated anywhere, which is why it was never checked.

When ordering genuinely matters

Ordering is worth paying for when the meaning of an event depends on the events before it. Three patterns qualify:

• State machines keyed to one entity. A subscription, an order, a shipment. The events are transitions, not snapshots, and applying them out of sequence produces a state the machine can never legally reach.
• Financial ledgers built from deltas. If you compute a balance by applying +50, -20, -40 in sequence with an overdraft check between each step, reordering changes which step trips the check.
• Inventory decrements and other non-commutative arithmetic. reserve 3 then release 3 is fine in either order only if the intermediate state is never read. It usually is.

And the honest inverse. Ordering buys you nothing when:

• The payload carries full state. If every order.updated webhook contains the complete order, applying the newest one wins regardless of arrival order. You need a version check, not a queue.
• The operation is commutative or idempotent. Cache invalidation, search reindexing, “send this to Slack”. Running them twice or backwards costs nothing.
• The data is append-only. Analytics events, audit trails, log shipping. They are sorted at read time, if at all.

Be specific about which of your destinations fall into which bucket. In most systems it is one or two, not all of them.

How ordered delivery works

Ordered delivery is a constraint on concurrency, expressed per ordering key — whatever identifies the entity whose history must stay intact, usually a customer ID, a subscription ID, or an order number pulled from a field in the payload.

For each key, the webhook gateway keeps exactly one delivery in flight. Event two is not attempted until event one is acknowledged with a success status or has exhausted its retries. Different keys still run in parallel, so customer_A and customer_B move independently. Order is preserved inside a key and nowhere else.

That is the design decision that matters. A key of customer_id gives you strict ordering per customer and full parallelism across your customer base. A key of source gives you one serial pipeline for every event from that provider, which is technically ordered and practically a bottleneck. The narrower the key, the less you pay. Queue systems make the same trade: AWS documents it as the message group in SQS FIFO queues, where throughput scales with the number of distinct groups.

The guarantee stops at your door. Ordered delivery controls the sequence of HTTP requests, not the sequence in which your handler finishes them. If your endpoint writes the request to your own queue and returns 200 so background workers can pick it up, you have re-parallelised the events the gateway just serialised, and the ordering is gone before anything reaches your database. FIFO delivery becomes FIFO processing only if the handler does the ordering-sensitive work — or at least commits it — before it acknowledges.

What ordering costs

Head-of-line blocking. This is the whole bill in one phrase. If the delivery at the front of a key’s queue is slow, everything behind it waits. If it is failing, everything behind it waits through the entire backoff curve — first attempt, second, fourth, capped — before the queue moves. Retries and the per-destination circuit breaker still apply in ordered mode, which is correct behavior and also means a tripped breaker now holds a queue rather than just one delivery.

Throughput per key collapses to serial. Without ordering, a key’s events are limited by your worker pool. With ordering, they are limited by the round-trip time of your destination, one delivery at a time. We are not going to hand you a benchmark number here: the only number that matters is your own endpoint’s response time multiplied by your events per key.

Latency becomes coupled. An unordered pipeline degrades gracefully: a stuck delivery is one stuck delivery. An ordered pipeline degrades in blocks, and the p99 you care about is no longer a property of a single request. None of this makes FIFO wrong. It makes it a deliberate purchase.

The simplest alternative: refetch the resource

The cheapest fix is to stop trusting the payload. Treat each webhook as a signal that something about an entity changed, discard the body, and read the current state from the provider’s API before you act on it. Whatever order the events arrive in, the read returns the latest truth. A late customer.subscription.updated cannot resurrect a cancelled account, because it triggers a read that comes back cancelled.

This is what Stripe means when it tells you to retrieve the object rather than trust the event, and it is what most teams land on after their first ordering bug. It handles duplicates for free as well: two reads of the same resource produce the same answer, so a redelivery is harmless without any bookkeeping. We made the wider case for it in treating a webhook as a signal rather than a payload.

What you pay is one API call per event, against the provider’s rate limit. At low volume that is nothing. At high volume, or when the provider exposes no read endpoint for what the event describes, it stops being free — and that is where the next two options start earning their keep.

The middle option: version-aware consumers

If the extra read is too expensive, check whether your provider already tells you the order. Most do, through a monotonic sequence number, a version field, or an updated_at timestamp on the resource. If it does, your consumer can reject stale events itself, with no second call:

on_event(event):
    key      = event.payload.subscription_id
    incoming = event.payload.updated_at   # or a version / sequence number

    current = store.get_version(key)
    if current is not null and incoming <= current:
        return 200   # stale or duplicate, already superseded

    store.apply(key, event.payload, version=incoming)

That is the entire pattern. It runs at full parallelism, needs no coordination in the gateway, and has a property FIFO does not: it is also a duplicate filter. A redelivered event carries a version you have already applied, so it is dropped by the same comparison. This is the practical reason why ordering and duplicates are separate problems — one check happens to handle both, but the guarantees are not the same thing.

Two caveats. Timestamps are only safe if they come from the provider and have enough resolution; two events in the same millisecond are a coin flip, so prefer an explicit sequence or version field when one exists. And the pattern needs snapshot payloads, where the newest event contains everything you need. If your events are deltas that must all be applied, discarding the stale one loses data, and you are back to needing real ordering.

Choosing between the three

Table: the three ways to survive out-of-order webhooks, and what each one asks of you.

Read it left to right and stop at the first column that works. If you can absorb the extra read, refetching is the least machinery for the strongest correctness. If you cannot, but your payloads carry full state and a version, the version check costs nothing at runtime. Only when every event must be applied, and applied in sequence, do you need the queue — and by then you are buying head-of-line blocking deliberately rather than discovering it.

Ordered delivery on Webhooker

Ordered delivery is available on the Team plan — it is not part of Free or Pro, and it is one of the few webhook features that is genuinely tier-gated rather than metered, which is worth knowing when you compare pricing models across vendors. It is configured per destination, so you can serialize the one endpoint that runs a state machine while the rest of your destinations keep delivering in parallel at full speed. That granularity is the point: turning FIFO on globally means paying head-of-line blocking for traffic that never needed ordering.

Everything else stays as it is. Delivery remains at-least-once, retries and the circuit breaker still apply, and the dead-letter queue still catches exhausted deliveries with their full per-attempt history. Ordering constrains when a delivery is attempted; it does not change how many times it may succeed. Pair it with an idempotent consumer, always. The surrounding per-destination delivery controls — transformations, header injection, outbound signing, rate limits — are the same on every plan, so you can start routing events on the free tier before deciding whether you need ordered delivery on the Team plan.

Frequently asked questions