Designing a Webhook Retry Policy

The first time a webhook retry policy cost us real money, it wasn't because a webhook failed. It was because a webhook succeeded, and then we sent it four more times. A payment partner's endpoint returned a 200 but took nine seconds to do it, our HTTP client timed out at eight, and our retry loop dutifully fired again. By the time the dust settled, a merchant had been credited five times for the same settlement, and I spent a Saturday writing a reconciliation script and an apology.

That is the thing about retries. They feel like a safety net, and mostly they are. But a badly designed retry policy is not a safety net, it is a slow-motion amplifier that turns one transient hiccup into a cascade. After more than a decade shipping payment infrastructure, I have come to treat the retry policy as a first-class part of the system design, not a checkbox on the delivery worker. Here is how I think about it.

Retries are a contract, not a feature

When you retry a webhook, you are making a promise to the receiver: "I will keep trying until you acknowledge me, and I will not lie to you about what happened." That promise only holds if both sides agree on what an acknowledgement means. The single most common failure I see is treating any non-2xx as a retryable failure and any 2xx as a permanent success, with nothing in between. Real endpoints are messier than that.

A 429 means slow down, not stop. A 500 might be transient or might be a bug that will fail identically forever. A connection reset halfway through the request body means you have no idea whether the receiver processed it. Your policy has to encode a point of view about each of these, and that point of view should be written down somewhere a human can read it, because six months from now someone will ask why event 8f3a retried for two days and you will want an answer better than "that's what the code did."

Idempotency comes before retries, always

You cannot design a retry policy in isolation. Retries only become safe once the receiver can tolerate seeing the same event twice, and that means every webhook must carry a stable idempotency key that never changes across retries. Not a timestamp, not a per-attempt UUID. The event ID, generated once when the event is born, and reused for every single delivery attempt.

I have watched teams get this subtly wrong by regenerating the payload on each attempt, which re-serialized a timestamp field and produced a different signature, which the receiver treated as a new event. The fix is to freeze the payload and the key at creation time and store them together. Retry the exact bytes, every time. If you find yourself rebuilding the request body inside the retry loop, stop and move that work upstream.

A retry without an idempotency key is not a retry. It is a second event that happens to look like the first one, and the receiver has no way to tell them apart.

Exponential backoff, with jitter that actually matters

Linear retries are a trap. If you retry every 30 seconds, a receiver that goes down for an hour gets 120 hammering requests the moment it recovers, right when it is most fragile. Exponential backoff spreads those attempts out: 10 seconds, then 30, then 90, then 5 minutes, then an hour, capped somewhere sensible. The math is simple and the effect is dramatic.

The part people skip is jitter. If ten thousand webhooks all failed at 14:00:00 because a shared dependency blinked, and they all use identical backoff, they will all retry at exactly 14:00:10, then 14:00:40, in perfect synchronized waves. You have built a thundering herd with a schedule. Adding randomized jitter to each interval smears those retries across the window and keeps the recovering endpoint from being knocked over again by your own good intentions.

public TimeSpan NextDelay(int attempt)
{
    // Base doubles each attempt, capped at 1 hour
    var baseSeconds = Math.Min(10 * Math.Pow(2, attempt), 3600);

    // Full jitter: pick a random point in [0, base]
    // so retries never line up across events
    var jittered = _rng.NextDouble() * baseSeconds;

    return TimeSpan.FromSeconds(jittered);
}

Full jitter, where you pick a random value between zero and the computed ceiling, is my default. It sacrifices a little predictability for a lot of spread, and in a fleet of thousands of concurrent deliveries that trade is worth it every time.

Classify the failure before you decide to retry

Not everything deserves a retry, and retrying the wrong things is how you burn a receiver's rate limit and your own worker capacity for nothing. I break responses into three buckets, and the policy branches on which bucket you land in:

• Retryable: timeouts, connection failures, 429, 502, 503, 504. These are usually transient. Back off and try again.
• Terminal: 400, 401, 403, 410, 422. A malformed payload or a revoked endpoint will fail identically forever. Retrying is just noise. Fail fast and surface it.
• Ambiguous: a generic 500, or a socket that died mid-flight. You genuinely don't know if it worked. Retry, but this is exactly why idempotency is non-negotiable.

That 410 Gone case is worth a special mention. When a merchant deletes their webhook endpoint, a well-behaved gateway returns 410, and that is your signal to permanently disable the subscription, not to keep pounding a dead URL for three days. We once had 40% of our retry queue devoted to endpoints that had been decommissioned months earlier, purely because nobody had taught the policy to listen for that signal.

Where retries end: the dead-letter decision

Every retry policy needs a hard stop. Infinite retries are not resilience, they are a queue that only grows. The question is what happens at the stop, and the honest answer is that "give up silently" is the worst possible choice for a payment system. If a settlement notification never reaches the merchant and nobody knows, you have a reconciliation problem that surfaces weeks later as an angry support ticket.

My rule of thumb is to retry across a window that matches the business meaning of the event, not an arbitrary attempt count. For a payment status change, I want retries spread across roughly 24 hours, because that covers a receiver's overnight maintenance window and gives their on-call a fair chance to fix things. After that, the event goes to a dead-letter store with its full history, and it triggers an alert plus a way to manually replay it once the endpoint is healthy again. Dead-lettered does not mean deleted.

Visibility is the real feature

The retry logic is maybe a third of the work. The rest is being able to answer questions about it. When a merchant emails saying "I never got the webhook for order 12345," you need to pull up every attempt: when it fired, what status came back, how long the response took, what the next scheduled attempt is. If your only record is a log line that rolled off after seven days, you are going to have a bad conversation.

I store delivery attempts as first-class rows, one per attempt, with enough detail to reconstruct the whole story. This table has saved me more incident reviews than any dashboard.

CREATE TABLE webhook_delivery_attempts (
    attempt_id      BIGINT IDENTITY PRIMARY KEY,
    event_id        UNIQUEIDENTIFIER NOT NULL,
    endpoint_id     BIGINT NOT NULL,
    attempt_number  INT NOT NULL,
    status_code     INT NULL,          -- NULL if connection failed
    response_ms     INT NULL,
    outcome         VARCHAR(20) NOT NULL, -- retryable / terminal / success
    attempted_at    DATETIME2 NOT NULL,
    next_retry_at   DATETIME2 NULL     -- NULL once terminal or delivered
);

With this in place, "show me every endpoint whose last ten attempts all failed" is one query, and it becomes an early warning that a partner's integration is drifting before they even notice. Observability turns a retry policy from a black box into something you can actually operate.

Respect the receiver, because they will remember you

Here is an opinion that not everyone shares: your retry policy should be conservative to the point of being slightly annoying to your own product team. It is tempting to retry aggressively so that delivery latency stays low, but you are a guest on someone else's infrastructure. If a receiver returns 429 with a Retry-After header, honor it exactly. If they don't send one, back off anyway. The fastest path to getting your IP range blocked is to treat another company's endpoint as a load test.

I also cap concurrency per endpoint, not just globally. One large merchant having a bad day should not be able to consume every worker in the pool and starve deliveries to everyone else. A per-endpoint concurrency limit of, say, five in-flight requests keeps one struggling receiver from becoming everyone's problem. This is the kind of thing you don't think you need until the night you do.

Test the paths that only fail at 3am

Retry logic is almost impossible to verify by staring at it, because the interesting behavior only shows up under failure. I insist on integration tests that stand up a fake receiver and script it to misbehave on purpose: return 200 but sleep past the timeout, drop the connection mid-body, return 429 three times then succeed, return 500 forever. Each of those maps to a bucket decision, and each should produce a specific, asserted outcome.

The slow-200 case is the one that got me all those years ago, and it is still the one teams forget. A receiver that succeeds after your client gives up is the single most dangerous scenario in the whole system, because the retry is technically correct and completely wrong. The only defense is idempotency on the receiver side plus a generous enough client timeout that you are not fighting normal latency. Test it explicitly, or ship it and find out live.

Conclusion

If I could tattoo one principle on every engineer building delivery infrastructure, it would be this: design the retry policy from the receiver's point of view, not your own. Your instinct is to make delivery fast and guaranteed. Their reality is that your retries land on their servers, consume their rate limits, and duplicate their side effects. A retry policy that is safe for them is almost always the one that is safe for you too, because the failure modes are shared. The counterintuitive part, and the thing I still have to argue for in planning meetings, is that the safest policy is usually the slowest one. Slow is a hard sell in a room full of people optimizing delivery latency down to the millisecond. Make the argument anyway. The engineer who ships the patient, conservative, slightly boring retry policy is the one who sleeps through the outage instead of writing apology emails on a Saturday.