Webhook delivery - retrying into somebody else's outage

Why this design

Payment providers, code hosts, CRMs and chat tools all send webhooks: an HTTP POST to a URL the customer registered, saying "this happened". It looks like one outbound request. It is really a delivery system with a strange dependency - the most important component, the receiver, belongs to somebody else and fails whenever it likes. The design question is how to keep a promise to a server you cannot see, without hammering it when it is down and without letting one broken customer slow down all the others.

What you will learn

• Why the event is stored before any delivery is attempted (at-least-once, never at-most-once).
• How 75M deliveries a day and a 300 ms median response turn into ~1,000 requests in flight - and 7,000 when one big customer hangs.
• Exponential backoff with jitter: eight attempts over about a day, and why the jitter matters as much as the backoff.
• Why receivers get duplicates, and how a signature and an event id let them verify and dedupe.
• The noisy-neighbour problem, and the per-endpoint limits and circuit breakers this board leaves out.

How to read this board

1. Top: EVENTS IN. Your product emits a domain event; the dispatcher looks up who subscribed and creates one delivery per endpoint.
2. Middle: THE ATTEMPT LOOP. The delivery queue, the red shared worker pool, the retry scheduler, and on the right, outside the boundary, the customer endpoint.
3. Bottom: RECORDS. Every attempt is logged; failed-for-good deliveries go to dead letters; the developer sees both in a dashboard and can replay.
4. Press play, then read pages 1 to 6, then the tables. The conversation to have is the red box: what happens to customer B when customer A is down?

Functional

• Customers register endpoints (an HTTPS URL) and choose which event types each receives.
• When an event happens, POST a JSON payload to every subscribed endpoint.
• A 2xx response within 10 seconds is success. Anything else - timeout, connection refused, 4xx, 5xx - is retried.
• Retry with backoff for about 24 hours (8 attempts), then mark the delivery failed and keep it for manual replay.
• Sign every payload (HMAC-SHA256 with a per-endpoint secret) and include a timestamp so receivers can reject forgeries and replays.
• A dashboard showing every attempt, with response code and time, and a "resend" button.
• Disable an endpoint automatically after several days of continuous failure, and email the owner.

Non-functional

• At-least-once: an accepted event is never silently dropped. Duplicates are allowed and documented.
• First attempt within 5 seconds of the event, p99.
• One customer's failing endpoint must not delay another customer's deliveries - stated here, and NOT met by this board.
• Never make our retries a denial of service against a recovering customer.
• Ordering is not guaranteed - each event carries its own timestamp and id.

Out of scope

• Ordered delivery per endpoint (needs one in-flight delivery per endpoint, much slower).
• Exactly-once delivery - impossible across a network to a server we do not control.
• Inbound webhooks (receiving them).