Drain the webhook retry backlog
Node · Node · intermediate · greenfield
Adds the retry-backlog drain job: it re-signs and re-sends every pending webhook delivery concurrently and reports a per-delivery outcome, with the existing per-delivery timeout and error isolation carried over. Ran it against a staging backlog of a few dozen deliveries and everything went out in one quick burst with correct results.
The webhook subsystem of a payments platform. The drain job runs after provider outages, when the backlog is at its largest; each delivery is an HTTPS POST to a customer endpoint. Node 20, global fetch.
Requirements
- `drainRetryBacklog(deliveries)` re-sends every pending delivery with `deliverOne` and returns per-delivery results `{id, ok, status}` aligned index-for-index with the input; a failed or timed-out delivery yields `ok: false` without affecting the others (`deliverOne` already isolates errors and carries a 5-second timeout).
- After an outage the backlog reaches 15,000-20,000 pending deliveries. The number of in-flight HTTP requests must stay bounded regardless of backlog size — a worker pool, a semaphore, or fixed-size chunks — so the number of concurrently active HTTP exchanges stays capped and receiving endpoints never get an unbounded burst.
- Deliveries may complete in any internal order as long as the returned array aligns with the input.
Files touched
- src/webhooks/drain.js
--- src/webhooks/drain.js
+const crypto = require('node:crypto');
+
+/**
+ * Webhook retry backlog drain. After an outage the backlog can hold
+ * thousands of pending deliveries; this job re-sends them all.
+ */
+
+function sign(body, secret) {
+ return crypto.createHmac('sha256', secret).update(body).digest('hex');
+}
+
+async function deliverOne(delivery) {
+ try {
+ const res = await fetch(delivery.url, {
+ method: 'POST',
+ headers: {
+ 'Content-Type': 'application/json',
+ 'X-Webhook-Signature': sign(delivery.body, delivery.secret),
+ },
+ body: delivery.body,
+ signal: AbortSignal.timeout(5000),
+ });
+ await res.body?.cancel(); // discard the body so the connection is released
+ return { id: delivery.id, ok: res.ok, status: res.status };
+ } catch {
+ return { id: delivery.id, ok: false, status: 0 };
+ }
+}
+