# How to build a queue-backed webhook sender on BullMQ > Record each event and one delivery per endpoint, queue a BullMQ job per delivery, and let workers POST with a timeout and log every attempt, one queue per endpoint. Source: https://voxgig.com/howto/build-a-queue-backed-webhook-sender-on-bullmq - Audience: api-producer - Level: intermediate - Languages: typescript - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Write the event row and one delivery row per subscribed endpoint, then add one BullMQ job per delivery, with the delivery id as the job id. Workers POST with a ten-second timeout, record every attempt with its status and response, and retry on the Standard Webhooks schedule. Give each endpoint its own queue, so a slow customer holds only its own concurrency. --- ## You will need Node 22.18 or later, and Redis for BullMQ in production. The samples run without a server: with no `REDIS_URL` set, they use the [PostgreSQL backend BullMQ ships](https://docs.bullmq.io/guide/postgresql) on PGlite, Postgres compiled to WebAssembly and run in process. With `REDIS_URL` set, `backend.ts` hands every field of the URL to the client: the user, the password, the database number, and the TLS that `rediss://` asks for. Signing follows the [Standard Webhooks specification](https://github.com/standard-webhooks/standard-webhooks/blob/main/spec/standard-webhooks.md), which the how-to on signing outbound webhooks covers in detail. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [BullMQ](https://docs.bullmq.io/guide/queues/global-concurrency) | You run Redis, or will, and want retries, backoff, and a concurrency cap per queue as options | A second data store to persist and watch, and a queue write outside your database transaction | You want no infrastructure beyond Postgres, or no sender to run at all | | [Inline in the request handler](https://github.com/standard-webhooks/standard-webhooks/blob/main/spec/standard-webhooks.md) | A prototype with one trusted receiver on your own network | Your response time becomes your customers' response time, and failures are never retried | Anything a customer can point at a slow or broken server | | [pg-boss](https://pgboss.io/api/jobs) | Postgres is already your database, and you want the job inserted in the event's own transaction | Every job is a row, so queue traffic becomes write load on your primary | Delivery volume would make that write load the database's main job | | [Svix](https://docs.svix.com/retries) | You would rather not operate a sender, and customers want a portal to inspect and replay | A vendor in the delivery path, and pricing by message volume beyond the free tier | Payloads may not leave your infrastructure, or volume makes per-message pricing dear | BullMQ gives you the concurrency cap and the retry schedule as configuration, and costs you Redis and a sweeper for deliveries whose queue write never happened. pg-boss removes the sweeper, because it can [insert a job inside the transaction](https://pgboss.io/api/jobs) that wrote the event, and puts the queue's write load on your database. Svix removes all of it and bills per message. ## Record the event and its deliveries first The request path writes rows, not HTTP. The event and one delivery per enabled endpoint go in together, so an event never exists without the deliveries that carry it. ```ts title="store.ts" createEvent(id: string, type: string, payload: unknown, now = Date.now()): string[] { this.db.exec('BEGIN IMMEDIATE') try { const fresh = this.db.prepare('INSERT OR IGNORE INTO events (id, type, payload, created_at) VALUES (?, ?, ?, ?)') .run(id, type, JSON.stringify(payload), now).changes === 1 const ids: string[] = [] if (fresh) { // The delivery id is also the BullMQ job id, which may not contain a ':'. Both parts // are percent-encoded, '-' included, so the '-' that joins them is the only one, and // two different event and endpoint pairs never share an id. const part = (s: string) => encodeURIComponent(s).replaceAll('-', '%2D') for (const ep of this.db.prepare("SELECT id FROM endpoints WHERE status = 'enabled' ORDER BY id").all() as { id: string }[]) { const deliveryId = `${part(id)}-${part(ep.id)}` this.db.prepare('INSERT INTO deliveries (id, event_id, endpoint_id, created_at) VALUES (?, ?, ?, ?)') .run(deliveryId, id, ep.id, now) ids.push(deliveryId) } } this.db.exec('COMMIT') return ids } catch (err) { this.db.exec('ROLLBACK') throw err } } ``` The delivery id doubles as the BullMQ job id, so adding the same delivery twice adds one job. BullMQ 6.3.9 refuses a custom job id that contains a colon, on Redis and on PostgreSQL alike, so both parts of the id are percent-encoded first. A test sends `order:1001` to an endpoint named `acct:7` to prove it. The encoding covers the hyphen as well, because the hyphen that joins the two parts must be the only one. With a bare hyphen, event `order-1` to endpoint `customer` and event `order` to endpoint `1-customer` would share an id, and the second event's transaction would fail. `INSERT OR IGNORE` on the event id makes a retried `emit` harmless as well. ## Give each endpoint its own queue BullMQ's global concurrency and rate limit apply per queue, across every worker process. Per-customer limits inside one queue are [groups, a feature of BullMQ Pro](https://docs.bullmq.io/bullmq-pro/groups/), the commercial edition. A queue per endpoint gets the same isolation from the open source package. ```ts title="sender.ts" async function queueFor(endpointId: string): Promise { const name = queueName(lane(endpointId)) let queue = queues.get(name) if (!queue) { queue = backend.queue(name) queues.set(name, queue) // The cap holds across every worker process, not only this one. await queue.setGlobalConcurrency(concurrency) workers.push(backend.worker(name, (job) => deliver(job), { concurrency, settings: { backoffStrategy: (attemptsMade: number) => retryDelay(attemptsMade, scheduleMs, random) }, })) } return queue } async function enqueue(deliveryId: string, endpointId: string): Promise { // The delivery id is the job id, so enqueueing the same delivery twice adds one job. await (await queueFor(endpointId)).add('deliver', { deliveryId }, { ...jobOptions, jobId: deliveryId }) } // Called from the request path: two inserts and a queue write, no HTTP. async function emit(event: { id: string; type: string; data: unknown }): Promise { const payload = { type: event.type, timestamp: new Date().toISOString(), data: event.data } const ids = store.createEvent(event.id, event.type, payload) for (const id of ids) await enqueue(id, store.delivery(id)!.endpoint_id) return ids } // A crash between the insert and the queue write leaves a delivery with no job. // Run this on a timer to put such deliveries back on their queue. async function sweep(olderThanMs = 60_000, now = Date.now()): Promise { const stranded = store.pendingSince(now - olderThanMs) for (const d of stranded) await enqueue(d.id, d.endpoint_id) return stranded.length } ``` [`setGlobalConcurrency`](https://docs.bullmq.io/guide/queues/global-concurrency) caps a queue across all workers, so two endpoint slots stay two however many processes you run. The retry waits come from a [custom backoff strategy](https://docs.bullmq.io/guide/retrying-failing-jobs) that follows the specification's example schedule, nine waits from five seconds to a day. The specification asks for random jitter without naming an amount, and the sample adds 20% either way.[^1] On Redis, BullMQ keeps that cap and every job under a key prefix, so the cap holds only across processes that share the prefix. `backend.ts` reads it from `BULLMQ_PREFIX` and falls back to `bull`, BullMQ's own default. The demo and the tests pass `isolate: true` for a prefix of their own, deleted on close, so a second run never meets the first run's job ids. ## Send each attempt with a timeout and log it Each job sends one signed POST, writes one attempt row, and lets BullMQ schedule the retry. ```ts title="sender.ts" async function deliver(job: Job<{ deliveryId: string }>): Promise { const d = store.delivery(job.data.deliveryId) if (!d || d.status !== 'pending') return 'skipped' if (d.endpoint_status !== 'enabled') { store.finish(d.id, 'cancelled') return 'endpoint disabled' } const n = job.attemptsMade + 1 const at = new Date() const started = performance.now() let status: number | null = null let error: string | null = null let response = '' try { const res = await fetch(d.url, { method: 'POST', headers: { 'content-type': 'application/json', 'webhook-id': d.event_id, // the same on every attempt, so the receiver can deduplicate 'webhook-timestamp': String(Math.floor(at.getTime() / 1000)), 'webhook-signature': new Webhook(d.secret).sign(d.event_id, at, d.payload), }, body: d.payload, redirect: 'manual', // a 3xx is a failed attempt, never a request to a host you did not check signal: AbortSignal.timeout(timeoutMs), }) status = res.status response = await readCapped(res, 200) } catch (err) { if (status === null) { error = (err as Error).name === 'TimeoutError' ? `no response within ${timeoutMs} ms` : `connection failed: ${(err as { cause?: { code?: string } }).cause?.code ?? (err as Error).message}` } } store.recordAttempt({ delivery_id: d.id, n, at: at.toISOString(), status, error, duration_ms: Math.round(performance.now() - started), response, }) if (status !== null && status >= 200 && status < 300) { store.finish(d.id, 'delivered') return 'delivered' } if (status === 410) { // The receiver has asked to stop. Disable the endpoint rather than retry. store.disableEndpoint(d.endpoint_id) store.finish(d.id, 'cancelled') throw new UnrecoverableError('410 Gone: endpoint disabled') } if (n >= (job.opts.attempts ?? 1)) store.finish(d.id, 'failed') throw new Error(error ?? `HTTP ${status}`) } ``` Three choices in that function follow the specification's advice on status codes. A 2xx is success and anything else is a failure. A 3xx is not followed, because a redirect sends your signed payload to a host nobody checked. A 410 disables the endpoint instead of retrying it.[^2] The fourth is the 200-byte cap on the logged response, because an endpoint can answer 200 and then stream for as long as you let it. The ten-second timeout matches [what GitHub gives receivers](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks). The specification recommends 15 to 30 seconds, and [Svix allows 30](https://docs.svix.com/retries). A shorter timeout frees worker slots sooner, and a receiver that follows the advice to acknowledge first needs milliseconds. ## Run one event against four endpoints The demo emits `invoice.paid` to four endpoints: one healthy, one that fails twice before succeeding, one that never answers, and one that answers 410. The schedule is scaled to waits of 50, 100 and 200 ms, and the timeout to 300 ms. ```bash node --no-warnings demo.ts ``` ```text output queue backend: postgres (PGlite) endpoint try result response ep_a 1 200 {"received":true} ep_b 1 503 upstream busy ep_b 2 503 upstream busy ep_b 3 200 {"received":true} ep_c 1 no response within 300 ms ep_c 2 no response within 300 ms ep_c 3 no response within 300 ms ep_c 4 no response within 300 ms ep_d 1 410 this endpoint was removed deliveries: ep_a delivered, ep_b delivered, ep_c failed, ep_d cancelled endpoints disabled: ep_d requests received: 9, all verified: true, distinct webhook-id values: 1 ``` The table is the attempt log a customer asks for when their integration missed an event. Each row also stores the attempt's start time and duration, which vary from run to run and are left out of the printout. Every request carried one `webhook-id` and verified against the receiver's secret with the `standardwebhooks` library. `ep_c` used all four attempts and failed; `ep_d` was switched off after one. The same file run with `REDIS_URL` against Redis 7.0.15, in the session that wrote this page and again in review, printed the same lines apart from the backend label. ## Check it worked The suite pins the cases a customer would report: the signature and its `webhook-id`, a timeout, a 410, a redirect, and an endless body. It also covers a duplicate event, an id holding a colon, an id holding a hyphen, the fields `backend.ts` reads from `REDIS_URL`, the shared key prefix, and the sweeper. ```ts title="sender.test.ts" test('the sweeper enqueues a delivery whose queue write never happened', async () => { const t = await setup(['ep_a']) // As if the process died between the insert and the enqueue. const [id] = t.store.createEvent('evt_t6', 'invoice.paid', { type: 'invoice.paid', data: {} }, Date.now() - 120_000) assert.equal(await t.sender.sweep(60_000), 1) await until(() => t.status(id) === 'delivered') assert.equal(await t.sender.sweep(60_000), 0) await t.done() }) ``` ```bash node --test sender.test.ts ``` ```text output # tests 12 # suites 0 # pass 12 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 5052.468398 ``` The sweeper test fakes the crash the queue design leaves open. The rows commit and the process dies before the queue write, so without the sweeper the delivery would stay pending for good. ## When it goes wrong The API's response time tracks the slowest customer, or one customer's outage delays everyone. Sending inline causes the first, and one shared queue causes the second. `pitfall.ts` shows both. ```bash node --no-warnings pitfall.ts ``` ```text output 1. POST /orders with three subscribed endpoints, one of them hanging (1 s timeout) inline answered in 1 s or more failed deliveries, never retried: 2 queued answered in under 1 s after the retries: ep_a delivered, ep_b delivered, ep_c failed 2. Three events to ep_a, ep_b and ep_c, where ep_c hangs (1 s timeout), concurrency 2 one queue for all endpoints ep_a and ep_b deliveries finished before ep_c's first timeout: 4 of 6 one queue per endpoint ep_a and ep_b deliveries finished before ep_c's first timeout: 6 of 6 ``` The inline handler waited out the hanging endpoint and dropped two deliveries for good. The queued handler answered at once and retried in the background. With one shared queue, two hanging deliveries to `ep_c` held both worker slots, so two of the six healthy deliveries waited a full timeout. With thousands of endpoints, the workers hold thousands of Redis connections. Each BullMQ Worker [opens a blocking connection of its own](https://docs.bullmq.io/guide/connections), so a worker per endpoint per process does not scale without limit. Group endpoints into a fixed number of lanes, or move to BullMQ Pro's groups. With BullMQ's default worker settings, retries on PGlite arrived about five seconds late, whatever the schedule said. An idle worker there noticed a delayed job only when its `drainDelay` wait of five seconds ran out, so `backend.ts` shortens it to 50 ms for the demos. In a run during review, four of the five retries to `ep_b` and `ep_c` came five seconds late with the default. Redis needs no such change: the same run there kept to the schedule. ## When not to do this Do not run this to send a few hundred webhooks a day to a handful of customers. [Svix's free tier](https://www.svix.com/pricing/) covers 50,000 messages a month, with a portal your customers can use to replay what they missed. Do not let a worker POST to any URL a customer types. The specification calls webhook senders especially exposed to server-side request forgery, and recommends sending through a proxy that refuses internal addresses. Do not drop the sweeper because the queue write rarely fails. It fails when a deploy kills the process between the commit and `add`, and without the sweeper that delivery is never sent. Do not add BullMQ to send webhooks if Postgres is your only data store. pg-boss, or BullMQ's own PostgreSQL backend, avoids a second system. BullMQ's [PostgreSQL guide](https://docs.bullmq.io/guide/postgresql) puts that backend at about 1.5 to 2 times fewer jobs a second than Redis on similar setups, depending on the workload, which is the cost to weigh. ## Related how-tos - [Limit concurrency in a message worker](/howto/limit-concurrency-in-a-message-worker) - [Verify a Standard Webhooks signature in PHP](/howto/verify-a-standard-webhooks-signature-in-php) - [Make queue consumer writes safe under redelivery](/howto/queue-consumer-writes-under-at-least-once) - [Acknowledge a webhook fast and process it from a queue](/howto/acknowledge-webhooks-fast-and-process-from-a-queue) ## Last verified Verified 2026-10-15 against Node 22.22.0 and 22.23.3, BullMQ 6.3.9, `ioredis` 5.11.1, standardwebhooks 1.1.1, `pg` 8.23.1, PGlite 0.5.10, and `pglite-socket` 0.2.13. Every output block is what the command before it printed, on both Node versions, with BullMQ on its PostgreSQL backend over PGlite. The Redis run is described, not captured. [^1]: The specification's example schedule, from the first attempt to the last, runs to 75 hours, 35 minutes and 5 seconds. [Svix's own schedule](https://docs.svix.com/retries) stops after eight attempts, with its two longest waits both ten hours, and so gives up a little over 27 hours in. The specification calls its schedule an example, which leaves every sender free to be impatient. [^2]: [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) defines 410 Gone as a condition that is "likely to be permanent" and meant to tell the recipient that the server's owners want remote links to the resource removed. A webhook subscription is a remote link with a retry schedule attached, so a sender that disables the endpoint is doing what the RFC asks of anyone holding one.