# Background Jobs Done Right: Queues, Retries and the Outbox Pattern > Background jobs fail in quiet ways. Idempotency, retries with backoff, dead-letter queues and the outbox pattern, with Postgres and BullMQ code. - Author: Tai Pham Phu - Published: 14 Oct 2026 - Tags: Backend, Architecture, System Design, Software Design, Queues - URL: https://phamphutai.com/2026/10/14/background-jobs-done-right/ A user signs up. Your code saves the account, then sends a welcome email, then creates a customer in the payment provider, then posts a message to the team's Slack. The email provider is slow today. The request takes nine seconds. The user clicks the button again. Now you have two accounts, or one account and an error page, or one account and no customer in the payment provider. Nobody knows which until a support ticket arrives. The usual fix is "put it in a background job." Right instinct. But a background job isn't where problems disappear. It's where they get quieter. This post is about the handful of patterns that make them loud again. In [Integrating Messy APIs](https://phamphutai.com/2026/10/16/integrating-messy-apis-booking-platforms-and-messaging/), I described "receive fast, process later" for webhooks: store the raw event, answer quickly, do the real work in a worker. This post picks up where that one stops. What happens inside "process later," and how do you get work into the queue without losing it on the way? --- ## Why work leaves the request path A job adds a queue, a worker process and a new way to fail, so be clear about why you're moving something. Work belongs in the background when it is: - **slow.** Generating a PDF, resizing images, calling an AI model, sending a batch of emails. The user shouldn't wait for it. - **flaky.** Anything that calls another company's API. It will time out sometimes, and you want to retry without the user clicking again. - **retryable later.** If it fails now, trying again in five minutes is a fine outcome. - **not needed for the response.** The user needs to know "your account exists," not "the welcome email was accepted by the mail server." Charging a card at checkout stays in the request: you can't show "order confirmed" and find out later the payment failed. A simple test: **if this step fails, should the user see an error right now?** If yes, keep it in the request. If no, it's a candidate for a job. --- ## Every job will run twice Most queues promise **at-least-once delivery**: every job will be processed, but some will be processed more than once. A worker finishes the work, then crashes before it reports success. The queue thinks the job never finished, so another worker runs it again. That's not a bug. It's the trade-off almost every queue makes, because "exactly once" across a network is close to impossible. So the rule is simple: **every job must be idempotent.** Running it twice must have the same effect as running it once. Three tools cover most cases. **1. Upserts instead of inserts.** If the job writes data, write it in a way that can safely repeat. In Postgres: ```sql INSERT INTO invoices (order_id, amount_cents, status) VALUES ($1, $2, 'issued') ON CONFLICT (order_id) DO NOTHING; ``` The unique constraint on `order_id` does the hard work. The second run changes nothing. **2. Idempotency keys for external calls.** Many payment and messaging APIs accept an idempotency key, a string you send with the request so the provider can recognize a repeat. Stripe is a well-known example. Derive the key from your own data, like `charge-order-8812`, never from a random value created inside the job, or every retry gets a new key and the protection is gone. **3. An "already processed" table.** When neither of the above fits, record what you've done: ```sql CREATE TABLE processed_jobs ( job_key text PRIMARY KEY, processed_at timestamptz NOT NULL DEFAULT now() ); ``` The job does its database work and inserts its key **in the same transaction**. A conflict means the job already ran, so roll back and stop. For side effects outside your database, like sending an email, this narrows the window but can't close it, which is why external idempotency keys matter. On many teams, idempotency only arrives after the first duplicate charge. It's much cheaper to design it in from the first job. --- ## Retries: backoff, jitter and knowing when to stop A job failed. Should you try again? Usually yes, but not immediately, not forever, and not for every kind of error. **Exponential backoff** means waiting longer after each failure: 1 second, then 2, then 4, then 8. If the other service is struggling, you give it room to recover instead of hammering it. **Jitter** adds randomness to that wait. Without it, a thousand jobs that failed at the same moment all retry at the same moment, and the other service falls over again. With jitter, the retries spread out. **Max attempts** puts a ceiling on it. After a set number of tries, the job is marked failed and stops. Then the important question: **which errors are worth retrying at all?** | Retry | Don't retry | |---|---| | Timeouts, connection resets | Validation errors (bad input stays bad) | | HTTP 429 (rate limited), 502, 503, 504 | HTTP 400, 401, 403, 404, 422 | | Database deadlocks or serialization failures | The record the job refers to was deleted | | A dependency that's briefly down | A bug in your code that throws every time | Retrying a 400 Bad Request five times just delays the moment someone notices. Fail it fast. Here's what that looks like with [BullMQ](https://docs.bullmq.io/), a popular Node.js queue built on Redis. Retry settings are passed when you add the job: ```ts import { Queue } from "bullmq"; const connection = { host: "localhost", port: 6379 }; export const emailQueue = new Queue("emails", { connection }); await emailQueue.add( "welcome", { userId: "usr_42" }, { jobId: "welcome-usr_42", // same id twice = second add is ignored attempts: 6, backoff: { type: "exponential", delay: 2000, jitter: 0.5 }, removeOnComplete: 1000, // keep the last 1000 completed jobs removeOnFail: false, // keep failed jobs so someone can look at them }, ); ``` With `type: "exponential"` and a 2-second delay, BullMQ waits roughly 2, 4, 8, 16 and 32 seconds between attempts, and `jitter: 0.5` shortens each wait by a random amount, up to half. A custom `jobId` also gives you cheap deduplication: BullMQ ignores an `add` whose ID already exists in the queue. Two details from the docs: custom IDs can't contain a colon, and can't be digits only. And the worker, which refuses to retry errors that will never succeed: ```ts import { Worker, UnrecoverableError } from "bullmq"; const worker = new Worker( "emails", async (job) => { const user = await db.users.findById(job.data.userId); if (!user) throw new UnrecoverableError(`user ${job.data.userId} not found`); const res = await mailer.send({ to: user.email, template: "welcome" }); if (res.status >= 400 && res.status < 500 && res.status !== 429) { throw new UnrecoverableError(`mailer rejected: ${res.status}`); } if (!res.ok) throw new Error(`mailer error: ${res.status}`); // retried }, { connection, concurrency: 5 }, ); ``` A normal `Error` uses the retry settings. An `UnrecoverableError` moves the job straight to failed, no matter how many attempts are left. --- ## Dead-letter queues, and actually looking at them A **dead-letter queue** (DLQ) is where jobs go when they've used up their retries. Some systems have a separate queue for this. In BullMQ, the "failed" set plays the same role, which is why the example keeps failed jobs with `removeOnFail: false`. A dead-letter queue nobody reads is just a slower way to lose data. Make it part of normal work: - **Alert when anything lands in it**, in a place the person on call actually sees. - **Store the error and the attempt count** with each failed job, so you can see why without digging through logs. - **Make replay a button or a one-line command.** Once you've fixed the bug, you should be able to push failed jobs back into the queue in seconds. This only works safely because the jobs are idempotent. - **Decide what "too old" means.** A failed "send welcome email" from three weeks ago probably shouldn't be replayed. Discard it on purpose. --- ## The dual-write problem Now the subtle part. Look at this handler: ```ts // Before: two writes, two systems, no shared transaction async function placeOrder(input: OrderInput) { const order = await db.orders.insert(input); // write 1: Postgres await queue.add("order-placed", { orderId: order.id }); // write 2: Redis return order; } ``` It looks fine. It isn't. Nothing makes these two systems succeed or fail together: - The database insert succeeds, then Redis is unreachable for two seconds. The order exists, but no confirmation email, no invoice, no stock update. Ever. - You flip the order and enqueue first. The job is added, then the database insert fails. A worker picks up a job for an order that doesn't exist. - You wrap both in a database transaction. The job is added, then the transaction rolls back. Same problem, harder to see. This is the **dual-write problem**: whenever one operation writes to two systems, one write can succeed while the other fails. Retries in the worker don't help, because the job was never created in the first place. --- ## The fix: the transactional outbox The **transactional outbox** pattern removes the second system from the request. Instead of writing to the queue, the handler writes a row to an `outbox` table, **in the same database transaction** as the business data. Either both rows are saved or neither is. A separate process, the **relay**, reads new outbox rows and pushes them to the queue. If the relay crashes or Redis is down, the rows simply wait. Nothing is lost, because the database is now the single source of truth.
REQUEST PATH Request ONE DB TRANSACTION orders row outbox row both commit, or neither poll Relay Queue Worker BACKGROUND mark published
The request only writes to the database. The relay moves outbox rows to the queue later, so a queue outage delays work instead of losing it.
The table is small: ```sql CREATE TABLE outbox ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), topic text NOT NULL, -- e.g. 'order-placed' payload jsonb NOT NULL, -- IDs, not whole objects created_at timestamptz NOT NULL DEFAULT now(), published_at timestamptz, attempts int NOT NULL DEFAULT 0, last_error text ); CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL; ``` The partial index keeps the relay's query fast even when the table holds months of published rows. (Clean up old published rows on a schedule, too.) The handler now writes both rows in one transaction. With [node-postgres](https://node-postgres.com/): ```ts // After: one transaction, one system async function placeOrder(input: OrderInput) { const client = await pool.connect(); try { await client.query("BEGIN"); const { rows } = await client.query( "INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2) RETURNING id", [input.customerId, input.totalCents], ); await client.query( "INSERT INTO outbox (topic, payload) VALUES ($1, $2)", ["order-placed", { orderId: rows[0].id }], ); await client.query("COMMIT"); return rows[0]; } catch (err) { await client.query("ROLLBACK"); throw err; } finally { client.release(); } } ``` And the relay, a small loop that runs in its own process: ```ts async function relayOnce(batchSize = 100) { const client = await pool.connect(); try { await client.query("BEGIN"); const { rows } = await client.query( `SELECT id, topic, payload FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT $1 FOR UPDATE SKIP LOCKED`, [batchSize], ); for (const row of rows) { await queue.add(row.topic, row.payload, { jobId: `outbox-${row.id}` }); } await client.query( "UPDATE outbox SET published_at = now() WHERE id = ANY($1)", [rows.map((r) => r.id)], ); await client.query("COMMIT"); return rows.length; } catch (err) { await client.query("ROLLBACK"); throw err; } finally { client.release(); } } ``` Call it every second or so. Three things do quiet work here: - **`FOR UPDATE SKIP LOCKED`** lets you run two relays without both grabbing the same rows. Each one locks a batch and skips rows another relay already holds. - **`jobId: outbox-${row.id}`** means that if the relay crashes after adding jobs but before marking them published, the next run tries to add the same jobs again, and BullMQ ignores the duplicates while they're still in the queue. - **The worker is still idempotent.** The outbox gives you at-least-once delivery into the queue, not exactly-once. That's fine, because you designed for it. In production, also record `attempts` and `last_error` when a publish fails, so a stuck row shows up instead of failing silently. --- ## Payloads should carry IDs, not objects Notice the outbox payload is `{ orderId }`, not the whole order. That's deliberate. ```ts // Before: a snapshot that goes stale await queue.add("send-receipt", { order: fullOrderObject }); // After: a pointer to the truth await queue.add("send-receipt", { orderId: order.id }); ``` Jobs can sit in a queue for minutes, or for days if they fail and get replayed after a fix. By then the order may have changed. A job with an ID loads the current state when it runs. A job with a snapshot acts on data that may no longer be true. IDs also keep payloads small, keep personal data out of the queue, and make a deleted record easy to spot: load it, find nothing, throw `UnrecoverableError`. The exception is when you genuinely need the state at a moment in time, like "the price the customer saw." Then store that snapshot in the database and pass its ID. --- ## Postgres-only queues You may not need Redis at all. The same `FOR UPDATE SKIP LOCKED` trick lets Postgres act as a job queue: workers select a batch of pending jobs, lock them, process them and mark them done, without stepping on each other. Libraries like pg-boss (Node.js) and Graphile Worker are built on this idea. The big advantage is that enqueueing a job becomes a plain insert **in the same transaction as your business data**, so the dual-write problem disappears without a separate outbox and relay. The trade-off is extra write load on your main database. For most small teams with modest job volume, that's not a problem, so start there and move to a dedicated queue when you have a measured reason. This is the same argument I make in [Start With a Modular Monolith](https://phamphutai.com/2026/10/14/start-with-a-modular-monolith/): fewer moving parts until you've earned more. --- ## Scheduled jobs and overlapping runs Cron jobs, jobs that run on a timetable, have a classic failure. The nightly sync usually takes ten minutes. One night it takes seventy, the next run starts, and two copies update the same rows. On more than one server it's worse: each server's scheduler fires. Fixes, from simplest: - **Let the queue own the schedule.** BullMQ has job schedulers: `queue.upsertJobScheduler("nightly-sync", { pattern: "0 0 3 * * *" }, { name: "sync" })` creates one schedule no matter how many servers call it, and the docs note it only creates the next job once the previous one has started. - **Take a lock for the run.** In Postgres, `SELECT pg_try_advisory_lock(42)` returns `true` for exactly one session at a time. If it returns `false`, another run is in progress, so exit quietly. Release it with `pg_advisory_unlock(42)` when done, or use the transaction-level `pg_try_advisory_xact_lock`, which releases itself on commit or rollback. - **Make the job itself safe to overlap.** Process "everything changed since the last successful run" and record that checkpoint at the end. Then even an overlap only repeats work, which your idempotent design already handles. --- ## Visibility: the queue should tell you when it's sick For each job, you want to be able to answer: **what's its status, how many attempts has it had, and what was the last error?** BullMQ stores all three on the job. If you roll your own, put them in columns, like the `attempts` and `last_error` columns on the outbox table. For the queue as a whole, the two numbers that matter most: - **Queue depth:** how many jobs are waiting. A steadily growing number means workers can't keep up, or have stopped. - **Age of the oldest waiting job:** the better signal. 5,000 jobs that clear in a minute is healthy. 3 jobs where the oldest is two hours old is not. Alert on both, plus the dead-letter count and the outbox's oldest unpublished row. That last one catches a relay that has quietly died, which is exactly the silent-absence failure I described in the [messy APIs post](https://phamphutai.com/2026/10/16/integrating-messy-apis-booking-platforms-and-messaging/). --- ## Graceful shutdown Every deploy stops your workers. Stop them cleanly: take no new jobs, finish the current ones, then exit. ```ts async function shutdown(signal: string) { console.log(`${signal} received, closing worker`); await worker.close(); // stops fetching, waits for active jobs await pool.end(); process.exit(0); } process.on("SIGTERM", () => void shutdown("SIGTERM")); process.on("SIGINT", () => void shutdown("SIGINT")); ``` BullMQ's `worker.close()` waits for active jobs to finish or fail, and it has no built-in timeout. So keep individual jobs short. Split long work into smaller jobs, and make sure your platform gives the process enough time between SIGTERM and a hard kill. If a worker dies anyway, BullMQ marks its active jobs as stalled and another worker picks them up. Once again, idempotency is what makes that safe. --- ## A checklist before you ship a job - [ ] Does this work really need to leave the request? Would the user need to see this error right now? - [ ] Is the job idempotent? Upsert, idempotency key or processed-jobs table? - [ ] Are retries set with exponential backoff, jitter and a max attempts limit? - [ ] Do permanent errors (400s, missing records) fail fast instead of retrying? - [ ] Do failed jobs land somewhere someone will see, with the error attached, and can they be replayed? - [ ] Is the job enqueued through an outbox, or in the same Postgres transaction, rather than as a second write? - [ ] Does the payload carry IDs instead of whole objects? - [ ] Can scheduled runs overlap? If so, is there a lock? - [ ] Are there alerts on queue depth, oldest job age and dead-letter count? - [ ] Does the worker shut down gracefully on deploy? None of this is clever. It's a set of habits that turn background jobs from a place where work goes missing into a place where work always ends up done, retried, or loudly failed. Those are the only three outcomes you want. --- *Sources: BullMQ documentation (retrying failing jobs, stop retrying jobs, job IDs, job schedulers, graceful shutdown) at docs.bullmq.io; PostgreSQL documentation on SELECT ... FOR UPDATE SKIP LOCKED and advisory locks.* *Related: [Integrating Airbnb, Booking.com and WhatsApp: Lessons From Messy APIs](https://phamphutai.com/2026/10/16/integrating-messy-apis-booking-platforms-and-messaging/), [Start With a Modular Monolith, Not Microservices](https://phamphutai.com/2026/10/14/start-with-a-modular-monolith/), and [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/).*