Running slow AI jobs in the background with queues

LLM calls, transcription, image generation and document analysis all share one trait: they are slow and unpredictable compared to a normal database query. Running slow AI jobs in the background with queues is the pattern that keeps them from dragging the rest of your app down with them. Put the work on a queue, let workers process it at a controlled pace, and tell the user when it is done.

The underlying lesson comes from the Buffdudes backend. There, heavy recurring jobs ran on cron, and as data grew some runs took longer than the interval between them, so a new run would start while the last one was still going. Moving that work into background services with controlled concurrency fixed it, as I describe in how I scaled a fitness platform to 60K users. AI workloads hit the same wall much sooner, because a single job can take seconds or minutes.

When a job belongs in the background

My rule of thumb: if the work can take more than a couple of seconds, can fail and need a retry, or calls a rate-limited external API, it goes on a queue. That covers most AI work:

• Summarizing or analyzing uploaded documents
• Transcribing audio
• Generating or processing images
• Batch extraction or classification over many records
• Anything an agent does in many steps

Short, interactive calls (a chat reply the user is watching stream in) can stay in the request. Everything else gets a job ID.

A Postgres job table is enough to start

You do not need new infrastructure for your first queue. If you already run PostgreSQL, a jobs table with FOR UPDATE SKIP LOCKED gives you a safe, concurrent queue with full visibility in SQL. A Redis-backed queue library is a fine choice too, especially if Redis is already in your stack. I'd start with whichever you already operate.

Here is the core of a Postgres-backed worker using pg:

import { Pool } from "pg";

const pool = new Pool();
const MAX_ATTEMPTS = 5;

async function claimJob() {
  const { rows } = await pool.query(`
    UPDATE ai_jobs
       SET status = 'running', attempts = attempts + 1, started_at = now()
     WHERE id = (
       SELECT id FROM ai_jobs
        WHERE status = 'queued' AND run_after <= now()
        ORDER BY created_at
        FOR UPDATE SKIP LOCKED
        LIMIT 1
     )
    RETURNING id, type, payload, attempts`);
  return rows[0];
}

export async function runWorker(handlers: Record<string, (payload: any) => Promise<unknown>>) {
  for (;;) {
    const job = await claimJob();
    if (!job) {
      await new Promise((r) => setTimeout(r, 2000));
      continue;
    }
    try {
      const result = await handlers[job.type](job.payload);
      await pool.query(
        `UPDATE ai_jobs SET status = 'done', result = $2, finished_at = now() WHERE id = $1`,
        [job.id, result]
      );
    } catch (err) {
      const giveUp = job.attempts >= MAX_ATTEMPTS;
      const delaySeconds = 10 * 2 ** job.attempts;
      await pool.query(
        `UPDATE ai_jobs
            SET status = $2, last_error = $3,
                run_after = now() + $4::int * interval '1 second'
          WHERE id = $1`,
        [job.id, giveUp ? "failed" : "queued", String(err), delaySeconds]
      );
    }
  }
}

SKIP LOCKED lets several workers poll the same table without ever claiming the same job. Run more worker processes to raise throughput, fewer to stay under a provider's rate limit. Concurrency becomes a number you control.

Retries, backoff and idempotency

AI APIs fail in ordinary ways: rate limits, timeouts, overloaded responses. The worker above retries with exponential backoff and gives up after a fixed number of attempts, leaving a failed row with the last error for someone to inspect. For the HTTP side of this (honouring retry headers, jitter, timeouts), see handling LLM API rate limits, retries and timeouts.

Two more things make retries safe:

• Idempotent handlers. A job may run twice if a worker crashes after finishing the work but before marking it done. Write results keyed by job ID, so a second run overwrites the first result.
• A reaper for stuck jobs. If a worker dies mid-job, its row stays running forever. A small scheduled query that requeues running jobs older than a sensible timeout fixes that.

Log every run, alert on the bad ones

The background is where failures go to hide. Nobody sees an error in a worker unless you make it visible.

The clearest example from my own work is a scraper I run nightly. Every run and every fetch is logged to run tables, and I get an email alert when a run is blocked or mostly fails. A preflight check runs before each run, failed requests retry with exponential backoff, and if requests are refused the run pauses, asks again, and then stops instead of hammering a server that has said no. It is a different workload from AI jobs, but the operational lessons transfer directly:

• Record each run and each unit of work with start time, end time, status and error. When something looks wrong, one SQL query tells you what happened.
• Alert on failure rates. One failed job is noise. A run where most jobs failed is a signal.
• Check before you start. A cheap preflight (is the API reachable, is the key valid, is there quota left) avoids burning a whole batch on a problem you could have seen in one request.
• Stop when the other side says stop. Repeated refusals or rate-limit responses mean back off and halt, never retry harder.

For AI jobs, add token usage and model latency to the per-job record. That turns the jobs table into your cost and performance dashboard for free.

Keep the user in the loop

A background job is invisible to the user unless the UI shows it. The flow I'd build:

1. The request creates a job and returns its ID immediately, with a 202 Accepted.
2. The UI shows a clear "processing" state tied to that job.
3. The client polls a status endpoint, or the server pushes an update when the job finishes.
4. Failures show a human message and a retry button.

For long jobs, report progress in stages ("transcribing", "summarizing") when you can. I cover these states in more depth in UX patterns for AI features.

Where I'd start

Add a jobs table, one worker process and exponential backoff, and move your slowest AI call onto it. Log every run with its status and error, and set an alert for runs that mostly fail. Once that is stable, tune concurrency against your provider's limits and add progress updates for the user.