LLM API rate limits, retries and timeouts in Node.js

Model APIs fail in ways your other dependencies mostly don't. They rate limit you by requests and by tokens. They get slow under load and a call that normally takes two seconds takes forty. They return a server error, then succeed on the next try. Handling LLM API rate limits, retries and timeouts in Node.js well is the difference between an AI feature that feels solid and one that falls over the first busy afternoon.

Sort errors before you retry them

The first rule of retries: only retry what can succeed on a second attempt.

• Retry: 429 (rate limited), 408, 500, 502, 503, 504, network resets and your own timeouts.
• Do not retry: 400 (your request is malformed), 401 and 403 (your key or permissions are wrong), 404, and 413 or similar "input too long" errors. Those need a code or config fix, so fail fast and log loudly.

Context changes the table. On a scraper I run nightly, a 403 from a store can be a temporary refusal, so that job retries 429 and 403 with exponential backoff. From a model API, a 403 means your credentials or permissions are wrong, and retrying it just wastes time. Classify per upstream, from what that upstream's errors actually mean.

Exponential backoff with jitter, and Retry-After first

When you do retry, wait longer each time and add randomness. Without jitter, every client that failed at the same moment retries at the same moment, and you rebuild the spike that caused the failure.

If the response includes a Retry-After header, use it. The API is telling you exactly how long to wait, which beats any formula.

Here is the helper I'd start from. It uses only fetch and AbortSignal.timeout, both built into modern Node.js:

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);
class NonRetryableError extends Error {}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export async function callModelWithRetry(body: unknown, maxAttempts = 4): Promise<unknown> {
  let lastError: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    // Full jitter: a random wait between 0 and the exponential cap.
    let waitMs = Math.random() * Math.min(20_000, 500 * 2 ** attempt);
    try {
      const res = await fetch(process.env.LLM_API_URL!, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: `Bearer ${process.env.LLM_API_KEY}`,
        },
        body: JSON.stringify(body),
        signal: AbortSignal.timeout(45_000),
      });
      if (res.ok) return await res.json();
      if (!RETRYABLE.has(res.status)) {
        throw new NonRetryableError(`Model API returned ${res.status}`);
      }
      lastError = new Error(`Model API returned ${res.status}`);
      const retryAfter = Number(res.headers.get("retry-after"));
      if (retryAfter > 0) waitMs = retryAfter * 1000;
    } catch (err) {
      if (err instanceof NonRetryableError) throw err;
      lastError = err; // network failure or timeout
    }
    if (attempt < maxAttempts - 1) await sleep(waitMs);
  }
  throw lastError;
}

Retry-After can also be an HTTP date. Number() turns that into NaN, the check fails and the code falls back to the computed backoff, which is a safe default.

Timeouts that match the work

Every model call needs a timeout. Without one, a hung connection holds a request, a socket and possibly a database transaction open indefinitely.

Size the timeout to the task. A short classification can have a tight one. A long generation needs more room, and if a user is waiting, it probably should be streamed so they see progress instead of a spinner. Also check the layers around your app: load balancers and reverse proxies have their own idle timeouts, and they will cut a long call off before your code's timeout ever fires.

Keep a total deadline on top of per-attempt timeouts. Four attempts at 45 seconds each plus backoff can add up to minutes, which is fine for a background job and unacceptable for an HTTP request. Interactive endpoints get fewer attempts. Slow work belongs in a queue, which I cover in running slow AI jobs in the background.

Retries must be safe to repeat

Retrying the model call itself is usually harmless; you might pay for an answer twice. The danger is in what happens around it. If your handler calls the model, sends an email and writes a row, and the retry wraps the whole handler, a timeout after the email went out means the user gets two emails.

• Retry the narrowest unit, the model call, and never the whole workflow.
• Use idempotency keys for side effects. Store a key per logical operation and skip work whose key is already recorded.
• Save results before acting on them, so a retry after a crash can resume from the stored output instead of generating a new one.

Break the circuit when the upstream is down

Backoff handles a blip. It does nothing useful during an outage, where every request burns its full retry budget, piles up and drags the rest of your app down with it.

The scraper taught me the pattern I now apply to model APIs. A preflight check runs before the job starts, so a run never begins against an upstream that is already refusing. When it gets refused, it pauses all requests for 15 minutes, then asks again before deciding to halt. If refusals keep coming across every target, the run stops instead of hammering. Every run and fetch is logged to run tables, and an email alert goes out when a run is blocked or mostly fails.

For an LLM feature, the same circuit breaker looks like this: count recent failures, and past a threshold stop calling the API for a cooldown period. During the cooldown, fail fast with a clear message or a fallback (a cached answer, a simpler non-AI path, a "try again shortly" state). After the cooldown, let one probe request through. If it succeeds, close the circuit.

Rate limit your own API too

Your users can generate load faster than your model quota allows. Limiting them at the edge protects both the quota and your budget. On a compliance and risk platform I work on, Nginx rate-limits the API at a level the app can sustain. A minimal version:

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        limit_req_status 429;
        proxy_pass http://app_upstream;
    }
}

Those numbers are placeholders. Set them from what your app and your model quota can actually handle, and add per-user quotas in the app for anything expensive. I go into the cost side of that in keeping LLM API costs under control.

The short version

Classify errors and retry only the retryable ones. Back off exponentially with jitter and honor Retry-After. Put a timeout on every call and a deadline on every request, make side effects idempotent, and add a circuit breaker so an outage fails fast. Then rate limit your own API so users can't spend your quota for you.