Keeping LLM API costs under control in Node.js

LLM features have a cost profile most web backends have never had. A normal endpoint costs roughly the same whether a user calls it once or a hundred times a day, give or take some database load. An LLM endpoint bills you per token, every time. Keeping LLM API costs under control in a Node.js backend is mostly about treating tokens like any other scarce resource: measure them, cap them and avoid spending them twice.

Here are the levers I reach for, in the order I reach for them.

Measure cost per feature first

You can't control what you can't see, and the provider's monthly invoice is far too coarse. It tells you the total. It says nothing about which feature, which prompt or which customer drove it.

So the first step is logging usage on every model call, tagged with the context your business cares about:

• Feature name. Ticket classification, chat, document generation.
• Tenant or user ID. So you can spot the one account generating most of the spend.
• Input and output tokens, from the usage data the model API returns.
• Model tier and whether the response came from cache.
• Latency and retry count, because retries are spend too.

Write it to a table or your log pipeline and build one query that groups spend by feature per day. That one view usually shows exactly where the money goes. I cover the wider logging setup, including how to handle personal data in prompts, in logging and observability for LLM features.

Put budgets on every call

Most cost blowups come from unbounded inputs or outputs. A user pastes a whole book into a text box. A conversation grows forever because every message resends the full history. A generation runs long because nothing told it to stop.

Budgets I set by default:

• Max output tokens on every request, sized to what the feature actually needs. A classification needs a handful of tokens; it should never be allowed thousands.
• Input limits enforced before the call. Validate length at the API boundary and reject or truncate with a clear message.
• Context trimming for chat: keep the recent turns verbatim, summarize older ones and drop what no longer matters.
• Per-user and per-tenant quotas, enforced in your backend. A daily token allowance per account stops one heavy user, or one buggy client stuck in a loop, from running up the bill for everyone.

Quotas also make pricing conversations easier for SaaS products. If you know the cost per action, you can put it in a plan.

Cache what repeats

Plenty of LLM calls repeat exactly: the same product description summarized for every visitor, the same FAQ answered, the same document classified twice because a job retried. Every repeat is money spent on an answer you already have.

On Buffdudes I put leaderboards behind Redis because they were read far more often than they changed, which I wrote about in scaling a fitness platform to 60K users. The same reasoning applies to model responses. Here is a minimal exact-match cache with ioredis:

import { createHash } from "node:crypto";
import Redis from "ioredis";
import { callModel } from "./llm"; // your wrapper around the model API

const redis = new Redis(process.env.REDIS_URL!);

// Bump this when the prompt template changes, so old answers stop matching.
const PROMPT_VERSION = "v3";

export async function cachedCompletion(
  modelTier: string,
  prompt: string,
  ttlSeconds = 60 * 60 * 24
): Promise<string> {
  const key =
    "llm:" +
    createHash("sha256")
      .update(`${PROMPT_VERSION}\n${modelTier}\n${prompt}`)
      .digest("hex");

  const hit = await redis.get(key);
  if (hit !== null) return hit;

  const output = await callModel(modelTier, prompt);
  await redis.set(key, output, "EX", ttlSeconds);
  return output;
}

Three things make this safe. The key includes the prompt version and model tier, so changing either never serves a stale answer. The TTL bounds how long an answer can live. And it only goes on deterministic features, where the same input should give the same output. Anything personalized needs the user or tenant in the key, or no cache at all. Semantic caching, matching similar prompts instead of identical ones, can save more and is also much easier to get wrong; I discuss the caveats in caching LLM responses with Redis.

Use the smallest model that passes

Model tiering is one of the largest savings available. Many tasks inside an AI feature are simple: classify intent, extract a few fields, check whether a reply is on topic. A smaller, cheaper model handles those well. Save the stronger, more expensive model for the steps that actually need reasoning or long, careful writing.

My approach:

1. Build a small test set of real inputs with known good outputs for each task.
2. Run it against the cheaper tier first.
3. Move up a tier only when the cheaper one fails the test set.

Chained pipelines make this easy, because each step can use a different tier. Routing can also be dynamic: try the small model, validate the output, and escalate to the bigger one only when validation fails.

Sometimes the cheapest model is no model. If a step can be done with a regex, a lookup table or a SQL query, do that.

Batch and defer what isn't interactive

Work nobody is waiting for should leave the request path. Nightly summaries, bulk classification, embedding a backlog of documents: these belong in background jobs, where you control concurrency and can batch several items into one call when the task allows it. Some providers also offer discounted asynchronous batch processing, which is worth checking for large offline jobs.

Background jobs also make retries cheaper to reason about. A request that times out and gets retried by the user, the browser and your own code can turn one answer into three bills. A queued job with one retry policy pays once.

The short version

Log tokens per feature and per tenant from day one. Cap every call's input and output, enforce quotas per account, cache deterministic answers in Redis with versioned keys and default to the smallest model that passes your tests. Move everything non-interactive into background jobs.