Caching LLM responses with Redis

Caching LLM responses with Redis is one of the simplest ways to make an AI feature faster and cheaper. Model calls are slow (often seconds) and billed per token, and a surprising share of requests in real products are repeats: the same question about the same document, the same summary of the same record. If the inputs have not changed, there is no reason to pay for the same answer twice. This post covers how I'd build an LLM cache in a Node.js backend, and where caching goes wrong.

The same lesson as any expensive read

On Buffdudes, the fitness platform I built the backend for, leaderboards were the expensive read: heavy aggregation across the whole user base, read constantly, barely changing second to second. Putting them behind Redis was one of the most effective fixes in scaling the platform to 60K users.

LLM calls fit the same profile, only more so. They are slower than any database query, they cost money on every call, and for many features the output for a given input does not need to change. Anything that is expensive, read often and changes rarely belongs in a cache.

Good candidates:

• Summaries of documents or records that have not been edited.
• Classification or tagging of a fixed piece of text.
• Embeddings of content and of popular search queries.
• Answers to FAQ-style questions over a knowledge base that changes on a schedule.

Poor candidates are open-ended chat turns with long, unique histories, and anything personalised where the cache key would include so much context that it never repeats.

Building an exact-match cache key

The cache key has to capture everything that affects the output, and nothing else. For an LLM call that usually means:

• The model identifier.
• A version for your prompt template (bump it whenever you edit the prompt).
• The system prompt and user input after normalisation.
• Generation parameters such as temperature and maximum output length.
• For RAG, the ids and versions of the retrieved chunks.

Hash all of that into a fixed-length key. Here is the core with ioredis and Node's built-in crypto. callModel is a thin wrapper around your provider's API.

import Redis from "ioredis";
import { createHash } from "node:crypto";

const redis = new Redis(process.env.REDIS_URL ?? "redis://localhost:6379");

type ModelRequest = {
  model: string;
  promptVersion: string;
  system: string;
  input: string;
  temperature: number;
};

declare function callModel(req: ModelRequest): Promise<{ text: string }>;

function cacheKey(req: ModelRequest): string {
  const normalised = [
    req.model,
    req.promptVersion,
    req.system.trim(),
    req.input.trim().replace(/\s+/g, " "),
    req.temperature,
  ];
  const hash = createHash("sha256").update(JSON.stringify(normalised)).digest("hex");
  return `llm:${req.promptVersion}:${hash}`;
}

export async function cachedCall(req: ModelRequest, ttlSeconds = 86_400) {
  const key = cacheKey(req);
  const hit = await redis.get(key);
  if (hit) return { ...JSON.parse(hit), cached: true };

  const result = await callModel(req);
  await redis.set(key, JSON.stringify(result), "EX", ttlSeconds);
  return { ...result, cached: false };
}

Building the key from an array (instead of stringifying an object) keeps the order fixed, so two logically identical requests always hash the same. Putting the prompt version in the key prefix also makes cleanup easy: after a prompt change, old entries simply stop being read and expire on their own.

TTLs and invalidation

Every cached LLM response needs an expiry. My defaults:

• Derived from immutable input (a summary of a specific document version): long TTL, days or weeks. If the document changes, its version changes, and so does the key.
• Derived from data that changes on a schedule (answers over a knowledge base re-ingested nightly): TTL a little shorter than the refresh interval, or explicit deletion after ingestion.
• Anything time-sensitive (stock, prices, status): short TTL, minutes, or no cache at all.

The cleanest invalidation is the one you never have to do. Put a content version or updated_at timestamp into the key, and edits naturally produce cache misses. Explicit deletes are for the cases where you cannot version the input.

Two more production details:

• Set a memory policy. Configure Redis with a maxmemory limit and an eviction policy such as allkeys-lru if it is a dedicated cache, so a burst of unique requests cannot exhaust memory.
• Coalesce concurrent misses. If 50 users open the same new document at once, you don't want 50 identical model calls. Keep an in-process map of pending promises per key, or take a short Redis lock with SET key value NX EX 30, so only one request calls the model and the rest wait for its result.

Semantic caching: proceed with care

A semantic cache matches requests by meaning instead of exact text: embed the incoming question, find a cached question within some similarity threshold, and return its answer. It raises hit rates for differently worded versions of the same question. If you have read embeddings explained for web developers, the mechanics will be familiar.

The risk is serving the right answer to the wrong question. "How do I cancel my order?" and "How do I cancel my subscription?" can sit very close in embedding space and need different answers. Thresholds that catch useful paraphrases will eventually catch a near miss like that, and the user gets a confident, wrong reply.

My rules if you do build one:

• Use it only for low-stakes, FAQ-like questions with a small set of well-understood answers.
• Scope it tightly: same tenant, same feature, same prompt version.
• Log every semantic hit with both questions, and review them regularly.
• Start with exact-match caching and measure the hit rate first. Often it is good enough.

Measuring whether it works

Track hit rate per feature, latency for hits against misses, and the model spend you avoided. Return a cached flag (as in the code above) and log it with your request traces. If a cache has a very low hit rate, it is costing you Redis memory and a network round trip for nothing, and you can remove it. Caching is one lever among several for keeping spend predictable; the others are covered in keeping LLM API costs under control in Node.js.

Where I'd start

Add an exact-match Redis cache in front of your most repeated, least personalised LLM call, with the model, prompt version, normalised input and parameters in a hashed key. Version inputs so edits invalidate themselves, set TTLs by how fast the source data changes, and coalesce concurrent misses. Leave semantic caching until you have hit-rate numbers that justify the risk.