Retrieval-augmented generation (RAG) in Node.js

Retrieval-augmented generation (RAG) is the pattern behind most useful "chat with your docs" features: before you ask the model a question, you look up the passages that are most likely to contain the answer and hand them over as context. Building RAG in Node.js is very approachable if you already run a Node.js and PostgreSQL backend, because you can do the whole thing with tools you already know. This guide walks through the pipeline the way I'd build it for a client product.

What RAG actually solves

A language model only knows what was in its training data plus whatever you put in the prompt. Your company handbook, your product docs and last week's support tickets are in neither. You have two options: fine-tune a model on that content, or fetch the relevant pieces at question time and include them in the prompt.

For almost every product I'd start with retrieval. It is cheaper, it updates the moment your data changes, and it lets you show the user exactly which sources the answer came from. Citations are the feature that turns "the AI said so" into something a user can check.

The pipeline has two halves:

• Ingestion (offline): load documents, split them into chunks, turn each chunk into an embedding vector, store the vector next to the text.
• Query (online): embed the user's question, find the nearest chunks, build a prompt with those chunks, ask the model to answer only from them.

Ingestion: load, chunk, embed, store

Start with the boring part, because it decides your quality ceiling. Pull the raw text out of your sources (Markdown, HTML, PDFs, database rows) and keep metadata with every piece: the source URL or file name, the section heading, a last-updated timestamp and, in a multi-tenant app, the tenant id.

Then chunk. A chunk is the unit you retrieve, so it should be small enough to be specific and large enough to make sense on its own. A few hundred words with a bit of overlap is a sensible first setting. I go deeper on the trade-offs in chunking strategies for RAG, and it is worth reading before you tune anything else.

Each chunk then goes through your provider's embedding endpoint, which returns an array of floats. Store the text, the metadata and the vector in one row. My default store is PostgreSQL with the pgvector extension, because the data already lives there and you get transactions, joins and tenant filtering for free. The schema and indexing details are in vector search in PostgreSQL with pgvector.

A few ingestion habits that save pain later:

• Hash each chunk's text and skip re-embedding when the hash has not changed.
• Store which embedding model produced each vector, so you can re-embed cleanly when you switch.
• Run ingestion as a background job with retries. Embedding thousands of chunks inline in a request handler will time out.

Query time: retrieve and prompt with sources

At query time you embed the question with the same model you used for the chunks, run a nearest-neighbour search, and assemble a prompt. Here is the core of it with pg and a thin fetch wrapper. The endpoint paths and response shapes are placeholders; swap in whatever your LLM provider exposes.

import { Pool } from "pg";

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function callModel(path: string, body: unknown) {
  const res = await fetch(`${process.env.LLM_API_URL}${path}`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.LLM_API_KEY}`,
    },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(30_000),
  });
  if (!res.ok) throw new Error(`Model API returned ${res.status}`);
  return res.json();
}

export async function answer(question: string, tenantId: string) {
  const { embedding } = await callModel("/embeddings", { input: question });

  const { rows } = await pool.query(
    `SELECT source, content
       FROM chunks
      WHERE tenant_id = $2
      ORDER BY embedding <=> $1::vector
      LIMIT 5`,
    [JSON.stringify(embedding), tenantId]
  );

  const context = rows
    .map((r, i) => `[${i + 1}] (${r.source})\n${r.content}`)
    .join("\n\n");

  const { text } = await callModel("/generate", {
    system:
      "Answer using only the numbered sources. Cite them like [1]. " +
      "If the sources do not contain the answer, say you don't know.",
    prompt: `Sources:\n${context}\n\nQuestion: ${question}`,
  });

  return { text, sources: rows.map((r) => r.source) };
}

Three details in there matter more than they look:

• The tenant filter lives in SQL. Never retrieve across tenants and filter afterwards. A leaked chunk in a prompt is a data leak.
• Sources are numbered. The model can cite [2], and your UI can map that back to a link.
• The model is allowed to say "I don't know". Without that instruction it will happily fill gaps from its own training data, which is exactly what you were trying to avoid.

Where RAG quality really comes from

When a RAG feature gives bad answers, the generation step is rarely the culprit. The usual causes sit upstream:

1. The right chunk was never retrieved. Log the retrieved chunk ids for every question. If the answer was not in the top five, no prompt tweak will fix it.
2. Chunks are cut in the wrong place. A table split in half or a heading separated from its paragraph retrieves badly.
3. Pure vector search misses exact terms. Product codes, error messages and names often need keyword matching. That is what hybrid search is for.
4. Stale data. If ingestion runs weekly and docs change daily, users get last week's answer with full confidence.

So the order I debug in is: retrieval first, chunking second, prompt last. Build a small set of real questions with known source documents, and check that the right chunk shows up in the results before you look at the generated text at all.

Production concerns

A few things separate a demo from something you can put in front of customers:

• Streaming. Answers take seconds. Stream tokens to the browser so the user sees progress.
• Caching. Identical questions against unchanged data can reuse the previous answer and its embedding.
• Access control. Store permissions with the chunk (tenant, team, role) and filter in the query. Retrieval is a read path and deserves the same authorization as any other read path.
• Prompt injection. Retrieved documents are untrusted input. If a user can upload a document, they can put instructions in it. Treat chunk text as data in your prompt and keep the model's permissions narrow.
• Cost. Embeddings are cheap per call but add up on re-ingestion. Hashing chunks and only re-embedding changes keeps this in check.

Where I'd start

Put pgvector in the Postgres you already run, ingest one well-structured source, and get the retrieval step returning the right chunks for twenty real questions before touching the prompt. Once retrieval is solid, add citations, streaming and access control, in that order. If you want help scoping a RAG feature for your product, that is the kind of build I do through my services.