Logging and observability for LLM features

LLM features fail in ways normal monitoring does not catch. The endpoint returns 200, the response time looks fine, and the output is wrong, empty, truncated, or quietly more expensive than last week. Without good logging and observability for LLM features, the first signal is a user complaint, and by then you cannot reproduce what happened.

This is the logging setup I'd put in place before an LLM feature reaches real users.

Log every call, as a record

Every model call should produce one structured record. Treat it like a row in a table you will query later, because you will. At minimum:

• Trace id that links this call to the user request that caused it.
• Feature name and prompt version, so you can compare before and after a prompt change.
• Model identifier, so you can compare before and after a model change.
• Input and output token counts.
• Latency, ideally both time to first token and total time if you stream.
• Outcome: success, validation failure, timeout, provider error, refused.
• Retry count.

Prompt text and output go in too, with the care described below.

Postgres is my default home for these records. You can answer real questions with SQL: which feature costs the most, which prompt version fails validation most often, what the p95 latency was yesterday.

A thin wrapper does most of the work

The easiest way to get consistent logs is to route every model call through one function. Here is a minimal version using fetch and pg:

import { randomUUID } from "node:crypto";
import { Pool } from "pg";

const db = new Pool({ connectionString: process.env.DATABASE_URL });

type CallOptions = {
  feature: string;
  promptVersion: string;
  traceId?: string;
  input: string;
};

export async function callModel(opts: CallOptions) {
  const traceId = opts.traceId ?? randomUUID();
  const started = Date.now();
  let outcome = "success";
  let output = "";
  let usage = { input_tokens: 0, output_tokens: 0 };

  try {
    const res = await fetch(process.env.LLM_API_URL!, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify({ input: opts.input }),
      signal: AbortSignal.timeout(60_000),
    });
    if (!res.ok) throw new Error(`provider_${res.status}`);
    const data = await res.json();
    output = data.output;
    usage = data.usage ?? usage;
    return { traceId, output };
  } catch (err) {
    outcome = err instanceof Error ? err.message : "unknown_error";
    throw err;
  } finally {
    await db.query(
      `INSERT INTO llm_calls
         (trace_id, feature, prompt_version, outcome, latency_ms,
          input_tokens, output_tokens, input_redacted, output_redacted)
       VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
      [traceId, opts.feature, opts.promptVersion, outcome, Date.now() - started,
       usage.input_tokens, usage.output_tokens, redact(opts.input), redact(output)]
    );
  }
}

function redact(text: string) {
  return text
    .replace(/[\w.+-]+@[\w-]+\.[\w.]+/g, "[email]")
    .replace(/\+?\d[\d\s-]{7,}\d/g, "[phone]");
}

The request and response shapes are placeholders for whatever your provider returns. The parts that matter: one trace id per request, a timeout on every call, and a log row written in finally so failures are recorded too. In production I'd write the log asynchronously or through a queue so a slow insert never delays the user.

Prompts and outputs: handle PII on purpose

Prompts and outputs are the most useful thing to log and the most dangerous. They contain whatever users typed, plus whatever your retrieval step pulled in: names, emails, health details, contract text.

My defaults:

• Redact before storing. Strip obvious identifiers (emails, phone numbers, card-like numbers) at write time. The regex above is a starting point; real redaction depends on your data.
• Separate the sensitive columns. Keep metrics (tokens, latency, outcome) in one table with long retention and full-text prompts in another with short retention and tighter access.
• Set retention. Delete or anonymise full prompt text after a fixed window. Metrics can live much longer.
• Respect tenant boundaries. On multi-tenant systems, logs need the same tenant scoping as the data they came from.
• Write it down. Your privacy policy and client contracts should match what you actually log.

Trace ids across multi-step flows

Many LLM features are several calls in a row: retrieve, draft, validate, maybe retry. A user sees one action. You see five log rows. A shared trace id, created when the request arrives and passed to every step, lets you reconstruct the whole chain with one query.

This matters most for agents and multi-step prompt chains, where a bad final answer usually traces back to a bad intermediate step.

Runs, fetches and alerts

The same discipline applies to batch work. For a scraper I run nightly, every run and every individual fetch is logged to run tables, and email alerts go out when a run is blocked or mostly fails. When something breaks, the alert says so and the rows that explain it are already in the database.

For LLM features, the equivalent alerts I'd set up:

• Validation failure rate above a threshold for a feature.
• Error or timeout rate spiking for the provider.
• Daily token usage above budget, per feature and per tenant.
• p95 latency above the feature's budget.

Alert on rates over a window. Single failures are noise; a jump from 1% to 15% is a signal.

What to look at weekly

Dashboards are only useful if someone reads them. A short weekly review covers most of it:

• Cost per feature and per active user, trending up or down. Keeping LLM API costs under control goes into the levers.
• The top validation failures, read by a person.
• A random sample of outputs, read by a person.
• Any user-reported issues, traced back to their log rows.

Reading real outputs regularly is the cheapest quality check available. It also feeds your evaluation set; logged failures make excellent test cases, as covered in how to evaluate an LLM feature before you ship it.

The short version

Route every model call through one wrapper that logs a structured row: trace id, feature, prompt version, tokens, latency and outcome. Redact and limit retention on prompt text. Log batch runs and their individual fetches the same way, and alert on failure rates, cost and latency over a window. Then read a sample of real outputs every week.