Building a voice AI assistant with speech-to-text

A voice AI assistant is three systems wearing one interface: audio capture, speech-to-text, and a language model that does something useful with the transcript. Each one is manageable alone. The engineering work is in the seams between them, where latency stacks up and errors compound.

Maslow is the project behind this post. It is an AI assistant platform with speech-to-text, built as a backend, admin and company dashboards, and a browser extension. I'll describe the architecture I'd use for a product like that, kept vendor-neutral, since the speech and model providers you choose will change faster than the structure around them.

The pipeline in one picture

Every voice assistant I'd build follows the same path:

1. Capture audio on the client (browser, extension or mobile app).
2. Upload it to your backend, authenticated as the user.
3. Transcribe it with a speech-to-text service.
4. Process the transcript with an LLM: answer, summarize, extract tasks, whatever the product does.
5. Return text (and optionally synthesized speech) to the client.
6. Store what the product needs, under the retention rules you've set.

The client never talks to the speech or model APIs directly. Every step after capture runs on your backend, which keeps keys private and gives you one place to apply auth, quotas and logging.

Capturing audio

In the browser, getUserMedia plus MediaRecorder covers most needs. A few things decide whether capture feels good:

• Ask for the microphone at the moment of use. A permission prompt on page load gets denied. One that appears after the user clicks a mic button gets accepted.
• Show that you are listening. A live level meter or pulsing indicator tells the user the mic is working before they've wasted a sentence.
• Pick push-to-talk or voice activity detection deliberately. Push-to-talk is simpler and more predictable. Automatic detection feels more natural but needs tuning for noisy rooms.
• Cap the recording length. An open mic left running is a surprise bill and a privacy problem.

The transcription endpoint

The backend endpoint takes raw audio, checks the user, enforces a size limit and hands the audio to transcription. Here is a minimal Express version, with the provider call wrapped in a small function so the rest of the code does not care which service sits behind it:

import express from "express";

const app = express();

async function transcribe(audio: Buffer, mimeType: string): Promise<string> {
  const res = await fetch(process.env.STT_API_URL!, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.STT_API_KEY}`,
      "Content-Type": mimeType,
    },
    body: audio,
    signal: AbortSignal.timeout(60_000),
  });
  if (!res.ok) throw new Error(`Transcription failed: ${res.status}`);
  const data = (await res.json()) as { text: string };
  return data.text;
}

app.post(
  "/api/voice",
  express.raw({ type: "audio/*", limit: "20mb" }),
  async (req, res) => {
    // auth middleware omitted; req.user is set upstream
    const text = await transcribe(req.body, req.get("Content-Type")!);
    res.json({ text });
  }
);

In production I'd add per-user rate limits, a check that the body is actually non-empty audio, and error handling that tells the client whether to retry.

The LLM step

Once you have text, the assistant behaves like any other LLM feature, with one difference: spoken input is messier than typed input. People restart sentences, say "um", and transcription mishears names and jargon. A few habits help:

• Tell the model the input is a transcript. A line in the system prompt saying "this is transcribed speech and may contain recognition errors" makes it more forgiving of odd phrasing.
• Feed domain vocabulary. If users talk about product names, people or technical terms, many speech-to-text services accept hint phrases, and the LLM prompt can include a glossary.
• Show the transcript. Let the user see what the system heard, and edit it before acting on anything important. Most "the assistant is dumb" complaints are really transcription errors the user could have caught.

Latency is the whole product

Voice raises user expectations. A two-second pause in a chat window is fine. The same pause after you've finished speaking feels broken. Total latency is upload time plus transcription plus model time plus response delivery, so attack each one:

• Compress audio on the client. Smaller uploads finish sooner on bad connections.
• Stream the model's answer. Show text as it is generated so the user sees progress within a second. I cover the server side in streaming LLM responses with Node.js and Server-Sent Events.
• Use a smaller, cheaper model for simple intents. Routing short commands to a fast model and saving the larger one for complex requests cuts both latency and cost.
• Push long jobs to the background. Transcribing an hour-long meeting should be a job with a status, never a request that holds a connection open.

Dashboards: the part nobody demos

A voice assistant for organizations needs more than the assistant. Maslow has admin and company dashboards, and that split is the pattern I'd recommend for any B2B assistant:

• Admin dashboard: platform operators manage companies, plans, usage and support.
• Company dashboard: each customer's own admins manage their users, settings and what the assistant can do for their team.

That second layer means company-scoped data everywhere: every transcript, setting and usage record belongs to a company, and every query is filtered by it. I've written about those patterns in multi-tenant SaaS architecture.

Privacy and retention

Voice recordings are personal data, and in a workplace they can capture conversations people did not expect to be stored. Decide early whether you keep raw audio at all. Often the transcript is enough, and deleting audio after transcription removes a whole class of risk. Make retention configurable per company, log access to transcripts, and keep audio in private storage.

Where I'd start

Build the narrowest loop first: a mic button, a transcription endpoint, one LLM action and a visible transcript. Measure end-to-end latency on a real phone network before adding features. Once that loop feels fast, the dashboards, retention settings and extra capabilities are ordinary product work.