Running a Python AI microservice alongside a Node.js backend

Most of my backends are Node.js and TypeScript. Most serious machine learning code is Python. Sooner or later a product needs both, and the question becomes how to run a Python AI microservice alongside a Node.js backend without turning the system into two half-owned codebases that blame each other.

FaceBloom is a good example of the shape. It is a social app with an AI face-scoring service: a Node.js/TypeScript backend, a dashboard, a Flutter mobile app, and a Python AI microservice. This post is the pattern I'd use for any product with that split, written as a guide.

Why not do it all in one language

You can call models from Node.js, and for plain LLM API calls that is usually what I do. The split earns its place when the AI side needs the Python ecosystem: image libraries, model runtimes, numeric code, or a researcher's notebook that has to become a service.

The trade is real, though. A second language means a second runtime, a second dependency tree, a second deploy, and a network hop on every call. So my rule is simple: keep the Python service as small and as stateless as possible. It does inference. Everything else stays in Node.js.

Draw the boundary around one job

The cleanest split I know gives each side a job it is good at.

The Node.js backend owns the product:

• Auth, users, sessions and permissions
• The database and every write to it
• Uploads, storage and file URLs
• Rate limits, billing and business rules
• The public API the mobile app and dashboard talk to

The Python service owns inference:

• Takes an input (an image reference, some text, a feature vector)
• Runs the model
• Returns a structured result with a version string

The Python service never sees a user session, never writes to the main database, and is never reachable from the public internet. Clients talk to Node.js. Node.js talks to Python. That one rule removes a whole category of security and consistency bugs, because there is exactly one place where "who is allowed to do this" gets decided.

Define the contract first

Two services in two languages drift apart quietly. Someone renames a field in Python, the Node.js side keeps compiling, and the bug shows up in production as undefined. I treat the request and response as a contract, write it down (an OpenAPI file or a JSON Schema works well), and validate it on the Node.js side every single time.

import { z } from "zod";

const ScoreResult = z.object({
  modelVersion: z.string(),
  score: z.number().min(0).max(1),
  labels: z.array(z.string()),
});

export async function scoreImage(imageUrl: string) {
  const res = await fetch(`${process.env.AI_SERVICE_URL}/score`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ imageUrl }),
    signal: AbortSignal.timeout(15_000),
  });
  if (!res.ok) throw new Error(`AI service returned ${res.status}`);
  return ScoreResult.parse(await res.json());
}

Three details in there matter more than they look:

• The timeout. Inference can hang. Without AbortSignal.timeout, one stuck request holds a Node.js connection open far longer than any user will wait.
• The parse. ScoreResult.parse turns a silent contract break into a loud, specific error at the boundary.
• modelVersion in the response. Store it next to the result. When you retrain or swap a model, you will want to know which version produced which score.

Pass file references between services

For image or audio work, the tempting move is to POST the raw file from Node.js to Python. I'd avoid it. Upload once to object storage, then send the Python service a short-lived URL or a storage key. The payloads stay tiny, retries become cheap, and both services can be scaled without caring about request body limits. I cover the storage side in more detail in my post on image pipelines with Sharp and S3.

Sync call or background job

If inference reliably finishes in a second or two, a direct HTTP call from the request handler is fine. Once it takes longer, or once traffic gets spiky, I move it behind a queue: Node.js records a pending job, a worker calls the Python service, and the result lands in the database. The client polls or gets a push. The full pattern is in running slow AI jobs in the background with queues.

The useful habit is designing the Python endpoint so it works either way. Make it idempotent on a job ID, so a retried request does not produce two different answers for the same input.

Running it in production

The operational side is where two-service setups usually hurt. A short checklist I'd hold any setup to:

• Health checks on both services. The Python service should expose a cheap health endpoint that also confirms the model is loaded, because a process that is up but has no model is the worst kind of up.
• Load the model at startup. Loading weights on the first request makes that request time out and makes autoscaling look broken.
• Private networking. The AI service lives on an internal address with no public route. A shared secret header on top is cheap insurance.
• One request ID across both. Generate it in Node.js, pass it as a header, log it in Python. Debugging a slow request across two services without it is guesswork.
• Separate deploys, pinned versions. Each service gets its own Docker image and CI pipeline. The Node.js side should tolerate the AI service being briefly unavailable during a deploy and show the user a clear error while it is down.
• Size the Python workers deliberately. Model inference is memory-heavy. Fewer, larger workers usually behave better than many small ones that each load their own copy of the model.

Where I'd start

Start with one inference endpoint, a written contract, a timeout and a validated response, and keep every product decision in Node.js. Add a queue when latency or traffic asks for it, and not before. FaceBloom and the rest of my work are on the projects page.