Deploying AI-powered Node.js apps on AWS

My default stack for deploying Node.js apps on AWS is GitHub Actions deploying to Elastic Beanstalk, with RDS for Postgres, Nginx in front of the app, and Redis alongside. It is the setup behind the Buffdudes backend, and I walked through the base pipeline in deploying Node.js to Elastic Beanstalk with CI/CD.

Adding AI features to an app on that stack changes a few things. Requests get slower and less predictable, a new and expensive secret appears, and some responses now stream for 30 seconds or more. The same architecture handles all of it once the settings in about five places are right.

Secrets: the model API key is the expensive one

A leaked database password is bad. A leaked model API key can run up a bill in hours. Treat it with at least the same care:

• Keep it out of the repo, the Docker image, and the client. Mobile and web clients call your backend, and only the backend calls the model API.
• Store it in AWS Secrets Manager or SSM Parameter Store and load it at startup, or set it as an Elastic Beanstalk environment property if your team is small and access to the console is tight.
• Use separate keys per environment, so a staging leak cannot touch production spend.
• Set a usage limit or alert on the provider side, if your provider offers one.

Deploy credentials: switch to OIDC

The other secret worth removing is the AWS key your CI pipeline uses to deploy. A long-lived access key stored in GitHub secrets is a standing risk: it works from anywhere until someone rotates it.

On the Buffdudes backend I moved deploys to GitHub OIDC instead of a stored AWS key. GitHub Actions requests a short-lived token, AWS checks it against an IAM role whose trust policy only allows this repository and branch, and the job gets temporary credentials for that run. There is no AWS key in GitHub at all. The setup is covered step by step in deploying to AWS from GitHub Actions with OIDC.

Timeouts: the part that actually breaks

A typical CRUD request finishes in milliseconds. An LLM call can take anywhere from one second to well over a minute, depending on output length. Every layer between the user and your code has its own timeout, and the shortest one wins.

On Elastic Beanstalk with an Application Load Balancer, the layers are:

1. The ALB idle timeout. It defaults to 60 seconds. If no bytes flow for that long, the load balancer closes the connection and the client sees a 504. You can raise it in the Elastic Beanstalk environment's load balancer settings.
2. Nginx proxy timeouts. proxy_read_timeout also defaults to 60 seconds. A non-streaming call that takes 70 seconds fails here even if the ALB allows it.
3. Node's own server timeouts. Node's HTTP server has requestTimeout, headersTimeout and keepAliveTimeout. A known gotcha: if keepAliveTimeout is shorter than the ALB idle timeout, the ALB can reuse a connection Node already closed, and you get intermittent 502s. Set keepAliveTimeout a few seconds above the ALB idle timeout.
4. Your outbound call to the model API. Always set an explicit timeout, for example with AbortSignal.timeout() on fetch, so a stuck upstream does not hold a connection open forever.

On Amazon Linux 2 platforms you can extend the Nginx config by adding files under .platform/nginx/conf.d/ in your app bundle. A minimal timeout bump looks like this:

# Raise only as far as your slowest legitimate request needs.
proxy_read_timeout 120s;
proxy_send_timeout 120s;

Raise these deliberately. A 10-minute timeout everywhere just means stuck requests hold workers for 10 minutes.

Streaming through Nginx and the ALB

Streaming is usually the better answer for user-facing AI features: the user sees output within a second or two, and because bytes keep flowing, idle timeouts stop being a problem.

Nginx buffers proxied responses by default, which breaks streaming: the user waits for the whole response, then gets it all at once. For Server-Sent Events, the simplest fix is to send an X-Accel-Buffering: no header from Node on the streaming route. Nginx reads it and disables buffering for that response only, leaving the rest of the app unchanged. If a stream might sit silent for long stretches, send a comment line every 15 to 20 seconds as a heartbeat to keep the ALB from closing it.

Long jobs belong off the request path

Some AI work is too slow for any request: processing a large document, generating a batch of images, running a multi-step pipeline. Raising timeouts to cover these is a trap.

The pattern I'd use: accept the request, write a job record, return a job id immediately, and process it in a background worker. The client polls or subscribes for the result. On Elastic Beanstalk that can be a worker environment or a separate process consuming a queue, with Redis or Postgres holding job state. I go through the options in running slow AI jobs in the background with queues.

This is the same lesson as moving heavy cron jobs to background services on Buffdudes: anything slow and unpredictable gets its own process with controlled concurrency, so it cannot starve the API.

Smaller things that matter

• Health checks. Keep the Elastic Beanstalk health check endpoint cheap. If it calls the model API, a provider outage will mark every instance unhealthy and trigger replacements.
• Rate limits at the edge. Nginx limit_req in front of AI endpoints protects both your bill and the provider's rate limits.
• Logging. Log latency, token usage and outcome for every model call so slow or failing requests are visible.
• Instance sizing. Most AI apps that call a hosted model are I/O bound, so they rarely need larger instances. Scale on request count or latency.

The short version

Keep the model API key in Secrets Manager or Parameter Store and deploy with GitHub OIDC so no AWS key lives in CI. Line up the ALB, Nginx and Node timeouts, with Node's keepAliveTimeout above the ALB idle timeout. Stream user-facing responses with Nginx buffering off, and move anything slower than a request into a background job.