Image processing pipelines with Sharp and S3

Almost every product I build ends up handling images: avatars, product photos, documents, AI-generated outputs. An image processing pipeline with Sharp and S3 is my default for all of them. Sharp is fast, runs in Node.js with no external services, and covers resizing, format conversion, rotation and compositing. S3 gives cheap, durable storage with signed URLs for anything private.

The clearest example from my own work is Fotobank, a stock-photo marketplace built with Next.js, Express and TypeORM, with Stripe for payments, S3 for storage and server-side image processing with Sharp. Photos are sold in tiers (Standard, Premium and Exclusive), and the pipeline only re-encodes photos that need it. That last rule is the one I'd carry to any media product.

Why re-encode only when needed

Every lossy re-encode throws away a little quality. For a stock-photo site, the original is the product a buyer pays for, so degrading it on upload is the worst possible default. For an AI app, the uploaded image is the model's input, and compression artifacts become part of what the model sees.

So the pipeline inspects every upload first and only rewrites the original when there is a concrete reason:

• The format is one you do not want to serve or store long term.
• The colour space is something browsers render badly, such as CMYK.
• The EXIF orientation flag says the pixels are stored rotated.
• The file is far beyond the maximum dimensions you support.

Everything else is stored byte-for-byte as uploaded. Derived files (previews, thumbnails) are always generated, because those are meant to be small.

The processing step

Here is the shape of it with Sharp. metadata() reads the header cheaply, and the decision happens before any pixel work:

import sharp from "sharp";

const SERVABLE = new Set(["jpeg", "png", "webp"]);
const MAX_EDGE = 12_000;

export async function processUpload(input: Buffer) {
  const meta = await sharp(input).metadata();
  if (!meta.width || !meta.height) throw new Error("Not a readable image");

  const needsReencode =
    !SERVABLE.has(meta.format ?? "") ||
    meta.space === "cmyk" ||
    (meta.orientation ?? 1) !== 1 ||
    Math.max(meta.width, meta.height) > MAX_EDGE;

  const original = needsReencode
    ? await sharp(input)
        .rotate()
        .resize({ width: MAX_EDGE, height: MAX_EDGE, fit: "inside", withoutEnlargement: true })
        .toColourspace("srgb")
        .jpeg({ quality: 95 })
        .toBuffer()
    : input;

  const preview = await sharp(input)
    .rotate()
    .resize({ width: 1600, withoutEnlargement: true })
    .webp({ quality: 80 })
    .toBuffer();

  const thumb = await sharp(input)
    .rotate()
    .resize(400, 400, { fit: "cover" })
    .webp({ quality: 70 })
    .toBuffer();

  return { original, preview, thumb, reencoded: needsReencode };
}

A few details worth calling out:

• rotate() with no arguments applies the EXIF orientation and resets it, so phone photos stop showing up sideways.
• Sharp drops metadata from its output by default. Previews and thumbnails carry no EXIF, which means no GPS coordinates leaking from a user's phone. An untouched original keeps whatever the uploader's file contained, so decide whether that is what you want. For user-generated content I usually strip it; for a photo marketplace, photographers may want their copyright fields kept.
• withoutEnlargement stops small images from being blown up into blurry large ones.

For a marketplace, the preview is also where a watermark goes, using Sharp's composite() to lay a logo or pattern over the image before encoding.

Laying out S3

I keep the bucket structure boring and predictable:

• originals/{assetId} in a private bucket or prefix. Never publicly readable.
• previews/{assetId}.webp and thumbs/{assetId}.webp, served through a CDN.
• One asset ID per upload, generated on the server. Never trust the client's filename as a key.

Originals are delivered through short-lived signed URLs generated after an authorization check. In a marketplace, that check is "has this user paid for this tier of this photo", which ties the download to the payment record. In an AI app it is "does this user own this asset". Either way, a leaked link expires on its own.

Uploads from browsers or mobile apps go straight to S3 with a presigned upload URL, so large files never pass through your API servers. The API then gets a small "upload complete" call with the key, and processing starts from there.

Run processing in the background

Image work is CPU-heavy. Resizing a large photo in an Express request handler blocks other requests on that process and makes upload latency depend on image size. I'd treat processing as a job: the upload completes, a job row is created, a worker downloads the original, runs Sharp, uploads the derived files and marks the asset ready. The UI shows "processing" until then. The job pattern is in running slow AI jobs in the background with queues.

Sharp uses a native library and its own thread pool, so the worker process deserves its own memory and concurrency settings. Processing a handful of images at once is usually the sweet spot; dozens in parallel can exhaust memory on a small instance.

The same pipeline for AI apps

AI features need this exact plumbing. Before an image goes to a model, you want it oriented correctly, in a predictable format and size, and free of location metadata. After a generation model returns an image, you want a stored original, a preview and a thumbnail, all private by default. The pipeline above serves both directions without changes.

Protecting the library

Two last things for media-heavy products. First, Fotobank's robots.txt opts out of AI training crawlers, which is a cheap first step for any site whose images are the product; I wrote that up in blocking AI training crawlers with robots.txt. Second, keep originals out of anything publicly listable, including sitemaps and open bucket policies.

The short version

Inspect before you touch, re-encode only when there is a concrete reason, always generate small derived files, and keep originals private behind signed URLs. Run the work in a background worker. If you are deploying this kind of Node.js service on AWS, my Elastic Beanstalk CI/CD post covers the deployment side.