Prompt chaining vs one giant prompt: when to split LLM calls

Every LLM feature eventually hits the same design question: do you ask for everything in one big prompt, or break the work into a sequence of smaller calls? Prompt chaining, where the output of one call becomes the input to the next, is often the right answer. It also adds latency, cost and new ways to fail. Here is how I decide between prompt chaining and a single prompt.

What a single prompt does well

A single prompt is the default for good reasons. It is one network round trip, one place to read when something goes wrong and one prompt to version. The model sees all the context at once, so it can make trade-offs across the whole task without anything getting lost in a handoff.

For tasks where the input is small and the output is a single coherent thing, I keep it to one call: summarize this ticket, classify this message, rewrite this paragraph in a friendlier tone. Splitting those adds moving parts and no quality.

The problems start when one prompt is quietly doing several different jobs.

Signs one prompt is doing too much

These are the symptoms I look for:

• The prompt has sections that contradict each other. "Be thorough" for one output and "be brief" for another, in the same instruction block.
• Outputs disagree with each other. You asked for three related artifacts and they use different names for the same thing.
• Fixing one part breaks another. Every prompt edit is a game of whack-a-mole across the whole output.
• You can't tell which step failed. The final answer is wrong, and there is no intermediate result to inspect.
• Some of the work needs different input. Part of the task needs the raw source material, and another part would be better off without it.

Two or more of those and I start drawing boxes.

The case for chaining: REQ Agent

REQ Agent, a CLI I built, turns raw client discovery material into client, designer and dev handoff documents. It uses two sequential LLM calls. The first reads the raw input and produces an internal Spec-Driven Development document plus a designer brief. The second takes only those outputs and writes the final client, design and dev documents.

I split it because the two jobs want different inputs. Understanding messy notes requires the notes. Writing three consistent documents works better from a clean, already-decided spec, without the tangents and abandoned ideas that live in raw discovery material. One prompt would have to do both at once, which is exactly the setup where related outputs drift apart. I wrote up the full design in the two-stage LLM pipeline post.

The split bought three things beyond output quality:

1. A checkpoint. The intermediate spec is a real artifact I can inspect, validate and, if needed, edit by hand before the second stage runs.
2. Validation against a source of truth. Because the spec exists separately, the final documents can be checked against it for feature coverage and naming consistency.
3. Smaller prompts. Each prompt describes one job, so it is shorter and easier to change without side effects.

The costs of chaining

Chaining is a trade, and the bill is real.

Latency adds up. Two sequential calls take roughly twice as long as one. For a CLI that generates documents, that is fine. For a chat reply a user is staring at, it may not be.

Errors compound. If stage one misreads the input, stage two faithfully builds on the mistake. Every link in the chain needs its own validation, or you are just moving the failure further from its cause.

Context gets lost in handoffs. Anything stage one doesn't pass forward is gone. That is sometimes the point (dropping noise), and sometimes a bug (dropping a constraint the client mentioned once).

Cost can go either way. More calls usually means more tokens. A chain can also be cheaper, when early steps like classification or extraction run on a smaller, cheaper model and only the final step needs a stronger one. I go into that kind of model tiering in keeping LLM API costs under control.

Rules I use to decide

My default is one prompt until a specific reason to split shows up. The reasons that justify a split:

• A step needs different input than the steps around it.
• You need a checkpoint a program or a person can validate before continuing.
• A step can run on a cheaper model or be replaced with plain code.
• Steps can run in parallel. Three independent extractions from the same document can be three concurrent calls, which costs about the same wall-clock time as one.
• The output is too long for one reliable response. Generating section by section from an outline is more reliable than one enormous answer.

"It feels more organized" is a weak reason, and so is a vague hope that more steps means more intelligence. Each split should map to a job you can name.

Keep the chain in code

A chain of prompts is still a program, and the control flow belongs in your code. I write each step as a function with typed input and output, validate between steps and keep the order fixed. When the model needs to decide which step comes next, you have moved from a chain into an agent, and that is a separate decision with its own risks. I compare the two in AI agent or deterministic workflow.

Store every intermediate output alongside the final result. When something goes wrong in production, the first question is always "which step?", and stored intermediates answer it in seconds.

The short version

Start with one prompt. Split when a step needs different input, needs a checkpoint or can run cheaper on its own. Validate between every link, keep the sequence in code and store the intermediate results. Two well-defined stages beat five vague ones.