Changing a Prompt in Production Without Breaking It

The prompt is the highest-leverage text in an AI app and the least protected. Nothing type-checks it, nothing fails when it's wrong, and a bad edit doesn't throw — it produces answers that are slightly worse in a way you won't notice for days. Treat a prompt change as what it is: a code change that ships in a deploy, with no gradual rollout available, because your app is one container and there's no traffic to split between two versions. The discipline that makes it safe is small: keep prompts in the repository rather than in a database, gate every change on the same eval set, log which prompt version produced each output, and make rollback a redeploy of the previous commit.

Keep the prompt in the repo, not in a database#

The tempting design is an admin page where you can tweak the prompt without deploying. It feels agile and it costs you every safeguard you have.

A prompt in the repository gets code review, a diff, blame, and history. A prompt in a database gets none of those, and worse, it drifts out of sync with the code that parses its output — you change the instructions to return a new field, and the parser that expects the old shape is still deployed.

There's a useful line to draw, and it's the same one that makes swapping models easy:

• Values go in environment variables. The model name, a temperature, a maximum length. Things with no logic in them.
• Logic goes in the repository. The prompt, the schema, the parser. Things that only make sense together.

A prompt is logic written in English. Ship it with the code that depends on it, through git push, and the two can never disagree.

A prompt change is a deploy, with everything that implies#

Because it lives in the repo, changing it rebuilds and redeploys. That means the container is replaced, in-flight requests are dropped, and there's about a second between the stop signal and the kill.

It also means there is no canary and no percentage rollout. One container serves everyone, so the new prompt is live for one hundred percent of users the moment it starts. There's no mechanism to give it to five percent first, and building one — a random branch on every request — means you're now running two prompts you have to reason about at once.

So the checking has to happen before the deploy, not after it.

Compare versions offline, not in production#

The right place to run two prompts against each other is your own machine, against fixed inputs.

This is what a small eval set earns its keep for. Twenty representative cases, both prompts, same model, and a comparison you can look at. For a prompt change specifically, watch four things:

• Format adherence. Did it still return parseable output every time? A prompt edit is the most common cause of JSON that suddenly won't parse.
• Refusal rate. New wording can make a model more cautious about requests it used to handle.
• Length drift. A prompt that asks for more detail gets more tokens on every single response.
• Tone. Harder to measure and the thing users notice first.

If you want a live comparison, run it as a second staging app with the same code and a different branch — that's the closest thing to A/B you get here, and it costs you a small amount of resource rather than your production traffic.

Log the version, not the prompt#

The question you'll eventually ask is "when did the answers get worse", and only your logs can answer it.

Put a short identifier — a version number, or the first eight characters of a hash of the prompt text — into the structured log line for every model call. Then a change in behaviour is attributable to a specific deploy instead of a vague feeling.

Log the identifier, not the prompt itself. Logs are capped and rotate, and writing a two-thousand-token system prompt on every request will evict everything else you needed within minutes.

Prompts only ever grow, and that's a bill#

Worth watching, because it's invisible until you look.

Every prompt edit is additive in practice. A model gets something wrong, you add a sentence telling it not to. Six months later the system prompt is four times its original size, and every one of those tokens is paid for on every single request.

Two habits keep it in check:

• Read the whole prompt occasionally, and delete instructions that no longer earn their place. Some were fixing a behaviour in a model you've since replaced.
• Check the token count when you edit. Two hundred extra tokens sounds like nothing and is a permanent per-request tax — see what an AI app actually costs for how quickly the API side dominates.

The instinct to add rather than rewrite is strong, because adding is safe and rewriting means re-testing. Rewriting is still occasionally the right call.

Rolling back, and what doesn't roll back#

Since it's a deploy, rollback is a deploy — redeploy the previous commit, or on the Docker Push path switch back to the previous image.

What comes back is the prompt. What doesn't:

• Anything the new prompt already wrote. Summaries, extracted fields, classifications sitting in your database were produced by the version you just reverted, and they stay as they are.
• Anything already sent. Messages posted, emails delivered, webhooks fired.

Which is the argument for logging the version alongside stored output — you can find and reprocess the affected rows only if you know which ones they were.

Write down why, not just what#

The diff shows what changed. It never shows why, and that's the part you'll need.

Keep a short note next to the prompt — a comment, or a line in a file beside it — recording what each awkward instruction is defending against. "Explicitly says not to include a preamble because the model kept adding one" saves the next person, quite possibly you, from deleting it as noise and reintroducing a bug you already fixed.

This matters more than usual for prompts, because unlike code, a prompt gives no hint about which of its sentences are load-bearing. It's the same reasoning as recording constraints in an app you'll come back to later.

FAQ#

Where should my prompts live?#

In the repository, alongside the code that parses their output. A prompt is logic, not configuration, and a database-stored prompt can drift out of sync with your parser.

Can I put my prompt in an environment variable?#

Prefer not to. Environment variables suit values like the model name; a prompt has no history, no review and no diff when it lives there.

Can I roll a prompt out to a percentage of users?#

No. One container serves all traffic, so a deploy makes the new prompt live for everyone at once. Compare versions offline before shipping.

How do I know a prompt change is an improvement?#

Run both versions against a fixed eval set and compare format adherence, refusal rate, output length and tone. Impressions from a handful of manual tries tell you very little.

Why did my JSON parsing break after a prompt edit?#

Prompt wording and output format are tightly coupled. Change the instructions and the shape can change with them, which is why the parser belongs in the same commit.

How do I find when the answers got worse?#

Log a prompt version identifier with every model call, so a behaviour change maps to a specific deploy.

Wrapping up#

Prompts sit in an awkward place: they behave like configuration, so people manage them casually, and they behave like code, so casual management breaks things quietly. Putting them in the repository resolves most of it in one move — you get review, history, rollback, and a guarantee that the prompt and its parser ship together. Add an eval run before each change and a version identifier in your logs, and the worst case becomes a bad afternoon you can precisely undo, rather than a slow decline nobody can date.