Switching Models Without a Rewrite

Sooner or later you'll want to change model — something cheaper, something better, or the one you're using gets deprecated. Whether that's a config change or a week of work is decided long before you try it, by whether model calls go through one place in your code or are scattered through it. The version worth building is small: one module that owns the prompt and returns your typed result, the model name in an environment variable, and your eval set as the gate. Then switching is a restart you can undo in thirty seconds — and you'll find out whether the cheaper model is actually cheaper, which is not the same question as its price per token.

Build the seam first#

One module. Everything else calls it and knows nothing about which provider you use:

# llm.py — the only file that knows about the provider
def extract_task(text: str) -> Task:
    ...  # prompt, call, validate, return a typed object

Callers get a Task. They don't see prompts, don't see raw responses, and don't handle provider-specific errors. That boundary is what makes a swap tractable, and it's the same boundary that makes validating output natural — parse into a real type at the edge, so nothing downstream deals with model quirks.

The prompt belongs in this module too. A prompt scattered across three call sites is three prompts to re-tune when you switch.

Make the model a variable, not a constant#

LLM_MODEL=<provider>/<model-name>

Set it as an environment variable rather than hardcoding it. Two things follow.

Switching is a restart, not a deploy. You change a value and the app comes back on the new model — no build, no push, nothing to review at midnight.

Rollback is the same action in reverse. If the new model is worse, you're one change away from the old behaviour, which is the property you actually want when you're uncertain.

Bear in mind the change recreates the container, so in-flight requests are lost. Switch when it's quiet, and remember any in-memory state resets with it.

What differs between models — it isn't just price#

The mistake is treating models as interchangeable except for cost. The things that actually break:

• Prompt sensitivity. Instructions tuned for one model can produce different structure or verbosity on another. Your careful phrasing is partly a fit to a specific model.
• Structured output. Support and strictness vary, so a schema honoured by one may need more validation and more retries elsewhere.
• Tool calling. Formats and reliability differ; a model that reliably picks the right tool is not a given.
• Context window. A smaller window can silently truncate conversation history you were relying on.
• Latency profile. Time-to-first-token matters enormously if you stream, and not at all if you don't.
• Refusals. Different boundaries mean requests that worked may now be declined — worth checking if your app handles anything sensitive.

Gate it on your evals, not on vibes#

This is where a small eval set pays for itself. Run the same cases against both models and compare:

• Pass rate, on the same fixed inputs. This is the number that matters.
• Retries needed. A model that fails validation more often costs more calls than its price suggests.
• Tokens used. A more verbose model can erase a per-token discount entirely.
• Latency, if users wait on it.

The cheaper model is only cheaper if the total comes out lower. Price per token, times tokens actually used, times attempts actually needed. I've seen that arithmetic reverse the obvious answer often enough to be worth doing rather than assuming.

If you don't have evals yet and you're considering a switch, building the set is the first step of the switch. Twenty cases is enough.

Roll it out like a deploy, because it is one#

• Try it in a staging app first, with the same code and a different LLM_MODEL. That's the cleanest A/B you'll get on a single container.
• Watch the first hour. Log which model produced each response, so a spike in validation failures is attributable rather than mysterious.
• Keep the old value written down. Not in your head — in the repository, next to the constraints you keep for future maintenance.
• Give it a week before deleting the fallback path.

Embeddings are a different problem#

Worth separating, because it's the one switch that isn't cheap.

Vectors from different embedding models aren't comparable. Switching means re-embedding your entire corpus, which is a real cost event proportional to how much you've indexed — and until it's done, your retrieval is meaningless.

So store the model name alongside every vector, and treat an embedding-model change as a migration with a plan rather than an environment variable you flip. More on that.

FAQ#

How do I make switching models easy later?#

Put every model call behind one module that returns typed results, and read the model name from an environment variable.

Is switching a deploy?#

If the model name is an environment variable, it's a restart — the container is recreated but nothing is rebuilt. That also makes rollback immediate.

Will my prompts work on a different model?#

Partly. Prompts are tuned to a model more than people expect, so expect to adjust structure and verbosity instructions.

How do I know the new model is better?#

Run the same eval cases against both and compare pass rate, retries, tokens and latency. One-off comparisons tell you almost nothing.

Is a cheaper model actually cheaper?#

Only if total spend drops. Multiply price per token by tokens actually used and attempts actually needed — verbosity and retries often cancel the discount.

Can I switch embedding models the same way?#

No. Vectors from different models aren't comparable, so you must re-embed everything. Store the model name with each vector so you know what needs redoing.

Wrapping up#

Model choice feels like a decision and behaves like a dependency — it changes underneath you, and you'll want to move for reasons that have nothing to do with your code. Everything here is about making that move boring: one module that owns the prompt and the parsing, one variable that names the model, one eval run that says whether it's better, and a rollback that takes a restart. Build the seam while you only have one model, because that's the only time it's free.