One Smoke Test Beats a Test Suite You'll Never Write

Ask an AI tool to build something and you get features. Ask it for tests and you usually get unit tests of pure functions — the parts least likely to break a deployment. Meanwhile the failures that actually take an app down are boring and structural: a missing import, an unset environment variable, a server bound to the wrong interface, a migration that won't apply. You don't need a test suite for those. You need one smoke test that starts your app the way the container will and checks it responds — and because a deploy here replaces a working container with no overlap, catching those before the push is the difference between a non-event and an outage.

What the smoke test is actually for#

Not correctness. Structure. It should fail on exactly the things that make a deploy worse than no deploy:

Every one of those has produced a "deployed successfully, doesn't work" report. None require knowing what your app is supposed to do.

Test the image, not the code#

The important detail: run the test against a built container, not against python app.py on the CI runner. A test that imports your module in the CI environment doesn't check the thing that goes wrong — your image.

- run: docker build -t smoke .
- run: |
    docker run -d --name smoke -p 8000:8000 \
      -e DATABASE_URL=sqlite:///tmp/test.db \
      -e API_KEY=dummy \
      smoke
    sleep 5
    curl -fsS --max-time 10 http://localhost:8000/health
    docker logs smoke
    docker rm -f smoke

That's the whole thing. curl -f fails the step on a non-2xx, the sleep gives a crash time to happen, and printing the logs means a failure arrives with its own diagnosis attached.

Two details worth keeping. Map the port from outside, so you're testing the same path the platform's router uses — a loopback-bound server fails here exactly as it would in production. And pass placeholder environment variables, because "works when configured" is what you're checking, not "works with your real keys".

For bots and workers, assert survival#

Most bots have no port, so there's nothing to curl. The equivalent check is that the process is still alive after a few seconds and said so:

docker run -d --name smoke -e DISCORD_TOKEN=invalid smoke
sleep 10
docker logs smoke | grep -q "app.start" || { echo "no startup line"; exit 1; }
[ "$(docker inspect -f '{{.State.Running}}' smoke)" = "true" ] || exit 1

This is where a startup log line pays for itself — it's the only evidence a port-less app got past configuration. With a deliberately invalid token you're testing that the app boots and reaches the point of trying to connect, which is the structural part.

If your bot crash-loops on a zip deploy the container stays up, so check for the repeating exit line too rather than trusting State.Running alone.

Wire it in before the deploy#

The ordering that makes this useful:

1. Smoke test on every push to your default branch.
2. Push to build to confirm the platform builds it too — that doesn't touch your running app.
3. Push to deploy only after both pass.

In GitHub Actions that's a needs: dependency between jobs, with the final step being an ordinary git push deploy. Remember fetch-depth: 0 on checkout, since a shallow clone can't be pushed.

The result is that your live container is only ever replaced by an image that has already started successfully somewhere else.

Ask your agent for this specifically#

"Write tests" produces the wrong thing. Ask for the smoke test in the terms above:

Write a CI job that builds the Docker image, runs the container with placeholder environment variables, waits a few seconds, and fails if the app isn't responding on its port. Print the container logs on failure. Don't write unit tests.

Then have it add a /health route if there isn't one — a handler returning 200 with no dependencies. Note that this is for your test, not for the platform: nothing here probes your app, so the route exists purely so your pipeline has something to ask.

If you want one more check, make the health route touch the database. Then "the app answers" and "the app can reach its data" are the same assertion.

Keep it under a minute#

The reason to stop at one test is that you'll actually keep it. A smoke test that runs in forty seconds gets run; a suite that takes fifteen minutes and needs maintenance gets skipped with --no-verify the first time you're in a hurry.

Resist the urge to grow it into a suite. When a real bug reaches production, add a specific test for that bug — that's how a useful suite accumulates, if it ever does.

FAQ#

What should I test before deploying an AI-built app?#

That the built image starts with production-like environment variables and answers a request. Structure, not behaviour.

Why test the container instead of the code?#

The failures that break deploys — missing dependencies, wrong bind address, unset variables — live in the image and its configuration, not in your functions.

My app has no HTTP port. What do I check?#

That the process is still running after several seconds and logged its startup line. A deliberately invalid credential is fine; you're testing boot, not auth.

Do I need a /health endpoint?#

Only for your own test. Nothing on the platform probes your app, so the route exists for your pipeline.

Where does this fit with the build branch?#

Smoke test in CI first, then push to build to confirm the platform builds it, then deploy.

Should I ask my AI tool to write tests?#

Ask for this specific job. A generic request produces unit tests of functions that rarely cause deployment failures.

Wrapping up#

The gap between prototype and something you can redeploy without flinching is smaller than "add testing" makes it sound. One job that builds the image, starts it with placeholder configuration, and asks it a question catches nearly everything that turns a deploy into an incident — and it does so before the old container has been thrown away. Write that one, keep it fast, and let the suite grow only when a real bug earns it a test.