# Shipping Is Half the Job > AI can write the code, but running it is a separate skill: environments, secrets, CI, staged rollouts, alerts, cost limits and a launch checklist. - Author: Tai Pham Phu - Published: 22 Oct 2026 - Tags: Vibe Coding, Software Design, Career, DevOps - URL: https://phamphutai.com/2026/10/22/shipping-is-half-the-job/ At 04:09 UTC on Friday, 19 July 2024, CrowdStrike pushed a small content update to its Falcon security software on Windows machines. It wasn't a new version of the program. It was a configuration file, the kind of thing the company shipped routinely to help detect new attacks. This one had a mismatch buried in it. A template expected 21 input fields, and the code that fed it supplied only 20. When Windows machines loaded the new file, the sensor tried to read the 21st value, read past the end of its memory, and crashed the whole operating system. CrowdStrike reverted the update at 05:27 UTC, 78 minutes later. By then, Microsoft estimated, 8.5 million Windows devices had been hit. Businesses around the world stopped. Microsoft published manual fix-up steps and sent hundreds of engineers to help customers bring machines back, because many couldn't recover on their own. Delta Air Lines cancelled more than 5,000 flights. Its CEO said the airline had to manually reset 40,000 servers, and that the outage cost it as much as $500 million. The bug itself was the kind that happens in every codebase. What turned it into a global event was **how it was shipped**: to everyone, at once. CrowdStrike's own root cause analysis says it plainly. Finding number six: "Each Template Instance should be deployed in a staged rollout." The fix they committed to was canary testing, then wider and wider "deployment rings," with time to watch the results before each step, and the ability for customers to choose when updates land. That's this whole post in one sentence. **Writing the code is half the job. Getting it to users safely, and knowing when it breaks, is the other half.** This is part 8 of *After Vibe Coding*. --- ## Why AI-built apps skip this part AI tools are very good at the first half. Describe a feature, get working code, click deploy. Some platforms make shipping a single button. The second half is mostly invisible in that flow: - **The AI writes code, not operations.** It doesn't know you have no staging environment, that your logs go nowhere, or that the API key in that file is your live one. - **"It works" means "it works on my machine, right now."** Production has real data, real traffic, slow networks, expired tokens and people who click twice. - **Mistakes in operations are expensive and fast.** A bad deploy, a leaked key or a runaway loop can hurt you in minutes. In [Built With AI in a Weekend](https://phamphutai.com/2026/10/22/built-with-ai-in-a-weekend-what-happened-next/) I wrote about an AI agent that deleted a live database. Most of these stories come back to the same gap: no separation between "where we try things" and "where real users live." None of this is hard. It's just a list of habits nobody puts in a prompt. Let's go through them. --- ## Environments: dev, staging, prod An **environment** is a complete copy of your app with its own settings, database and keys. - **Dev:** your laptop or a personal cloud sandbox. Fake data. Break anything. - **Staging:** a copy of production that only you and your team use. Same code, same kind of infrastructure, separate database, test payment keys. - **Production (prod):** real users, real money, real data. The rule that matters most: **nothing in dev or staging can touch prod data or prod keys.** Separate databases, separate API keys, separate accounts where you can. If your AI coding agent has credentials, it gets dev credentials, never prod ones. For a tiny team, staging can be cheap. Many hosts give you preview deployments per branch. Pair that with a small staging database seeded with realistic fake data, and you have somewhere to click through a change before users do. --- ## Config and secrets **Config** is anything that differs between environments: database URLs, API keys, feature switches. **Secrets** are the config values that would hurt if they leaked. The basics: - Config comes from environment variables, not from code. - `.env` files are in `.gitignore`. Commit a `.env.example` with fake values instead. - Production secrets live in your host's secret settings or a secret manager, not in a chat, a doc or a screenshot. - Anything with a `NEXT_PUBLIC_` (or similar) prefix is shipped to the browser. Never put a secret there. - If a secret leaks, rotate it. Deleting the commit isn't enough; it's already been seen. AI tools love to hardcode keys "for now." Catch it in review. A second line of defence is to **validate config when the app starts**, so a missing or wrong value fails loudly at deploy time instead of quietly at 3 a.m.: ```ts // src/config.ts import { z } from "zod"; const Env = z.object({ APP_ENV: z.enum(["dev", "staging", "prod"]), DATABASE_URL: z.string().url(), STRIPE_SECRET_KEY: z.string().startsWith("sk_"), SENTRY_DSN: z.string().url().optional(), }); const parsed = Env.safeParse(process.env); if (!parsed.success) { console.error("Invalid environment:", parsed.error.flatten().fieldErrors); process.exit(1); } export const config = parsed.data; // Cheap guard against the most expensive mix-up. if (config.APP_ENV !== "prod" && config.STRIPE_SECRET_KEY.startsWith("sk_live_")) { throw new Error("Live Stripe key outside production"); } ``` Every other file imports `config`, never `process.env` directly. Now the AI has one obvious place to add new settings, and you have one place to review them. --- ## CI/CD basics **CI (continuous integration)** means every change is checked automatically: it builds, the types check, the linter passes, the tests pass. **CD (continuous delivery)** means a change that passes is deployed by a script, not by hand. You don't need much. A GitHub Actions workflow like this covers most small apps: ```yaml # .github/workflows/ci.yml name: ci on: pull_request: push: branches: [main] jobs: check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 cache: npm - run: npm ci - run: npm run typecheck - run: npm run lint - run: npm test - run: npm run build deploy-staging: needs: check if: github.event_name == 'push' && github.ref == 'refs/heads/main' runs-on: ubuntu-latest environment: staging steps: - uses: actions/checkout@v4 - run: ./scripts/deploy.sh staging env: DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }} ``` Add a `deploy-production` job that needs `deploy-staging` and uses a GitHub environment called `production` with a required reviewer. Now production deploys need one human click, after staging is green. Two things matter more than the tool: - **Every deploy goes through the pipeline.** No "quick fix" pushed from a laptop. The day you skip it is the day it breaks. - **You can roll back in one step.** Know the command or button that puts the previous version back, and try it once on a quiet day. The tests that run in CI are the ones from [part 6](https://phamphutai.com/2026/10/22/tests-are-how-you-talk-to-the-ai/). The lint rules that guard your architecture are from [part 7](https://phamphutai.com/2026/10/22/from-prompting-to-architecture/). CI is where they stop being good intentions. --- ## Staged rollouts and feature flags CrowdStrike's lesson applies at every size: **don't send a change to everyone at once.**
STAGED ROLLOUT staging fake data internal your team 5% 25% 100% watch watch watch Between steps, watch errors, latency, support. Not healthy? Flag off or roll back. Few users saw it. ALL AT ONCE 0% 100% one step, no time to look, everyone finds the bug together
A staged rollout gives each step time to show a problem while only a few users are exposed; an all-at-once release doesn't.
For a small web app there are two easy ways to do this. **Staged deploys.** Some hosts can send a percentage of traffic to a new version. If yours can, use it for risky changes. **Feature flags.** Ship the code switched off, then turn it on for yourself, then a few users, then everyone. Twitter's 2013 engineering post (the one from [part 7](https://phamphutai.com/2026/10/22/from-prompting-to-architecture/)) describes exactly this: deploy code "in the fully 'off' and safe setting," then gradually turn it up. You can use a flag service, or start with a tiny one of your own: ```ts // src/flags.ts import { createHash } from "node:crypto"; type Flag = { enabled: boolean; percent: number; allow?: string[] }; export function isOn(name: string, flag: Flag, userId: string): boolean { if (!flag.enabled) return false; // kill switch if (flag.allow?.includes(userId)) return true; // you and your team first const hash = createHash("sha256").update(`${name}:${userId}`).digest(); return hash.readUInt32BE(0) % 100 < flag.percent; // same user, same answer } ``` Store the flag values somewhere you can change without a deploy (a database row or your flag service), and you have a kill switch: if the new checkout breaks, turn it off in seconds. Flags have a cost. Pete Hodgson's well-known guide on Martin Fowler's site calls them "inventory which comes with a carrying cost." Each one is an `if` that doubles the paths through your code. **Remove a release flag within a week or two of reaching 100%.** Put the removal on your list the day you add the flag. --- ## Logs, metrics and alerts that matter When something breaks, you need to answer two questions fast: *what* is broken, and *why*. ### Logs Logs answer "why." Make them **structured** (JSON, not free text) and **connected** (every line from one request shares a request ID): ```ts // src/log.ts import pino from "pino"; export const log = pino({ level: process.env.LOG_LEVEL ?? "info", redact: ["req.headers.authorization", "*.password", "*.token"], }); // in a request handler const reqLog = log.child({ requestId, userId }); reqLog.info({ bookingId, totalCents }, "booking created"); reqLog.error({ err, bookingId }, "payment failed"); ``` Note the `redact` list. AI-written logging often dumps whole request bodies, passwords and tokens included. Logs get copied, shared and kept for months. Keep secrets and personal data out of them. ### Metrics and alerts Metrics answer "what." Google's SRE book names four "golden signals" worth watching for any user-facing system: **latency, traffic, errors and saturation** (how full your busiest resource is). For a small app that means: how slow are requests, how many are there, how many fail, and is the database or queue close to its limit. The same chapter has the two best rules I know about alerts: - **Alert on symptoms, not causes.** "Checkout error rate above 2% for 5 minutes" is a symptom users feel. "CPU at 80%" might be nothing. - **"Every page should be actionable."** If an alert wakes you up and there's nothing to do, delete it or make it a daily summary. Noisy alerts train you to ignore the real one. For a tiny team, five alerts are plenty: the site is down, the error rate spiked, a core flow (sign-up, checkout) stopped succeeding, a background queue is backing up (see [Background Jobs Done Right](https://phamphutai.com/2026/10/14/background-jobs-done-right/) for queue health), and spending is unusual. --- ## Error tracking An error tracker (Sentry, Rollbar, Bugsnag and others) catches exceptions in your server and browser code, groups duplicates and tells you which release introduced them. Set it up on day one. It's usually a few lines. Then: - **Tag every event with the release** (the git commit), so you can see "this started with deploy X." - **Upload source maps** so browser stack traces point at your real code, not minified gibberish. - **Check it after every deploy**, for ten minutes. That's your personal canary. - **Paste the full trace into your AI tool** when debugging. A real stack trace with context beats a description every time, as in [part 3](https://phamphutai.com/2026/10/22/debugging-when-the-ai-is-out-of-ideas/). --- ## Cost alarms and spending limits A surprise bill can kill a small product faster than an outage. AI products add a new risk: every LLM call costs money, and a bug or an abuser can make thousands of calls an hour. Know how your provider's budgets actually work, because many don't stop anything: - **AWS Budgets** sends alerts on actual or forecast spend, and can run "actions," such as applying a policy that blocks new resources. But AWS notes its budget data updates up to three times a day, and that there can be a delay between incurring a charge and getting the notification. You can blow past your number before the email arrives. - **Google Cloud** says an alerts-only budget "doesn't automatically cap" usage or spending. It also offers spend cap budgets for eligible services, which pause those services until you lift the cap. Check whether yours qualify. So use layers: 1. **Budget alerts at 50%, 80% and 100%** of what you expect to spend, going to a channel you read. 2. **Provider spend limits** on LLM and API accounts where they exist. Use separate keys per environment, so a staging bug can't drain the production budget. 3. **Your own limits in code.** Rate-limit expensive endpoints per user, and keep a hard daily cap on your own AI spend: ```ts // Before each LLM call: reserve the estimated cost, refuse past the daily cap. const DAILY_LIMIT_CENTS = 2_000; export async function reserveAiSpend(db: Db, estimateCents: number) { const day = new Date().toISOString().slice(0, 10); const { rows } = await db.query( `INSERT INTO ai_spend (day, cents) VALUES ($1, $2) ON CONFLICT (day) DO UPDATE SET cents = ai_spend.cents + EXCLUDED.cents RETURNING cents`, [day, estimateCents], ); if (rows[0].cents > DAILY_LIMIT_CENTS) { throw new Error("Daily AI budget reached"); } } ``` The upsert is atomic, so two requests at once can't both sneak under the limit. When the cap is hit, the app degrades gracefully and alerts you, instead of quietly spending all night. [Treat the LLM Like an Unreliable API](https://phamphutai.com/2026/10/14/treat-the-llm-like-an-unreliable-api/) has more on timeouts and retries around these calls. --- ## Backups Backups get their own section in [part 5, Data Outlives Code](https://phamphutai.com/2026/10/22/data-outlives-code/), so just the short version here: automatic daily backups (plus point-in-time recovery if your database offers it), stored away from the main account, and **a restore you have actually tested**. A backup you have never restored is a hope, not a backup. --- ## On-call for a team of one to five "On-call" sounds like a big-company thing. It just means: when the app breaks, who finds out, and what do they do? - **One person owns alerts each week.** Rotate. If you're solo, that's you, so make the alerts few and meaningful. - **Write a one-page runbook.** How to check the site, how to roll back, how to turn off a flag, how to restore a backup, who to contact at your host. Write it calm, read it panicked. - **Have a way to tell users.** A simple status page or a pinned message. Silence makes outages feel worse than they are. - **After any real incident, write a short, blameless review.** What happened, how you found out, what you'll change. CrowdStrike's root cause analysis is a big-company version of the same thing. Yours can be ten lines. - **Protect sleep.** If you got paged at night for something that could wait until morning, fix the alert the next day. --- ## A launch checklist Before real users touch it: - [ ] Dev, staging and prod are separate, with separate databases and keys. - [ ] Config is validated at startup. No secrets in code, in the repo or in browser bundles. - [ ] Every deploy goes through CI: typecheck, lint, tests, build. - [ ] You know how to roll back, and you've done it once. - [ ] Risky features ship behind a flag, with a kill switch. - [ ] Logs are structured, carry a request ID and leave out secrets. - [ ] An error tracker is installed and tagged with releases. - [ ] There are a handful of symptom-based alerts, and each one is actionable. - [ ] Budget alerts, provider spend limits and your own per-user and daily caps are set. - [ ] Backups are automatic, and a restore has been tested. - [ ] There's a one-page runbook and someone owns alerts this week. --- ## How to build this skill while using AI AI tools are good at this work when you ask for it explicitly, because it's well-documented and repetitive. The skill is knowing what to ask for and checking the result. - **Ask for operations as a feature.** "Write a GitHub Actions workflow for this repo that runs typecheck, lint, tests and build on every PR." Then read every line and make sure you know what each step does. - **Ask the AI to review a deploy, not just code.** "Here's my config, workflow and env example. What could leak, what could break in production, and what isn't monitored?" - **Practise failure on purpose.** On staging, break the database URL, kill the queue worker, throw an error in checkout. Did an alert fire? Did the error tracker catch it? Could you roll back? - **Read real postmortems.** CrowdStrike's root cause analysis is twelve pages and very readable. Many companies publish theirs. Each one teaches a failure you don't have to experience yourself. - **Keep your own AI agent away from prod.** Give it dev credentials only. That one habit prevents the scariest stories. --- ## Key ideas - **The CrowdStrike outage was a rollout failure as much as a bug.** The fix they committed to was staged deployment with time to watch between steps. - **Separate dev, staging and prod**, with separate data and keys. AI agents get dev access only. - **Validate config at startup**, keep secrets out of code and browser bundles, and rotate anything that leaks. - **Every deploy goes through CI, and rollback is one step.** - **Ship risky changes behind flags**, roll out gradually, and remove flags once they're at 100%. - **Alert on symptoms users feel**, make every alert actionable, and install an error tracker on day one. - **Budget alerts often don't stop spending.** Add provider limits and your own caps in code. *Sources: CrowdStrike, ["External Technical Root Cause Analysis: Channel File 291"](https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf) (6 August 2024) and ["Technical Details: Falcon Content Update for Windows Hosts"](https://www.crowdstrike.com/en-us/blog/falcon-update-for-windows-hosts-technical-details/) (July 2024); David Weston, Microsoft, ["Helping our customers through the CrowdStrike outage"](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/) (20 July 2024); NPR, ["Delta's CEO says the CrowdStrike outage cost the airline $500 million in 5 days"](https://www.npr.org/2024/07/31/nx-s1-5058652/delta-crowdstrike-outage-500-million-dollars) (31 July 2024); Raffi Krikorian, Twitter Engineering, ["New Tweets per second record, and how!"](https://blog.x.com/engineering/en_us/a/2013/new-tweets-per-second-record-and-how) (16 August 2013); Pete Hodgson, ["Feature Toggles (aka Feature Flags)"](https://martinfowler.com/articles/feature-toggles.html) (2016, revised 2017); Rob Ewaschuk, Google SRE book, ["Monitoring Distributed Systems"](https://sre.google/sre-book/monitoring-distributed-systems/); AWS, ["Managing your costs with AWS Budgets"](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html) and Google Cloud, ["Create, edit, or delete budgets and budget alerts"](https://docs.cloud.google.com/billing/docs/how-to/budgets) (both checked October 2026).* *Related: [From Prompting to Architecture](https://phamphutai.com/2026/10/22/from-prompting-to-architecture/), [The Junior Developer's Roadmap for 2027](https://phamphutai.com/2026/10/22/junior-developer-roadmap-2027/), and [Background Jobs Done Right](https://phamphutai.com/2026/10/14/background-jobs-done-right/).*