Free chapter: Outages I caused, and what each one taught me

Every outage in this chapter was preventable, and none of them felt that way while it was happening. A one-person product doesn’t have a team to catch mistakes, so each one became a rule. Most of those rules are now written where my AI agent reads them every session. That’s the real subject of this chapter: turning a bad afternoon into a guardrail.

They’re in order of how much they hurt.

9.1 The “quick check” that took the site down

August 2026. Site fully down.

In the early days, the live site ran straight out of the project folder. A production server (next start) served pages from a build folder called .next.

My agent wanted to check a small change, so it started a development server (next dev) in the same folder. The development server rebuilds that .next folder continuously, in a different format from a production build. The live server suddenly found files missing or half-replaced. Pages failed with “module not found”, and the site went down.

The worst part: it had done the same thing once already that session, without harm. “Just a quick check” was the reasoning both times. The second time it broke production.

What changed:

A written rule: never run a development server in the folder production serves from. It’s in the agent’s instructions and in its memory, with the incident linked.
Worktrees for all experiments. A git worktree is a second, separate copy of the project with its own build folder. Every change now gets built and tested in one.
Later, production moved to its own machine entirely, so a slip on the development box can’t touch it at all.

Lesson: the instinct “this is too small to need the safe setup” is exactly when you need it. Write that sentence into your agent’s rules.

9.2 A build that died halfway

October 2026. Site broken until rebuilt.

The nightly deploy built the new version in place: in the same .next folder the live site was reading. A production build starts by deleting that folder, then compiles a new one.

One night the build tried to fetch a web font, the font server didn’t answer, and the build died about 20 seconds in. The live site was left serving a half-written folder: no build ID, chunks from two different builds. Visitors got errors.

An earlier, smaller version of the same problem: two builds running at once in the same folder (one of them a test sweep) corrupted it in August.

What changed:

Build out of place. Every build now happens in a separate folder. Tests and checks run there. Only when everything passes is the finished build swapped in, followed by a rolling restart.
One build at a time. The deploy refuses to start if another build is running.
A failed build changes nothing. The live site keeps serving the last good version.

Lesson: a deploy should have exactly two outcomes: the new version, or the old one untouched. Anything in between is an outage waiting for a bad night.

9.3 The reboot that took the tunnel with it

August 2026. Down for about an hour, late evening.

At the time, the site ran in a virtual machine on my home PC. Visitors reach it through a Cloudflare Tunnel: a small program on my server that dials out to Cloudflare, so I never open a port to the internet.

Windows rebooted the host PC for updates, three times in half an hour. Each time, the virtual network came back with a new random address range, and its built-in DNS (the service that turns names into addresses) often came back dead. The tunnel program couldn’t look up Cloudflare’s address, so it never connected. Cloudflare showed visitors a 530 error.

The site came back the moment I set the server’s DNS by hand.

What changed:

DNS on every server is pinned to public resolvers, permanently, not left to whatever the virtual network offers.
No configuration anywhere uses a hard-coded address for the home VMs, because those addresses change on every reboot.
A note I now check first: a 530 on a tunnelled site means the tunnel isn’t connected. Suspect the server’s DNS or network before the app.

Lesson: learn the one error code that means “my side, not theirs”. It saves the first half hour of every outage.

9.4 A network that dropped out every six minutes

September–October 2026. Weeks of short drops, a few real outages.

This was the hardest. The site mostly worked, but the tunnel logs filled with “no recent network activity”. Bursts of 10 to 30 seconds with no network, roughly every six minutes, day and night. It went from one or two a day to over a hundred a day in under a week. Most of the time some of the tunnel’s four connections survived. When all four dropped at once, the site was really offline.

What made it hard was the shape of it: delay, not loss. Pings mostly got through, eventually. That signature looks like bufferbloat, like a busy ISP, like the virtual network, like the network card’s power saving. I suspected each in turn.

Then we measured, one layer at a time:

The servers never froze: a timer loop on each ran without a gap.
Both virtual machines lost the internet and the home network in the same seconds, so it wasn’t the internet provider.
A connection between the two virtual machines on a private switch kept flowing through a burst, so it wasn’t the virtualization software.
The host PC itself lost pings to the router in the same seconds. The problem sat between the PC and the router.

The bursts started the night of a Windows update reboot. They stopped, for a while, after the router went down and came back. When production later moved to a dedicated machine plugged straight into the router, the same bursts followed it. That pointed at the router side, not my machines.

What changed:

Two tunnel connectors instead of one, so one dropping doesn’t take the site with it.
A watchdog that restarts a tunnel copy if it stays disconnected, one at a time.
A modem and router reboot, which helped. The home network was later reorganized, with the old Wi-Fi router turned into a simple access point.
My note says, in bold: don’t re-diagnose the network card, the power saving or the ISP; it’s already been ruled out, with the evidence listed.

Lesson: when a fault looks like everything, stop guessing and cut the system in half. Test one boundary at a time, and write down what each test ruled out, so you never pay for it twice.

9.5 A week of lost email nobody noticed

October 2026. Inbound email down for 8 days.

When production moved to its new machine, I changed one DNS record to point at it. Two minutes earlier, Cloudflare’s audit log shows a “system” change deleting one of the mail records for the domain. No action of mine is logged as the cause.

Outgoing email (receipts, trial emails) used a separate service and kept working. But every email to the domain bounced for eight days. I found out only because a test of school sign-up codes, sent to my own domain, never arrived.

The fix was one button in Cloudflare: “Enable Email Routing”.

What changed:

After any DNS change, a one-line check that the mail records are all there. It’s in the deploy notes and the agent’s memory.
The weekly health report now checks the mail records too.

Lesson: the outage you don’t monitor lasts longest. List what would break silently, such as inbound email, backups and scheduled jobs, and give each one a check.

9.6 The safe restart that wasn’t

October 2026. Down for about two minutes.

A security patch for Node.js came out. Updating it on the server meant moving the process manager (PM2, which keeps the app running) onto the new version. The documented command for that is pm2 update.

On my server, PM2 itself runs under systemd, the system’s service manager. During the update, the old PM2 process exits. Systemd saw it exit, decided the service had stopped, and ran its stop command, which kills all PM2 processes, including the new one. The whole app went down, and visitors saw a 502 for about two minutes until I restored the saved process list.

What changed:

The rule: on production, restart the service through systemd, never with pm2 update.
After any restart at that level, check the process list and the live site straight away.
Save the process list before every reboot.

Lesson: a command that’s safe on your laptop can be dangerous in production, because production has an extra layer watching it. Know what supervises your supervisor.

9.7 Turning incidents into guardrails for your agent

An AI agent doesn’t remember yesterday unless you make it. Mine doesn’t learn from an outage by itself. These are the places a lesson goes so it sticks:

An incident note in the repo (docs/incidents/): what happened, the timeline, the cause and the fix. Written the same day, while it’s fresh.
A rule in the agent’s instruction file (CLAUDE.md), if it’s something to never do again. One or two sentences with a link to the incident. For example: “Never run next dev in the production folder.”
A memory note, for facts about the environment that aren’t in the code. For example: “The virtual network changes its address range on every host reboot.”
A test, where possible. The deploy has a check that the build folder is complete before it swaps anything in. A test doesn’t depend on anyone remembering.
A line in the knowledge vault, so the next big change starts from what’s already known.

Two habits make this work:

Write down what you ruled out, not just what fixed it. Half the cost of the network drops was re-testing things already cleared.
Correct notes that turn out to be wrong. A stale note is worse than none, because the agent will trust it.

9.8 Checklist

Never run a development server where production serves from; use worktrees
Build out of place; swap only a finished, checked build; one build at a time
Pin DNS on servers; never hard-code addresses that can change
Know what a 530 (or your host’s equivalent) means
Run at least two tunnel connectors, restarted one at a time
When a fault looks like everything, test one boundary at a time and record what you ruled out
Check mail records after every DNS change; monitor inbound email
Know what supervises your process manager before restarting it
For every incident: a note, a rule, a memory, and a test if you can

Next chapter: free pages as the front door, and how to tell whether a change to one did anything.