Backup · Recovery · Practice
A Backup Is Just an Opinion Until It Restores
Choi Areum · 10 March 2026 · 5 min read
The phrase that changed our cohort culture was a half-joke from a participant in 2024: "a backup is just an opinion until it restores." We adopted it. Every recovery program we run now starts with a verification drill, not a theory chapter. The drill is small — pick one production-like dataset and restore it into a clean instance from scratch. Time it. Write down what surprised you.
Operators sometimes resist this exercise because it can feel like an audit. The fix is partly social: position the drill as a rehearsal that benefits the operator, not as evidence for management. Nobody wants to discover a gap in their own backup chain in front of an executive. Most do want to find it on a Wednesday afternoon, with a mentor on the call, in time to fix it before it matters.
Two patterns recur. First, many teams have automated backups but no archived test of restore time, which means they cannot answer the question their leadership is most likely to ask after an incident: "how long would it have taken?" Second, the documentation about how to restore is almost always older than the most recent change to the backup pipeline. The verification drill resolves both at once.
We close every cohort with a written verification report — short, honest, useful at handover. If you take only one thing from this post, take that template into your next operations meeting. Quiet, repeatable verification is what calm restores look like.
Continue reading
-
21 Apr 2026
Reading Autovacuum Like a Quiet Letter
Most operators learn autovacuum as a list of parameters. Here is the small mental model we use in our cohorts to read it as a conversation between the planner and the storage layer.
-
17 Feb 2026
gh-ost in a Mid-Volume Estate
Notes from a recent corporate cohort on introducing gh-ost-style online schema changes into a team that had been doing them by hand for years.