There is a particular kind of project that looks finished and is not. The demo works. The client signs off. Everyone shakes hands, the invoice clears, and the team that built the thing moves on to the next one. For about a year, nothing goes wrong. Then a certificate expires, or a dependency the world has moved past finally breaks, or someone needs to add a field to a form and nobody knows where the form lives. That is the moment the client discovers what they actually bought. Not a system. Access to a system, for as long as the people who built it are willing to answer the phone.
We build software for a living, so this is an awkward thing to write down. The most profitable version of our business is the one where every platform we ship can only be maintained by us. But we have inherited enough of those platforms, built by other people, to know exactly what they feel like from the inside, and it is not a business we want to be in.
A platform decays on a delay
The reason abandonment is dangerous is that the failure is not immediate. If a system fell over the day the builders left, nobody would tolerate the arrangement. Instead it degrades quietly. Dependencies drift out of support. Small operational knowledge (the fact that this cron job must run before that one, the reason nobody touches the settings table on a Friday) lives in the heads of people who are no longer in the room. The system keeps running because it was running when they left, and running systems have inertia. Then something forces a change, and the change touches a part of the machine that no living person understands, and now it is a crisis, and the crisis has a year of accumulated neglect stacked behind it.
So when we talk about handover, we are not talking about a folder of PDFs delivered on the last day. We are talking about whether the system can survive us leaving. There are two tests we use for that, and they are both slightly humiliating to run.
The runbook test
Can a new engineer, someone who has never seen the codebase, deploy it to production on their second day, using only what is written down? Not with a call to us. Not with a message to the person who set up the pipeline. Just the documentation and the repository.
Most handover docs fail this instantly, and they fail in a very specific way: they describe the system as it was designed, not as it must be operated. They tell you the architecture. They do not tell you that the deploy will hang the first time because an environment variable is missing from the secrets store and the error message will not mention it. The gap between what a system is supposed to do and the exact sequence of commands that make it do the thing is where all the real operational knowledge lives, and it is precisely the part that never gets written down, because the people writing it already know it and cannot see that they know it.
The only fix we have found is to make the documentation load-bearing. A README you must actually read in order to run the thing does not rot, because the moment it is wrong, someone hits the wrong instruction and fixes it. Docs that sit off to the side, describing the system for the benefit of some imagined future reader, rot within weeks and nobody notices until that reader arrives. So we push as much as we can into the path of doing the work. The runbook is not a document about deployment. It is the thing you follow to deploy, every time, including us.
The restore test
The second test is shorter to state. An untested backup is not a backup. It is a belief that you have a backup.
We have never once seen a first restore go cleanly. Not once. Something is always wrong, and it is always something stupid: the backup captured the data but not the schema migrations, or the restore user does not have permission to recreate the extensions, or the dump is technically complete but takes so long that the recovery-time promise in the contract turns out to be fiction. You find these things only by doing the restore. Reading the backup script tells you nothing. The script runs fine. It is the restore that fails.
So we do the restore. On a fresh environment, from the backup, with a stopwatch, and we write down what broke and how long it took. And then, because the first one always exposes something, we fix that thing and do it again. It is dull work and it is the single highest-leverage hour in the whole engagement, because it converts a belief into a fact.
Walking into someone else's undocumented system
A fair amount of our support work starts with a system we did not build and nobody documented. The engineers who wrote it are gone. What exists is the code, a production database, and a set of assumptions the client holds that may or may not be true.
The first two weeks of that are not glamorous. It is mostly reading code and writing down what it does. Not what it should do, not what the client believes it does, not what a sensible person would have built. What it actually does, including the parts that are clearly bugs that some downstream process now quietly depends on. We got burned on this once. Early in a takeover we found a scheduled job that looked broken, an obvious off-by-one in how it batched records, and one of us tidied it up. It was not broken. A reconciliation report three steps downstream had been silently compensating for that exact behaviour for two years, and correcting the job broke the report, and the report was the thing the finance team actually looked at. We spent a bad afternoon on that. The lesson stuck: when you inherit a system, you document reality first and pass judgement later. The map comes before the roadworks.
Post-incident reviews, and why we bother writing them down
When something goes wrong in a system we run, we write it up afterwards. Not for the client to wave at their board, though it is useful for that too. We write it because an incident is the only time a system tells you the truth about itself, and if you do not capture that while it is fresh, you have paid the cost of the outage and thrown away the only thing you got in return.
A post-incident review that is worth the hour it takes should contain:
- A timeline in plain wall-clock time: when it started, when we noticed, when a human began working it, when it was resolved. The gap between started and noticed is usually the embarrassing one, and it is the one worth closing.
- What the users actually experienced, in their words, not in metrics.
- The technical cause, followed honestly down to the thing underneath the thing. "The disk filled up" is not a cause. Why did it fill, and why did nothing warn us before it did.
- What we changed so this specific failure cannot recur silently, with an owner and a date, not a good intention.
- And no name attached to the mistake. The question is what let a person make that error, not who made it. You get honest reviews only when they are safe to write.
A minimal runbook, if you have nothing
If you are inheriting a system tomorrow and want somewhere to start, this is roughly the skeleton we write into every handover. It is not clever. It is just the set of questions that hurt to answer under pressure, answered in advance.
- How do I deploy a change to production, as a numbered list of commands, tested by someone who did not write it.
- How do I roll that deploy back when it goes wrong at 11pm.
- Where do the logs live and what does a healthy system look like there, so you can tell normal from broken.
- What are the scheduled jobs, in what order do they matter, and what happens if one does not run.
- How do I restore the database from a backup, with the last date this was actually done written next to it.
- Who do I wake up, and what is the one thing I must not do while half-asleep.
The commercial part
There is an uncomfortable truth underneath all of this. Making ourselves replaceable is, on paper, a bad business move. We spend real hours writing the runbook that means the client could, in principle, fire us and carry on. We hand over the keys along with the instructions for changing the locks.
But the clients we have built those handovers for are the ones who come back. Not because they are trapped. Because when the next thing needs building, the vendor who left them a system they could actually run is the obvious call, and the vendor who left them a hostage situation is the one they are quietly trying to escape. Lock-in wins you the renewal you were going to get anyway and loses you every one after it. We would rather be asked back than kept.