The Plain Record

What things really cost, and why.

What Happens When a Backup Nobody Has Ever Restored From Finally Gets Tested?

Posted on by Gordon Achebein Technology4 min read

A small converted office above a detached garage at the end of a driveway with low sun on the windows
A small converted office above a detached garage at the end of a driveway with low sun on the windows

A design office of four people lost a workstation to a failed drive on a Tuesday, which should have been a two hour inconvenience because the machine had been backing up to an external disk every night for a little over two years. The green icon had been in the corner of the screen the entire time. When somebody finally connected the external disk to a working machine, the most recent complete file on it was twenty six months old, and everything after that was either missing or a zero byte placeholder that opened into nothing.

What Was Actually Happening

The backup software had been installed correctly and had been running every night as instructed. Some months in, the office had reorganized its project folders and moved active work into a new directory, and the backup job pointed at the old paths. It continued to run, continued to complete, and continued to report success, because from its perspective it was faithfully copying the folders it had been told to copy, which by then contained almost nothing.

A second failure compounded the first. The external disk itself had developed bad sectors during the second year, and the software logged read errors while still reporting the overall job as completed. Neither problem was exotic and neither required bad luck. Both are ordinary, both are common, and both were entirely invisible to anybody looking at a status icon rather than at a restored file. The two failures also had nothing to do with each other, which is worth noticing, because it means the office was not unlucky once but running an arrangement that had no way of reporting either kind of trouble.

Why Nobody Noticed for Two Years

Every signal the office had was a signal about the backup process rather than about the backup itself. The icon was green because the job finished. The log said complete because the copy operation returned successfully. Nobody had reason to doubt any of it, and there was no moment in two years when anyone had a reason to open the external disk and look, since the whole point of a backup is that it sits untouched.

This is the structural problem with backups generally, and it explains why the failure is so common in small offices. A system that is only exercised during a disaster is a system whose faults are discovered during a disaster. Guidance from the National Institute of Standards and Technology treats recovery testing as a separate control from backup itself precisely for this reason, and that separation is the whole insight: copying data and being able to get it back are two different capabilities, and only one of them is being verified by a green icon.

The Test That Would Have Caught It

The test takes about ten minutes and requires nothing technical. Pick a file that was definitely modified within the last week, restore it from the backup to a temporary location, open it, and confirm that it contains the recent work. That single exercise would have failed on the first attempt in month four and revealed the entire problem while it was still a configuration error rather than a loss, at a point when correcting it meant changing one setting in a dialog box.

Done quarterly, with a note of the date, it also catches the slower failures: the disk going bad, the cloud account whose payment method expired, the machine that stopped running the job after an operating system update. None of those announce themselves either, and all of them are caught by the same ten minutes. Writing the date down is what turns the habit into a record, and a record is what allows somebody to notice that the last successful test was fourteen months ago rather than assuming it was recent.

What Recovery Actually Cost

A specialist recovered a meaningful portion of the failed drive, at a cost roughly equal to several years of any backup service the office might have chosen. What could not be recovered was recreated, which took two people the better part of three weeks and produced work that was, by their own account, slightly worse than the original because the reasoning behind certain decisions had gone with the files. The client relationships survived because the office told everybody immediately and explained exactly what had happened, which turned out to matter more than the technical failure did. The schedule did not survive, and two projects moved by a month.

The Rule the Office Uses Now

Two copies in the house on unlike hardware and a third held well away from it is the arrangement they adopted, and it is the standard advice for a reason. The addition they made themselves is the one that matters more: a recurring calendar entry every quarter, assigned to a named person, requiring an actual restore of an actual recent file, with the date written in a shared document. Nobody trusts a status indicator in that office any more.

The green icon in the corner of the screen was never lying. It was answering a question nobody should have been asking, which is whether the copying process completed, rather than the only question that mattered, which is whether the work could be brought back. Two years of correct answers to the wrong question is a fair description of most backup arrangements in small offices, and the difference between those and a genuine one is ten minutes, four times a year.

About Gordon Achebe

Gordon writes about what lasts, what does not, and why.

View all posts by Gordon Achebe

About the author

Gordon Achebe

Gordon writes about what lasts, what does not, and why.

More from Gordon Achebe