The Plain Record

What things really cost, and why.

The drive was backing up for two years. Nothing on it could be restored

Posted on by Gordon Achebein Technology3 min read

A small converted office above a detached garage at the end of a driveway with low sun on the windows
A small converted office above a detached garage at the end of a driveway with low sun on the windows

A two person bookkeeping practice, working out of a converted room above a garage, lost the contents of a working computer to a failed drive in the middle of March. Inconvenient timing and not a catastrophe, because there was a backup. There had been a backup for two years. The little icon had been green the entire time.

What was on the backup was a mirror of the same folder as it existed that morning, which is to say a mirror of a folder that had been emptied by the failure. This is a common ending and it is worth walking through, because every step in it looked correct from the inside.

What was actually happening

The setup was file sync, meaning a service that keeps a folder identical across a computer and a remote server. Sync is genuinely useful. It is not a backup, and the distinction is exactly the one that mattered here.

A backup keeps versions of a file over time. Sync keeps one version, the current one, everywhere. When a file is deleted, corrupted, or encrypted, sync does its job perfectly and propagates that state to every copy within minutes. The faster and more reliable the sync, the more thoroughly the damage is copied.

Many sync services do retain deleted files and older versions for a window, which is what saves most people. In this case the retention window had passed for the files that mattered, because the drive had been failing quietly for weeks and the corruption had been syncing the whole time.

Why nobody noticed

Because nothing ever told them to look. The service reported success, which was accurate. The failure was in what was being backed up and how far back the history went, and neither of those is visible on a status icon.

This is the ordinary shape of backup failure. Almost nobody loses data because a backup stopped running. They lose it because the backup was running against the wrong folder, or excluded a database file that was open, or kept a week of history when they needed a month.

The test that would have caught it

Ten minutes, twice a year, and it is the entire discipline. Pick a real file, something you would genuinely miss. Restore it from the backup to a different location. Open it. Confirm it is the version you expected and not an empty shell.

Then do the harder version once: ask what you would do if the computer were gone entirely, right now. Where is the restore coming from, who has the account credentials, and how long would it take. An operation that cannot answer those three questions does not have a tested backup regardless of what is running.

That framing is not original to this office. The recovery material published by the National Institute of Standards and Technology rests on the same point, written for organizations a thousand times the size: a control nobody has exercised is an assumption, not a control.

What recovery cost

Some of it was recoverable from other places, which is usually true and usually forgotten in the first panic. Email held sent copies of most client documents. The accounting software kept its own cloud copy. Bank records were retrievable.

What was gone was two years of working files, notes, and the local database that tied them together. Rebuilding took several weeks of evenings during the busiest season of the year, and the practice had to tell clients that certain documents would need to be sent again. The financial cost was modest. The cost in credibility and in March evenings was not.

The rule they use now

Three copies of anything that matters. Two of them on different kinds of storage. One of them out of the building. It is an old convention in the trade and it survives because it addresses the three ways data actually disappears: hardware failure, human error, and something that happens to the building.

In practice that meant the sync service kept for what it is good at, a proper versioned backup service running alongside it, and an external drive rotated to a relative's house every few months. Nothing exotic and nothing expensive.

The change that mattered most was smaller than any of that. Twice a year, on a date in the calendar, somebody restores a file and opens it. That is the whole difference between a backup and a hope, and it takes less time than a coffee break.

About Gordon Achebe

Gordon writes about what lasts, what does not, and why.

View all posts by Gordon Achebe

About the author

Gordon Achebe

Gordon writes about what lasts, what does not, and why.

More from Gordon Achebe