As I’m in the process of slowing shutting down other sites I’ve worked on (due to time constraints), I’m reposting them here. Here’s an article from “Behind The Frontline” which was going to be a cross between I Work With Fools, Worse Than Failure and Dilbert.
Category: Tech Scares. Author: Myself.
Plusnet’ed (PLUS.net.EDD) v. – an operator error which results in loss of significant amounts of customer data.
Origin: The loss of 700Gb of customers emails from Plusnet’s mail servers in July 2006 (Source: The Register)
Today, I thought I’d setup a new backup solution for our customers with the aim being to be able to offer them a sort of “instant rollback” style service. So we install additional hard drives into all the servers – no problem.
Now just to format (prepare them for use) and “mount” them (make the computer recognise them). No problems.
Hang on – what’s this beeping noise on the monitoring systems? Why is my screen turning red? Oh, the monitoring system says a test page on the server is down, not to worry, it’s probably a false alert. I’ll just check it manually.
“404 Not Found”
That’s odd. Let’s double check that link, yep, it’s correct. Let’s try another page. Still not found. Worrying.
Let’s check the server. Login. “cd /home/” (move to where all the customer data is) and ls (list files). Nothing.
Crap. Where has all that data gone and am I know 100% certain I’ve wiped and formatted the correct drive. (at this point, panic sets in).
Let’s just reboot to be sure. Reboot. Still no /home/ data. Crap. But no other data is affected, so let’s just try unmounting home (tell the computer to “forget” about it) and remount it. There’s the data! Why did the computer start thinking the data should be on the new blank drive?!?
I really really started thinking that I had made a major mistake and wiped nearly 200Gb of customer data: and since it was a backup system going into operation, the old backup system had been taken offline a while earlier.
It’s now 2 hours later and my pulse is only now beginning to stablise!