bjoernvold.com

Here is the sequence of unfortunate events

Monday 16th

Main and UPC Power failure at data center (DS)
MD3000+MD1000 dual controller, dual host storage array containing 2TB of RAID10 volumes, containing basically everything, comes up confused
Working remotely, identify the key to restoration: 100gb, 50% of an XFS partition
Lacking an entire “dark” duplicate storage stack to bring up, get Dell involved

Tuesday 17th

Dell support says they don’t know the cause, but we must wipe entire array, do firmware upgrades, and start again. I don’t trust this gear.
Check backups, mail: ok, nfs:ok, site files: ok. The sql backup is incomplete.
Since all physical drives are green, and RAID10, so contain two copies, we request all 13 that contain this and other volumes, by carefully pulled, labelled, and shipped to Lab #1 (Colorado). Lab #1 (and #2) say they see MD3000 problems frequently. I am still relaxed at this point.
DS employee uses his initiative to decide that all disks are hot-pluggable, so pulls them all for shipping, with the equipment powered up. Getting a bit more tense.
Recovery info is presented to Lab #1 identifying LVM2 uuid, storage array config, etc

Wednesday 18th:

Lab #1 receives and images all drives ok

Thursday 19th:

Lab #1 fails to answer direct questions on whether scan of retrieved images shows missing UUIDs or not. Getting more tense still.
Estimated “success” bill rises to $8k

Friday 20th:

Lab #1 wants the “good half” of the missing filesystem (even though it is on the disks they have) but do not possess a fast internet connection so it must be FedEx’d. Starting to get a headache.
DS write the “good half” of the missing XFS filesystem to a USB drive and FedEx it, by dropping it into a “monday pickup” box. Four letter words.

Monday 23rd:

Bring up an old snapshot of the database that was not stored on storage array, in order to service search engine traffic to forums etc ( the majority of our traffic daily)
Lab #2 give a quote of $6k to $18k, and ask for the entire equipment stack, both storage arrays and the host as well. This is despite the missing chunk contained within just 6 drives. What a good business to be in!

Tuesday 24th (last few entries are my timezone dates, a day ahead):

Give Lab #1 a couple of days to deliver the goods, and ask for drives to be shipped to Lab #2 as a backup plan. We’re in “wait mode”.


postscript:
If you have extracted a virtual disk from the physical disks in a disk group, from a borked MD3000 before, feel free to drop your business card off to adminhelp (at) dslreports.com. At least as a backup to the backup plan.

viking60 wrote:..
In this case I would say:
If you use RAID 10 with all that redundance why do you send all your disks away?

Beeing shocked and wondering"who has pissed in my pants"! may look dynamic, but does not cut it.

This is not force majeure .

Update 27th April:
Whenever we’ve had the possibility of a good or bad outcome, we get dealt the bad outcome. Inspection of all disks has so far not supported any better theory than the SAN (storage array) over-wrote enough data to make full restoration a fragmented process, at best. I would like to dump technical detail as it seems a ludicrous scenario, but cannot until the analysis is complete. I’m not going to be convinced until we get more than just one professional opinion.

In the event things drag on for longer I could unlock the site for “normal” use, leaving space for whatever data recovery turns up, however this in itself is tricky. Meanwhile I will free up more older but still useful pages. This is no help to members who wish to use restricted forums or get to their instant messages, or post in forums, but it at least heads off some of the search engine damage created by unavailable URLs.

I hope we’ll catch a break in the next few days and can post something more optimistic!

Update: 7th May
I am told by the second lab the missing data is intact up to the time of the power fail. I have to now wait for payment to go through and return by courier of the media. Then there will be time spent checking the data, and restoration of both data and hardware without adding too much downtime. For the curious: this event has cost $28,000 in lab recovery fees. This is not including any money that we must now spend on new hardware, does not include financial impact of downtime, permanently lost traffic, and so on.