Hitachi Data Systems fingered in Sidekick disaster

Probably an incorrect protection resync

One real easy way to destroy data permanently is to have do a data resync of a mirror in the reverse direction while having an active filesystem against it.

i.e. lets say before you do a upgrade you mirror the drive as a rollback so you have a point in time to roll back to. If an admin does a restore rather than a resync, you'll be rolling the disk back to a date a month ago live (which is something you can do with arrays), while the filesystem didn't know you did that and will continue writing to filesystem with new data. So you'll have a filesystem with old data and new files and a corrupted filesystem. If you have some form of a database (i.e. oracle, ms sql, etc) which is most likely in this configuration even filesystem forensics looking for file EOF markers wouldn't work. If you are replicating the data it would copy over the corruption as well.

Hardware won't stop you from a bad admin (we had this type of thing happen to us by a contractor 10 years ago)

What is wrong with Microsoft?

What is wrong with Microsoft? This company is utterly inept. Blaming this on Hitachi? The problem has gotta be Microsoft's. The behaviour of Microsoft is bizarre and unbelievable. As the article suggests, how is it even possible that Microsoft was able to lose terabytes of data?

At least Microsoft should come clean about what really happened. We deserve a better explanation that what Microsoft has given so far.

Microsoft's share of the mobile phone market is declining rapidly, and this latest incident will further alienate Cellphone network providers (eg T-Mobile) as well as customers, and hasten Microsoft's mobile decline into irrelevance.

Business as usual

I bet this was the result of a misunderstanding/miscommunication. No way in hell all three (or four?) companies involved employ total amateurs too dumb to have a backup or five. In my experience , when too many companies with different storage policies have access to one dataset, its usual conflicting policies badly communicated that lead to several people overwriting their version of the same data, each thinking that the other is keeping not a backup, but "a version we should be able to merge the rest with later", or something to that effect; have this go some rounds of " uh-no,-do-that -backwards-and-then-the-other-way-round", and suddenly all versions are useless.

The sad thing is that, as far-fetched as it sound, it's happend more than once where i work.

(not one of the companies named in the article, fortunately.)

Let me guess...

PlusNet all over again.

Typing format in the wrong window during an upgrade which requires formatting master, moving data and formatting secondary.

It gets highly entertaining when you have a window opened to the secondary at the wrong time and execute a format there.

What could go wrong?

Amazing that ALL of your backups are stored in the same set of hardware. And that an upgrade can wipe it ALL out?

Was there no replicated database at a separate location?

Heh

Backups, backups, backups! ...oh wait

HDS...very sour taste!

We've been in talks with HDS lately for some kit but it all went a bit sour, they simply couldn't really be bothered to deal with us, not retruning calls, hardly any bartering. We were looking at about £2M, reasonably small by most standards but by contrast EMC couldn't do enough to get the order. HDS left a very sour taste and it looks like their services division are not much better than pre-sales!

"What kind of server or SAN crash would actually delete terabytes of data on a SAN's disk drives?"

One that wiped out the one and only copy of a non-recoverable master encryption key, perhaps?

Well..

Well, It depends what went wrong on their SAN. I mean yeah, the bytes are probably still on the drives, I doubt it went screwy enough to overwrite everything.

BUT.. What if the logical mapping is gone? How do you know what belongs to which partition? Kind of like when your partition table gets overwritten, except you're dealing with hundreds of drives.

Backups are "Old School"

An actual quote from a customer.

More ways to fail

I assume everything's in some proprietary database. It's probably not stored as plain text/numbers - with that much data, compressing each record will save a lot of space and disk accessing - so a low-level scan of the disks won't necessarily produce anything usable if the main index has been trashed. RAIDs and such are only a defense against physical drive failure, not data corruption. If the nightly (or whatever) backup routine is allowed to run before everything has been fixed, well, there goes the backup too. Some people keep a series of backups, but that requires a lot of disks, and who actually expects such a catastrophic failure?

I'm sure someone can come up with a witty comment about the wisdom of acquiring a company called 'Danger', or even naming it that in the first place.

Hardly Data Storage

So Bob, tell me how an "EMC Clariion can lose all it's data pretty easily". Go on entertain me.

Cue the circus music

and imagine Microsoft, Danger, Hitachi, T Mobile etc chasing each other in circles around the server room.

doo do doo doo do do do do, do do

duh!

Microsoft + Cloud = Danger

It appears

"It appears that the direct cause of the Sidekick data loss may have been storage area network remedial work outsourced to Hitachi Data Systems."

To you perhaps. To me, it appears that Microsoft are looking for someone else to blame for never making an offsite backup. It also appears that El Reg are happy to back Microsoft up on this, based on some astroturfing they did on Engadget. Hey, here's a juicy rumour, let's print it as observed fact. Twats.

You know as well anyone else that whatever HDS fucked up should have resulted in no more than some downtime while they recovered from backup. The non-existence of this backup is the direct cause of the data loss. But no, it's never management's fault.

How the fuck can you not see right through this story?

Massive problems like this are never *caused* by predictable things such as SAN upgrade fuckups or fuses blowing on plugs. They are *caused* by organizations' refusal (on the basis of cost) to mitigate against events which "can't happen" but invariably do.

tape

Tape is a pain in the arse and far from trendy, but a set of tapes (preferably several) in a firesafe is a "backup". I wouldn't use the term "backup" to describe anything else.

Speculators.

Right, lets start with the facts : Er, data was lost.

Reuters report a server failure damaging two databases, Engadget reports "rumours" and that "the upgrade runs into complications" so very little in the way of facts to work with there.

There is no meat to say if a database was corrupted or portions dropped with no backup , if a filesystem with a flatfile store was the "database" and the FS was corrupt or if a disk array related failure occured and it was wiped, ruined, overwritten or otherwise lost by a tired technician. Theres not a single mention of array technology for replication anyway but HDS get the slagging off.

Who knows why it fell over and why no backups on tape exist (assuming that tape is not out of fashion yet), maybe a disgruntled techie burnt the lot!

I just find it amusing that a company called Danger was purchased by Microsoft, who probably migrated everything onto M$-$QL and M$ Server products and the whole lot fell over, a bit like Microsoft touting the London Stock Exchange on their products as a reference site only for it to suffer total trading outages a few months later.

I'll join in and speculate they should have been on some form of Unix with proper databases as my money says Wintel / MS-SQL problems are hidden in the middle somewhere.

I'm a Geezer Me - I'll nick Anything!

"Amazing that ALL of your backups are stored in the same set of hardware."

Maybe, Juust Maybe, they had the entire database in some derelict building outside a caravan site in, f.ex., Slough and the local lads & ladettes simply burgled the place for the copper wiring.

This kind of thing Never Happened Before, right??

Wait, DRAM?

You're telling me those Sidekick smartphones *don't* use Flash memory?

I'd never use a phone like that. I've lost too many contact lists on older phones, and I definitely don't want that to happen again.