I worked for IBM for many years. When Warwick was built they specified there should be two entirely independent cables bringing mains power to the site (quite big fat cables, clearly). 'twas done..

Some time after it opened - IIRC before the "hosting paying punters systems" game started - it all went "pufffttt" and the power dropped - no juice from either cable. everyone runs round in circles panicing & the systems are brought back up on generators... those that weren't UPS'd.

After the inevitable enquiry it transpired that although there were two separately routed cables into the site///.................. they both led back to the same electricity sub-station. Brilliant!

theartfullodger

Have been the unfortunate engineer on duty when this sort of thing happened. Fortunately we had enough battery power to keep the site running until our facilities engineer could get on site and figure a way to get things working.

I can't tell you how fun it is standing in the pouring rain trying to get the generator to come on line (you could manually start it but the UPS wouldn't take the input from it) The engineer ended up pulling a sodding great fuse out of the main panel which failed one of the phases at which point the the whole thing just sprang into life. I wasn't paid enough to reach into a three phase power distribution panel I can tell you....

So I can have sympathy for them. Sometimes you can test everything as much as you want but until the power goes out for real you just don't know.

<rant>

Here, we have to get customers approval for this type of drills. Time and again, no customer wants to run any risk related to "let's switch off the mains and see the generator humming".

Catch-22, anyone?

</rant>

Being on the flightpath to Birmingham International, my understanding was the many of the critical Warwick services were real-time mirrored (albeit onto older kit) in other sites, precisely because of the risk of a 'jet crash' scale calamity. I suppose it depends what level of backup the customer is paying for, though. I'd be interested if customers such as TUI were affected, for instance. It was always considered a little ironic that TUI would pay for such a high level of redundancy, given that in all likelihood, it would be one of their jets, doing the crashing!

Either way, many of those systems can take upwards of an hour to reboot (some are rebooted once every year, as a matter of course, and it's quite an operation, whenever it's that time of year - even when you have a schedule to work to). So bringing the whole lot back from an unexpected sudden death, in stages, would cause a lot of downtime.

It is a must in our organisation... every Thursday our mains power drops out and the generators take over for a while. The last time something did go wrong the whole thing pancaked, but at least we learnt from it. Chances are that our systems won't go down...

IBM clearly was a bit too confident that everything worked and would continue to.

Hmmm... that said, interesting that IBM has a massive data centre in Warwick - The town's not really big... ;-)

My impression is that the power outage was quite short - nothing like 7 hours. The rest of the time was spent in bringing systems on line in an orderly manner. I would be surprised if any readers of The Register did not understand this. Certainly all the customers would be aware of it -it's stanfard T's & C's used in all contracts covering this type of service provision.

The big issue is what were customer expectations, based on the specifications of the MG sets and the UPS.

It is my experience that standby MG sets are far too frequently just that: they standby while everybody is expecting them to start. So the UPS should be specified to be large enough to allow a manual start of the MG after main failure. But it would be very rare - and unacceptably expensive at a site the size of Warwick - to specify UPS that can support the failure of a prime or singleton MG set while mains power is also off. The mean time to fix a failed MG set is certainly some hours. A better solution is always to have multiple MG sets, even though there is extra synchronisation costs involved - with each set capable of supporting a reduced operation of the site.

I would hope the tested the generator. But, I've heard of at least several large instances where the generator test itself caused an outage -- generator caught fire, burned out some switch it shouldn't, etc.

I would also not judge too quickly -- they summarize it as "the generator failed", but that doesn't necessarily mean it didn't start; possibly the hardware failed that hooks the generator to the data center; maybe there was some power surge when the power went out, damaging the generator etc.; maybe the generator ran but was not working under load. Any of these would not have been found even if it was tested.

@AC, "they both led back to the same electricity sub-station. Brilliant!" That's hilarious. This does happen with data as well; I remember reading about some data center that ordered like 6 or 8 different redundant connections so a backhoe wouldn't knock them off line. Well, one did... these providers were claiming they had different physical connections but actually had something like different wavelengths on the same fiber, or different fibers in the same bundle, so indeed a backhoe knocked out the primary and all backups.

" No, one would expect a world class data processing outsourcer like this to have multiple site datacentre rendundancy configured to work seemlessly in active-active or active-passive failover configuration."

That really depends on what kind of data center they are. The case where they were hosting SAP for someone, I'd agree, IBM should have been taking care of redundancy. In some cases, hosting amounts to "put your boxes in our datacenter", it's up to the customer to then get redundant boxes, put them in multiple data centers, and arrange for failover.

I think when doing generator testing with live equipment on the line you wouldn't just flip off the main power to test to see if the generator comes up, you should be able to do that without having to force the UPS to take the full load.

When testing UPS systems you should be able to fire up the generators so that in the event the UPS fails the generator power is already flowing and there should be no disruption of service.

Of course if the automatic switch over fails then you lose power to the UPS..that happened at a local Internap facility here a couple of years ago, the UPS worked, the generator worked, but the automatic switch that flipped the UPS from grid to generator power failed, the batteries were already needing replacement and didn't last as long as people thought and they failed.

And of course it's ideal to have everything connected to two independent power sources so you can test one at a time. And run metered power strips(at least) so you can ensure that you are not overloading your circuits, most data centers won't alert you to this unless your a serious abuser of this(long after you've effectively lost redundancy by oversubscribing the line too much)

And I for one wouldn't want to be at a data center where there's only 30 seconds of UPS power, I would want at least 15 minutes, in the event of a problem 30 seconds is not enough time of course for on site personnel to respond. I think data centers that use flywheel UPSs are just nuts.

Now if my equipment is fully redundant across sites and losing a site isn't a big deal then it's not a big deal, but organizations that have that level of availability are very few and far between.

Running generators as a test has its own problems, as BT found out many years ago. Policy was to start the diesel generator every Monday to make sure it worked. It did.

Then when the power failed one day, the generator started fine, ran for about 30 minutes, and died.

It turned out that the constant cold starts & shutdowns had had the same effect as short runs in a car, the generator was well coked-up inside. Once it got up to running temperature all that glowing carbon caused havoc.

After a full cylinder-head-off rebuild the test policy was changed to a test run once a month, up to full temperature for a few hours.

Here's some slightly better maths: five 9s is 00.001% time down, which is 315.36 seconds per year, or just over five minutes. So a 7 hour outage is still within your uptime requirement for the next century with a bit to spare...

Also to comment i assisted several times external power cutting situations (like some road works and a machine digging and cutting the power lines) and the Data Centre being kept running for 12+ hours with the diesel power generators and UPS, and later through the second external power line connected to a different sub-station.

5 hours to bring the systems up and running don't seem to be a catastrophic situation as probably the (business) data loss was minimal or nule and the true impact was stopping electronic business for some hours which also happens a lot with normal ATM debit payments with having these disruptions...

Correction:

"... which also happens a lot with normal ATM debit payments WITHOUT having these disruptions..."