No uninterruptible power supply?
UK flights CRIPPLED by system outage that shut ALL London airspace
All London airspace was closed to incoming and departing traffic for just under an hour on Friday afternoon due to a computer outage at the National Air Traffic Services – Blighty's air traffic control authority. According to the European Organisation for the Safety of Air Navigation, a machine failure resulted in all airspace …
COMMENTS
-
-
-
Friday 12th December 2014 16:07 GMT smudge
http://www.allen-diesels.com/case-studies/civil-aviation-authority.php
Two 16kV, 5MW diesel engines, it seems.
Note that this is not a comment on Allen Diesels, nor am I suggesting that they are in any way at fault. I merely came across that page when searching around t'interweb to see what sort of backup power supply Swanwick has.
-
-
Friday 12th December 2014 16:26 GMT smudge
There was a data centre (in Denmark or the Netherlands, I think), where the power supply failed.
The diesel generator came up as it should, and powered everything for a couple of days, by which time it needed refuelling.
The diesel tanker duly arrived - and crashed into the outbuilding, completely trashing the generator inside.
-
Friday 12th December 2014 16:35 GMT Xpositor
Quite a few years ago when I worked for a bank that was subsequently taken over by a Spanish bank, we had a UPS test one weekend. Everything worked flawlessly. Unfortunately, mid-way through the day on the Monday, all power in the building failed, knocking out the mainframes. Some herbert had forgotten to switch back over to mains power, and the generator duly ran out of diesel.
-
Friday 12th December 2014 17:10 GMT Phil O'Sophical
Adding to the diesel stories, I know of a site that had diesel backup which was regularly tested. Once a month, for many years, the generator was started, fuel levels verified, etc., Unfortunately the diesel was run for about 2 minutes each time, which had the same effect as lots of short journeys in a car. When the power failed years later the generator started and took over the load flawlessly...until it warmed up. Once hot the thoroughly coked-up engine misfired, wouldn't keep speed, and spluttered to a halt. It took a head-off engine rebuild to get it back online.
-
-
-
Friday 12th December 2014 19:39 GMT HelpfulJohn
"The Dish".
A brilliant film. Very moving in places and full of understated Ozz/UK style humour. The Oz politicians are wonderfully inept, witless and charming.
The American from NASA is so USAlien he's just barely credible but he is a perfect foil for the more relaxed Ozlanders, also a great straight man.
http://en.wikipedia.org/wiki/The_Dish
And the radio telescope's dish is still in the middle of a sheep meadow.
-
-
-
Friday 12th December 2014 16:31 GMT Anonymous Coward
@smudge
Reading the page you linked to, it is the most unfortunate bit of trumpet blowing imaginable, since it is telling us that "nothing can go wrong".
My own UPS horror story? Power cuts out briefly, battery UPS comes on line, there is a thump and a roar as the Diesel generators cut in. Then after a minute white smoke is noticed emerging from the stack by the Diesel shed, the lights flicker and there is an enormous bang. Lights go out.
The maintenance department, the last time the standby generators were serviced, cleaned out the old oil and omitted to replace it. Subsequent inspections had apparently failed to notice that absence of oil level.
Very soon after, the post of facilities manager was vacant.
-
Friday 12th December 2014 17:06 GMT 142
@ arnaut
A similar thing happened, at a massive scale, at the Dublin Amazon AWS data centre a few years back http://www.siliconrepublic.com/enterprise/item/23084-mystery-surrounds-outage-at
They blamed a lightning strike initially but it appears to have been poorly configured failover gear.
-
Friday 12th December 2014 18:53 GMT Dave Henderson 1
Re: @smudge
Bit of a corner-cut stand-by genset if it didn't have the basics of a low oil / low pressure cut-out before it went clunk. Even 30 years ago a decent panel wouldn't have allowed it to run up and hold without pressure being present. Then again, I used to see the most stupid cost-cutting on vital things that might only be needed once in a blue moon - then the kind of thing above would happen.
-
-
-
-
-
Friday 12th December 2014 16:12 GMT Anonymous Coward
Re: It would appear to be worse...
Redundancy means nothing - I work in engineering, and no matter how you plan, something always goes tits-up from little incidents.
Have a read here:
Logging in on my Slack notebook today was:
Linicks Law: If Murphy's law can fail, it will.
*OK wiki crap, but it reflects the paper I read on redundant redundancy once.
-
Friday 12th December 2014 16:25 GMT Bartholomew
Re: It would appear to be worse...
I've seen where they have slowly added more load, over years, than the system was designed to handle. The zero crossover switches, switch flawlessly to they synchronised backup generators and, then fail because they are not rated to carry the larger currents. Then you need to call the sparks in to jury rig a temporary workaround solution, which takes time.
-
-
-
Friday 12th December 2014 16:55 GMT Hans Neeson-Bumpsadese
Generator testing
I remember one site where they tested the backup generator religiously for a few minutes every week. When a major power outage did occur and they needed to run of the genny, it failed within the hour...all that testing had run the fuel down, and nobody had thought about topping it up.
-
-
-
Friday 12th December 2014 16:41 GMT Anonymous Coward
Big UPS
Each of the backup generators is rated for 5MW. And what is the holdup life on your home PC UPS? Typically they hold up long enough for a controlled shutdown.
Let's be optimistic and specify a 4 hour hold up time. At 5MW that's 20 000kWH. For comparison, a typical 110AH leisure battery rated at 12V can supply C/10 for about 4 hours before suffering serious damage. That's 44AH * 12V = approx 500WH.
20 000kWH = 40 000 standard leisure batteries. That's nearly half a million times more than the 10AH unit in a typical PC UPS.
tl;dr: storing large amounts of electrical power is very expensive, that's why we still drive fossil fuel cars.
-
Friday 12th December 2014 16:49 GMT I ain't Spartacus
Jim Willsher,
What's the battery life on your UPS, to keep your PC alive? Oh, plus the phones, radios, lights, power to the other hundred PCs, links to all the radar data you need for ATC, links to airports etc? That probably takes a tad more battery than just to give 1000W at 240V to a single desktop.
Presumably the UPS has to keep everything up long enough for the generator to fire up to keep providing power. That's assuming something hasn't gone wrong with the internal power wiring, in which case there's external power coming in, it just can't be distributed (and neither can the power from the genny).
-
-
Friday 12th December 2014 16:29 GMT Inventor of the Marmite Laser
At least some stuff seems to be arriving/leadving Luton:
http://www.london-luton.co.uk/en/flights/
Interestingly NATS says:
NATS can confirm that a technical problem has been reported at Swanwick air traffic control centre.
UK airspace has not been closed, but airspace capacity has been restricted in order to manage the situation. We apologise for any delays and our incident response team has been mobilised.
Every possible action is being taken to assist in resolving the situation and to confirm the details.
Further information will be released as it becomes available.
-
Friday 12th December 2014 16:09 GMT David Pollard
If memory serves
Wasn't there a problem with the power supply a few years ago? As I dimly recall, there was a component like a smoothing capacitor in the common feed and it was this that failed. The power supply went down and the UPS came up and was connected to a short; or something along those lines. Everything worked perfectly apart from an unlikely fault that no one had foreseen.
-
Friday 12th December 2014 16:17 GMT Anonymous Custard
12 more sleeps 'til Santa?
Well all I can say is they've got 12 days to fix it, or my kids won't be happy if Rudolph and co don't get clearance to land.
That said one of my kids won't be happy anyway, as the storm and power cut we had last night took out the micro-SD card in my Pi, so about 3 weeks worth of her Minecraft tweaking (since the last back-up) looks like it maybe a gonner.