back to article Telecity's engineers to spend SECOND night fixing web hub power outage

Telecity engineers will tonight make a second attempt to fix broken power systems at its critical Sovereign House internet facility. They will be working 24 hours after their first efforts failed and resulted in a second power outage. The internet service giant had planned to repair and fully fix the power supply in the early …

  1. Anonymous Coward
    Anonymous Coward

    One of our VOIP services was killed by this yesterday afternoon but it was working again this morning and it is still working now (18 November 13:33)

    1. Anonymous Coward
      Anonymous Coward

      Good nice to see they supply 5 9's for telephony, the carrier industry standard...Oh wait.

      1. Anonymous Coward
        Anonymous Coward

        Good nice to see they supply 5 9's for telephony, the carrier industry standard...Oh wait.

        Due to a transposition error Telecity designed Sovereign House for nine fives, and as a result the UPS and backup power were an empty cardboard box with wax crayon writing that says "N+1 power backup". Luckily the finance department didn't make the same error, so customers were billed for a full fat resilient service.

        Makes you wonder why the clowns of DCMS are committing £1.2bn for nationwide broadband rollout so that farmers can watch ovine grumble in HD, when a single data centre snarl up can mung 10% of UK internet traffic, for want of a sub £5m investment in a power system that simply works.

  2. Alister

    Amazon AWS EU-West1

    We run some monitoring software on a VM instance in Amazon's EU-West cloud, which we use to keep a check on our servers which are in various datacentres around the UK. Yesterday at about 14:00 GMT we started getting buried in alert emails, as the AWS instance couldn't see any of our stuff in Union Solutions, 6Degrees, Node4 and various other providers. We ended up turning the monitoring off, and weren't able to start it again until past 19:00 GMT last night, and it's still a bit flaky today.

  3. Tom Chiverton 1

    Best source of updates and reasons seems to be Jason's tweet stream!

  4. Dan Paul

    Cheap datacenter providers ......

    who don't provide UPS on your servers aren't worth the powder to blow them to hell!

    ALL their uptime promises are a lie.

    However, the person who chose them should have to live with the outcome.

    1. Lee D Silver badge

      Re: Cheap datacenter providers ......

      Honestly, how long do you think a UPS will last, of any size?

      And I've seen enterprise-grade UPS just switch off with full battery. They have to do so if they detect cross-phases, bad earths and other faults on the line or equipment or they will, quite literally, kill people and start fires. They are by no means a solve-all device.

      There is no amount of money you can pump into something to build a resilient service. It's just not possible. Or we quite literally would have no need for backups, failovers, hot-spares, etc.

      The people who are running services on JUST ONE DATACENTER are the idiots here, lying to their customers that they will stay up. There's a reason that Amazon datacenter has such a name. It's just one of very many, and you can migrate instances away to unaffected datacenters, or spin up your backup/failover instances.

      But a UPS? Pfft. Even in a datacenter they are there only to level out spikes and hold things up until the generators kick in, and the generators can only work for as long as everything else is safe and there's diesel enough. One engineer on the floor who could get zapped, and you are REQUIRED to switch everything off. Proven by the guy who DIED in a datacenter only a few weeks ago because he tried to work live on the system because the datacenter managers didn't want to turn off the power.

      Guess what the downtime is for a death once the emergency services arrive and tell you in no uncertain terms to make sure its safe for them for the entire duration of their investigation?

      1. John Stoffel

        Re: Cheap datacenter providers ......

        All decent data centers have a UPS, it only needs to be online for 10 minutes at most before the generator(s) spin up to take the load. This sounds more like a transformer or two got fried somehow.

        As for your claim about someone dying, care to post some particulars or point to a source for this claim? I don't doubt that it's hjappened at some point in time... but showing your sources before making claims like this is always better. Otherwise you get flamed.

        Me? I'm just showing you the coals and saying "be a shame if they got dumped on you... accidentally like...". LOL

        1. Lee D Silver badge

          Re: Cheap datacenter providers ......

          The Register originally, I believe! If not, Slashdot.

          Article was about the company that the contractor was working for being fined into oblivion by the HSE.

          http://press.hse.gov.uk/2015/engineering-and-facilities-firms-sentenced/

  5. Anonymous Coward
    Anonymous Coward

    Power problems

    Shirley renting a BFG (as in Generator, not Gun) to park in the street, plus assorted power distribution equipment and a ton of extension cords would be a short-term crutch?

    Granted, there are logistical problems like maintaining security and safety while running on temp power. Hiring security for access control and firewatch would get around much of that.

    1. Anonymous Coward
      Anonymous Coward

      Re: Power problems

      I worked for a company that tried to run a datacentre on a Generator from the street using lots of power extension leads. Didn't work out so well.

      Fixed wall mount air conditioning is a requirement to keep it cool otherwise the air temp quickly rises over 50 deg C.

      Portable Air con units also running on the Generator are not a solution to a problem caused by being a cheapskate in the first place......

      Annon because who would actually admit to doing something so daft even if it was a LONG time ago....

    2. Terry Barnes

      Re: Power problems

      That entirely depends what the problem is. Sometimes the issue is getting the current from the generator to the racks. An outage of this length almost guarantees that it's not just a case of a blocked fuel filter or stuck changeover relays.

      Most buildings of this type have existing external connections for a mobile generator and a reserved parking space all ready to go.

      It's not something you can bodge because once the power's back on no-one will let you turn it off again for a long time so a long outage indicates to me that something significant has broken and is taking time to repair properly. That could be a distribution board or a busbar or something of that order.

      If you bodged temporary power with extension leads it would be a) quite dangerous but also b) a permanent solution until you can agree a changeover back to normal power with every single customer affected. I'd trust a generator for a week or so, I wouldn't like to be relying on it six months later.

  6. K

    This took us out for a period yesterday..

    We've resilient leased lines with BT and Virgin (routed through Timico), this initially affected our internet traffic about 1pm, for a brief period, then struck with vengeance about 2:30pm. Fortunately we've not epxerienced any problems today.

  7. tin 2

    Another source reckoned customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for.

    LOLOL yeah more fool them for thinking they got it more like!

    1. Terry Barnes

      "customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages "

      If you don't have site resilience you don't have resilience - you have protection.

      It's the right call for a lot of business needs though - it's a case of balancing the cost of achieving very high uptime versus the cost of being down. Most importantly, don't write contracts for your customers that promise availability that isn't at least equalled by the contracts you have with your suppliers.

  8. Anonymous Coward
    Anonymous Coward

    That accompanying photo...

    ...of the woman doing a mini-scowl, scrunching up he nose. I'm not sure you should use it. It's most distracting. She's pretty sexy. I'd be very happy to try and fix her critical power hub.

    1. Anonymous Coward
      Anonymous Coward

      Re: That accompanying photo...

      That accompanying photo... .

      is just sexist dinosaur clickbait?

      Ladies, do you have moments like this: "that's a dull article on <subject>, but the bloke in the photo looks a bit of alright, I'd be happy to administer his systems"?

    2. Anonymous Coward
      Anonymous Coward

      Re: That accompanying photo...

      The 1970's called and would like their ancient sexist crap back.

      1. Anonymous Coward
        Anonymous Coward

        Re: That accompanying photo...

        The 1970's called and would like their ancient sexist crap back.

        Out of curiosity, does this comment mean that IT articles must only be accompanied by pictures not featuring people? Or must only feature geeky/old men? What if the geeky/old men were black or look a bit weird or a bit camp? Any of that allowed? And what about unattractive women? Would Ann Widdecombe be non-sexist on account of her, well, appeal? And what if there's people who actually fancy her? They might be a tiny minority, but we wouldn't want anyone harbouring the wrong sort of thoughts?

        1. Anonymous Coward
          Anonymous Coward

          Re: That accompanying photo...

          "Out of curiosity, does this comment mean that IT articles must only be accompanied by pictures not featuring people?"

          No, it means that an industry welcoming to skilled people of all sexes wouldn't have websites where men comment about women thus;

          "She's pretty sexy. I'd be very happy to try and fix her critical power hub."

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like