back to article BT broadband goes TITSUP - cripples Scots, Geordies, Northern Irish

Some of the telcos which depend on BT's network to offer broadband services to their customers have been reporting a major outage affecting thousands of people in the North of England, Northern Ireland and the two biggest cities in Scotland. ISP Zen Internet told its subscribers that BT's 20CN and 21CN circuits were currently …

COMMENTS

This topic is closed for new posts.
  1. Roger Kynaston
    Pint

    First rule in a major outage

    Take the phone off the hook until you know what has gone wrong. That way you can concentrate on diagnosing it without answering idiot questions from know nothings.

    Beer because I am sure the BT engineers (or whoever they have outsourced it to) will need lots after today.

  2. GettinSadda
    Mushroom

    Aberdeen too

    Aberdeen went out too, but came back on an hour or so back

  3. Anonymous Coward
    Anonymous Coward

    No mention of BT Openreach? As I understand it BT Openreach are the wholesaler. BT Broadband (in your tttle) is for when BT is the ISP - with end products such as BT Infifinty. No wonder Joe Public thinks that they can only buy their broadband direct from BT.

    As an aside, Plusnet FTTC in Greater Manchester seems unaffected - it would be interesting to know the topology to understand why Preston has taken the hit.

    1. Anonymous Coward
      Anonymous Coward

      BT Openreach do the "last mile" between exchange and punter

      BT Openreach do the "last mile" between exchange and punter (oversimplified, but...).

      For retail ISPs such as AAISP, the connectivity between the exchange and the ISP is provided (and in some ways managed) by another piece of BT, BT Wholesale.

      Yes it's complicated.

      Best just to think of BT as the default provider of frequently overpriced and underperforming voice and Internet services to the naive and ill-informed.

      1. Test Man
        Stop

        Re: BT Openreach do the "last mile" between exchange and punter

        You sure?

        I'm pretty sure that BT Openreach are the ones that maintain the entire physical connection, whereas BT Wholesale are the ones that sell/lease the various services that operate over these connections wholesale to retail ISPs, etc.

        So basically if there was a problem between the ISP and the exchange, BT Openworld would be the guys to fix it, same if there was a problem between the exchange and your home.

        1. supermoore

          Re: BT Openreach do the "last mile" between exchange and punter

          Backhaul from the exchange is owned and managed by tier 1 service providers, rather than Openreach.

          It's a complicated mix of ownership. In some cases BT will be leasing fibre off Virgin, in other cases the other way round. There's also C&W fibres in the mix. It's a complex peering arrangement, much like the connections on from these networks to the internet.

          But no, Openreach don't own backhaul or the exchange kit (DSLAM/MSAN) that connects to it. Openreach are strictly HDF onwards towards the customer.

      2. cortland

        Re: BT Openreach do the "last mile" between exchange and punter

        Lucas WHO?

  4. Anonymous Coward
    Anonymous Coward

    woohoo!

    thank $Deity my internet needs are not supplied by anything to do with BT past the local exchange !! BE LLU is canine dangley bits !!

  5. Anonymous Coward
    Anonymous Coward

    normal service

    It's back on in Orkney. My ISP rides on BT wholesale, I rang them and was told that five other ISPs were down.

    Strangely, my neighbour uses the Post Office and his connection stayed up

    1. Chandy

      Re: normal service

      > Strangely, my neighbour uses the Post Office and his connection stayed up

      TCP over Parcelforce. Most reliable protocol ever, bar the odd dropped packet.

      1. Anonymous Coward
        Anonymous Coward

        Re: normal service

        Ha, have a thumbs up. Though I'd disagree with "Most reliable protocol ever, bar the odd dropped packet". There's a reason they're known as Parcelfarce even within the Royal Mail.

  6. Anonymous Coward
    Anonymous Coward

    North west Scotland too

    Went out at 9:02, back at 11:11 (thanks Nagios.) Just when I decided to vote "Yes" in 2014, BT lets me know the error of my ways. Must be connected 'cos the "No" campaign is called Better Together - BT geddit? NB for some commentards - some of this post may not be entirely serious.

  7. Anonymous Coward
    Anonymous Coward

    Plusnet are owned by BT?

    I did not know that. That's them crossed off the list as a potential isp

    1. FlossyThePig
      Holmes

      Re: Plusnet are owned by BT?

      Why? It's a bit like saying I won't buy a Ferrari because they are part of the Fiat group

      1. Anonymous Coward
        Anonymous Coward

        Re: Plusnet are owned by BT?

        How?

        Ferrari vs Fiat - different designers, different tooling, different parts etc

        ISP - The same infrastructure, same procedures, same everything except the name and run by the same people. If you have had to deal with BT you'd understand

        1. edge_e
          Thumb Up

          Re: Plusnet are owned by BT?

          Different prices, different call centers, different staff. BT have, quite sensibly, been completely hands off the plusnet business since they bought it.

          I'd happily recommend them to any non geek.

          1. dephormation.org.uk
            Big Brother

            Re: Plusnet are owned by BT?

            "Different prices, different call centers, different staff. BT have, quite sensibly, been completely hands off the plusnet business since they bought it."

            Codswallop. Most of the Plustnet Plc directors are ex-BT Retail...

            John Petter - Managing Director, BT Consumer

            Matthew James Davies - ex BT Payment Services Limited

            Simon Curry - ex BT Business Direct Limited

            Andy Wilson - previously Director of Revenue and Margin Assurance at BT Retail

            The company is 100% owned by BT.

        2. Test Man
          FAIL

          Re: Plusnet are owned by BT?

          So in other words:

          Ferrari vs Fiat - different designers, different tooling, different parts etc

          Plusnet v BT - different infrastructure (bar the fact that they both run over BT Wholesale lines), different procedures, different everything.

          1. Anonymous Coward
            Anonymous Coward

            Re: Plusnet are owned by BT?

            Plusnet v BT - whether they're different or not depends on the context.

            From a broadband user point of view: chalk and cheese. Different pricing, different packages, different facilities (how many BT Retail users know what a 'static IP' is), different processes, different call centres, etc.

            From a competitor's (and ideally from a regulator's) point of view: same shareholders, same CEO. Same company.

            1. Anonymous Coward
              Unhappy

              Re: Plusnet are owned by BT?

              I would have recommended Plusnet until recently.

              UK call centres usually excellent and (*usually...) good reliability.

              However, I asked for an upgrade to fibre broadband, they gave me an afternoon slot for the engineer. Having took a half day off, the engineer didn't turn up til 8pm, only to tell me that I lived too far from the exchange.

              Plusnet were ready for taking the fibre subscription direct debit until I told them. Then when they reverted to my usual 1mb rural broadband speed, they had doubled the price. Upon questioning this, I was told that it was because I lived too far from the exchange and they hadn't realised!

              Complaint soon after / threatening to go to Sky Broadband (would work out cheaper but I would worry that the customer service would be VirginMedia-like) and they reinstated the previous charge for the original broadband.

  8. Anonymous Coward
    Anonymous Coward

    Post Office have started a TalkTalk migration [oh, the horror!] which I understand is rolling so some customers may have been migrated already.

    This won't have had anything to do with Openreach either - all they do is manage and deploy field infrastructure [fibre ducts, cabinets, copper cabling etc.] and connecting exchange kit and premises lines, they don't control or manage BT's actual core data centre kit or networks such as routers.

  9. xeroks

    Plusnet are owned by BT?

    @ac

    I've been with Plusnet long before they were bought over. I have been surprised - and pleased - that their customer service has remained as good as it always was.

    Also their product still has all the stuff BT users don't care about, like fixed IP addresses, php webspace etc.

    Having said that, my wife is at home without no internets this morning.

    1. Anonymous Coward
      Anonymous Coward

      Re: Plusnet are owned by BT?

      "Having said that, my wife is at home without no internets this morning."

      Well, as long as snow doesn't stop the milkman ;)

  10. Anonymous Coward
    Anonymous Coward

    Plusnet are indeed owned by BT

    http://www.theregister.co.uk/2006/11/16/bt_bids_plusnet/

    One of the things it has enabled BT to do is continue to scam the halfawake regulators at Ofcon.

    The majority of the public, like your good self, is unaware that Plusnet is BT Sheffield.

    So, if someone looking for a provider has decided that BT are too overpriced, or their overseas customer service is intolerable, or whatever, they can look elsewhere, and quite possibly find that Plusnet appears to offer tolerable service at quite low prices (certainly a lot less than Big BT).

    In particular, boutique ISPs from AAISP to Zen look massively expensive in comparison with Plusnet/BT Sheffield, although the service isn't really on the same planet either.

    So although Big BT aren't supposed to use predatory pricing against their competition, BT Sheffield apparently still can, as far as the regulators are concerned.

    Marvellous.

  11. Neill Mitchell

    How did they tell their customers?

    Via their website? Ummm.

    I just had this with BT. My Infinity went down and I was told on the phone that I could report the fault more quickly via their website...

    1. Tom Wood

      Re: How did they tell their customers?

      Mobile broadband?

    2. Peter Gathercole Silver badge
      Facepalm

      Re: How did they tell their customers?

      I'm sure that somewhere in the package documentation you got there was an alternative dial-up service that you can use when ADSL is not working.

      What do you mean! You no longer have a V90 modem?

  12. dak

    We were off

    We were off from about 9:30 to 11:30 but it's working again now in South Lanarkshire.

    This follows another outage on 14th November that took out much of Central Scotland because of a fault in an exchange at Irvine.

    Whatever happened about the Internet routing around damage?

  13. Anonymous Coward
    Anonymous Coward

    "Whatever happened about the Internet routing around damage?"

    We're not talking about "the Internet" here, strictly speaking.

    The failing connectivinty in question is provided by BTwholesale's 21st Century Network. It doesn't seem to understand the concept of "resilience" very well, does it. At least one of the ISPs who depend on BTw for connectivity to their broadband customers has even publically accused 21CN's design of "being unfit for purpose". (Yes that would be AAISP)

    1. dak
      Thumb Up

      Re: "Whatever happened about the Internet routing around damage?"

      Thanks for that.

      And I'm definitely going to find a way to use the word "connectivinty ".

  14. Anonymous Coward
    Anonymous Coward

    Thank *insert your favorite deity here* for 3G.

    My phone started ringing off the hook at just after 9am. At least with a 3G dongle I was able to get to the BT status page and find out just enough to keep the heat off my back.

    1. Anonymous Coward
      Anonymous Coward

      Re: Thank *insert your favorite deity here* for 3G.

      "At least with a 3G dongle I was able to get to the BT status page "

      unless you're on Vodafiddle trying to access 3g data services from Portsmouth island, which hasn't worked reliably for the last 3 months!

      Looks like the punters are going to have to organise their own network reliability plans, as the carriers don't seem capable.

  15. William Boyle
    FAIL

    How not to do large-scale safety-critical systems

    Just one router did this? What? No hot fail-over? This is the danger of a mono-culture in ubiquitous tech. A single point of failure will always fail... I build tools to monitor very large scale network systems, and our gear can continue operating at full load after 2 failures at any point in the system - hardware or software. The worker drones at BT aren't at fault here - they are doing their job to the best of their considerable abilities. It is management that should be taken out to the woodshed because it is they who likely said that redundant routers, load-balancers, or whatever would cost too much...

    1. TomS_
      Stop

      Re: How not to do large-scale safety-critical systems

      Or, how about: we're selling these services dirt cheap to residential Internet users, with no SLA, we dont need to offer a fully resilient network because thats not conducive to the low cost product that the masses want.

      But... if you're a business or enterprise customer and want to pay a bit more for a different kind of service, one that includes an SLA, then sure, we'll provision it in a redundant way such that maybe, and only maybe, the most catastrophic of events will cause an outage.

      People seem to get all uppity when they buy something, agree to the terms under which it will be delivered, and then it fails and they wonder why.

      Sure its not the worker drones fault, but I say its not managements fault either. Its the consumers fault for wanting to pay jack squat and expect something gold plated. If you want uninterrupted Internet access, youve got to pay more than a couple of 10'ers a month...

  16. Jabba
    Happy

    Our Zen fibre went down, I re-routed us via our Demon ADSL backup... did the trick although twas a wee bit slower....

  17. xeroks

    Maintenance scheduled for this morning

    Don't know if it's related or not, but Plusnet were "performing some software upgrades on one of our RADIUS authentication servers." this morning, according to their customer service site. It was expected to be a 5 minute task.

    Given that other ISPs were affected, I'd guess it's unlikely to have been the cause of today's problem.

    They post warnings about future work like this all the time, and rarely are there any probs.

  18. skuba*steve
    FAIL

    Single Point of Failure

    Seemingly (according to our discussions with the 20CN support desk this morning) BTO had installed a line card at the Edinburgh Exchange which then took down every card in the chassis; it _should_ have failed over to a juniper device, but didn't, thus resulting in the outage we experienced.

    We lost circuits across the NE, NI and Scotland - SPOF fail!

    1. GreyWolf
      Devil

      Re: Single Point of Failure

      And the supplier of said card? By any chance, was it the preferred supplier to the 21CN project that could not deliver what they had promised during the tender process? A supplier from a very large country in the distant not-West?

      1. skuba*steve

        Re: Single Point of Failure

        BT had rasied a TAC case with Cisco, so I'me guessing it's cisco gear; it was affecting traffic on 20 and 21CN, across ipstream connect and WBC - we're guessing something in the distribution layer, but no RFO from BT as yet.

    2. TomS_

      Re: Single Point of Failure

      This sounds like a recent bug that existed in Juniper T series routers, whereby installing an FPC would cause an upset somewhere in the system (I dont recall exactly where), and cause the entire box to stop forwarding traffic.

      The company I work for had to (until recently when we got an updated JunOS that fixes the issue) entirely remove all traffic (gracefully) from a router before installing an FPC in case it went south and caused an ungraceful interruption to the network.

      Assuming this is the same kind of problem, this should have allowed failover to occur had that been set up correctly, however. Things like routing protocols would drop adjacencies and simply route around the faulty router.

  19. phuzz Silver badge
    WTF?

    BT-ception

    A couple of the shops that we look after dropped offline, so I took to the phone to find out what had happened.

    Knowing that we used BT Enterprise, I rang BT.

    However, I was told that although we clearly had an account with BT, for some reason the DSL account belonged to plusnet, so they couldn't help.

    Fearing some dodgy business I rang up plusnet, only to be told, that actually, plusnet resell their own service to other ISPs, including....BT Enterprise.

    I was about to ring Enterprise when it all started working again

    If anyone can even explain wtf is going on with this I'd appreciate it, right now it sounds like the telecoms version of Inception.

    1. Anonymous Coward
      Anonymous Coward

      Re: BT-ception

      Did you read the stuff here about Plusnet being part of BT these days? Did the Plusnet staff not tell you that too (maybe it didn't sink in)? Did your BT Enterprise/Workplace logon at http://workplace.btenterprise.com/main.html not shed any light on the matter?

      Workplace was Plusnet's mostly-automated system for running a retail ISP in as close as they could get to zero-touch style, e.g. little need for useless and counterproductive script jockeys in call centres out East. At the time BT bought Plusnet, Workplace was said to be an important factor in the decision.

  20. pullenuk

    dont diss

    Plusnet

    My neighbour running bt infinity and I'm on plusnet version. Mine is always faster and better on ping. Even called plusnet at 11.30PM to sort a billing issue, sorted in 5 mins.

    May be owned by bt, doesn't run like them

This topic is closed for new posts.

Other stories you might like