back to article Cause of Parliamentary downtime on Microsoft Office 364½ revealed

A major Microsoft Office 365 outage in the Houses of Parliament that left up to 3,000 users without email was due to Microsoft failing to inform itself about a technical change, The Register can reveal. The outage occurred on 23 June and resulted in a total of 13 hours of downtime, the Parliamentary Digital Service said in …

  1. TheTick
    Facepalm

    I'm a raging libertarian capitalist, but even I see the problems of having all the emails of our Parliamentarians on the servers of a foreign company, no matter how much they say they will never, ever take a little peek...

    Why on earth hasn't the government set up an IT division to provide public sector organisations, including Parliament, with an internally managed email solution/file services etc? An NHSMail for government.

    1. Anonymous Coward
      Anonymous Coward

      Leading by example?

      Presumably the "5eyes" would be jolly pleased if some of their "allies" or, even awesomer, other governments are hapless enough to follow suit.

    2. Uberseehandel

      You can host Exchange 365 on a private box (real or virtual or cloudy)

      You need to get out and about more

      1. Hans 1

        >You can host Exchange 365 on a private box (real or virtual or cloudy)

        1. you forgot the fanboy icon

        2. this was MS-administered kit (presumably in the cloud), hence the outage and outrage

      2. Anonymous Coward
        Anonymous Coward

        ou can host Exchange 365 on a private box (real or virtual or cloudy)

        You need to get out and about more

        If you would have left out the personal insults and had bothered to spend some more time doing tech research you would have discovered that parliamentary email even goes through TWO third parties - they use MessageLabs filtering. Let me translate that for you: the US has two separate opportunities to get a copy of ever incoming and outgoing email.

        1. TheVogon

          "the US has two separate opportunities to get a copy of ever incoming and outgoing email."

          It's at least 3 goes. Don't forget we also give them a direct slurp feed of all our telecoms and satellite links.

          Probably our secret services want them to have a copy anyway so that they can "keep an eye" on things by proxy without breaking any British laws...Much like the Americans do with their citizens via GCHQ.

        2. Benchops
          Black Helicopters

          > the US has two separate opportunities to get a copy of every incoming and outgoing email

          Don't you grieve --

          they're on our side, I believe.

      3. TheVogon

        "You can host Exchange 365 on a private box (real or virtual or cloudy)"

        There is no such thing as "Exchange 365".

        Office 365 which we are talking about here does not have a private hosting option.

        1. Uberseehandel

          A private cloud hosting environment for Office 365 is entirely possible - that is an Exchange server on a private virtual server.Who looks after the machines the servers runs on is a separate matter. An organisation like ATOS can run the servers, but they are still private to parliament.

    3. NeverMindTheBullocks

      @TheTick

      There is such a service in place.

      FCO Services ( the services division of the Foreign Office) provides and supports a formally Accredited application services platform that includes Office 265, delivered via the PSN and Internet specifically for use with material that may need delivering to out of the way places and embassies in countries that may be somewhat less than respectful of our National Security. Or simply for handling more sensitive material at home.

      1. Anonymous Coward
        Anonymous Coward

        Re: @TheTick

        a formally Accredited application services platform

        Accredited by or for what? I assume that's one of the users of xGSi?

      2. Anonymous Coward
        Facepalm

        Re: @TheTick

        FCO Services want 10 times the price per mailbox of MSFT though...

      3. Trigonoceps occipitalis

        Re: @TheTick

        From the new, truthful marketing department:

        "Office 265"

    4. RISC OS

      my guess...

      ...some influential MP somewhere has a second job working for MS... or his son-in-law or daughter does.

      1. Anonymous Coward
        Anonymous Coward

        Re: my guess...

        my guess...

        ...some influential MP somewhere has a second job working for MS... or his son-in-law or daughter does.

        Oh, it is far less subtle than that, and it happened during the Blair government. What is interesting is that much of that has been scrubbed off the Internet. It's so impressively done that it suggests that a lot of money was involved.

        1. Anonymous Coward
          Anonymous Coward

          Re: my guess...

          "far less subtle than that, and it happened during the Blair government. "

          Oooh you little tease.

          Give us a few hints (or if it's already public, some search keywords).

          Then maybe it can be added as an Appendix if/when the Chilcot report comes out.

          1. Anonymous Coward
            Anonymous Coward

            Re: my guess...

            Oooh you little tease.

            Give us a few hints (or if it's already public, some search keywords).

            Maybe he or she doesn't want end up as Dr Kelly?

            1. An ominous cow heard

              Re: my guess...

              "Maybe he or she doesn't want end up as Dr Kelly?"

              Fair comment.

              Plus for all we know the hints may have been "submitted and awaiting moderation since [forever]" or the original post may have been removed without trace or whatever. It's the Interweb, it works in mysterious ways.

    5. Hans Neeson-Bumpsadese Silver badge

      "Why on earth hasn't the government set up an IT division to provide public sector organisations, including Parliament, with an internally managed email solution/file services etc?"

      Like this? http://www.theregister.co.uk/2015/08/03/uk_justice_email_insecure_protocols/

    6. Alan Brown Silver badge

      "Why on earth hasn't the government set up an IT division to provide public sector organisations,"

      Because such organisations get takenover by mindless jobsworths (or worse) before they even rollout their first connection, resulting in costs being 10 times higher than if done by a competitive company.

      Nice idea, if you can keep the trough-wallowers out, but the reality is that such outfits end up full of jobs for the boys. Government/civil service corruption is alive and well in the UK, just disguised a little better than in "some other countries"

      1. Anonymous Coward
        Anonymous Coward

        Because such organisations get takenover by mindless jobsworths (or worse) before they even rollout their first connection, resulting in costs being 10 times higher than if done by a competitive company.

        Nowadays, yes, but once upon the time there was the CCTA, and they had some seriously good people. At some point, Slashdot had an interview with "the queens webmaster", and that was remarkably sane (also in context of some of the questions). I doubt you would get such clear and competent answers now.

      2. Anonymous Coward
        Anonymous Coward

        If my experiences tromping the globe are anything to go by, corruption everywhere is about the same. It's only the perception that differs world-wide. The mechanisms vary as to why there is a perceived difference.

    7. Smoking Gun

      Office 365 has recently been cleared for NHS too.

      1. Anonymous Coward
        Anonymous Coward

        >Office 365 has recently been cleared for NHS too.

        Shit. So now medical care is formally on the list of mechanisms the government uses to spy on us. Via it's reciprocal (yeah, as if!) arrangement with the Yanks.

        Is there anything they don't use?

  2. Robert E A Harvey

    QOTD

    "it wasn't working quite correctly"

    brilliant!

    1. Anonymous Coward
      Anonymous Coward

      Re: QOTD

      QOTD

      "it wasn't working quite correctly"

      brilliant!

      Yes, I'm in total awe of such brilliant, deeply deductive powers. I got immediate flashbacks to Benedict Cumberbatch playing Sherlock here

      /dripping sarcasm

  3. Herby

    Welcome to the cloud...

    Sorry we run it and it might be down when you really want to use it.

    You see, "the cloud" is just an euphemism for "someone elses computer that you have no control over". If I were in charge of Parliament (I'm not the sovereign, sorry) I'd have the servers under direct control. Of course if you are the "queen of america" (or believe you are) you DO have your own server and control all of its actions, which makes it easier to erase "bad things" when you want to.

    1. Anonymous Coward
      Anonymous Coward

      Re: Welcome to the cloud...

      What's any of this got to do with Annie Lennox?

      1. Fred Flintstone Gold badge

        Re: Welcome to the cloud...

        What's any of this got to do with Annie Lennox?

        I think it's more Nine Inch Nails (yes, I know it's originally by David Bowie, I just like this version better :) ).

  4. Anonymous Coward
    Anonymous Coward

    A good start!

    Now for a stretch goal: render Parliament incommunicado for the rest of the year...

    Joking aside, the loss of emails is a bad sign. That suggests that they had to roll back the data as well, rather than just sorting out a config/deployment mess. In the glorious cloudy world of tomorrow that ain't meant to happen

    1. RISC OS

      Re: A good start!

      yeah... they were "lost" (today would be a good day to delete bad emails)

  5. RISC OS

    Why MS...

    ...surely it would be better to host our data on the NSA cloud directly rather than via MS?

    1. Frank N. Stein

      Re: Why MS...

      Isn't MS the NSA Data Cloud?

  6. ratfox
    Happy

    Nice title

    It got a laugh out of me

    1. Peter Gathercole Silver badge

      Re: Nice title

      Maybe El Reg should keep a running downtime count for the Government use of MS Office Online, and use the running total every time they have to carry a story like this!

  7. Steve Davies 3 Silver badge
    Big Brother

    It is called Lookout for a very good reason

    Perhaps a few more people in HMG might like to be told that.

  8. Hans 1
    Facepalm

    Office 364½ ?

    It launched in May, fell over in June ... why, oh WHY do they go with the redmondRetards™?

    1. Anonymous Coward
      Anonymous Coward

      why, oh WHY do they go with the redmondRetards™

      Answer was provided higher up in the comments. That has had real consequences for years.

  9. Anonymous Coward
    Anonymous Coward

    Change Management

    It's not rocket science to have an automated notification system that informs all relevant parties when changes are made. You could also have a simple web page on a screen that scrapes a table in a db, to inform the email blinded.

  10. BenBell

    Oh the Irony

    one part of an entity not talking to another.. resulting in a large cock-up... you couldn't write it any better,

  11. Anonymous Coward
    Anonymous Coward

    It can never fail...

    One day at work I had a frustrated manager come up to me and ask for help on a system I don't support, scheduall.

    Nothing I could do, but he didn't like it when I laughed at his comment of:

    "But we were told it could never fail"

    Managers, where would we be without them!

  12. Tezfair
    FAIL

    Nice to see our Gov supporting UK businesses.

  13. swampdog
    WTF?

    They wouldn't even need the cloud if the muppets stopped storing BLOB's in the databases. Store links.

    Massive PDF scans. Attach to email. Copy in dog. All slightly different but massive.

  14. P. Lee

    As much as I'd like to slate MS for this

    I'm probably bothered less by the downtime than other factors.

    Privacy probably isn't an issue - I assume private infrastructure/servers and the MessageLabs stuff will only be external which would be going over SMTP on the interwebs anyway.

    However, I'm always a bit concerned at over-centralisation. Over centralisation with a foreign company having tentacles into your infrastructure seems like a bad plan. Really, how hard can it be to run a mail server? Just because you can put everything on a single service, should you? Could you not at least make a show of supporting some British industry - even if all you did was pay them to bring some FLOSS solution up to certified government standards? A dual-vendor policy might be a good idea when MS come knocking suggesting you install W10 & Cortana...

  15. Dominion
    Facepalm

    ITIL?

    Was the change managed using ITL? LOL!

  16. Alan Brown Silver badge

    Costs

    "The Register understands the transition slashed costs by hundreds of thousands of pounds, halving monthly user fees from roughly £25 per month to £12 per month."

    This says a lot more about the raging inefficiencies within Parliament than it does about Office 365 (especially since Google and MS's big selling point about these services is "It's FREE!")

    The security ramifications of all parliament's mail being on servers in Ireland don't bear thinking about - and similar experience with outsourcing this shit shows that what you may save in upfront costs you lose in added support costs.

    The only valid reason to outsource this kind of thing is in order to declare the existing ISD management structure redundant and start over with competent bods, then pull it back in as rapidly as possible.

    FWIW, now that Goo's dumped gmail for ISPs, I expect to see gmail for businesses go soon afterwards - and the only reason that MS even offers this service is because Google does. Just remember that if the service is free then you're the product.

  17. Ted Treen

    But surely...

    ...anything which stops or impedes the denizens of The Palace of Westminster from communicating with each other and furthering their little schemes is a good thing for the rest of us.

  18. StuHarg

    Email Continuity Services

    You would think it would be mandatory for all Gov organisations to have an Email Continuity service in place like the one www.spambrella.com provide. Single point of failure using Office365 isn't new it has been happening across the USA throughout June and July.

    1. Mike007 Bronze badge

      Re: Email Continuity Services

      Their "backup MX" service won't help if the primary provider is still receiving email but then deletes it from their infrastructure. A backup MX is also not needed for a 13 hour outage, as the origin mail servers should spool the messages for several days before considering them undeliverable and notifying the sender to try again.

  19. Anonymous Coward
    Anonymous Coward

    Why the FOI surely if we know parliament are using orifice 365 then when there is an outage they won't be able to email their expenses etc...?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like