NHS trusts and other customers of IT giant 2e2 are stuck between a "rock and a hard place" after their worst nightmares of cloud computing came true. Hospital bosses and private businesses relied on the integrator to operate their outsourced cloud systems - but 2e2's administrators demanded extra cash last week to keep those …
Itz de cloud
Those IT managers who committed themselves to this service - no doubt hoping that bonuses would come their way based on cost savings - and who had no business continuity plan in place deserve everything they get. And that includes the public services numpties who have been pushed this way by those pratts in the cabinet office.
Re: Itz de cloud
An addendum. People seem to have a bizarre understanding of SLAs and the underlying impact of a failure of their service provider to meet KPIs. Fundamentally if your business really relies on a service you need a guaranteed (externally - not just from your provider, even if they be BIG) ability to either get the service back or recoup the losses. Few organisations actually understand their costs or risk in this area. Probably because most of their managers are financially illiterate - and that really includes their accountants.
I'll read this when I get home - I should have stopped laughing by then.
Hi Lee and all
I know you work in education so you will like this one.
I used to have to help with IT purchase requests in a fairly large FE college a decade or so ago. Eportfolios were on the radar. Got some suppliers in. Demos, nice draggy droppy. Asked each supplier 'if you go bankrupt, how do we get our students' work back'?
I was really surprised at the reaction. 3 out of 5 said 'noone has asked us that before' and looked worried. I kid you not. One said 'we will send you a copy on a hard drive' and the other one said that they had funds in an escrow account to provide data migration.
We went with a spatch-cocked internal approach which actually worked ok. Our liability. It was about 40Gb of data max.
My bursar and I were discussing this years ago. It's the sort of thing that occurred to him just because of the different businesses he has to deal with. Occasionally, someone DOES go bankrupt, and the clients are left to pick up the pieces. As soon as any mention of putting necessary services (i.e. not just "run our website", which we can hand off to anyone within a day or two if there's a problem, and even have "backup" domains for, nor "provide online content" - but actually real services like thin-clients, running our library, etc.), then the question quickly arises about what happens if they go bankrupt.
Hell, I get staff who complain when BBC Schools change the content they have on their website, like I'm personally responsible for them phasing out 10-year-old Realplayer streams that the teachers have had in their lesson plans since before I started and not providing a replacement. They honestly expect me to "do something about it" (I don't know what, because all I can do is send a sternly-worded letter to the BBC, same as them - the way they talk you'd think I could just claw back years of archived copyrighted video material without the BBC being involved at all - hell, even Archive.org doesn't mirror RP streams).
So when we put something in the hands of an external business, we have to basically assume that one day they won't be there. It's not a problem - companies come and go - but we just plan what to do in that case, but we *do* have to plan.
My bursar in fact just read this post of yours, and we weren't surprised at all. The correct answers are obviously the last two - escrow accounts and DIY. Like RAID, backups, and everything else, you should never assume that something WON'T go wrong - you should just plan for it going wrong and finding out how you would cope with that. You can't stop things going wrong, but you can stop them being big problems if they do.
Whenever I see the word "cloud"
I mentally replace it with "clown".
Before everyone jumps in saying its a disgrace.......
Its costing £300k per day to keep what is left of the company alive. And there's no money in the pot.
All of the data in those datacentres is valuable, and important - so this is the only way to keep available.
The alternative was to turn the power off, and flog the kit to the bailiffs.
was not to leave your very precious data in the hands of a set of (incompetent) pirates
The only disgrace is those companies who don't have a Business Continuity Plan that has their data and services available from an alternate location in a reasonable time frame.
Shit happens. Plan for it. And while you can't plan for the unknown, you should have a plan for loss of a site.
To all those CIO's who put their eggs in one basket? Congratulations, your bonus this year is an omelette.
based on observations of senior management in many private and public sector roles
£300k/day? That's (on workdays alone) £60M/year, which is IMHO suspiciously large..
"The alternative was to turn the power off, and flog the kit to the bailiffs."
Yeah, man, suddenly that data centre becomes ooooh so valuable and important! Just a few months ago everything had to be done cheaper, with even cheaper contractors, on even cheaper support contracts.
I'm not having a rant at 2e2 here, it's this @$%£&#! attitude that a data centre or a eCommerce website is just a bunch of machines hanging off a bunch of cables, which can be maintained by ever cheaper support staff...by wood worms, if necessary.
And the bailiffs, what will they look at? The hardware, of course - like, EMC SAN with all kinds of unencrypted data still on it for a grand. Who cares that some customer has no eCommerce website anymore....
Im sure 75% of that cost will be going to the administrators.
In another article is was stated that the majority of the kit was leased from HP... so nothing to flog to the bailiffs there.
I think a lot of BCP plans consider data from an alternate location. What they don't consider is from alternate supplier.
Take the average retailer, I'm fairly certain most do not pay for 2x web hosts, 2x logistics suppliers, 2x fulfillment providers etc. None of which could easily be switched within days to an alternate provider if their main provider got in such mess that an administrator shut everything down within 48 hrs.
Outsourcing nightmare - not cloud
Cloud has nothing to do with it. Failing to understand and plan for the risks of a supplier's business falling over, by the people responsible for the decision to outsource in the first place (not to mention the people running the failed supplier) has everything to do with it. This kind of cloudwashing is as heinous as vendors cloudwashing their services. Very sloppy.
Re: Outsourcing nightmare - not cloud
Precisely....surprising to see how many fail to really understand the nature of services and contingency + continuity, failing to get the right terms of KPI's in place and picking dodgy suppliers. Insolvency of clients or suppliers has been there since the year dot...whether products or services.
And this is why cloud services are stupid.
One word - Dops.
are things that generally rain all over you.
Both in the real and virtual world.
Your cheque is in the post...
Many companies would be unable to pay this amount, in less than three months.
The Accounts Office would first need a business case, three competitive quotes, a review by the budget committee, proforma invoice, two signatures, a purchase order number...
That's assuming they even understand the urgency of the situation.
It's heartening to read
a lot of commentards get the concept of business continuity, and due diligence. I have zero sympathy for any company that suffers as a result of this, and hope the incompetent managers who caused that suffering are ejected at very high velocity. If any publicly funded organisation is affected then there should be sackings. We're constantly being told why the NHS needs such highly paid managers - I would argue it's to avoid shit like this, not walk straight into it.
If you outsource a critical part of your business operations, then you need to plan for the day it isn't there. Simples. And that process should have been part of the initial outsource, not a sticking plaster over a bad deal. If, as part of your outsourcing you discover you can't plan for such a day (say a monopoly supplier) then you really need to ask yourself if you should be outsourcing that process at all.
Having seen a proper due diligence in action, I know that it's not uncommon for big clients to request bank statements, audited accounts, historic headcounts, and a whole load of minutia before they consider spending a single penny with a supplier.
Personally (and I'm not even a business continuity specialists, I just know a few). My first question to 2e2 would have involved escrow for the hosting, so that even if 2e2 went down, their datacentres would have provision to overrun until clients had recovered their data. But then I didn't go to the right public school.
I really wish the mainstream media would show an interest in this. I'd love for one of the tv companies to do a What If type of documentary of someone like Capita/Accenture collapsing, who have their fingers in way more pies than 2e2 managed to get involved in.
Re: What if?
Don't worry, they'll be under greater scrutiny now than ever before....
Re: What if?
Everything would suddenly become more efficient!
Good on AWE for telling the Admins to stuff their charges....but at the same time a shame as it would have been interesting t see the mexican stand off otherwise - calculator wielding Administrator stood by the power switch versus AWE, with Nuclear weapons...there must be other customers in the health care and blue light sector who can stand their ground...and as far as the £250k staff costs, that must be pre-slash and burn..?
no one has mentioned...
The NHS IT service is probably one of the most complex you can work in due to the hundreds of different clinical systems in use.
One thing that no one has mentioned is where clinical systems are hosted by suppliers, what protection is there then, you don't have the option of hosting jointly eleswhere.
This whole fiasco needs to be in the national press as a huge wakeup call for everyone.
Whilst some criticise the NHS IT managers for this, a lot of decisions to outsource were made by financial directors as a cost cutting excercise, who paid hundreds of thousands to consultancy companies to recommend this strategy, just Google Sussex Partnership and 2E2 to see who the culprits are.
"Healthcare specialists xxxxxxx Consulting supported the Trust throughout this service transition including complex multi-agency stakeholder engagement, requirements specification, the OJEU procurement process and the subsequent commercial negotiation and transition of services to 2e2"
And yet this company is still bragging about the fact on their website trying to sucker other NHS organisations into the same disasterous decisions
Re: no one has mentioned...
Ahh here's an interesting read...
Hind sight is a wonderful thing.
Re: no one has mentioned...
It would appear that most of these cash strapped NHS Trusts were ill advised by the so called "Consultants" who at this moment is still advertising "how good they are" by means of publishing a list of all the NHS Trusts' logos, giving the impression that they are NHS friendly, which I don't doubt, as they say the proof is in the pudding. The pudding tastes like ***t.
I'm surprised, that NHS hadn't ask for their logos to be taken down. This is a clear case of "hit and run" like so many things these days. Directors who took on a post only for the good times, stitched the org up, and move on after 1-2 years of so called contracting/consultancy job, disappeared to another org with an improved CV that says "I have done this great outsourcing deal or transformation" but seems to have been rather economic on the truth and have now left the org only to have some other poor sods to pick up the pieces....hmmmm what example could I possible cite?
Let's see , how about Sussex Partnership Trust. They placed their Trust on contractors who's only there for 1 reason (unpopular this may be, we all know the truth don't we). I'm surprised no one had sought comments from the IT director and consultancy firm who had "organised" the deal. It's time to put an end to all these "hit and run".
You only have to look at the issues when the Detroit Police Department outsourced its operation to OCP, to see its not a good idea...
So extortion is legal if you're skint?
You heard it here first. Spend it if you've got it.