Testing and rollout

I'm sure it was tested, but possibly not load tested enough, or perhaps there were problems with the production architecture and the test enviroment did not fully model. It looks like this was a hardware upgrade as well as a software upgrade.

During the development of IF there was extensive performance testing done. They have a stateless server architecture with 8 or more application servers running independently (you can read about it in a book called Enterprise SOA if you are interested, there's a case study on IF).

It is possible to upgrade one server independently of the others, but only if network or database schema changes are not involved. During a software upgrade is is almost always possible to roll back the upgrade, hardware upgrades do not always have that option. I bet there are some very tired people in Edinburgh now. Probably in meetings that they'd rather not be in too.

IF's track record has been very good until now though.