my money would be on:
:Goldman Sachs :~{
Routine WHAT NOW? Bank of England’s CHAPS payment system goes TITSUP
The CHAPS system used to handle most bank-to-bank transfers in the UK has collapsed, with all mortgage payments currently on pause. We were told that the problem was due to an update over the weekend, when the details of new CHAPS users were being added and old ones removed. This is meant to be a routine operation, but a …
-
-
Monday 20th October 2014 12:45 GMT We're with Steve
9/1 - New name not added to the legacy "Tea Rota Application Package" system (running on legacy Tandem) that no one thought was used anymore but some Charlie was using to do a lookup for short company name on as it was easier than filling in a change request to use "Core Names Thingy 2.1" and senior management had promised some Treasury wonk it would be live by the end of the week (we will fix it after go live, right Steve?).
-
Monday 20th October 2014 13:52 GMT Danny 14
found it
#*****************************************************************************************
# don't forget to comment out the following before the next update!
# Not fixed CHAPS import fully yet.
#*****************************************************************************************
-
-
-
Monday 20th October 2014 13:18 GMT fixit_f
Not a huge issue
Sounds like all payments will be made today and they're extending the late clearing cutoff for delayed payments
http://www.chapsco.co.uk/media/press_releases/-/page/3143/
Just means that estate agents and conveyancers might have to work a bit later so those house keys can still get handed over!
-
Monday 20th October 2014 13:35 GMT Anonymous Coward
Coincidence?
Barclays Bank had an online banking login notice on Sunday evening saying they were doing maintenance. What should have been an instant bill payment elicited a warning that it was accepted - but that the actual transfer would be delayed "For security checks" until Tuesday.
Coincidence?? or just a false sensitivity to a small transfer on Saturday to prove the destination account - followed by a larger transfer on Sunday?
-
-
-
Monday 20th October 2014 21:16 GMT GBE
Re: Steady on, CHAPS!
<pedant>
quote: an American might say that CHAPS in its current state was "about as much use as the tits on a boar hog".
I've heard that many times, but never with the word "hog" on the end
In common parlance, "boar" means adult male hog, so the "hog" is redundant and makes the listener think you might think there's such a thing as a "boar sheep" or "boar cow".
In less common usage, "boar" means adult male <certain other mammals>, in which case tits are just as useless.
</pedant>
-
-
-
-
Monday 20th October 2014 21:12 GMT Anonymous Coward
Was it still on NonStop? Something stopped.
Like many similar systems, BoE RTGS was on NonStop (Tandem/CPQ/HP) (until at least 2012). A Tandem usergroup whitepaper says so, as does a public profile on a well known Facebook-for-businesspeople of a NonStop person with RTGS experience who left BoE a year or two ago.
Who can tell me what the system now runs on, in which country it is hosted, and who wrote and approved the DR plan?
-
-
-
Tuesday 21st October 2014 12:00 GMT Anonymous Coward
Re: Legacy system ?
"It almost sounds like the IT bod Chap who actually knew what he was doing left years ago.
left was off-shored a few years ago."
That's why I'm hoping someone can tell us where the system now is.
Courtesy of public data on Facebook-for-professionals (no account needed), I apparently know the name of the man in charge of the infrastructure till a couple of years ago, and I also apparently know where he now works (still working with NonStop - it's a relatively small world).
Among the many things I don't know is why he left, or what the sucession plan was.
Some of these details may emerge in the inquiry.
-
-
-
Tuesday 21st October 2014 09:07 GMT Anonymous Coward
CHAPS
the only time I've ever had anything to do with CHAPS was a few years ago after I put a few grand inheritance into my Barclays account after my old man had died. Put it in to the current account and then wanted to move it on to a saving's account in another bank. Anyway paid the cash in then couldn't move it out as Barclays had put a limit on the amount of money that you could move out of the account and it was quite a small limit, grrrrrrrrrrrrrrrrr Trip in to branch to be told the only way to move it was via CHAPS which of course they wanted something like £30! ROBBING TW&TS
-
Tuesday 21st October 2014 12:54 GMT Anonymous Coward
Why no love for NonStop.. :)
this or part of this is still on NonStop (good job IMHO) but not a Non-Stop/Himalaya those are a little old now and no longer supported by HP.
CHAPS runs on a NonStop system which IIRC is smaller than the test systems used by say VocaLink or the Banks. This is ok though CHAPS only does like 200K transactions per day VocaLink do 12-14m (real time) transactions per day.
Nothing at all wrong with NonStop.. it has lowest TCO for this kind of environment. This outage wasn't an infrastructure failure it was a failure to implement change.
You can be Active Active on the most reliable hardware available on the planet (they are) but if you F**K up an application or configuration change it all goes down.
Should have been trivial to roll back though.. looks like it wasn't
Common causes of system failure #1 Operator, #2 Software Change
Thats like 90% of failures taken care of right there.. then comes power failure, network failure and infrastructure failures.
Seems odd however that a systems that only has to be available BoE office hours went down.. FPS is 24/7/365 and still manages 100% availabilty (core systems) given this bonus they should have been able to veryify the change prior to the opening bell.
Cheers (Anonymous Availability Nerd..)
-
Tuesday 21st October 2014 17:56 GMT Anonymous Coward
Re: Why no love for NonStop.. :)
"CHAPS runs on a NonStop system"
That's as maybe, but isn't RTGS (the system that failed) distinct from CHAPS?
Other than that, yes, this all sounds a bit unfortunate; properly designed and managed NonStop setups (especially office-hours-only ones) shouldn't be falling over like this due to (apparently) lack of proper testing.
Bedside reading for availability fans (this month including a 6page summary of, and a link to a free copy of, NonStop for Dummies):
http://www.availabilitydigest.com/
-
Wednesday 22nd October 2014 15:07 GMT NonStopBoy
Re: Why no love for NonStop.. :)
Good point I was unclear..
I am pretty sure RTGS is also run on NonStop.. I have had discussions with programmers of this systems at NonStop events in recent times.
Good link to the Availability Digest, Dr Bill is a good friend of mine and his site is always very informative.. I especially Like the Active Active 60's style.. write up he did for NASA historical work. Brilliant.
-
-
Biting the hand that feeds IT
