back to article Embattled 123-reg flings six months' free hosting at angry customers

Customers still unable to access their websites following a mega cock-up at hosting site 123-reg over the weekend have been offered six months' free VPS and backup recovery services as a sweetener. The biz 'fessed up to customers this week that a script containing a catastrophic error which was run on Saturday (16 April) took …

  1. This post has been deleted by its author

    1. g e

      "doing everything in our power "

      Except getting the chequebook out for compo

      They should at least refund the previous year to those customers at the bare miminum.

    2. msknight
      Pint

      Re: And second prize is...

      You owe me a new keyboard...

      For that to be the first comment was an absolute classic.

    3. Bibbit

      Re: And second prize is...

      Good work. Laughed my socks off and forgot my hangover for a few minutes.

  2. DaddyHoggy

    Isn't six months free hosting by the company that just nuked your website like getting a free meal voucher at a restaurant that nearly killed you with food poisoning...?

  3. Dwarf

    Salt into the wounds

    So, 123 lost the web sites contents - their fault for not testing things properly

    The customers haven't got backups - customers fault for not doing backup in the first place.

    Then the customer get free hosting for 6 months.

    Er, so what will they be hosting then some form of "This customer has not created a site" holding page ?

    I wonder if anyone who has only static files has tried to recover from archives such as "the way back machine" or similar archives. See https://archive.org/web/

    1. AMBxx Silver badge

      Re: Salt into the wounds

      6 months to setup and test your hosting at an alternative. Then 123 will be bought by TalkTalk - looks like a good cultural fit.

    2. Peter X

      Re: Salt into the wounds

      If you were in the sorry situation of having no backups of your own, then way-back-machine or google-cache might be your best option. EXCEPT way-back-machine won't show anything if the robots.txt file is no longer being served (or indeed, if it's set to block bots).

      Anyone know what 123-reg have been serving for sites that have been nuked? It would be interesting to see if they do serve a robots.txt file.

  4. CraPo

    Does MarcoMasala work at 123-reg?

    http://serverfault.com/questions/587102/monday-morning-mistake-sudo-rm-rf-no-preserve-root

    1. bluesxman

      Re: Does MarcoMasala work at 123-reg?

      I assumed it was one and the same when I saw the first reports of this 123 debacle. Curiously prescient. Though Marco came across somewhere between "trollish" and "massively incompetent".

      1. CraPo

        Re: Does MarcoMasala work at 123-reg?

        Turns out it was actually "viral marketing", which is a form of trolling I guess.

  5. TeeCee Gold badge
    Meh

    Shit....

    ...is still shit even when it's free, you know?

    1. wolfetone Silver badge

      Re: Shit....

      But it's free shit.

      1. Rich 11

        Re: Shit....

        It could be freebird shit but I still wouldn't buy it.

  6. localzuk Silver badge

    6 months and then...

    Will they delete all your stuff again after the 6 months are up? If I delete all the data at my job, I would lose my job and be sued...

    1. Geoff May

      Re: 6 months and then...

      You are lucky, my boss has a shotgun ...

  7. Pascal Monett Silver badge

    "this is something we take extremely seriously"

    Yeah, NOW you do.

    But you didn't take it seriously enough BEFORE the brown stuff got widely dispersed, did you ?

    And the irony of it all. 67 out of 115000 servers. A 0.0006% mishap that is positively trashing their reputation. I think other hosting services would do well to sit up and take notice. Operating at minimum financial cost is something that can cause great financial cost down the line.

    1. Rich 11

      Re: "this is something we take extremely seriously"

      Operating at minimum financial cost is something that can cause great financial cost down the line.

      Unfortunately this argument rarely sways beancounters unless something tragic happens to the Finance server in the fortnight before the final budget meeting...

      1. Captain DaFt

        Re: "this is something we take extremely seriously"

        "Unfortunately this argument rarely sways beancounters unless something tragic happens to the Finance server in the fortnight before the final budget meeting..."

        I smell a BOFH script!

    2. AndyS

      Re: "this is something we take extremely seriously"

      Does anyone really believe the 67/115,000 number? Someone suggested that they were cherrypicking numbers - 67 physical machines, out of the 115,000 virtual machines, which would probably be a few percent of their business (how many VMs on a server?).

      1. tin 2

        Re: "this is something we take extremely seriously"

        I for one absolutely do not.

  8. RayzorWire

    Dear 123reg customer, have you tried: http://archive.org/web/

  9. Anonymous Coward
    Anonymous Coward

    My VPS instances are back, so I don't have to restore from my own backups. While that's a relief for me, the data in my DB is essentially read only. I feel sorry for those that have lost data that they can't restore from backup.

  10. Chris Priest

    I'm still waiting to see if they can restore mine, have built a new one temporarily, not had any email offering any compensation yet.

  11. AlexS

    Laugh a minute.

    I know a lot of ex customers who decided to go with these guys.

    I said to them just remember to take your own backups. They said they didn't need to.. Getting phone calls now hahaha!

    (Evin grin)

  12. Chris King
    Mushroom

    Dear 123-reg, that loud BANG! you just heard...

    ...was another bunch of customers passing the sound barrier to get away from you as fast as possible.

    If I had been a customer affected by this cock-up, I wouldn't be wringing my hands and whining about compensation, I'd be looking for someone more competent and moving everything away at the first opportunity.

    Six months of FREE!!! crap is still six months of crap, and probably comes with strings attached. No thank you.

  13. Androgynous Cow Herd

    If you pay peanuts

    You are probably working with monkeys.

    1. Dwarf

      Re: If you pay peanuts

      The librarian rules ook ?

  14. TS15
    FAIL

    30%...

    So, 30% of servers have *now* been restored.

    Um, what about the other 70% ?

    Gone for good? Vague hope of bare metal recovery?

    Best of luck to anyone who's been caught up in this mess.

    1. Wibble
      Facepalm

      Re: 30%...

      They're the 30% of customers who had a DR solution in place?

  15. Mark 85

    So why would anyone in their right mind stay with them? Same for Talk-Talk and others. It seems that some beancounter types only look at what's going out for services and forget the old adage: "Screw me once, shame on you. Screw me twice, shame on me.".

    1. KeithR

      I've hosted (and continue to host) half a dozen sites with them for the thick end of ten years.

      So far, I've had one day's downtime across the lot of 'em.

      That's why I'M staying put.

      Clear enough?

      You lot really don't do "sense of perspective", do you?

  16. jtaylor

    While I understand the anger, I'm not sure it does any good to assign blame for this.

    I am not familiar with 123-reg, but they sound like a low-cost operation. As long as they were transparent about what they sold, and delivered what was sold, they're an honest business. Backups aren't free -- if they were, then every customer would have their own backups anyway, right? Yes, they screwed up and should fix their mistake, but mistakes do happen, and it's not clear that this was more than a very unfortunate mistake.

    Likewise, customers often don't understand the relevant differences between different products. I see this a lot in consulting. Someone can hire a larger firm for a lower hourly rate to do the same work. If that's all you see, the choice is pretty simple. Even if they have a comparison list of the differences in what they're buying, that's usually in some form that makes sense to the seller, not the customer. I don't like to pay extra for "magic beans" either.

    Yes, this sucks. Yes, it makes 123-reg look unreliable. Yes, it makes some of their customers look naïve. Learn how to prevent this in the future. That responsibility falls on all parties, not just 123-reg.

    Oh, and I do plan to use this as an example of what can go wrong when you don't understand a product.

    1. Adam 1

      You could nearly mount that argument if the failure was caused by a tsunami hitting their data centre. The script was run by them for them with no customer benefit. They did it in a prod environment without any fallback plan and without giving notice to their customers. Inadequate precautions were taken. Blame is the right response here.

      1. jtaylor

        Adam, I completely agree that 123-reg could be blamed. I also think that customers could be blamed. My point is that blame doesn't help.

        "The script was run by them for them with no customer benefit." Do you suggest that hosting companies should not do this?

        "in a production environment without any fallback plan" I suspect most companies run scripts in prod. The lack of a sufficient fallback plan was indeed a serious mistake.

        "without giving notice to their customers" Do you suggest that hosting companies should notify their customers any time a script is run that touches their service?

        "inadequate precautions were taken." Hindsight is wonderful. Is your point that in the future, be sure that all precautions are adequate?

        "Blame is the right response here." It's a valid response. I just don't see how it improves things for the future.

        I used to work with a large financial services company that was known for blaming people (and firing them.) It made the staff wonderfully careful, right up the point where things began to break. At that critical point, the clever folks ran like hell, leaving the less gifted people, junior staff, and contractors to solve the problem while dealing with managers on a witch-hunt. It was in nobody's interest to understand the real causes of a problem (either you were safe, or already fired.)

        1. Anonymous Coward
          Anonymous Coward

          I'll address this, as a senior lead in a very large global professional services firm. (Anon for reasons)

          "The script was run by them for them with no customer benefit." Do you suggest that hosting companies should not do this?

          "in a production environment without any fallback plan" I suspect most companies run scripts in prod. The lack of a sufficient fallback plan was indeed a serious mistake.

          "without giving notice to their customers" Do you suggest that hosting companies should notify their customers any time a script is run that touches their service?

          "inadequate precautions were taken." Hindsight is wonderful. Is your point that in the future, be sure that all precautions are adequate?

          These three points are simple, a proper change control process should cover changes within your internal environment and also handle vendor work. We receive multiple updates per day from vendors in changes they are making within their environments that may impact us, be it internal DNS changes, VM maintenance, mailbox maintenance. This is all built into the contractual terms and makes it a hell of a lot easier to unpick things when something does go wrong.

          1. KeithR

            CLEARLY - given that this is the first such catastrophic fuck-up in 123-Reg's history that I'm aware of (and I've been a customer for about ten years) - they've been getting things right so far.

            I don't believe FOR A SECOND that if they were really routinely recklessly fucking about in the live environment that a similar SNAFU wouldn't have happened years ago.

  17. John Tserkezis

    <i."While working on this issue is our current priority, we have also put a series of measures in place to ensure that this does not happen again in the future."</i>

    They sacked the guy who did it?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like