Furious customers tear into 123-reg after firm's mass deletion woes The fall-out continues from hosting service 123-reg's major weekend cockup, which knocked several customers offline – with several telling us the error has effectively deleted their businesses. As already revealed, on Saturday customers' virtual servers vanished after the hosting firm ran a script containing a catastrophic …
Someone has lost their website, you don't have the 50GB back up file to restore their website, but it's alright because you've fixed the cron job so that deleting servers doesn't happen again? And that last bit is meant to restore faith in your company?
OH YOU'RE DIDO HARDING IN DISGUISE!
OH YES YOU ARE!
HARDING IN DISGUISE!
YOU UTTER CLOWN!*
*To be sung along to the tune of "Devil in Disguise", by Mr.Elvis Presley.
This post has been deleted by its author
To run rm -rf with parameters in a script.
First, you test for the parameters to have valid values
if [[ -z "$1" || -z "$2" ]]
then
echo "Variables not set"
else
echo "rm -rfi $1/$2"
fi
as well as turning whatever command you're trying into an echo statement. Similarly, you test for reasonable values retrieved from the database.
Then, you run this on a sacrificial environment.
Then, you run this on a sacrificial environment, capturing the output. You inspect this to contain the commands you expect.
Then, you run the captured output as a script. After you've verified that it does what you intended for the first couple of lines, you ctrl-c the script, remove the '-i' and re-run it.
Then, you verify the restorability of your backups.
Then, you verify the restorability of your backups again.
Only then you run the script on your live environment.
.
You can't really trust those test -z's... if $1 and $2 are space characters, or slashes, or periods, (and I bet there are more), you still lose. And you probably can't afford to perform that careful testing procedure for every new set of $1 and $2 values.
Writing "rm -rf $variable" does require a certain hubris.
I'm a software developer and speaking frankly when I have to code a deletion against any persistent storage it always give me the heebie jeebies. I'd always far, far rather have an 'I Am Active' flag that I can just clear.
I've been coding for thirty years and spent fifteen of those as a data recovery engineer and writing DR software. These things are what experience is made of.
And you probably can't afford to perform that careful testing procedure for every new set of $1 and $2 values.
That's why you write the resulting commands into a file, which you inspect before running it. I also mentioned checking the results from the database for sane values
And if you can't afford testing potentially disastrous stuff like this, you shouldn't be sitting at a keyboard from which you can issue such commands.
"That's why you write the resulting commands into a file, which you inspect before running it. I also mentioned checking the results from the database for sane values"
And take the path variable that you've constructed and feed it to ls, just to see if what you expect happens.
Those -z's actually work fine if you have quoted the variables (as the OP did). It breaks if you forget the quotes though - which lots of people do. It makes me cringe every time I see this.
Shell scripts are a rubbish environment for writing any sort of robust task. You can get slightly saner behaviour with:
#!/bin/sh -ue
Then:
* Your script will abort if you try to use an unset variable in an expansion
* Your script will abort if one of the commands it runs gives a non-zero return code which you don't test for
But at the end of the day, you would be *far* better to use a real programming language.
Watch out, using sh -e or set -e is usually crap. It is not that unusual to execute commands in the shell which are expected to return non zero in the ordinary course of events; e.g.:
#!/bin/sh
set -e
x=`expr 0 \* 1`
echo "$x"
The echo command is never executed, but there is no diagnostic. That'll give you a bad day debugging. It's the same behavior in POSIX sh and bash.
You're right about sh -u or set -u, though. It's a lot easier than testing every use of a variable with [ -z "$var" ], and does give you a built-in diagnostic when it aborts. As others noted, neither one is a complete protection, of course.
$1 and $2 referring to how much people paid for the service, I presume ? Let's face it - 123 is hardly a 'premium' establishment and and company big or small not backing up their own files (Whether entrusted to 123, MS, AWS or a cowboy down the road) is not doing it right.
@Destroy All Monsters:
Code this shit explicitly.Using a programming language.
Any programming language.
Not murderhack "how is this interpreted NOW?" bash.
Oh come now. This class of error has absolutely zero to do with using bash specifically, or even using a shell based on POSIX sh. It has everything to do with the hazards of constructing arguments dynamically, no matter what the language. MySQL, python, C, go, or anything else: they all entail the same hazard.
On a much larger scale than what this guy did:
http://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question
Original post where he pretends to delete everything:
http://serverfault.com/questions/587102/monday-morning-mistake-sudo-rm-rf-no-preserve-root
Is 123-reg a "you gets what you pays for" company? (i.e. rock-bottom)
I accept that the company has actually issued these statements and I accept that El Reg is reporting it. But I honestly find it no easier to believe than if I saw a cow begin to levitate in front of me.
A company could just not be this incompetent. It just doesn't compute in my brain.
A company could just not be this incompetent. It just doesn't compute in my brain.
Oh it does compute in my brain. But maybe that's because I believe that dark energy and stupidity are one and the same. The very force that appears least organised and most chaotic, most elusive is yet the most prevalent in the universe, and the very thing that holds it altogether. I think a small white cartoon dog beat me to this important theorem, though:
http://dilbert.com/strip/1996-07-21
"A company could just not be this incompetent. It just doesn't compute in my brain."
*sigh* Quoting somebody earlier in the thread:-
I've been coding for thirty years and spent fifteen of those as a data recovery engineer and writing DR software. These things are what experience is made of.
In other enviroments with highly destructive managers however, "You know, these old guys cost us three times more than these new guys we pay the minimum wage to, and they keep arguing that things I tell my staff to do are irresponsibly dangerous and demand that we do things slowly because they say it's "safer". Let's make them all 'redundant' and get a bunch of kids to do the job, they can google anything they don't know these days and their far faster."
12 Months later. What, you mean that you accidentially deleted all of the servers?!?!! Wasn't somebody supervising... Ah. Well, never mind just do a press release blaming a DDOS/hackers/global warning and and recover the backups. What, you mean we that the backups were in the cloud and deleted with the live enviroment!?! Shit. Well, how do we recover from this? You don't know? Then what do we do?!
I put these bozos in the same bracket as Farthosts. I have migrated many customers from both providers over the years. Not so much for reasons of reliability (although both have been shaky at times) but the customer service was always dreadful and lack of many technical features.
I definitely understand choosing cheap providers because of cost - if you won't die over a day or so of downtime then thats a perfectly valid decision to make. But if downtime is going to cost your business serious money, it's probably better to not be using budget mass-market web hosts where you are one account of 10,000 and instead have a managed service, or better still run from multiple providers. Compare the loss due to downtime vs the cost of improving availability.
One thing I would say is separate out your DNS hosting, domain registration and web hosting. Easier to juggle things around in times of brownout. And if the data is important to you, sort your own backups.
.. is what happens when you go for the lowest bidder.
Exactly that. Ditto for "free", by the way, because that just suggests you're paying with something you have far less control over. If you pay with money you can at least stop payments.
Having said that, most small business don't even know what sort of risk they are running when they use services like this. The ones that survive this will have learned a valuable (sorry about the pun) lesson.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
