Reply-all email lightning storm STRIKES TWICE at Cisco Cisco is tackling another email storm today, just weeks after its servers were deluged with millions of "reply-all" messages being sent by the networking giant's staff - with many requesting they be removed from the list. The Register has heard from Cisco workers who are currently in the, er, eye of the storm, which is …
I once had the good fortune to work for a large operation that connected many smaller satellite operations to its e-mail directory, and it turned out that there were suddenly two of me. The real me in NYC, and someone with the same name in Upper Bumfuck, NY.
I ended up having to write a robot to deal with the floods of misdirected time sensitive and legally sensitive mail people would send me that was intended for the other one. I tried engaging the other guy to get the word out, but he remained willfully thick and insulting, until I changed the robot to stop forwarding the mail to him and simply bin it with a notification to the sender that they had the wrong guy. Things reached a head when I got notification that my brand new computer was scheduled for replacement. I managed to sort that out with his manager and I think I got his replacement machine delayed a month too. Hope so.
One particular manager had a group who always felt the need to say something when she mailed out stuff, even if it was just "Me Too" (Reply to all of course). One Monday I arrived to find my mailbox had buckled under multiple copies of huge PDFs and Word docs as everyone replied to all about each others replies about how wonderful it was. I also had sixty four notifications from the admins that my e-mail was turned off because of misuse.
So I wrote a special robot just for that manager. It recognized every member of the claque and forwarded all mail sent to me by one of them to her (which meant she got two copies for each reply), along with her original (making three copies) and an explanatory cover note ("Wrong Stevie again, Ms Manager"). Three misdirected mailings and she finally got the message (I calculate she got it a minimum of 192 times in fact).
Matters weren't helped by shirt-thick mail admins who would not grant update permissions to people so they could fill out their address book entries in a more helpful way than the minimalist default of name, phone number and SMTP mail. One of the admins offered to change MY email address when approached to help solve the issue. When I pointed out it wouldn't solve the problem but would break the address books of everyone I had a legitimate business relationship with outside our mail directory, they grudgingly agreed to add "NYC" to my address card for me.
Reminds me of the time I was in Tokyo during the late 90's with dial-up email and some moron at US HQ who I had never heard of sent out an email to everyone in the entire company with some nice pictures of some other person who I had never heard of who had apparently just had a birthday party in an office I had never been to.
Outlook at the time could not be told to not download that email so it sat their trying to download megabytes on a 28K dial up link.
Yeah, thanks for that.
Even with broadband that still happens. My roomie works in an office with a very limited mail space. They are expected to monitor mail, archive/download and delete for the server. Periodically secretaries for high level mucky-mucks send out "All Hands" type messages with html wallpapers, flashing GIFs, or overly large PDF fliers for upcoming celebrations. In fact just the other day her "funny for the day" comment when she got home was one such message had taken more than a fifth of her mail space that morning. Mind you, there are cases in which she is legitimately expecting largish images to be coming in so she can look at photos of stuff to determine failure modes of equipment.
Reminds me of something classic I did at school:
In outlook web access three friends and I set up an e-mail rule each,
If a received e-mail had "SPAM" in the subject then forward it to the other three and then delete it
one person then sends one e-mail and sits back. The admin realised after a few 10's of minutes why the servers were bogged down and tried to *e-mail* us to ask us to stop. Naturally given the rules we didn't even get that e-mail for 5 hours.
Not sure how I didn't receive any punishment. Nor when we used a similar technique to bombard a different friend with 15,000 e-mails in 10 minutes
Anyone care to enlighten me as to how a flood of emails add up to that much damage? I mean, that's some fairly hefty pocket change for what, a few minutes pressing delete one morning? A mild nuisance?
Or is this one of those silly-bugger accountant sums from never-never land where you say 35,000 employees (citation needed) each had to spend (citation needed) an 30 minutes (citation needed), at an average cost to the company of $35/hour (citation needed)...
If you count the lost productivity from the entire workforce being without email for at least half a day, you'll probably get somewhere around that figure. Then if you add any on-call workers and their expenses claims that will add a bit more. It's not so much that everyone had to delete a thousand emails, but that the servers had to cope with 34 million emails within a few minutes and promptly crashed in flames.
One of the guys accidentally sent out a message to all 50k+ employees and even worse had read receipts turned on (he was finance and they always have them on by default)....So within seconds he realised what he had done so he tried to do a recall.
Then followed about 2 days of him sat there unable to do any work as every couple of seconds up would pop a notification of his email being read (followed by a "fail to recall").
OK so I could have killed the response messages and turned off his notifications etc. but no 'cos he wouldn't learn anything that way... :)
Yeah, and Kalamazoo is on I-94, just a couple hours from Chicago1 and a direct shot from Toronto, via Port Huron. And it's not far from the junction with I-69.
If you want out of the way, try holding the exam in Marquette.
1Or a couple of days, depending on traffic, since this is Chicago we're talking about.
"Just as night follows day, some of those staffers committed the mortal sin of hitting reply all to the reply-all message requesting to be removed from the list."
LOL The office trolls, they just cant resist an opportunity like that.
"Other staffers than piled in by berating the "idiots" who failed to realise they were creating yet another email storm."
LOL^2 A combination of The Gullable people that dont know about the office trolls yet, and the office trolls who (only barely) managed to resist the first round opportunity temptation of "please remove me from this mailing list, me too, and me" in wait of the inevitable 2nd round of "dont reply all!".
If you're lucky, it all happened in another timezone while you were asleep, or drinking by yourself at home infront of the tv after dinner, and you get to review the whole shitstorm over your morning coffee to pick whether you want to be part of LOL or LOL^2, with the innocent defence of "oh sorry i didnt see those other e-mails yet, my INBOX was full of ...".
But seriously, think of all the innocent photons and electrons needlessly excited across ocean floors and continents to deliver all this rubbish. No one ever thinks of the poor photons and electrons....
"Whoever sent out the original email is at fault. When you send an email to a large list you should use the Bcc: field for the list name"
That would depend on what you're trying to achieve. If your intent is to broadcast information then you're right. If, however, your intent is to ask something from a large group of people, and the answers may prompt others to answer with more detail then it's entirely appropriate to not only use the To field but also to use the reply all function. Sometimes this is the only way to get useful and complete info, and at work I often resend a mail reply back out to the group to get further feedback because people did a direct reply to me.
Anonymous because this post is going to be deeply unpopular - it is at work generally because it's nerds I send them to who don't like to communicate.
"That would depend on what you're trying to achieve": NO.
If you want to share info with a lot of people and get their input: REPLY, with the list as said in BCC.
If you want the above, but share the input with all the others: GET A WIKI/FACEBOOK/FORUM/whatever. You can even stick in a poll. It's like the future, you know.
As pointed out, you are wrong. You are one of the arseholes that cause these problems - there are much more effective tools designed for multiple inputs that have been around for a long time now.
(The trouble is, you modus operandi is strangely familiar - do you work for a British university?)
You can bet they'll never make that mistake again. Ever.
Or the sender was an innocent victim, having stepped away from their desk to fill their waterbottle for their drink-4L-of-water-a-day-and watch-the-pounds-dissappear-from-your-gut diet , forgetting to lock the screen in their enthusiasm for the aqua aquisition, and an opportunistic office troll took advantage of their oversight and sent the email while they were away...
(You can bet they'll never make that mistake again. Ever.).
There's an old story of the 'email laser' said to have occured at IBM, back in the early days of email.
Employee A went on holiday, setting an email rule to forward all email to his co-worker, Employee B.
Employee B, however, was also away - and set an email rule to reply to all incoming email with an out-of-office message.
As the rule was enforced by the email server, there was no network delay - the rate of back-and-forthing was limited only by the email server's own processor speed. The problem was noticed when it ran out of disk space.
I'm a Cisco employee and I could have used my phone as a vibrator for all of the action it got today.
There were two mail storms - the first was about some mundane technical question emailed to 5 people as well as (for some reason) a mailing list which is for everyone who works in engineering under one of the VPs - 99% of it was the usual crap like "how am I on the list' and "please unsubscribe me" as apparently no one in Cisco has the mental capacity to look at the list of recipients and see that one of them has "mailer list" next to it.
The second one was started when someone emailed a blank email, subject "unsubscribe" to the same list, followed by hundreds of other people doing it thinking it'd amount to anything. You can't unsubscribe from this list, it is automatically generated and permanent for as long as you work under this VP, and even if you could this is not how you do it.
I haven't got an email for a while so I guess someone in San Jose has got to work and finally put an access control list on the damn thing. Not sure why the average employee is theoretically able to email about half of the company.
Anon just bcuz.
In fact, come to think of it, for several years Outlook has also warned people that they are about to mail a lot of people if they use a big group. Presumably because Microsoft found it cheaper to change the software than watch this kind of nonsense. Cisco ought to upgrade :)
This multi-billion dollar giant only finally upgraded to the latest Exchange as of last year. Before then it was on good old 2003.
The ignore button was mentioned by someone but it was drowned by the sea of idiots (and by people changing the subject which would probably defeat the filter anyway). I counted probably close to 500 emails before they stopped flowing.
There are rumors that the original email that started it all didn't have the mailing list CC'ed when the sender sent it, and that Outlook or Exchange mangled it to include it. I can believe that, the original email was certainly not worthy of being sent to a VP's mailing list.
I'm also a cisco employee and can confirm the ordeal lasted from UK morning time till about 4pm. I see it starting with a very innocent email on a technical discussion. As for the ignore button in outlook. A lot of staff have their emails forwarded to work phones. I don't have a blackberry myself but I have the feeling that it doesn't have the 'ignore' button. One email to the list from a US staff complained about being woken up at 3am by a non stop vibrating phone. Also many of the replies changed the subject like to 'PLEASE STOP THE SPAMMING' or 'I DONT WANT TO BE ON THIS LIST'. I ended up creating a rule in outlook to ignore everything sent to that VP's mailing list.
Change the subject line
Add confirm read receipt
Add a forwarding rule to your home email.
The read confirmations were showing up FOUR MONTHS LATER
More fun than putting two telephones together on an all staff conference call to make it nice and echoey
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
