back to article Cisco email accidentally sent to 1000s of employees causes message list MAYHEM

Ever become stuck in a seemingly infinite loop of emails from colleagues hitting the "reply-all" option and asking to be removed from a mailing list containing the addresses of thousands of employees? If the answer to that question is a resounding yes followed by a massive sigh, then you ought to have some sympathy for Cisco, …

COMMENTS

This topic is closed for new posts.
  1. Swarthy

    Two birds - One stone...

    Lay-offs and email storms:

    Just lower the ax on the ones that replied-all to unsubscribe.

    1. Anonymous Coward
      Anonymous Coward

      Re: Two birds - One stone...

      There were a number of emails suggesting exactly that...

      Personally the whole stormed brightened my day and a happier worker works harder...

      Anon for obvious reasons...

      1. Anonymous Coward
        Anonymous Coward

        Re: Two birds - One stone...

        Ah, memories.

        Working in a london council IT dept in a minor install/support role - somewhere in the region of 5000 users and email accounts. Someone managed to send an email to all staff, and the usual shenanigans as described in the article kicked off. We thought it was funny till the sysadmins pointed out how much bandwidth and storage it was eating up.

        We talked to the top man in the support dept, who talked to the CEO/top man in the council. A plan was formed.

        All staff email - anyone who continues to respond to the erroneous reply to all email will be terminated for gross misconduct and misuse of company network, or words to that effect that made it sound like a real threat (there was some nuance and explanation of how this was perfectly legal, the bandwidth is affecting the several dozen remote offices on WAN links, no-one can get any work done, you have been warned, etc), signed, the CEO under advisement of IT Operations.

        Twenty minutes later, another email - three people have been summarily dismissed for wilful gross misconduct and have been escorted off their respective premises by security.

        And about thirty seconds later, all the network monitoring equipment stopped flashing red and making scary flashing lights.

        Of course, no-one had been sacked - although some had been noted for written warnings (they had been sending 'yeah right' emails - not big, not clever, probably didn't do their promotion prospects any good). But it gave the network admins time to clear up the mail servers and dump all the crap out of the queues and mailboxes.

        I don't believe it has recurred since to such a large degree since (for a start, there is no longer an "easy to acci-click" all staff distribution group - which is how it apparently happened in the first place, this was at the height of the terrorism times just after 7/7 so local govts all had status updates on terrorism alert statuses etc) - and old hats are wary of checking recipients before hitting 'reply to all'.

        I work in a far, far smaller operation these days - maybe two dozen users. I can live with reply to all abuse there because I'm the sysadmin, and as such I'm the one who gets to belittle people for doing it ;-)

        Anon for a few reasons...

  2. chairman_of_the_bored
    FAIL

    Nothing new

    This - if true - is an almost exact replica of what happened at HP a few years ago. In open-plan offices, if was hilarious to listen to the pings of incoming emails, especially from laptops left unattended by their users.

    1. agentgonzo

      Re: Nothing new

      These days a lot of people have blackberries that buzz when there is an incoming email. The guys on tech support in China were not amused...

      1. Crazy Operations Guy

        Re: Nothing new

        "These days a lot of people have blackberries that buzz when there is an incoming email. The guys on tech support in China were not amused..."

        Depends on where they were keeping their BlackBerries...

    2. Anonymous Coward
      Anonymous Coward

      Re: Nothing new

      This not only happened a few years ago at HP, it still happens with monotonous regularity to this day. It is unbelieveable the number of numbskulls in the company that still click on "Reply to All" to be unsubscribed from a distribution list - and they are supposed to be IT "professionals".

      1. Mr Nobody 1

        Re: Nothing new

        Not only does this still happen with monotonous regularity within HP, there is also the office sweepstake to see who can complete the 'mandatory training with questions so stupid you would have to have a common sense bypass to fail''. I think the record for the 30 mins on Health and Safety was around 3.

  3. qwertyuiop
    Alert

    Brings back memories

    Many years back whilst working for another company we arrived in the office one Monday morning to find the mail servers on the floor and in need of life-saving treatment having become "full". Hours later when normal service had been restored and the full post-mortem began we discovered what had happened.

    At the time we had no remote access into email so one Thursday a contractor, prior to going home for a three day weekend, set up an auto-forward rule in his mailbox to forward any incoming mail to his Hotmail account and an account with another ssupplier (can't remember who). This despite a total ban on auto-forwards to outside the company.

    Everything was fine throughout Friday; emails arrived and were forwarded to his two private accounts. Unfortunately at some point on Saturday morning his Hotmail inbox became full and so the next time it received a message from his work account it replied saying that it was undeliverable because the mailbox was full.

    So our system duly forwarded that to his Hotmail account...

    Which replied that it was full...

    Repeat this loop until some time later his other account became full...

    ...and started sending replies that it couldn't accept incoming messages because it was full. Which our server dutifully forwarded to BOTH accounts which BOTH replied that they were full...

    And some time on Sunday our mail servers gave up because THEY were full.

    Oh how we laughed! Or maybe we didn't. The contractor had his contract terminated that day.

    1. Tim K

      Re: Brings back memories

      The guy was fired for that? Really? That is 100% the sysadmin's fault there - a ban isn't worth the paper it's written on unless it's been technically implemented.

      1. Gerardo McFitzpatrick-O'Toole

        Re: Brings back memories

        Come on, Tim - the sysadmin should probably have prevented it an/or mitigated the effects, but this doesn't remove the contractor's responsibility, or the responsibility of anyone that misuses a company network, whether well-meaningly or maliciously.

      2. The Serpent

        Re: Brings back memories

        "The guy was fired for that? "

        He used his privileges to set up a facility which was specifically banned by the company he worked for. That is very dodgy ground even on the limited information given in the original post. There may be additional circumstances such as lost business due to lack of the email service. Round my way such an arrangement would result in many, many breaches of the data protection act. All of which would probably justify a short, sharp trip across the car park.

  4. Craigie

    size

    "At that point four million emails had been sent, generating over 375GB of network traffic".

    I make that 94Mb per email.

    Hmmm....

    1. agentgonzo

      Re: size

      Your maths is bad. 94Kb per email (which is what it averaged out as as the reply-all had all the previous text/images in it as the afternoon went on).

      1. Omgwtfbbqtime
        Facepalm

        Re: size

        Would surprise me if each line started:

        >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

      2. Anonymous Coward
        Anonymous Coward

        Re: size

        Your maths is also bad...

        94Kb * 4000000 = 47GB

        I assume you meant 94KB average per email, which would be about right.

        But note that the 375GB was 'network traffic', which presumably includes the transmission and separate reception stages (e.g. SMTP & POP) and headers/overhead, so each email itself would need to be somewhere around 40KB for that.

        ... but maybe the email store was on a SAN, in which case that network traffic might also be counted... I'm going to stop here.

        1. agentgonzo

          Re: size

          Maths was correct, holding down of shift was incorrect... I meant 94KB (kilobytes) rather than Kb (kilobits).

          1. Anonymous IV

            Re: size

            @agentgonzo: Maths was correct, holding down of shift was incorrect... I meant 94KB (kilobytes) rather than Kb (kilobits).

            Probably you actually meant 94kB!

            If I remember correctly the kilo (k) is the only SI prefix for powers of ten great than one (apart from the never-encountered hecto-h) which is in lower case (compare mega-M, Giga-G, tera-T).

            Another instance of where a standard is flawed...

            </pedant>

    2. Raumkraut

      Re: size

      > I make that 94Mb per email.

      I blame the top-posters.

      1. JeffyPoooh
        Pint

        Re: size

        Me too. Top posters are idiots.

        > I make that 94Mb per email.

        I blame the top-posters.

  5. AndrueC Silver badge
    Facepalm

    nag staff about completing an online training module by the end of the month.

    One of the curses of working for a big corporate. I once had to complete a course four times because of various system glitches.

    Keeping HR happy.

    1. dillbertiniantype

      Me too.

      It happened to me too. Actually it was a training course on sexual harassment. I wonder why they made me do that four times. Needless to say I no longer work there.

  6. simbr

    When this happened to me it wasn't in the relatively safe confines on an office but out on the wild wild web. Must have been 10 years ago I came back from a holiday to find a newsletter I'd subscribed to had suddenly behind a two-way mailing list and exploded...

  7. Anonymous Coward
    Anonymous Coward

    This happened at the BBC whilst I was there.

    Someone replied to a distribution list that they really shouldn't have been able to send to, something to do with "playing pictionary with the helpdesk team" (it was an article in the newsletter they were replying to).

    Cue a storm of reply alls asking to be removed alongside other trying to be helpful. Eventually the DL got disabled.

    It made it up to Salford where I then saw pictures on Facebook with printouts on the wall.

    I still have the e-mails, I still browse through them for amusement.

  8. Brian Miller

    Happened a few times at Microsoft

    The first time was the "Me too" incident in, what, 1998 or 1999? Somebody was testing distribution lists on Exchange, and had simply grabbed a bunch of names and slapped it on the list just to create a really huge list. Someone else, whose name was on that list, sent out an email to the whole list: "Who owns this list, what is it for, and why am I on it?" Someone else replied, "I want to know, too." Then someone else replied, "Me, too!" Followed up with, "Me, three!"

    Many inane replies later, and the mail servers had ground to a stop, and took hours to clear. They made up t-shirts to commemorate the event.

    I've seen something similar happen a few times since then. Somebody accidentally uses the wrong distribution list, or else sends out an important bulletin, and then some idiot always replies with, "Could you please remove me from this list?" and on it goes.

  9. I ain't Spartacus Gold badge

    I remember this happening once. Someone sent one of those chain emails about a missing child in the US. To the entire global address list. The person who replied in a grumpy and abusive manner, of course, hit reply to all... There were many emails in my inbox over a space of about 15 minutes, in which the perpetrator attempted to retract their offensiveding message, their boss then apologising, HR then informed all that the aforementioned person had now 'left the company'. Oh deary, deary me.

    I do remember some idiot doing an email to about 40 people, with delivery and read receipts and an attachment to fill out - as these people were notorious for not providing the information. Of course the idiot forgot to include the attachment, and so within five minutes 40 delivery receipts, 10 out of office messages, 10 read receipts and 5 replies appeared in my his inbox, and the thing had to be sent out again with apologies, and 40 more delivery receipts, 10 read receipts, 5 answers... And a partridge in a pear tree.

  10. Anonymous Coward
    Anonymous Coward

    Keeping buzzing, you happy worker bees!

    Ahhhh, you can tell the level of job unsatisfaction at a company, by how readily the proles will take an opportunity like this to enjoy just a few moments of sadistic levity by 'accidentally' or 'helpfully' reply-alling too, thus adding to the carnage and giving them temporary respite from their numbing jobs. Being nagged to complete a (very likely) health and safety, or information handling, online training module? Oh what joy...

    1. king of foo

      Re: Keeping buzzing, you happy worker bees!

      I make a point of hitting reply all

      ...and copying my **** boss in at the same time...

  11. fixit_f

    Well Cisco are the network people....

    ..... and they presumably use their own kit for their infrastructure. Just call it "stress testing" and voila, a happy outcome.

  12. My Alter Ego
    Devil

    Did this to a guy in an previous company

    He was a very sleazy towards a friend of mine. There was also a bug with the Lotus Mail client which wouldn't lock if you used the Super+M combination, so when we left his desk we sent an email to everyone in Dublin, London, Toronto & Tokyo which requested a receipt (which wasn't optional). His inbox didn't last two minutes!

    Ah, good days.

  13. Anonymous Coward
    Anonymous Coward

    Email Storms

    These email storms happen at least once per anum at my workplace (a large recognizable company with a name related to a large South American river). Half the fun is reading all of the responses egging everyone on - like needing to put "unsubscribe" in quotes or other even more silly, extra complex email procedures that have the effect of generating even more fun - especially if the thread forks with a new subject line. Experiencing the joy of others' trolling is secondary to the joy found in the anger of those who don't have the mental capacity to properly use the "rules" feature on their email client to sort out or delete the threads.

  14. Steve Renouf
    Facepalm

    Bcc anyone?!?

    How the F**K can they reply-all if they've all been Bcc'd as they should be. What? They weren't Bcc'd? Heads should roll!

    1. ecofeco Silver badge
      Facepalm

      Re: Bcc anyone?!?

      And there you have it.

      The instant I see a company email come in with over 100+ recipients in CC instead of Bcc, I'm on the phone to the original sender.

      Needless to say, this doesn't make my boss happy. (proving yet again that being "proactive" and "taking ownership" usually means you will get fired)

    2. Captain Scarlet Silver badge

      Re: Bcc anyone?!?

      Its always hidden (Even in Lotus Notes although simple rule to make it show be default on your default policy) which doesnt help.

      I have said to members of staff before use BCC only to be told "I know how to send emails!"

    3. Anonymous Coward
      Anonymous Coward

      Re: Bcc anyone?!?

      Some companies have LISTS which is just a name and nobody even really knows who is on it, except maybe the list manager, who may or may not be the person listed as such if you can find the subscribe/unsubscribe file. Then other lists are added to lists and you end up with 3 or 4 copies of everything, but you can't unsubscribe from the lower order ones because you DO want the mail that should come on that normally. The last storm we had was small, only about 5000 on the list.

  15. Jess--

    worst one I came across was between 2 mail servers within the same company which had no attachment size restrictions in place.

    everything worked perfectly (for years) until an idiot decided to send a 30gb file as an attachment before leaving on a friday evening.

    the initial mail server accepted it and started sending it onto the 2nd server.

    the second server started to receive the email with no problems and everything appeared fine... for 5 minutes.

    5 minutes after the initial sending from one server to the other had started the first mail server went back over its outbound queue and found this 30gb email that hadn't been delivered (because it was still transferring) and resent it.

    5 minutes later it found 2 undelivered emails and started resending them

    <5 minutes per step>

    1 email (30gb)

    2 emails (60gb)

    4 emails (120gb)

    8 emails (240gb)

    16 emails (480gb)

    32 emails (960gb)

    64 emails (1920gb)

    128 emails (3840gb)

    256 emails (7680gb)

    512 emails (15360gb)

    1024 emails (30720gb)

    2048 emails (61440gb)

    4096 emails (122880gb)

    <end of 1st hour>

    By saturday morning the second mail server had died completely having used up 4tb of previously empty storage while the 1st email server was unable to display how many emails were in its oubound queue.

    1. The Vociferous Time Waster

      Re:

      Jess, I call bullshit. What mail system sends mail it has not received yet. Did you put 4tb because it sounds impressive rather than realistic?

      1. Tom 260

        Re: Re:

        The first server was doing all the sending (and resending), it had received the entire mail from the user's computer/terminal.

        With all the chaos generated by users, who needs email worms like "I love you" to bring down a system!

      2. Jess--

        Re: Re:

        The sending mail server was exchange (I don't know the version) the receiving server was Mailenable (used internally by that dept.)

        the 4tb was 3 2tb drives in a raid5 array.

        As Tom pointed out the user had submitted the email to the 1st server and it had submitted fully, and it was the 1st server that was doing the stupid re-send even though the transfer was already in progress.

    2. Captain Scarlet Silver badge
      Facepalm

      I bet thats Lotus Domino Server sending between two different Notes Networks!

      Ten years ago my work place also used rather silly schedules to send mail from server to server, one server had to have 5 emails in the queue before it triggered the notes agent to send mail to the relevant servers. This meant the last person to send an email from that office was very likely not to have it sent until the next week.

  16. mdava

    Reply-all should require more input

    Reply-all is fine, but it is proved time and time again that it's used inappropriately.

    I think all email clients should require that when you click "reply-all" you have to individually say "yes" to each of the email addresses that you are responding to.

    I can't think of a valid reason to reply-all to more than 20 people, so doesn't seem unworkable.

    1. king of foo

      Re: Reply-all should require more input

      Hear hear, I'd also add that distribution lists should only be allowed as bcc, at the same time 'reply all' only working for to and cc (:. ignoring bcc), both having a maximum number of recipients ; your 20 sounds about right. Job done.

  17. Alan 6

    I used to work for a small division of Honeywell, a company with a couple of hundred thousand employees worldwide.

    Last year somebody managed to send an email newsletter to everyone in the company, the all users group address is limited access with only very senior IT personnel allow to use it, so how this message got out is anyone's guess. One person replied to all, asking to be removed from the email list, this prompted other people to do the same, then more and more, it took the best part of a week for the IT bods to finally remove the emails clogging up all the company's email servers worldwide...

    1. intrigid

      Anyone's guess? Uh, my guess is that the list WASN'T restricted at all, considering that everyone was able to use it!

      1. Alan 6

        Everyone wasn't able to use it, the first reply all ripped the names from the list and put them all in to To: box, all 200,000 or so of them...

  18. herman

    Database backend?

    Any self respecting mail server with a database backend will only save ONE copy of a message sent to a list.

    Soooo, what kind of crappy mail servers are used by the likes of Cisco, HP, MS and Amazon?

    1. Anonymous Coward
      Anonymous Coward

      Re: Database backend?

      A ZFS server with dedup turned on would have dealt fairly well with this scenario.

    2. Anonymous Coward
      Anonymous Coward

      Re: Database backend?

      Microsoft infamously removed the de-dup feature of MS Exchange in order to improve performance. So, your "self respecting mail server" departmental solution standards are just so '90s. Also, when email is distributed over tens to hundreds of DB backends, the de-dup becomes less viable.

      http://blogs.technet.com/b/exchange/archive/2010/02/22/3409361.aspx

      Note: a 10,000 user distribution list is a SINGLE email address, so the various "are you sure" schemes only work if the users aren't used to just clicking "OK/Yes/Agree/Continue". If you can't count on them to not try to unsubscribe from the HR system controlled "all-users" or "department x users" email lists - you can't count on them to just click "no".

  19. NuttEffect
    FAIL

    Sorry...

    Me too.

  20. Bill Stewart

    I accidentally triggered one of these many years ago. Somebody sent a CraigShergold-gram to the building list (about 5000 people) one Friday afternoon, and I thought for a moment about whether I should send a "Please don't reply, especially to the entire list, here's the explanation" reply on not. This was back in the days when most of our mail was on departmental Vaxes, and some of it was on a big Unix-like mainframe system, some running SMTP but some running UUCP, and there wasn't a snopes.com to point people to, just alt.folklore.urban on Usenet.

    I decided that it was probably better to send it than not, and of course I started getting bouncegrams from people on vacation, etc. About half an hour later, a friend called, asking if I'd meant to send five copies of the mail, spaced five minutes apart. "Umm. no..." "Thought not, enjoy the rest of the afternoon cleaning things up." Apparently the mainframe in the basement had forwarded out the message to everybody, decided that something hadn't worked, and re-queued it to try again later. I went down to the basement where the building sysadmins lived, apologized, and we spent a couple of hours trying to find the problem and clean up the mess (simply stopping the mail server and clearing out its queue wasn't close to good enough.) Never did find out what was wrong, and of course my bouncegrams were starting to include "Can't deliver message; mailbox full" as various departmental Vax disks filled up. And Monday morning we started with a couple rounds of "Stop sending me this junk at work" "Get me off this mailing list" "Don't send that to the entire list, dummy, just the sender (sent to the entire list, of course)", but none of them broke the mail relay this time.

  21. Number6

    The Delete key?

    For a one-off I use the delete key. For more persistent irritating stuff like "your timesheet is now (over)due" I set up a mail forwarding rule to auto-delete. That's how I would have dealt with a storm like that described, although I might have gone back and laughed at some of the emails in the deleted items mailbox later.

  22. Ed_UK

    <Reply-all> may not be the problem

    During the last international mail-storm here , I watched hundreds of "unsubscribe" emails being sent to the entire list. My first thought was that many colleagues were foolishly hitting Reply-all. In fact, they were actually (still foolishly) using Reply.

    Most list-servers in my humble experience simply treat a Reply as a Reply-to-list; you need to use a different admin address if you want out. List members tend to forget or lose this vital piece of info. At work, entire departments may get signed up to new lists, with little warning.

    Some list-servers at least may trap any unsub requests, but it doesn't seem to be default.

    Still, that's my observation as a user, not an IT admin of any sort.

  23. Lloyd

    Similar experience

    I worked for a subsidiary of a certain Blue Bank and some noob sent an email to the "all Company" group (which covered every UK branch, head office, etc) asking a question (something like "does anyone know where the printer ink is"). Cue much "why is this coming to me?", "please stop emailing me" emails, which was stopped very, very quickly by an email from the CEO's PA, which simply introduced herself and stated that everyone should stop responding, the email was obviously an error and to leave it alone. It was a very clever email, it came across as a school teacher scolding children but with the added backing of "carry on and you'll be dragged in to see the CEO" but managing to leave the threat heavily implied without being threatening. kudos to her the trail stopped dead and everyone got on with their lives.

  24. randomHandle

    Be my friend

    Or the person at a Scottish financial institution who somehow managed to import the entire corporate directory into their Facebook contacts and then sent everyone a friend request. The recipients were surprisingly unfriendly.

    The Management subsequently insisted that the directory should be hidden from everyone.

    1. I Am Spartacus

      Re: Be my friend

      Oh how I wish I had put the coffee cup down for this.

      This thread has brightened a rather dull day by making my laugh all through lunch. Bed for the screen, but great for the diet!

  25. Anonymous Coward
    Anonymous Coward

    ICL, during my industrial year back on the early '90s...

    Some management type managed to send a message to every email address in the entire company - using OrificePower - so the X.400 email address list was so big that servers keeled over all over the place attempting to deliver them all.

    Once the storm was finally over, said manager decided to send out an apology - to the exact same list.

    I don't know if he was ever allowed to use email again after that...

  26. Chris T Almighty
    Happy

    I feel sorry for the Spooks

    Their job of cataloging every email in the world isn't made any easier by this sort of nonsense. :o)

This topic is closed for new posts.

Other stories you might like