back to article BOFH: Everyone deserves a little DOWNTIME

Isn't it always the way that when you're hours from the office sporting only a smartphone with a dodgy data connection that something critical claps out back at work. And then something's wrong with the VPN so you have to gain access through a convoluted chain of remote desktop, ssh and telnet links just to end up with a …

  1. joeW
    Pint

    Back to Friday BOFH?

    Much better, thank you.

  2. Maverick
    Pint

    at last

    back to Friday were it bloody well should be

    the world is at peace again

  3. Marcus Aurelius
    Joke

    When will we hear....

    ...the pitter patter of tiny little BOFHs?

    1. Anonymous Coward
      Anonymous Coward

      Re: When will we hear....

      Depends on how Amsterdam goes I guess

  4. Dr. Mouse

    Thank you!

    Thank you, dear Vultures, for restoring BOFH to his rightful place on a Friday. Just what I needed after this week...

    To the story, great stuff again, Simon! Remote management through a convoluted chain of remote desktops, SSH connections and port forwarding into the LOMs, why does that sound so familiar? And figuring which server by trying them at random until the one your want disappears. All with people calling you (with me, normally to tell me there's a problem, which I already know, but nobody has botherred to pass the damn message on).

    And Gina is still on the scene... Hope the enjoy the trip to Amsterdam :)

    1. Robert E A Harvey

      Re: Thank you!

      I think Gina is putting him off his stride. How did all these chairweights get his phone number?

      1. Rick Giles
        Linux

        Re: Thank you!

        "chairweights"

        Love it!! It fits my users to a T.

  5. Longrod_von_Hugendong

    Best one...

    for quite a while, I am sure we have all experienced the 'it will be done much quicker if I am not hassled ' phone call.

    1. Bassey

      Re: Best one...

      "....for quite a while"

      It was good but without anyone "having an accident" or, at the very least, the Accounts Team having all their work backed up to null (Hey, the backups were fast!) before a complete server outage I'm not sure it could count as "Best".

      1. auburnman
        Mushroom

        Re: Best one...

        I'm sure one or two names will have wound up in BOFH's 'dish served cold' file as a result of today's episode. And with the inevitable Christmas party episode coming up they'll surely be on the hit list.

      2. Captain Scarlet Silver badge
        Terminator

        Make one up

        The other ip he commanded to reset released the safety control on the accountant eating vacuum cleaners

        1. wowfood

          Re: Make one up

          I figured what that ip he just disabled did. It was a firewall they installed after I flooded the Enrichment Center with a deadly neurotoxin, to make me stop flooding the Enrichment Center with a deadly neurotoxin. So get comfortable while I warm up the neurotoxin emitters.

    2. Anonymous Custard

      Re: Best one...

      I am sure we have all experienced the 'it will be done much quicker if I am not hassled ' phone call.I am sure we have all experienced the 'it will be done much quicker if I am not hassled ' phone call.

      I find the most effective reply is "do you want me to stand here and talk to you about doing some work to fix this, or shall I just get on with actually doing it?". It usually works surprisingly well, even allowing for your normal audience types.

  6. Joel 1

    The one advantage...

    The one advantage of doing it over GPRS - if the data network is being accessed, you can't make phone calls, or more critically, have inbound calls to interrupt you.

    Occasionally slow and steady wins the race...

    1. petur

      Re: The one advantage...

      Are you sure? I remember it the exact opposite, an SMS or call interrupting the data. Off the memory, because with 3G+ this is no longer an issue....

      1. Dr. Mouse

        Re: The one advantage...

        Yes, IIRC GPRS did allow calls through. It was before that, when it basically acted like a modem (can't even remember what it was called now) which would tie up the line when you were internetting.

  7. Alistair
    Pint

    spreadsheetDNS

    sysadmin win.

    (yes I have had to, given a faulty nic in an LB in front of 4 dns servers *sigh*)

    And having a copy on the phone is total win.

    Having a cop pull up behind you, parked on the shoulder while you fix things, is kinda cool, especially when she says "Thanks for pulling off the road to do that."

    Oh - and the GPRS bit? -- depends on which slicing your network uses by the way.

    1. Anonymous Coward
      Anonymous Coward

      Re: spreadsheetDNS

      UK cops frown upon using the hard shoulder as a remote login location.

      1. Anonymous Coward
        Anonymous Coward

        Re: spreadsheetDNS

        And of course the real problem here is answering work calls when you're off duty.... A real bofh wouldn't have answered, or wouldn't carry a cell phone, alas.

  8. Inventor of the Marmite Laser Silver badge

    Did anyone notice the small surge of acceleration

    As the world shifted ever so slightly on its axis to once again point, with unerring accuracy, at BOFH on a Friday.

    There is a God after all.

    1. Michael H.F. Wilkinson Silver badge
      Happy

      Re: Did anyone notice the small surge of acceleration

      "There is a God after all"

      Yes, he is called Simon

  9. b166er

    Excellent, cheers!

    "I was just wondering if you could tell me when things would be back as there's people wanting to work."

    Fuck that one really grinds my gears!!! No fucking shit, Sherlock!

    1. stucs201

      re: people wanting to work

      More likely people quite happy to have a break, but wanting it to be known they have a good excuse for slacking off.

    2. Anonymous Coward
      Anonymous Coward

      well if you're doing a good job you shouldnt hear that very often. what with redundancy n stuff.

      If you hear it a lot because its not your job to ensure it dosent happen - just to take the shit when it does - LEAVE

    3. Toltec

      "I was just wondering if you could tell me when things would be back as there's people wanting to work."

      I estimate it will take me about two hours to come up with an estimate for that. Would you prefer me to do that or continue fixing the problem?

      1. JimC

        I find it works nicely if you say "my order of magnitude estimate is 10 hours", which seems to be so much more acceptable than "more than an hour and less than one hundred"...

    4. Geoff May

      My response to that is "Do you want me to fix the problem or talk to you?"

  10. D@v3
    Pint

    it might be the quick pint i had a lunch, but

    "Well off the top of my head I'd say it was something like a 78 per cent chance."

    "So 78 per cent?"

    "No, LIKE 78 per cent. So it could be 7 or 8 or 78 per cent. It's hard to tell with all these calls."

    made me laugh, out loud.

    So glad i have a quiet afternoon ahead of me.

  11. Clamps Silver badge

    "Unless I just turned off the monitoring system." haha brilliant. On a par with

    "Damage Report!"

    "The damage control machine broken sir!"

    1. Anonymous Coward
      Anonymous Coward

      We had something on those lines. A dev company that was hosting one of our servers. Their monitoring server started pushing out alerts of failing kit. The engineer logged in remotely and rebooted the kit and went back to sleep.

      We woke up our account manager at 8:30am to tell him our server was down, and so was their website and probably all their other clients. They had an air-con fail (they only had a single unit we found out later). No more alerts as the monitoring server was now fried!

      We soon moved the box out of their hands.

      1. Anonymous Coward
        Anonymous Coward

        Fried server

        I have heard the account of how a company that shall be nameless had an aircon failure one weekend which was automatically reported to the people who managed the physical hardware. So two engineers were dispatched in a hurry. On arrival, the security guard told them that they were not allowed in because they were not on his list of people to be admitted at weekends. While they argued virtually the entire server farm fried.

        This was the same company who, a couple of years later, did not want us to host a service for them in the cloud "because it was unreliable" - then discovered how much work they would have to do to accommodate our software and databases on their shiny new server farm, and suddenly decided that perhaps we should cloud host for them.

        As I say, I can't say who they were and where they were based, but the medical acronym NFN may give a clue.

  12. Skymonrie
    Pint

    painful

    Reading through that has given me a headache...it's too true.

    my weeks sysadmin nightmare working remotely

    I just spent half a week having an email back and forth with someone who insisted a system didn't work regardless of it working (and being shown to work through screen recorders and tutorials) with every variant and setup under the sun.

    After finally convincing them to let me setup a remote desktop connection to their computer to see first hand, we did the run through and, surprise surprise, it worked perfectly. They just read a single (something they had written) word wrong and thought they were in the wrong place. If they had even filled in the "problem report" form provided they would have seen their own stupidity.

    Love that we're back to weeklies, have a beer on me!

  13. Anonymous Coward
    Anonymous Coward

    "with the Eurostar special fare it was actually CHEAPER"

    Over a decade ago, 4 of us were scheduled for training and told to locate the cheapest option. We figured out a place in the Bahamas would work out cheaper than London but it didn't fly with management.

    1. Anonymous Coward
      Anonymous Coward

      Re: "with the Eurostar special fare it was actually CHEAPER"

      I on the other hand was once asked to do an evaluation to site a new factory, before China would even have been considered. As well as a couple of development zones in the UK, I did a comparison with the Bahamas. The Bahamas came out top; low taxes, cheap loans, workforce with UK educational standards, and as the products were small and light hybrid assemblies transport costs were not a problem.

      I presented it, the MD looked at the numbers longingly and then said "I just can't see myself getting it past group management."

  14. Anonymous Coward
    Anonymous Coward

    Lucky South Easters!

    Bah! Lucky South Easters / Londoners, with your excellent links to all the interesting cities of mainland Europe!

    On this backwater of the isles, a jolly to London is the best we can hope for!

    1. Robert E A Harvey

      Re: Lucky South Easters!

      Jolly? to London?

      Whitley Bay or Brown's pie shop in Lincoln would be a Jolly. That Lunnon is a Penalty.

  15. YetAnotherPasswordToRemeber

    "I was going to say I don't know," I respond, "but now it's I don't know PLUS the time of this phone call, PLUS the time to navigate back to the remote desktop app."

    "So you don't really have any idea?"

    "I don't know exactly what's wrong yet, so no."

    I guess we've all have this discussion!!

  16. Andrew Moore

    I particularlly loved...

    ...the "tell us how long it will take to fix BEFORE you've even had a chance of looking at the problem yet"

  17. Mike Hartley
    Paris Hilton

    So BOFH did hook up with Gina

    Paris - because she's probably jealous right now...

  18. bed

    Don't saw the branch off the tree while sitting on it

    GPRS can be handy once you go to a remote site to swap out a router and use a different IP range then get into a chicken and egg situation because the router back at base needs to have it outgoing 'route' command changed at the same time. 3G/GPRS dongle things use (used ?) more GSM channels than when using a phone as a modem so gave you more bandwidth.

    Good rant. Responding to the "The Internet is not working" phone call always resulted in a de-prioitisation in the dynamically reallocatable prioritisation schedule matrix.

  19. Boris the Cockroach Silver badge
    Happy

    I feel his pain

    for too many times I've been buried down the back of a machine trying to work out whats gone wrong in the el-cheapo antique machine I've been lumbered with fixing when the inevitable call comes from the management

    "When will it be up and running?"

    My current boss is pretty good though.... does'nt instantly fire me when I shout back that he should f*** off and die ...

  20. ecofeco Silver badge
    Thumb Up

    Too many times

    "So you don't really have any idea?"

    "I don't know exactly what's wrong YET, so no."

  21. Tempest8008
    Coat

    All I know...

    All I know is that if he was doing all this stuff from his Smartphone in the US or Canada he'd have built up such a huge data bill that he'd probably be into the tens of thousands of dollars by the time he was done.

    Oh, for a sensible European style system of cellular roaming...

  22. Anonymous Coward
    Anonymous Coward

    Was this written in 1998?

    PDC? BDC? Still running NT4 at BOFH premises? :-)

    1. Anonymous Coward
      Anonymous Coward

      Re: Was this written in 1998?

      Of course! Who else are they going to find that can make it work. I can recall calling my mother back in the U.S. from the Philippines about a problem that was driving me nuts. After reporting all the symptoms, she told me where to kick. If that fixed it, replace this component, otherwise replace that. She was dead on. Awesome Mom!

    2. Tom Maddox Silver badge
      Thumb Up

      Re: Was this written in 1998?

      I was going to make the same comment. It probably makes for excellent excuse fodder, though: "We've got a BSOD on the PDC, so we need to TTFO it PDQ so we can see if it's DOA."

      "Why not upgrade?"

      "STFU."

  23. Fatman

    BOFH FAILed here

    Sorry, but I don't buy this shit!!!!

    BOFH's ALWAYS leave themselves a BACK DOOR if they are any good.

    It can be nothing more than an old analog modem connected to a RS-232 port on a POTS line.

    Years ago, I worked (according to TPTB, it might be a case of 'fucking off' rather) at a radio station. 1970's cheapo telemetry equipment often went bonkers at the most inopportune time, and if the transmitter site based remote unit lost "sight" of its studio counterpart, it was programmed to kill the transmitter, shutting down the station. That were (IIRC) the rules (according to the Federal Crucifixion Commission) in force at the time.

    Getting control meant a 25 mile drive out to the transmitter site, and power cycling the remote unit to 'reset' it. The station's engineer didn't like to do that, so he created this 'black box' that connected to a POTS line. You "call" the 'black box', it answers with a tone, you punch in a (user selectable) authorization code to get a second tone. Then you could press one of the 12 buttons on the DTMF pad and send a command to open/close a relay. He had the "*" button set to cause a one second pulse to trigger a power drop relay on the remote units' power connection. Power goes out, and one second later, it comes back on. Another minute for the two ends to handshake, and all is good. The studio end was programmed to flash a light on the audio control board informing the operator that the telemetry unit had gone off line, and the operator better power cycle the remote end within 15 minutes of the light coming on, or else.....

    Nothing pisses off a station owner than the prospect of DEAD AIR.

    Now, our BOFH should NOT allow this disaster to be "wasted". Time to get out the spreadsheet, and con persuade the Boss into approving the necessary kit to improve resilience in such situations.

    1. Anonymous Coward
      Anonymous Coward

      Re: BOFH FAILed here

      Eh? He explicitly says he *is already* using the backdoor. Unless he was referring to Gina, but I don't think so.

  24. Herby

    Question to pose...

    How many unlisted numbers have modems that answer on the 10th ring?

    Those that respond with the game of "Global Thermonuclear War" excepted.

  25. chivo243 Silver badge
    Thumb Up

    Damn

    I still saw it on a Saturday...

    On another note, I hate the interview that seems to occur during a crisis. How long bla bla, are you aware bla bla. Can you expedite it bla bla.

    I would really to ask in the middle of the crisis just because:

    How is the family? How is that 50ft boat? Did you get the BMW recall for the ass warmer? I see the school your kids attend received another award. Are you still boning your receptionist? I like 50's in non sequential order.

  26. wayne 8

    but now it's I don't know PLUS the time of this phone call

    I actually told that to the dot Indian manager who was the fifth or sixth person to appear at my cube to inform me of the critical issue I was trying to resolve. He has since ascended to reporting to the dot Indian CIO and I was never promoted since he has blocked my manager's promo recommendation.

    Still, I would say it again.

    Bootnote: That same guy suggested last Wednesday that for "stability for Black Friday" a nightly refresh job not be run on the eve of BFD. It took several hours to trace the resultant crashing servers filling the logging disk to the lack of the refresh.

  27. Kiwi
    Go

    That's why...

    That's why I ride a motorbike (well, aside from it just being so much fun :) ), that's why I get as much time out in the country as I can (far away from cell towers - or so people believe ;) ), and that's why the smartest phone I'll ever own is my V3 RAZR. Next phone will be as dumb as I can get.

    Generally can't be reached. If I can, nothing I can do about it anyway.

    Like the icon.. Go.. Go far. Go fast :)

  28. ortunk

    ...this guy I used to work with would come in emergencies and ask me "when can we get an answer" (not an ETA, then he would stand guard outside my office and tell everyone to sod off for whatever time frame I gave him...

    ...best piece of sysadminwork I ever did was during his guard...

  29. ortunk
    Paris Hilton

    Paris because I don't knw how to delete this post...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like