back to article Outsourcer didn't press ON switch, so Reg reader flew 15 hours to do the job

Welcome again to On-Call, our weekly wander through readers' recollections of their ramblings to customer sites after being called out to fix stuff. This week, reader Tim shares a tale from “About 7 years ago when I was working for a software-as-a-service company based in the UK.” Things were going well and the company was …

Page:

  1. Anonymous Coward
    Anonymous Coward

    Not sure I can top the story

    but I have been on a 30 hour round trip in Europe to configure a printer as the hawdware outsourcer was not permitted access to PC's that could execute its commands...

  2. chivo243 Silver badge
    Trollface

    Let me guess

    'I have power' lights and the 'I am running' lights.”

    Amber and Green?

    I would have loved to see the looks on the "other" guy's faces

    1. Voland's right hand Silver badge

      Re: Let me guess

      I had that one as well on a couple of occasions.

      It is not just Dell. Nearly all servers which have WOL and/or a management controller of some sort on board are like that - they will indicate that the power is connected, but not power on unless you tell them to.

    2. Anonymous Coward
      Anonymous Coward

      Re: Let me guess

      Easy to solve - ask the guy to use his phone to take a short video of the server front and back (or even a photo), and email it to you. All the information you need to say the words "Press that button there".

      Plus - if the servers aren't built or working at all, you don't even need this evidence to say "Press the button...." without any cost or risk at all

      1. I. Aproveofitspendingonspecificprojects

        How come Tim wasn't asked to stay another day to find someone less intellectual than a barnacle to take care of future issues.

      2. jonathanb Silver badge

        Re: Let me guess

        Isn't "have you tried turning it off and on again" usually the first item on the level 1 tech support script?

        1. Cynical Observer
          Facepalm

          Re: Let me guess

          @jonathonb

          ... my guess they would have been stumped. As the machines were not on, turning them off would have been impossible and having failed to complete step one, they could not countenance proceeding to step two under any circumstances.

    3. Chris Parsons
      Headmaster

      Grammar Nazi

      Guys', if plural.

    4. Eddy Ito
      Mushroom

      Re: Let me guess

      To be fair the inventor of "off" leds should be drawn and quartered followed by serious torture. Many years ago my mum bought a Sony receiver that had a red "off" led which turned green when it was powered on. The problem was up to then nearly everything had red "on" leds which went dark when powered off. Imagine my surprise when I pressed the 1/0 button to turn it "off" only to be met with Aida with the volume set at 11. Funny thing, not many devices still have that arrangement but unfortunately the evil bastards have discovered blue leds for damn near everything and they seemingly pump them at 20 watts.

      1. illiad

        Re: Let me guess ( off leds..)

        only answer is thick tape to cover them... likewise 'disk activity' leds next to power leds(mine are tape covered!)

  3. mix
    FAIL

    Surely some kind of telephone support would have worked it out? Are the servers on? Yes they have lights on, what colour are the lights? etc.

    1. Peter 26

      Exactly. This is incompetence from both sides.

      The people trying to login are so incompetent they can't do basic troubleshooting, so they pass it on to the techie. But unfortunately the techie although knowledgeable enough to fix the issue, is socially incompetent, so doesn't call the company to "clarify" the situation beforehand and save a trip.

    2. Anonymous Coward
      Anonymous Coward

      Re: what colour are the lights?

      ... of course, if you are red/green colourblind, working out whether that more-or-less greeny coloured light is a "green" or an "amber" one can be tricky.

      1. Anonymous Coward
        Anonymous Coward

        Re: what colour are the lights?

        In my experience with Dell servers, the only way to be sure they're on is if the fans are doing an impression of screaming banshees.

        1. TRT Silver badge

          Re: what colour are the lights?

          But what if you're deaf AND deuteranopic?

          1. illiad

            Re: what colour are the lights?

            to repeat....

            if color blindness is involved, ** get a Janitor *** or **guy from the street** to tell you what color it is!!!!!!

          2. Yugguy

            Re: what colour are the lights?

            Deuteranopic? Unable to see the fifth book in the bible?

      2. chivo243 Silver badge
        Pint

        Re: what colour are the lights?

        "Eventually management gave up"

        The whole management team and all the rack monkeys plus the on site rep are colorblind at the same colocation? What are the odds? I guess anything is possible ;-/

        We won't calculate the odds of them all being deaf too...

        1. Doctor Syntax Silver badge

          Re: what colour are the lights?

          "Eventually management gave up"

          This is usually when things start to improve.

      3. Anonymous Bullard

        Re: what colour are the lights?

        if you are red/green colourblind, working out whether that more-or-less greeny coloured light is a "green" or an "amber" one can be tricky.

        In that case, the answer to "What colour are they?" should be "Hold on, let me get my friend..."

        1. Anonymous Coward
          Anonymous Coward

          Re: what colour are the lights?

          I made this point on a different thread, but if you are red/green colourblind, a little piece off a broken brake light lens solves the problem. If it's green, it is dark (LED), or possibly dark blue (bulb), through a red lens.

      4. Martin-73 Silver badge

        Re: what colour are the lights?

        Not sure why you got the downvotes, the green is amberish, and the amber is greenish. So yeah, it's a VERY hard call to tell which is which. Not sure why this choice of colours is so dang popular

        1. KA1AXY

          Re: what colour are the lights?

          Am a colorblind hardware designer, can verify. Which is why I scrupulously avoid using bicolor LEDs, and prefer the "traffic light" green to the old yellow-green type in my designs.

    3. chris 17 Silver badge

      Knowledge vs complexity gap

      knowing how to rack a server is one thing

      troubleshooting why it won't power up once power cable(s) is/are plugged in and the lights are on is another.

      http://blog.ipspace.net/2012/03/knowledge-and-complexity.html

      http://blog.ipspace.net/2015/11/can-you-afford-to-reformat-your-data.html

      loads of out source companies relying on unskilled and inexperienced staff to provide services that on paper are cheaper but in the long run are poor value, like in this case requiring an engineer to cross a continent to power on kit.

      1. Anonymous Coward
        Anonymous Coward

        The cable monkeys at our place are very good. I wouldn't trust 90% of them to do any work with PC's let alone servers.

        Anon because apparently I am not supposed to call them cable monkeys, I am supposed to call them engineers (bollocks).

    4. phuzz Silver badge

      It's been a couple of years since I last played with a Dell server, but I'm pretty sure that the light behind the power button flashes orange when the server is off, and lights up bright blue when you turn it on.

      So even if you were colour blind the flashing should have been a give away.

      1. John Tserkezis

        "the light behind the power button flashes orange when the server is off, and lights up bright blue when you turn it on. So even if you were colour blind the flashing should have been a give away."

        It's going to be pretty much impossible to win this argument, because even if for colour blindness, there are no standards for power/running indicators - there is nothing obvious. I've seen it go every which way, even separate marked leds.

        At the end of the day, if you don't know what the bloody led does on a Dell, or any other type of box, you just don't know.

        Even checking the fans is iffy. I've seen domestic grade equipment among enterprise gear, and on those, the fans are rather subtle - I've resorted to a torch at the back to see if I can see the blades turning.

        So, as per the other post, this is a fail on multiple sides.

        1. Anonymous Coward
          Anonymous Coward

          RTFM!

          comletely agree, fail

    5. illiad

      and if color blindness is involved, ** get a Janitor *** or **guy from the street** to tell you what color it is!!!!!!

    6. maffski

      Depending on the generation of Dell server it can be more problematic - they have an ID light that can be toggled blue/amber (iirc) by a button.

  4. Anonymous Coward
    Anonymous Coward

    External IP KVM or DRAC?

    Was it an external KVM? Why didnt they have DRAC or IPMI enabled, would have been a little easier then to check the problem with the servers and power them on remotely.

    1. Philip Storry

      Re: External IP KVM or DRAC?

      Because DRAC/ILO/IPMI look very expensive, so the bean counters dislike them.

      When you're building any infrastructure that has quite a few servers, the additional cost of DRAC/ILO/IPMI soon adds up to a hefty bill.

      Everybody here knows that when you factor in the potential costs - longer outages, and time saved when called out - they're actually pretty good value for money. Not wasting time having to go to the data centre to deploy the Mk I Finger O' Doom is pretty handy. An IP KVM was a useful alternative, but the lack of the power feature made it very much an inferior solution - which was reflected in the pricing of the two technologies.

      But try telling that to the guy who doesn't understand, and is wondering why every server is more expensive by a three figure sum...

      The drive to virtualisation has often been justified solely just on the basis of shaving that cost off each server (and having standardised drivers/devices on your servers). As you scale up, it becomes a significant saving.

      1. Anonymous Coward
        Anonymous Coward

        Re: External IP KVM or DRAC?

        IIRC the basic DRAC functionality, including power control, are free or nearly. The Enterprise version with remote console and the like is to be paid separately.

        While a KVM is usally better for remote control than the ActiveX/Java "virtual KVM" of DRAC, for basic tasks and to check/control hardware, on-board management systems are very useful especially if the machines are away from you. Virtualization doesn't help, you still need a way to manage the hw the hypervisor are running on...

        Also the actual versions of on-board management system can host the server drivers and automatically install them at setup.

      2. Anonymous Coward
        Anonymous Coward

        Re: External IP KVM or DRAC?

        I've previously used an IPKVM and an IP power strip - they worked out considerably cheaper than DRAC/etc cards in each server at the time.

        1. steamrunner
          Mushroom

          Re: External IP KVM or DRAC?

          Did your IPKVM and power strip combo support remote-mounting media so that you could re-install the server from bare metal remotely? Did it alert you to hardware issues with the server? Was it able to automatically reboot your server (ASR) when it detected something was wonky or crashed? Did it provide access to the integrated management (hardware) log on the server when the server was offline? Did it allow you to power-adjust the CPUs to stay within power requirements/tolerances? Did it allow you to automatically power down servers at 'quiet' times and then start them up again when needed? I'm Hazarding a guess the answer to most of those is "no".

          So, as ever, one pays one's money for the features and things one particularly needs. Some people only need basic power control and KVM, some people need more. Price alone is an invalid comparison - we're IT people, we know this.

          Personally, I'm only interested in some, but not all, of the things mentioned above, but experience has been a hard task master and now if anyone around our place provides/specs a server without proper iLo/DRAC-level management on board they get a very stern Paddington-Bear-Stare-grade talking to. It's a philosophy that's saved our bacon (or at least saved a lot of wasted time) enough times to be warranted for us. Meanwhile, our physical KVM (where needed) does just that and nothing more. YMMV, as they say... :-)

          S.

      3. DainB Bronze badge

        Re: External IP KVM or DRAC?

        That's why I love Sun/Oracle hardware, iLOM is right there on motherboard and you can't get rid of it no matter how many cents you want to save. Makes management of lights off datacenters across the planet a breeze.

        1. Anonymous Coward
          Anonymous Coward

          Re: External IP KVM or DRAC?

          HP standard servers also...

          1. Sgt_Oddball

            Re: External IP KVM or DRAC?

            I think you even get it on the gen 8 micro servers too, which stack very neatly on each other for little towers of power (they also make a half decent workstation but that's another matter)

  5. Known Hero

    Ẁhy does nobody use skype ?

    1. Anonymous Coward
      Anonymous Coward

      Ẁhy does nobody use skype ?

      Because, since Microsoft acquired it, it is not very good any more.

      1. Jess

        Re: Ẁhy does nobody use skype ?

        > Because, since Microsoft acquired it, it is not very good any more.

        Here is a useful alternative (compatible with standard sip, though video has to be browser to browser or phone to phone)

        https://www.getonsip.com/

        1. John Tserkezis

          Re: Why does nobody use skype ?

          "https://www.getonsip.com/"

          I must be the only one who reads T&Cs. No commerical use for getonsip.

          "How can they tell" you ask?

          Yeah right, that's what's making the bulk of software writers ONLY offer rediculously expensive subscription models for everyone.

          Thanks for your attitude.

  6. CT

    I was sent to Estonia in order to fix a font-related problem. Failed to fix it on the spot, but had a lightbulb moment that resulted in days of rework.

    1. AndyS

      I had cornflakes for breakfast.

      1. Pookietoo

        I had a nutty crunchy breakfast bar.

  7. BenR

    Sadly, this barely surprises me.

    1. Quortney Fortensplibe
      WTF?

      Hold the Front Page

      AndyS: I had cornflakes for breakfast.

      BenR: Sadly, this barely surprises me.

      Well, I'm gobsmacked. I always had him down as a Wheetabix man.

      1. Pookietoo
        Headmaster

        Re: Hold the Front Page

        "Weetabix"

        1. Quortney Fortensplibe
          Unhappy

          Re: Hold the Front Page

          ..."Weetabix"...

          I stand corrected. Thank you, sir.

          I would downvote myself, for shame, but it's not allowed, unfortunately.

    2. I. Aproveofitspendingonspecificprojects

      I had a fishy serving of microwaved mackerel rolled in some sort of pepper and tasted of bones and hot burned fish. Then I had to make some soup to wash out the bloody taste of damned breakfast shit.

      Sometimes before coffee sets in I make poor life choices.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like