This just wouldn't have happened if he had written the passwords for everything on a post-it note and stuck it above his desk - like everyone else does....

Nah! That wouldn't be "confusion" that would be a "hardware error". I reckon it's that they had trouble interpreting the Mayor's handwritten version of "Th3M4y0R1S4f5ckw1T".

How do you do that then?

Mark: Best comment ever on El Reg!

I cannot se how having access to the harware loosing passwords could be such a big problem.

I once hade to take back a Unix machine from a customer who had stopped paying for the machine.

Asking the boss for the root password he smiled and said "sorry I just forgot it".

I could have left it at that but I had to boot the machine from a floppy, mound the HDD and erase the root password.

The boss did not smile anymore.

There must be ways to deal with Windows too.

As any sensible Desktop Support Operator knows, all you need to do is talk nicely to your nearest (insert flavor of Unix here)-using geek and (s)he will be able to furnish you with a password hacking tool... sorry, emergency boot disk.

Anon as I'm at work and the Big Bosses would go uber-ballistic if they realised just how fekkin stupid we really think they are.

That they couldn't find a hacker in the Bay area, if not California that could crack the passwords? Instead they go pleading to the culprit?

Clear case of incompetent bureaucracy.

SF is a BIG city so their budget must be large enough to suggest he had a team rather than be working alone - what were they doing while he was setting all this up?

I was torn between the S&C (a hacker could have sorted them out) and Paris. Paris got it in the end (oooerr) to represent the administration...

Or they realised the I was a 1...

"He is accused of blocking all access to the city's network and routers by resetting passwords."

I think we all know what this means - the passwords were all "admin" or "password" and no-one in SF thought to try them.

Otherwise those passwords would have been easily available at a safe nearby.

I know what the back of people who are "good at it" looks like...

... aren't they all 'terrorists' now? It's probably a lot easier all round for the city authorities to lock up one bloke until he tells them the password, rather than prove that an outside hacker could get through their security.

Pretending that access to the system is impossible without the correct password gives the impression the system is, if nothing else, impregnable to unauthorised users. Getting someone else to hack in and set it right would have the US press howling in full-on 'Chicken Licken' mode that any 'terrorist' could have done the same - cue the banning of 'War Games' and every IT professional going on a 'no fly' list.

My money's on the mayor telling our man that they'd already got in, but the trial would go a lot easier if the fiction was maintained.

Or were using a Mac (fanboys or technopleges) to log in and the password had a # in it.....

that'd be the 1 in c1sco then?

He was in charge of WAN routers, all Cisco gear, and the passwords were all for those routers, there were no servers nor any desktops involved.

Apparently, the Ciscos were configured such that password recovery was turned off, or something like that. This was all in an online article a few days ago where another IT guy working there gave some further details.

Well, duh!

"Give us the passwords, and we can talk about cutting the bail to something sensible. That is, if you want to have a last little bit of freedom before all this becomes your second home. You do, don't you? Or have you come to enjoy Big Bubba's night-night 'cuddles'?"

Paris could have worked that one out for herself.

I can't vouch for veracity of this, but here is apparently the inside story...

(from infoworld, linked by geekpress.com)

http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-network-lockout_1.html

I'm surprised Homeland Security didn't fly him off somewhere and have the sh*t beaten out of him.

I used to be responsible for Cisco password security at a rather large multi-national many years ago and we had numerous cases of Network Engineers setting up routers and forgetting to update the password file. (Wonderful flat text file available to some 500+ users who could easily copy it to floppy......I know as my Manager and I did once. Left the building, went to lunch, and no one knew. Informed the 3rd line manager and he just grunted at us.)

As routers with lost passwords were at customer sites we had one of two options to recover them.

1. Use the Cisco Configuration Tool for dragging back the config, editing it, and then uploading it to the router again. (Cisco wouldn't allow us to have it, but we had the IBM versions which worked great.)

2. Send an engineer to site at a cost of £100 per router and get them to manually locally download the config to their laptop, reset the passwords, and upload the new config.

Surely they could have done the above ???

Even Paris could have done better.

Yep you've hit the nail on the head - the guy disabled the password recovery mechanism which locks out access to ROMMON which would be the only way of traditionally recovering the hardware (the config is destroyed regardless). Basically this guy had the keys to the kingdom.

Whilst it is obviously crazy that all of this was entrusted to one guy (what if he died unexpectedly?) based on my experience of configuring Cisco equipment for corporates I would say it wasn't that unexpected for one guy (or girl) to end up with absolute control over the network. Suits seem to generally only care about the network staying up, not the particulars of how it is administered, until - of course - the s**t hits the fan.

The problem was that the sysadmin was paranoid.. to the point where he wouldn't even write the router configuration to the router's flash memory. (Yes, if the power failed the router would lose its configuration unrecoverably. Maybe it was safe from hackers but it wasn't safe for hardware failure.. stupid sysadmin!)

Apparently he didn't give anyone the password or write it down because he didn't trust them.

After all this, I'm confused as to why he's still pleading NOT guilty...?

Paris, 'cos she wasn't guilty either, just a little confused.

he probably used to post on TheRegister style comments pages as well

Its that all admin passwords are either GOD or SEX....

I wonder if they tried those?

Paris cause i reckon all her password are related to sex...

the Abu-Graib way.

http://www.catb.org/~esr/jargon/html/R/rubber-hose-cryptanalysis.html

>For "After initial confusion" read "After they turned off the Caps Lock"

ROFL!

Real keyboards don't have Caps Lock...

http://www.pfusystems.com/hhkeyboard/hhkeyboard.html

Why he is there now,..

Middle Manager: The network is unmaintainable while only you hold the passwords and configs. Please arrange to document these in a suitable manner for other staff.

Senior Engineer: No, I do not believe you or any of the other staff have the necessary skills to maintain this network.

[Lots of back and forth]

Middle Manager: Last chance, documentation or suspension.

Senior Engineer: Suspension.

[More waiting]

Middle Manager: Passwords and config please?

Senior Engineer: No

Middle Manager: Last chance, documentation or incarceration.

Senior Engineer: Incarceration.

[More waiting]

Middle Manager: Passwords and config please?

Senior Engineer: No

Middle Manager: Last chance, documentation or prosecution?

Senior Engineer: Documentation

[Try passwords]

Middle Manager: Proper passwords and config please?

Senior Engineer: No

Middle Manager: Last chance, proper passwords or prosecution?

Senior Engineer: Proper passwords

LESSON: All Senior Engineers are still only cogs in a larger machine.

Why he did it,…

Middle Manager: Please provide passwords to Junior to allow him to make changes.

Senior Engineer: Those changes are outside his ability to perform, and are an unacceptable risk.

Middle Manager: I don’t think your job is as complex as you make it out to be. Passwords please.

[Receive passwords]

Middle Manager: Junior, please make this network change with the passwords I have provided.

[Network crash – 36 hours for Senior Engineer to recover]

Director: What the heck happened last week?

Middle Manager: Senior Engineer made a mistake, despite being told it was not a sound change to make.

LESSON: All Middle Managers are cnuts.

Assuming the Hard Disks aren't encrypted, with physical access to the machines you can:

Windows:

Reset the Local Machine and Active Directory passwords by modifiying SAM

Extract hashes from SAM and crack the passes using Rainbow Tables.

*nix:

Reset the passes by modifying /etc/shadow.

Crack /etc/shadow to get plain-text passwords.

I'd put money on the HDs not being encrypted, its a drawn out, expensive process with very little actual ROI.

Who wants to bet this chap is one of, if not the only person managing the system. He probably set it up as well. This is a storm in a teacup, exacerbated by the City's unwillingness to properly staff their infrastructure.

I make the following prediction:

Now the dullards in SF have the passwords the fibrewan network will work no more.

Up until Childs handed over the passwords the network was working great, you just could not make any alterations to it. Now the city has the passwords some PFY will be given the job of making an apparently minor change that will result in partial or total breakdown.

Mark My Words, your Doomed SF!

What does SF stand for? SanAndreas' Fault? Send fail? Systems failure? Sentry fled? Soft Fu....errr ....geddit.