The Register® — Biting the hand that feeds IT

Comments on: Irate sysadmin locks San Francisco officials out of network

Get the vendor there. 

Posted Tuesday 15th July 2008 15:26 GMT

If they paid multiple millions of dollars, even as devalued as the dollar is right now, the vendor should be happy to send an engineer over to pop the cork on these systems. It should be easy for someone familiar enough with the systems who has physical access.

hahahahahahaha 

Posted Tuesday 15th July 2008 15:35 GMT

Happy

hahahahahahahahahaahahahahahahahahahahahahaahahahahahahahah

Strange... 

Posted Tuesday 15th July 2008 15:38 GMT

Pirate

I don't know why, but for some reason I find myself rooting for this guy. Give 'm hell Terry ;o)

(Obviously ignoring the massive disruptions and costs, not to mention the consequenses wrought upon the innocents... ).

Clearly didn't work hard enough in the first place 

Posted Tuesday 15th July 2008 15:38 GMT

To have a single point failure like that and have no possible way to restore if it went down in any way, such as this, just shows cost cutting as usual and incompetence.

I don't support what he has done, but it just demonstrates how often that companies, and cities, are not ready for failures or recovery.

No BOFH 

Posted Tuesday 15th July 2008 15:40 GMT

Paris Hilton

He should have read the BOFH and PFY's exploits before trying this. He may have been successful at keeping his job with their tactics.

Paris because even she can keep a job. Party girl is a job, right?

Interesting... 

Posted Tuesday 15th July 2008 15:40 GMT

Coat

I always did wonder what the real name of the BOFH was...

I'd say he was living up to his name 

Posted Tuesday 15th July 2008 15:41 GMT

Coat

but I know many a child who would be insulted if I did.

If only... 

Posted Tuesday 15th July 2008 15:44 GMT

our own government systems were this secure...

lol 

Posted Tuesday 15th July 2008 15:44 GMT

Flame

Hes not been busy at work eh? Looks like hes been pretty busy to me ;o)

<- Burn baby burn

@Christopher 

Posted Tuesday 15th July 2008 15:54 GMT

"It should be easy for someone familiar enough with the systems who has physical access."

If the sensitive data is encrypted (as it should be) then getting at it without the password will not be easy.

Robb is right, it sounds as though there is a procedural failure here.

Aren't BOFH articles supposed to be posted under 'Odds....' 

Posted Tuesday 15th July 2008 16:07 GMT

"city officials' fears that Childs may have an accomplice "... AKA PFY?

I Just Hope 

Posted Tuesday 15th July 2008 16:18 GMT

Paris Hilton

... that they haven't been cycling their backup tapes too closely, a week ago? Wonder if anyone's attempted a restore yet.

Paris for the obvious reasons.

@AC 

Posted Tuesday 15th July 2008 16:28 GMT

Flame

The BOFH's name, as anyone who has read his exploits will know, is Simon. Strangely, the name of the IT guy in one episode of "The Office" was also Simon.

Could they be related?

San Fookedcisco 

Posted Tuesday 15th July 2008 16:40 GMT

San Fran is about to be the cracking hot-spot of the world, how many are going to be moving their target sights over to that city right now.

With people trying to open access left right and center, there will be other cracks for people to slip in, and trojan the system. And it is right next door to silicon valley, that is a coup for any cracker worth his salt, the proximity and the side effect hacks that would be made possible are pretty tempting.

The firing looks politically motivated not performance related, are they saying his performance was so bad he couldn't pull of a stunt like this?

And who is to say the access codes he gave were not the right ones as he knew it, and the system is not already hijacked. As the dust starts to settle the damages on this one are going to be colossal.

One does sort of 'root' for the underdog, but unless he has a cunning and devious plan up his sleeve, he is in for some rough treatment.

Sysadmin is an ironic position 

Posted Tuesday 15th July 2008 16:40 GMT

IT Angle

Its kinda wierd being a DBA or sysadmin. If you HAVE done your job properly, you should be sitting back and reading the paper. That means the systems are running properly and you aren't running around putting out fires and fixing problems. You are retained for your experience and ability to plan upgrades or recover from disasters, not to be constantly busy. It isn't terribly clear in what way he was a poor performer, or was it that someone objected to his salary?

Should have been on Friday... 

Posted Tuesday 15th July 2008 16:56 GMT

Obviously this happened because there was no BOFH episode on Friday last, something had to make up for it.

If done according to the script, there is a piece of code that forwards "release papers" at the proper time. Then air fare to a proper vacation spot, with a money transfer.

Oh, well, scripts don't go according to plan always.

Can't be used as a BOFH story... 

Posted Tuesday 15th July 2008 17:28 GMT

Coat

NO cattle prods were used in the production of this story (Or passwords)... :D

Mine is the coat with the password in the pocket...

Once a BOFH, always a BOFH 

Posted Tuesday 15th July 2008 17:56 GMT

Thumb Up

Has anyone even bothered to lift the keyboard up and look underneath it? That's where you usually find the post-it pad with all the required passwords on. Or, they could just try P@55w0rd , guaranteed to unlock most companies' networks....

Reminded me of the proper way for him to have done it:

http://www.i-resign.com/uk/letters/dilbert_resign.asp

locked-out 

Posted Tuesday 15th July 2008 17:59 GMT

Gates Halo

Sounds like just another mICROsOFT update to me.

RE: Sysadmin is an ironic position 

Posted Tuesday 15th July 2008 18:00 GMT

Thumb Up

Interesting point in that I wonder if this is all a stunt by him to prove (now to prove in court) that he was in fact doing his job, and he was well qualified for it; albeit in this case now he is a criminal, vandal, and saboteur.

I have found that many logical and genuinely smart people are completely and hopelessly socially inept.

locked-out 

Posted Tuesday 15th July 2008 18:22 GMT

Sounds like just another microsoft update to me.

A bastard operator perhaps... 

Posted Tuesday 15th July 2008 19:31 GMT

... but NOT a bastard operator from hell! In any case, he's a tosser for 1) losing a (very) gainful position, and 2) doing it in a way that will disqualify him from employment for life.

Still, the lad's got style.

@San Fookedcisco 

Posted Tuesday 15th July 2008 19:44 GMT

Please stop. You can't rely on silly hollywood movies for all your knowledge, your nonsense is painful.

It's San Fran ... 

Posted Tuesday 15th July 2008 19:48 GMT

Stop

$126,000 is NOT that much in that area. The cost of living is extremely high.

More federal BS 

Posted Tuesday 15th July 2008 20:03 GMT

Thumb Down

Bullshit. Millions of dollars, my arse.

The only reason they're bumping up the supposed damages is down to the fact that they want to turn it into a federal case. If I remember correctly I don't think the feds will get involved unless the damages involved are over $50,000 for each occurance. I'm sorry, but refusing to hand over the admin password for a router doesn't cost $50 grand to fix in anybody's book - even via the "think of a number and double it, and add 20% contingency" style contracts the US government uses.

In any case, what kind of dumb-ass management allows a single administrator complete access to the entire network. Should be using centralised access based on securID or RADIUS anyway. While you're at it guys, sack the clearly neglegent operations manager.

Bleh.

I think ... 

Posted Tuesday 15th July 2008 20:38 GMT

... he's innocent. They're just too thick to log in.

Have they checked they don't have CAPS LOCK on?

A cunning plan. 

Posted Tuesday 15th July 2008 21:32 GMT

Paris Hilton

"One does sort of 'root' for the underdog, but unless he has a cunning and devious plan up his sleeve, he is in for some rough treatment."

The best cunning plans involve not getting caught. Failing that technique, we can at least hope he was devious enough -and his employers stupid enough, to fall into a trap he set that will exonerate him by wiping all the evidence, or changing it enough to make the case fail.

I find it hard to imagine politicians and civil servants being cluebatted up enough to cope with that.

sitting back and waiting 

Posted Tuesday 15th July 2008 21:40 GMT

Alert

interesting POV Damian. if you've done your job properly you can just sit back and wait for other things to come along. yes. no firefighting etc. but you can be sure that if you ARE in that position, then the PHB will ensure that LOTS of other work DOES

come your way - after all, you're good and can get the things done that the other idiots you work with cannot do. hmmm. some level of incompleteness in all tasks must therefore figure in your schedule?

Hmmm... 

Posted Tuesday 15th July 2008 22:00 GMT

Coat

I think that there is a BOFH at work here, or possibly a PFY... it's just not as obvious as it appears...

1. You have someone who for some reason isn't that well liked, possibly even the 'head' of IT. Just think about it.. he was being diciplined for 'poor performance at work'... let's face it, there isn't a manager born that actually knows what his IT department does. If someone knows that he is a poor performer (might coencide with being a nethack expert) you can bet your bum that it's because someone has been slipping words in the big boss' ear.

2. The entire system becomes locked, and there is only 1 person who could have locked it down (or so it seems)...

3. That person, after he gets arrested, cooperates with the police and gives up the password(s), but for some reason or other, it (they?) don't seem to work anymore... he must be lying... surely?

4. The police seem to think that there might be someone else involved as well, but can't seem to prove anything...

5. I'm just surprised that this poor fellow hasn't became stuck in a lift yet.

I wouldn't be that surprised that after this guy gets porridge some clever pimply faced youth somehow manages to crack the password and gets a promotion to top it all off ;o)

Non-Disclosure Clause 

Posted Wednesday 16th July 2008 01:56 GMT

Coat

It's probably in his employment agreement that he must never disclose his password. As you would expect it to be in any system admins agreement. I know its in mine.

Mines the one with padlocks on the pocket zippers.

Shit, I wish I made his salary 

Posted Wednesday 16th July 2008 02:34 GMT

Coat

I'd be pretty happy to just do my job (which is sysadmin) <sigh> I may be coming ungruntled... What a fool.

Call him Terry Rist 

Posted Wednesday 16th July 2008 04:13 GMT

Can't get the password voluntry? Charge him with being a terrorist, then the Yanks can legally torture him to make him talk. An orange jumpsuit, a plane ride to GITMO, a few goes at water boarding will make him spit out the pword.

That is why torture is legal, right, your are always garunteed to get all the "intelligence" you needed? It is 100% reliable?

Snooping around 

Posted Wednesday 16th July 2008 05:18 GMT

Alert

If you Google "San Francisco FiberWAN" and pore through the first 9-10 pages, you find a pdf for a feasibility study done in early 2007. What's interesting is in the middle of the 200 page document, one finds a discussion of the FiberWAN system.

Briefly, here's what it appears to be. S.F. has had fiber cable laid to many of it's outlying offices, to connect them to the datacenter. At the core of everything in the datacenter are several huge Cisco Catalyst 6500s, a SAN or two to hold most of the critical data, and connections to most if not all of the servers. So it seems like this is more than a WAN, it looks like it includes the datacenter LAN as well.

If Childs has locked out everybody from even seeing the router configs, and has also locked down or destroyed any and all backups thereof, somebody is in a heap o' trouble.

You can't just reconstruct all those VLAN (etc.) definitions from looking at what box is connected to what switch port.

(Disclaimer: I'm a software guy, I don't do wires, but this looks really icky).

Can you say: “Scapegoat”? 

Posted Wednesday 16th July 2008 05:21 GMT

Coat

I’m betting some VIP stepped on the fiber and kinked it.

I’ve yet to see a local government that didn’t treat their IT department like a redheaded stepchild and it would be no wonder the thing collapsed. I’m just counting the days until it happens where I work.

Never let suits in the datacenter…

@ Damian Wheeler 

Posted Wednesday 16th July 2008 07:25 GMT

Paris Hilton

@ Damian Wheeler said: It isn't terribly clear in what way he was a poor performer, or was it that someone objected to his salary?

HE was probably the one who objected to his salary. He wasn't getting paid anywhere near as much as many of the police and firemen in San Francisco. Where else can a policeman make $180,000 a year with overtime, then retire with his pension based on the last full year's pay, including overtime... at age 50.

Paris, because she understands money

Re: Sysadmin is an ironic position 

Posted Wednesday 16th July 2008 07:55 GMT

Similarly a manager's job (should) be like that too: they don't exist because all the result of them should be, for the worker, that there are no problems. It should be only when the manager isn't there you notice what they did.

Unfortunately, that isn't all that impressive a job (and it's the squeaky wheel that gets the grease), so most managers become PHB-like and ensure EVERYONE knows they are there. Politicians do the same thing (pass laws because that's all that people can see them do, so they do it).

Problem is that the more management above you, the more, as AC pointed out, your PBH will find work for you to do.

FUBAR from the start 

Posted Wednesday 16th July 2008 08:16 GMT

Dead Vulture

I mean, seriously, they figured out that this guy had done some nice empire-building, and nobody thought "Hey, let's get a warrant and put a couple taps in and capture his passwords, hm?" Sounds like a combination of a child named Childs and Law Enforcement Amateur Hour. Morons abound!

Dead vulture, because that guy ain't no kind of BOFH

@Andy Taylor 

Posted Wednesday 16th July 2008 08:31 GMT

Linux

The BOfH is (most likely) named after the author, Simon Travaglia.

http://en.wikipedia.org/wiki/Simon_Travaglia

@Snooping around 

Posted Wednesday 16th July 2008 08:31 GMT

Coat

Cisco boxes are the easiest thing to crack open if you have physical access. Take the words 'piss' and 'easy' and try and make a suitable phrase :-)

If the Yanks want me to do it, that'll be $126000 plus plane fare,hotels,food,car hire etc.

An absolute bargain in any useless,devalued currency.

Mine's the one next to the travel bags.

What they said v What they meant 

Posted Wednesday 16th July 2008 08:46 GMT

Paris Hilton

What they said: "city officials said they were making progress in regaining control of the system, which is up and running but inaccessible."

What they meant: "the guy didn't SuperGlue the power buttons, which we've now located and we can power the servers up and down at will. Still can't get in but they go on and off nicely".

What they said: 'Mayor Gavin Newsom, said he was "confident that [the Department of Technology] is doing everything necessary to maintain the integrity of the city's computer networks"

What they meant: "Hey, if we can't get in then probably no other bugger can. That means our systems are "integrous", right ?"

What they're going to say: "Welcome to Information Retrieval".

What they're going to mean: "Welcome to Guantanimo. One orange jump-suit each. Line on the left".

Paris, 'cause one of those suits must look good on someone, surely !?

Anonymous accusations? 

Posted Wednesday 16th July 2008 09:43 GMT

Since when did city officials have the right to smear someone in public without disclosing their names? Surely the first principle of justice is that a man has the right to know those accusing him?

Isn't it the one 

Posted Wednesday 16th July 2008 10:05 GMT

Coat

where he gets ate by the raptor whilst in his mac?

I'll hire the guy 

Posted Wednesday 16th July 2008 10:11 GMT

back on Tuesday as a consultant......

Send for the thumbscrews 

Posted Wednesday 16th July 2008 10:11 GMT

Sounds like Homeland Security's wet dream. Remember the hypothetical scenario that experts told us "never happens" - the "ticking bomb" that must be "diffused" as the media people have taken to saying? Well this may be as close as we have got yet.

Let's see how long it takes them to get him on a waterboard. But what if he's actually forgotten the password?

Ummm. 

Posted Wednesday 16th July 2008 10:50 GMT

"the system, which is up and running but inaccessible"

How do they know it's running if they can't access it?!

"the system, which is up and running but inaccessible" 

Posted Wednesday 16th July 2008 12:12 GMT

hmm... Looking at earlier in the article:

"Meanwhile his former bosses were unable to access San Francisco's new multimillion-dollar FiberWAN"

So, at the start the system was inaccessible. It didn't say it was down. So what has changed?

What they are saying is "We haven't done jack shit".

BTW, I agree that this BOFH was probably set up by his PFY, who will now receive a promotion, a huge pay rise, and exclusive access to the cattle prod. Meanwhile, BOFH gets time in the slammer to plot his revenge...

I also agree that it was probably triggered by the lack of a BOFH article this week. I nearly flipped out from withdrawal myself. Shame on you Simon, causing so much damage!

lol'd 

Posted Wednesday 16th July 2008 12:22 GMT

Go

Ha Ha frisco got owned you go brother!

@Steve Evans 

Posted Wednesday 16th July 2008 12:32 GMT

The green LED is on, obviously.

If the water's pouring, and if the password is 

Posted Wednesday 16th July 2008 12:44 GMT

"mmmmmppffffttyoubastarffffffmmmgarglemmpptffftttretchfffspitthtpttaaaarrrgh"

then he's really in trouble, 'cos they won't realise when he's coughed it up.

still getting paid 

Posted Thursday 17th July 2008 09:34 GMT

thats because the payroll records are on there.

not me ! 

Posted Friday 18th July 2008 01:06 GMT

if he gives out the password it means he did it.

Denial of Service Attack? 

Posted Wednesday 23rd July 2008 15:28 GMT

I thought that denial of service attacks were launched by an external agents.

Don’t Miss

email symbolStill sending naked email? Get your protection here

Security How-to Buckle your seatbelt, encrypt your bits

Google's Satan phoneT-Mobile G1 Google Android-based smartphone

Review Operating System 1, Hardware 0

Ubuntu teaser Ubuntu 8.10 - All Hail new Network Manager

Review The good kind of UI theft

OpenOffice_logoOpenOffice 3.0 - the only option for masochistic Linux users

Review And linear optimizing Mactards