"Chancellor Alistair Darling also said yesterday that the disaster actually strengthened arguments in favour of ID cards."

As expected.

If they did not have this data breach they would have had to create it. Now they have the best argument in favour of a national database and ID cards. They can even make them voluntary. If you have an ID and you are on the national database you have nothing to worry about. If you do not...

I bet that if we dig further we will find someone even above the "senior manager" pushing for this.

Have we found out the file format and what form the password protection took (and how the password was communicated) yet?

I have to say I am surprised that removing some fields from what is presumably a text file or Access database does not seem an insurmountable problem to me, but in this exciting, outsourced, friction-free world in which we live running an AWK script over a text file is probably a £10K call-off on a contract.

"the disaster actually strengthened arguments in favour of ID cards."

here's a few observations :

1) how not to secure my data .com

2) MSFT security -- not

3) more humans = less security

and

4) let's form a committee to hire a big consulting company to spend big money to reinvent the wheel

No doubt whoever runs the HMRC system charges £5k for a custom SQL query that any decent PFY could do in 20 minutes, given sufficient liquid incentivisation...

The government is trying to offload the blame to "junior" staff. My question is, why if they were handling 25 million records were they junior staff?

The very fact is, no one at junior level should be expected to handle that amount of data of that importance so regardless of who was at fault it's still senior staff to blame for putting junior staff in that position in the first place.

"A junior official from the Child Benefit IT department has reportedly been suspended and sent to a nearby hotel with a minder in order to protect his or her identity."

As long as 1 in 25mil peoples identity is safe. When do we all start getting our own minders?

<quote>If they did not have this data breach they would have had to create it.</quote>

And what makes you think that this wan't a contrived Security Breach for this reason????

When this story broke I wrote to my MP asking for a bunch of details, including the file format and the current password protection policy.

I'll forward onto El Reg any info I recieve

"A junior official from the Child Benefit IT department has reportedly been suspended and sent to a nearby hotel with a minder in order to protect his or her identity."

First impression: They lost 7.5 million of our identities but the junior official's identity is protected by letting them stay in a hotel with a bodyguard at the taxpayer's expense. Brilliant.

Second impression: I'm not actually at all vindictive about the junior official in question, since we all know it was neither their decision nor their fault. Some people might not be of the same mind but I can't imagine anyone wanting to commit actual physical harm to them. What the minder is really there for is obviously to make sure they don't spill the beans to the media about who was actually responsible. Now that is, quite frankly, downright creepy. If it was me I would be saying "Fine, if you're going to sack me, sack me, and get your goon out of my way so I can start looking for a new job."

without it being "too expensive".

I think we need a technical explanation for why someone couldn't just grab the fields the NAO needed.

Heck, I can buy a CD-ROM on the high street with names & addresses from the electoral roll, and that must be a comparable number of records.

Select Name, NIN, anything else of no interest to Russian mafia boys

from BloodyHugeDisk.Claimants

Telling us they couldn't strip out the innocuous stuff without calling in Arfur Andersdone (with a silly accent over the second "d") and paying them big wonga sounds like a gormless bluff.

The government has been selling our details to outfits little better than criminals for several years and continue to maintain their right to do so.

Don't believe me? All you have to do is set yourself up as a parking enforcement company, sign on with the SIA (Don't worry - they won't do any checks on you. They didn't on those 5000 illegal immigrants, did they?)

After that, all you do is collect a few likely looking car registrations and send them with a cheque to the DVLA. They'll give you the registered keeper name and address for a fiver a time. Sweet.

He just about predicted that crap like this could and would happen in his book "Plundering the public sector". He was so right all along.

Go David!

"the NAO asked for the data be "desensitised" but this was rejected on grounds of expense"

How hard is it to run PGP over a file before sticking on the CDs?

Well, it would take the education of senior politicians - which can be hard.

More importantly it would take educating senior politicians that MS tools and (laughable) security are not all they're cracked up to be - and since MS bunged the British Library £100 million ( http://news.bbc.co.uk/1/hi/technology/4402442.stm ) that's *very* hard indeed!

"Alistair Darling also said yesterday that the disaster actually strengthened arguments in favour of ID cards"

did he now?

fucking how's that, then?!?! i love the way he can just come out with crapola like that and not have to actually back it up with facts or even theories. at the end of it all, they didn't take appropriate steps with sensitive data. it should have been crypted to buggery, 4096 bit stuff. delivered by armed guard, if necessary.

GRRRR!!

Having worked with government departments for many years I would bet my right testicle on the fact that it was an access database which was being used as a local copy for that office's use. We all know how secure they are...

If the chancellor is telling the truth - that only one junior official was to blame - that is much more worrying that the Tories' version.

How is that a junior official has access to the entire database, can copy it onto CD and, presumably, put it in his pocket (rather than the post) and take it home to sell to anyone he wants to? With no audit trail or management oversight. If this is what happened, heads should roll.

If a senior official was involved (which is likely) the chancellor is lying or misinformed and even more heads should roll, including his.

Poor old Steve McLaren looks like a hugely successful mastermind in comparison with this sorry lot.

Working in baks for many years, if a similar incident had happened, even of 1/100th of the magnitude, they'd be frogmarching everyone that had his figerprinted on this f-up out of the building with a binbag, and not a lot of sign of full pay and hotel rooms.

I'm glad people are now starting to think about the only victim in this fiasco. The poor guy or gal who sent it out on the 18th of October. Was this or was this not the last day of the postal strike? Seems to me that internal unregistered post was the only option available on the day.

"Has a minder in a nearby hotel" - suicide watch more like.

From a technical point of view, the operation is very simple

However, as mentioned, if IT is outsourced, this sort of one-off is not usually contractually defined, so you have to launch the procedure for non-standard work, which is normally slow and painful (in my experience it almost always involves having to explain what you want to non-technical people as a first step) and leads to all sorts of farcical situations, or gets bypassed

To be fair to the outsourcer, if it's not in the contract, why should they do it except following contractually agreed steps and getting paid for it, same as any other contract?

Kirsty Young's next guest on Desert Island Discs is Alistair Darling. However, the programme will be shorter than usual because he has lost four of the eight records

From the bbc story:

“He said the NAO wanted only limited child benefit records but was told in an e-mail from a senior business manager in March that to remove more sensitive information was too costly and complex.”

Right. Costly and complex – it’s a database right – so a simple query could have been written – may have taken time to run if it had to pull out all that information – actually the query might have been pretty complex given that im hoping the table structure was set up properly with all the correct joins and normalisation etc (though I wouldn’t hold my breath). Also – 2 cds? Whats that - about 1.6GB of data? Sounds a bit small for 25million records considering when I worked in the hospital the database I used that held the data for the patients that had been seen in the hospital – considerably less than 25 million I might add – was sitting at around 2GB when I left.

Im also assuming that they will have information officers whose job it is to respond to requests like that – who may be familiar with a little known application called crystal reports.

Though what happened probably went something like this:

Scene 1 - HM customs & revenoo office, basement where the IT people are kept

Non-Existent High Level Civil Servant (NEHLCS): “Ho there laddo! A mate of mine at the Audit Office,old school chum actually, jolly good sort [insert long winded anecdote about old pull-my-finger Smythe]… Well he asked for a bit of information that we have. I’ll have my secretary send you the details shouldn’t take a clever lad like you long eh?”

Scapegoat: “Umm… well maybe. Depends what it is they are looking for. Oh and could you sign the authorisation for me to access the data please as well – you know.. for the security audit thingy we are supposed to do…”

NEHLCS: “Well he did say its dashed urgent. No need to bother about that security tosh now – don’t worry I will do it later. You just get that info he wants and send it over to them toot sweet.”

Scene 2 – 20 minutes later

NEHLCS: “Well laddo have you managed to get that info I asked for?”

Scapegoat: “Umm… Just writing the query now. Its actually quite complicated because…”

NEHLCS: “Argh! Non of your technical mumbo jumbo! I don’t understand that rubbish anyway! Can you not do it faster?”

Scapegoat: “Not really boss. It all takes time, and because of the amount of data it will take a while to run when it is ready anyway”

NEHLCS: “Bugger. He did say he needed it soonest. I know,” (Self satisfied smile)”Send it all.” 

Scapegoat: “Umm… All of it? Are you sure? I don’t think that we are allowed to do that…”

NEHLCS: “Nonsense! We are the Government! We can do what we like – it is just sharing information anyway. Will be a lot easier when we have that big central database” (Scapegoat shivers and turns pale) “Stick it all on a cd and send it down to them. Let that git Smythe get the stuff out of it himself. Always was a lazy bugger.”

Scapegoat: “Umm… send it how? And it will take time to burn it onto cd anyway”

NEHLCS: “The post you daft sod! How else! Actually better use that courier service we use – stick it in their bag – royal mail are probably on strike again. Ruddy socialist slackers!”

Scapegoat: “Recorded delivery right you are”

NEHLCS: “Oh no! we are trying to save money here. It will be fine in the normal bag”

Scapegoat: “Riiiiiight….lf you could just sign this form saying that you have authorised a copy of the ENTIRE SYSTEM….”

NEHLCS: “No time! Of to see the minister for a few ummm… policy thingies. Pop it in the post there is a good chap.”

Scene 3 – A month later

NEHLCS: “You there! What happened that bit of data you sent to the Audit office! They haven’t got it yet!

Staffer: “…”

Scapegoat: “Umm… actually that was me. I posted it like you asked”

NEHLCS: “Well they have no record of getting it. Where is the tracking slip”

Scapegoat: “….”

NEHLCS: “Well?”

Scapegoat: “you instructed me to send it by normal mail”

NEHLCS: “…”

NEHLCS: “Send it again – This time recorded delivery. Those buggers at the Audit Office probably lost it.”

NEHLCS leaves basement

Scapegoat: “Boss. The boss had me do something and it stuffed up so now im telling you – we’ve lost a copy of the database.

IT Boss: “*@!£%^&*&I*UO(*U”

Ok, so nobody has yet specified whether the were "ordinary" CDs or not. But you'd have to go some to fit 25 M records onto 2 CDs. Rough calculations seem to suggest that each record would have to be between 50 and 160 bytes ( this is back of the envelope stuff ).

Presumably, since it wasn't encrypted, it also wasn't compressed so a small record with name, address, NI number, DOB, bank details might be :

fred bloggs,23 the road,truro,cornwall,tr5 4tr,ab123456b,020304,12345679,010163

That's 79 bytes - many records would be bigger than this and that's just CSV ( no allowance for file format /separators etc ).

Did they /really/ get all the data on 2 CDs ?

On a less cynical note - don't they have the intarweb in Govt ? If someone wanted a gig of secure data off of me I'd fire it over a VPN or something ( after encryption ! ).

There's no chance these muppets will /ever/ be able to run an ID card scheme securely !

Obligatory speculation : CSV file, zipped with password. This is very, very common when shifting bulk, high value, personal data to/from outsourced functions or external organisations. Sad, but true*. But who knows how these muppets go about things.

Not so speculative part : Additional expense ? For unticking two fields (sort code, account number) in the database table export wizard used to dump the CSV (or whatever format) file ? At worst, setting up a duplicate query with those fields removed ? Fuck off. And there simply isn't any way it would have been much more complicated than this** no matter what's on the back end. (And it will be SQL Server or Oracle, I'd guess Oracle 'cause HMRC (or their outsourced pixies) have some experience in Oracle data warehousing)

I've seen this done (and done it myself) a hundred times, and I don't recall it ever being a chargeable extra.

Definitely not speculative : If a "junior official" is in a position to make such decisions, and access such data without some managerial supervision, then whatever else the gov might claim, HMRC really do have *serious* systemic problems with their IT, security and management processes, this is beyond question.

Grr!

*In which case your whole 'security' policy is largely predicated on the integrity of the physical transport process.

**OK, it's a little more involved in Oracle with no third party tools, but come on, it's what ? Seven lines of sqlplus ?

"That's why we don't like seeing work off-shored. It raises all kinds of security issues about sensitive data and the worry is that it could get into the wrong hands."

I may be from 'daan saf' but Tyne and Wear is hardly off shore!

Anyone else think the 'unions' are just using this to bolster their own self importance?

HMRC has for years been cutting staff and costs, for one reason or another. It has been for years the practice to de-skill tasks so that they can be given to poorly paid and badly trained temporary staff or E grades.

This whole process has been driven by successive chancellors, such that now most of the people who are qualified to do it have too much to do.

Knowing many people in HMRC, tells me that most staff are though demotivated, surprisingly, conscientious and hard working. That HMRC works despite its management, not because of it, and that they despise their politically appointed senior management and their advisors who know nothing about the practicalities of running HMRC. So in that they are probably no different to the vast majority of us who work in large organizations.

Why will it happen again, well, the politicians will insist on a knee jerk reaction, rapid changes to procedures which will be implemented by a new management that knows nothing about the business. With luck people will be trained, but the training won't be kept up because of cost, and so on it goes.

The general standards for handling sensitive data in government are actually very good, and easy to understand and far better than anything you see in the private sector.

"Chancellor Alistair Darling also said yesterday that the disaster actually strengthened arguments in favour of ID cards."

What, so the Gov't can send even more of our personal data around the country in a completely unsecured manner?

Also, what kind of screwed up database system have the HMRC got running, that simply removing the fields that they don't need from a data dump involves a prohibitively high expense of time and manpower???

Most (Microsoft) Office monkeys could do this in Excel, let alone the kind of over-priced Oracle-type monster that they've got, so why can't... wait, nevermind.

The easily could of used DVDs. They probably wouldn't know the difference.

... To filter the data.

[sql]

begin tran stripoutconfidentialdata

select ninumber from bigconfidentialchildbenefittable

commit tran stripoutconfidentialdata

[/sql]

Export to CSV and encrypt.

That will be £10,000 consultancy fee please...

"Chancellor Alistair Darling also said yesterday that the disaster actually strengthened arguments in favour of ID cards. "

I'd be very interested to read how he could elaborate on this one. El Reg to call him ? Worth the cost, IMHO.

Even in my banana republic, the press would collapse from laughter on this one ...

"Chancellor Alistair Darling also said yesterday that the disaster actually strengthened arguments in favour of ID cards."

He's a bigger muppet than he looks if he thinks anyone is going to swallow that line.

I also have a lot of sympathy with the union position and I don't like off-shoring of sensitive data, but when the unions come out with statements like this:

"That's why we don't like seeing work off-shored. It raises all kinds of security issues about sensitive data and the worry is that it could get into the wrong hands."

Didn't the big fucking irony alert go off in their minds?

Now their opponents can simply argue that an Indian data centre might lose the data, but it'll definitely be cheaper.

Without knowing what exactly was on the CD's (suspect probably DVD's, but we all know how good the media are at reporting what the actual facts are, but don't let the facts get in the way of a good story), I honestly don't know what all the fuss is about in regards to Identity fraud.

Given how much information you have to provide now adays, information in general circulation is relatively easy to come by, but when fraud has it's rewards it is generally because people don't follow process, and get lazy.

Now, ok, in this case some data has gone missing. I suspect that someone is going to come back from holidays and go, oh, oops, what I am I spose to do with all these DVD's I have three copies of. Probably not the smartest way, to have transferred the data, and well, suspect that they were too lazy to bother encrypting the data. But really, the hype over this is just stupid.

Given that all of us who are on the Child Benefit records have now been advised to check and recheck our statements and credit history, are HMRC going to cover the cost of getting credit reports from Experian and Equifax for everybody affected? Only fair, I would say ...

I've worked as a contractor for several government agencies, and I've seen this thing plenty of times.

I am allowed to send restricted data (which I assume this data is classed as) through the Royal Mail without any encryption required- simply because the Royal Mail is a 'trusted' organization and the Government must support it.

Laughably if I want to send confidential data (the next security level up from restricted) it can still be sent unencrypted through the mail but must be secured inside two envelopes. No, seriously, that's it- possibly when thieves open the first envelope they get confused and think its a game of pass the parcel...

I assume that the internal mail system of the HMRC is also 'trusted' and that the data is only restricted so technically no-one from the IT side has done anything wrong.

If someone was to use the freedom of information act to request the HMRC's policies on handling and transferring different security levels of data they could confirm this... *nudge nudge*

what toss.......

thats the problem with so called clueless"IT experts" , read "the reg" and think that they are Alan Turing and Bill gates rolled into one! -

"it was probably an access database"

id like to see a access database loaded with 2m million anythings and then still be able to export it to disk!

its probably same guy making this statemnt, who drew up the scurity and file and data transfer policy - stuff it in a jiffy bag and courier it up the M1 gov'nor...

otherise, well its either that or get a ad-hoc request through the GSI.

as for the lowgrade uncivil servant, he was probaly just following procedures - why dont we see a senior uncivil servant or a minister taking a jump of waterloo bridge?

It seems to me that despite the outrage that we all feel about this incident, absolutely nothing is going to happen to put it right. The government will sit tight and wait for the fiasco to blow over unless public pressure is so great that they're forced to do a U-turn. Are we there yet? Nope. So nothing's gonna change.

I've never wanted to chuck a rotten tomato at a politican before but right now Mr Brown deserves an entire truck load.

Really? I wouldnt have thought of that. Thanks for clearing that up.

because we thought he'd run away, and that nasty Press gang would find him and make him blub up and implicate us big boys.

Is anyone counting how many government ministers are trotting out this preposterous story about "self-empowered" junior officials? My guess is they would have tried to pin it on the office cleaners had it involved anyone other than the NAO.

A lot of commentators are asking why a junior official had access permissions to the entire database. Perhaps because that is his job?

I suspect that his role is to support a database application. He may have amazing technical skills and years of experience, but because it's a hands on role, he's "junior". Senior staff don't get there hands dirty, they go to meetings, think about blue skies, rub shoulder with politicians and issue instructions like "Send all our data through the post on a password protected disk because encryption software and secure networks are too expensive" and "I want it done by lunchtime!"

Would you really want to give such clueless senior civil servants full access to the data?

I never thought anything else.

The NAO didsn't contact him direct and say "Oi mate, bung us a copy of the entire database on two CDs", did it?

bung the magic phrase into eBay:

The Missing 2 disc special edition

and you can see the cunning CD swiping swine have disguised 25 million kiddie benefit claimant addresses as a Ron Howard / Tommy Lee Jones / Cate Blanchett double DVD injuns 'n' redemption romp.

Special edition, indeed.

Brown and Darling are taking full advantage of the public’s ignorance when it comes to the workings of the civil service. Their use of the word “Junior Official” is highly misleading. I’ve worked in the Civil Service since the late 80’s and I’m not entirely sure what a Junior Official is – it’s not on any pay scale I’ve ever read. I can only assume they meant someone who is not a Senior Civil Servant (SCS). If this is true, of the ½ million civil servants in the UK – 4000 of them are SCS. That means 99.2% of ALL civil servants are “Junior Officials”. The wording has clearly been used to give the impression of an office junior – an incompetent temp. And in this case, and what is more worrying, a rogue one. This is very misleading. It is simply not possible for an individual to act independently and download an entire database from the Child Benefit system (a “live-load run”). This requires very special permissions from management. There is no way this individual is guilty of acting independently – it’s impossible. If he’s guilty of anything, he’s guilty of trusting that the courier service, TNT, would do their job. Nothing more. Hardly a sacking offense.

Last night's Newsnight said that the NAO asked for anonymised data, but the Revenue refused as it would be too much work for their IT support run by...

...EDS!

Also, are we sure that Alistair Darling isn't a Marxist out to overthrow the capitalist system? In just a couple of months he has completely destroyed the nation's trust in banking, something generations of Trotsykites had never managed.

A few years ago a student friend had a summer job working with data that was either census or electoral register (I forget which now). His job, scheduled to last 6 weeks, was to go through the data finding households with more than 5 adults.

Being a computer science student, he realised that this could be done in a matter of seconds with an awk script, or similar. But if he had done that, he would have been congratulated and then fired since there was nothing else for him to do. So he spent the allocated 6 weeks doing it manually, as instructed.

This wasn't HMRC, but I imagine that the same sort of thing happens in most large organisations.

This is government - think mainframes, think generations older, think VME, IDMSX, COBOL!

All these "just write a query" idi0ts have no idea!

The Government is getting what it deserved - a level of service commensurate with the resources allocated. An analogy with monkeys and peanuts springs to mind. I used to be a civil servant, latterly in the old C&E and am well aware of the cuts in staff and the direct and indirect effects.

The joke is that senior management always meet the targets for cutting staff and at the same time produce an "Assurance" that the Revenue is being protected and that all is well. To do anything else would put a bonus at risk. The remaining staff have to carry the can for the top level incompetence.

Isn't that the title given to someone who is subsequently found dead (under suspicious circumstances) and is later said to have committed suicide?

Why didn't he run the script, then just *look* like he was doing it manually & give them the info piecemeal?

If enough of us got really worked up, we could chuck these toerags out. However, it requires that we actually do something other than puff impotently on websites such as this. As George Mikes said "Other countries have revolutions, the English have satire"

Or as the Governator put it in "Red Heat"

"Pud dee politishuns up against dee wall and shute dem"

James Belushi - "No; the lawyers wouldn't let us"

"Shute dee lawyers furst"

"Being a computer science student, he realised that this could be done in a matter of seconds with an awk script, or similar. But if he had done that, he would have been congratulated and then fired since there was nothing else for him to do. So he spent the allocated 6 weeks doing it manually, as instructed."

Surely the correct approach in these situations is:

1. Write the script and run it

2. Spend rest of the summer getting paid to sit back, play Minesweeper or (if you're lucky) browse the web.

3. Hand in your results (and if you're feeling nice, the script) at the end of the contract, having been paid the full amount.

(For career bonus points, shave up to a third off the boss's expectations. They will be amazed, you will get 4 weeks pay for a day's work; the next guy still gets room to "improve" the process further still, everyone's happy, right? For geek bonus points, spend the rest of the summer benchmarking and optimising the code until it's mathematically impossible to tighten...

We've all been there, surely?)

My expatriate brother in France turned 65 a couple of years ago. So he told the DHSS at Newcastle his current address for the first time in years, to get his pension.

His comment on the current fiasco: "It was notable that I received the El Gordo scam when ONLY the DHSS had my new address."

By accident or design these systems will leak data. This time the sheer scale of the leak makes it difficult to cover up, but steady leakage has its dangers too.

I don't think it would be as easy as a "filter" as some people have suggested. If you are doing an audit, you might want to see correlations in the data, such as how many people with the same name in the same house claimed more than x.

To do that you would have to change everyone's names, but not randomly, and have to be sure you ended up with the same statistics (i.e. don't change all Smiths and Joness to Williams). And the same with NI numbers, and addresses, and how to you make the postcodes anonymous and still useful in a geographical search, etc.

Sure there is a sliding scale on how much work you can do on this, but to do it right (and I can imagine the civil service being a do it completely right or ignore it place) would be a small project in and of itself.

All of this would mean that you can pass it onto the NAO and not worry about the security clearance for their DBA. (We face similar problems when sending copied of our live system to the vendor when trying to debug problems.)

Not encrypting - criminal. Saying making it "desensitised" has a cost - completely true.

MR said "...but the Revenue refused as it would be too much work for their IT support run by......EDS!"

You're wrong dude - the story coming out is that it was supposedly *too expensive* for HMRC to - as Nick Brice points out - do a task that was outside of the agreed contract. So by trying to do it on the cheap, it's now costing them more! At least the HMRC head guy had the cojones to jump ship.

Why is EDS involved when they're supposed to be with Cap Gemini? Or are HMRC just too useless to grasp the concept of totally replacing one supplier with another.

Interesting office comment - anyone pick up on how fast the civil service unions managed to spin this to their own ends, almost if they knew it was going to happen? Now I find that very suspicious given that my experience of civil service unions (I used to be in one!) is that most of them couldn't find their a*s with a map and a big sign pointing at it.

Not that I'm suggesting for 1microsecond that they arranged it, but it's very convenient timing for them.

I feel sorry for the poor staff stuck "in the trenches" receiving abuse day-on-day. Us members of the public are not the only victims here, (although hopefully the two disks will turn up - [temporarily] lost in the post)

Ill second that, Ive seen the same procedure in use with gov restricted cd's. Only we were told when we queried it that we had to use two envelopes for restricted items, and the inner envelope was to be addressed to ourselves, so in the event of a breach of the outer, hopefully they returned it to the address on the inner. We used to encrypt the contents against policy and email the pass to the recipient (ie by a independent means), but only because we cared slightly more about our reputation than the usual EDS mob.

Of course what would happen in practice is that anything that caused the outer to get ruptured would do the same to the inner envelope.

Government take on the chocolate fireguard if you ask me...

Think what you like, but the points made about 'just' executing a query are still valid.

Even if you have to fire up Microfocus COBOL workbench to do it, the principle still holds.

You obviously don't have much exposure to COBOL, a language which was designed to make writing and executing just such queries easy *. A system I worked on three years ago on ICL minis (yes, original ones at that) and which was largely written in COBOL sometime around 1976, was happily able to export a selection of fields to a CSV file for transfer to more modern kit.

In fairness COBOL programs often take a while to write because you need some downtime to recover from the psychosis inducing whitespace and indent rules and the enforced boilerplate. But the principle is very much the same.

The only question is why doing this should incur unacceptable delay and/or cost. Most likely because this counts as additional work under the outsourcing contract, the terms of which (esp in government departments) are so nit picking as to induce hysterics in all but the most fearsome contract experts, and are in fact designed to exploit just such situations.

This is a totally separate issue from the technology involved, and part of the "systemic failure" that Darling et al are so fiercely denying.

*Although weather that design goal was met is still a subject for some often strenuous debate, and I can easily imagine several COBOL coders have just spit coffee through their noses.

I contacted newcastle a few years ago and soon after got scam letters for the Spanish national lottery too. Hmmmmm.......

A jolly good piece of free encryption software is available at:

http://www.truecrypt.org/

So easy to use even a Senior Official could.

No - we just build a large ship - a "B" ark if you please*. Load it up with all the politicians, throw in all the PHBs, senior management teams, outsource account managers, client executives and third party consultants and every other golgafrinchan we can find and set it going off round in circles round the Atlantic.

We could even leave instructions on how to turn the autopilot off, set up lots of hidden cameras and have ourselves a nice reality TV program. We could have minutes of fun watching the useless sods form committees and steering groups (sic), project teams, publish newsletters, set up war rooms etc whilst they try to decide what to call the project to find someone to open the instruction manual.

In the meantime we would then get on with the job of delivering a proper service to who-ever our particular bunch of customers are.

* to Douglas Adams, thank you. Remembering that those in charge are just a load of useless bloody loonies is sometimes the only thing that gets me through the day.

On the one hand, take this with a pinch of salt because it's second-hand info.

On the other, like many governments and security forces around the world, HMRC do (as far as I can tell from some Googling) use Lotus Notes/Domino. Because out of the box it's roughly 1000 times more secure than anything MS have ever produced - ** when the database is on a server ***.

I'm told (as I say, second-hand, mate of a mate who works at HMRC) that the disks had two Notes databases on them.

Now, if you encrypt a Notes database when you replicate it locally and then send on the ID file that was used, or generate one-off encryption keys and send them on to be imported into the recipient's ID file then you have one damn secure database. Current version supports 2048-bit keys.

If you set "enforced ACL (access control list)" then you have something that looks secure but can be opened with a little effort by anyone with a Lotus certification.

If you don't do either then your last best hope is that the crims in posession of the disks don't know what a .nsf file extension is and/or don't know where to get a copy of Notes from!

Take this as I do - To Be Confirmed - but adds a little interest to the mix.

Speculate no more:

the BBC have published some of the correspondence and emails:

http://www.bbc.co.uk/blogs/nickrobinson/Informationrelatingtochildbenefitdata.pdf

It refers to a previous request which was sent in 100 zipped files on 2 CDs, and to send the password(s) in a separate email.

At this point you hope they used winzip v9 or later (with AES, or 7zip) and non-short password(s). Winzip v9 gives a max effective key length of 160 bits whether 128,192 or 256 bit AES key length is used (uses HMAC-SHA1, 1000 iterations, in a key derivation function -see RFC2898). Crack programs available will struggle with anything more than short password when faced with a zip encrypted using AES.

Unfortunately, it's more likely that Windows XP own built in zip was used which I think just uses the old (non AES, more easily cracked) Zip 2.0 compatible password protection.

It is just as likely that he was downloading a file generated by some pre-existing batch process. As child benefit has been around for over 30 years it is quite likely that there was already a job set up to extract this sort of information from the system. While it is easy for IT professionals to think of numerous ways that the data could have been cleansed after the download it is important to remember that the individual in question was probably a lowly qualified administrative officer who was probably following a set of written instructions parrot fashion. His breach of the HMRC procedures was probably not that he was acting outside his remit by downloading the data and sending it out unencrypted but that he used the internal post rather than registered mail as the mechanism. As there were postal strikes around the time that the event occurred he may even have been give verbal instructions by his manager to use a non standard route ( note to all civil servants advise your supervisor in writing every time you depart from set practise so he/she can not deny the fact later). The fact that this young and probably lowly paid individual ( say £13,000 p.a) is being scapegoated by politicians and the senior managers in HMRC for what is really the failure of a poorly designed and implemented IT process just goes to show how low some of the top people in government are prepared to stoop nowdays. To describe the official round of blamestorming and buck passing as pathetic does not really start to give full expression to my contempt for these people.

"We used to encrypt the contents against policy and email the pass to the recipient (ie by a independent means), but only because we cared slightly more about our reputation than the usual EDS mob."

Actually - as an EDSer (although nothing to do with the DII or Govt businesses) I'd just correct this. EDS' policy is pretty damn clear on what you're supposed to do with this sensitivity level of data. At the minimum it's to be "safeguarded" (dumb US speak for encrypted I guess) and "delivery to the it's intended recipient only must be ensured" (proof of delivery and pack tracking). I've heard of folks encrypting a file, and then doing again (with different pass phrases obviously), then phoning one passphrase to the destination person, and only giving them the other when they confirm that they got the file. Think that's a bit anal myself. :)

So if it had been one of my colleagues responsible, then the mandated punishments are disciplinary hearing, sacking and probably a spell in court.

But, and I realise that there's folks at El Reg who'll either dispute this, on this occasion the fault lies 100% with HMRC, so please don't try laying the blame at EDS' door. :-P

Still no idea...

Microfocus Cobol - on a mainframe?

I'm talking systems that are 20+ years old, on big (clue "VME") mainframe boxes, with all the controls and constraints that brings - develop, test, sign-off, code control, PTE test, sign-off, schedule, run... easy to charge a few man-days for even a simple scan in that sort of environment.

Never worked on the system in question, but I have worked on similar, and know how it is - all too well!

Even if it's an antediluvian mainframe system, running off an extract dataset with just the fields you want should take a decent techie oh, all of five minutes.

No procedures to follow, no source code versioning, no compile even, if you're lucky, and certainly no testing required, just bash off a quick little number in SAS, or Easytrieve or whatever you have to hand, and bish bash bosh, loads 'a' data wiv the dodgy bits left aaht.

There are different ways of treating name & address data to avoid DPA issues, by the way.

The obvious way is to encrypt the stuff to death.

The less obvious way is to use a commercial software package that obfuscates specific fields with gobbledegook; point it at the names and it'll "intelligently" change them so it's no longer possible to identify the people involved, but to a human eye they are still obviously names of some sort. Point it at the addresses, and it'll do a similar job.

In this case, it seems the names needed to remain unchanged, but the bank details could have been discombobulated by software. Of course, setting up a run of this sort of software is less than trivial if you want to do a good job: I wonder if this is what was being quoted as "too expensive"?

Within the pdf the email dated/timed 13 March 2007 13:11 seems a bit strange - "I hope you make sense to you than us however".

But what is the URAC mentioned in that email? In context it appears to be record/field layout/descriptor but I've never seen that term before.

BTW When NAO state the HMRC Process Owner "was a copy recipient of an email", please note that they do not state he was ONLY CC'd on one email in the entire email exchange.

The last statement limits the Process Owner's involvement, but the first leaves involvement undefined while implying a limit. (I've read a lot of Civil Service reports.)

According to the BBC 6 more CDs are missing. These contain audio files of customer complaints. It seems that they were shipped by TNT but never arrived.

TNT apparently said "it was impossible to say whether the CDs had ever entered TNT's system."

Err Excuse me - are TNT a courier service or are they a bunch of cowboys. How can a courier company get into a situation where they cannot say if they've ever actually picked something up. Sounds like they are as useless as the person who decided to use them to ship confidential data. Remind me never to use them if they are so clueless

Maybe all these "lost" CDs are actually just sitting on a shelf somewhere waiting to be picked up?