Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Archive for March, 2006

Government widens click-use licence.. but wants more money from its users

Friday, March 31st, 2006

The Office of Public Sector Information has widened the organisations that will be able to license “core data” through the click-use licence, according to the What’s New page:

From 1 April 2006 the Click-Use Licence is being extended to cover information produced by public sector organisations such as local authorities, the NHS, police and fire services.

However, while that’s welcome – because click-use should mean that you can use data licensed that way as you want, without worries about whether it’s for commercial re-use, private enjoyment or (even) undercutting an existing commercial service – there is a fly in the ointment. Two, actually.

First, it will be voluntary for these organisations to provide data in this way. They don’t have to do it.

Secondly – and this is perhaps the killer – local authorities (which have the potential to provide so much mapping and postcode data which could be patched together, because in many cases they originate it) are now under pressure to bring down council tax by whatever means they can; which includes selling information. Obviously, if you’re a councillor considering the best route to keep the council tax bill down, one thing you’ll probably cross off your list quite early is making data available for free. Anyone who knows an argument to the contrary, please give it..

Media: why Norwich Union had to map the UK all over again

Thursday, March 30th, 2006

In today’s Guardian, Why a £5m mapping project had to double up on data explained why the restrictive data practices of the Environment Agency and Ordnance Survey cost Norwich Union £5m when it decided to draw up a map to assess flood risks a few years ago.

Did the Environment Agency have the data Norwich Union needed? It did. Would it make it available? Ah, that’s a different question. Which meant that Norwich Union spent £5m (which has to come out of its customers’ pockets, in the end) to create a brand new flood map – essentially duplicating the work that already existed.

What’s the cost of sales and marketing in the government organisations?

Sunday, March 26th, 2006

Here’s a thought. We have our list of target organisations.

Now, could someone who has a few moments download their annual reports (really just the accounts) and look at their cost of sales and cost of marketing?

By our argument, the cost of sales is an unnecessary overhead which could be cut virtually to zero (to be replaced by cost of distribution of electronic data formats).

Cost of marketing would still exist: the Ordnance Survey, for example, would still want people to be aware of its different offerings, and its position as a prime supplier. (You can disagree – make the case in the comments.)

Does subtracting the cost of sales (and perhaps legal costs…) from the organisations’ costs substantially reduce their costs, and thus mean that it would be easier for them to “trade” without having to make sales? It’s a thought…

That’s better: see what’s been commented on recently

Sunday, March 26th, 2006

I’ve installed a number of (free) WordPress plugins – specifically, fuzzy recent comments, fuzzy recent posts and fuzzy recently updated by Semiologic.

This means (if you look down on the right-hand side) that you can see what the latest posts are, which pages have recently been updated (this only refers those publicly visible within the /blog directory) and who’s been commenting on what. I think that this will – until I get the forums up – give us the chance to follow the discussions going on here more effectively.

Your feedback always welcome, of course. I am trying the forums; it’s a question of getting the headers consistent with the rest of the site so that people won’t disappear into some sort of black hole there. Sure, the site isn’t consistent anyway, but improvement can be incremental.

(As I write this I’m watching the BBC Panorama program on the mad money-go-round within the NHS which describes a “double whammy” for going into debt… under the rules the government sets. They have to pay back the debt and lose it from their budget of the next year. One can see why the trading funds aren’t keen to be there.)

The money-go-round: how often does the government charge itself for its own data?

Friday, March 24th, 2006

An interesting email dropped into the inbox today..

One point which does not seem to have come across in the printed discussion is the question of other Government departments having to pay the OS to use OS data. This might seem bizarre – it is – but that is how things have come about. I was involved in negotiations with a number of Government departments and OS over a Service Level Agreement. Those of us who understood the huge potential benefits of GIS and linking datasets across departments were crying out for OS data but could not afford the fees OS wanted. Very many meetings were held between departments and OS, which resulted in reporting back to our own departments, then instigating discussions with sponsor departments to see if they would fund the use of data. The toing and froing was incredible. What it has cost the taxpayer in staff time far outweighs the income OS (and the Treasury) might receive. SLA’s now exist. Their terms are commercially sensitive. The cost is unbelievable. The whole process bogged down GIS development for 5 years and the enthusiasm of many hard working civil servants was drowned in the process.

(The sender requested anonymity).

This is an interesting point, because it’s clear from the Budget that Labour wants to push off any costs it can out of government – that is, not put any cash into organisations, and indeed to sweat the assets as hard as it can.

But if more than half of the OS’s £100-odd million in “revenues” is actually coming from other parts of government which aren’t “revenue-generating” (Defra? the Environment Agency – it uses OS maps to ilustrate its flood maps) or even private organisations which charge back to government (Capita uses postcodes for the congestion charge and for the TV Licence databases; would the cost of the contract to government be lower if the postcode or OS data were available free?) then something strange is going on.

That is, money is washing around in a sort of slush pile, and you never quite get to see where the music stops. If OS charges Capita for data, and Capita charges the government to run the TV licence, we arguably see no net benefit from charging for the data; it’s just an administrative exercise that makes work for the accounts people and the lawyers at OS.

Sure, Capita adds value to the data; that’s the idea of a private company. (No loans jokes please..) But this circular motion between government departments of money that only exists theoretically seems to me the weakest point of the government’s case for trading funds and charging for data – and remember, it’s that case which we’re trying to demolish.

Forums? Hold your breath.. a bit..

Friday, March 24th, 2006

Thanks for all the comments about the need for forums (which is getting clearer and clearer as we don’t have a “recent comments” or “recent posts” link in the right-hand bar, though I’d like to implement them.)

I’m evaluating the Drupal forum software, and hope to put something in soon. Thanks too for the advice on wiki software.

What do we want from the Met Office and the BGS?

Thursday, March 23rd, 2006

A thread started at the request of “weatherman”: what info precisely do we want from

And what useful data do they already provide for free?

Comments welcome.

Today in The Guardian: Berners-Lee talks; who should we be chasing?

Thursday, March 23rd, 2006

Today in the print edition of The Guardian, the Technology section has two pieces in the Free Our Data campaign:

Unfortunately the table in the second piece hasn’t reconstituted at all online – I may have a go at putting it in a more readable HTML form, at least for use here.

And anyone who would like to suggest some wiki software (thanks for the pointers to forum software) is welcome to, since I think we’ll need it to corral all the data about who does what and what they own and charge.

Another day, another budget; but we need monetary arguments

Wednesday, March 22nd, 2006

I spent a long train journey last night reading an enormous email conversation about 2002’s withdrawal by the Ordnance Survey of the Panorama product (which provided comparatively cheap – though not free – data on height).

It was clear that the reason why the OS was withdrawing the product was simple: it competed with the higher-quality and more expensive products that it had generated more recently itself. Trouble was, people were still plumping for the cheap Panorama one.

So the arguments that were being made to the OS – that it ought to make the data available, that others would host the data for free if the OS would only release it to the public domain – clearly weren’t going to work.

What we have to recognise in this campaign is that it will not be won by appealing to the better nature of the OS or the Treasury. They don’t have one.

The only argument that wins is the economic one: that the Treasury (in particular) will receive more money through the data being free than by it being charged for. That means you have to demonstrate that there will be enough mapping companies, and that they’ll be big enough, that their tax revenues will exceed £105 million (the OS’s revenues, I think, in the past year).

OBviously here the Peter Weiss paper is the best illustration, but it’s only an illustration – at present. We need better illustrations tailored to the UK.

I’m considering the way forward at present. One tactic that seemed logical is to do an FOI request to each of the “target” organisations asking what data they charge for, and how much they charge, and how much revenue they get from them.

Other thoughts?

Paying twice for data? Through your council, you might be paying EIGHT times

Tuesday, March 21st, 2006

This morning an interesting email dropped into the Guardian’s inbox. It’s quoted here in full with the permission of the author (see end).

I look after all the maps for the council where I work and yes, even government departments and councils etc have to pay for Ordnance Survey data.

Local government has interesting scenarios where the taxpayer will pay three times or more for Ordnance Survey Data. One of the most interesting scenarios is Planning Applications.

  • 1st payment to OS: if a member of the public wants to submit a Planning Application they can buy a site plan map, usually from the council (cost of about £25 for an 4 x A4 sheets) or other OS licenced data reseller.
  • 2nd Payment to OS: the Planning Authority (local council) also have to buy their map base from Ordnance Survey every year. Part of what is called the Mapping Services Agreement (MSA) [and a whole other debate hangs around the MSA – CA].
  • 3rd Payment to OS: the member of the public also pays for Ordnance Survey data as part of their normal taxes.
  • There is also a 4th Payment (which is the biggest scandal) that goes to Ordnance Survey and the Post Office, to use our council-created and council-maintained Local Land and Property Gazetteer (LLPG) or local address database. Even though all the councils create and maintain their own address gazetteer, we have to pay the OS and Post Office for the privilege of using that address data.

    The OS says that it owns the copyright of the position of the address, and the Post Office says it own the copyright of the address (because it adds the postcode). Councils therefore have to pay a per-click cost to OS and Post Office to use the council-created addresses on our own website address lookup facilities.

    The irony about all this is that the local council creates the address in the first place (Street Naming and Numbering sections) and gives (for FREE) this information (including site plan) to the Post Office and Ordnance Survey – so they are in essence charging the local council for its own information. Therefore the public have to pay the Council to create the address (Street Naming and Numbering dept) and then pay again to the OS and the Post Office for the right [for the council] to use it.

    Now, if the council wants to use Addresses (LLPG) created by a neighbouring council(s) (eg for cross-council working/Partnership working etc) the cost goes up even more. The tax payer has to pay the council to create the address data (Street Naming and Numbering dept), OS and Post Office to use the address, and again if used on a web address search facility.

    Then a neighbouring council will have to pay extra money to the OS and Post Office to use the same address (if it’s not in its postcode area) and finally pay again to use the address in a address search facility on its website.

    In total the tax payer could pay up to five times for the one address that the council gives to the OS and Post Office for FREE.

    Finally, taxpayers putting in a planning application online for one building/address using OS data could be paying a total of up to eight times for that one address/building data. Now that is scandalous.

    Chris Hancox, GIS Officer, at a council in the Anglian region

    [Editor’s note: some deletions made at writer’s request]

    Got something to say? Get in touch; and suggest some forum software

    Monday, March 20th, 2006

    We’ve been asked how one can get in touch. It’s pretty simple: email (I’ll give that rather than my Guardian email simply because I have better spam-filtering and mail-organising options on the Gmail address).
    If you’ve got something that needs saying, we’re always interested. I note that some people think forums, rather than a blog, would be a better organisational structure; that’s probably true. If anyone wants to suggest some PHP/MySQL-based forum software, I’m all ears. “Free” is a good price too..

    Why isn’t the BBC on this list? Because..

    Sunday, March 19th, 2006

    A comment here brings up the question of

    “Let us not foreget Auntie – the dear old British Broadcasting Corporation (BBC). The BBC must have one of largest archives of historical material in the World, that should be freely available, for research , if not commercial use.”

    So why aren’t we mentioning the BBC in this campaign? Couple of reasons, actually. (1) It doesn’t raise its money from taxes. No, it doesn’t. It raises (almost all of) its revenues through the licence fee, which is payable if you own a television. You are under no obligation to buy a TV. (In fact there’s presently a weird loophole where you can watch TV on a PC without needing a licence.) That’s rather different from organisations which get their cash from us.

    In addition to that, the BBC is opening up its archives in all sorts of ways. The BBC Backstage project is full of exciting ideas which are being done in all sorts of ad-hoc no-payment ways: take the scheme which brings the BBC’s banned news Persian service to people in Iran via instant messenger. The plans for charging are on the back burner, for the most part. It’s not a closed organisation at all.

    In fact, if you can think of some useful data that you reckon the BBC should make available, it probably already has..

    Tim Berners-Lee agrees: our data should be free

    Friday, March 17th, 2006

    I’ve heard from a contributor that in a talk he gave earlier this week, Sir Tim Berners-Lee – you may recall him from films such as “Honey, I invented the World Wide Web” – agrees with the idea that Ordnance Survey data (in particular) should be available to us all, for our use, without cost. We’ll have a full piece about what he said in next week’s Guardian Technology, if you can bear to wait. (Actually, you’ll have to. Sorry.)

    The interesting point about Sir Tim, of course, is that he could have patented his work in developing the hypertext protocol (what if CERN had had a requirement that workers’ ideas were patented?) and perhaps made a lot of money – although equally, the Web would not have been taken up with the same excitement if one had had to pay a licence fee for every web page served or link clicked. Sir Tim said as much in 2004 (original article seems to have disappeared.)

    The content you’re looking at now is an illustration of how freeing publicly-paid and generated data – in this case, how to implement the hypertext protocol (http to you and me), developed at a taxpayer-funded particle physics laboratory – can lead to individual and commercial implementations whose value far exceeds those that could have been realised by charging.

    We’re glad to know that we’re thinking along the same lines as the man who invented the Web. But can anyone think of a counter-example – of patented or copyrighted ideas around data that have been taken up on a huge scale? To win the argument for this campaign, we need to consider any counter-example.

    (Note: “patented or” removed in above because we’re not talking about patents in this campaign, but about copyright walls surrounding public data, and want to focus on that. – CA)

    Trouble commenting? Be patient..

    Friday, March 17th, 2006

    If you’ve put a comment (as many of you have – thank you) to a post and found it hasn’t appeared, or you’ve had to fill out a captcha* (where you have to say what letters appear in an image), please be patient. This is just the result of the comment spam protection used on the site, which is usually very effective.

    To bypass it, you can log in (see the Login link on the right) – you choose your user name and password – which validates you in the eyes of the system. The aim is to make it possible for anyone to comment, while keeping spammers out. Thanks again for your patience, and for all the contributions.

    * actually, I should have written captcha™. If you read the linked entry, it seems that Carnegie-Mellon has trademarked that acronym.

    Which organisations should we be chasing? Let’s make a list..

    Thursday, March 16th, 2006

    One thing that we haven’t done yet – at least, not in full – is to draw up a list of organisations that are government-owned (ie the government is the only shareholder) and which collect and then sell back our data. Often they use the excuse of being “trading funds” – ie told by the Treasury to go out and earn their keep – to claim that they’re not government-funded. But it’s rather like a child still living with its parents. If the trading funds went massively into debt, would the government – of any political colour – shut down Ordnance Survey or the UK Hydrographic Office? We think not. So, they’re really taxpayer-owned. That’s our data!

    So, who is there? Off the cuff we can think of the Ordnance Survey; UK Hydrographic Office; Highways Agency (does it have an equivalent in Scotland?); Post Office. Who else is there?

    Once we do that, we can begin to frame precisely what access we do want, and create a campaign statement that can be framed in precise terms. That’s the sort of thing that ministers, MPs and senior civil servants find compelling.

    Contributions welcome, as always.