Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Archive for December, 2009

Fun facts from the DCLG / OS consultation

Tuesday, December 29th, 2009

A few things that strike us as we read through the consultation and impact assessment (links in previous posts).

Impact assessment:

Ordnance Survey generates most of its revenue from business and the public sector; in 2008/9 they each accounted for 46 per cent of the organisation’s total revenue. Consumers, through the sale of paper maps in retailing channels, accounted for the remaining 8 per cent of sales.

Impact assessment:

Ordnance Survey generates revenues from its products through licensing arrangements either directly with customers, or indirectly through licensed partners and through retail distributors. The direct customer channel accounts for two-thirds of Ordnance Survey’s trading revenue and includes various collective purchase agreements and major private sector users such as the utility companies. Approximately 25 per cent of Ordnance Survey’s trading revenue is generated though the indirect partner channel.

Impact assessment:

Separately, there are imbalances in Ordnance Survey’s current pricing model which may be causing inefficient allocation of resources. Firstly, Ordnance Survey currently charges private sector customers of its large-scale products significantly more than comparable government customers. The higher prices being paid by the private sector may potentially have restricted consumption to the less price sensitive users, impacting the economic benefit to the economy. Secondly, the payment allocation mechanism employed by government generates a weak price signal to Ordnance Survey from individual government users within the collective agreements.

Now that’s a really interesting one. Private sector pays more than government? I hadn’t heard that before. Payment mechanism generates a weak price signal?

Impact assessment:

[OS] already has a cost reduction programme underway as part of its existing business strategy, but any long-term strategic option would seek to introduce a framework that enhances cost transparency and provides incentives to pursue further efficiency gains.

More as we come across them…

Impact assessment of making OS ‘mid-scale’ data free puts cost at 47m-58m pounds

Wednesday, December 23rd, 2009

What interesting reading the impact assessment of the DCLG consultation on making OS data free is. Clearly some arms have been twisted in the Treasury to make it happen – Liam Byrne, chief secretary to the Treasury, almost surely in the driving seat there.

On the option being chosen (which is explicitly not the one that was examined in the “Cambridge study”, which looked at the benefits of releasing large-scale data, not the “mid-scale” data being proposed) the cost seems to be that government costs rise somewhat, while costs to the commercial sector fall.

From the document (on the impact assessment page):

ANNUAL COSTS

Lost OS revenue from OS Free data being made free: £19-24m (govt would fund this on a cost plus basis, amounting to £6-9m).

Increased government charges for large-scale data: £28-34m (price rebalancing based on number of datasets used by public and private sector).

One-off (Transition) Yrs: £ tbc

Average Annual Cost (excluding one-off): £47-58m

Total Cost (PV) £391-482m

Other key non-monetised costs by ‘main affected groups’ Transition costs to Ordnance Survey, government departments and businesses of moving to new model. There would be impacts on third party providers (see Competition Assessment, Annex 1).

ANNUAL BENEFITS

Description and scale of key monetised benefits by ‘main affected groups’: gain to business and consumers from OS large-scale data being made cheaper: £28-34m if assume price rebalancing is revenue neutral.

Gain from OS Free data being made available: £19-24m.

Average Annual Benefit (excluding one-off) £47-58m

Total Benefit (PV) £391-482m

Other key non-monetised benefits by ‘main affected groups’: The lower charges to businesses and consumers for large-scale data, and the free data should increase demand and hence welfare. Entry and innovation should occur in the market for geographical information. These welfare benefits have not been quantified (Pollock report focuses on releasing large-scale data).

And finally:

Key Assumptions/Sensitivities/Risk: Modelling assumptions: some substitution from paid-for to free data; lost revenue by OS due to competition from new derived products. Not yet determined how the revenue shortfall will be covered from government (i.e. who will pay and how). So for now assume no change in demand, but will estimate this for the final IA.

Price Base Year 2009

Time Period Years 10

Net Benefit Range (NPV) –

NET BENEFIT (NPV Best estimate) £0

Hurrah! Ordnance Survey consultation is live!

Wednesday, December 23rd, 2009

Thanks to a little bird at an interested organisation, we now know that the DCLG has opened its consultation on OS data.

It’s at http://www.communities.gov.uk/publications/corporate/ordnancesurveyconsultation, where we learn that the closing date is 17 March 2010 (and the opening date is today, 23 December 2009).

Consultation paper on the Government’s proposal to open up Ordnance Survey’s data relating to electoral and local authority boundaries, postcode areas and mid scale mapping information.

The consultation document itself weighs in at 2.2MB of PDF and 91 pages.

As ever, let us know your thoughts.

Update: and don’t miss the Impact Assessment paper – here’s the PDF of the Impact Assessment – which for some strange reason isn’t linked from the main page.

UEA CRU climate data is a free data issue too

Tuesday, December 22nd, 2009

I’ve been researching the apparent hack of the University of East Anglia’s Climate Research Unit (CRU), where a huge amount of email going back more than a decade, plus huge numbers of documents, have been released onto the internet – they’re indexed on various sites in searchable form and through Wikileaks, for example.

What I find interesting is some of the discussion around it. There have been multiple freedom of information (FOI) requests to the CRU from people who want to examine the underlying data used to make the analysis about human-driven global warming.

You’d think it would be straightforward. Science operates by data leading to theory leading to prediction leading to test against data, with a parallel process of independent test against the same data. So you’d think that access to the data would be a key thing.

Lots of Freedom of Information requests have thus come into the CRU demanding (that’s the word) the original data used for the papers. But the CRU has turned them down. Why? Because, UEA says, it came from weather organisations which charge for their datasets – and restrict those datasets’ redistribution.

Read for yourself at the CRU Data Availability page:

Since the early 1980s, some NMSs [national meteorological services], other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties.

And:

In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.

The inability of some agencies to release climate data held is not uncommon in climate science. The Dutch Met Service (KNMI) run the European Climate Assessment and Dataset (ECA&D, http://eca.knmi.nl/) project. They are able to use much data in their numerous analyses, but they cannot make all the original daily station temperature and precipitation series available because of restrictions imposed by some of the data providers

CRU insists it wants to make the data available:

We receive numerous requests for these station data (not just monthly temperature averages, but precipitation totals and pressure averages as well). Requests come from a variety of sources, often for an individual station or all the stations in a region or a country. Sometimes these come because the data cannot be obtained locally or the requester does not have the resources to pay for what some NMSs charge for the data. These data are not ours to provide without the full permission of the relevant NMSs, organizations and scientists. We point enquirers to the GHCN web site. We hope in the future that we may be able to provide these data, jointly with the UK Met Office Hadley Centre, subject to obtaining consent for making them available from the rights holders. In developing gridded temperature datasets it is important to use as much station data as possible to fully characterise global- and regional-scale changes. Hence, restricting the grids to only including station data that can be freely exchanged would be detrimental to the gridded products in some parts of the world.

The problem arises because the centre has been running in this way since the 1980s – before the internet reached even most universities, and when the culture of “pay for data” (because it was so hard to acquire, and so jealously guarded) was much more ingrained.

But it is a problem that needs to be overcome. The CRU has all sorts of PR difficulties because it hasn’t grasped this nettle – which needs to be grasped so that it can finally get past any questions about its research. There are people who aren’t satisfied at being told that the data needed to investigate a scientific paper can’t be passed on because of long-lost contracts. (We wouldn’t be very impressed by that if we were told it either.)

Paying for public data: it’s never a good idea. Especially when it creates problems like this.

Consultation update: still invisible, but asked in Parliament

Friday, December 18th, 2009

Ordnance Survey says it’s for the Department of Communities and Local Government that’s in charge of the consultation over making its data free….

According to this Parliamentary answer, DCLG thinks so too:

The question:

Mark Field (Cities of London & Westminster, Conservative)

To ask the Secretary of State for Communities and Local Government with reference to the announcement of 17 November 2009 on the Making Public Data Public initiative, when he expects to begin the consultation regarding access to Ordnance Survey data.

The answer from the DCLG minister responsible:

Ian Austin (Minister of State (the West Midlands), Regional Affairs; Dudley North, Labour)

We expect the consultation to be launched during the week beginning 14 December 2009.

That’s this week. This week is almost over. What, it takes a week to launch a consultation? There are international experts who can do it quicker. Meanwhile I tried phoning the DCLG press office (no reply on multiple lines) and emailing it (no response).

Helluva way to organise a consultation.

Anyone seen a consultation?

Friday, December 18th, 2009

The Department of Communities and Local Government is, apparently, in charge of the consultation over making OS data free.

The plan was of course that the consultation would begin “in December”.

December’s here and we haven’t seen much sign. Anyone else? We’ve put a call in to find out…

(If you need reminding about the case for making data free, see our other pages on the site, such as the articles page or the “Case for free“. Perhaps we do need to update them in the light of the studies of the past couple of years…)

Postcodes to be free? But which ones?

Wednesday, December 9th, 2009

The BBC has a piece saying that “postcodes” will be free from 2010:

Currently organisations that want access to datasets that tie postcodes to physical locations cannot do so without incurring a charge.

Following a brief consultation, the postcode information is set to be freed in April 2010.

….

The dataset that is likely to be freed is that which ties postcodes to geographic locations. Many more commercial organisations use the Postcode Address File (PAF) that ties post codes to addresses. Currently access to either data set incurs a charge.

In October 2009 the Royal Mail took legal action that cut off the access many websites had to PAF data.

(You might remember that one.)(

Sites that used the postcode feed included Job Centre Pro Plus, HealthWare (locates nearby pharmacies and hospitals), Planning alerts.com (monitors planning applications), Straight Choice (finds out who sent political leaflets).

That’s quickly contradicted however by the email that came around from the Royal Mail, noted by Steve Feldman, coming from Giles Finnemore, Head of Marketing at the Address Management Unit of the Royal Mail:

You may be aware of a story on the BBC website today that Government is planning to give anyone free access to postcode data.

Access to postcodes is already, and will continue to be, free to every citizen via www.royalmail.com/postcodes4free.

(Which is a nonsense. It’s true, but it’s also nonsensical, because the postcodes4free page requires registration and will only give you a limited number of postcode lookups in a 24-hour period. Which, if you think about it, is absurd: why does the Royal Mail want to make it difficult to address letters? You need to have an address list if you want to generate postcodes; if you didn’t have the postcodes, where did you get the addresses?)

For the avoidance of doubt PAF(r), the Postcode Address File, remains the intellectual property of Royal Mail and is supplied and used under licence. The new and recently published licences come into effect from April 2010. There are no plans for that to change.

Maintaining a world class postal address file requires significant ongoing investment and it is right that organisations who obtain value from using the file pay to do so.

We are aware of no plans for Government to pay Royal Mail for businesses and organisations to use our address file.

And it’s also contradicted by the Royal Mail’s press release page, which at present (December 9) has nothing about postcodes.

However, it may well be that the PostZon file – or more precisely, the long/lat lookups for every postcode – will be available for free next April.

Daily Telegraph: making stuff free can create revenues

Wednesday, December 9th, 2009

Hey, look, even the Daily Telegraph – hardly a home of the idea of the free lunch – is reprinting Breaking Views pieces which point out that making data free brings bigger benefits.

UK map giveaway throws bread upon the waters pretty much sings from our songbook:

The Met Office and the Ordnance Survey are unlikely candidates to stimulate another revolution. The weather forecasts may be accurate (sometimes) and the maps beautiful, but as businesses, neither is going anywhere. This is no surprise, since neither is really suited to becoming a proper commercial enterprise.

Yet the data they own is, literally, invaluable. Made freely available, all sorts of would-be entrepreneurs could exploit it to build businesses beyond the dreams of the public sector. The slightly geeky approach needed to be a successful internet entrepreneur is commonplace among mapaholics and weather nuts. Given the raw material, they could make a thousand businesses bloom.

The proposal unveiled this week is vague – a consultation document is promised later this month. The ability of the civil servants to emasculate any good idea should never be underestimated. But this is one whose time has come.

Given their tiny profits, selling off the Ordnance Survey and Met Office would raise minimal amounts. Giving away the data will undermine profits, but the benefits in terms of corporate taxes should be much larger.

Thanks. We knew you couldn’t keep a good idea down.