Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Archive for 2006

One victory for free data: Statute Law database goes online

Friday, December 22nd, 2006

Nicely timed just ahead of Christmas, the Department for Constitutional Affairs has put its Statute Law Database online. You can search for any extant law, and various others – including how upcoming legislation will affect existing laws.

We’re happy this is going to be free for anyone (even if it’s not quite user-friendly.. we await MySociety getting their hands on it).

But impressively enough it does go all the way back to the Statue of Malborough of 1267 which points out that

Fermors, during their Terms, shall not make Waste, Sale, nor Exile of [M2 House,] Woods, Men, nor of any Thing belonging to the Tenements that they have to ferm, without special Licence had by Writing of Covenant, making mention that they may do it; which thing if they do, and thereof be convict, they shall yield full Damage, and shall be punished by Amerciament grievously

Free Data: the idea rises above the government horizon; government quizzed on OFT report

Thursday, December 21st, 2006

In this week’s Guardian Technology, Mike Cross notes that the Office of Fair Trading report (which we’ve referred to before) has some interesting analysis of other countries’ models for public data charging. In Commercial case for free data rises overseas, he examines its coverage of the models used in the US, Sweden and Australia.

In Sweden, public agencies can compete with private organisations, and use their own “raw” data to generate “refined” data without transparency in their accounts.

In Australia and the US, by contrast, the study finds that governments prevent public agencies from competing directly with the private sector. There has not been a single complaint from a private operator in either country for the past five years.

There is a price to pay, the international study suggests. In the US, it finds that, while the supply of free data from the US Geological Survey has stimulated private investment, the private sector has now overtaken the government agency, whose maps are now used only as a last resort. Part of the problem may be that an agency forced to provide data for free has no incentive to make its data relevant to the market. The study suggests that, in a free-data environment, regular surveys of users may be necessary to ensure that an agency’s products are still needed.

And just as an added extra, the UK government is now being asked by the opposition what it will do about Ordnance Survey following the OFT’s determination:

James Arbuthnot, for the Conservatives, asked the DCLG what’s going to be done.

Unfortunately the answer is completely unhelpful, apart from saying that “Ordnance Survey officials will be working with officials in my department and in other Departments and organisations mentioned in the report to assist the Department of Trade and Industry in its coordination of the official Government response to the report.”

More detail from the OFT report: where next for public data?

Thursday, December 14th, 2006

Guardian Technology today examines the Office of Fair Trading’s conclusion that data restrictions cost the economy £500m:

Although stopping short of endorsing Technology Guardian’s Free Our Data proposal, the report says the Treasury should investigate the benefits of making those public sector databases freely available for commercial re-use. Confusion in government policy is stifling the re-use of data – and that policy “could be better informed by a proper assessment of whether [information] be provided for free”.

It really is time that the Treasury and/or Department of Trade and Industry did a proper study into the effects of properly splitting some data suppliers/gatherers into “raw” and “refined” sides, and the benefits and costs of making supply of raw data free. British Geological Survey already does this, and we’ve not heard any complaints about it. Ordnance Survey doesn’t, and… well, anyway.

But there’s an underlying conclusion that isn’t made explicit in the report. It’s this: trading funds are unsustainable (or barely so) in a model where everyone, including the private sector, has equal access to raw data, and the “commercial” side of public-sector organisations competes directly with the private sector. (Why would you set up a public-sector company to compete with a private-sector one? It can’t compete effectively because of things like pensions, which are more expensive in the public sector.)

So Ordnance Survey, as a trading fund, can only work if it gets some special access to raw data. But that goes against the government’s own rules on data trading. This can’t continue. Either Ordnance Survey is recognised as a special case (because it collects data for all of the country and processes it) and brought back into the taxpayer-funded fold – which its chief executive Vanessa Lawrence thinks won’t work – or it’s going to suffer a very slow death by a thousand complaints and reports. The OFT waves the big stick of the Competition Commission at it; the next few years are going to be tough, in policy terms. And that’s before one gets into the subject of what the Inspire directive means for OS.

OFT says more competition for public sector information would generate £1 billion extra annually

Thursday, December 7th, 2006

While everyone’s been focussing on the Gowers report (into copyright), the Office of Fair Trading – with perhaps questionable timing – has just published its own report into the commercial use of public sector information.

And it thinks there should be more, and that there should be less competition from the public sector information holders (PSIHs).

The key comment comes from John Fingleton, OFT Chief Executive, who said:

‘This is ground-breaking work for the OFT, looking at hidden markets in the economy. These monopoly public sector bodies cost the UK economy £500 million in lost opportunities. Our recommendations will help to make this valuable public asset more easily available for commercial uses which will benefit the economy and consumers.’

£500 million? The taxes on that would easily cover the £50 million in private sector funds that the Ordnance Survey needs, wouldn’t it?

Here’s the press release text:

“The OFT’s market study into the Commercial Use of Public Information (CUPI) is published today, and has found that more competition in public sector information could benefit the UK economy around £1billion a year.

Download a copy of the report (PDF, 707KB).

Examples of public sector information include weather observations collected by the Met Office, records held by The National Archives used by the public to trace their family history, and mapping data collated by Ordnance Survey. The underlying raw information is vital for businesses wanting to make value-added products and services such as in-car satellite navigation systems.

Public sector information holders (PSIHs) are usually the only source for much of this raw data, and although some make this available to businesses for free, others charge. A number of PSIHs also compete with businesses in turning the raw information into value-added products and services. This means PSIHs may have reason to restrict access to information provided solely by themselves.

The study found that raw information is not as easily available as it should be, licensing arrangements are restrictive, prices are not always linked to costs and PSIHs may be charging higher prices to competing businesses and giving them less attractive terms than their own value-added operations.

The report has also found that much of the legislation and guidance which aims to ensure access to information is provided on an equal basis, lacks clarity and is inadequately monitored. As a result the full benefits of public sector information are not being realised.

The OFT concludes that PSIHs should :

  • make as much public sector information available as possible for commercial use/re-use
  • ensure that businesses have access to public sector information at the earliest point that it is useful to them
  • provide access to information where the PSIH is the only supplier on an equal basis to all businesses and the PSIH itself
  • use proportionate cost-related pricing and to account separately for their monopoly activities and their value-added activities so that PSIHs can demonstrate that they are providing and pricing information fairly and in a non-discriminatory manner, and
  • enable the regulator (Office of Public Sector Information) to monitor PSIHs better, with improved enforcement and complaints procedures.

    Implementing these recommendations could double the value of public sector information to the UK economy to £1billion a year, and benefit consumers by providing a wider range of competitively priced goods and services.

    Competition watchdog reports

    Thursday, December 7th, 2006


    The Office of Fair Trading’s long awaited market study into the commercial use of public sector information makes interesting reading. It appeared too late for this week’s edition but we’ll be taking a close look next week. It doesn’t recommend “free data” – that possibility was never in its remit – but it does list a catalogue of obstructive behaviour and opaque pricing.

    One public sector agency is singled out as a special case because of the problems encountered by users of its information, and for the way “previous attempts by regulators and other bodies to influence [its] behaviour…. have met with resistance”. If you can’t guess which one, and are in a hurry to find out, read The Commercial Use of Public Information, (OFT 861), Office of Fair trading.



    Ed Parsons to leave Ordnance Survey: but why?

    Tuesday, December 5th, 2006

    Ed Parsons, the go-ahead chief technology officer of Ordnance Survey, is to leave at the end of this month.

    The reasons aren’t clear; there’s simply a posting on his blog which reads (in part, quoting the “agreed statement” that went out inside OS):

    Ed Parsons is leaving his post as Chief Technology Officer of Ordnance Survey to pursue new challenges in the increasingly dynamic Geographic Information (GI) Industry.

    Since his arrival in June 2001, Ed has developed Ordnance Survey’s IT strategy and has led OS Research labs. Ed has been instrumental in moving the organisation’s focus from mapping to the creation of geographic information.

    Ed’s comment:

    I am not is a position to add any more to this statement, but of course I am sorry to be leaving a great group of very committed GI professionals, the future for me is not completely clear at this point – but whatever it turns out to be, you will read it here as it happens !!

    Odd that he would be leaving with no definite plans, but we’re sure that there are going to be plenty of companies looking to hire him – he’s got experience they’ll find invaluable.

    Our take? The fact that he has his own blog shows he’s not your average member of a public sector organisation. We found him an interesting person who was certainly prepared to engage in the debate about free data and constantly looked for what the future holds in mapping. Who will succeed him, and will they bring that same drive to OS?

    How the Danes get it right with address data

    Thursday, November 30th, 2006

    In today’s Guardian, Mike Cross writes about how Denmark has shown that pooling public data can be done – easily:

    The server’s owner, the National Agency for Enterprise and Construction, also licenses bulk data to commercial re-users, such as estate agents and finance businesses.

    About 15 commercial users pay an annual subscription of about £5,000 and then about 10p per megabyte, which the government says is the marginal cost of connecting them and supplying data.

    Before 2003, commercial users had to request individual property details, at a fee of about 50p each. Under the new arrangements, demand for data has soared, says Ulrik Roehl of the agency.

    “When we started, we thought there might be around 10 distributors, but we exceeded that in two years,” he says. The distributors download about 65m property details a month.

    Reducing charges encourages takeup? Simple supply and demand, of course. And within government, there are no charges.

    We’re looking for more foreign examples so we can begin to make the point to government. Anyone else pitch in on how other countries do it?

    How does Australia charge for government data?

    Thursday, November 30th, 2006

    We’re intrigued by a comment on Ed Parsons’ blog (where he is talking about the model used by the island of Jersey, which is part, sort of, of the UK; he notes that “In contrast to the rest of the UK, Jersey with a single layer of government, has just got on a built a single land and property address database which is widely adopted and has become the standard for government use… Half of the possible 8000 government employees access the sole corporate geospatial Intranet and on the sister Island of Guernsey the utilities companies are beginning to publish their assets to single password protected website.”)

    In the discussion that follows (which we have weighed in to), there’s a comment that

    The model use in Jersey is similar to the models that now dominate in Australia. The heavy cost recovery models of the 1990s are now giving way to more sensible and effective understanding on pricing and access to spatial data. In most Australian states web portals provide citizens with free access to basic spatial information (either at state of local government level). The cross charging across government and to the public is declining as it was shown that it cause dysfunctional behaviour and held back the potential of SI. Commercial users are required to pay licence fees but at reasonable pricing levels which will still stimulates economic growth.

    Can anyone point us to more detail about how Australia implements its charging models?

    How the Inspire directive was watered down

    Thursday, November 23rd, 2006

    This week’s Guardian Technology looks at the lobbying that led to the Inspire directive moving from one which would have mandated free data access between governments (and users?) in Europe, to one which allows organisations that charge for data to continue to do so.

    Read Britain poised for victory in Brussels (which due to deadlines had to be completed before the vote and text was completely finalised).

    However, the reaction has been mixed. Ed Parsons, chief technology officer of Ordnance Survey, writes on his personal blog (which reflects his views, not necessarily those of OS):

    I would caution anyone reading too much into these early reports, the devil will be in the detail here.. and we should also not forget that INSPIRE is about a lot more than the licensing regime.

    From now on the technical experts can get on with drafting the principles around which the infra-structural components that will allow spatial data to be shared can be built – in my mind the really important part of INSPIRE

    the creation of metadata; the technology of interoperability; the development of data services; mechanisms to promote national co-ordination.

    We’ll have to see quite what shape of devil is in those details. By the way, does anyone know of any bloggers within other relevant organisations, such as UK Hydrographic Office?

    Inspire decision

    Wednesday, November 22nd, 2006


    The European parliament and council of ministers have finally agreed a compromise wording to the Inspire directive designed to harmonise spatial information around Europe. The directive had become a cause celebre in the movement to make public sector data freely available. Broadly, the European parliament backed our position, while the council of ministers was opposed.

    Here’s today’s announcement of the compromise, hammered out on Tuesday night (after this week’s Technology Guardian went to press).

    “The European Parliament and Council reached agreement last night on the contents of the proposed INSPIRE Directive, which aims to harmonise spatial information across Europe.

    “Key points resolved during the final stages of the discussions between the institutions included the principles according to which citizens should be allowed to examine the official maps and other spatial data covered by the directive, and rules for granting authorities access to data held by other authorities. Copyright issues were also dicussed in detail.

    “The directive will oblige EU member states to improve the administration of their map services and other spatial data services according to common principles. This will give Europe’s citizens better opportunities to find useful information about the environment and other issues from the internet. It will also enable the authorities to benefit more from information compiled by other official organisations.

    “Data search services designed for the public will generally be free of charge, although the directive allows fees to be charged for access to data that has to be updated frequently, such as weather reports.”

    Looking from a parochial point of view, the compromise seems to satisfy the UK government’s two objections to amendments voted through by the parliament last summer: that they would compromise national security and put trading funds such as Ordnance Survey out of business.

    Satu Hassi, a Green MEP from Finland who was closely involved with the negotiations, told me this morning that while she was not 100% satisfied with the outcome, the compromise at least puts some limits on data charges. In particular, it prohibits what she calls “arbitrary charging” – a finance ministry cannot suddenly decide to double the price of an information asset. 

    One disappointment was the loss of an amendment that would have forced governments to make available data about radioactivity. Sounds like a worthy target for the Free our Data campaign. 

    To sum up? Well, obviously the outcome isn’t what we’d have hoped for. Inspire isn’t going to end the absurd practice of public bodies spending time and effort negotiating rights and paying royalties for using data already owned by the taxpayer. (In Hassi’s words: “a ridiculous zero-sum game”.)

    We’re not downhearted, however. Thanks to Inspire, the argument for freeing public sector information has been made at ministerial level in every government in Europe. It is on the mainstream agenda.

    And, strongly as we feel about free data, let’s not lose sight of the main story. Inspire is about building new tools for understanding continent-wide impacts of climate change and pollution.  That seems like a good idea, even if some of the nuts and bolts aren’t the ones we’d have chosen ourselves.

    There’s more in tomorrow’s Technology section. Feedback welcome, as always.


    Postcode charges threatens split between councils and Post Office

    Thursday, November 16th, 2006

    Worryingly, the government’s insistence that every chunk of data somehow be turned into an asset in itself – rather than an asset to whoever uses it – is creating fissures between councils, which generate addresses, and the Post Office, which charges them for using postcodes.

    Read more at A one-way street to postcode madness in today’s Guardian:

    Councils say they provide lists of street names and numbers for free – but Ordnance Survey and Royal Mail treat their data as a commercial asset and charge other public bodies to make it available to the wider public.

    …Royal Mail says that the sums are tiny: authorities pay 0.5p a click, or a flat fee per domain. However, councils, under constant pressure to meet new centrally set financial targets, have little slack in their budgets. The final straw is that from October next year, the charge will double. Jennie Longden, head of address management at Royal Mail, says that these are the first price increases since 1995.

    The result, though, could be a grassroots rebellion. David Heyes, address manager at Wigan metropolitan borough council, Greater Manchester, says he is “very uncomfortable” with the click fee.

    ….Datastandard, a web community for professionals, has suggested charging Royal Mail between £250 and £1,000 for notifications of changes to local gazetteers. “I suggest Royal Mail pass on some of their costs to Ordnance Survey, but that’s for them to sort out,” said Robert Kimber, of Luton council.

    Would that be good for free data? No – it’s moving in the opposite direction. What’s needed is a minister or two to bang some heads together. Unfortunately it seems the heads that need banging are within government – possibly inside the Treasury – to stop this madness.

    Inspire countdown

    Thursday, November 9th, 2006

    November looks like crunch month for the European Inspire directive, which in the form voted by the European Parliament earlier this year would mandate a “free data” regime. Here’s the latest compromise proposal, from the excellent

    We’ll be monitoring developments, and especially the UK government’s positon, here.

    Should Ordnance Survey be split into two?

    Thursday, November 9th, 2006

    Following our earlier post about the Ordnance Survey losing the NIMSA contract, this week’s Guardian Technology investigates: Survey subsidy wiped off the map – and talks about the death of NIMSA to a number of people. Robert Barr of the University of Manchester mentions some possibilities:

    Ending subsidies to Ordnance Survey raises another possibility: that a future government might consider outright privatisation – an option considered and rejected in the 1990s. This would be a disaster for free data. Barr suggests an alternative approach: splitting the organisation into two. One division would operate a national geospatial database, funded by the taxpayer and made available to all, while the other would compete freely in the marketplace for maps and other “value added” products.

    Another model could be Canada’s Geobase project, where since 2001, mapping agencies at different levels of government – federal, provincial and municipal – have agreed to share and make available geospatial data under so-called “copyleft” royalty-free licences. The database, available at the Geobase portal, includes administrative boundaries and height data, which have both been subjects of anguished controversy in Britain.

    We repeat: we don’t think it would be at all good for Ordnance Survey to be privatised. For the free data model to work, you need a publicly-funded government agency.

    NIMSA is dead: bad news for Ordnance Survey – and free data?

    Thursday, November 2nd, 2006

    A terse announcement from the new Department for Communities and Local Government (DCLG) says that the National Interest Mapping Services Agreement (NIMSA), which “funds mapping services which are in the national interest, but would not otherwise be provided by the market as they are not economically viable”, is not going to be renewed when its seven-year term expires at the end of 2006.


    DCLG considers that: (a) it is appropriate for some of the services which have been supported by NIMSA, to be procured directly by those public sector bodies who require them, either individually or collaboratively; and 

    (b) it is appropriate for DCLG to continue supporting some national interest geographic activities but on a much smaller scale than previously.

    And the effects on users?

    Ordnance Survey has already indicated to DCLG that they are willing to continue to provide a ‘Mapping for Emergencies’ helpline service and national interest coastal survey work. Ordnance Survey will separately be advising users of any impacts of the decisions made by DCLG. 

    We’d like to know what OS thinks those effects will be. (They’re welcome to say here in the comments, anonymously or not.) For OS, NIMSA might not have been a huge part of its income (anyone have a number?) but it did prove that it was worth having, because the market wouldn’t do it. (And we’ve heard a lot about market failure having extremely large knock-on effects recently.)

    What’s faintly worrying about this is that it implies that DCLG doesn’t see OS as an important organisation. The Free Our Data contention is that OS is important, and that mapping everywhere, not just the bits that the market today thinks are important, matters. What we disagree about is the funding model.

    Ordnance Survey gets its lobbying in against INSPIRE

    Thursday, November 2nd, 2006

    A long interview in the Times details Vanessa Lawrence’s discomfort at the idea of INSPIRE, which would compel government organisations in Europe to swap their data for free – and in effect compel them to drop any system whereby they charge for intellectual property.

    OS has been doing some subtle lobbying (read the long Daily Mail hagiography of the organisation here; interesting how it makes so much of the funding model “that means it doesn’t cost the taxpayer a thing”) recently.

    INSPIRE, though, might be immune to lobbying. That hasn’t stopped a right-wing group called the Taxpayers’ Alliance from picking up the piece and suggesting that OS is talking the truth, and that it’s absurd to suggest that OS should be publicly funded, instead of taking funds from public organisations.

    However, although the Taxpayers’ Alliance seems to fight rather shy of finding out what other taxpayers think (there’s no way to comment on its blog), Heather Brooke at Your Right To Know (who deals with Freedom of Information and related issues) spears it neatly in her own post on the YRTK blog:

    Ordnance Survey is a monopoly, so claims about profitability, product quality, dividends and running costs are moot – we have nothing to compare them against. Would it be just as good value if the OS cost £200 million to run and paid a 8% dividend? Economic theory gives us a good idea of what actually to expect – price maximisation, inefficency and stagnation. It’s easy to turn a profit when you got your capital assets for free and can charge whatever you feel like.

    Meanwhile, INSPIRE is going to be voted on November 21. Counting down..