Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Free Our Data: now on Twitter

May 15th, 2008

You may or may not have heard of Twitter, which creates a social network (sorry) around sending SMS-length messages via the web or mobile. The idea is that you can send messages to the world in general, or communicate publicly with friends, or privately with friends. You can think of it as real-time micro-blogging (ie, very short but to the minute).

If you want to understand more about the service (which is free), you can read Making the most of Twitter, at the Guardian.

But the point here is that there is a Free Our Data twitter account, which will principally be used for notifying when there are new blog posts here. (We tend to note any stories on the Guardian too, though I may add that to the feed.) It shouldn’t be overwhelming. And we’ll aim to follow those who follow us, so that we know what’s going on.

You’ll find the page at http://www.twitter.com/freeourdata. (Congratulations Ian Sealy - our first follower!)

In today’s Guardian: the problem with route data for glider pilots

May 15th, 2008

Today in the Guardian, with Access to data needs to take off, we look at the slightly unusual (in the context of this campaign) case of glider pilots, in particular, and the hassles that they have in trying to load flight data, in particular about where they may and may not fly.

The wrinkle being that the data is available - but not in a convenient machine-readable format that they could load into their GPS-based systems. And responsibility for deciding whether to do that is split between the Civil Aviation Authority - which is a government-appointed regulator - and National Air Traffic Services, a privatised organisation in which the government is the largest minority shareholder, along with commercial airlines and staff.

The point being, if you want someone to do something that would streamline things, where do you press?

Britain must love flying. How else to explain the fact that the space over the north of London is, as Andrew Watson puts it, “one of the most complicated bits of three-dimensional airspace on the planet”? And as the spokesman for National Air Traffic Services (Nats), which looks after air traffic control for commercial services, notes, there’s also the south of England, home to commercial airports such as Heathrow, “executive” airports such as Biggin Hill and Farnborough, and 60-odd other airports used by pilots of light aircraft, parachutists, hang-gliders and gliders. It’s very, very busy.

But as Watson, a glider pilot, points out, that busy-ness leads to a peculiar conflict over data. Specifically, useful access to data about where you may and may not fly in the UK. And with about 10,000 glider pilots and 3,000 gliders in the UK making some 500,000 flights a year and able to range across the country during a single flight - that access to data matters.

The Civil Aviation Authority (CAA) sets the rules about who can fly where. As Watson explains: “The UK is covered by a three-dimensional pattern of airspace into which various kinds of aircraft may or may not be allowed to enter.” All pilots must carry a chart showing that 3D space if they fly more than five miles from their airfield. (In practice, they all carry the authoritative CAA chart, published annually.)

That might seem straightforward - except that the boundaries can change from week to week or even day to day. That means pilots must check the CAA and Nats websites for the “Notices to Airmen” (or Notams, which presumably apply to female pilots too) about temporary airspace restrictions.

What Watson finds perplexing is that though you need to know those airspace “boundaries” - which are unrelated to commercial mapping data - the CAA does not publish machine-readable versions.

It’s a fiendishly complex situation, though what’s really puzzling is why NATS and the CAA, which would benefit from glider (and other private) pilots being able to get this data easily into their machines (without the possibility of transcription errors), preventing dozens of intrusions into controlled airspace every year, don’t do it.

Then again, it’s a funny world, flying. As I learnt while researching it, Nats and the CAA don’t accept GPS as a navigation device. Why? “Because it’s intentionally degraded,” I was told (by Nats). That is, your position isn’t perfectly accurate. But only to 10m, surely? Or even 100m? Is that really going to make all the difference when you’re thousands of feet (or metres) up?

So here’s the Ordnance Survey’s public task, for the record

May 15th, 2008

Since we asked where the Ordnance Survey (OS) public task is, we’ve been pointed to it. (Thanks again, Dan.)

So we thought that, since it seems elusive elsewhere on the web, we’d put it here for the record. And for the record, this is extracted directly from the report of the Select Committee on Communities and Local Government. No doubt there’s some copyright issue, but we can’t quite see why acknowledging a source and repeating it correctly, in brief, is a problem, especially with an organisation that we (as citizens) own.

So here’s the PDF of the Ordnance Survey public task; and below, the HTML. If you spot any errors, please tell us and we’ll aim to correct them.


(From the written evidence to the Select Committee on Communities and Local Government, Fifth report of session 2007-8, printed January 21 2008)

Ordnance Survey’s Public Task is to:

— Collect and maintain uniform datasets with national coverage forming the official record of the natural and built environment of Great Britain, containing:

— detailed mapping of the built and natural landscape (topography);86

— high resolution address data;

— transport networks (including road, rail, waterways, tracks and paths);

— terrain and height data;

— administrative and electoral boundary information; and

— geographical names (including features with imprecise extents such as “Dartmoor”, “The Cotswolds”, “Forest of Bowland”, “South Downs” etc).

— Provide a nationally consistent cartographic portrayal of the topography of Great Britain at various scales including large scales.

— Maintain a definitive three-dimensional satellite-based geodetic87 reference framework of Great Britain that underpins the national datasets and facilitates the integration and analysis of location-based information from many sources.

— Make the content of the datasets widely available in forms that are accessible to customers of all types for wider benefit. This will be in the form of products which are the direct output of those datasets. As part of its Public Task and in order to fulfil its trading fund obligations, Ordnance Survey will charge all customers for the licensing and use of its products.

— Conduct its activities efficiently and effectively to maximise the broader economic value of its data, whilst complying with trading fund requirements.

— Ensure that its data is capable of supporting the principles underlying the Digital National Framework in underpinning the association and integration of third party geospatial information.

In order to fulfil its Public Task, Ordnance Survey is required to collect and maintain topographic data at the following scales:

— High Change Geographies88: 1:1250

— Rural Areas89: 1:2500

— National cover: 1:10000, 1:25000, 1:50000, 1:250000

The above datasets are required to fulfil Ordnance Survey’s Public Task to ensure that a comprehensive, nationally consistent version of each exists in the public interest.

Annex 1A sets out:

— Ordnance Survey’s datasets which fulfil its Public Task;

— the products that are currently the direct output of those datasets; and

— the rationale for including the various data within the Public Task.

The minimum levels of accuracy and revision required for those datasets are at Annex 1B.

The referencing systems and data collected and maintained by Ordnance Survey contribute to the development and integration of geographic and location based information collected and used by government, business and individuals.

As the National Mapping Agency of Great Britain, Ordnance Survey represents Great Britain overseas as experts on Survey, Mapping and Geographic Information. It provides a focus for the provision of public sector information into pan-European and international collaboration by National Mapping Agencies and the European Directive on the Infrastructure for Spatial Information in Europe (INSPIRE).

In discharging its Public Task, Ordnance Survey:

— Seeks to maximise both the accessibility of, and the broader economic benefit arising from the use of the data. It therefore creates products directly from these datasets and makes them available, including through commercial licensing, to government and business customers and consumers.

— Takes into account the views of customers (as well as, inter alia, technological changes and its trading fund requirements) to ensure that the range and content of these products meets their changing needs, and makes changes to content, accuracy and revision policies as may be necessary to ensure that the datasets and products remain fit for purpose. This may result in adding or withdrawing products from availability from time to time, as well as enhancing content andfunctionality.


Notes:

86 Topography: Including defining the surface shape and composition of the landscape, comprising both natural and artificial features.

87 Geodetic: Relating to the scientific discipline that deals with the precise measurement and representation of the earth, its gravitational field, and other related phenomena. Within Ordnance Survey geodetic-quality information forms the high precision framework that ensures the correct positioning of all mapping and other data against the National Grid.

88 Predominantly urban areas and areas of significant development.

89 Predominantly rural settlements and developed agricultural land.


Where is Ordnance Survey’s public task set out, exactly? And why is it paying an external PR company? (updated)

May 12th, 2008

As Mike Cross has posted, there is some interesting triangulation going on through Parliamentary questions on how the Ordnance Survey’s public task was determined. (There seems to be a groundswell of questions from Conservative MPs inquiring into it.) These are generally batted away with standard civil service responses - basically saying “well, we did it how we did it.”

So we wondered: where can the OS’s public task document be found?

Not on the OS website, where you simply find

For the purposes of the PSI regulations, Ordnance Survey’s Public Task is defined as embracing everything we do from time to time to fulfil our obligations under the Ordnance Survey Trading Fund Order 1999 (SI 99/965) and the Ordnance Survey Framework Document 2004.

This is akin to saying “whatever we do, it’s a public task, because we do it.”

Nor on the Shareholder Executive site (the SE is the government arm which determines what its trading fund should do), where the relevant page says:

Ordnance Survey’s objective is to deliver its Public Task. This can be summarised [emphasis added - CA] as to:

  • collect and maintain uniform geographic datasets with national coverage, and provide nationally-consistent mapping

  • maintain a definitive three-dimensional satellite-based geodetic reference framework to enable correct positioning of mapping and other data against the National Grid
  • ensure that its data is capable of supporting the principles underlying the Digital National Framework, to allow integration and association with other geospatial information
  • make the content of its datasets widely available in forms that are accessible to all customers for wider benefit, and charge its customers for the licensing and use of its products
  • conduct its activities efficiently and effectively to maximise the broader economic value of its data, while complying with trading fund requirements and creating long-term shareholder value.
  • Yes, that’s an improvement, but we don’t want summarised - we want full. Where is it? Anyone?

    (Bonus link: OS has paid an organisation called Mandate £42,076.20 + VAT (that’s £49.400) since August 2007 for “consultancy and advice on Corporate Communications and Public Affairs.” Interesting to see that Mandate says that its skills in public affairs includes “Winning over the decision makers who matter”.

    Wonder why OS would think it needs those services?)

    Update: Dan MacDonald has provided a pointer to the OS’s public task in the comments - but he points out that “it’s only available if you trawl thru’ the pdf version of the evidence given at http://www.publications.parliament.uk/pa/cm/cmcomloc.htm“. Thanks, Dan.

    Crime mapping for London, Boris? We’ll start the clock now

    May 5th, 2008

    Boris Johnson’s election as mayor of London means that we can see how committed - and effective - the Tories are in their claims to want to provide more data from councils and government to the public.

    Among the pledges made by Johnson as part of his election campaign was to introduce crime mapping to London.

    David Cameron, the Tory leader, wants every police force in the country to record every crime online, every month, in map form. There’s also a pamphlet (PDF) on their proposals.

    How long will it take in London, Boris was asked? “We’ll start on day one, I’ll go to the Met and say listen, this is a fantastic idea…let’s see what we can do to put this on the web so people can look at exactly what’s going on in their neighbourhood, and use that tool to drive down crime.”

    David Davis, shadow home secretary, says “the police will get a massive reduction in red tape, targets..”

    Given that Boris didn’t get the final say-so until midnight on Friday, and Monday May 5th is a holiday, perhaps Tuesday will be the “day one” when he’ll stroll into New Scotland Yard and mention this. We’ll keep count. (Apparently he went to work on Monday. So the clock is well and truly started.)

    If it arrives, it’ll certainly be a win for free data. Interesting questions: will it be on Google Maps or Ordnance Survey’s OpenSpace, or some other provider’s maps? Will it be redistributable? What sort of copyright will it have?

    The proposals have considered privacy issues, and suggest:

    We believe

    there should be three kinds of map for each area for three categories of offences:

    (a) where privacy is less of an issue: at their exact location with a pinpoint showing exactly where the offence occurred;
    (b) where privacy might be an issue: identifying a 300 metre long street section within which an offence occurred; and
    (c) where privacy is an issue: identifying the nearest whole street within which an offence occurred.

    (Assaults would be in category (a); sexual attacks and domestic violence in (c), for example.)

    The copyright issue - and reusability - may have been determined: from the pamphlet:

    As our new regime spreads more crime information into the public arena, we expect people will want to create their own crime maps on the web of their own neighbourhood. That will be a matter for individuals, social entrepreneurs, Neighbourhood Watches and others. Once the appropriate statistics are freely available, it would be comparatively simple from a technical point of view for citizens themselves to overlay the statistics onto an online street map. There are already many different mapping systems of the UK available to the public online, including Yahoo, Google, and Streetmap.

    But what about that terrible ogre for Britons - house prices?

    It is sometimes pointed out that crime mapping, in identifying areas of high crime, affects the level of house prices in those areas. In fact, crime mapping can actually raise house prices if identifying areas where crime is rising then leads to effective action being taken to cut crime. In our view crime mapping will provide a clear and powerful incentive to affected residents to complain and insist on effective crime fighting solutions to reduce the crime. And when crime is reduced this typically increases house prices - rather than the reverse.

    So, we’ll start the clock tomorrow at 8am, when mayors ought to be arriving in their offices to start work.

    Government answers (cagily) on free data questions

    April 22nd, 2008

    The Green Party has been doing some sterling work trying to get the government to answer questions arising from the Cambridge economics study into the potential benefits of making data sets available for free.

    We draw your attention to the exchange in the Lords (where the Greens have a representative, Lord Beaumont of Whitley) over this. Though you may find the answers uninspiring, at first:

    Lord Beaumont of Whitley asked Her Majesty’s Government: Whether they intend to make the Ordnance Survey’s MasterMap available free of financial or legal restrictions. [HL2714]

    The Parliamentary Under-Secretary of State, Department for Communities and Local Government (Baroness Andrews): As announced in the Budget, the Government will look closely at public sector information held by trading funds including Ordnance Survey, to distinguish more clearly what is required by government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead up to the next spending review, the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term. In the mean time Ordnance Survey will continue to generate the revenue it requires to cover its costs, to fund investments and to provide a return to government, from sales of paper mapping and from licensing use of the Crown copyright and Crown database rights in its data, including OS MasterMap.

    This is the same “customers pay a fair contribution” line we’ve been hearing since the report came out (first used by DBERR, as we recall).

    Of course the key to really good questions in Parliament is to ask something that the government can’t disagree with, but which isn’t part of its policy at present - because that puts it into the logical bind in which it should take up that policy. The case of economic advice given to the government is the classic one. (But you hear it all the time at Prime Minister’s Questions - the ne plus ultra of this game - in which opposing MPs try to expose gaps like this.)

    Undeterred, Lord Beaumont pressed on soon afterwards (April 3):

    Lord Beaumont of Whitley asked Her Majesty’s Government:

    • Whether they intend to make unrefined information held by trading funds available free of legal and financial restrictions, as recommended in the recently commissioned study, Models of Public Sector Information Provision via Trading Funds; and [HL2713]
    • Why certain financial information in the Ordnance Survey and United Kingdom Hydrographic Office sections were redacted as confidential in the recent Models of Public Sector Information Provision via Trading Funds report; and [HL2715]
    • Whether they intend to make the Met Office’s unrefined information available free of legal and financial restrictions, following the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds (p76) that this would provide a net benefit to society of £1.03 million; and [HL2733]
    • Whether in light of the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds that there is a net benefit to society if trading funds release unrefined information at marginal cost, they will review the financial and legal restrictions on all unrefined information held by public bodies. [HL2734]
    • Whether they will define the “public tasks” for trading funds in order to identify information that should be released free of financial and legal restrictions, as highlighted in the recently commissioned study Models of Public Sector Information Provision via Trading Funds (chapter 3, paragraph 3.49). [HL2735]

    Lord Davies of Oldham:

    As announced in the Budget, the Government will look closely at public sector information held by trading funds to distinguish more clearly what is required by Government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead-up to the next spending review the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term.

    For central government bodies other than trading funds, the clear policy is that raw information should, subject to any statutory provision, be freely available or provided at the marginal cost of dissemination.

    In drafting the report Models of Public Sector Information Provision via Trading Funds, Cambridge University relied on the co-operation of and provision of data from trading funds. Some of the financial data provided was commercially confidential and was therefore not published in the report. This did not alter the overall conclusions of the report.

    Tom Chance, a coordinator for the Green Party (who helped draft the questions, notes on his blog that the commitment to make information freely available “subject to any statutory provision” is encouraging:

    That’s good to know, and backs up the Green Party’s case for making it accessible as well, e.g. Parliamentary procedure in an open, machine-readable format rather than plain HTML, or key data on domestic energy use in one place as a canonical source rather than being scattered across different sections of government departments (Defra, BERR, CLG, etc.)

    One other thing he remarks on (about the Guardian’s mapping):

    It would be nice if we could supply those guys with a decent set of OpenStreetMap graphics for use in articles rather than using non-free sources too!

    It’s actually an avenue we’re exploring - I’ve swapped emails with Steve Coast, who says that a change in the licensing for OSM may make this possible in the near future. It would certainly cut some of our mapping costs.

    If you get free data, what will you do with it?

    April 19th, 2008

    Our challenge to you: if you get free data, what will you do with it?

    The question has some urgency because if you can think of what you’d like to do with data from the Land Registry, Companies House or the Met Office, then you could be in line to be the first to benefit from it - and show the benefits of making more data free.

    The Cambridge report noted three sets of data that could be made free with minimal revenue impact: Land Registry, Companies House, and Met Office.

    Let’s revisit them, so you can think what to do with them.

    For Land Registry, the analysis only looks at “Property Data Services” - which are the ‘Property Price Data’ and ‘Polygons’, whose respective revenues were £893k, the majority of which was from a bulk form of the product, and £405k. (That compares to Land Registry’s total revenues of XXX, 86% of which comes from compulsory registrations.)

    Those, it should be noted, are tiny compared to its revenues. Land Registry’s fee income in 2006/7 was £474million (in 2005/6, £395m). Its costs are very high, but it still had an operating surplus of £96m - nearly as large as Ordnance Survey’s entire revenues.

    The reason: you’re obliged to tell LR when you buy or sell land or put a charge (such as a mortgage) on registered land.

    The property data could surely be used for some imaginative analysis - though note that Land Registry bans the use of its data for unsolicited mailshots. (An interesting question is how, if one moved to a free data model, one would spot uses which broke rules like that. Would you drop the rule, or include intentionally fake data which would tip you off if you received a mailshot addresses to it?)

    Companies House is next: £72m revenue, again almost all from obligatory registrations. (Although search is the most profitable area - as you’d expect: it’s easier to search data than to accept and check it.) Unfortunately most of the data there is marked confidential in the analysis - but again, one can imagine that it might be useful to find people who are persistent directors of companies that aren’t acting lawfully…

    Finally there’s the Met Office. Can anyone think what you’d do with a lot of weather data?

    More analysis and suggestions of how to use these three organisations’ output data - if it were free - are welcome.

    A national address gazette - but copyright problems persist

    April 19th, 2008

    Today’s Guardian notes the arrival of the NLPG (national land and property gazetteer) in a commercial form, created by local authorities and Intelligent Addressing.

    Yours, for

    between £15,000 and £20,000 a year. Profits will be shared among local authorities to help them keep data up to date.

    So let’s delve briefly:

    The gazetteer is not the only address database on the market. The state-owned Ordnance Survey also offers addresses as part of its MasterMap digital geographical database of Britain.

    Depending where you stand, this is either healthy competition or a wasteful duplication of effort. In recent years, a tortuous dispute over the licensing of intellectual property in state-generated address databases has exposed some of the damaging consequences of public agencies trying to compete with each other in the information business. Last year, the dispute exposed a hole at the heart of the government’s information strategy when the Advisory Panel for Public Sector Information said it was unable to rule on the matter.

    So what do we advise?

    Technology Guardian’s Free Our Data campaign urges a simple solution - that a taxpayer (or otherwise centrally funded) basic database of addresses be made available to all comers, for free. Despite some advances in the campaign, including the support of the Cabinet Office minister responsible for government IT, we have a long way to go.

    Cambridge economics report (briefly) debated in Parliament

    April 2nd, 2008

    We note (via theyworkforyou’s excellent email alerts system) that the issue of trading funds - and specifically, the Ordnance Survey model - was briefly debated in the Commons yesterday.

    The main protagonists: Iain Wright, who says he is

    the Minister for Ordnance Survey responsible for the shareholder relationship between the Department and the agency, dealing with strategic and day-to-day issues arising in connection with its activities, particularly in terms of financial and Government matters. My ministerial colleague the noble Baroness Andrews leads for the Department on issues relating to the purchase of Ordnance Survey products and services.

    Robert Key (Con, Salisbury) asks:

    there is continuing confusion between [OS's] public duty and the private competition that it has to have as a trading fund. The pan-government agreement, which regulates how different Government Departments and agencies use Ordnance Survey, came to an end yesterday. We have no news of what is going to be put in its place, so will he tell us? When will the regulatory framework be updated and amended to bring an end to all this confusion, which is getting in the way of Ordnance Survey’s excellent work?

    The reply:

    In respect of his important point about the pan-government agreement, that was established, as he is aware, to ensure that the Government have access to mapping data in order to develop and implement policy at a reasonable price. We are looking into that, and I will update the House accordingly.

    Then more interestingly from David Taylor (Lab, NW Leicestershire):

    There is an argument that [OS] information should be made more freely available, free of charge. Has he read the book which was published alongside the Budget, “Models of Public Sector Information via Trading Funds”—quite a racy read—and which rebuts the claim that a move to free data would damage the work of Ordnance Survey? It should be made freely available to citizens of this country, and that can be done in a way that produces funds rather than absorbs them.

    The minister’s reply:

    As a fellow accountant, I can imagine that I would find it racy as well. [Ah, Parliamentary humour - CA] My hon. Friend raises an important point about the provision of data. He said that Ordnance Survey breaks even as a trading fund. In fact, it provides about £6.2 million in surplus that is then passed back to the public purse via dividends. That is to be encouraged. The business model, with changing market conditions and technology, is being considered and, as Minister with responsibility for Ordnance Survey, I will continue to do so. [Emphasis added - CA]

    Interesting: that the Cambridge report is now getting debate time, that the free data model is being considered, and that the minister responsible is considering the business model “with changing market conditions”.

    Could a surcharge on planning applications fund free data from Ordnance Survey?

    April 1st, 2008

    Since we’re playing with hypothecated taxes - which we should point out is not a central tenet of the Free Our Data campaign; we think governmnent should fund this centrally, but recognise that in tight economic times it’s hard to persuade government to spend more rather than doing revenue-neutral “experiments” - let’s examine the suggestion (made elsewhere) that OS’s free data should be funded by a surcharge on planning applications.

    After all, the logic goes, planning applications are voluntary; and they require OS detail. (One has to submit a number of copies of OS maps of the area the application applies to.)

    [A disclosure up front: I have a planning application in process. However, the following simply emerges directly from the available statistics.]

    Let’s recall the amount that needs to be raised, according to the Cambridge study, to compensate OS for the loss of revenues from selling its “raw” or “public task” data: £12m - £30m.

    Now, how many planning applications were there for the UK? For England and Wales, the total applications received looks like this:

    This is only for England and Wales, of course, but shows that planning applications, even at their busiest, are an order of magnitude less than Land Registry transactions. True, this excludes Scotland, but even if that has as many planning applications as England and Wales (which seems unlikely), that would only be a total of about 1.3m applications in the busiest time.

    That in turn would imply a surcharge per application received of roughly £10 to £30. Whether that’s small or large compared to the cost of the overall transaction (which might be putting up an extension on a house or a change in use, requiring no actual building) isn’t clear.

    (Locally, making an application costs between £135 - £265. It may vary in other locations. The surcharge would also have to be collected from multiple councils, which would impose an administrative overhead; by comparison, there’s only one Land Registry.)

    But as an actual amount per transaction, it’s comparatively large, meaning it would be very sensitive to variations in the number of applications. The figures there vary from 501,000 to 689,000 - a 28% or 37% variation, depending which you take as your numerator.

    Why might this be? Perhaps Land Registry transactions include commercial purchases, which occur more frequently. And also that people buy and sell houses more frequently than they do things to them. Either way, it’s doesn’t look like the optimum way forward. It’s certainly a lot more expensive - per transaction - than a Land Reggistry surcharge.

    Year or Quarter

    Planning
    applications
    received (,000)

    1997/98

    505

    1998/99

    501

    1999/00

    521

    2000/01

    545

    2001/02

    582

    2002/03

    635

    2003/04

    674

    2004/05

    689

    2005/06

    645

    2006/07*

    644

    * provisional. Source: DCLG

    Land Registry surcharge could fund free OS data surprisingly cheaply

    March 27th, 2008

    Sold signs outside houseOne suggestion that has been made by Robert Barr (of Manchester Geomatics) and echoed recently by Ed Parsons on his blog (though I think Ed came up with it independently) is that Ordnance Survey’s non-refined data (that is, the stuff it does as part of its public task, which the Cambridge economics study of trading funds interpreted to be its MasterMap and Large Scale Topo) could be made available for free by making up any funding shortfall from a surcharge on Land Registry transactions.

    The reasoning: most LR transactions involve OS mapping.

    According to the study, that would cost between £12m and £30m in foregone revenue.

    So how much would you have to add to Land Registry transactions to make up that amount? It sounds like an awful lot of money to generate.

    Here are the figures I’ve culled from the Land Registry’s performance data for the past three years on the number of transactions.

    Number of registrations 2004/5 2005/6 2006/7 Mean 04-06 As % of total
    first registrations 297,405 309,609 304,391 303,802 4.3
    discharges 2,486,875 2,502,318 2,605,620 2,531,604 35.8
    mortgages 2,680,128 2,627,999 2,723,530 2,677,219 37.9
    transfers for value 1,378,200 1,270,867 1,480,819 1,376,629 19.5
    leases 167,234 173,610 197,546 179,463 2.5
    Total 7,009,842 6,884,403 7,311,906 7,068,717 100
    Total w/o discharges 4,522,967 4,382,085 4,706,286 4,537,113 64.2

    With millions of transactions, it looks like raising £12m - £30m wouldn’t actually be too hard. “Discharges” are the ending of a claim to a legal title - generally, though not always, the end of a mortgage. They attract no fee at present. Other LR charges range from £2 (for a search) to £700 (for first non-voluntary registration of a pricey parcel of land). Most of the charges, though, are £20 - £40 and upwards.

    So to find the £12m that the trading funds report suggests OS would lose solely from non-discharge transactions would mean adding £2.65 to the cost of each LR transaction.

    If we take the loss in revenue to OS as £30m, then it means adding £6.61 to each transaction. It’s not more than the cost of any transaction (except searches - which aren’t the same as the “searches” one does when buying a house; those go through your local authority), and compared to the cost of the typical transaction - say, the average £180,000 house purchase - it’s peanuts.

    Right - that’s the analysis done. Now we just need to find a minister who is in charge of Land Registry and Ordnance Survey and can tweak the legislation (it doesn’t need primary legislation, surely?) to make these changes. And we’re done.

    This analysis also appears (without the fun table) in today’s Guardian: Land Registry holds key to free OS.

    In the Guardian: ‘we’re now in sight of victory’; so what next?

    March 20th, 2008

    Following on from the trading standards report, today’s Guardian examines what it could mean, and what the government - and other - response so far has been.

    In sight of victory notes about the study that:

    The findings will be hard to dismiss. Unlike previous studies, they are based on hard figures from the trading funds affected. It also takes a holistic view, for example taking into account the overall cost to society of the extra taxation needed to pay for free data.

    And then there’s the reaction:

    Britain’s leading expert in public sector information policy, Professor Richard Susskind, chair of the Advisory Panel on Public Sector Information, described the report as “precisely the kind of detailed, systematic and rigorous economic analysis of trading funds and PSI re-use that APPSI has been recommending since 2003. We hope this represents the beginning of a new era of open and sophisticated thinking about the economics of PSI.”

    Michael Nicholson, chair of the industry association Locus, said the study “breaks new ground. It is the first time the consequences of exploiting public sector data have been reviewed from a socio-economic perspective in this way”.

    Ordnance Survey, the agency that would be most affected by a change in policy - and whose raw data the report says would benefit the UK most - said the study is “an important input to the wider debate”.

    Ed Parsons, the OS’s former chief technology officer, noted that “the ludicrous merry-go-round of government departments paying [each other] would disappear, reducing costs and increasing the use of geographic information (GI) within government … the relatively small GI industry in the UK would flourish”.

    (We should also note that Ed is not in favour of an OS that is totally directly funded. In the same blog post he says that

    OS in the position where its continued operation and the quality of its data is reliant on a subsidy from government [would be] a disastrous position which could result in a USGS like reduction in funding if political priorities change.

    We take that point on board - and we’ll deal with that in coming weeks.)

    So what does government, which now has stacks of data about free data’s benefits, say?

    Central government was less enthusiastic in its responses. The department for Business, Enterprise and Regulatory Reform, which oversees the running of trading funds, implied pricing policy would continue as before. While the government would set out clearly what information it needs for its public task, to ensure that such information “is made available as widely as possible for use in actual and potential downstream markets” the principle of recovering costs will remain in the next spending review (after 2011).

    Yes, you can groan now.

    “The underlying principle will be that information collected for public purposes will be made available at a price that balances the need for access while ensuring customers pay a fair contribution to the cost of collecting this information.”

    In other words, DBERR knows much better than economists using data gleaned from the organisations involved what’s right for the economy. It’s a bit shocking, really.

    Over to you: what do you think should be the next steps in the campaign? Our first thought is to examine how effective a system where Land Registry registrations part-fund Ordnance Survey, since registrations are (in economic lingo) inelastic. Anyone know where the data about the number of registrations via LR for the past 10 years could be found?

    Trading Funds report: final totals: economy +£179m, gov’t -15.4m

    March 19th, 2008

    So here are the tallies, according to the Trading Funds report, of how making bulk data free would benefit the economy, and how much it would cost the government. (You can find the figures in Chapter 7 of the report.)

    Name Product Gross benefit (£m) Net cost to government (£m) Net gain (£m) Ratio of return on investment (net gain/net cost)
    Ordnance Survey Large Scale Topo 168 12 156 13
    DVLA Anonymised Bulk & Mileage data 4.3 0.582 3.7 6.36
    Companies House Bulk Data and Image 2.6 0.681 1.9 2.79
    Land Registry Property Price, Polygon GIS 2.3 1.1 1.2 1.09
    Met Office wholesale data 1.2 0.260 1.03 3.96
    UK Hydrographic Office Digital UK Charts & Publications 1.08 0.744 0.338 0.45
    Total 179.48m 15.37m 164.2m 10.68
    source: models of PSI provision by Trading Funds report, DBERR

    The figure for Ordnance Survey is dramatic, but understandable: as it says itself, it underpins huge amounts of economic activity. The point that this study makes is that making its raw data free would allow even more economic activity - creative, useful, beneficial, taxable - to occur.

    (The UKHO is unlike the other trading funds, as the report points out, because it takes in raw data from outside organisations, which it may not be allowed to make available royalty- or payment-free. Which goes some way to explaining the answer to the question I was asked earlier.)

    Advance notice: Mike Cross talking to BCS on April 2 in London

    March 19th, 2008

    Just in case you’ve got a slot in your diary: Michael Cross, the co-founder of the Free Our Data campaign, will be talking at the British Computer Society’s Geospatial Specialist Group on Wednesday April 2 at 6.20pm in London.

    As the page says, “Michael will be explaining the rationale behind, and taking questions about, the sometimes controversial campaign, now in its third year.”

    Better to be sometimes controversial than always bland, though, that’s what we say.

    The page says that “NB This event is free, but registration is required. If you would like to attend this event please register by sending an email to GSG.Event@Googlemail.com”. It’s not clear if you need to be a member of the BCS or its Geospatial Specialist Group to come along - but presumably the registration email will tell you.

    We suspect that Mike will talk a little about the strength of the official reports that have all, so far, backed more or less the campaign’s standpoint; none has demonstrated that the wider economy would lose out by adopting our system.

    Trading Funds report: is PSI the new electricity and roads?

    March 14th, 2008

    Still chugging through the Trading Funds report (because it is 154 pages, after all): here we are at chapter 6.

    Where we find (PDF p107-108):

    With the development of the ‘knowledge’ economy, driven in large part by improvements in digital technology, the supply of data by trading funds can be seen as an analogous activity in ‘information’ sector to the supply of physical infrastructure in the form of power and electricity, transport (roads, trains etc), and telecommunications. This comparison is illuminating in a variety of ways.

    First, existing utilities often have similar cost structures where large fixed costs are combined with low marginal costs. Related to this, many of them, at least in some areas of their activities, have ‘natural’ monopolies just as trading funds may do in some areas of their business. Utilities are usually providing ‘essential’ infrastructure which, if not directly essential to government, are essential to the general economy – this could be seen as similar to the ‘public task’ of trading funds.

    For a combination of these reasons many of these utilities are regulated and have been now for some time and one might think that these regulatory experiences would have something to offer when considering the situation of trading funds (few, if any, of which have any independent regulation at the present time).

    This is an interesting point. It’s been clear for a long time that trading funds tend to be natural monopolies (it would make no sense - as we have pointed out - to have two competing national mapping agencies, nor to privatise Ordnance Survey, because the latter would create the situation where mapping would be a cherry-picking exercise).

    Two things: first, which trading funds are tightly regulated? One does hear the phrase of “capture”, in which a trading fund is able to persuade its controlling minister that it’s doing everything just right, and because it’s handing over the money every year the minister doesn’t have any cause to come down heavily on it - quite the opposite: don’t annoy the Treasury is one of the first rules of Being A Government Minister.

    Of course, utilities - certainly electricity and gas, and increasingly roads and rail - aren’t free. But then again, it costs money to generate and distribute electricity and gas, or build roads and rail. By contrast, the cost of running online distribution is falling all the time; the amount of data that the Met Office distributes every day - a few gigabytes, it told the report’s authors - would cost a few pounds per day to distribute if held on Amazon’s S3 system. That’s pretty cheap.