Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Archive for the 'Ordnance Survey' Category

Goodbye Gordon Brown: but thanks for the data … and the campaign goes on

Wednesday, May 12th, 2010

Gordon Brown’s stint as prime minister is over. But we can thank him for one thing he left behind: the commitment by the Treasury to fund free data from the Ordnance Survey (by my understanding, for at least five years – which I think is at least what it will take for really useful commercial applications to emerge from the availability of the data).

That’s a huge step. When Mike Cross and I started the Free Our Data campaign in March 2006, Tony Blair was prime minister. We knew that there was a strong reason for it, but it took time to get traction. Our first meeting with a minister was with Baroness (Cathy) Ashton at the Ministry of Justice; she didn’t seem too interested.

Once Gordon Brown came into office and there was a change at ministerial level, things changed dramatically. We got audiences and found ministers who were largely sympathetic. Brown too understood the idea – which simply took off when he found himself sitting beside Tim Berners-Lee at a dinner and started making conversation.

Brown asked: “What’s the most important technology right now? How should the UK make the best use of the internet?”

To which the invigorated Berners-Lee replied: “Just put all the government’s data on it.”

To his surprise, Brown simply said “OK, let’s do it.”

Now the Conservatives and Liberal Democrats are in power. The Conservative manifesto contains a pledge to access to government data:

Drawing inspiration from administrations around the world which have shown that being transparent can transform the effectiveness of government, we will create a powerful new right to government data, enabling the public to request – and receive – government datasets in an open and standardised format. independent estimates suggest this could provide a £6 billion boost to the UK economy. We will open up Whitehall recruitment by publishing central government job vacancies online, saving costs and increasing transparency.

That £6bn number comes, I believe, from the Cambridge study – though that included making OS Mastermap data free. I don’t think that that will be done, given the commitment to extra public spending it would involve in the short term and the long-term payback it would need.

I also think that while it’s nice to publish central government jobs online, there will be problems in how you slice it so that people can find the jobs they want. You might find that it’ll become something that other sites – and of course businesses – will exploit as a raw data feed and sell access to, or improve. (Yes, I know it’s also a scheme which is about chopping the funding of the Guardian’s public-sector jobs supplement off at the knees; there have been elements within the Tory party which have wanted to do this for years.)

In short: the campaign continues, but the Con-Lib coalition has indicated that it has a lot of the right instincts. Once we know which ministers we need to lobby – and once they know what their viewpoints are – we’ll be pushing the campaign again. There’s still so much data in there which needs to be freed.

Ordnance Survey data goes free: yes, we had noticed (just hadn’t written it…)

Friday, April 9th, 2010


OS OpenData Meridian 2 rendered with OpenStreetMap’s Mapnik. Photo on Flickr. Map data: Ordnance Survey.

Apologies for not writing about this sooner; we have been busy and all sorts.

So, go get your Ordnance Survey data.

As you’ll surely have noticed, Ordnance Survey has made a chunk of data available for free personal or commercial reuse, under a licence that equates to Creative Commons Attribution – you have to say what’s in it.

Disappointingly the 1:25K and 1:50K datasets (Explorer and Landranger) were not released – this has been a huge disappointment to paper mapmakers and others who wanted to do some innovative things with them. The Ramblers’ Association in particular had really been hoping for those.

Great stuff with the Boundary Line data, CodePoint (postcode to lat/long) and Meridian data. CodePoint on its own is saving lots of people lots of money – Cyclestreets reckons that’s £2,000 of money it doesn’t have to lay out annually to run its site.

OpenStreetMap is still discussing the implications of the licence and whether it wants to import the data. But it has set up a parallel project which uses it.

More visualisations via OSM:

For us, this doesn’t feel like the end of the road; it’s much more like the beginning, where now we have to see what people are going to do with it to prove the usefulness of the idea. And of course there are so many more potential data sources that should go free – tide times and flood maps being two which spring to mind. Come on, UK Hydrographic Office and Environment Agency.

We welcome your opinions.

Tristram Cary of Getmapping responds on the OS consultation: ‘British mapping is fundamentally flawed’

Tuesday, March 9th, 2010

Time for a different viewpoint on the OS consultation. Tristram Cary is head of Getmapping, which shot to prominence from its AIM-listed position in February 2002 via a story perhaps best expressed in the opening paragraph of this story in the Independent:

The Queen has found herself unwittingly on a collision course with the Government as a result of a legal dispute involving a small aerial mapping company.

How?

Getmapping, an AIM-listed aerial photography specialist in which the Queen has a stake, is taking legal action against the Government-owned Ordnance Survey claiming it has abused its market position and breached fair pricing policies. Ordnance Survey is part of the beleaguered Stephen Byers’ Department of Transport.

Ah, yes, those halcyon days when Stephen Byers was beleaguered, and OS was in DoT. (It’s interesting to ask why it was there, and why it moved, but that’s for another day.)

Cary and Getmapping sued OS:

Getmapping has spent £6m developing an aerial map of Britain which it sells over the internet to estate agents, local authorities and civil engineers. It finished the so-called Millennium Map in May 2001 having agreed the previous September that Ordnance Survey could sell it as part of its own services.

However, last May Ordnance Survey decided it wanted a higher specification aerial map and put the contract out to tender. Getmapping claims Ordnance Survey was planning to undercut the Getmapping product even though its higher specification map would cost more to develop. It says Ordnance Survey would therefore be cross-subsidising in order to compete unfairly with the private sector.

At the time I found this lawsuit very interesting; I’d say that it was, for me, the genesis of the entire “free our data” concept, certainly regarding OS’s behaviour. (Since some might wonder, I’ll specify: I had and have absolutely no relationship, commercial or otherwise, with Getmapping or Mr Cary.)

The lawsuit was eventually settled – which in effect meant Getmapping was forced to exit the stock market:

[Getmapping] finally agreed to drop its abuse case in June and paid £125,000 to OS to cover its legal costs although Mr Cary is still angry about the unfair advantage he feels that OS gets from not having to separately account for its commercial arm from its public sector work. The government agency’s commercial activities do not pay a market rate for OS copyright material, Mr Cary claims.

(See also: OS pledges to fight after judge calls Getmapping case “very weak” (June 2002); OFT ruling on the case (May 2002); formal statement at end of case (June 2003)

Water under the bridge? Well, Getmapping is still around (and has been providing aerial data to sites such as Microsoft’s Live Maps). And Tristram Cary recently added a comment to an earlier posting, which we thought would be worth reprinting here, since not everyone subscribes to our full comments feed.

So here’s Getmapping’s take on the consultation (though it’s not their formal response, which is of course far more detailed.)

Tristram Cary writes:

I take a very different view of the OS Consultation and I would like to outline my thoughts here. If there is an opportunity to provide a fuller explanation in the Free Our Data blog then I would be happy to do so.

I also would be very happy for anyone to base a response to the OS Consultation on these ideas.

I think that the way that British Mapping is managed is fundamentally flawed, and that as a result British mapping is not nearly as good as it could be. The main flaws are:

a) OS has become detached from the main sources of mapping change (in particular Local Government). When Local Government authorises a change it does not inform OS and OS therefore has to find out about the change from secondary sources. For example OS does not have access to the Local Land and Property Gazetteers (LLPGs) from Local Authorities which list all changes which affect the NLPG. As a result OS maps lag far behind the ‘truth’ on the ground (typically 6-12 months) and there is a substantial waste of surveying effort, as changes are logged and mapped twice (once by the Local Authority and once again by OS).

b) Maps are Value Added Products. Maps are combination of various low level ‘Elemental Datasets’ superimposed for convenience. Some of these Elemental Datasets are Natural Government Monopolies (NGMs) (eg District Boundaries, SSSIs, street names etc), but others are not (eg coastlines, rivers, height data). Thirty years ago it made sense for maps to be the lowest level of Core Geographical data, but in the modern era of GIS and digital mapping it makes more sense to maintain each Elemental Dataset separately and to allow the private sector or individual users to combine them to form maps. Why? Mainly because the role of government should be minimised to encourage competition and customer choice – this implies that the Government should only look after the Natural Government Monopoly datasets such as Boundaries, SSSIs street names etc. Over the last ten years good progress has been made in opening up the mapping market to genuine competition, and there are now several sources of OS independent mapping including NavTeq, Bartholemews, UK Maps and our own People’s Map. The great downside of giving away map products as free PSI is that it will kill the competition for those maps. It would be much better to give away the Natural Government Monopoly data ONLY and allow the free market to develop all the higher-level mapping products.

c) The Derived Data Trap. The OS is currently in a dominant (almost monopoly) position for the provision of large scale maps. OS maintains that position partly by claiming IPR on data which a user derives using an OS map as a reference, even if the data does not appear on the OS map in the first place. For instance if a utility draws its pipeline network using an OS map as a reference than OS says that if the utility stops paying its annual licence fee to OS then the utility must delete all its derived pipeline data and recreate it from scratch. This acts as a effective barrier to the creation of a truly competitive market for maps for two reasons. First the main customers (government and utilities) are locked in because if they change map supplier then they face having to re-create their derived data. And second because the independent map suppliers cannot get hold of any derived data without paying OS a fee and negotiating licence terms. Boundaries are a good example of this. The Boundary Commission cannot supply its boundaries to the private mapping sector because they are drawn on top of OS Maps and are therefore considered to contain OS copyright.

Because of all these issues, Getmapping’s view on the OS Consultation is as follows:

a) All Natural Government Monopoly (NGM) data should be made available free of charge to the market as PSI, but without any non-NGM data included

b) No maps which contain non-NGM data should be made available free of charge as this will severely damage competition, customer choice and quality in the market

c) In the medium term responsibility for maintaining NGM data should be transferred from OS to the natural owners of the data (eg Local Government for street naming and numbering) and the maintenance of the data should become an integral part of the change process, funded by the organisation which causes the change. For example, if somebody extends their house, then the builder should commission an independent surveyor to make a detailed survey of the as-built change to a specified format and accuracy. This survey would be paid for by the owner of the building and it would be used as the formal NGM update record. An intelligent properly-connected process of this kind would allow maps to be kept up to date within hours of a change, not within 6-12 months

d) It follows that in the medium term OS’s role should become a QA and government-procurement role. This is because if NGM data is maintained by the ‘owning’ government department and made available as PSI then the private mapping sector will be able to make high quality maps to suit all market demands. I believe that the quality, choice and prices of maps would all improve as a result.

Your comments are welcome. The “derived data” is especially interesting, particularly as it applies to utilities. We’d love to hear from utilities or other customers whether this is indeed the implication of the contract – because we suspect it is, given previous form on the derived data question.

And while we’re talking about Land Registry.. here comes UKMap

Friday, March 5th, 2010

Verrry interesting post over at the UKMapping blog. We’ll include the full content because it’s short and very relevant.

Great news was received today at UKMap HQ. After much review and testing Land Registry have confirmed [and are happy for us to tell people] that they are happy to accept registrations based on UKMap.

Great news for all those consultants, government types and property professional using UKMap.

Full text below

“Following a review of UKMap, Land Registry is able to confirm that UKMap meets Land Registry’s requirements as a mapping base on which registration applications can be made and is comfortable accepting registration applications based on UKMap. Such registration applications must still follow relevant guidance as set out in Public Guide 40.”

What does this mean? That LR doesn’t necessarily have to rely on Ordnance Survey maps for registrations. That, too, pours sand into any engines that might be getting started looking to fund OS through higher LR transaction fees. Savvy users would just go to UKMap.

And an earlier post on that same blog includes these interesting new clients:

  • London Fire, serious users with a serious need to get better mapping. Now they have that with UKMap.
  • London Borough of Islington – managing their green environment, UKMap’s trees and Land Use makes all the difference for them.
  • Mott Macdonald – detailed city centre mapping, all in glorious 3D.
  • Promap – the UK’s leading mapping portal for the Land and Property market have announced they are taking UKMap

London councils and emergency services? That’s what I think you call an inroad into OS’s market, isn’t it?

Free Our Data response to the DCLG consultation on OS Free

Friday, February 26th, 2010

We’ve written up our response to the DCLG consultation, and emailed it over this afternoon. Make sure you respond too! Deadline is 17 March – a Wednesday, for no obvious reason.

Feel free to crib from this or build on it. Your comments welcome (though of course they’ll only be useful after the fact…)


Response to Ordnance Survey consultation from the Free Our Data campaign, 25 February 2010

Introduction
The Free Our Data campaign was co-founded in March 2006 by Charles Arthur and Michael Cross with the aim of persuading government that non-personal datasets created by government-owned agencies and companies and organisations should be made available for free reuse without licence restrictions.

The rationale for this approach is that citizens have already funded the existence and collection of these agencies through taxes paid over past years. (This includes historical data; Ordnance Survey, for example, has been a trading fund for some time but on its incarnation as a trading fund immediately used data previously collected at public expense.) Furthermore, the private and non-profit sector can imagine better ways of using data than government can because they have a direct interest in using it – but price and licensing are significant barriers to the development of those applications.

The campaign is apolitical. It is not aligned, associated with or funded by any political party or outside group; its (very small) costs are paid by the co-founders out of pocket.

We are delighted that government has chosen to accept the rationale behind the campaign’s logic with its plans to create the OS Free products. Our only caution is that it must ensure that the model used to fund it can be widely applied to other non-personal datasets within government. OS Free should not be a one-off, but instead should be the basis for a wider sharing of data.

One final, general point: the Free Our Data campaign believes that Ordnance Survey provides an excellent map-generating service and must remain a government-owned asset whose public task includes the continual mapping of the UK’s geography and built environment. Any moves to privatise any part of its operation would be retrograde and threaten both OS’s future usefulness and the UK’s economy. We would oppose such moves.

Question 1:What are your views or comments on the policy drivers for this consultation?
The need to reduce overt government spending, allied to the growth in personal computing power owned and controlled by the public at large, creates an entirely new opportunity to let citizens analyse, understand and benefit from the data that the government collects on their behalf. This is a two-way process.

Clearly the UK government is rapidly recognising the benefits of transparency – that actions are not just seen to be done, but that the reasoning for the actions can also be interrogated and understood. This is one key policy driver. (Hereafter PD1.)

There is a second policy driver (hereafter PD2): the need to reduce the public sector deficit in coming years. This is best done through a reduction in public spending and an increase in tax revenue from the private sector.

There is also an untapped private-sector entrepreneurial market whose entire existence depends on the successful implementation of this consultation and future ones like it. When government-collected data is treated as a limited asset which must be priced to create an artificial shortage, government constrains the private sector which generated the taxes used by the government to create the data. Clearly, that then constrains the tax base, because not all companies (extant or proposed) can afford to buy the data. Therefore total taxes are lowered by pricing data. This is inefficient, and constrains entrepreneurship based around the effective use of data.

Therefore making government-owned data like this free for reuse (including commercial reuse) will bring in larger tax revenues as long as HMRC is vigilant in collection of owed taxes from individuals and companies.

How the consultation will reduce public sector spending in the context of the Ordnance Survey’s financial model (as a trading fund) is less obvious, but still exists. The consultation iterates costs of making these mapping data free. However, it does not iterate the potential benefits through reduced costs to local councils, police forces and local health authorities, for example, of being able to provide map-linked data on public websites, without paying, at the Landranger and Explorer scale; this has been a consistent bugbear to local councils, to police forces and public health observatories which want to share their work with the public.

Making such data free also obviates the legal examination of any instance in which those bodies wish to share their work – a cost which is also unpriced in the consultation document. While these costs may not match the millions of pounds directly attributable in lost revenue from sales of Explorer and Landranger-scale data, they are significant in the cultural sense too – because they enable those bodies to operate in a more transparent manner as well, satisfying PD1 above.

Question 2: What are your views on how the market for geographic information has evolved recently and is likely to develop over the next 5-10 years?
The geographic information market has been completely transformed in the past 10 years by
-the opening of GPS (Global Positioning System) data to the world by the US military (an excellent example of treating “information as infrastructure”, in which the US government bears the cost of supplying, in effect, location data to non-US-taxpaying people in the UK and elsewhere); and
-the ability to create “crowdsourced” maps, such as OpenStreetMap (OSM), which are accessible via the internet without copyright restriction for consultation, addition or editing.

In the future the GI market will be further transformed by
-the growing number of smartphones with built-in GPS
-the falling cost of mapping large areas with great precision due to improving satellite photography systems
-mapping information, including road and route data, becoming a commodity, where only value-added forms can be effectively charged for.

Sales of satnav devices provide a clear indication that “knowing where you are” is a key piece of information for people: estimates suggest that between 4 million and 7.5m such devices have been sold in the past 10 years in the UK alone.

The commoditisation of route and road information, which has previously been supplied only by Ordnance Survey, will continue. If satnav makers decided that the OSM mapping was good enough, and that the pricing model it offers (of zero cost), they might choose to use its database (or even to improve its database while using it) and neglect the OS version. Under the present OS funding model, the only way for OS to recover its costs would be to raise costs to its existing clients, including the public sector – which would not fit PD2.

Therefore it is essential that OS does provide the OS Free data to encourage the growth of the UK geographical information sector, and develops its own high-quality mapping as part of the public task for which it exists.

Question 3: What are your views on the appropriate pricing model for Ordnance Survey products and services?
Given the name of the campaign, our obvious answer would be “all should be free”. But we recognise that there are pragmatic and political problems with this.

The question assumes a great deal about OS products and services, and its charging regimen. However as noted in the consultation OS has consistently declined to separate out the costs and revenues and profits of its “raw” and “value-added” products and services, which makes it difficult to take anything but a Gordian Knot approach to finding appropriate models.

The question would be better framed as “which products and services should OS produce, and what should it charge for, and how should the charging regime be set?”

The more logical approach is to ask what OS’s public task should be, what products and services flow from that, how far those should be self-financing (using, say, a trading fund method) and what other products and services are seen as a public good which should be funded out of general taxation.

OS’s public task is clearly to map the geography of the UK; arguably this also includes the built environment. MasterMap provides an appropriate starting point for the public task, comprising a detailed scale of the UK.

For the moment we find the proposed model – with MasterMap and non-OS Free products’ prices aligned for the public and private sectors – to be equable. However as costs of updating maps and built environment detail falls (due to pervasive GPS feedback systems such as smartphones and cheaper satellite imagery allied to automated updating of map databases) this may need review to see whether more detailed scale products closer to MasterMap level can also be offered free.

Question 4: What are your views and comments on public sector information regulation and policy, and the concepts of public task and good governance as they apply to Ordnance Survey?
PSI regulation and policy suffers from the problem that where public organisations decline to comply with it, neither method of enforcement is satisfactory.
-If OPSI or other organisations demand compliance using non-legal recourse (e.g. asking for “good practice”), the non-complying organisation can ignore it; or
-if OPSI or other organisations seek legal recourse for compliance, the exercise is extremely costly for all concerned and is concluded so slowly due to legal process that private organisations in particular are at risk of going out of business first. (The instance of Getmapping’s complaint against OS in the early 2000s is illustrative.)

It is absurd that OS has written its own definition of its public task – with or without the consent of its minister in DCLG. With the release of OS Free, it is time for the job of defining OS’s public task, which impinges on huge parts of British life and the economy, to be put in the hands of a body entirely outside OS.

Question 5: What are your views on and comments on the products under consideration for release for free re-use and the rationale for their inclusion?
It is essential that there should be both raster graphics and vector graphics. The former allow easy use on websites to create Google Maps-style interfaces (where the map can be “dragged” to a location). The latter allow dynamic scaling. Though no rationale has been offered for their inclusion, they seem to fit the “mid-scale” requirement.

The inclusion of Code-Point and Boundary-Line datasets, with licences that allow free reuse (including commercial reuse) is essential to the creation of useful, effective and profit-generation applications.

Question 6: How much do you think government should commit to funding the free product set? How might this be achieved?
This is a key question – and how the government chooses to implement this will demonstrate whether it is truly committed to the idea that “information is infrastructure” by creating a model of funding that will be applicable to other data-collecting trading funds and parts of citizen-funded government, or if it is simply choosing a short-term fix for the problem of the desperate need for free access to OS data.

It is easiest to start by indicating what the government should not do.

– It should not raise prices within government for the non-free OS datasets above those charged to commercial organisations outside government. This would create tensions under which government organisations would naturally seek third-party solutions to reduce their costs (because of PD2). That would undermine income for OS and jeopardise the quality of all its data. In extreme cases, price rises might deter local authorities and other public bodies from using high quality geographic information to deploy their resources more efficiently and end up costing the public purse more in the long term.

– It should not raise prices for commercial organisations above those charged to government for the same datasets. This too will tend to exacerbate any drift to third-party solutions for high-value datasets. (Although it should be expected that these will occur naturally due to new entrants in the market.)

Therefore government should commit exactly the “funding gap” that making the datasets mentioned free will cause – apart from the paper maps. OS will presumably continue to sell paper maps, and will be able to rely on its brand to benefit from their sales and consequent profits. Therefore Treasury should fund the “gap” in revenues out of general tax funding, rather than by levying greater charges for OS data from other public sector sources.

The government’s own argument that “information is infrastructure” should be applied here. Roads, for example, are physical infrastructure. Government sees their provision as a public good and commits to fund their building from general taxation. It does not charge higher road tax prices to government-owned vehicles to offset the fact that government has built the roads and provides free access to them. (Nor is road tax hypothecated towards road-building.)

In the same way, other public sector organisations that use OS data should not be charged over the amount that private sector groups would be, and their payments should not be hypothecated towards any “funding gap”. The amounts being discussed – ¬£19-¬£24m pa – are comparatively small when set against overall public spending.

The benefits, admittedly, are difficult to enumerate. It is possible that, as with GPS, the benefits will not be immediately visible, and may not appear in the same place as the investment. It would therefore be sensible for government to commission regular studies to evaluate the growth of business predicated on use of the OS Free products.

By adopting a “non-hypothecation” approach to funding OS Free, government will be greatly simplifying the process required for the subsequent release of other datasets from other government-owned bodies. The pressure to release OS Free arose because the trading fund model is too restrictive: it cannot prime the market.

To draw an analogy, the search engine Google could not be profitable if it were to use Microsoft’s Windows to power its multiple thousands of servers that store its index of the internet. It would have to pay a Windows licence on each of those servers, and for each additional one. The cost would outweigh its profits. Instead, Google uses the free Linux operating system for those servers. We are suggesting that using the OS trading fund model for products is akin to licensing Windows: it limits the size of the market for their use, and the speed with which companies can grow while using those products.

Question 7: What are your views on how free data from Ordnance Survey should be delivered?
The key to the datasets being useful will be (a) availability (b) reliability ( c) accessibility.
Availability: where are the datasets stored? If OS hosts the files, it will need to create an entirely new system to support hundreds or thousands of concurrent accesses. That is inefficient, and outside OS’s remit. It would be more sensible for the datasets to be uploaded to a cloud facility such as Amazon’s S3 storage or Google’s cloud facility where copies could be downloaded. This is a comparatively low-cost solution where OS would only have to pay for downloads, rather than setting up its own hosting service.

Furthermore, it is clear that the datasets will be subject to change over time. It would be inefficient to upload a complete set every day, for example. A more effective method would be to upload a “diff” file of differences from the previous full upload every so often (daily, weekly, monthly). This would reduce the total amount that would be needed to for an up-to-date download and simultaneously create new opportunities for applications showing what has changed on a map or dataset over time. A full dataset incorporating the diffs from the last full upload could be provided every, say, six months.

Reliability: so users can be confident that the files come from OS, they should be cryptographically signed.

Accessibility: the files should be made available in formats that are readable using open-source software: that will ensure that they will be usable by the widest possible range of users and applications.

Question 8: What are your views on the impact Ordnance Survey Free will have on the market?
Resellers of OS data will not be pleased. But this will force them to focus on value-added services rather than promulgating a system which perpetuates the extension of copyright limitations that are not sustainable in the age of the internet.

Some map providers have already cut their prices in response to the expectation of OS Free. As in Canada (in the example cited in the consultation) we should expect that mapmakers will take the opportunity to create specialised maps for different niche groups (climbers, walkers, and other outdoors pursuits are likely to be the first to take advantage of this).

The provision of CodePoint will galvanise a market that has been held back by the problems of creating fast, cheap and legal lookups for geocodes. Although organisations such as Yahoo offer them, using those leaves providers dependent on outside groups, when they would prefer to do their own lookup. CodePoint is an essential part of the package.

The provision of Boundary-Line will be highly important in the forthcoming election. It will also be important for online organisations which depend on mapping electoral constituencies.

Question 9: What are your comments on the proposal for a single National Address Register and suggestions for mechanisms to deliver it?
The absence of a working National Address Register (due mainly to intellectual property claims by publicly owned bodies) has created the absurd situation of the Office for National Statistics being forced to spend millions of pounds creating a one-off register for use in the 2011 census and then discarding it afterwards.

Government should retain the ONS census for future reuse and treat it as a resource with huge ability to create value for the economy.

Question 10: What are your views on the options outlined in this consultation?
Option 1 – allowing OS to continue with its planned “hybrid” strategy – is deeply unsatisfying. The strategy proposal has received no proper oversight; it has not been debated in Parliament; its financial assumptions are at best weak and at worst flawed; and the creation of an “attached” private company that would sell OS-branded goods is anticompetitive because it offers no transparency on pricing, while having sole advantage of the OS brand.

Option 2 – releasing large-scale data for free reuse – would cross a Rubicon. Although the Free Our Data campaign would support this, we are concerned that government and Treasury has not shown sufficient commitment to the idea of vote-funded data collection and parsing by OS, and that this strategy could endanger the long-term future of OS. Furthermore, it could undermine would-be commercial competitors, and would create substantial upheaval in the geographic information market. Change is good, but too much change can be unpalatable.

Option 3 – releasing “mid-scale” data as suggested, and considering a transition to further release – seems to offer a path towards the long-term future of OS while providing the opportunity to prove the benefit that would accrue to the private sector, and thus the Treasury through tax receipts, of freeing data. We find this the most pragmatic approach – but reiterate that the government’s aim should be to pursue a path where it releases data for the use of citizens without cost impairment.

Question 11: For local authorities: What will be the balance of impact of these proposals on your costs and revenues?
N/A.

Question 12: Will these proposals have any impact on race, gender or disability equalities?
We see no impact on those inequalities.

Charles Arthur & Michael Cross, 26 February 2010.

Data.gov.uk: now that’s what we call a result

Monday, January 25th, 2010

The official launch yesterday of data.gov.uk, with an index of 2,500 datasets provided by government departments, is fantastic news – and a significant milestone for the Free Our Data campaign.

It’s worth remembering how far we’ve come since 9 March 2006, when we kicked off the campaign in Guardian Technology with Give us back our crown jewels:

Imagine you had bought this newspaper for a friend. Imagine you asked them to tell you what’s in the TV listings – and they demanded cash before they would tell you. Outrageous? Certainly. Yet that is what a number of government agencies are doing with the data that we, as taxpayers, pay to have collected on our behalf. You have to pay to get a useful version of that data. Think of Ordnance Survey’s (OS) mapping data: useful to any business that wanted to provide a service in the UK, yet out of reach of startup companies without deep pockets.

This situation prevails across a number of government agencies. Its effects are all bad. It stifles innovation, enterprise and the creativity that should be the lifeblood of new business. And that is why Guardian Technology today launches a campaign – Free Our Data. The aim is simple: to persuade the government to abandon copyright on essential national data, making it freely available to anyone, while keeping the crucial task of collecting that data in the hands of taxpayer-funded agencies.

And further on:

[The consultancy] Pira [carrying out a study for the EU] pointed out that the US’s approach brings enormous economic benefits. The US and EU are comparable in size and population; but while the EU spent €9.5bn (£6.51bn) on gathering public sector data, and collected €68bn selling and licensing it, the US spent €19bn – twice as much – and realised €750bn – over 10 times more. [Peter] Weiss [who wrote a study comparing the US and UK] pointed out: “Governments realise two kinds of financial gain when they drop charges: higher indirect tax revenue from higher sales of the products that incorporate the … information; and higher income tax revenue and lower social welfare payments from net gains in employment.”

Happily, that argument has been driven through Whitehall by the efforts of Tim Berners-Lee and Professor Nigel Shadbolt. I interviewed Berners-Lee for the Guardian: see the video or read my account of how they did it.

So is that it? Is the campaign over? No, not at all. There are plenty of holdouts: UK Hydrographic Office is complicated (because it buys in third-party data which it then resells), yet even so one would think there should be information that it collects about British coastal waters which could be released as having public benefit.

Similarly postcodes, where there is some notable opposition to making any of the datasets free. The easiest one would be PostZon, which simply holds geolocations for each postcode plus data about which health and administrative boundary it lies inside; that’s nothing like as extensive (or valuable) as the full Postcode Address File (PAF).

But there’s really strong resistance against making anything from the Royal Mail available for free, and one detects Lord Mandelson’s hand in this.

If you haven’t yet had your say on the OS consultation, Harry Metcalfe has created a terrific tool for doing precisely that at osconsult.ernestmarples.com. Go along and make your views heard.

Fun facts from the DCLG / OS consultation

Tuesday, December 29th, 2009

A few things that strike us as we read through the consultation and impact assessment (links in previous posts).

Impact assessment:

Ordnance Survey generates most of its revenue from business and the public sector; in 2008/9 they each accounted for 46 per cent of the organisation’s total revenue. Consumers, through the sale of paper maps in retailing channels, accounted for the remaining 8 per cent of sales.

Impact assessment:

Ordnance Survey generates revenues from its products through licensing arrangements either directly with customers, or indirectly through licensed partners and through retail distributors. The direct customer channel accounts for two-thirds of Ordnance Survey’s trading revenue and includes various collective purchase agreements and major private sector users such as the utility companies. Approximately 25 per cent of Ordnance Survey’s trading revenue is generated though the indirect partner channel.

Impact assessment:

Separately, there are imbalances in Ordnance Survey’s current pricing model which may be causing inefficient allocation of resources. Firstly, Ordnance Survey currently charges private sector customers of its large-scale products significantly more than comparable government customers. The higher prices being paid by the private sector may potentially have restricted consumption to the less price sensitive users, impacting the economic benefit to the economy. Secondly, the payment allocation mechanism employed by government generates a weak price signal to Ordnance Survey from individual government users within the collective agreements.

Now that’s a really interesting one. Private sector pays more than government? I hadn’t heard that before. Payment mechanism generates a weak price signal?

Impact assessment:

[OS] already has a cost reduction programme underway as part of its existing business strategy, but any long-term strategic option would seek to introduce a framework that enhances cost transparency and provides incentives to pursue further efficiency gains.

More as we come across them…

Impact assessment of making OS ‘mid-scale’ data free puts cost at 47m-58m pounds

Wednesday, December 23rd, 2009

What interesting reading the impact assessment of the DCLG consultation on making OS data free is. Clearly some arms have been twisted in the Treasury to make it happen – Liam Byrne, chief secretary to the Treasury, almost surely in the driving seat there.

On the option being chosen (which is explicitly not the one that was examined in the “Cambridge study”, which looked at the benefits of releasing large-scale data, not the “mid-scale” data being proposed) the cost seems to be that government costs rise somewhat, while costs to the commercial sector fall.

From the document (on the impact assessment page):

ANNUAL COSTS

Lost OS revenue from OS Free data being made free: £19-24m (govt would fund this on a cost plus basis, amounting to £6-9m).

Increased government charges for large-scale data: £28-34m (price rebalancing based on number of datasets used by public and private sector).

One-off (Transition) Yrs: £ tbc

Average Annual Cost (excluding one-off): £47-58m

Total Cost (PV) £391-482m

Other key non-monetised costs by ‘main affected groups’ Transition costs to Ordnance Survey, government departments and businesses of moving to new model. There would be impacts on third party providers (see Competition Assessment, Annex 1).

ANNUAL BENEFITS

Description and scale of key monetised benefits by ‘main affected groups’: gain to business and consumers from OS large-scale data being made cheaper: £28-34m if assume price rebalancing is revenue neutral.

Gain from OS Free data being made available: £19-24m.

Average Annual Benefit (excluding one-off) £47-58m

Total Benefit (PV) £391-482m

Other key non-monetised benefits by ‘main affected groups’: The lower charges to businesses and consumers for large-scale data, and the free data should increase demand and hence welfare. Entry and innovation should occur in the market for geographical information. These welfare benefits have not been quantified (Pollock report focuses on releasing large-scale data).

And finally:

Key Assumptions/Sensitivities/Risk: Modelling assumptions: some substitution from paid-for to free data; lost revenue by OS due to competition from new derived products. Not yet determined how the revenue shortfall will be covered from government (i.e. who will pay and how). So for now assume no change in demand, but will estimate this for the final IA.

Price Base Year 2009

Time Period Years 10

Net Benefit Range (NPV) –

NET BENEFIT (NPV Best estimate) £0

Hurrah! Ordnance Survey consultation is live!

Wednesday, December 23rd, 2009

Thanks to a little bird at an interested organisation, we now know that the DCLG has opened its consultation on OS data.

It’s at http://www.communities.gov.uk/publications/corporate/ordnancesurveyconsultation, where we learn that the closing date is 17 March 2010 (and the opening date is today, 23 December 2009).

Consultation paper on the Government’s proposal to open up Ordnance Survey’s data relating to electoral and local authority boundaries, postcode areas and mid scale mapping information.

The consultation document itself weighs in at 2.2MB of PDF and 91 pages.

As ever, let us know your thoughts.

Update: and don’t miss the Impact Assessment paper – here’s the PDF of the Impact Assessment – which for some strange reason isn’t linked from the main page.

Consultation update: still invisible, but asked in Parliament

Friday, December 18th, 2009

Ordnance Survey says it’s for the Department of Communities and Local Government that’s in charge of the consultation over making its data free….

According to this Parliamentary answer, DCLG thinks so too:

The question:

Mark Field (Cities of London & Westminster, Conservative)

To ask the Secretary of State for Communities and Local Government with reference to the announcement of 17 November 2009 on the Making Public Data Public initiative, when he expects to begin the consultation regarding access to Ordnance Survey data.

The answer from the DCLG minister responsible:

Ian Austin (Minister of State (the West Midlands), Regional Affairs; Dudley North, Labour)

We expect the consultation to be launched during the week beginning 14 December 2009.

That’s this week. This week is almost over. What, it takes a week to launch a consultation? There are international experts who can do it quicker. Meanwhile I tried phoning the DCLG press office (no reply on multiple lines) and emailing it (no response).

Helluva way to organise a consultation.

Anyone seen a consultation?

Friday, December 18th, 2009

The Department of Communities and Local Government is, apparently, in charge of the consultation over making OS data free.

The plan was of course that the consultation would begin “in December”.

December’s here and we haven’t seen much sign. Anyone else? We’ve put a call in to find out…

(If you need reminding about the case for making data free, see our other pages on the site, such as the articles page or the “Case for free“. Perhaps we do need to update them in the light of the studies of the past couple of years…)

Daily Telegraph: making stuff free can create revenues

Wednesday, December 9th, 2009

Hey, look, even the Daily Telegraph – hardly a home of the idea of the free lunch – is reprinting Breaking Views pieces which point out that making data free brings bigger benefits.

UK map giveaway throws bread upon the waters pretty much sings from our songbook:

The Met Office and the Ordnance Survey are unlikely candidates to stimulate another revolution. The weather forecasts may be accurate (sometimes) and the maps beautiful, but as businesses, neither is going anywhere. This is no surprise, since neither is really suited to becoming a proper commercial enterprise.

Yet the data they own is, literally, invaluable. Made freely available, all sorts of would-be entrepreneurs could exploit it to build businesses beyond the dreams of the public sector. The slightly geeky approach needed to be a successful internet entrepreneur is commonplace among mapaholics and weather nuts. Given the raw material, they could make a thousand businesses bloom.

The proposal unveiled this week is vague – a consultation document is promised later this month. The ability of the civil servants to emasculate any good idea should never be underestimated. But this is one whose time has come.

Given their tiny profits, selling off the Ordnance Survey and Met Office would raise minimal amounts. Giving away the data will undermine profits, but the benefits in terms of corporate taxes should be much larger.

Thanks. We knew you couldn’t keep a good idea down.

Outrageous. Incredible. International expert was spoken word only even within OS

Friday, November 27th, 2009

Let’s just remind ourselves what it was that Sir Rob Margetts, chair of Ordnance Survey, said at the launch of OS’s proposed new strategy (which is now in little pieces all over the floor since Gordon Brown and Tim Berners-Lee announced the end of derived data and the freeing up of mid-scale mapping, but anyway) back in April:

“We came to conclusion that the cost to government in the first five years [of a free data model] would be between £500m and £1 billion. That wasn’t the only reason that we discarded it. We did, with outside help, a review of equivalent organisations around the world.“

Who, I then asked, was the “outside help”? OS responded:

With regard to the International Comparison of Geographical Information Trading Models Study, outside help was provided by senior officials of those Institutions contacted.

In the case of the United States of America, as senior officials of the United States Geological Survey (USGS) were unavailable, Mr. David Cowen, Distinguished Professor Emeritus at the University of South Carolina, kindly provided us with an in-depth overview of the state of public sector GI data in the United States, including USGS. Mr Cowan is a former chair of the Mapping Science Committee of the United States National Research Council and is chair of the National Research Council’s Committee for the study of Land Parcel Databases.

The document was also reviewed by an internationally recognised expert in Geographical Information and National Mapping who agreed with the analysis and conclusions.

This latter bit intrigued us. An “internationally recognised expert”, eh? Except it turned out that he or she did not want to be identified, although he or she works or has worked full-time for a foreign mapping agency, and read the study for free. And that OS transacted everything with the expert by spoken word:

A copy of the report was provided to the person concerned and engagement on this matter was conducted orally with no permanent record made of these conversations.

And now in response to my latest Freedom of Information request for

copies of all emails and/or documents internally relating to the decision to choose this person – for example, discussion of who would be suitable candidates or who would not be suitable candidates to carry out the review of the report

OS replies:

There was no decision process in place to find suitable candidates. An opportunity presented itself to request the opinion of a global expert in this field which was undertaken orally. The resultant opinion was expressed orally and there was no permanent record made of these conversations.

So here’s what happens. You have a report. You happen to bump into an old mate. “Hey, want to read my report?” you say. “Sure,” they say. They read it. “Seems OK,” they say. You go back to your office and tell people “I met X who says it’s fine.” Even though the report is a thrown-together farrago of disconnected information about various national mapping agencies and their charging methods, combined with an unrelated chunk of poorly displayed data about national GDP versus national R&D expenditure, which cannot by any reasonable measure be claimed to justify anything about any charging model.

This then becomes “The document was also reviewed by an internationally recognised expert in Geographical Information and National Mapping who agreed with the analysis and conclusions.”

If there is anyone at Ordnance Survey who is prepared to defend this course of events, could they please get in touch? Or even the international expert, who is very welcome to comment anonymously to explain whether they think OS’s representation of their opinion is justified. Comments are open.

Gordon Brown announces OS maps to be free online

Wednesday, November 18th, 2009

Do we scent victory? Hell yeah. It seems that the prime minister plus the chief secretary of the Treasury plus the inventor of the world wide web collectively outrank Vanessa Lawrence, and so Gordon Brown was able to declare at a seminar at No.10 yesterday (to which I was invited, thanks for asking) that

Today, some of you may know, we are opening up Ordnance Survey information – one of the first recommendations that Tim Berners‑Lee made to us with Nigel Shadbolt in the work that they are doing. We are making Ordnance Survey material available to the up to a certain level in a way that it was not available free of charge before.

The Guardian has the story: Ordnance Survey maps to go free online:

The government is to explore ways of making all Ordnance Survey maps freely available online from April, in a victory for the Guardian’s three-year Free Our Data campaign. The move will bring the UK into line with the free publication of maps that exists in the US.

Gordon Brown announced the change at a joint event in London today with Sir Tim Berners-Lee, the inventor of the World Wide Web, who is now information tsar advising on the handing over of private government data to the public.

The government has been inspired by the success of crime mapping where “data openness” is helping citizens assess the safety of geographical areas.

Today’s announcement will be followed by a speech, due next week by the chief secretary to the Treasury, Liam Byrne, explaining how the freeing up of data, alongside the scaling back of other functions of central government, could lead to a “smarter state”.

Our understanding is that Liam Byrne was key in getting this pushed throughL since it involves financial risk – OS won’t be getting that income – the Treasury has to approve it.

Key points: it involves “mid-scale” maps from 1:10,000 upwards; and it kills off the “derived data” rows that government departments and everyone else has been having for so long. Derived data will have a stake through its heart.

Oh, and – this “free” will extend to being free for commercial use. That’s right, you’ll be able to build a business with it. Though it’s not clear yet whether you’d be able to take the maps and create *printed* ones. Must ask about that.

There will be a consultation starting in December. We’d urge any customer of OS to add in their views. And we’d urge any would-be customer who would otherwise not use the data to add their views too.

Quite where this leaves OS’s “hybrid strategy” isn’t clear. And OS doesn’t seem very clear about it either. Vanessa Lawrence wasn’t at No.10, and nor was anyone from OS, which seems surprising – you’d think they’d want to bask in the reflected glory of being praised by Tim Berners-Lee for the quality and usefulness of their data, surely?

When we asked this morning how much foregone revenue this means (since obviously giving away maps you used to charge for means less income), OS said it was “not in a position to make any comment at this time”, which seems surprising, again, because you’d think that it would have given Liam Byrne very clear indications of how far the roof would fall in if it were to do this.

Norwegian mapping authority frees up maps

Saturday, November 14th, 2009

Oh, those crazy Norwegians. They’re giving away maps for free. Within limits – it’s limited to “individuals and non-profit associations”. So that’s half a step towards free data.

Interesting that this is also including maritime information – the stuff that the UK Hydrographic Office sort-of provides; it’s rumoured to be on the list for privatisation, however.

And of course Norway was one of the countries in the OS’s woeful International Comparison report. Apparently it gets an unknown amount of government grant, but operates a “full cost recovery” system (6.1.4 in the report).

Arguably this is similar to the OS’s OpenSpace project, which is free-ish availability of data for individuals and non-profits… as long as they don’t get too big. What’s not clear in the Norwegian example is what its rules on “derived data” are. That would be interesting to know from any Norwegians.

From this Google translation page of the Associated Press story in Norwegian:

(AP) Soon, you can use much of the information at the Norwegian Mapping Authority on your private website, and take up battle with Google Maps.

From 1 December this year, you can retrieve detailed information from the Norwegian Mapping Authority, and bake it into your own web pages. Totally free, if you are an individual or operate a non-profit association.

It follows the Norwegian Mapping Authority in the footsteps of Google and the Norwegian Meteorological Institute, who shares the information it holds on to the free public use.

There is talk of detailed maps and geographic information that the Norwegian Mapping Authority has been sitting for a long time, and that until now only been available for those willing to pay.

“The data has been available through the programs that we manage, but now it will therefore be direct access to the service,” explains Erland Røed the Norwegian Mapping Authority to VG Nett.

Like for example, Google’s maps will be extended the opportunity to mix together information from several sources, and then add this on top of the map from the Norwegian Mapping Authority.

You should read the whole page (so I’m not going to copy it all here). But it does sound like the Norwegian Authority is getting with the program in a big way.

….At the State Map Verks web pages, users can decide how utstnittet of the map should look like, the layers of information to be included and finally get a clip of code.

…The system located at the bottom of the State Map Verks solutions [is] also at the forefront of the development of open software.

“Operating systems and databases we use are free software, so this is done by the book. It’s gone out 80 million map images of this year, says Røed. “The public and private individuals who have gone on our site has generated a lot of use before it is released freely, “he says.

By releasing data free on the way the Norwegian Mapping Authority do hope that the users themselves to come up with good solutions. A few examples of how users can generate more information from the maps of the Norwegian Mapping Authority is sports.

See chart to the Norwegian Mapping Authority, for example, be used as background during a real-time tracking of Færderseilasen with all available information as a true marine gear.

Such tracking is technically possible even with the Google service Google Maps, but without details that a real chart can offer.At the State Map Verks map is the depth measurements, lighthouses, beacons and all the relevant information for sailors, where Google Maps only shows the blue sea.

Similarly shows the topographical maps of the Norwegian Mapping Authority highly detailed rendering of the terrain, long more accurate than Google Maps.

There is also a reader discussion of site boundaries, which has a translation too.