Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Archive for March, 2010

And quote of the day goes to…

Wednesday, March 24th, 2010

A simply marvellous response from Martin Daly, chief technology officer of CADCorp, commenting on the AGI’s response to the Prime Minister (read it here, because you’ll want to).

He remarked on Twitter:

Loving this AGI letter re OS … with pp-d signature. It takes some cojones to pp a letter to THE FREAKING PRIME MINISTER.

And you know what? He’s absolutely right. The second signature is pp’d. “Yeah, Prime Minister, I want you to read it, but you know what? I can’t actually be there to sign it. Can’t think how I could electronically put my signature on it either. So I got that person in the office who does stuff to sign it.”

Roll on, digital age. But mind you bring my quills too.

ESRI to government: aren’t you being a little hasty in making this OS data free?

Wednesday, March 24th, 2010

ESRI has sent an open letter to government – with a number of co-signatories who are in effect competitors – fretting about the proposal (commitment really) to make a number of OS datasets free.

Below is the letter, which I’ve blockquoted to make it clear what’s the letter and what’s not. I’ve inserted some comments, based on my personal knowledge; and of course some of this is coloured by my advocacy of the Free Our Data campaign.

Dear Sir,

Following the Prime Minister’s announcement on November 17th 2009 which set out his vision for Making Public Data Public, a consultation has been taking place on three options for making certain Ordnance Survey datasets available for free with no restrictions on re-use. Submissions to the consultation must be received by March 17th.

We the undersigned, represent companies who employ 630 staff in the UK GIS industry. We represent over 50% of this market and as such are effectively competitors. Whilst each of our companies has submitted an individual response to the consultation, we are writing this letter to express our shared, serious concerns about both the manner in which this consultation is being undertaken and the potential negative impacts that could result, not for our companies, but for the Ordnance Survey and for the UK economy.

Before saying anything else, we wish to record our full support for the role of the Ordnance Survey as a world leader in the collection, maintenance and distribution of the highest quality geographic information and mapping.

CA: That’s in common with the Free Our Data campaign, which has always praised the OS’s work in collecting data and said that it should remain in government ownership.

The Government press release entitled “Re-mapping the future for Ordnance Survey – making public data public” stated that “…any change would be implemented from April 2010.” We insist that following the close of the consultation on 17th March, adequate time be allowed for a full analysis of the submissions, prior to any decision being taken. We are very concerned that the decision to release a selection of mid- and small- scale products for free appears to have already been made by the Prime Minister’s office.

CA: Actually, it was made before the consultation – he announced it back in November. The consultation was in effect asking if more should be done, or if there might be some overwhelming reason why the old OS model should be retained, with all the fun of derived data restrictions and so on that have given us so much enjoyment over the years.

The consultation document states that were this decision to be taken, it would “drive improved transparency and accountability of government and, by facilitating greater innovation, create new economic and social value”. Whilst this may drive transparency and create social value, we do not consider that any significant economic value would result.

CA: This is an interesting assertion, but it’s not backed up by any evidence. It was trivial for us to show that it costs more to restrict the use of the CodePoint database by JobCentreProPlus than it actually benefits the economy (because the fees paid to lawyers are greater than the cost of the database licence, and of the benefits that would be paid to someone who can’t find a job). As for mid-scale data, it’s similarly easy to show that people who are put off by (a) licence prices and (b) derived data rules will embrace the economic possibilities of free data.

Another question might be: what’s the specific difference between social value and economic value?

We work on a daily basis with a very wide range of public and private sector customers who need large scale Ordnance Survey data in order to improve the effectiveness and efficiency of their organisations. Currently the cost of this data puts it beyond the reach of many organisations, particularly in the private sector. If we are to create real economic value for the UK, it is the availability of large scale datasets, at reasonable prices, that needs to be addressed, rather than releasing mid- and small- scale datasets for free.

CA: the consultation does point out that the public sector has been getting its data cheap, and the private sector in effect overcharged. That’s likely to be realigned – but equally, some public sector organisations may opt out of some OS products.

The consultation document says very little about the revenue impact on the Ordnance Survey of releasing datasets for free. The Ordnance Survey is a Trading Fund which is completely self-funded from the revenues it receives from licensing its datasets. If any of these datasets were to be made freely available, this lost revenue would need to be replaced. We would be extremely concerned if this were achieved by increasing the charges for large scale datasets to either the public or private sector or both.

CA: basically, the concern is that prices of the large-scale [MasterMap level] data will be increased to pay for the free stuff.

We cannot see how any decision to release Ordnance Survey datasets for free can be made without putting the necessary changes in place to compensate the Ordnance Survey for this lost revenue. In the present economic climate we cannot see the Government or Ordnance Survey public sector customers making up this short-fall and our experience tells us that the private sector will not pay more. Furthermore it is already evident that as a result of this proposed change, Ordnance Survey is feeling considerable pressure to generate additional revenue streams, through the rapid introduction of additional innovative and/or automated product lines. These pressures may encourage Ordnance Survey to risk its long term strategies of high quality geospatical data production for the sake of short term revenue gains.

CA: well, this may be one of those extremely rare times where being a journalist, with the access it sometimes provides to decision-makers, is more useful than being in business. From speaking to ministers and the people who are working on making the data free, I’m confident that the Treasury has accepted the idea that it needs to fund the shortfall. (But there’s also an interesting question to be posed to ESRI: if OS data is so price-inelastic – that is, people will pay anything for them – and the GIS data so useful, then why can’t ESRI and the others contemplate a rise in OS pricing? After all, they’d all be using the same datasets, have the same prices, and so be able to pass them on to customers equally, because the usefulness of the datasets would have the same price inelasticity. I suspect ESRI’s sources on this are within Ordnance Survey. Mine are within Whitehall. Whitehall is where the money lives.

If the Government is serious about making Government geospatial data more readily available, it also needs to look beyond the Ordnance Survey. Current restrictions on the availability of postal address file data from The Royal Mail and Communities and Local Government identifiers for properties and streets need to be removed. Making all such data more readily available would lead to greater innovation and play an important role in placing the UK at the forefront of the knowledge economy in the twenty-first century.

CA: that sounds like a complaint about derived data, but it’s not quite clear what “restrictions… need to be removed” actually means.

We appear to be moving towards a decision to release a selection of mid- and small- scale Ordnance Survey products for free, without due regard being given to the future funding of the Ordnance Survey or whether this will drive real economic growth for the UK. This gives us serious cause for concern.

Dr Richard Waite, managing director, ESRI (UK) Ltd; Howard Papworth, executive director SG&I Western Europe, Intergraph; Dr Michael Sanderson, executive chairman, 1Spatial Group Ltd; Mike O’Neill, managing director, Cadcorp Ltd.

CA: what’s interesting about this is the lack of numbers. I’d have thought there was enough detail in the consultation that these companies could have set out their worries in financial terms – especially with a budget coming. I find the lack of hard numbers intriguing.

OS data: ‘you will like it’

Tuesday, March 23rd, 2010

Yesterday Gordon Brown fired a sort-of starting gun for the election, at least in part, by trying to get the geek vote: loads more non-personal government data to be made available for free commercial reuse, bus franchises to be obliged in future to make timetables available to others, tons of broadband. (Read the speech.)

The interesting clause, of course, being this one:

And following the strong support in our recent consultation, I can confirm that from 1st April, we will be making a substantial package of information held by ordnance survey freely available to the public, without restrictions on re-use. Further details on the package and government’s response to the consultation will be published by the end of March.

(While I was typing this, I had a call from a PR representing a group of users of OS data who are upset that within days of the consultation ending Brown seems to be heading off down one of the routes that is not the “leave it as it was” one. More to come…)

Afterwards I asked Nigel Shadbolt and Michael Wills – the latter being the minister at the Ministry of Justice who was, we think, the first inside government to give serious consideration to the ideas of the Free Our Data (read the interview with him from July 2007).

So, I asked, what’s going to be in the free OS package? Raster data? Or vector data too? To what scale? From Explorer (1:25,000) upwards, or what?

Furrowed brows. But plenty of assurances that “you will like it”. Await more in the coming days…

Ed Parsons collects a list of OS consultation responses – with Goldilocks and elephants galore (updated)

Saturday, March 20th, 2010

Ed Parsons, once upon a time the chief technology officer of Ordnance Survey but now at Google, has blogged about the OS consultation, which he describes as a fairy tale:

The consultation understandably and in my mind quite rightly has remained focused on the specifics of making OS data free, and in the great tradition of Civil Service options papers offers a Goldilocks Choice; one too cold, one too hot and one option just about acceptable.

Option 1 appears to maintain the status-quo and I don’t see anyone outside the Romsey Road distortion field supporting it.

Option 2 is perhaps something that may be achievable in the long term with continued technological change and changing market requirements, however at the moment this would put Ordnance Survey in a position where it’s current operational processes are financially unsustainable.

So Option 3 represents the obvious compromise, some small scale data for free while allowing the cash cow of MasterMap to continue to fund a reduced but largely similar OS to the one we have today.

He also points to the elephants in the room – the issue of derived data (which is mentioned only once in the whole consultation, but which impinges crucially on so much geospatial work), and the lack of a national address register:

This is an issue bigger than Ordnance Survey, although OS has had its part to play in the current mess. This really does need strategic leadership from the centre.

Helpfully, he’s also collected a list of so-far published responses to the consultation, which we’ll shamelessly republish here. One large search engine’s response seems to be missing though… must be an oversight.

Let us (and him) know if there are more public responses to the consultation. We’re expecting to hear something relevant on Monday, by the way. More closer to the time.

…and APPSI comes out swinging

Thursday, March 11th, 2010

The government’s Advisory Panel on Public Sector Information has come out with a strong response to the OS consultation.

Headline points: look at the picture, not just at OS, resolve the “fundamental contradictions” in information policy and move towards a free data regime. “In particular, OS should not have any intellectual property rights in derived data.”

Oh, and sort out the “national scandal” of the lack of a comprehensive free address register.

There’s more here. http://bit.ly/91MY2q

It’s a good read.

Tristram Cary of Getmapping responds on the OS consultation: ‘British mapping is fundamentally flawed’

Tuesday, March 9th, 2010

Time for a different viewpoint on the OS consultation. Tristram Cary is head of Getmapping, which shot to prominence from its AIM-listed position in February 2002 via a story perhaps best expressed in the opening paragraph of this story in the Independent:

The Queen has found herself unwittingly on a collision course with the Government as a result of a legal dispute involving a small aerial mapping company.

How?

Getmapping, an AIM-listed aerial photography specialist in which the Queen has a stake, is taking legal action against the Government-owned Ordnance Survey claiming it has abused its market position and breached fair pricing policies. Ordnance Survey is part of the beleaguered Stephen Byers’ Department of Transport.

Ah, yes, those halcyon days when Stephen Byers was beleaguered, and OS was in DoT. (It’s interesting to ask why it was there, and why it moved, but that’s for another day.)

Cary and Getmapping sued OS:

Getmapping has spent £6m developing an aerial map of Britain which it sells over the internet to estate agents, local authorities and civil engineers. It finished the so-called Millennium Map in May 2001 having agreed the previous September that Ordnance Survey could sell it as part of its own services.

However, last May Ordnance Survey decided it wanted a higher specification aerial map and put the contract out to tender. Getmapping claims Ordnance Survey was planning to undercut the Getmapping product even though its higher specification map would cost more to develop. It says Ordnance Survey would therefore be cross-subsidising in order to compete unfairly with the private sector.

At the time I found this lawsuit very interesting; I’d say that it was, for me, the genesis of the entire “free our data” concept, certainly regarding OS’s behaviour. (Since some might wonder, I’ll specify: I had and have absolutely no relationship, commercial or otherwise, with Getmapping or Mr Cary.)

The lawsuit was eventually settled – which in effect meant Getmapping was forced to exit the stock market:

[Getmapping] finally agreed to drop its abuse case in June and paid £125,000 to OS to cover its legal costs although Mr Cary is still angry about the unfair advantage he feels that OS gets from not having to separately account for its commercial arm from its public sector work. The government agency’s commercial activities do not pay a market rate for OS copyright material, Mr Cary claims.

(See also: OS pledges to fight after judge calls Getmapping case “very weak” (June 2002); OFT ruling on the case (May 2002); formal statement at end of case (June 2003)

Water under the bridge? Well, Getmapping is still around (and has been providing aerial data to sites such as Microsoft’s Live Maps). And Tristram Cary recently added a comment to an earlier posting, which we thought would be worth reprinting here, since not everyone subscribes to our full comments feed.

So here’s Getmapping’s take on the consultation (though it’s not their formal response, which is of course far more detailed.)

Tristram Cary writes:

I take a very different view of the OS Consultation and I would like to outline my thoughts here. If there is an opportunity to provide a fuller explanation in the Free Our Data blog then I would be happy to do so.

I also would be very happy for anyone to base a response to the OS Consultation on these ideas.

I think that the way that British Mapping is managed is fundamentally flawed, and that as a result British mapping is not nearly as good as it could be. The main flaws are:

a) OS has become detached from the main sources of mapping change (in particular Local Government). When Local Government authorises a change it does not inform OS and OS therefore has to find out about the change from secondary sources. For example OS does not have access to the Local Land and Property Gazetteers (LLPGs) from Local Authorities which list all changes which affect the NLPG. As a result OS maps lag far behind the ‘truth’ on the ground (typically 6-12 months) and there is a substantial waste of surveying effort, as changes are logged and mapped twice (once by the Local Authority and once again by OS).

b) Maps are Value Added Products. Maps are combination of various low level ‘Elemental Datasets’ superimposed for convenience. Some of these Elemental Datasets are Natural Government Monopolies (NGMs) (eg District Boundaries, SSSIs, street names etc), but others are not (eg coastlines, rivers, height data). Thirty years ago it made sense for maps to be the lowest level of Core Geographical data, but in the modern era of GIS and digital mapping it makes more sense to maintain each Elemental Dataset separately and to allow the private sector or individual users to combine them to form maps. Why? Mainly because the role of government should be minimised to encourage competition and customer choice – this implies that the Government should only look after the Natural Government Monopoly datasets such as Boundaries, SSSIs street names etc. Over the last ten years good progress has been made in opening up the mapping market to genuine competition, and there are now several sources of OS independent mapping including NavTeq, Bartholemews, UK Maps and our own People’s Map. The great downside of giving away map products as free PSI is that it will kill the competition for those maps. It would be much better to give away the Natural Government Monopoly data ONLY and allow the free market to develop all the higher-level mapping products.

c) The Derived Data Trap. The OS is currently in a dominant (almost monopoly) position for the provision of large scale maps. OS maintains that position partly by claiming IPR on data which a user derives using an OS map as a reference, even if the data does not appear on the OS map in the first place. For instance if a utility draws its pipeline network using an OS map as a reference than OS says that if the utility stops paying its annual licence fee to OS then the utility must delete all its derived pipeline data and recreate it from scratch. This acts as a effective barrier to the creation of a truly competitive market for maps for two reasons. First the main customers (government and utilities) are locked in because if they change map supplier then they face having to re-create their derived data. And second because the independent map suppliers cannot get hold of any derived data without paying OS a fee and negotiating licence terms. Boundaries are a good example of this. The Boundary Commission cannot supply its boundaries to the private mapping sector because they are drawn on top of OS Maps and are therefore considered to contain OS copyright.

Because of all these issues, Getmapping’s view on the OS Consultation is as follows:

a) All Natural Government Monopoly (NGM) data should be made available free of charge to the market as PSI, but without any non-NGM data included

b) No maps which contain non-NGM data should be made available free of charge as this will severely damage competition, customer choice and quality in the market

c) In the medium term responsibility for maintaining NGM data should be transferred from OS to the natural owners of the data (eg Local Government for street naming and numbering) and the maintenance of the data should become an integral part of the change process, funded by the organisation which causes the change. For example, if somebody extends their house, then the builder should commission an independent surveyor to make a detailed survey of the as-built change to a specified format and accuracy. This survey would be paid for by the owner of the building and it would be used as the formal NGM update record. An intelligent properly-connected process of this kind would allow maps to be kept up to date within hours of a change, not within 6-12 months

d) It follows that in the medium term OS’s role should become a QA and government-procurement role. This is because if NGM data is maintained by the ‘owning’ government department and made available as PSI then the private mapping sector will be able to make high quality maps to suit all market demands. I believe that the quality, choice and prices of maps would all improve as a result.

Your comments are welcome. The “derived data” is especially interesting, particularly as it applies to utilities. We’d love to hear from utilities or other customers whether this is indeed the implication of the contract – because we suspect it is, given previous form on the derived data question.

And while we’re talking about Land Registry.. here comes UKMap

Friday, March 5th, 2010

Verrry interesting post over at the UKMapping blog. We’ll include the full content because it’s short and very relevant.

Great news was received today at UKMap HQ. After much review and testing Land Registry have confirmed [and are happy for us to tell people] that they are happy to accept registrations based on UKMap.

Great news for all those consultants, government types and property professional using UKMap.

Full text below

“Following a review of UKMap, Land Registry is able to confirm that UKMap meets Land Registry’s requirements as a mapping base on which registration applications can be made and is comfortable accepting registration applications based on UKMap. Such registration applications must still follow relevant guidance as set out in Public Guide 40.”

What does this mean? That LR doesn’t necessarily have to rely on Ordnance Survey maps for registrations. That, too, pours sand into any engines that might be getting started looking to fund OS through higher LR transaction fees. Savvy users would just go to UKMap.

And an earlier post on that same blog includes these interesting new clients:

  • London Fire, serious users with a serious need to get better mapping. Now they have that with UKMap.
  • London Borough of Islington – managing their green environment, UKMap’s trees and Land Use makes all the difference for them.
  • Mott Macdonald – detailed city centre mapping, all in glorious 3D.
  • Promap – the UK’s leading mapping portal for the Land and Property market have announced they are taking UKMap

London councils and emergency services? That’s what I think you call an inroad into OS’s market, isn’t it?

A spoke in the wheels of Land Registry transaction fees to pay for Ordnance Survey?

Thursday, March 4th, 2010

An interesting debate about the prospect of redundancies at Land Registry on Wednesday brought Michael Wills, of the Ministry for Justice – who is also in charge of the (trading fund) Land Registry – to the chamber.

It’s a long debate, but here’s the nitty-gritty:

Transaction levels, which are the key factor for the Land Registry, will have fallen from 16.1 million in 2007-08 to a projected 10 million in 2009-10. The Land Registry receives no central funding because it is a trading fund; it depends on the fees that it receives for services rendered. It made a loss of £130 million in 2008-09 compared with a surplus of about £70 million in 2007-08.

Very interesting – and this comes against a backdrop where it’s being suggested that government is being charged too little for OS data (compared to the private sector); and where there have been suggestions (hi, Robert Barr) that we should have an excess added to LR transactions in order to fund OS.

I think this really needs to be costed. It would be interesting too to see how much Land Registry pays to OS…

What should be done with Royal Mail’s PAF? (From 1999)

Tuesday, March 2nd, 2010

This letter first appeared in the Financial Times on Friday July 30 1999. (You can see it at http://twitpic.com/z7y0m – which we can’t embed it for technical reasons.) We’re using it here because it’s rather interesting historically – especially for the signatories, and particularly one of them. You’ll notice, of course, that it didn’t work out like they asked….


Sir
The financial and commercial freedom that plc status will confer on the Post Office may be a welcome contribution to national competitiveness. However, one Post Office asset, namely the Postcode Address File (PAF), a computerised and maintained list of all postal delivery addresses and their postcodes, is too important a part of the national information infrastructure to be handed over without safeguards.

At present the Post Office receives new address information from local authorities, attaches additional information to optimise the address for postal delivery and allocates a postcode. This information is compiled into the PAF, which is copyrighted and published by the Post Office. IT is also made available through a number of Value Added Resellers (VAR). These data are used by thousands of commercial enterprises, large and small, for the maintenance of customer records and for a wide range of marketing and logistical purposes.

As a public corporation the Post Office has handled its monopoly position, as the national compiler of postal addresses, responsible. However, some have questioned the price of the information and the control the Post Office has exercised over its reuse and resale. Once the Post Office is a plc, directors tasked with maximising shareholder value could be tempted to extract further advantage from the PAF by restricting competitors’ access to the data, placing constraints on the operations of VARs, or charging royalty payments for the use of addresses in other contexts.

To ensure that the Post Office cannot succumb to such temptations as a plc, we would propose that the production, maintenance and placement of the PAF in the public domain should become a regulatory requirement for the Post Office in exchange for the privilege of retaining monopoly rights for the delivery of letters. This would ensure that the national address file becomes a public good to be used for the benefit of all, rather than an unregulated private asset.

Robert Barr, sr lecturer, school of geography, University of Manchester
Keith Dugmore, managing director, Demographic Decisions
Philip Good, managing director, Hopewiser
Robert James, independent consultant
Vanessa Lawrence, chair, Association for Geographic Information
Christopher Roper, director, Landmark Information Group
Richard Webber, director, Experian