Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Freeing the oceans’ data: new Estonian startup aims to do just that

July 10th, 2012

Ben Rooney writing in the Wall Street Journal, in an article titled “Public Ocean Data Made Available“:

An Estonian start-up is offering a single point of access to much of the world’s available oceanographic data.

Marinexplore, based in Tallinn, has pulled together a large number of public and commercial datasets, to provide over three petabytes (3 million gigabytes) of public ocean data.

“The tools and working processes for ocean exploration have changed little over the last 15 years,” said Rainer Sternfeld, founder and CEO of Marinexplore.

That’s a hell of an ambitious plan. It’s not just the cost of getting oceanographic data out there; it’s also the challenge of getting around the multitude of copyright entanglements in them. When the Free Our Data campaign was in full swing (or doing its jazz hands if you prefer), we prodded at the UK Hydrographic Office a few times – principally over tide data, since that’s hardly what you’d call “international” – but soon discovered that the people there are even more aware of the problems in disentangling quite who owns what. Making the UKHO data free might end up just meaning less data availability altogether.

“Most of the data comes from public sources”, the story says, which is telling in itself. There are lots of sources, but the problem often comes about in determining which is canonical – that is, which ones you really trust. One of the challenges for European countries has been making sure that simple things like sea levels are the same across borders. It’s a bit embarrassing if the data say it dips or rises by a few centimetres when you take one step along the shore from one country to another.

Which is why it’s no surprise to find that near the end of the story we learn that

Most of the data is from public data sets such as the US NOAA (National Oceanic and Atmosphere Administration) stations

. Yes, that’s the US taxpayers-pay-government-collects-and-gives-back model again.

This is course means that the datasets when it comes to locations such as the UK may be severely restricted both in use, detail and timeliness. We haven’t examined it yet – if anyone gets a change to look at Marinexplore’s data and assess it, please let us know.

(As a sidenote: isn’t it fascinating how the the US government and its citizens are perfectly happy to run a system where everyone contributes to the cost of collecting data about the oceans and then gets it back for free at the point of use – even if some won’t use it, for example because they live in landlocked states – but cannot countenance the idea of everyone contributing to the cost of healthcare which they then get back for free at the point of use? One is the peoples’ right to what their tax dollars are spent on; the other is, apparently, socialism. Beats us. We still like the US model for data payment/collection/use.)

    The following posts may also be related (the database thinks...):
  • None

Environment Agency… sells our data?

June 15th, 2011

We asked about a year ago whether we could declare the campaign done, finished, over. And the answer was clearly no, we can’t.

We said at the time that we’d consider it done when the Environment Agency began making its data available for free.

Well, the big red phone has been flashing in the Free Our Data batcave. Things are not how they should be. We’re hearing reports that it is selling data.

Here’s an email we received today:

“As you might know, the Environment Agency in England and Wales manage/collect/produce several datasets. Back in 2007 you wrote an article about ononemap using EA’s flood maps and how they forced the website admin to remove it from ononemap website.

(Yes, we remember that.)

“Unfortunately, EA has gone far beyond that.” (We’ve redacted the rest for confidentiality reasons.)

We’ll file a Freedom of Information request to see if we can get more details. Stay tuned.

    The following posts may also be related (the database thinks...):
  • None

Vive les données publiques ouvertes!

July 13th, 2010

Many thanks to the organisers of the excellent LIFT10 conference in Marseille for inviting us to tell the story of our campaign. There’s an account here:

Best of luck to our co-campaigners in France and elsewhere.

    The following posts may also be related (the database thinks...):
  • None

Is the campaign won? What do you think?

June 4th, 2010

Choices by anemoneprojectors.

Choices. Photo by anemoneprojectors on Flickr. Some rights reserved
Simon Rogers writes at the Guardian about the release of a snapshot of the Treasury’s COINS (Combined Online Information System):

“known universally as Coins [it] is the most detailed record of public spending imaginable. Some 24m individual spending items in a CSV file of 120GB presents a unique picture of how the government does its business.”

Although that’s because it was presented in UTF-16, which would be great if it were encoding Mandarin (in-joke?), but which actually meant that every second byte was blank – so the Guardian’s developers simply did a quick conversion to UTF-8, halving the size at a keystroke.

He notes:

“This is a different kind of database. It shows how the government actually works; the millions of tiny items that make up the billions of public expenditure every year. It could well be the government’s largest database: if you know of anything of equivalent size and complexity let us know – we can’t come up with anything.”

And then he comments:

“It was only 2006 that the Guardian launched the Free Our Data campaign to push for the government to release public data that we’ve paid for but was previously hidden behind paywalls or official secrecy. Now, that battle is won.”

That’s an interesting one. For me, Coins isn’t actually the marker for the point where we can hang up our campaigning clothes. The Ordnance Survey data release was a huge step. But actually, for me the point at which I’ll feel we’ve really reached the parts that we want to reach is when the Environment Agency makes its flood map data available for free commercial re-use. (I haven’t asked Mike Cross. Perhaps he’ll pitch in.)

But what do you think? It’s your campaign too. Is everything that needs to be done, done? Or is there more to be done, and if so, what?

    The following posts may also be related (the database thinks...):
  • None

Local council? Want to publish your data? Here’s how

June 2nd, 2010

In a very timely fashion, has come up with a blogpost explaining for any local authorities who want to know (and are listening/reading) how to publish itemised local authority expenditure.

Publishing raw data quickly is an immediate priority, but in the medium term local authorities should work towards structured, regularly updated data published on the Web using open standards. Subject to other issues below, our immediate advice to local authorities is:

  • Users will be interested in the core information held in the accounts system – such as expenditure code, amount paid, transaction date, beneficiary, and payment reference number. The expenditure code has to be explained and steps taken to help users identify the
  • As a first stage, publish the raw data and any lookup table needed to interpret it in a spreadsheet as a CSV or XML file as soon as possible. This should be put on the council’s website as a document for anyone to download. Or even published in a service such as Google Docs
  • There is not yet a national approach for publishing local authority expenditure data. This should not stop publication of data in its raw, machine-readable form. Observing such raw data being used is the only route to a national approach, should one be required
  • Publishing raw data will allow the panel and others to assess how that data could/should be presented to users. Sight of the data is worth a hundred meetings. Members of the panel will study the data, take part in the discussion and revise this advice.
  • As a second stage, informed by the discussion, the panel and users can then give feedback about publishing data (RDF, CSV, etc) in a way that can be consistent across all local authorities involving structured, regularly updated data published on the Web using openstandards.

There’s more too, about experiences of some of the councils which have published data. Very well worth reading.

    The following posts may also be related (the database thinks...):
  • None

The open data flood begins here: Cameron writes to central and local government

June 2nd, 2010

OK – stand by: the data flood is about to get started.

David Cameron has declared his intention to make a lot more data open. First there was the podcast (with, helpfully, transcript), and then a letter to government departments.

Remember that this is following on from the Government Transparency page of the Coalition programme. (We added our own comment there, as you’ll see.)

The podcast first – which was recorded on the Saturday as David Laws was caught in the web of expenses:

“If there’s one thing I’ve noticed since doing this job, it’s how all the information about government – the money it spends, where it spends it, the results it achieves – how so much of it is locked away in a vault marked sort of private for the eyes of ministers and officials only.

“I think this is ridiculous. It’s your money, your government, you should know what’s going on.

“So we’re going to rip off that cloak of secrecy and extend transparency as far and as wide as possible. By bringing information out into the open, you’ll be able to hold government and public services to account. You’ll be able to see how your taxes are being spent. Judge standards in your local schools and hospitals. Find out just how effective the police are at fighting crime in your community.

“Now I think that’s going to do great things. It’s certainly going to save us money.”

That’s just an extract – it’s quite forceful, though of course there’s a big gap between speaking forcefully and actually getting a result. (It’s called politics.)

Then there’s his letter that he wrote to government departments:

Central government spending transparency

  • Historic COINS spending data to be published online in June 2010.
  • All new central government ICT contracts to be published online from July 2010.
  • All new central government lender documents for contracts over £10,000 to be published on a single website from September 2010, with this information to be made available to the public free of charge.
  • New items of central government spending over £25,000 to be published online from November 2010.
  • All new central government contracts to be published in full from January 2011.
  • Full information on all DFID international development projects over £500 to be published online from January 2011, including financial information and project documentation.

    Local government spending transparency

    • New items of local government spending over £500 to be published on a council-by-council basis from January 2011.
    • New local government contracts and tender documents for expenditure over £500 to be published in full from January 2011.

    Other key government datasets

    • Crime data to be published at a level that allows the public to see what is happening on their streets from January 2011.
    • Names, grades, job titles and annual pay rates for most Senior Civil Servants with salaries above £150,000 to be published in June 2010.
    • Names, grades, job titles and annual pay rates for most Senior Civil Servants and NDPB officials with salaries higher than the lowest permissible in Pay Band 1 of the Senior Civil Service pay scale to be published from September 2010.
    • Organograms for central government departments and agencies that include all staff positions to be published in a common format from October 2010.

    Can’t wait, personally. COINS will be an amazing resource – see what the Guardian’s Datablog has to say on that.

    An interesting side note of the sort of that needs to be done can be found among the comments on that Government Transparency page, from “Algernon Arry”:

    “Has the policy of forcing councils to publish details of their spends over £500 been thoroughly thought through. A Government committed to reducing waste and here it is proposing that local councils add to their bureaucratic processes. If this is forced onto local councils I forecast that they will need an army of bureaucrats to carry it out.”

    “Logistically, think of all the times when a payment greater than £500 will be made. A couple come immediately to mind. What about everytime a child or a vulnerable person is taken into care, we all know the cost of nursing/care homes do you expect their details to be given out? (Probably covered by Data Protection legislation, so that will have to be changed.) Or everytime an order is placed to repair an piece of defective tarmac on a road surface. And in both cases what added value does it provide, and honestly how many residents are actually ineterested [sic]. Most councils have scrutiny and audit committees where these issues can be debated.

    “I don’t want to pay a penny more in Council-tax, so the Government can make an instant saving by dropping this stupid idea.”

    The idea that it will take extra human effort to make this happen, and that it will have to be done by hand – rather than being a question of adding an output stream between the existing accounting system and the web – may also have some traction among some of the more hidebound councils, one suspects. Data wranglers beware; it may require some concrete examples.

    • The following posts may also be related (the database thinks...):
    • None

    Goodbye Gordon Brown: but thanks for the data … and the campaign goes on

    May 12th, 2010

    Gordon Brown’s stint as prime minister is over. But we can thank him for one thing he left behind: the commitment by the Treasury to fund free data from the Ordnance Survey (by my understanding, for at least five years – which I think is at least what it will take for really useful commercial applications to emerge from the availability of the data).

    That’s a huge step. When Mike Cross and I started the Free Our Data campaign in March 2006, Tony Blair was prime minister. We knew that there was a strong reason for it, but it took time to get traction. Our first meeting with a minister was with Baroness (Cathy) Ashton at the Ministry of Justice; she didn’t seem too interested.

    Once Gordon Brown came into office and there was a change at ministerial level, things changed dramatically. We got audiences and found ministers who were largely sympathetic. Brown too understood the idea – which simply took off when he found himself sitting beside Tim Berners-Lee at a dinner and started making conversation.

    Brown asked: “What’s the most important technology right now? How should the UK make the best use of the internet?”

    To which the invigorated Berners-Lee replied: “Just put all the government’s data on it.”

    To his surprise, Brown simply said “OK, let’s do it.”

    Now the Conservatives and Liberal Democrats are in power. The Conservative manifesto contains a pledge to access to government data:

    Drawing inspiration from administrations around the world which have shown that being transparent can transform the effectiveness of government, we will create a powerful new right to government data, enabling the public to request – and receive – government datasets in an open and standardised format. independent estimates suggest this could provide a £6 billion boost to the UK economy. We will open up Whitehall recruitment by publishing central government job vacancies online, saving costs and increasing transparency.

    That £6bn number comes, I believe, from the Cambridge study – though that included making OS Mastermap data free. I don’t think that that will be done, given the commitment to extra public spending it would involve in the short term and the long-term payback it would need.

    I also think that while it’s nice to publish central government jobs online, there will be problems in how you slice it so that people can find the jobs they want. You might find that it’ll become something that other sites – and of course businesses – will exploit as a raw data feed and sell access to, or improve. (Yes, I know it’s also a scheme which is about chopping the funding of the Guardian’s public-sector jobs supplement off at the knees; there have been elements within the Tory party which have wanted to do this for years.)

    In short: the campaign continues, but the Con-Lib coalition has indicated that it has a lot of the right instincts. Once we know which ministers we need to lobby – and once they know what their viewpoints are – we’ll be pushing the campaign again. There’s still so much data in there which needs to be freed.

      The following posts may also be related (the database thinks...):
    • None

    Ordnance Survey data goes free: yes, we had noticed (just hadn’t written it…)

    April 9th, 2010

    OS OpenData Meridian 2 rendered with OpenStreetMap’s Mapnik. Photo on Flickr. Map data: Ordnance Survey.

    Apologies for not writing about this sooner; we have been busy and all sorts.

    So, go get your Ordnance Survey data.

    As you’ll surely have noticed, Ordnance Survey has made a chunk of data available for free personal or commercial reuse, under a licence that equates to Creative Commons Attribution – you have to say what’s in it.

    Disappointingly the 1:25K and 1:50K datasets (Explorer and Landranger) were not released – this has been a huge disappointment to paper mapmakers and others who wanted to do some innovative things with them. The Ramblers’ Association in particular had really been hoping for those.

    Great stuff with the Boundary Line data, CodePoint (postcode to lat/long) and Meridian data. CodePoint on its own is saving lots of people lots of money – Cyclestreets reckons that’s £2,000 of money it doesn’t have to lay out annually to run its site.

    OpenStreetMap is still discussing the implications of the licence and whether it wants to import the data. But it has set up a parallel project which uses it.

    More visualisations via OSM:

    For us, this doesn’t feel like the end of the road; it’s much more like the beginning, where now we have to see what people are going to do with it to prove the usefulness of the idea. And of course there are so many more potential data sources that should go free – tide times and flood maps being two which spring to mind. Come on, UK Hydrographic Office and Environment Agency.

    We welcome your opinions.

      The following posts may also be related (the database thinks...):
    • None

    And quote of the day goes to…

    March 24th, 2010

    A simply marvellous response from Martin Daly, chief technology officer of CADCorp, commenting on the AGI’s response to the Prime Minister (read it here, because you’ll want to).

    He remarked on Twitter:

    Loving this AGI letter re OS … with pp-d signature. It takes some cojones to pp a letter to THE FREAKING PRIME MINISTER.

    And you know what? He’s absolutely right. The second signature is pp’d. “Yeah, Prime Minister, I want you to read it, but you know what? I can’t actually be there to sign it. Can’t think how I could electronically put my signature on it either. So I got that person in the office who does stuff to sign it.”

    Roll on, digital age. But mind you bring my quills too.

      The following posts may also be related (the database thinks...):
    • None

    ESRI to government: aren’t you being a little hasty in making this OS data free?

    March 24th, 2010

    ESRI has sent an open letter to government – with a number of co-signatories who are in effect competitors – fretting about the proposal (commitment really) to make a number of OS datasets free.

    Below is the letter, which I’ve blockquoted to make it clear what’s the letter and what’s not. I’ve inserted some comments, based on my personal knowledge; and of course some of this is coloured by my advocacy of the Free Our Data campaign.

    Dear Sir,

    Following the Prime Minister’s announcement on November 17th 2009 which set out his vision for Making Public Data Public, a consultation has been taking place on three options for making certain Ordnance Survey datasets available for free with no restrictions on re-use. Submissions to the consultation must be received by March 17th.

    We the undersigned, represent companies who employ 630 staff in the UK GIS industry. We represent over 50% of this market and as such are effectively competitors. Whilst each of our companies has submitted an individual response to the consultation, we are writing this letter to express our shared, serious concerns about both the manner in which this consultation is being undertaken and the potential negative impacts that could result, not for our companies, but for the Ordnance Survey and for the UK economy.

    Before saying anything else, we wish to record our full support for the role of the Ordnance Survey as a world leader in the collection, maintenance and distribution of the highest quality geographic information and mapping.

    CA: That’s in common with the Free Our Data campaign, which has always praised the OS’s work in collecting data and said that it should remain in government ownership.

    The Government press release entitled “Re-mapping the future for Ordnance Survey – making public data public” stated that “…any change would be implemented from April 2010.” We insist that following the close of the consultation on 17th March, adequate time be allowed for a full analysis of the submissions, prior to any decision being taken. We are very concerned that the decision to release a selection of mid- and small- scale products for free appears to have already been made by the Prime Minister’s office.

    CA: Actually, it was made before the consultation – he announced it back in November. The consultation was in effect asking if more should be done, or if there might be some overwhelming reason why the old OS model should be retained, with all the fun of derived data restrictions and so on that have given us so much enjoyment over the years.

    The consultation document states that were this decision to be taken, it would “drive improved transparency and accountability of government and, by facilitating greater innovation, create new economic and social value”. Whilst this may drive transparency and create social value, we do not consider that any significant economic value would result.

    CA: This is an interesting assertion, but it’s not backed up by any evidence. It was trivial for us to show that it costs more to restrict the use of the CodePoint database by JobCentreProPlus than it actually benefits the economy (because the fees paid to lawyers are greater than the cost of the database licence, and of the benefits that would be paid to someone who can’t find a job). As for mid-scale data, it’s similarly easy to show that people who are put off by (a) licence prices and (b) derived data rules will embrace the economic possibilities of free data.

    Another question might be: what’s the specific difference between social value and economic value?

    We work on a daily basis with a very wide range of public and private sector customers who need large scale Ordnance Survey data in order to improve the effectiveness and efficiency of their organisations. Currently the cost of this data puts it beyond the reach of many organisations, particularly in the private sector. If we are to create real economic value for the UK, it is the availability of large scale datasets, at reasonable prices, that needs to be addressed, rather than releasing mid- and small- scale datasets for free.

    CA: the consultation does point out that the public sector has been getting its data cheap, and the private sector in effect overcharged. That’s likely to be realigned – but equally, some public sector organisations may opt out of some OS products.

    The consultation document says very little about the revenue impact on the Ordnance Survey of releasing datasets for free. The Ordnance Survey is a Trading Fund which is completely self-funded from the revenues it receives from licensing its datasets. If any of these datasets were to be made freely available, this lost revenue would need to be replaced. We would be extremely concerned if this were achieved by increasing the charges for large scale datasets to either the public or private sector or both.

    CA: basically, the concern is that prices of the large-scale [MasterMap level] data will be increased to pay for the free stuff.

    We cannot see how any decision to release Ordnance Survey datasets for free can be made without putting the necessary changes in place to compensate the Ordnance Survey for this lost revenue. In the present economic climate we cannot see the Government or Ordnance Survey public sector customers making up this short-fall and our experience tells us that the private sector will not pay more. Furthermore it is already evident that as a result of this proposed change, Ordnance Survey is feeling considerable pressure to generate additional revenue streams, through the rapid introduction of additional innovative and/or automated product lines. These pressures may encourage Ordnance Survey to risk its long term strategies of high quality geospatical data production for the sake of short term revenue gains.

    CA: well, this may be one of those extremely rare times where being a journalist, with the access it sometimes provides to decision-makers, is more useful than being in business. From speaking to ministers and the people who are working on making the data free, I’m confident that the Treasury has accepted the idea that it needs to fund the shortfall. (But there’s also an interesting question to be posed to ESRI: if OS data is so price-inelastic – that is, people will pay anything for them – and the GIS data so useful, then why can’t ESRI and the others contemplate a rise in OS pricing? After all, they’d all be using the same datasets, have the same prices, and so be able to pass them on to customers equally, because the usefulness of the datasets would have the same price inelasticity. I suspect ESRI’s sources on this are within Ordnance Survey. Mine are within Whitehall. Whitehall is where the money lives.

    If the Government is serious about making Government geospatial data more readily available, it also needs to look beyond the Ordnance Survey. Current restrictions on the availability of postal address file data from The Royal Mail and Communities and Local Government identifiers for properties and streets need to be removed. Making all such data more readily available would lead to greater innovation and play an important role in placing the UK at the forefront of the knowledge economy in the twenty-first century.

    CA: that sounds like a complaint about derived data, but it’s not quite clear what “restrictions… need to be removed” actually means.

    We appear to be moving towards a decision to release a selection of mid- and small- scale Ordnance Survey products for free, without due regard being given to the future funding of the Ordnance Survey or whether this will drive real economic growth for the UK. This gives us serious cause for concern.

    Dr Richard Waite, managing director, ESRI (UK) Ltd; Howard Papworth, executive director SG&I Western Europe, Intergraph; Dr Michael Sanderson, executive chairman, 1Spatial Group Ltd; Mike O’Neill, managing director, Cadcorp Ltd.

    CA: what’s interesting about this is the lack of numbers. I’d have thought there was enough detail in the consultation that these companies could have set out their worries in financial terms – especially with a budget coming. I find the lack of hard numbers intriguing.

      The following posts may also be related (the database thinks...):
    • None

    OS data: ‘you will like it’

    March 23rd, 2010

    Yesterday Gordon Brown fired a sort-of starting gun for the election, at least in part, by trying to get the geek vote: loads more non-personal government data to be made available for free commercial reuse, bus franchises to be obliged in future to make timetables available to others, tons of broadband. (Read the speech.)

    The interesting clause, of course, being this one:

    And following the strong support in our recent consultation, I can confirm that from 1st April, we will be making a substantial package of information held by ordnance survey freely available to the public, without restrictions on re-use. Further details on the package and government’s response to the consultation will be published by the end of March.

    (While I was typing this, I had a call from a PR representing a group of users of OS data who are upset that within days of the consultation ending Brown seems to be heading off down one of the routes that is not the “leave it as it was” one. More to come…)

    Afterwards I asked Nigel Shadbolt and Michael Wills – the latter being the minister at the Ministry of Justice who was, we think, the first inside government to give serious consideration to the ideas of the Free Our Data (read the interview with him from July 2007).

    So, I asked, what’s going to be in the free OS package? Raster data? Or vector data too? To what scale? From Explorer (1:25,000) upwards, or what?

    Furrowed brows. But plenty of assurances that “you will like it”. Await more in the coming days…

      The following posts may also be related (the database thinks...):
    • None

    Ed Parsons collects a list of OS consultation responses – with Goldilocks and elephants galore (updated)

    March 20th, 2010

    Ed Parsons, once upon a time the chief technology officer of Ordnance Survey but now at Google, has blogged about the OS consultation, which he describes as a fairy tale:

    The consultation understandably and in my mind quite rightly has remained focused on the specifics of making OS data free, and in the great tradition of Civil Service options papers offers a Goldilocks Choice; one too cold, one too hot and one option just about acceptable.

    Option 1 appears to maintain the status-quo and I don’t see anyone outside the Romsey Road distortion field supporting it.

    Option 2 is perhaps something that may be achievable in the long term with continued technological change and changing market requirements, however at the moment this would put Ordnance Survey in a position where it’s current operational processes are financially unsustainable.

    So Option 3 represents the obvious compromise, some small scale data for free while allowing the cash cow of MasterMap to continue to fund a reduced but largely similar OS to the one we have today.

    He also points to the elephants in the room – the issue of derived data (which is mentioned only once in the whole consultation, but which impinges crucially on so much geospatial work), and the lack of a national address register:

    This is an issue bigger than Ordnance Survey, although OS has had its part to play in the current mess. This really does need strategic leadership from the centre.

    Helpfully, he’s also collected a list of so-far published responses to the consultation, which we’ll shamelessly republish here. One large search engine’s response seems to be missing though… must be an oversight.

    Let us (and him) know if there are more public responses to the consultation. We’re expecting to hear something relevant on Monday, by the way. More closer to the time.

      The following posts may also be related (the database thinks...):
    • None

    …and APPSI comes out swinging

    March 11th, 2010

    The government’s Advisory Panel on Public Sector Information has come out with a strong response to the OS consultation.

    Headline points: look at the picture, not just at OS, resolve the “fundamental contradictions” in information policy and move towards a free data regime. “In particular, OS should not have any intellectual property rights in derived data.”

    Oh, and sort out the “national scandal” of the lack of a comprehensive free address register.

    There’s more here.

    It’s a good read.

      The following posts may also be related (the database thinks...):
    • None

    Tristram Cary of Getmapping responds on the OS consultation: ‘British mapping is fundamentally flawed’

    March 9th, 2010

    Time for a different viewpoint on the OS consultation. Tristram Cary is head of Getmapping, which shot to prominence from its AIM-listed position in February 2002 via a story perhaps best expressed in the opening paragraph of this story in the Independent:

    The Queen has found herself unwittingly on a collision course with the Government as a result of a legal dispute involving a small aerial mapping company.


    Getmapping, an AIM-listed aerial photography specialist in which the Queen has a stake, is taking legal action against the Government-owned Ordnance Survey claiming it has abused its market position and breached fair pricing policies. Ordnance Survey is part of the beleaguered Stephen Byers’ Department of Transport.

    Ah, yes, those halcyon days when Stephen Byers was beleaguered, and OS was in DoT. (It’s interesting to ask why it was there, and why it moved, but that’s for another day.)

    Cary and Getmapping sued OS:

    Getmapping has spent £6m developing an aerial map of Britain which it sells over the internet to estate agents, local authorities and civil engineers. It finished the so-called Millennium Map in May 2001 having agreed the previous September that Ordnance Survey could sell it as part of its own services.

    However, last May Ordnance Survey decided it wanted a higher specification aerial map and put the contract out to tender. Getmapping claims Ordnance Survey was planning to undercut the Getmapping product even though its higher specification map would cost more to develop. It says Ordnance Survey would therefore be cross-subsidising in order to compete unfairly with the private sector.

    At the time I found this lawsuit very interesting; I’d say that it was, for me, the genesis of the entire “free our data” concept, certainly regarding OS’s behaviour. (Since some might wonder, I’ll specify: I had and have absolutely no relationship, commercial or otherwise, with Getmapping or Mr Cary.)

    The lawsuit was eventually settled – which in effect meant Getmapping was forced to exit the stock market:

    [Getmapping] finally agreed to drop its abuse case in June and paid £125,000 to OS to cover its legal costs although Mr Cary is still angry about the unfair advantage he feels that OS gets from not having to separately account for its commercial arm from its public sector work. The government agency’s commercial activities do not pay a market rate for OS copyright material, Mr Cary claims.

    (See also: OS pledges to fight after judge calls Getmapping case “very weak” (June 2002); OFT ruling on the case (May 2002); formal statement at end of case (June 2003)

    Water under the bridge? Well, Getmapping is still around (and has been providing aerial data to sites such as Microsoft’s Live Maps). And Tristram Cary recently added a comment to an earlier posting, which we thought would be worth reprinting here, since not everyone subscribes to our full comments feed.

    So here’s Getmapping’s take on the consultation (though it’s not their formal response, which is of course far more detailed.)

    Tristram Cary writes:

    I take a very different view of the OS Consultation and I would like to outline my thoughts here. If there is an opportunity to provide a fuller explanation in the Free Our Data blog then I would be happy to do so.

    I also would be very happy for anyone to base a response to the OS Consultation on these ideas.

    I think that the way that British Mapping is managed is fundamentally flawed, and that as a result British mapping is not nearly as good as it could be. The main flaws are:

    a) OS has become detached from the main sources of mapping change (in particular Local Government). When Local Government authorises a change it does not inform OS and OS therefore has to find out about the change from secondary sources. For example OS does not have access to the Local Land and Property Gazetteers (LLPGs) from Local Authorities which list all changes which affect the NLPG. As a result OS maps lag far behind the ‘truth’ on the ground (typically 6-12 months) and there is a substantial waste of surveying effort, as changes are logged and mapped twice (once by the Local Authority and once again by OS).

    b) Maps are Value Added Products. Maps are combination of various low level ‘Elemental Datasets’ superimposed for convenience. Some of these Elemental Datasets are Natural Government Monopolies (NGMs) (eg District Boundaries, SSSIs, street names etc), but others are not (eg coastlines, rivers, height data). Thirty years ago it made sense for maps to be the lowest level of Core Geographical data, but in the modern era of GIS and digital mapping it makes more sense to maintain each Elemental Dataset separately and to allow the private sector or individual users to combine them to form maps. Why? Mainly because the role of government should be minimised to encourage competition and customer choice – this implies that the Government should only look after the Natural Government Monopoly datasets such as Boundaries, SSSIs street names etc. Over the last ten years good progress has been made in opening up the mapping market to genuine competition, and there are now several sources of OS independent mapping including NavTeq, Bartholemews, UK Maps and our own People’s Map. The great downside of giving away map products as free PSI is that it will kill the competition for those maps. It would be much better to give away the Natural Government Monopoly data ONLY and allow the free market to develop all the higher-level mapping products.

    c) The Derived Data Trap. The OS is currently in a dominant (almost monopoly) position for the provision of large scale maps. OS maintains that position partly by claiming IPR on data which a user derives using an OS map as a reference, even if the data does not appear on the OS map in the first place. For instance if a utility draws its pipeline network using an OS map as a reference than OS says that if the utility stops paying its annual licence fee to OS then the utility must delete all its derived pipeline data and recreate it from scratch. This acts as a effective barrier to the creation of a truly competitive market for maps for two reasons. First the main customers (government and utilities) are locked in because if they change map supplier then they face having to re-create their derived data. And second because the independent map suppliers cannot get hold of any derived data without paying OS a fee and negotiating licence terms. Boundaries are a good example of this. The Boundary Commission cannot supply its boundaries to the private mapping sector because they are drawn on top of OS Maps and are therefore considered to contain OS copyright.

    Because of all these issues, Getmapping’s view on the OS Consultation is as follows:

    a) All Natural Government Monopoly (NGM) data should be made available free of charge to the market as PSI, but without any non-NGM data included

    b) No maps which contain non-NGM data should be made available free of charge as this will severely damage competition, customer choice and quality in the market

    c) In the medium term responsibility for maintaining NGM data should be transferred from OS to the natural owners of the data (eg Local Government for street naming and numbering) and the maintenance of the data should become an integral part of the change process, funded by the organisation which causes the change. For example, if somebody extends their house, then the builder should commission an independent surveyor to make a detailed survey of the as-built change to a specified format and accuracy. This survey would be paid for by the owner of the building and it would be used as the formal NGM update record. An intelligent properly-connected process of this kind would allow maps to be kept up to date within hours of a change, not within 6-12 months

    d) It follows that in the medium term OS’s role should become a QA and government-procurement role. This is because if NGM data is maintained by the ‘owning’ government department and made available as PSI then the private mapping sector will be able to make high quality maps to suit all market demands. I believe that the quality, choice and prices of maps would all improve as a result.

    Your comments are welcome. The “derived data” is especially interesting, particularly as it applies to utilities. We’d love to hear from utilities or other customers whether this is indeed the implication of the contract – because we suspect it is, given previous form on the derived data question.

      The following posts may also be related (the database thinks...):
    • None

    And while we’re talking about Land Registry.. here comes UKMap

    March 5th, 2010

    Verrry interesting post over at the UKMapping blog. We’ll include the full content because it’s short and very relevant.

    Great news was received today at UKMap HQ. After much review and testing Land Registry have confirmed [and are happy for us to tell people] that they are happy to accept registrations based on UKMap.

    Great news for all those consultants, government types and property professional using UKMap.

    Full text below

    “Following a review of UKMap, Land Registry is able to confirm that UKMap meets Land Registry’s requirements as a mapping base on which registration applications can be made and is comfortable accepting registration applications based on UKMap. Such registration applications must still follow relevant guidance as set out in Public Guide 40.”

    What does this mean? That LR doesn’t necessarily have to rely on Ordnance Survey maps for registrations. That, too, pours sand into any engines that might be getting started looking to fund OS through higher LR transaction fees. Savvy users would just go to UKMap.

    And an earlier post on that same blog includes these interesting new clients:

    • London Fire, serious users with a serious need to get better mapping. Now they have that with UKMap.
    • London Borough of Islington – managing their green environment, UKMap’s trees and Land Use makes all the difference for them.
    • Mott Macdonald – detailed city centre mapping, all in glorious 3D.
    • Promap – the UK’s leading mapping portal for the Land and Property market have announced they are taking UKMap

    London councils and emergency services? That’s what I think you call an inroad into OS’s market, isn’t it?

      The following posts may also be related (the database thinks...):
    • None