Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Archive for September, 2009

Time for local government to think harder about opening its data

Wednesday, September 30th, 2009

Chris Taggart gave a presentation earlier this month to APPSI – the Advisory Panel on Public Sector Information – about opening up local government data.

Even without the actual talk (is it online anywhere in some form?), the slides make compelling reading. Local government, of course, can sometimes be just as bad as central government (or indeed trading funds) about hanging grimly on to its data, enforcing dubious or unnecessary copyright, and basically making peoples’ lives hard when it should be making it easier.

You can also read my thoughts on how local government could open itself up in an article for Society Guardian here, which has attracted some useful comments – and links to interesting sites.

But now, here’s the lecture. Flash required, of course.

Do you know where your postboxes are?

Tuesday, September 15th, 2009

As an example of how getting data out there can just be plain useful, let’s return to one of the winners of the Show Us A Better Way competition (remember that?).

Prizewinner: postbox locations.

Obstacle: Royal Mail wouldn’t release the data of the location of its 116,000 postboxes.

Solution: Freedom of Information request.

Obstacle: incomplete geographic information in the response (a postcode, not long/lat, plus a mystical Royal Mail reference per box); no collection times.

Solution: FOI request for the collection times and a bit of data marriage.

Obstacle: still don’t know where the postboxes actually are.

Solution: crowdsource it! Get people to pinpoint the locations of what they think are the postboxes onto an OpenStreetMap map. So far about 26,000 have been done – have you done the ones near you?

Obstacle: Royal Mail says it still holds all the rights to the locations of the postboxes.

Solution: actually, you don’t really need a solution. Toothpaste is notoriously hard to put back into the tube.

And as Matthew Somerville pointed out to us, knowing the locations of the postboxes means that one might be able to do “travelling salesman” analyses on the routes – which could have huge potential savings for the Royal Mail. How much does it spend on fuel and time doing collections every day? How much might it save with a proper analysis? Who knows? We won’t until we see all the postboxes put in their place.

And that’s why it’s better to rely on making government data available – free, in both senses of the word – than to try to create artificial “value” from it by charging.

Price does two things: it implies that what you are pricing has value; and it puts a barrier between the thing being “sold” and its potential users. If the users don’t want it enough, they won’t ever go across the barrier. If you take down the barrier, then you get every user you could ever get. And some of them will do really useful things with your product – that’s possible if it’s data.

Sounds like a good idea: Sir Tim Berners-Lee goes to Downing Street to talk open data

Tuesday, September 15th, 2009

Well, Sir Tim Berners-Lee (he invented the web, you know) seems to be getting stuck in. He has gone to Downing Street along with Nigel Shadbolt (whose name always reminds of a Harry Potter character – apologies: he’s actually professor of artificial intelligence at the University of Southampton) to talk to Gordon Brown.

About what?

Mr Berners-Lee and Mr Shadbolt presented an update to Cabinet on their work advising the Government on how to make data more accessible to the public.

Gordon Brown has already spoken publicly about his aim of making the UK a world leader in opening up government information on the internet, an important element of Building Britain’s Future.

He could have asked us. We’d have told him back in 2006. Or 2007. Or 2008.

Sir Tim Berners-Lee told Cabinet about the goal of delivering a single online access point to Government information, similar to the one introduced by the Obama administration in the US.

Don’t we sort of have that already through the work of OPSI and its data portal? Sometimes it seems like the work of Carol Tullo and John Sheridan et al has just been swept down a plughole – or perhaps memory hole, a la 1984.

He also spoke about proposals to extend the “open data” approach, ensuring greater transparency in government and improving the efficiency of public services.

It would be interesting if the “efficiency of public services” meant “to stop different bits of government squabbling over the data they collect like children in a playground and instead start to share it freely, rather as we adults advise children to do so they can discover the benefits of sharing”.

But there’s a suspicion it’s really code for “cut public services while saying what’s being cut will be replaced by something else at some time in the future”.

The Government hopes the data project will benefit the UK by creating jobs, driving new economic growth and allowing the re-use of government data to encourage the development of new, innovative information-based businesses and services.

Hold on just a moment there. The government hopes all these things, does it? Is that because it’s taking the Cambridge study seriously, and looking at its potential benefits to the economy? So we’re not going to see terrible approximations like the OS’s “hybrid” strategy, then?

It is also expected to help increase the transparency of government and empower citizens to get more out of public service by tailoring it to their needs.

What I don’t like here is the description of it as a “data project” as though it were something that sat apart from what should actually be a process – and a core process at that. It shouldn’t be “what part of this data shall we release” but “is there any of this that shouldn’t be released?”

After the update from Sir Tim and Professor Shadbolt, The Prime Minister confirmed his full support for the next phase of their work.

It would be nice to know what that next phase included. Anyone seen a copy of the timetable?

Transport for London really doesn’t like Ordnance Survey: response to consultation

Friday, September 11th, 2009

Good grief, the people at Transport for London (TfL) really aren’t happy with Ordnance Survey, or with its consultation or plans.

Thanks to the efforts of Christopher Osborne – see the thread where he gets OS to release this, the response from TfL to the OS’s “strategy consultation” via Freedom of Information requests at, and marvel at OS’s ability to try to keep stuff secret – we can now see that TfL really wants to do more with map data, but feels that OS is standing in the way.

We wonder how this is being received in Southampton.

We’ve taken the PDF and rendered it as a table, but you can download it here or view it as (possibly better formatted) HTML here..

I’ve added emphasis to some of the tastier responses..

Response from Transport for London to the OS Business Strategy Consultation

TFL: We are generally disappointed in the scope of this consultation and regret that it is not more wide ranging TfL has major concerns with the issues of Derived Data which over-ride many of the items of this consultation We do not think that OS has paid sufficient attention to its many Government customers and their Public Task We regret the enormous cost in both time and resources that Government bodies have to spend on procurring OS data.

OS: Goal 1: Promote innovation for economic benefit and social engagement TFL response
OS: Supporting Government’s objectives to make data more accessible and to encourage innovation by individuals and commercial companies, Ordnance Survey will promote the innovative use of geographic information and its potential application. TFL: OS appears to wish to promote innovation by launching such services in house, experience suggests that this is not the best way and that they should concentrate on making their data freely available and allow the market to innovate.
OS: Ordnance Survey will provide support to all of its users in their use of geographic information so that they in turn can support their customers and the wider public. TFL: We question whether OS is prepared to listen to their users to determine what support is needed, and fear that it imagines that one size will fit all.
OS: Key components of this goal are: TFL: Many Govt bodies could contribute to this Goal if they were not inhibited by the onerous and restrictive licence conditions imposed on their use of derived data
OS: An extended OS OpenSpace service
OS: This will provide additional data and usage rights to support the creation of any new publicly accessible application. It will provide greater access to free use of a number of Ordnance Survey products from 1:10,000 scale through to 1:1 million scale. It will also include official boundaries information. TFL: Provision of OS data on OpenSpace does not make it freely available. Users require access to boundaries in all formats and applications. The derived data issue is a continuing problem, even on the OpenSpace website gallery, demonstration users warn users not to create derived data on the site. There seems to be considerable interest in the BoundaryLine data from other respondents to this consultation. BoundaryLine has been frequently quoted as the “definitive” boundary dataset for UK, why then are the boundaries not coincident with the Boundaries shown on the OS MasterMap product.
OS: This service will benefit individual developers and organisations such as commercial companies, local community groups, national special interest groups and smaller charities that will be able to develop applications as long as there is no direct commercial gain from the specific application itself. Advertising and sponsorship alongside the application will be encouraged. TFL: In the OpenSpace FAQs it is stated that Govt bodies may use the API but it doesn’t seem to be encouraged, given the OS advice on the use Google maps by OS licensees we would suggest that this should be a priority.
OS: There will be limits within the system to ensure that the new OS OpenSpace service has a minimal impact on existing commercial users of the data but these limits will be applied in a more collaborative way. TFL: Govt bodies interested in using the OpenSpace Service would need greater assurance on the use of limits
OS: An upgrade path from innovation through to commercial services
OS: Through the creation of an innovation ladder Ordnance Survey will provide a clear path for those that wish to progress their ideas from early-stage right through to commercial development. The existing developer programme will be opened up to a wider developer community and financial entry barriers will be significantly reduced. TFL: Can the OS appoint an independent body (AGI?) to create the “innovation ladder” so that the disincentives that are built into the current usage of OS data are not carried forward? Can the OS then use the lessons learnt to overhaul their existing licensing terms and conditions?
OS: Removing minimum royalty charges for partner licenses
OS: For organisations licensing data as value-added resellers new or existing – there will be NO minimum royalty for the first year and a minimum royalty of only 1,000 per year thereafter. Where Ordnance Survey is providing a national dataset of its most valuable data (OS MasterMap Topography Layer) the minimum royalty of 1,000 will apply from year one in order to cover the additional costs of supplying that data. TFL: The partnership licenses appear to be wholly geared towards the commercial sector and seem to be completely inappropriate for the use by Govt bodies which is sometimes required. They certainly do not feel like partnership agreements as all the terms are in favour of the OS.
OS: Develop the Ordnance Survey Innovation Network
OS: Ordnance Survey wishes to create a friendly on-line community of commercial and non-commercial developers, partners and resellers who can share ideas about innovative uses of Ordnance Survey data and other data. The network will help individuals as well as small and large organisations to develop ideas, deliver projects and even commercialise their ideas, through making connections with other network members who can help them with the development of technology, services, products or seed financial funding. TFL: Can the OS look at the lessons that could be learnt from previous projects such as CODES, the Collection of Data from External Sources? In this project external bodies were requested to provide OS with data that could be incorporated into OS products. Data was supplied free of charge, OS required assurances and warranties that the data was supplied free of any external IPR etc, and that they were granted a non exclusive royalty free, irrevocable, licence to copy, sell, adapt, distribute etc etc. OS then went on to try to sell the data back to the organisations that supplied it. The whole experience did not seem very friendly and took a very long time to bear any fruit.
OS: Goal 2 Increase the use of Ordnance Survey data
OS: It is recognised that the framework under which Ordnance Survey prices and licenses its data and services has become complex and unwieldy. Working with Shareholder Executive and OPSI, this is being comprehensively reviewed as part of the drive to improve focus on the customer. TFL: TfL would be slightly reassured that this review by the Shareholder Executive and OPSI would lead to a desirable outcome if the OS included its customers in this review
[CA: woooah!]
OS: The overall aim of the work is to make data more easily available and encourage innovation and competition in the market. TFL: The OS has frequently in the past used the ‘level Playing Field’ argument to inhibit any negotiation on their terms and conditions of contract. Clear, concise and transparent terms and conditions would assist here. Innovation has been stifled in the past by OS insisting that any use of there data must fit into one of their existing contracts.
OS: Under the new framework, Ordnance Survey will continue to license its data both indirectly and directly to end customers. Partners will continue to have appropriate licences so that the price they pay reflects the market value of their end product. However, Ordnance Survey wishes to re-examine the boundary between the different licensing routes to check they best support the current market and rationalise the suite of licences available to partners. TFL: Many users of OS data, and derived data wish to make their information available free of charge. If OS wish to licence their data so that the price paid by the user reflects the market value of the end product, does that mean that the OS data will also be free of charge?
OS: In particular, Ordnance Survey expects over time to offer many more products to commercial partners so that they can distribute them to end customers in parallel to Ordnance Survey itself. It is also examining how to give greater flexibility over the rights customers have to use data internally in their own business and better facilitate the use of data within the public sector that contains shared public sector intellectual property. TFL: We can see no advantage in the OS competing with the private sector for the distribution of mapping products, because on a level playing field they will lose, and if it is not level everyone will cry foul. OS should concentrate on the collection of data and leave the distribution to the private sector. In the public sector there should be total freedom for Govt bodies to define and complete their public role.
OS: Ordnance Survey is also reviewing all of its licence documents with the aim of making them shorter and easier to understand. TFL: The OS should appoint an independent body to review their licence documents and pay them based on the number of words deleted.
OS: The pricing structure that accompanies the new licensing framework will be transparent and reflect the work that is being undertaken to reduce costs across the organisation.
Ordnance Survey recognises that there is a desire for rapid change to the current system. However, it wants to ensure that it takes decisions about its pricing and licensing model with a thorough understanding of the impact it will have on customers, Ordnance Survey and others in the market. For that reason, comments are very welcome here on how the licensing framework should be reformed as part of the new strategy.
TFL: One of the major problems for users is the derived data issue which both restricts what the user can do with the derived data but also requires that the OS data and the derived data are deleted at the end of the licence term. One way to overcome this would be to make licences perpetual, so that having “purchased” the OS data the user can keep it, and any derived data for ever. OS needs to differentiate between a commercially derived product such as Geographers A-Z map and a Govt body’s derived data which may consist of discrete points lines or polygons describing their area of interest. It must be recognised that the current rules on derived data make OS map data legally unusable for many Govt bodies and they have to resort to increasingly complex and costly methods to avoid creating an OS derived data issue in completing their public task.
OS: It is expected that work on a new pricing and licensing framework and the plan for its rollout will be complete by October 2009. Ordnance Survey has asked the Office of Public Sector Information, together with Shareholder Executive, to be involved in the work from today to ensure that its thinking is independently challenged.
OS: GOAL 3: Support the sharing of information across the whole of the public sector
OS: Accurate and high-quality geographic information is playing an increasingly important role in the delivery of better public services and ensuring the Government is held to account by citizens. The public sector at both a national and a local level is an important part of Ordnance Survey’s customer base and it is committed to ensuring they receive data and services that best meet their requirements. TFL: The Public Sector would be best served by open competition in the supply of mapping data services, but all previous procurement have been skewed by the near monopoly situation of OS. The Public Sector has to go through lengthy and costly procurement processes to access their data.
OS: In particular, it is recognised that public sector bodies need to be able to share data including data with shared intellectual property (IP) with each other in support of the development of policy and its implementation at national and local level. TFL: The Public Sector also need to share data with the public, who ultimately pay for all this, and with the commercial sector, who will also pay if we can get through the minefield of licensing.
OS: The Pan-Government and One Scotland agreements for the provision of geographic information to central Government in England and Wales and central and local Government in Scotland came into force on 1st April 2009. Ordnance Survey is delighted to be part of both of these agreements that will better enable the delivery of public services.
Ordnance Survey recognises that it will face increasing competition from commercial rivals to deliver geographical information services to the public sector and others. Ordnance Survey welcomes this and aims to continually improve its offering to provide the best value-for-money for all its customers.
TFL: By competing with commercial rivals to deliver GI products, OS can no longer maintain its UK definitive role. For example in the Address field there are now two competing products both effectively paid for by the public purse and neither definitive, such that for the 2011 census additional public funds will be spent harmonising and incorporating other data to produce a comprehensive address list, which is a plain waste of public money and resources. Surely a better strategy would be for one public body, such as the OS, to collect GI data, using the commercial sector as appropriate. The data should then be made freely available to the Public Sector and marketed through commercial partners who create GI products.
OS: Following the success of bringing the public sector, including the NHS and Emergency Services, into a single agreement in Scotland (One Scotland Agreement) Ordnance Survey is now in discussion with central and local Government in England and Wales to determine how the new Ordnance Survey business strategy might support a similar goal. This would provide a common platform for use in delivering core public services using geographic information from Ordnance Survey and its competitors. TFL: The Ordnance Survey always insists that their agreements are “commercial confidential to the parties”, therefore other Govt bodies are unable to compare the success or benefits of one form of agreement with another. Greater openness and transparency would assist here.
OS: As part of this work, Ordnance Survey will also be discussing with Local Authorities, Land Registry, Royal Mail and others how it can align its methods of data capture and provision to ensure efficiency for the public purse.
OS: Goal 5 Enhance value through the creation of an innovative trading entity
OS: The geographic information market is undergoing huge change as a result of advances in technology. There is less and less distinction between viewing data on a computer, via the internet or on a mobile phone. Data collection methods are also changing radically, with technology driving costs down and users increasingly making their own contributions back to the original source data. TFL: Data collection methods are changing rapidly and driving costs down, but we question whether they are driving standards up. OS maps of London 100 plus years ago show street furniture, drainage gullies, lamp post, the internal building layouts of public buildings, all features long gone from their latest offerings. While the overall accuracy may have improved, the quality provided is actually quite low and worse than previously claimed.
OS: If it is to thrive in this market and continue to fulfil its core public duties, Ordnance Survey must adapt to this, just as it did 30 years ago in the move from paper to digitally-based information.
It is believed that there are greater opportunities for the Ordnance Survey brand particularly in the consumer marketplace. To explore and grow these opportunities in a transparent and timely manner, Ordnance Survey will establish a wholly-owned subsidiary company. This subsidiary will further Ordnance Survey’s ability to offer new and innovative services to government, business and individual customers alike.
The new company will initially explore the development of an enhanced brand presence through an extended consumer leisure portal. Research has strongly suggested that a richer website operating under the Ordnance Survey Company name and brand, will attract a far larger proportion of leisure enthusiasts. It is envisaged that this company will explore the opportunities afforded by the use of the Ordnance Survey brand for affiliate merchandising.
The new subsidiary is part of the drive to ensure that Ordnance Survey is sustainable for the medium-term and value is generated for the taxpayer. It will operate on the same terms in relation to Ordnance Survey data as any commercial rival.
TFL: As a large user of OS mapping in the public sector TfL suffers significant restrictions on its use of OS data because it is not licensed to engage in activities that are regarded as “competing activities” with OS’s commercial rivals. We are concerned as to how this will develop if OS is going to enter the market with this new trading company. There are a series of complex competition issues which arise from monopolistic or dominant undertakings seeking to enter a downstream market on a commercial basis. TfL is extremely concerned by this prospect and questions whether effective arms’ length arrangements will really be put in place. At a wider level, we also question whether this is really the most effective strategy for deriving overall public benefit.

International man/woman of mystery: the silent, uncommunicative type, apparently

Thursday, September 3rd, 2009

The tale of the International Man/Woman of Mystery – who “approved” the Ordnance Survey’s international comparisons study (which personally I thought was woeful) – just gets more and more incredible, or less and less credible.

Two more FOI responses came back today. You’ll recall that we previously established that the IM/WOM works, or has worked, for a foreign mapping organisation, and is a foreign national.

So we asked: Please could you provide the following information, none of which identifies the person:

1) which NMA employed or employs them, and is the employment ongoing?

2) if the employment is no longer ongoing, when did it end?

3) Is the person male or female?

To which the response came back:

I can confirm that Ordnance Survey does hold this information, however I regret to inform you that your request falls within the Personal information exemption under section 40 (2) (a) of the Freedom of Information Act 2000 (FOIA). We believe this exemption applies because in our reasonable opinion the provision of the information requested would prejudice the anonymity of the individual and would serve to substantively assist in identifying them and remove their anonymity. As has been pointed out previously the individual does not wish their identity to be disclosed, therefore we will not be releasing this information to you. With regard to this matter, we will also take the same approach with any further requests that, in our opinion, prejudice the anonymity of, or substantively assist in identifying the individual concerned.

Possibly to be expected, but any one of the answers would arguably not have identified the person, and have been useful.

And so we tried a different tack. There must have been some discussion with this person setting up their examination of the study, right? So we asked for internal emails and other communication with them. (It would be redacted, of course…)

Back comes the reply:

Thank you for your email dated 5 August 2009 requesting: copies of all OS emails, letters and notes from any telephone conversations to or from the internationally recognised expert that were made in relation to their review of this study (for example approaching them to carry out the review, sending them the review, receiving their comments on the review).

I regret to inform you that Ordnance Survey is unable to help you with your enquiry as we do not hold this information. A copy of the report was provided to the person concerned and engagement on this matter was conducted orally with no permanent record made of these conversations. [Emphasis added – CA]

So basically they called a Good Friend up and asked them to cast an eye over it. No emails to or from? No phone conversations or letters? No written or emailed comments?

Is it just me, or is that incredible for a study that’s being prepared for the minister in order to consider the best method of funding the national mapping agency of Great Britain?

There are some other areas that we can probe. But this is just amazing. A study best described as sloppy, and a review that can only be described as secretive. And this is the new model for Ordnance Survey?