Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Archive for the 'Government reports' Category

In the Guardian: what happens to the Postcode Address File in a Royal Mail split ownership?

Monday, March 9th, 2009

With Lord Vold… Mandelson looking to persuade a private partner, likely TNT, to take a minority shareholding in the Royal Mail, the interesting question arises of what happens to the ownership of the intellectual property of the Postcode Address File (PAF).

After all, if you buy into a company, you’d probably want to see more efficient use of its assets. (That’s part of the plan in the shareholding selloff.) Would that mean that TNT or similar would start trying to “sweat the assets” of the PAF?

In What does the Royal Mail sell-off mean for postcodes data we investigate this briefly. The problem is that nobody – including the Turner Report into the future of the Royal Mail – seems to have considered this. Neither PAF nor intellectual property nor postcodes are mentioned at all in the Turner report.

PAF is profitable –

in August 2007 the postal regulator Postcomm revealed that PAF operations made a profit of £1.58m on revenues of £18.36m, all but £4m from resellers.

However marketing organisations, which use PAF, don’t like the idea of those assets being sweated.

“The reason for getting the private sector involved [in the Royal Mail] is to improve efficiency,” said Robert Keitch, director of media channel development at the Direct Marketing Association. Raising PAF prices would make it harder to check addresses and increase the need for manual checks by postal staff, he suggested.

Our opinion?

The Free Our Data campaign has consistently suggested that the PAF – linked to map data – should be made available for free, without copyright restrictions, due to its growing importance for location-based services. The comparatively small cost of running it, especially without the costs of administering its sales and checking for violations of licences, could perhaps be borne through a levy on address or name changes, or simply through the tax revenues that could be gained from new companies set up to take advantage of the datasets. However, it is unknown whether Mandelson will recommend that.

We await developments.

Tories demand publication of Trading Funds review, back free data principles

Thursday, February 19th, 2009

photo: Paul Downey

The Conservative Party is demanding that the government publishes the Trading Funds Review, while also giving its backing to the concept of free data – although it hasn’t quite gone as far as backing the entire principles of the Free Our Data campaign.

At a debate last week hosted by the Policy Exchange (a broadly right-wing thinktank), Charles Arthur – that’s me – Steven Feldman, Ed Parsons, Shane O’Neill and – crucially – the shadow skills minister Adam Afriyie debated the idea of free data and what it could mean for government.

Perhaps one problem was that there wasn’t anyone to represent the other side of the debate. I had thought that Steven Feldman would oppose the idea – but he didn’t, and doesn’t.

But the meat of the evening was in Afriyie’s speech. Now, the Free Our Data campaign isn’t party political; the more sides back it, the more likely we are to get change.

So, to Afriyie’s speech, which he presented pretty much as it’s on the site:

The campaign to ‘free our data’ is an important one – all the more so at a time when our economy is in deep recession.

And further:

So my vision is for a more open, innovative and better connected society: a society where access to communication technology creates more powerful citizens and a less controlling state; a society where the free-flow of public information energizes entrepreneurs and social innovators.

Data will be the fuel of this new economy. And as the repository of the country’s public data, the Government already has a vital presence in the field.

A key point:

First, I turn to non-trading fund data. This is the deep well of raw data which lies unpublished, unavailable and untapped in dusty government vaults.

This includes raw data for crime maps and school league tables. It could include hospital performance statistics and patterns of carbon emissions. It might include the coordinates of every mobile phone mast, held by Ofcom, or the location of accident black spots, held by the Department for Transport.

One little point: the mobile phone mast data doesn’t actually belong to the government. It really belongs to the mobile phone companies, but I think that the government has an interest in holding it. This was one of the topics that OnOneMap brought to the fore.

Afriyie continued:

There is any number of data sets below the radar, and I’m sure members of the audience will have their own ideas. Much of it can be mashed-up and re-used, adding value for the economy and society.

And now to the crunch stuff.

Second, I turn to trading fund data. This is information which government agencies sell to generate income. The Ordnance Survey, the Met Office, the Hydrographic Office and the Land Registry are good examples.

Now let me be clear: we see nothing intrinsically wrong with asking users to pay for a service. Nor do we oppose attempts to replicate efficient, business forces within the public sector.

But we are instinctively cautious of government monopolies. And that puts us in a position to ask whether there could be a better way to generate more jobs, more wealth and more enterprise for the nation by doing things differently.

Take Ordnance Survey as an example. While there is a case for the government having responsibility for mapping the nation’s territory, with growing opportunities in the digital economy it’s not so obvious that the government should also run map shops. Conservatives do not usually take the view that retail businesses are best run by governments.

That’s an interesting point. It’s indicative of the importance and breadth of what the trading funds do that they are still owned and run by the government.

Afriyie notes that there has been plenty of studies about the benefits of free data models, including the Cambridge study and more recently the Trading Funds Assessment, carried out last summer and autumn – just as markets and banks properly went into meltdown.

But so far there have been at least five government reports. And some – like the long delayed Trading Fund Assessment – have been announced with great fanfare only to fade with a whimper.

Indeed we are now told, bizarrely, that the Trading Fund Assessment will not be published and will be treated as private advice to Ministers. There’s open government for you! But we do welcome to the Trading Fund Assessment. The situation should certainly be scrutinised. But the review should not be an excuse for kicking the issue into the long grass.

So I’m calling on the Government to publish the review, and let us have the debate.And let’s be clear: it should not be an excuse for half-hearted measures.

It is of course important to read the whole speech (this is only a few extracts from it) to get the full idea of what the Conservatives are – and aren’t – saying here. It’s a nudge in the direction of a free data model, and Afriyie’s reaction on the night seemed to indicate that he thinks it’s a good idea. But it’s a long way between the idea and the implementation. We’re not done yet.

(Paul Downey has a good summing up of the event too. It’s his picture at the top.)

Power of Information blog suggests what should be freed up; Geographic Strategy seems to agree

Friday, November 28th, 2008

Over at the Power of Information Task Force blog, its chairman Richard Allan has been busy – perhaps following on from the Guardian story about the problems with derived data.

He’s got some trenchant opinions about what should be available to people (and companies?) wanting to work with geographic data for free:

All government administrative boundaries – e.g. constituencies, wards, super output areas, health authorities, school catchments etc.

All point data for the location of public service outlets – e.g. schools, hospitals, public toilets, daycare centres etc.

I can’t think of any good reasons why such data should not be declared as free for re-use in all senses of the word, i.e. that no license fee should be payable but also that no restrictions should be placed on how it is re-used so we stop worrying about Google Maps terms and conditions etc. for this class of data.

The major advantage in doing so is that anyone who wants to experiment with this data, both inside and outside government, is able to get on with innovating without having to worry about legal problems.

We, of course, agree. (We’d go further. But there’s a place and a time.)

There are many interesting comments on the post, though perhaps the most interesting comes from Robert Barr:

Street centre lines, all address points, perhaps even land parcel boundaries (I know, Land Registry not OS) could all be provided on the basis that those who cause the data to change are charged not those who use it. That ensures both a fair charge for the one off service (with some element for maintaining the servers) rather than speculative and counter productive attempts at deriving revenue from data sales.

It works for the Land Registry and the Domain Registration System on the Internet, why not OS?

The interesting point is that the recently-released National Geographic Strategy seems to hint at something similar. On page 21 (and onward) we find this:

Each dataset owner (both Core Reference Geographies and other location-related datasets) should simplify their licensing arrangements so as to facilitate the sharing of data to realise greater overall value. This is in line with the Government’s response to the Power of Information Review recommendations and with the sharing arrangements required for INSPIRE.

Although that’s then ameliorated by the next-but-one paragraph:

The simplification should take account of the trading nature of the owners of the Core Reference Geographies and should not duplicate the Government’s separate review of the pricing of public sector information by trading funds. The simplification should also ensure that Crown Copyright is protected appropriately.

We’ll have to wait to see what emerges. But it seems like an approach which sweeps away the “derived data” idea is winning through.

Ordnance Survey business model “to be considered”; national geographic strategy coming Tuesday

Monday, November 24th, 2008

The Pre-Budget Report isn’t usually required reading for Free Our Data, but this time around it was, partly because expectations had been raised by the Sunday Times story suggesting that OS and other trading funds were in line for privatisation. (I was doubtful. I just don’t think privatisation is in this administration’s DNA. Everything it’s done with banks, after all, goes in completely the opposite direction.)

Here’s the relevant extract from the PBR (1.7MB PDF)

Re-use of public sector information from trading funds

4.54 The HM Treasury/Shareholder Executive assessment of trading funds has considered the potential for innovation and growth from increasing commercial and other use of public sector information. It will shortly publish some key principles for the re-use of this information, consider how these currently apply in each of the trading funds and how they might apply in the future, and the role of the Office of Public Sector Information in ensuring that Government policy is fully reflected in practice. For the Ordnance Survey, this will involve consideration of its underlying business model. Further details will be announced in Budget 2009. [emphasis added]

As Ed Parsons, former chief technology officer at Ordnance Survey and now a map guru at Google, points out:

This is not about privatisation – this is about how the OS trades.. how it charges for data, and its relationship with other departments. This has been on the cards for a while, although I think the issue with derived data no doubt moved this up the agenda a bit!

Yes, the issue of derived data really has stirred things up within Whitehall. And we hear that on Tuesday the national geographic strategy for the UK – often promised, never yet seen – will be published. That might be interesting too.

Update: we’ve been pointed to some more mentions of OS and other trading funds. Seems the Sunday Times wasn’t completely wrong about some being in line for privatisation – but OS still isn’t.

The relevant section is in Box 6.4 (“Operational Efficiency Programme – Asset strand”), on page 119 of the report. My comments in [italics]

Gerry Grimstone is heading the asset strand of the Operational Efficiency Programme (OEP), and will be working with departments, agencies, and the Shareholder Executive to consider, for a number of Government assets, the potential for alternative business models, commercialisation, new market opportunities and, where appropriate, alternatives to public ownership. The work includes:


• a review of British Waterways’ model for managing its canal-side property portfolio which  will assess how best public value might be delivered from these assets in the medium term; [partial or complete privatisation? What does BW do that needs to be in government hands?]


• options for the Queen Elizabeth II conference centre, after a recent study concluded there was no public policy rationale for the Government to own it; [looks like a selloff]


• a strategic review into the future business model of the Ordnance Survey, that will take into account its role as a public sector information provider, together with providing value for money for the taxpayers; [doesn’t look like a selloff because of the “role as a PSI provider”]


• work with the Land Registry to explore ways to improve its operating framework; [again, no selloff; operating frameworks are how you get stuff done, not whether you’re private or not]


• widening the scope of the study of capacity requirements at the Dartford Crossing to include the potential to realise value for the taxpayers and, in addition, continuing to explore options  for the commercialisation of other transport assets; [“realise value” for taxpayers sounds like a selloff]


• a study to explore the potential benefits of alternative future models for the Royal Mint; [unclear]


• reviews of the Met Office, Oil & Pipeline Agency, and Defence Storage & Distribution Agency examining alternative business models. [notable that it doesn’t mention Met Office’s role as a PSI provider, which is rather suspicious]


Budget 2009 will report on progress, and will also take into account market conditions and the views of relevant stakeholders. Gerry Grimstone will work with Lord Carter of Coles, who heads up the OEP Property workstrand, in ensuring that any appropriate efficiencies in relation to property associated with these assets are taken in account.

There’s also some interesting mentions at the end: read in full to see how the market is affecting thoughts of selloffs – or not:

Departments are also working to achieve efficiencies on other Government assets:


• the Ministry of Defence will shortly publish its response to a recent consultation on its plans to release and share parts of its electromagnetic spectrum holdings, with the release process for initial spectrum bands beginning in Spring 2009;


• the Government continues to explore options for realizing value from its stake in Urenco;


• a study of the Forestry Commission’s portfolio in England is being launched to examine options for delivery of public value from the estate in the long term; [=selloff]


• a major redevelopment by Covent Garden Market Authority will aim to put it on a sustainable financial footing, enabling the Government to achieve its long-term objective of disengagement.[=selloff]


In addition, it was concluded in October not to pursue a sale of the Tote in light of current market conditions, and that it should be retained in public ownership for the medium term, to be brought to the market when conditions are likely to deliver value for the taxpayers and the racing industry. [=no selloff because there are no buyers.]

Privatisation, of course, is all very well if you can find a buyer. But in a falling, or stagnant, market, you can’t get the best price. Alastair Darling would be in a weak position if he sold any assets off now only to see the market rise before an election – which would open him to accusations of selling the country short.

Today in the Guardian: ‘free data’ ministers still in place, but face uphill challenge

Thursday, October 9th, 2008

The dust has settled from the ministerial reshuffle of last week, and we’re happy to see that the ministers whose views about access to government data chime with ours – particularly Tom Watson in the Cabinet Office and Michael Wills at the Ministry of Justice – remain in place. Note too that Shriti Vadera, who was at DBERR, now has a higher-profile role which is expected to have quite an impact. (Baroness Vadera, a former City high-flier – remember them? – is understood to hold strong views on, inter alia, making government data more easily accessible.)

In today’s Guardian we note the fact that that hasn’t been shuffled around, and the new challenge that ministers pushing the free data idea face: how do you persuade a government that has just melted down the golden rule in order to quasi-nationalise high street banks at a cost of around £500bn, with what looks like a shrinking economy on the way, that it should forgo hundreds of millions of pounds in tax funding to pay to make data free?

(One answer might be: because it’s cheaper to do that than pay the unemployment benefits and other consequent costs if companies that pay for government data go bust.)

In Free data faces a tough challenge in the new parliamentary season, Michael Cross notes that

Decisions about the future of such trading funds will need to be made soon. A much-delayed government-wide strategy for geographical information is due for publication this autumn. It has spent the past year being bounced between Civil Service desks as its ideas are aligned with Britain’s commitments to open access to environmental data under the EU Inspire Directive. More interesting for the Free Our Data campaign will be the outcome of a review by the government’s Shareholder Executive into the trading fund model.

Oh, yes, that review. Today’s bonus link: Parliamentary Question from the Tories’ shadow minister for innovation and skills, Adam Afriyie: who has the review team met?

The answer:

the Shareholder Executive team has heard the views of around 20-25 stakeholders from the private sector, as well as others from the public and third sectors. The private sector stakeholders have included customers, suppliers and competitors of the trading funds, small UK-based companies, large multinationals and representatives of trade associations and interest groups.


It would not be appropriate to name the organisations individually because their contributions to and comments on the assessment have been to inform advice to Ministers and were on a non-attributable basis.

Interactive crime maps for everyone by Christmas, says Home Office

Monday, July 28th, 2008

Despite the fact that Parliament has risen (so that it’s officially the silly season – hey, was that a UFO flying past?), the Home Office is still busy at it. Today, it’s put out a press release saying that

Every neighbourhood in England and Wales will have access to the latest local crime information through new interactive crime maps, Home Secretary Jacqui Smith announced today.

The rollout of interactive crime maps follows the announcement made by the Home Secretary earlier this month, as part of the Policing Green Paper, that every police force in the country has now delivered monthly crime information to the public on their websites. New interactive crime maps will take the rollout of local crime information to the next level.

By the end of the year every police force area will produce crime maps which will allow the public to:

* see where and when crime has happened, down to street level for some crimes;

* make comparisons with other areas; and

* learn how crime is being tackled by their local neighbourhood policing team.

We think that the last of those is going to be very interesting indeed, since for senior police officers it will (in a nice phrase I heard on a related topic from a civil servant recently) “hold their feet to the fire”. (Strange how one has to summon images of torture when trying to get some public services to change..)

Coincidentally, we’ve had some interesting emails on the topic: one from (which, you’ll find, does some crime mapping).

And another reader wrote in at length:

“About 3 – 4 years ago I worked temporarily in Bury MBC’s Housing Department. There was a man in the Chief Executive’s department who had a GIS containing 3-years-worth of police crime data. He could rustle you up a map of recorded crimes, varying by type and date, for any local area you chose, on request. So it can’t be that difficult to do it.

“In the early 1970’s I was Area Housing Manager at Speke in Liverpool. My office was in the middle of this Council-built area some 6,000 houses and flats and the local police station was just across the street. This was long before we had computers for anything except (batch processed) rent accounting and it was before “defensible space” became an idea in good currency amongst urban designers.

“Following a disturbing interview with a widow with three children whose chronic poverty had been made even worse by being burgled 5 times in 6 months, I enlisted the help of the station sergeant. I gave him a 1:2500 plan of the estate and, at my request, he went through the station’s day book for 6 months past, putting a red felt-tip dot against the address of each recorded burglary.

“He returned the plan to me saying “I’ve done what you asked and it looks like a bad case of measles, but I’m none the wiser.” As soon as I saw the plan I was immediately the wiser. The “measles” were overwhelmingly clustered around particular styles and types of dwellings, and the 3-storey walk-up open-plan flats, where the widow lived, were many times more likely to be burgled than (say) the semi-detached houses.

“I subsequently extracted £30,000-worth of additional fencing from my bosses to enhance security. (Quite a lot in 1974.)

“The point of the story is not that I was cleverer than the police sergeant; I’m sure I wasn’t. The point is that a policeman’s eyes see a residential area one way, and a housing manager sees it another. Who knows what might be achieved if lots of people could see the data and bring their distinctive perceptions and intelligences to its analysis and interpretation?

What indeed? Simon Dickson is a bit dubious about how easy it will be for government to do this; Steven Feldman (who I think we could fairly call a sceptic about Free Our Data – which is fine; an unopposed theory has no strength) has pointed out that postcodes sometimes give more detail away than you’d think (personally, I suspect that domestic violence will be excluded from these visible crime stats).

So we’ll wait to see. By Christmas? Sounds fun.

(Crossposted with the Technology Guardian blog)

And now, OPSI sets up an “unlock that data” channel

Monday, July 7th, 2008

The Office of Public Sector Information (OPSI) goes from strength to strength. After its chief Carol Tullo spoke out in Europe about the importance of greater access to data, OPSI has set up a web page where you can request data sets you want released:

As the regulator for public sector information re-use, we know that people can encounter problems from time to time getting hold of the information they need in the formats they want. Difficulties can include problems with charging, licensing or the data standards that public sector information is provided in.

These problems aren’t about access (which is dealt with under Freedom of Information legislation), but all the other issues which can occur when you want to do something with public sector information – copy it, remix it with other data or add value and republish it. If you are trying to re-use some public sector information, but the data you need is locked-up, this service is for you.

How it works:

  1. You describe the public sector information asset you want unlocked for re-use, and post a request to the service. We’ll check through your request and if it’s OK (e.g. not a Freedom of Information request) we’ll post it here.
  2. Others can see your request and support it, either by adding a comment or by voting. The more support a request has, the better the chances of unlocking the information you want to re-use.
  3. We’ll contact the public sector information holder and see what can be done to unlock the information for re-use. To keep things simple, if the problem relates to an issue specifically covered by the Re-use of Public Sector Information Regulations or the Information Fair Trader Scheme, we’ll treat it accordingly – so you won’t need to make a separate complaint. We’ll post back our findings here.

And there’s already one request in there, for access to OS electoral boundary details, which I recall is an issue that comes up again and again – it was certainly mentioned at the RSA/Free Our Data debate nearly two years ago.

The problem, as detailed by “Matthew”:

I find it odd that if I want to know the actual boundary of the ward or constituency I am in (co-ordinates, not just an image), I have to pay Ordnance Survey lots of money for their Boundary-Line product. I would have thought that, given it’s quite important to know which MP or councillors I’m going to have the option of electing, that this information should be freely available as part of a healthy democracy; it’s compiled by the various publicly funded Boundary Commissions/Committees as far as I know.

His ideal solution:

I think the actual data rather than just images of the boundaries should be available, so that people can create things using the data – you can’t do anything with images besides display them. For example, I can’t create a Google map (using their My Maps feature) of my ward marking on where and when councillors hold their surgeries, and other local amenities. I can’t create an application that asks people to select where they live on a map and it tell them if their Parliamentary constituency will be changing at the next general election, what it’s changing to, and what difference that makes to them.

I am aware of the website, but this is extremely hard to use – you have to know the name of your area before you can enter a postcode, you can’t look up by e.g. ward name, and it only provides images of the boundaries.

More power to his, and OPSI’s, elbow.

This is all terrifically encouraging, especially along with the Show Us A Better Way competition using government data for imaginative (and perhaps commercial) mashups. Have you got your entry in yet?

The government wants you to show it a better way (and will pay £20,000)

Wednesday, July 2nd, 2008

As an idea, Free Our Data has now begun to gain some traction in government – and even, as the whole saga over crime mapping in London shows, with the Conservatives.

Now the Power Of Information taskforce, which includes Tom Watson, the Cabinet Office minister we interviewed a while back, has started a new initiative (though competition is just as good a word) at

Ever been frustrated that you can’t find out something that ought to be easy to find? Ever been baffled by league tables or ‘performance indicators’? Do you think that better use of public information could improve health, education, justice or society at large?

The UK Government wants to hear your ideas for new products that could improve the way public information is communicated. The Power of Information Taskforce is running a competition on the Government’s behalf, and we have a £20,000 prize fund to develop the best ideas to the next level.

To show they are serious, the Government is making available gigabytes of new or previously invisible public information especially for people to use in this competition.

And in case you wondered if it involves puttings CDs from HMRC into envelopes..

Rest assured, this competition does not include personal information about people.

There is a set of examples – such as crime mapping, Fixmystreet, and a pointer to others such as (which “compiles obscure information about subsidies under the Common Agricultural Policy and puts it in one place, to make it much easier to see where farm subsidies are going across Europe.”)

The team signs off with a flourish:

We’re confident that you’ll have more and better ideas than we ever will. You don’t have to have any technical knowledge, nor any money, just a good idea, and 5 minutes spare to enter the competition.

There’s already a list of submitted ideas, which includes a Road Works API, FixMyTransport (“where people with shared public transport problems could come together to get things improved”), Rate My Bus, and others.

Come on, people – tell us your ideas, then go and enter them on the site (or vice versa) and win the funding. It would be fantastic if a Guardian Tech reader could win this.

Update: just to point to some of the resources you can use (among many, many, many): mapping information from the Ordnance Survey, medical information from the NHS, neighbourhood statistics from the Office for National Statistics and a carbon calculator from the Department for Environment, Food and Rural Affairs (Defra). And these are in API form, which means they’re all ready for mashup goodness.

Although not, it seems, the Postcode Address File (though the Edubase file, with school addresses, does include postcodes).

Trading Funds review: terms of reference

Wednesday, June 25th, 2008

Things are starting to happen, through the work of the Power of Information taskforce.

Here’s the latest: the terms of reference for the study demanded by the POI report reviewing the trading fund model have just been published.

The study will be jointly led by Treasury and DBERR; Yvette Cooper and Baroness (Shriti) Vadera are the lead ministers, though the Cabinet Office POI taskforce secretariat will work closely with them. (We’re still waiting on the timescale.)

From the POI Taskforce blog:

the Taskforce will be assisting with this review, particularly looking at the value of the data held by the funds and whether the current business models and licencing arrangements are sustainable.

If you had to set our priorities what areas would you have us look at? Is there any research we should be looking at? Please kick off the discussion in the comments.

From the TORs themselves, part of the detailed work will include

With regards to a priority group of TFs (Ordnance Survey, Met Office, UK Hydrographic Office, Companies House, DVLA, and Land Registry) the exercise will aim to produce a detailed and definitive pricing and access policy for information held/created by TFs and the optimal constitutional structure for the several TFs to maximise benefit to the UK economy whilst maintaining public policy objectives.


changes in the pricing policy and licensing regime around information held by TFs may impact on their trading performance and value. Therefore for those TFs affected, the assessment will
  • provide an assessment of the business model or constitutional reforms needed to meet the Government’s commitment on access to information collected for public purposes by downstream markets; and
  • provide an assessment of the options for pricing the release of this information

We think this is going to be fun.

Cabinet Office Taskforce blogs… and you should read it

Wednesday, June 25th, 2008

Richard Allan and Tom Watson – the former a former Lib Dem MP who is now chair of the Power of Information Taskforce, the latter a Cabinet Office minister who set up the taskforce – have set up a blog for news of the progress that the taskforce is making.

It’s fascinating stuff, and we’ve been remiss in not noting more of it here.

For instance, there’s a proof-of-concept for crime mapping, including a presentation where the slides say things like “Is it safe to park my car here?” “Has crime in my area gone up or down?” and “How can I do something about it” – for as it points out, “Call to action [is] almost completely missing in existing mapping propositions”.

(On crime mapping, The Register is echoing the Times reporting today that the Information Commissioner thinks there are “privacy issues”. We feel sure we’ve heard this before – oh, because we actually looked into it, and found there isn’t a real objection. The comments on the Times piece are interesting, because they indicate that people *want* crime mapping, and think the objections of “the property industry” and the ICO are trivial. However one of the comments on the Register piece is interesting:

I’ve seen this … the Met Version. Working. With real data.

It had nothing to do with BoJo the clown. It was already being worked on for ages before, BoJo just made it a plegde and got lucky in the sense that it was already working before the election


The only thing that concerns me is that a visual representation will encourage house prices in certain areas to go into freefall as the red areas (high crime) are highlighted. Funnily enough anywhere near a train station (Underground or National) seems to have higher ratings than other parts of the boroughs but overall the map has average crime levels.

I suppose we will have to wait and see if this makes the police take less action in the red areas to reduce reported crime or tackle stupid things in the whole of London to raise the baseline average……


Which is a long diversion from the POI blog. Where you’ll find plenty of fascinating information, such as the old model for Parliamentary data:

and the new model such as can be used by theyworkforyou:

Get on and add it to your feeds. This is important stuff. And, even more importantly, it’s government being done transparently: when you can see what people are thinking, it helps you to influence it.

(Update: the comments feed isn’t obvious, but you can get it here er, here (thanks, Simon in the comments.)

Crime mapping coming more widely as government gets on board

Wednesday, June 18th, 2008

New guidance will mean that there will be more crime mapping: a paper published by the Cabinet Office and written by Louise Casey, the government’s crime adviser (the one, you’ll recall, who said that some anti-binge drinking schemes were nonsense) notes, inter (very many) alia, that

Police forces are due to provide standardised local information on crime, starting from Summer 2008, as part of the Government’s new crime strategy3. Some are already providing local information but what will be available from the Summer of 2008 is likely to remain highly variable. We hope police forces will draw on the evidence in this review to develop the information they provide over the next year.

In particular, there is strong public demand for consistency in the content and presentation of information about crime across the country and a strong focus on action. In a survey of members of the public for the review we found that:

• 72% of the public said the format of police websites should be the same across all police forces; and

• 87% wanted to see the same format used by all forces for the information they provide.

Beyond this, we believe there is scope for better presentation of comparative information on crime and the performance of the police and other criminal justice agencies which would be of interest to the public. With advances in mapping technology, there are several examples of crime information available on websites that allow the public to bring up crime information mapped onto a neighbourhood. [Emphasis added – CA]

Mapping and interactive reporting tools are useful and careful consideration should be given to their development and presentation. [Emphasis added – CA] We believe some consideration should also be given to standardising the information they provide on crime, based on best practice, so that consistent types of information are presented to the public in a recognisable and user-friendly format. While the focus of existing sites is local, some consideration should also be given comparisons between areas. An end aim could be to ensure that information is available on a national basis, consistent between areas. This would raise the profile of such information with the public – and a consistent format would make sense to a more mobile population. [Emphasis added – CA]

Jacqui Smith, the secretary of state at the Home Office, responded:

“We plan to publish monthly local crime data and we will take forward the report’s recommendations on local crime mapping and making sure every household receives ‘Crime Watch’ style information about the local fight against crime.

Not sure where that leaves police officers like Brian Paddick who think it’s all too unbearable to countenance the public seeing crime data. But the idea has now been so thoroughly floated, it’ll be next to impossible to simply bury it.

There are of course wrinkles. Stuart Grimshaw, who tipped us off to this announcement, has written a letter to his MP asking what formats the data will be provided in. Which matters, rather a lot. Do we want to scrape PDFs or websites? No, we want a decent XML feed, please. Not hard. Perhaps the police could be issued with geotagging systems – take a photo at the crime, send it with geotags (which can be done with GPS-enabled phones). Aggregate data, remove precision as required by class of crime. Send to web server with RSS output. Job done.

APPSI to examine free data model, it says

Wednesday, June 11th, 2008

The Advisory Panel on Public Sector Information has put out its latest annual report. APPSI, you’ll recall, is now headed by David Rhind, the previous head of the Ordnance Survey (who also testified to the Treasury on the next census). But this report was signed off by Richard Susskind.

Kable has a short story on it:

Outgoing chair of APPSI, Professor Richard Susskind, said: “2007 was a pivotal year for the UK in relation to the re-use of PSI. Above all, we saw a marked increase across central government in the level of debate over the re-use of PSI. In particular, APPSI warmly welcomed the growing interest amongst ministers.”

The annual report itself is more interesting: re the Cambridge report, it says

APPSI can already confirm, however, that we welcome the tone and rigour of the Cambridge study – it is the kind of detailed and systematic economic analysis of trading funds and PSI re-use that we have been recommending since 2003; and we hope this represents the beginning of a new era of open and sophisticated thinking about the economics of PSI.

That’s encouraging. More transparency helps. And as for pressure upwards:

We intend, more frequently than we have in the past, to provide practical briefings to our Minister at the Ministry of Justice. These will cover key issues such as evidence, statistics and data relating to the impact of PSI; the governance of PSI and principles underpinning its re-use; the enforcement of the PSI Regulations; models and case studies clarifying the economics of PSI; the findings of ongoing horizon scanning by APPSI; and the adequacy and scope of information management activities across the public sector.

Basically, much more focus on on the economics of PSI. It also says it will “follow up” on progress from the recommendations of the reports into PSI such as The Power Of Information and the Cambridge study. And to that end…

To stimulate and widen debate about the future exploitation of PSI, we will conduct an initial inquiry into the implications of introducing a regime under which public bodies would be subject to some kind of obligation to make their PSI available for re-use. There is no such obligation today under the PSI Regulations.

Which is telling, isn’t it?

In the Guardian: how rows over intellectual property are causing problems for the 2011 census

Friday, June 6th, 2008

2001 Census: Royal Mail worker sorts envelopes
A Royal Mail worker sorts envelopes in the 2001 census. Copyright: The Guardian.

In this week’s Guardian Technology we look at the 2011 census – and beyond: the Commons Treasury Select Committee produced a report earlier this year pointing out that the £500m 2011 census is being hampered by rows between the Ordnance Survey, Post Office and local authorities about who owns the intellectual property in addresses, as the Office for National Statistics needs a comprehensive, accurate national address register to carry out its work.

The article, Traditional census is ‘obsolete’ also looks at what could follow: a rolling census carried out by noting peoples’ movements through address registers and so on.

An extract:

Angela Eagle, exchequer secretary to the Treasury, was pressed by Mark Todd, the Labour MP for South Derbyshire who sits on the committee, over the failure to create the register – particularly because, as he repeatedly pointed out, all the intellectual property lies within the public sector. Eagle responded that: “I would not underestimate the difficulty of the issues surrounding [a single national address register].” Todd suggested that a Gordian knot approach – cutting through the complexity at a stroke of legislation – might work. “We can all hope,” Eagle responded. But in the meantime, the government’s Office for National Statistics (ONS) will prepare its own register – a needless duplication.

The Free Our Data campaign would certainly back a Gordian approach: slice through the 10 years of rows by making the data available for free, without copyright restrictions, where they are gathered by government-owned organisations.

While Todd is no fan of the free data model, we can all agree with his frustration at the way that rows about data owned by public-sector organisations are holding back the development of another public-sector resource, an accurate census. The report recommends that the government “remove any outstanding obstacles to the production of an address register”.

But what are those obstacles? Only they know. Todd said: “I don’t know whether it’s the trading fund status of Ordnance Survey, legalistic barriers or failure of the will by government.”

You can find the Treasury select committee report here.

Why the police are against crime mapping – and what it tells us

Friday, May 30th, 2008

The BBC’s Today programme reported this morning that the Information Commissioner has, at least temporarily, held up the plan by the new London mayor Boris Johnson to introduce crime mapping to London. Crime maps, you’ll recall, were one of Johnson’s manifesto pledges. (Note: I can’t find anything to this effect on the ICO news page – anyone got a link?)

Apart from wondering why the manifesto pledge wasn’t taken for a quick spin by Johnson and the Tories past those who might have an interest (such as the Information Commissioner), the justification given afterwards by Brian Paddick, a former deputy assistant commissioner at the London Metropolitan Police (and the Liberal Democrat candidate for the mayor’s job), was illuminating. Why? Because it showed how deep the thinking that “we know better than you” runs inside British administration.

(You can hear the clip here; requires Real Player.)

Paddick said that the police already use crime mapping internally, and that from time to time the police chiefs for various areas would be called together to account for why they weren’t doing so well.

And how effective were those meetings in getting anything done? “It depended who was chairing them,” Paddick said. Good chair = police chief shamed, or driven, to do something. Bad chair = not much happens.

So why was he against it? “There are signif downsides to that [public crime mapping] process, potential to stigmatise areas, to create ghettos… and to make underreporting ofcrime even worse – people not reporting.”

The following is a rough, first-pass transcript.

Q: But it’s just telling people where crime happens.
Paddick: But we should be focusing on wht will improve efecti of police. The New York success of accountability didn’t come through making the crime data available to the public, but to Comstat, where police commanders were compared to their peers in open forum. We have tried that in London wher have half dozen commanders – asked those who do better what the secret is so they can tell the others…. The police already use crime mapping data themselves, using it in a sophisticated way. The only difference between what happens and what Boris Johnson is suggesting is that of making it public.

Q So the police already have mapping street by street to decide where to deploy resources – all Boris Johnson is suggesting is to make it public.
Paddick: yes… there are systems to hold police commanders accountable though meeetings. Making crime maps available down to street level is a lot of pain for very little gain.

Q But if we do what New York did, why might we not get better results?
Paddick: It has to be said that Comstat process – that is, holding local police commanders to account in one room and account for why crime had gone up or down – whether or not that worked depended who was chairing the meeting. It’s not a very British thing to hold people to account in front of their peers. It had mixed results depending on how the chair held them to account. (Emphasis added – CA.)

Q I thought it was about using it in clever ways, overlaying demographics, the location of porn pawn [thanks Stuart in comments] shops, to lead to more effective deployment of police?
Paddick: …. yes there is potential to use crime mapping more effectively to deploy resources, but the basic principle is available to the police and they are using it at present.

Well, wouldn’t the court of public opinion possibly be quite a good chair for the biggest possible meeting? Accountability is an uncomfortable process, yes. We see again and again that organisations dislike it. MPs don’t like having their expense claims made public – and when they are, they fight against them. In the same way, the police probably don’t like the idea that their full-time (one hopes) efforts to stop and solve crimes might be overlooked by the people to whom it’s actually happening.

But the fact is that this is a process that is happening in society, and any organisation that tries to ignore it does so at its peril because it loses the trust of the public. Journalism (which is a topic I know about) is going through this process: readers can cross-check what we write, point out where we make errors, bring new information we didn’t have. The principle that I try to work on with blog posts like this one is that when I write it, I know more than the average reader. But that implies that half the readers know more about the topic already – and if they can be encouraged to pitch in (as, happily, people do here) then we all benefit, because we have even more information.

The police may not like the idea that their work is made visible, and some people may feel uncomfortable with the idea of knowing about crimes being reported in their vicinity. (Then again, house prices are falling fast enough anyway. Crime maps won’t make a difference.) But the idea that the police are smarter about crime and criminal patterns than all of the population who might be looking at a crime map is, frankly, insulting. The average police officer is probably much smarter about crime than most of us. But it’s that group of the wider population that can find patterns that they can’t who they should be recruiting. If getting police commanders together in a room makes them o better, try making the room so big it covers the country. Then we’ll see some change.

Postscript: Wait, what’s this? In a story from earlier this month, with Gordon Brown pushing all sorts of policy initiatives, we find this:

Mr Brown will say: “My aim is to ensure we utilise all the innovation at our disposal to improve public services in this country and to give more power to those who use them.”

He is to praise government “successes” such as introducing broadband into every school, electronic border controls and electronic data records in hospitals.

There will also be pledges to push ahead with neighbourhood “crime mapping”, video identification of suspects, electronic school report cards, and online GP appointment booking.

(Emphasis added – CA.)

Oh, and a bonus link from Heather Brooke’s website (for she it is who has fought so hard to get MPs’ expenses made public): Police PR spending:

We found that police forces across the UK are spending £39m each year on press and PR – enough to fund an extra 1,400 full time officers and more than enough to cover the annual police pay rise withheld by the Government. The force at the top of the league (Police Service Northern Ireland) spends eight times more per person on PR than the lowest (Derbyshire). Meanwhile, forces spend nearly ten times more on PR (what police want us to know) than on FOI (what we want to know).

Government answers (cagily) on free data questions

Tuesday, April 22nd, 2008

The Green Party has been doing some sterling work trying to get the government to answer questions arising from the Cambridge economics study into the potential benefits of making data sets available for free.

We draw your attention to the exchange in the Lords (where the Greens have a representative, Lord Beaumont of Whitley) over this. Though you may find the answers uninspiring, at first:

Lord Beaumont of Whitley asked Her Majesty’s Government: Whether they intend to make the Ordnance Survey’s MasterMap available free of financial or legal restrictions. [HL2714]

The Parliamentary Under-Secretary of State, Department for Communities and Local Government (Baroness Andrews): As announced in the Budget, the Government will look closely at public sector information held by trading funds including Ordnance Survey, to distinguish more clearly what is required by government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead up to the next spending review, the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term. In the mean time Ordnance Survey will continue to generate the revenue it requires to cover its costs, to fund investments and to provide a return to government, from sales of paper mapping and from licensing use of the Crown copyright and Crown database rights in its data, including OS MasterMap.

This is the same “customers pay a fair contribution” line we’ve been hearing since the report came out (first used by DBERR, as we recall).

Of course the key to really good questions in Parliament is to ask something that the government can’t disagree with, but which isn’t part of its policy at present – because that puts it into the logical bind in which it should take up that policy. The case of economic advice given to the government is the classic one. (But you hear it all the time at Prime Minister’s Questions – the ne plus ultra of this game – in which opposing MPs try to expose gaps like this.)

Undeterred, Lord Beaumont pressed on soon afterwards (April 3):

Lord Beaumont of Whitley asked Her Majesty’s Government:

  • Whether they intend to make unrefined information held by trading funds available free of legal and financial restrictions, as recommended in the recently commissioned study, Models of Public Sector Information Provision via Trading Funds; and [HL2713]
  • Why certain financial information in the Ordnance Survey and United Kingdom Hydrographic Office sections were redacted as confidential in the recent Models of Public Sector Information Provision via Trading Funds report; and [HL2715]
  • Whether they intend to make the Met Office’s unrefined information available free of legal and financial restrictions, following the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds (p76) that this would provide a net benefit to society of £1.03 million; and [HL2733]
  • Whether in light of the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds that there is a net benefit to society if trading funds release unrefined information at marginal cost, they will review the financial and legal restrictions on all unrefined information held by public bodies. [HL2734]
  • Whether they will define the “public tasks” for trading funds in order to identify information that should be released free of financial and legal restrictions, as highlighted in the recently commissioned study Models of Public Sector Information Provision via Trading Funds (chapter 3, paragraph 3.49). [HL2735]

Lord Davies of Oldham:

As announced in the Budget, the Government will look closely at public sector information held by trading funds to distinguish more clearly what is required by Government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead-up to the next spending review the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term.

For central government bodies other than trading funds, the clear policy is that raw information should, subject to any statutory provision, be freely available or provided at the marginal cost of dissemination.

In drafting the report Models of Public Sector Information Provision via Trading Funds, Cambridge University relied on the co-operation of and provision of data from trading funds. Some of the financial data provided was commercially confidential and was therefore not published in the report. This did not alter the overall conclusions of the report.

Tom Chance, a coordinator for the Green Party (who helped draft the questions, notes on his blog that the commitment to make information freely available “subject to any statutory provision” is encouraging:

That’s good to know, and backs up the Green Party’s case for making it accessible as well, e.g. Parliamentary procedure in an open, machine-readable format rather than plain HTML, or key data on domestic energy use in one place as a canonical source rather than being scattered across different sections of government departments (Defra, BERR, CLG, etc.)

One other thing he remarks on (about the Guardian’s mapping):

It would be nice if we could supply those guys with a decent set of OpenStreetMap graphics for use in articles rather than using non-free sources too!

It’s actually an avenue we’re exploring – I’ve swapped emails with Steve Coast, who says that a change in the licensing for OSM may make this possible in the near future. It would certainly cut some of our mapping costs.