Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens

Archive for May, 2008

Why the police are against crime mapping – and what it tells us

Friday, May 30th, 2008

The BBC’s Today programme reported this morning that the Information Commissioner has, at least temporarily, held up the plan by the new London mayor Boris Johnson to introduce crime mapping to London. Crime maps, you’ll recall, were one of Johnson’s manifesto pledges. (Note: I can’t find anything to this effect on the ICO news page – anyone got a link?)

Apart from wondering why the manifesto pledge wasn’t taken for a quick spin by Johnson and the Tories past those who might have an interest (such as the Information Commissioner), the justification given afterwards by Brian Paddick, a former deputy assistant commissioner at the London Metropolitan Police (and the Liberal Democrat candidate for the mayor’s job), was illuminating. Why? Because it showed how deep the thinking that “we know better than you” runs inside British administration.

(You can hear the clip here; requires Real Player.)

Paddick said that the police already use crime mapping internally, and that from time to time the police chiefs for various areas would be called together to account for why they weren’t doing so well.

And how effective were those meetings in getting anything done? “It depended who was chairing them,” Paddick said. Good chair = police chief shamed, or driven, to do something. Bad chair = not much happens.

So why was he against it? “There are signif downsides to that [public crime mapping] process, potential to stigmatise areas, to create ghettos… and to make underreporting ofcrime even worse – people not reporting.”

The following is a rough, first-pass transcript.

Q: But it’s just telling people where crime happens.
Paddick: But we should be focusing on wht will improve efecti of police. The New York success of accountability didn’t come through making the crime data available to the public, but to Comstat, where police commanders were compared to their peers in open forum. We have tried that in London wher have half dozen commanders – asked those who do better what the secret is so they can tell the others…. The police already use crime mapping data themselves, using it in a sophisticated way. The only difference between what happens and what Boris Johnson is suggesting is that of making it public.

Q So the police already have mapping street by street to decide where to deploy resources – all Boris Johnson is suggesting is to make it public.
Paddick: yes… there are systems to hold police commanders accountable though meeetings. Making crime maps available down to street level is a lot of pain for very little gain.

Q But if we do what New York did, why might we not get better results?
Paddick: It has to be said that Comstat process – that is, holding local police commanders to account in one room and account for why crime had gone up or down – whether or not that worked depended who was chairing the meeting. It’s not a very British thing to hold people to account in front of their peers. It had mixed results depending on how the chair held them to account. (Emphasis added – CA.)

Q I thought it was about using it in clever ways, overlaying demographics, the location of porn pawn [thanks Stuart in comments] shops, to lead to more effective deployment of police?
Paddick: …. yes there is potential to use crime mapping more effectively to deploy resources, but the basic principle is available to the police and they are using it at present.

Well, wouldn’t the court of public opinion possibly be quite a good chair for the biggest possible meeting? Accountability is an uncomfortable process, yes. We see again and again that organisations dislike it. MPs don’t like having their expense claims made public – and when they are, they fight against them. In the same way, the police probably don’t like the idea that their full-time (one hopes) efforts to stop and solve crimes might be overlooked by the people to whom it’s actually happening.

But the fact is that this is a process that is happening in society, and any organisation that tries to ignore it does so at its peril because it loses the trust of the public. Journalism (which is a topic I know about) is going through this process: readers can cross-check what we write, point out where we make errors, bring new information we didn’t have. The principle that I try to work on with blog posts like this one is that when I write it, I know more than the average reader. But that implies that half the readers know more about the topic already – and if they can be encouraged to pitch in (as, happily, people do here) then we all benefit, because we have even more information.

The police may not like the idea that their work is made visible, and some people may feel uncomfortable with the idea of knowing about crimes being reported in their vicinity. (Then again, house prices are falling fast enough anyway. Crime maps won’t make a difference.) But the idea that the police are smarter about crime and criminal patterns than all of the population who might be looking at a crime map is, frankly, insulting. The average police officer is probably much smarter about crime than most of us. But it’s that group of the wider population that can find patterns that they can’t who they should be recruiting. If getting police commanders together in a room makes them o better, try making the room so big it covers the country. Then we’ll see some change.

Postscript: Wait, what’s this? In a story from earlier this month, with Gordon Brown pushing all sorts of policy initiatives, we find this:

Mr Brown will say: “My aim is to ensure we utilise all the innovation at our disposal to improve public services in this country and to give more power to those who use them.”

He is to praise government “successes” such as introducing broadband into every school, electronic border controls and electronic data records in hospitals.

There will also be pledges to push ahead with neighbourhood “crime mapping”, video identification of suspects, electronic school report cards, and online GP appointment booking.

(Emphasis added – CA.)

Oh, and a bonus link from Heather Brooke’s website (for she it is who has fought so hard to get MPs’ expenses made public): Police PR spending:

We found that police forces across the UK are spending £39m each year on press and PR – enough to fund an extra 1,400 full time officers and more than enough to cover the annual police pay rise withheld by the Government. The force at the top of the league (Police Service Northern Ireland) spends eight times more per person on PR than the lowest (Derbyshire). Meanwhile, forces spend nearly ten times more on PR (what police want us to know) than on FOI (what we want to know).

In today’s Guardian: how Inspire was watered down

Thursday, May 22nd, 2008

Inspire sounded like it would be such a great idea: data would be swapped for free between governments, everyone would benefit.

But practice and theory tend not to be next-door neighbours. And so it has proved with Inspire, which as this week’s Guardian story – An Inspired debate about access – points out, hasn’t lived up to early hopes.

First, some very good news. Civil servants revealed last week that the British government has begun work on a system to make all the geospatial data it holds on the natural environment available for free inspection and re-use. Now the bad news. In this context, “free” means we will still have to pay to download much key data, especially if it is to be published or otherwise used commercially.

The British government has until May 2009 to “transpose” Inspire’s measures to UK legislation: the Department for Environment, Food and Rural Affairs (Defra) is leading the work. In a culture where public agencies jealously guard the intellectual property in their databases, and often have to recover the costs of running them, the new regime may face some opposition.

“It is clear that Inspire will require organisations to collaborate better,” David Lee, one of the Inspire project team at Defra, said at the Public Sector conference in London last week. He revealed that the department is considering creating a geoportal, which would allow anyone to track down any geospatial data set, regardless of who owns or maintains it. “The vast majority of transactions to members of the public will be free,” Lee said.

However it is not quite as simple as that. While the Inspire directive requires “metadata” (machine-readable data about data) to be provided free of charge, access to the databases themselves may be charged, especially if it is for commercial use.

And so that dichotomy continues: commercial use versus “private” use. Everyone, it seems, should have Chinese walls in their minds – how else can you be sure that knowledge gained in one’s personal life won’t be exploited in their commercial existence?

Free Our Data: now on Twitter

Thursday, May 15th, 2008

You may or may not have heard of Twitter, which creates a social network (sorry) around sending SMS-length messages via the web or mobile. The idea is that you can send messages to the world in general, or communicate publicly with friends, or privately with friends. You can think of it as real-time micro-blogging (ie, very short but to the minute).

If you want to understand more about the service (which is free), you can read Making the most of Twitter, at the Guardian.

But the point here is that there is a Free Our Data twitter account, which will principally be used for notifying when there are new blog posts here. (We tend to note any stories on the Guardian too, though I may add that to the feed.) It shouldn’t be overwhelming. And we’ll aim to follow those who follow us, so that we know what’s going on.

You’ll find the page at (Congratulations Ian Sealy – our first follower!)

In today’s Guardian: the problem with route data for glider pilots

Thursday, May 15th, 2008

Today in the Guardian, with Access to data needs to take off, we look at the slightly unusual (in the context of this campaign) case of glider pilots, in particular, and the hassles that they have in trying to load flight data, in particular about where they may and may not fly.

The wrinkle being that the data is available – but not in a convenient machine-readable format that they could load into their GPS-based systems. And responsibility for deciding whether to do that is split between the Civil Aviation Authority – which is a government-appointed regulator – and National Air Traffic Services, a privatised organisation in which the government is the largest minority shareholder, along with commercial airlines and staff.

The point being, if you want someone to do something that would streamline things, where do you press?

Britain must love flying. How else to explain the fact that the space over the north of London is, as Andrew Watson puts it, “one of the most complicated bits of three-dimensional airspace on the planet”? And as the spokesman for National Air Traffic Services (Nats), which looks after air traffic control for commercial services, notes, there’s also the south of England, home to commercial airports such as Heathrow, “executive” airports such as Biggin Hill and Farnborough, and 60-odd other airports used by pilots of light aircraft, parachutists, hang-gliders and gliders. It’s very, very busy.

But as Watson, a glider pilot, points out, that busy-ness leads to a peculiar conflict over data. Specifically, useful access to data about where you may and may not fly in the UK. And with about 10,000 glider pilots and 3,000 gliders in the UK making some 500,000 flights a year and able to range across the country during a single flight – that access to data matters.

The Civil Aviation Authority (CAA) sets the rules about who can fly where. As Watson explains: “The UK is covered by a three-dimensional pattern of airspace into which various kinds of aircraft may or may not be allowed to enter.” All pilots must carry a chart showing that 3D space if they fly more than five miles from their airfield. (In practice, they all carry the authoritative CAA chart, published annually.)

That might seem straightforward – except that the boundaries can change from week to week or even day to day. That means pilots must check the CAA and Nats websites for the “Notices to Airmen” (or Notams, which presumably apply to female pilots too) about temporary airspace restrictions.

What Watson finds perplexing is that though you need to know those airspace “boundaries” – which are unrelated to commercial mapping data – the CAA does not publish machine-readable versions.

It’s a fiendishly complex situation, though what’s really puzzling is why NATS and the CAA, which would benefit from glider (and other private) pilots being able to get this data easily into their machines (without the possibility of transcription errors), preventing dozens of intrusions into controlled airspace every year, don’t do it.

Then again, it’s a funny world, flying. As I learnt while researching it, Nats and the CAA don’t accept GPS as a navigation device. Why? “Because it’s intentionally degraded,” I was told (by Nats). That is, your position isn’t perfectly accurate. But only to 10m, surely? Or even 100m? Is that really going to make all the difference when you’re thousands of feet (or metres) up?

So here’s the Ordnance Survey’s public task, for the record

Thursday, May 15th, 2008

Since we asked where the Ordnance Survey (OS) public task is, we’ve been pointed to it. (Thanks again, Dan.)

So we thought that, since it seems elusive elsewhere on the web, we’d put it here for the record. And for the record, this is extracted directly from the report of the Select Committee on Communities and Local Government. No doubt there’s some copyright issue, but we can’t quite see why acknowledging a source and repeating it correctly, in brief, is a problem, especially with an organisation that we (as citizens) own.

So here’s the PDF of the Ordnance Survey public task; and below, the HTML. If you spot any errors, please tell us and we’ll aim to correct them.

(From the written evidence to the Select Committee on Communities and Local Government, Fifth report of session 2007-8, printed January 21 2008)

Ordnance Survey’s Public Task is to:

— Collect and maintain uniform datasets with national coverage forming the official record of the natural and built environment of Great Britain, containing:

— detailed mapping of the built and natural landscape (topography);86

— high resolution address data;

— transport networks (including road, rail, waterways, tracks and paths);

— terrain and height data;

— administrative and electoral boundary information; and

— geographical names (including features with imprecise extents such as “Dartmoor”, “The Cotswolds”, “Forest of Bowland”, “South Downs” etc).

— Provide a nationally consistent cartographic portrayal of the topography of Great Britain at various scales including large scales.

— Maintain a definitive three-dimensional satellite-based geodetic87 reference framework of Great Britain that underpins the national datasets and facilitates the integration and analysis of location-based information from many sources.

— Make the content of the datasets widely available in forms that are accessible to customers of all types for wider benefit. This will be in the form of products which are the direct output of those datasets. As part of its Public Task and in order to fulfil its trading fund obligations, Ordnance Survey will charge all customers for the licensing and use of its products.

— Conduct its activities efficiently and effectively to maximise the broader economic value of its data, whilst complying with trading fund requirements.

— Ensure that its data is capable of supporting the principles underlying the Digital National Framework in underpinning the association and integration of third party geospatial information.

In order to fulfil its Public Task, Ordnance Survey is required to collect and maintain topographic data at the following scales:

— High Change Geographies88: 1:1250

— Rural Areas89: 1:2500

— National cover: 1:10000, 1:25000, 1:50000, 1:250000

The above datasets are required to fulfil Ordnance Survey’s Public Task to ensure that a comprehensive, nationally consistent version of each exists in the public interest.

Annex 1A sets out:

— Ordnance Survey’s datasets which fulfil its Public Task;

— the products that are currently the direct output of those datasets; and

— the rationale for including the various data within the Public Task.

The minimum levels of accuracy and revision required for those datasets are at Annex 1B.

The referencing systems and data collected and maintained by Ordnance Survey contribute to the development and integration of geographic and location based information collected and used by government, business and individuals.

As the National Mapping Agency of Great Britain, Ordnance Survey represents Great Britain overseas as experts on Survey, Mapping and Geographic Information. It provides a focus for the provision of public sector information into pan-European and international collaboration by National Mapping Agencies and the European Directive on the Infrastructure for Spatial Information in Europe (INSPIRE).

In discharging its Public Task, Ordnance Survey:

— Seeks to maximise both the accessibility of, and the broader economic benefit arising from the use of the data. It therefore creates products directly from these datasets and makes them available, including through commercial licensing, to government and business customers and consumers.

— Takes into account the views of customers (as well as, inter alia, technological changes and its trading fund requirements) to ensure that the range and content of these products meets their changing needs, and makes changes to content, accuracy and revision policies as may be necessary to ensure that the datasets and products remain fit for purpose. This may result in adding or withdrawing products from availability from time to time, as well as enhancing content andfunctionality.


86 Topography: Including defining the surface shape and composition of the landscape, comprising both natural and artificial features.

87 Geodetic: Relating to the scientific discipline that deals with the precise measurement and representation of the earth, its gravitational field, and other related phenomena. Within Ordnance Survey geodetic-quality information forms the high precision framework that ensures the correct positioning of all mapping and other data against the National Grid.

88 Predominantly urban areas and areas of significant development.

89 Predominantly rural settlements and developed agricultural land.

Where is Ordnance Survey’s public task set out, exactly? And why is it paying an external PR company? (updated)

Monday, May 12th, 2008

As Mike Cross has posted, there is some interesting triangulation going on through Parliamentary questions on how the Ordnance Survey’s public task was determined. (There seems to be a groundswell of questions from Conservative MPs inquiring into it.) These are generally batted away with standard civil service responses – basically saying “well, we did it how we did it.”

So we wondered: where can the OS’s public task document be found?

Not on the OS website, where you simply find

For the purposes of the PSI regulations, Ordnance Survey’s Public Task is defined as embracing everything we do from time to time to fulfil our obligations under the Ordnance Survey Trading Fund Order 1999 (SI 99/965) and the Ordnance Survey Framework Document 2004.

This is akin to saying “whatever we do, it’s a public task, because we do it.”

Nor on the Shareholder Executive site (the SE is the government arm which determines what its trading fund should do), where the relevant page says:

Ordnance Survey’s objective is to deliver its Public Task. This can be summarised [emphasis added – CA] as to:

  • collect and maintain uniform geographic datasets with national coverage, and provide nationally-consistent mapping
  • maintain a definitive three-dimensional satellite-based geodetic reference framework to enable correct positioning of mapping and other data against the National Grid
  • ensure that its data is capable of supporting the principles underlying the Digital National Framework, to allow integration and association with other geospatial information
  • make the content of its datasets widely available in forms that are accessible to all customers for wider benefit, and charge its customers for the licensing and use of its products
  • conduct its activities efficiently and effectively to maximise the broader economic value of its data, while complying with trading fund requirements and creating long-term shareholder value.
  • Yes, that’s an improvement, but we don’t want summarised – we want full. Where is it? Anyone?

    (Bonus link: OS has paid an organisation called Mandate £42,076.20 + VAT (that’s £49.400) since August 2007 for “consultancy and advice on Corporate Communications and Public Affairs.” Interesting to see that Mandate says that its skills in public affairs includes “Winning over the decision makers who matter”.

    Wonder why OS would think it needs those services?)

    Update: Dan MacDonald has provided a pointer to the OS’s public task in the comments – but he points out that “it’s only available if you trawl thru’ the pdf version of the evidence given at“. Thanks, Dan.

    Crime mapping for London, Boris? We’ll start the clock now

    Monday, May 5th, 2008

    Boris Johnson’s election as mayor of London means that we can see how committed – and effective – the Tories are in their claims to want to provide more data from councils and government to the public.

    Among the pledges made by Johnson as part of his election campaign was to introduce crime mapping to London.

    David Cameron, the Tory leader, wants every police force in the country to record every crime online, every month, in map form. There’s also a pamphlet (PDF) on their proposals.

    How long will it take in London, Boris was asked? “We’ll start on day one, I’ll go to the Met and say listen, this is a fantastic idea…let’s see what we can do to put this on the web so people can look at exactly what’s going on in their neighbourhood, and use that tool to drive down crime.”

    David Davis, shadow home secretary, says “the police will get a massive reduction in red tape, targets..”

    Given that Boris didn’t get the final say-so until midnight on Friday, and Monday May 5th is a holiday, perhaps Tuesday will be the “day one” when he’ll stroll into New Scotland Yard and mention this. We’ll keep count. (Apparently he went to work on Monday. So the clock is well and truly started.)

    If it arrives, it’ll certainly be a win for free data. Interesting questions: will it be on Google Maps or Ordnance Survey’s OpenSpace, or some other provider’s maps? Will it be redistributable? What sort of copyright will it have?

    The proposals have considered privacy issues, and suggest:

    We believe

    there should be three kinds of map for each area for three categories of offences:

    (a) where privacy is less of an issue: at their exact location with a pinpoint showing exactly where the offence occurred;
    (b) where privacy might be an issue: identifying a 300 metre long street section within which an offence occurred; and
    (c) where privacy is an issue: identifying the nearest whole street within which an offence occurred.

    (Assaults would be in category (a); sexual attacks and domestic violence in (c), for example.)

    The copyright issue – and reusability – may have been determined: from the pamphlet:

    As our new regime spreads more crime information into the public arena, we expect people will want to create their own crime maps on the web of their own neighbourhood. That will be a matter for individuals, social entrepreneurs, Neighbourhood Watches and others. Once the appropriate statistics are freely available, it would be comparatively simple from a technical point of view for citizens themselves to overlay the statistics onto an online street map. There are already many different mapping systems of the UK available to the public online, including Yahoo, Google, and Streetmap.

    But what about that terrible ogre for Britons – house prices?

    It is sometimes pointed out that crime mapping, in identifying areas of high crime, affects the level of house prices in those areas. In fact, crime mapping can actually raise house prices if identifying areas where crime is rising then leads to effective action being taken to cut crime. In our view crime mapping will provide a clear and powerful incentive to affected residents to complain and insist on effective crime fighting solutions to reduce the crime. And when crime is reduced this typically increases house prices – rather than the reverse.

    So, we’ll start the clock tomorrow at 8am, when mayors ought to be arriving in their offices to start work.