Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Archive for July, 2008

Interactive crime maps for everyone by Christmas, says Home Office

Monday, July 28th, 2008

Despite the fact that Parliament has risen (so that it’s officially the silly season – hey, was that a UFO flying past?), the Home Office is still busy at it. Today, it’s put out a press release saying that

Every neighbourhood in England and Wales will have access to the latest local crime information through new interactive crime maps, Home Secretary Jacqui Smith announced today.

The rollout of interactive crime maps follows the announcement made by the Home Secretary earlier this month, as part of the Policing Green Paper, that every police force in the country has now delivered monthly crime information to the public on their websites. New interactive crime maps will take the rollout of local crime information to the next level.

By the end of the year every police force area will produce crime maps which will allow the public to:

* see where and when crime has happened, down to street level for some crimes;

* make comparisons with other areas; and

* learn how crime is being tackled by their local neighbourhood policing team.

We think that the last of those is going to be very interesting indeed, since for senior police officers it will (in a nice phrase I heard on a related topic from a civil servant recently) “hold their feet to the fire”. (Strange how one has to summon images of torture when trying to get some public services to change..)

Coincidentally, we’ve had some interesting emails on the topic: one from Zubedpi.com (which, you’ll find, does some crime mapping).

And another reader wrote in at length:

“About 3 – 4 years ago I worked temporarily in Bury MBC’s Housing Department. There was a man in the Chief Executive’s department who had a GIS containing 3-years-worth of police crime data. He could rustle you up a map of recorded crimes, varying by type and date, for any local area you chose, on request. So it can’t be that difficult to do it.

“In the early 1970’s I was Area Housing Manager at Speke in Liverpool. My office was in the middle of this Council-built area some 6,000 houses and flats and the local police station was just across the street. This was long before we had computers for anything except (batch processed) rent accounting and it was before “defensible space” became an idea in good currency amongst urban designers.

“Following a disturbing interview with a widow with three children whose chronic poverty had been made even worse by being burgled 5 times in 6 months, I enlisted the help of the station sergeant. I gave him a 1:2500 plan of the estate and, at my request, he went through the station’s day book for 6 months past, putting a red felt-tip dot against the address of each recorded burglary.

“He returned the plan to me saying “I’ve done what you asked and it looks like a bad case of measles, but I’m none the wiser.” As soon as I saw the plan I was immediately the wiser. The “measles” were overwhelmingly clustered around particular styles and types of dwellings, and the 3-storey walk-up open-plan flats, where the widow lived, were many times more likely to be burgled than (say) the semi-detached houses.

“I subsequently extracted £30,000-worth of additional fencing from my bosses to enhance security. (Quite a lot in 1974.)

“The point of the story is not that I was cleverer than the police sergeant; I’m sure I wasn’t. The point is that a policeman’s eyes see a residential area one way, and a housing manager sees it another. Who knows what might be achieved if lots of people could see the data and bring their distinctive perceptions and intelligences to its analysis and interpretation?

What indeed? Simon Dickson is a bit dubious about how easy it will be for government to do this; Steven Feldman (who I think we could fairly call a sceptic about Free Our Data – which is fine; an unopposed theory has no strength) has pointed out that postcodes sometimes give more detail away than you’d think (personally, I suspect that domestic violence will be excluded from these visible crime stats).

So we’ll wait to see. By Christmas? Sounds fun.

(Crossposted with the Technology Guardian blog)

Want the Postcode Address File for free? Just ask (updated)

Monday, July 21st, 2008

Some more remarkable achievements by the Showusabetterway website – the competition set up by the UK government asking people to suggest ways to use its data to create mashups and new services, and offering a £20,000 prize for a winner. (Or possibly winners. But read on.)

The latest win: the Royal Mail is joining in, offering its Postcode Address File. Yes, you can argue that it ought to make that available for free anyway, but let’s change the world one piece at a time.

To get the full file, all you need to do – as the site explains – is to email the Royal Mail.

For full access you should email the Address Management Unit at address.management@royalmail.com Put ‘Show Us A Better Way’ in your subject heading so they know to prioritise your request.

Please also in the email say (a) the format you’d like it in (given on the details page) and your physical address, so they can send you the data on CD if you want it.

(The link above will fill in the email with the subject line pre-filled.)

This is a hell of an achievement. As I understand it, the licences will only be valid through to the end of July, so be quick. But if you’ve ever needed to see what the full PAF looks like, here’s your chance.

Obviously, we would not condone using it in ways that breach the Royal Mail licence. We’re aiming to do this legally. But it’s definitely another success for the Power of Information taskforce and Tom Watson in the Cabinet Office. He said he’d have a go on June 29th; now he’s achieved it. Three weeks? For government and licensing regimes, that’s fast.

Guardian praises Free Our Data (OK, well, not so surprising..)

Friday, July 18th, 2008

This morning’s Guardian has as its third “leader” (the opinion slot where the paper points to issues of the day), which is always “in praise of…”.

And today it’s In praise of “Free Our Data”. Hey, we’re chuffed.

The piece itself says (in part)

Businesses and others could use the data to map cheaply where crimes happen, or how much traffic is on the roads. Enthusiasts for cliff-climbing could share tidal forecasts. Those against argue that the Ordnance Survey’s work is not entirely paid for by taxpayers, or warn that it could lead to the privatisation of all data collection. These are serious points, and they should be taken into account. But the momentum is in favour of freeing up data; Cabinet Office minister Tom Watson boasts that he wakes up and immediately thinks “How can I free another dataset?” One hopes that is not literally true, but the sentiment is appreciated.

I don’t know, I like the idea of Tom Watson getting up having thought about a new dataset to make available. Heaven knows there are plenty of them.

But please go to the site and join in on the comments, which includes one with some interesting points about the British Library. (I’m not certain of the funding status of the BL, so don’t know if it would fall under the FOD umbrella or not.) Opinions? I lean towards the idea that the BL’s manuscripts are pre-existing data, and so there has to be some sort of cost involved in getting them into digital form…

Crime mappers are doing it for themselves

Thursday, July 17th, 2008

Today in the Guardian’s Technology section Heather Brooke – who was one of the key drivers behind getting access to MP’s expenses – writes (as part of the Free Our Data campaign) in Met keeps crime stats under lock and key about how the Metropolitan Police insist that (a) they’re not going to release data for crime mapping (b) even if they did, they keep it amalgamated on such a level that it wouldn’t be any use to anyone.

The Met also cites privacy as a reason not to release location specific crime data. Yet the Data Protection Act does not prohibit personal information being disclosed, even if one considers anonymised crime reports “personal”; and Boris Johnson’s pledge was only ever to publish crime data by street level, not by exact address. The law’s purpose is to ensure that disclosure is for a legitimate purpose. State-mandated ignorance benefits no one.

Crimes are not a great secret, particularly not violent crimes – such as the spate of stabbings in the UK in recent months – though without access to the raw data, how can we know how and where it’s rising? [Richard] Pope [of planningalerts.com] thinks the main problem is that the police are not technically savvy, citing an encounter at a meeting between locals, the council and the police where the Met admitted it couldn’t provide incident detail broken down by area – so the council ended up paying the Met just to get this information.

But people aren’t necessarily waiting for the police. Take this mashup generated by MapMan which looks at that topic du jour, knife crime.

Via the Digital Urban blog, here’s London Teenage Murders 2007, Knife Assaults and Regeneration Areas: Mapped – A Clear Pattern Emerges:

Created using Google MyMaps the list has been compiled via various websites (such as http://www.capitalradio.co.uk/article.asp?id=532062) with street names identified in related press articles and plotted on the map. Actual position within the street will not be accurate, but the street names themselves should be. Note the map relates to all murders, not just knife related incidents.

Using MapTube [URL corrected] the map can be overlaid with other data sets, such as a map uploaded detailing assault using a knife or sharp objects extracted from all hospital admissions (2007). The map is based on data with a cause code of ICD-10 X99 (assault by sharp object) and excludes all codes that may indicate accidental injury (ICD10 – W25, W26), self inflicted (ICD10 – X78) and undetermined intent (ICD10 Y28).

Figures are directly age standardised per 100,000 population with CI’s – Actual counts were excluded in the map due to disclosure surrounding low numbers. By overlaying the two maps you begin to get a picture of the extent of knife crime and the number of murders in London.

Each link is clickable for more information. Such data should really be available via either the http://www.london.gov.uk/ or http://www.met.police.uk/ along with other locations of crime in the city. It may be alarming to see such incidents mapped but this is the city we live in and the public should have a right to view exact locations of crime in their neighbourhoods.

There’s plenty more: they then overlay urban deprivation and find an interesting correlation with the number of teenage murders.

OK, so you might find that obvious. But it also tells you where the energies need to be focussed – and whether parents in Hampstead or Notting Hill really need to worry about the possibility of their child being a victim.

(One other thing: the gender of the victims. I suspect it’s overwhelmingly male too, isn’t it?)

Anyhow, this is all stuff that’s been done at zero cost to the police. Maybe if they think they’re overcome with data, we could help them out some more. Make the data available for free, and we’ll help you for free.

(crossposted with the Guardian Technology blog)

Ordnance Survey seeks a chairman/woman. But why?

Wednesday, July 9th, 2008

How interesting: we note from the EPSIPlus blog (a bit late – since the job application has long since closed, so if you were wanting to do this, you’ve missed the boat) that Ordnance Survey is seeking a non-executive chair.

How intriguing. I think I’m right in saying that none of the other trading funds is chaired; and as the advert itself says, “It is within the plan to modernise the governance of Ordnance Survey; as a result Ordnance Survey are seeking to appoint the first Non-executive Chair in the organisations’ [sic] 217-year history.”

We hear that the appointee will probably be chosen sometime this month.

So what sort of person are they looking for?

The ideal candidate will be an experienced Chair who understands how to build commercial opportunities in the public sector and who has the intellect to take forward a challenging debate about Ordnance Survey’s future strategy. S/he will have experience of change.

Of change? Change, at OS? Why? How utterly fascinating.

The ad itself (click for larger version) says the role requires that they “develop and champion a clear and compelling strategy to a broad range of stakeholders; ensure the board is effective in delivering a strategy balancing the nation’s interest with commercial imperatives [emphasis added – CA]; scrutinise performance and governance structures in line with owner’s objectives. Evaluate board skills mix and performance.”

As if that wasn’t interesting enough..

Here are the “key responsibilities” laid out in the document:

The key responsibilities are to:

  • Ensure that the Board as a whole is effective in developing a strategy and corporate business plans for Ordnance Survey, scrutinising its performance against the endorsed plans and acting in the best interests of the Department for Communities and Local Government as shareholder, while balancing the need for Ordnance Survey to act in the nation’s interest within in a commercially competitive environment;
  • Ensure that the shareholder receives full and timely feedback on the organisation’s business performance, its progress against plans, the future development of the Corporate Plan, and any other issues requiring attention;
  • Ensure the maintenance of an effective board, with an appropriate balance of skills and experience, including key appointments as required. The Chair will be part of the selection panel for the recruitment of any new Chief Executive and Non-executive Directors;
  • Ensure appropriate governance arrangements are established and implemented in line with best practice and the requirements of a public body;
  • Actively contribute to the management of relationships with Ordnance Survey’s stakeholders both in Whitehall, the devolved administrations and beyond, and represent Ordnance Survey as appropriate with customers and industry players;
  • Acts as a source of advice and support on business issues to the Chief Executive and other Executives as necessary.
  • The Chair is responsible for upholding good governance at Ordnance Survey. S/he will ensure appropriate and effective Board sub-committees exist and will, in consultation with the Chief Executive, determine Board meeting frequency and agenda. A key role is to ensure that all Non-executive Directors are effective in the support and challenge they provide to the Executive team.

The Shareholder Executive, working for the Department for Communities and Local Government, takes a close interest in the performance management of Ordnance Survey. The Chair is expected to work constructively with senior Shareholder Executive officials.

The candidate is expected to have the usual abilities concomitant with these jobs – bulging address book, Cabinet ministers and heads of industry mobile numbers on speed dial, ability to leap tall buildings and to cure sick animals with their magic touch, that sort of thing.

On its face, it doesn’t look like the successful candidate will die from overwork: three or four days a month, which earns an annual remuneration of £40,000 – £50,000. But of course that would be to ignore how important this job will be. We’re looking forward to seeing who it is.

Obviously, if you’ve applied, do feel free to share the experience..

The postcode debate, summed up beautifully on Tom Watson’s blog

Monday, July 7th, 2008

Tom Watson MP, the Cabinet Office minister who is also the political wing of the Power Of Information taskforce, started an interesting debate on his blog, when he noted a comment by Simon Dickson about the usefulness of the Postcode Address File.

He mused, “I’m going to spend some time trying to understand just why [PAF] can’t be available for free or at marginal cost. Feel free to air your views in the comment section.”

And boy, did people air their comments. It’s worth reading in full, but I think the prize – at least the Free Our Data prize for stating the value of the free data model – goes to Greg, who (in a long and well-argued comment) sums up by responding to “Mitch” (an earlier commenter who had worked in the Royal Mail on updating the PAF):

The points you make, Mitch, are unfortunately so reminisecent of the innovation-stifling opinions of inward-facing bureaucrats which have been such a major contributor to Britain’s loss of economic advantage over the years. Examples which are now so clear include the fact that we invented public-key encryption long before the US, but kept it a government secret rather than using it to gain an edge in commerce; or that Frank Whittle invented the jet engine only to find that closed-minded bureaucrats couldn’t see it working. Bureaucrats are rarely the best people to judge whether something has a place in propelling innovation and competitiveness. The fact that there’s so much energy on my side of the postcodes debate [arguing to make it available for free] says it all.

Mitch; you should be proud that you worked on a world-leading data source. It’s just such a shame its wings are crippled by its owners.

We love it when people state the benefits so clearly. The whole thread is worth reading, though, for the vigour of the arguments on both sides.

And now, OPSI sets up an “unlock that data” channel

Monday, July 7th, 2008

The Office of Public Sector Information (OPSI) goes from strength to strength. After its chief Carol Tullo spoke out in Europe about the importance of greater access to data, OPSI has set up a web page where you can request data sets you want released:

As the regulator for public sector information re-use, we know that people can encounter problems from time to time getting hold of the information they need in the formats they want. Difficulties can include problems with charging, licensing or the data standards that public sector information is provided in.

These problems aren’t about access (which is dealt with under Freedom of Information legislation), but all the other issues which can occur when you want to do something with public sector information – copy it, remix it with other data or add value and republish it. If you are trying to re-use some public sector information, but the data you need is locked-up, this service is for you.

How it works:

  1. You describe the public sector information asset you want unlocked for re-use, and post a request to the service. We’ll check through your request and if it’s OK (e.g. not a Freedom of Information request) we’ll post it here.
  2. Others can see your request and support it, either by adding a comment or by voting. The more support a request has, the better the chances of unlocking the information you want to re-use.
  3. We’ll contact the public sector information holder and see what can be done to unlock the information for re-use. To keep things simple, if the problem relates to an issue specifically covered by the Re-use of Public Sector Information Regulations or the Information Fair Trader Scheme, we’ll treat it accordingly – so you won’t need to make a separate complaint. We’ll post back our findings here.

And there’s already one request in there, for access to OS electoral boundary details, which I recall is an issue that comes up again and again – it was certainly mentioned at the RSA/Free Our Data debate nearly two years ago.

The problem, as detailed by “Matthew”:

I find it odd that if I want to know the actual boundary of the ward or constituency I am in (co-ordinates, not just an image), I have to pay Ordnance Survey lots of money for their Boundary-Line product. I would have thought that, given it’s quite important to know which MP or councillors I’m going to have the option of electing, that this information should be freely available as part of a healthy democracy; it’s compiled by the various publicly funded Boundary Commissions/Committees as far as I know.

His ideal solution:

I think the actual data rather than just images of the boundaries should be available, so that people can create things using the data – you can’t do anything with images besides display them. For example, I can’t create a Google map (using their My Maps feature) of my ward marking on where and when councillors hold their surgeries, and other local amenities. I can’t create an application that asks people to select where they live on a map and it tell them if their Parliamentary constituency will be changing at the next general election, what it’s changing to, and what difference that makes to them.

I am aware of the election-maps.co.uk website, but this is extremely hard to use – you have to know the name of your area before you can enter a postcode, you can’t look up by e.g. ward name, and it only provides images of the boundaries.

More power to his, and OPSI’s, elbow.

This is all terrifically encouraging, especially along with the Show Us A Better Way competition using government data for imaginative (and perhaps commercial) mashups. Have you got your entry in yet?

England and Wales schools database: available here in SQL format

Friday, July 4th, 2008

As part of the government’s Show Us A Better Way competition, it has made available all sorts of databases and datasets and APIs that haven’t previously been available – such as the list of all the schools in England and Wales.

Our only quibble with the latter was that it was only provided in Excel format – which as one commenter points out is a proprietary format (though free programs like OpenOffice will open it), and anyway to really begin doing useful things with such data you need to stuff it into a database; which calls for SQL format.

Never fear, Free Our Data is here. We’ve imported the data from the Excel file into a MySQL database and exported it as an SQL file which has all the required CREATE TABLE commands, with the data.

Grab a copy.
To make sure you’ve got the correct version (in case it gets copied and used elsewhere):

the MD5 checksum of the zip file is 3f46d71d84f6047ee0162d12a9456901

and of the SQL file itself (once unzipped) is 1021643b2c1c71773f20c7a4fbd1b8e1 .

The government wants you to show it a better way (and will pay £20,000)

Wednesday, July 2nd, 2008

As an idea, Free Our Data has now begun to gain some traction in government – and even, as the whole saga over crime mapping in London shows, with the Conservatives.

Now the Power Of Information taskforce, which includes Tom Watson, the Cabinet Office minister we interviewed a while back, has started a new initiative (though competition is just as good a word) at Showusabetterway.com:

Ever been frustrated that you can’t find out something that ought to be easy to find? Ever been baffled by league tables or ‘performance indicators’? Do you think that better use of public information could improve health, education, justice or society at large?

The UK Government wants to hear your ideas for new products that could improve the way public information is communicated. The Power of Information Taskforce is running a competition on the Government’s behalf, and we have a £20,000 prize fund to develop the best ideas to the next level.

To show they are serious, the Government is making available gigabytes of new or previously invisible public information especially for people to use in this competition.

And in case you wondered if it involves puttings CDs from HMRC into envelopes..

Rest assured, this competition does not include personal information about people.

There is a set of examples – such as crime mapping, Fixmystreet, and a pointer to others such as farmsubsidy.org (which “compiles obscure information about subsidies under the Common Agricultural Policy and puts it in one place, to make it much easier to see where farm subsidies are going across Europe.”)

The team signs off with a flourish:

We’re confident that you’ll have more and better ideas than we ever will. You don’t have to have any technical knowledge, nor any money, just a good idea, and 5 minutes spare to enter the competition.

There’s already a list of submitted ideas, which includes a Road Works API, FixMyTransport (“where people with shared public transport problems could come together to get things improved”), Rate My Bus, and others.

Come on, people – tell us your ideas, then go and enter them on the site (or vice versa) and win the funding. It would be fantastic if a Guardian Tech reader could win this.

Update: just to point to some of the resources you can use (among many, many, many): mapping information from the Ordnance Survey, medical information from the NHS, neighbourhood statistics from the Office for National Statistics and a carbon calculator from the Department for Environment, Food and Rural Affairs (Defra). And these are in API form, which means they’re all ready for mashup goodness.

Although not, it seems, the Postcode Address File (though the Edubase file, with school addresses, does include postcodes).