Free Our Data: the blog

A Guardian Technology campaign for free public access to data about the UK and its citizens


Archive for 2008

Show Us A Better Way competition closes Sept 30: get your entries in!

Wednesday, September 24th, 2008

As the title says: on Tuesday 30th, the competition at Show Us A Better Way closes, and judging will begin.

The idea, just as a reminder, is to come up with innovative ways to use government-collected datasets, in order to demonstrate the value that can be generated from them.

There are already hundreds of entries – which means that the judging process will be rather complex. In fact, if anyone has ideas for how the competition could use crowdsourcing (aka voting) in order to choose the finalists for consideration by a judging panel, we’d be interested to hear so we can pass them on.

Ordnance Survey’s lobbying, part 2

Thursday, August 28th, 2008

In Guardian Technology of August 21 we reported on Ordnance Survey’s hiring of a lobbying company called Mandate, and how it had kept watch on MPs and organisations which seemed to be interested in the whole “free data” concept.

Ordnance Survey responded to the story: this is a reprint in full of its letter. Following this, how a story in today’s Guardian Technology examines the contradictions between what OS says it does (and what its minister, Iain Wright, responded in his original Parliamentary answer – that it’s “consultancy and advice on Corporate Communications and Public Affairs”) and what the email track seems to suggest.

First, the letter:
“We are more than happy that the Guardian has shown this interest in the way that Ordnance Survey communicates about the important work that we do.
“Ordnance Survey data helps underpin life in Britain. It is relied on by business and society, from battling the effects of climate change to the sat nav in millions of cars. Our data is mapped down to the nearest few centimeters and updated up to 5000 times a day. It is this consistent level of quality, currency and detail that makes it so vital for public services, ranging from emergency planning to the delivery of everyday services on the ground.
“It is because Ordnance Survey data is so vital that parliamentarians and other important stakeholders expect us to communicate with them about our work. That is why we engage with politicians from all parties who care about the services that we provide. We have a duty to inform them on our role collecting the data needed to map every feature on the landscape, and how we intend to maintain the quality of this sophisticated data going forward.
“We’re committed to the best possible communications with all our stakeholders, now and in the future.
“Nicole Perry head of public affairs, Ordnance Survey”

And so to today’s story, Ordnance Survey defends its use of lobbying company:

The Free Our Data campaign agrees with Perry that the need to educate opinion-makers about geographical data in the digital age is an important part of its public task. However, a study of the 361 printed pages of correspondence between OS and Mandate, obtained under the Freedom of Information Act, suggests that its publicity campaign strays into broader areas of government policy. In particular, on the question of whether it and other information agencies should continue to be run as trading funds, required to cover their costs by charging for access to data. (The Free Our Data campaign argues that this policy hampers state and community initiatives to make better use of data.)

Among the papers we received is an account of a seminar on trading funds, organised on April 29 by Locus, an industry body which represents users of public sector information (and which retains its own lobbyist, a firm called Quintus Public Affairs). In an email to Perry, a Mandate executive reveals that she attended the seminar, accompanied by a colleague “Eleanor”, and reports back “on comments from the meeting that you should be aware of”. These include the views of Locus’s chairman, “Bryan Carlsberg” (sic – his name is Carsberg) that member companies should talk to the Conservative party on this issue “as they are currently looking for proposals for their manifesto”.

We’re just trying to square that with Iain Wright’s suggestion that this is “consultancy and advice on Corporate Communications and Public Affairs”. It seems very like, well, straying into areas of policy.

And there’s also the question of quite what is recommended.

On April 24 this year, Mandate alerted Perry that a Conservative MP, Greg Clark, had tabled a question about the relationship with Mandate. The email urged Perry to “please rest assured” that Clark had asked many such questions, and that the information needed in response is “minimal”. We will see whether Ordnance Survey’s minister follows that advice.

I had always thought that it was the responsibility of departments and government agencies to seek to answer Parliamentary Questions as fully as possible; if this is not done and the minister answering is not sufficiently briefed, it can be extremely embarrassing, initially for the minister. Is “minimal” advice sufficient? And overall, has Mandate really been good value for money?

One other thing we’ve heard:

Ordnance Survey’s use of a lobby firm to engage in the free data debate is likely to be on the agenda at the next meeting of the government’s Advisory Panel on Public Sector Information next month.

We’ll look forward to the minutes of that meeting.

Ordnance Survey’s lobbying, part 1

Thursday, August 28th, 2008

Coming late to posting this (I blame holidays), but Mike Cross entered an FOI request after we noticed in May that it had paid a lobbying company called Mandate about £49,000 for “consultancy and advice on Corporate Communications and Public Affairs”.

Except that that description seems rather askew from what we found in the emails (released on paper, and redacted – you know, blanked out – to protect the names of individuals in Mandate and Ordnance Survey). Our thanks by the way to Greg Wright, then shadow minister for the Cabinet Office and MP for Tunbridge Wells, who asked the question (not on our behalf; we’ve no idea why he asked it, though it seems to have been well-informed). Iain Wright of DCLG answered it.

And so to our first story on the topic, which appeared in Technology Guardian under the headline “Ordnance Survey hires PR company to lobby politicians” (can you tell our lawyers checked it first?):

The correspondence reveals that Ordnance Survey (OS) is targeting MPs from Westminster and devolved assemblies, civil servants and leading figures in the free data debate. The agency openly attends party conferences and other political events to promote the value of geographical data. However, earlier this year a Parliamentary question revealed that it had paid a company called Mandate £42,076.20 plus VAT since August 2007.

However, it refused to release emails on backup tapes on the grounds of cost, £11,250. The correspondence released – mainly between Nicole Perry, head of public affairs, and Mandate executives whose names have been blanked out – reveals a busy programme of meetings with politicians, especially those who have asked questions in Parliament about OS’s corporate affairs, or about free data.

Among MPs named are Labour’s David Taylor – “you might recall that he’d (sic) raised the issue of free data” – Conservatives Anne McIntosh and Paul Beresford, and several Welsh Assembly members. According to Mandate, Robert Kee (Conservative, Salisbury) “is a big supporter of OS, so I don’t think this [a Parliamentary question] is anything to worry about”.

This story (of which that’s only part; go to the story itself to read in full) generated a response from OS, which will be dealt with (and we’ll publish the letter in full) in the next post up.

Met Police put up first version of crime mapping system

Monday, August 25th, 2008

Apologies for coming late to this; I’ve been away (and Mike Cross has mislaid his passwords to the blog).

Anyway: the Met Police have made their first version of the much-promised crime mapping system available. It’s at http://maps.met.police.uk/ and says it has been developed “in conjunction with the Metropolitan Police Authority and the Mayor of London”.

And the test version does come with a sort of data health warning:

Please note, that whilst every effort is made to record the details of crime and its location as accurately as possible, there are occasions when victims are unable to provide the actual location of a crime. In these instances, the site will not be able to display all the crime reported to the police.

So we make that, since 5 May 2008 when Boris Johnson came to power promising crime mapping, a total of 101 days to get to a beta implementation. As political fulfilment goes, that’s really not bad.

There are observations and criticism: Simon Dickson is only half-surprised that it’s built on Google Maps, not Ordnance Survey’s OpenSpace (“Here’s a extra-high-profile government mapping application, and they’ve made a conscious – and entirely predictable – decision not to build it using the tool provided by the government’s own mapping agency.”); though Tom Loosemore, writing in his personal capacity, comments that

The biggest missed opportunity is the lack of proper profile for your local coppers (aka your “Safe Neighbourhood Team”). The site should make it dead easy for your to contact them, and challenge/shape their priorities. After all, even coppers work for you…

True, though it’s still early days. My principal criticism is that it simply shows crimes against “average”. If you go for a postcode (the first half, eg SW12 is enough) then you get total figures for an area, but that too isn’t helpful – there’s no idea of whether that covers a large area (is Balham, where I used to live, larger than Wandsworth, which apparently has far more crime yet is still “average”?).

Basically, it’s still keeping the information inside the police station walls, and I don’t think that’s enough. This information doesn’t have to be personalised, but it does need to be localised – in fact, made precise.

Update: there’s a Guardian story which has some quotes from police people involved:

A Met spokesman emphasised that this version of the map is a test phase and will be subject to a technical review.

“The software development will enhance the service that we currently provide regarding the number, rate and geographical location of defined crime types within the capital,” the spokesman said.

Ordnance Survey appoints first (non-exec) chair: Sir Rob Margetts

Thursday, August 7th, 2008

So, what does anyone out there know about Sir Rob Margetts’s web 2.0 credentials? Because that’s what he’s going to have to show now that he’s going to be the Ordnance Survey’s new, first non-executive chair, as announced in a formal press release today.

Sir Rob, it says, is a CBE, Fellow of the Royal Society of Engineers and FIChemE (institute of chemical engineers). The appointment is initially for three years.

Biography:

Sir Rob began his career with the ICI Group in 1969, progressing through a number of appointments within the group prior to joining the Board in 1992 and being Vice-Chairman from 1998-2000. Since 2000 he has been Chairman of Legal & General Group plc and in 2006 also became Chairman of Ensus Ltd.

(In case you’re wondering, Ensus aims “to become a leading provider of bioethanol to the European transport fuels market. Made from natural products such as wheat and sugar beet, bioethanol offers a renewable and environmentally friendly alternative to oil for petrol driven vehicles.”)

He is also a non-executive director of Anglo American plc and Chairman of the Energy Technologies Institute. In addition, Sir Rob was Chairman of the Natural Environment Research Council (NERC) from 2001-06.

To be honest, on first glance it’s hard to see precisely what he’s bringing to this party. After all, what OS needs now is someone who can chart a course through the new web 2.0 world, where what matters is being able to see the best ways to exploit (in the most general sense) its intellectual property, amidst a changing political (small p), economic and online climate.

But back to the press release:

Of his appointment, Sir Rob comments: “I am delighted to have been invited to chair Ordnance Survey. It is a great privilege to join this organisation, which provides so much benefit to its users and enjoys an excellent reputation. I much look forward to working with Vanessa and Board colleagues as well as staff, business partners and customers.”

Vanessa Lawrence, who remains Ordnance Survey’s Accounting Officer and continues to report directly to the Minister, welcomes the appointment, saying: “I am delighted that Sir Rob will be joining us in the new role of Non-Executive Chair. With his outstanding record of business leadership and organisational development, I know that his knowledge, skills and expertise will be invaluable as we move forward in the coming years.”

Love the bit pointing out that Vanessa Lawrence continues to report directly to Ian Wright at DCLG. Why was that felt necessary? In case the media got the “wrong” idea about what this appointment means, one must presume. But then what’s the point of a chair over whose head the chief exec can appeal, since both are appointed by the same person?

Sir Rob will report to Shareholder Executive, the body that advises Ministers and senior officials on the government’s “shareholding” in organisations like Ordnance Survey. His appointment brings Ordnance Survey in line with other Trading Funds with Non-Executive Chairs who have already modernised their governance structures, such as the Met Office and UK Hydrographic Office.

Make no mistake: this is a shakeup of how OS functions. It’s a political (small p) shock to its system, but what will be interesting will be to see precisely how that plays out.

Anyone got any more information about Sir Rob Margetts, and in particular his approach to online intellectual property, business models and the web?

Show Us A Better Way offers £20,000 for developing prototypes

Monday, August 4th, 2008

The (British) government’s Show Us A Better Way competition has added another £20,000 string to its bow: this time, the prize – in addition to the original one – is available for prototypes of ideas that have been made on the site.

It offers a number of sites which will help with mashups, and links to useful code resources. Well worth investigating, since you don’t have to be one of the people with the ideas in order to benefit greatly from them.

Interactive crime maps for everyone by Christmas, says Home Office

Monday, July 28th, 2008

Despite the fact that Parliament has risen (so that it’s officially the silly season – hey, was that a UFO flying past?), the Home Office is still busy at it. Today, it’s put out a press release saying that

Every neighbourhood in England and Wales will have access to the latest local crime information through new interactive crime maps, Home Secretary Jacqui Smith announced today.

The rollout of interactive crime maps follows the announcement made by the Home Secretary earlier this month, as part of the Policing Green Paper, that every police force in the country has now delivered monthly crime information to the public on their websites. New interactive crime maps will take the rollout of local crime information to the next level.

By the end of the year every police force area will produce crime maps which will allow the public to:

* see where and when crime has happened, down to street level for some crimes;

* make comparisons with other areas; and

* learn how crime is being tackled by their local neighbourhood policing team.

We think that the last of those is going to be very interesting indeed, since for senior police officers it will (in a nice phrase I heard on a related topic from a civil servant recently) “hold their feet to the fire”. (Strange how one has to summon images of torture when trying to get some public services to change..)

Coincidentally, we’ve had some interesting emails on the topic: one from Zubedpi.com (which, you’ll find, does some crime mapping).

And another reader wrote in at length:

“About 3 – 4 years ago I worked temporarily in Bury MBC’s Housing Department. There was a man in the Chief Executive’s department who had a GIS containing 3-years-worth of police crime data. He could rustle you up a map of recorded crimes, varying by type and date, for any local area you chose, on request. So it can’t be that difficult to do it.

“In the early 1970’s I was Area Housing Manager at Speke in Liverpool. My office was in the middle of this Council-built area some 6,000 houses and flats and the local police station was just across the street. This was long before we had computers for anything except (batch processed) rent accounting and it was before “defensible space” became an idea in good currency amongst urban designers.

“Following a disturbing interview with a widow with three children whose chronic poverty had been made even worse by being burgled 5 times in 6 months, I enlisted the help of the station sergeant. I gave him a 1:2500 plan of the estate and, at my request, he went through the station’s day book for 6 months past, putting a red felt-tip dot against the address of each recorded burglary.

“He returned the plan to me saying “I’ve done what you asked and it looks like a bad case of measles, but I’m none the wiser.” As soon as I saw the plan I was immediately the wiser. The “measles” were overwhelmingly clustered around particular styles and types of dwellings, and the 3-storey walk-up open-plan flats, where the widow lived, were many times more likely to be burgled than (say) the semi-detached houses.

“I subsequently extracted £30,000-worth of additional fencing from my bosses to enhance security. (Quite a lot in 1974.)

“The point of the story is not that I was cleverer than the police sergeant; I’m sure I wasn’t. The point is that a policeman’s eyes see a residential area one way, and a housing manager sees it another. Who knows what might be achieved if lots of people could see the data and bring their distinctive perceptions and intelligences to its analysis and interpretation?

What indeed? Simon Dickson is a bit dubious about how easy it will be for government to do this; Steven Feldman (who I think we could fairly call a sceptic about Free Our Data – which is fine; an unopposed theory has no strength) has pointed out that postcodes sometimes give more detail away than you’d think (personally, I suspect that domestic violence will be excluded from these visible crime stats).

So we’ll wait to see. By Christmas? Sounds fun.

(Crossposted with the Technology Guardian blog)

Want the Postcode Address File for free? Just ask (updated)

Monday, July 21st, 2008

Some more remarkable achievements by the Showusabetterway website – the competition set up by the UK government asking people to suggest ways to use its data to create mashups and new services, and offering a £20,000 prize for a winner. (Or possibly winners. But read on.)

The latest win: the Royal Mail is joining in, offering its Postcode Address File. Yes, you can argue that it ought to make that available for free anyway, but let’s change the world one piece at a time.

To get the full file, all you need to do – as the site explains – is to email the Royal Mail.

For full access you should email the Address Management Unit at address.management@royalmail.com Put ‘Show Us A Better Way’ in your subject heading so they know to prioritise your request.

Please also in the email say (a) the format you’d like it in (given on the details page) and your physical address, so they can send you the data on CD if you want it.

(The link above will fill in the email with the subject line pre-filled.)

This is a hell of an achievement. As I understand it, the licences will only be valid through to the end of July, so be quick. But if you’ve ever needed to see what the full PAF looks like, here’s your chance.

Obviously, we would not condone using it in ways that breach the Royal Mail licence. We’re aiming to do this legally. But it’s definitely another success for the Power of Information taskforce and Tom Watson in the Cabinet Office. He said he’d have a go on June 29th; now he’s achieved it. Three weeks? For government and licensing regimes, that’s fast.

Guardian praises Free Our Data (OK, well, not so surprising..)

Friday, July 18th, 2008

This morning’s Guardian has as its third “leader” (the opinion slot where the paper points to issues of the day), which is always “in praise of…”.

And today it’s In praise of “Free Our Data”. Hey, we’re chuffed.

The piece itself says (in part)

Businesses and others could use the data to map cheaply where crimes happen, or how much traffic is on the roads. Enthusiasts for cliff-climbing could share tidal forecasts. Those against argue that the Ordnance Survey’s work is not entirely paid for by taxpayers, or warn that it could lead to the privatisation of all data collection. These are serious points, and they should be taken into account. But the momentum is in favour of freeing up data; Cabinet Office minister Tom Watson boasts that he wakes up and immediately thinks “How can I free another dataset?” One hopes that is not literally true, but the sentiment is appreciated.

I don’t know, I like the idea of Tom Watson getting up having thought about a new dataset to make available. Heaven knows there are plenty of them.

But please go to the site and join in on the comments, which includes one with some interesting points about the British Library. (I’m not certain of the funding status of the BL, so don’t know if it would fall under the FOD umbrella or not.) Opinions? I lean towards the idea that the BL’s manuscripts are pre-existing data, and so there has to be some sort of cost involved in getting them into digital form…

Crime mappers are doing it for themselves

Thursday, July 17th, 2008

Today in the Guardian’s Technology section Heather Brooke – who was one of the key drivers behind getting access to MP’s expenses – writes (as part of the Free Our Data campaign) in Met keeps crime stats under lock and key about how the Metropolitan Police insist that (a) they’re not going to release data for crime mapping (b) even if they did, they keep it amalgamated on such a level that it wouldn’t be any use to anyone.

The Met also cites privacy as a reason not to release location specific crime data. Yet the Data Protection Act does not prohibit personal information being disclosed, even if one considers anonymised crime reports “personal”; and Boris Johnson’s pledge was only ever to publish crime data by street level, not by exact address. The law’s purpose is to ensure that disclosure is for a legitimate purpose. State-mandated ignorance benefits no one.

Crimes are not a great secret, particularly not violent crimes – such as the spate of stabbings in the UK in recent months – though without access to the raw data, how can we know how and where it’s rising? [Richard] Pope [of planningalerts.com] thinks the main problem is that the police are not technically savvy, citing an encounter at a meeting between locals, the council and the police where the Met admitted it couldn’t provide incident detail broken down by area – so the council ended up paying the Met just to get this information.

But people aren’t necessarily waiting for the police. Take this mashup generated by MapMan which looks at that topic du jour, knife crime.

Via the Digital Urban blog, here’s London Teenage Murders 2007, Knife Assaults and Regeneration Areas: Mapped – A Clear Pattern Emerges:

Created using Google MyMaps the list has been compiled via various websites (such as http://www.capitalradio.co.uk/article.asp?id=532062) with street names identified in related press articles and plotted on the map. Actual position within the street will not be accurate, but the street names themselves should be. Note the map relates to all murders, not just knife related incidents.

Using MapTube [URL corrected] the map can be overlaid with other data sets, such as a map uploaded detailing assault using a knife or sharp objects extracted from all hospital admissions (2007). The map is based on data with a cause code of ICD-10 X99 (assault by sharp object) and excludes all codes that may indicate accidental injury (ICD10 – W25, W26), self inflicted (ICD10 – X78) and undetermined intent (ICD10 Y28).

Figures are directly age standardised per 100,000 population with CI’s – Actual counts were excluded in the map due to disclosure surrounding low numbers. By overlaying the two maps you begin to get a picture of the extent of knife crime and the number of murders in London.

Each link is clickable for more information. Such data should really be available via either the http://www.london.gov.uk/ or http://www.met.police.uk/ along with other locations of crime in the city. It may be alarming to see such incidents mapped but this is the city we live in and the public should have a right to view exact locations of crime in their neighbourhoods.

There’s plenty more: they then overlay urban deprivation and find an interesting correlation with the number of teenage murders.

OK, so you might find that obvious. But it also tells you where the energies need to be focussed – and whether parents in Hampstead or Notting Hill really need to worry about the possibility of their child being a victim.

(One other thing: the gender of the victims. I suspect it’s overwhelmingly male too, isn’t it?)

Anyhow, this is all stuff that’s been done at zero cost to the police. Maybe if they think they’re overcome with data, we could help them out some more. Make the data available for free, and we’ll help you for free.

(crossposted with the Guardian Technology blog)

Ordnance Survey seeks a chairman/woman. But why?

Wednesday, July 9th, 2008

How interesting: we note from the EPSIPlus blog (a bit late – since the job application has long since closed, so if you were wanting to do this, you’ve missed the boat) that Ordnance Survey is seeking a non-executive chair.

How intriguing. I think I’m right in saying that none of the other trading funds is chaired; and as the advert itself says, “It is within the plan to modernise the governance of Ordnance Survey; as a result Ordnance Survey are seeking to appoint the first Non-executive Chair in the organisations’ [sic] 217-year history.”

We hear that the appointee will probably be chosen sometime this month.

So what sort of person are they looking for?

The ideal candidate will be an experienced Chair who understands how to build commercial opportunities in the public sector and who has the intellect to take forward a challenging debate about Ordnance Survey’s future strategy. S/he will have experience of change.

Of change? Change, at OS? Why? How utterly fascinating.

The ad itself (click for larger version) says the role requires that they “develop and champion a clear and compelling strategy to a broad range of stakeholders; ensure the board is effective in delivering a strategy balancing the nation’s interest with commercial imperatives [emphasis added – CA]; scrutinise performance and governance structures in line with owner’s objectives. Evaluate board skills mix and performance.”

As if that wasn’t interesting enough..

Here are the “key responsibilities” laid out in the document:

The key responsibilities are to:

  • Ensure that the Board as a whole is effective in developing a strategy and corporate business plans for Ordnance Survey, scrutinising its performance against the endorsed plans and acting in the best interests of the Department for Communities and Local Government as shareholder, while balancing the need for Ordnance Survey to act in the nation’s interest within in a commercially competitive environment;
  • Ensure that the shareholder receives full and timely feedback on the organisation’s business performance, its progress against plans, the future development of the Corporate Plan, and any other issues requiring attention;
  • Ensure the maintenance of an effective board, with an appropriate balance of skills and experience, including key appointments as required. The Chair will be part of the selection panel for the recruitment of any new Chief Executive and Non-executive Directors;
  • Ensure appropriate governance arrangements are established and implemented in line with best practice and the requirements of a public body;
  • Actively contribute to the management of relationships with Ordnance Survey’s stakeholders both in Whitehall, the devolved administrations and beyond, and represent Ordnance Survey as appropriate with customers and industry players;
  • Acts as a source of advice and support on business issues to the Chief Executive and other Executives as necessary.
  • The Chair is responsible for upholding good governance at Ordnance Survey. S/he will ensure appropriate and effective Board sub-committees exist and will, in consultation with the Chief Executive, determine Board meeting frequency and agenda. A key role is to ensure that all Non-executive Directors are effective in the support and challenge they provide to the Executive team.

The Shareholder Executive, working for the Department for Communities and Local Government, takes a close interest in the performance management of Ordnance Survey. The Chair is expected to work constructively with senior Shareholder Executive officials.

The candidate is expected to have the usual abilities concomitant with these jobs – bulging address book, Cabinet ministers and heads of industry mobile numbers on speed dial, ability to leap tall buildings and to cure sick animals with their magic touch, that sort of thing.

On its face, it doesn’t look like the successful candidate will die from overwork: three or four days a month, which earns an annual remuneration of £40,000 – £50,000. But of course that would be to ignore how important this job will be. We’re looking forward to seeing who it is.

Obviously, if you’ve applied, do feel free to share the experience..

The postcode debate, summed up beautifully on Tom Watson’s blog

Monday, July 7th, 2008

Tom Watson MP, the Cabinet Office minister who is also the political wing of the Power Of Information taskforce, started an interesting debate on his blog, when he noted a comment by Simon Dickson about the usefulness of the Postcode Address File.

He mused, “I’m going to spend some time trying to understand just why [PAF] can’t be available for free or at marginal cost. Feel free to air your views in the comment section.”

And boy, did people air their comments. It’s worth reading in full, but I think the prize – at least the Free Our Data prize for stating the value of the free data model – goes to Greg, who (in a long and well-argued comment) sums up by responding to “Mitch” (an earlier commenter who had worked in the Royal Mail on updating the PAF):

The points you make, Mitch, are unfortunately so reminisecent of the innovation-stifling opinions of inward-facing bureaucrats which have been such a major contributor to Britain’s loss of economic advantage over the years. Examples which are now so clear include the fact that we invented public-key encryption long before the US, but kept it a government secret rather than using it to gain an edge in commerce; or that Frank Whittle invented the jet engine only to find that closed-minded bureaucrats couldn’t see it working. Bureaucrats are rarely the best people to judge whether something has a place in propelling innovation and competitiveness. The fact that there’s so much energy on my side of the postcodes debate [arguing to make it available for free] says it all.

Mitch; you should be proud that you worked on a world-leading data source. It’s just such a shame its wings are crippled by its owners.

We love it when people state the benefits so clearly. The whole thread is worth reading, though, for the vigour of the arguments on both sides.

And now, OPSI sets up an “unlock that data” channel

Monday, July 7th, 2008

The Office of Public Sector Information (OPSI) goes from strength to strength. After its chief Carol Tullo spoke out in Europe about the importance of greater access to data, OPSI has set up a web page where you can request data sets you want released:

As the regulator for public sector information re-use, we know that people can encounter problems from time to time getting hold of the information they need in the formats they want. Difficulties can include problems with charging, licensing or the data standards that public sector information is provided in.

These problems aren’t about access (which is dealt with under Freedom of Information legislation), but all the other issues which can occur when you want to do something with public sector information – copy it, remix it with other data or add value and republish it. If you are trying to re-use some public sector information, but the data you need is locked-up, this service is for you.

How it works:

  1. You describe the public sector information asset you want unlocked for re-use, and post a request to the service. We’ll check through your request and if it’s OK (e.g. not a Freedom of Information request) we’ll post it here.
  2. Others can see your request and support it, either by adding a comment or by voting. The more support a request has, the better the chances of unlocking the information you want to re-use.
  3. We’ll contact the public sector information holder and see what can be done to unlock the information for re-use. To keep things simple, if the problem relates to an issue specifically covered by the Re-use of Public Sector Information Regulations or the Information Fair Trader Scheme, we’ll treat it accordingly – so you won’t need to make a separate complaint. We’ll post back our findings here.

And there’s already one request in there, for access to OS electoral boundary details, which I recall is an issue that comes up again and again – it was certainly mentioned at the RSA/Free Our Data debate nearly two years ago.

The problem, as detailed by “Matthew”:

I find it odd that if I want to know the actual boundary of the ward or constituency I am in (co-ordinates, not just an image), I have to pay Ordnance Survey lots of money for their Boundary-Line product. I would have thought that, given it’s quite important to know which MP or councillors I’m going to have the option of electing, that this information should be freely available as part of a healthy democracy; it’s compiled by the various publicly funded Boundary Commissions/Committees as far as I know.

His ideal solution:

I think the actual data rather than just images of the boundaries should be available, so that people can create things using the data – you can’t do anything with images besides display them. For example, I can’t create a Google map (using their My Maps feature) of my ward marking on where and when councillors hold their surgeries, and other local amenities. I can’t create an application that asks people to select where they live on a map and it tell them if their Parliamentary constituency will be changing at the next general election, what it’s changing to, and what difference that makes to them.

I am aware of the election-maps.co.uk website, but this is extremely hard to use – you have to know the name of your area before you can enter a postcode, you can’t look up by e.g. ward name, and it only provides images of the boundaries.

More power to his, and OPSI’s, elbow.

This is all terrifically encouraging, especially along with the Show Us A Better Way competition using government data for imaginative (and perhaps commercial) mashups. Have you got your entry in yet?

England and Wales schools database: available here in SQL format

Friday, July 4th, 2008

As part of the government’s Show Us A Better Way competition, it has made available all sorts of databases and datasets and APIs that haven’t previously been available – such as the list of all the schools in England and Wales.

Our only quibble with the latter was that it was only provided in Excel format – which as one commenter points out is a proprietary format (though free programs like OpenOffice will open it), and anyway to really begin doing useful things with such data you need to stuff it into a database; which calls for SQL format.

Never fear, Free Our Data is here. We’ve imported the data from the Excel file into a MySQL database and exported it as an SQL file which has all the required CREATE TABLE commands, with the data.

Grab a copy.
To make sure you’ve got the correct version (in case it gets copied and used elsewhere):

the MD5 checksum of the zip file is 3f46d71d84f6047ee0162d12a9456901

and of the SQL file itself (once unzipped) is 1021643b2c1c71773f20c7a4fbd1b8e1 .

The government wants you to show it a better way (and will pay £20,000)

Wednesday, July 2nd, 2008

As an idea, Free Our Data has now begun to gain some traction in government – and even, as the whole saga over crime mapping in London shows, with the Conservatives.

Now the Power Of Information taskforce, which includes Tom Watson, the Cabinet Office minister we interviewed a while back, has started a new initiative (though competition is just as good a word) at Showusabetterway.com:

Ever been frustrated that you can’t find out something that ought to be easy to find? Ever been baffled by league tables or ‘performance indicators’? Do you think that better use of public information could improve health, education, justice or society at large?

The UK Government wants to hear your ideas for new products that could improve the way public information is communicated. The Power of Information Taskforce is running a competition on the Government’s behalf, and we have a £20,000 prize fund to develop the best ideas to the next level.

To show they are serious, the Government is making available gigabytes of new or previously invisible public information especially for people to use in this competition.

And in case you wondered if it involves puttings CDs from HMRC into envelopes..

Rest assured, this competition does not include personal information about people.

There is a set of examples – such as crime mapping, Fixmystreet, and a pointer to others such as farmsubsidy.org (which “compiles obscure information about subsidies under the Common Agricultural Policy and puts it in one place, to make it much easier to see where farm subsidies are going across Europe.”)

The team signs off with a flourish:

We’re confident that you’ll have more and better ideas than we ever will. You don’t have to have any technical knowledge, nor any money, just a good idea, and 5 minutes spare to enter the competition.

There’s already a list of submitted ideas, which includes a Road Works API, FixMyTransport (“where people with shared public transport problems could come together to get things improved”), Rate My Bus, and others.

Come on, people – tell us your ideas, then go and enter them on the site (or vice versa) and win the funding. It would be fantastic if a Guardian Tech reader could win this.

Update: just to point to some of the resources you can use (among many, many, many): mapping information from the Ordnance Survey, medical information from the NHS, neighbourhood statistics from the Office for National Statistics and a carbon calculator from the Department for Environment, Food and Rural Affairs (Defra). And these are in API form, which means they’re all ready for mashup goodness.

Although not, it seems, the Postcode Address File (though the Edubase file, with school addresses, does include postcodes).