One of the surprises of the Ordnance Survey OpenData release at the beginning of April was the inclusion of the Code-Point Open dataset, which lists the location of all postcodes in England, Wales and Scotland. This was clearly a very important dataset because of the way postcode geography drives many services and activities in the UK. Before the release, the costs of using postcodes in geographical analysis were prohibitive for many small organisations.
So how usable is this free Code-Point data? The principle of ‘do not look a gift horse in the mouth’ doesn’t apply here. The whole point of releasing the data is to make it as useful as possible to encourage innovation, so it should be made available in a way that makes it easy to reuse. I evaluated it while analysing a dataset of 11,000 volunteers’ postcodes that I received from a third sector organisation.
The download process is excellent and easy, apart from the fact that there is no clear and short description of the products in a non-technical manner next to each product. To find a description, you need to go to the product page – so you are at least 2 clicks away from the product details. It would be better to have a link from each product and include a brief description in the download page. We will see in a second why this is important…
The next step was the download itself and the opening of the zip file, which was clear and easy. There is an oddity with all Ordnance Survey data that they have a redundant sub-directory in them – so in this case the data resides under \codepo_gb\Code-Point Open\ . The fact that the files is broken up into postcode area instead of one big file of 157MB is fine, but it can be helpful to remind users that they can concatenate files using simple commands – this is especially necessary to less tech-savvy users. So an explanation for Windows users that you can open the Command window using ‘cmd.exe’ and run ‘type a.csv b.csv > common.csv’ can save some people plenty of time.
But the real unpleasant surprise was that nowhere in the downloaded package is there a description of the fields in the files! So you open the files and need to figure out what the fields are. The user manual is hides 4 clicks away from the download page and luckily I knew that the ‘user manual’ is stored under ‘technical information’ on the product page, which is not that obvious at first visit. Why not deliver the user manual with the product ?!? The Doc directory is an obvious place to store it.
The user manual reveals that there are 19 fields in the file, of which 9 (half!) are ‘not available in Code-Point Open’ – so why are they delivered? After figuring out the fields, I created a single line that can be attached to the files before importing them to a GIS:
Postcode,Positional Quality,PR Delete,TP Delete,DQ Delete,RP Delete,BP Delete,PD Delete,MP Delete,UM Delete,Easting,Northing,Country,Regional Health Authority,Health Authority,County,District,Ward,LS Delete.
Of course, all the fields with ‘Delete’ in the name mean that they should be deleted once imported.
Interestingly, once you delete these fields, the total size of Code-Point Open drops from 157MB to 91MB – which means that it can save the Ordnance Survey bandwidth and carbon emissions by making the file smaller.
Another interesting point is that the user manual includes detailed instructions on how to change the postcode to a ‘single spaced postcode’. The instructions are for Excel, Mapinfo and ArcGIS. This is the type of information that can help end-users start using the data faster. Finally, you can use this wonderful information to create lovely maps.
All these problems are minor, apart from the description of the fields which is a major usability error. Similar analysis can be carried out for any of the Ordnance Survey datasets, to ensure that they are useful to their users. There are some easy improvements, such as including the user manual with the distribution, and I’m sure that, over time, the team at the Ordnance Survey will find the time to sort these issues.
Po Ve Sham - Muki Haklay's personal blog 
Interesting article. I think what has excited me more is the ONS releasing NSPD Open: http://www.ons.gov.uk/about-statistics/geography/products/geog-products-postcode/nspd/index.html
This does however exclude any PAF content 😦 Also, there is a charge of £200 a year for 4 update disks. Someone will put this on a web-server at some point (which you can do legally!).
The OAC User Group plan to use this data to build some coding tools which are almost ready to launch.
LikeLike
It will be interesting to see if NSPD will be released in an easy to use form – and by that I mean usable for people that are not full time researchers. The provision of coding tools, done properly, can make life even easier – interfaces like http://www.uk-postcodes.com/ are helpful for people who want to utilise postcodes without dealing with the data itself, but the post is more about the user experience when you need the data itself.
LikeLike
Muki,
Good post and certainly I agree with you on this. A couple of days after Code Point Open came out I did some work to covert it to kml and cross reference for Yahoo’s WOEID’s. I had a similar experience, my steps are detailed in this blog post:
http://blogs.snowflakesoftware.com/news/2010/04/first-derived-dataset-from-os-open-data.htm
The string manipulation of the Post Code is a pain, I would imagine that the single spaced code is the most common use-case so why not clean it up from the start.
A lot of the docs around Open Data could be better (Boundary Line is another that needs some TLC) but to be fair to OS you can understand why there’s some shortcuts especially given the timeframe they had to get the data out. Hopefully the docs around Open Data will improve over time, so I’ll add my vote to yours for a refresh sometime soon.
Ian
LikeLike
Hi Muki, Great post, thanks for writing it. I work in the OS press office and I’ll make sure the OS OpenData and Code Point Open teams see your feedback. Constructive criticism is really important, so thanks again.
LikeLike
I’ve just created a Wiki page on data.gov.uk about OS Code-Point following their suggested structure, and bringing in some information from the OpenStreetMap wiki. Also linked to this blog post.
The wiki is open access, although you have to register, and I encountered a few bugs in the registration process actually. Even so, I’d suggest that is a good place to pool our collective knowledge of these data sets, and link to example uses and alternative data formats.
LikeLike