A comparison of my analysis of OpenStreetMap (OSM) quality evaluation to other examples of quality evaluation brings up some core issues about the nature of the new GeoWeb and the use of traditional sources. The examples that I’m referring to are from Etienne Cherdlu’s SOTM 2007 ‘OSM and the art of bicycle maintenance’, Dair Grant’s comparison of OSM to Google Maps and reality, Ed Johnson’s analysis this summer and Steven Feldman’s brief evaluation in Highgate.

Meridian 2 and OSM in the area of Highgate, North London

Meridian 2 and OSM in the area of Highgate, North London

The first observation is of the importance and abundance of well georeferenced, vector-derived public mapping sites, which make several of these comparisons possible (Chedlu, Dair and Feldman). The previous generation of stylised street maps is not readily available for a comparison. In addition to the availability, the ease with which they can be mashed-up is also a significant enabling factor. Without this comparable geographical information, the evaluation would be much more difficult.

Secondly, when a public mapping website was used, it was Google Maps. If Microsoft’s Virtual Earth had also been used, it would arguably allow a three-way comparison as the Microsoft site uses Navteq information, while Google uses TeleAtlas information. Using Ordnance Survey (OS) OpenSpace for comparison is also a natural candidate. Was this familiarity that led to the selection of Google Maps? Or is it because the method of comparison is visual inspection, so adding a third source makes it more difficult? Notice that Google has the cachet of being a correct depiction of reality, which Etienne, Dair and Bob Barr demonstrated not to be the case!

Thirdly, and most significantly, only when vector data was used – in our comparison and in parts of what Ed Johnson has done – a comprehensive analysis of large areas became possible. This shows the important aspect of the role of formats in the GeoWeb – raster is fabulous for the delivery of cartographic representations, but it is a vector that is suitable for analytical and computational analysis. Only OSM allows the user easy download of vector data – no other mass provider of public mapping does.

Finally, there is the issue of access to information, tools and knowledge. As a team that works at a leading research university (UCL), I and the people who worked with me got easy access to detailed vector datasets and the OS 1:10,000 raster. We also have at our disposal multiple GIS packages, so we can use whichever one performs the task with the least effort. The other comparisons had to rely on publically available datasets and software. In such unequal conditions, it is not surprising that I will argue that the comparison that we carried out is more robust and consistent. The issue that is coming up here is the balance between amateurs and experts, which is quite central to Web 2.0 in general. Should my analysis be more trusted than those of Dair’s or Etienne’s, both of whom who are very active in OSM? Does Steven’s familiarity with Highgate, which is greater than mine, make him more of an expert in that area than my consistent application of analysis?

I think that the answer is not clear cut; academic knowledge entails the consistent scrutiny of the data, and I do have the access and the training to conduct a very detailed geographical information quality assessment. In addition, my first job in 1988 was in geographical data collection and GIS development, so I also have professional knowledge in this area. Yet, local knowledge is just as valuable in a specific area and is much better than a mechanical, automatic evaluation. So what is happening is an exchange of knowledge, methods and experiences between the two sides in which both, I hope, can benefit.

The Metropolitan Police Authority has released a beta version of their new Crime Mapping application, showing the generalised level of crime (burglary, robbery and vehicle) for Lower-level Super Output Areas (LSOAs). The application uses generalised boundaries of LSOAs , and use a clear classification of the level of crime. Interestingly, the Show Us a Better Way website includes several suggestions for crime mapping – so there is an ongoing public interest.

This is not surprising, based on my own experience with CamStats, which was developed in collaboration between me, Kate Jones and Dave Ashby for Camden Police in late 2003, with the website operating from early 2004 until late 2007.

As you can see from the slideshow above, the information that CamStats provided is richer than what is available today. CamStats was based on static maps, and was very easy to produce – we designed it so a team administrator (with no GIS skills) could compile monthly and annual statistics simply by copying a data file to a processing machine, and then clicking one button in Mapinfo Professional which called Mapbasic, Perl scripts and other utilities to create, process and map the data and compile the HTML pages for the website into one zip file. All the user had to do was transfer the zip file to the Met web team who easily updated the webserver by unzipping the files. The fact that it was running for three years without any request for support is something that Kate and I are justifiably proud of.

Notice that CamStats provided options to see different geographical units, different forms of visualisation and to view the information in tabular and chart forms. Users could even download the aggregate counts for each area to compile there own reports. This was particularly useful for a number of community groups in Camden.

There is no question that the use of Google Maps, which provide context for the statistics is a huge usability improvement over our implementation. However, it will be interesting to see how long it will take the Met team to reach the functionality and ease of use CamStats provided …

The UrbanBuzz team that recorded some of the community showcase presentations during the Mapping Change for Sustainable Communities event in June, have now released the videos on YouTube. The videos are showing the posters and explanations for our work in Marks Gate, Pepys Estate and Hackney Wick. The presentations are by members of the communities, with some help from the project team.

This is the video for Marks Gate, where the focus was on community perceptions about their environment and how it can improved:

The next one shows the noise mapping work in the Pepys Estate (and I had the pleasure of assisting Caroline to explain the mapping):

And the final one shows the historical mapping in Hackney Wick:

The three videos give a good overview of the community mapping projects that were carried out within Mapping Change for Sustainable Communities, and the Environmental Justice projects with London 21 and London Sustainability Exchange .

The Environment Agency has released the new version of their ‘What’s In Your Back Yard?’ application, which allows you to find out about pollution incidents, water quality, and flood risk in your area. Similar to the rest of the website, it is still designed for 800×600 screen resolution, although not really – the legend spills to the bottom and requires scrolling down.

The application takes a lot of the new interaction metaphors that we’ve become accustomed to in Web Mapping 2.0 – slippy maps that can be panned easily, obtaining information by point and click on the map – unlike the cumbersome way in which it was handled in the previous version. There are also other improvements such as more data.

Yet, there remain some of the most common mistakes that I have pointed to in the past – the maps are still small and too generalised to really see what is in your area, they still have the weird scale statement at the top of the map (what does it means that the scale is 1:650,000?).

Also, notice that the slippy map implementation is such that, after you move the map, it disappears and then reappears – it takes about 3 seconds which is enough to make it difficult to relocate yourself on the map. That’s actually not a good implementation – the whole point of slippy maps is that only a small area of the screen is being refreshed, making it easy for users to keep the image in their short-term memory.
The last aspect that I would expect in such an application released in 2008 is the ability to download the data from the application (see the Power of Information report) or to have a well-controlled API as in OS OpenSpace.

So, it’s a move in the right direction, but a long way to go …

OSM quality evaluation

7 August, 2008

In the past year I have worked on the evaluation of OpenStreetMap data. I was helped by Patrick Weber, Claire Ellul, and especially Naureen Zulfiqar who carried out part of the analysis of motorways. The OSM data was compared against Ordnance Survey Meridian 2 and the 1:10,000 raster as they have enough similarity to justify a comparison. Now, as the fourth birthday of OSM is approaching, it is good time to evaluate what was achieved. The analysis shows that, where OSM was collected by several users and benefited from some quality assurance, the quality of the data is comparable and can be fit for many applications. The positional accuracy is about 6 metres, which is expected for the data collection methods that are used in OSM. The comparison of motorways shows about 80% overlap between OSM and OS – but more research is required. The challenges are the many areas that are not covered – currently, OSM has good coverage for only 25% of the land area of England. In addition, in areas that are covered well, quality assurance procedures should be considered – and I’m sure that the OSM crowd will find great ways to make these procedures fun. OSM also doesn’t covered areas at the bottom of the deprivation scale as well as it covers areas that are wealthier. The map below shows the quality of coverage of the two datasets for England, with blue marking areas where OSM coverage is good and red where it is poor.

Difference between OSM and OS Meridian for England

Difference between OSM and OS Meridian for England

The full report is available here, and if someone is willing to sponsor further analysis – please get in touch!

The paper itself have been published – Haklay M, 2010, “How good is volunteered geographical information? A comparative study of OpenStreetMap and Ordnance Survey datasets” Environment and Planning B: Planning and Design 37(4) 682 – 703

Follow

Get every new post delivered to your Inbox.

Join 2,697 other followers