Po Ve Sham - Muki Haklay’s personal blog

Back in 2005, when I worked with Artemis Skarlatidou on an evaluation of public mapping websites, we came up with a simple test to check how well these search sites perform: Can a tourist find a famous landmark easily?

The reasoning behind raising this question was that tourists are an obvious group of users of public mapping sites such as Multimap, MapQuest, Yahoo! Maps, Microsoft’s Virtual Earth or Google Maps. Market research information presented by Vincent Tao from Microsoft in a seminar a year ago confirmed this assumption.

During the usability evaluation, we gave the participants the instruction ‘Locate the following place on the map: British Museum: Great Russell Street, London, WC1B 3DG’. Not surprising, those participants who started with the postcode found the information quickly, but about a third typed ‘British Museum, London’. While our participants were London’s residents and were used to postcodes as a means of stating an address precisely, a more realistic expectation from tourists is that they would not use postcodes when searching for a landmark.

In the summer of 2005 when we ran the test, the new generation of public mapping websites (such as Google Maps and Microsoft Virtual Earth) performed especially bad.
The most amusing result came from Google Maps, pointing to Crewe as the location of the British Museum (!).

The most simple usability test for a public mapping site that came out of this experiment is the ‘British Museum Test’: find the 10 top tourist attractions in a city/country and check if the search engine can find them. Here is how it works for London:

The official Visit London site suggests the following top attractions: Tate Modern, British Museum, National Gallery, Natural History Museum, the British Airways London Eye, Science Museum, the Victoria & Albert Museum (V&A Museum), the Tower of London, St Paul’s Cathedral and the National Portrait Gallery.

Now, we can run the test by typing the name of the attraction in the search box of public mapping sites. As an example, here I’ve used Yahoo! Maps, Google Maps, Microsoft’s Virtual Earth and Multimap. With all these sites I’ve imitated a potential tourist – I’ve accessed the international site (e.g. maps.google.com) and panned the map to the UK, and then typed the query. The results are:

Notice that none of these search engines managed to pass the test on all the top ten attractions, which are visited by millions every year. There is a good reason for this – geographical search is not a trivial matter and the semantics of place names can be quite tricky (for example, if you look at a map of Ireland and the UK, there are two National Galleries).

On the plus side, I can note that search engines are improving. At the end of 2005 and for most of 2006 the failure rate was much higher. I used the image above in several presentations and have run the ‘British Museum Test’ several times since then, with improved results in every run.

The natural caveat is that I don’t have access to the server logs of the search engines and, therefore, can’t say that the test really reflects the patterns of use. It would be very interesting to have Google Maps Hot Trends or to see it for other search engines. Even without access to the search logs though, the test reveals certain aspects in the way that information is searched and presented and is useful in understanding how good the search engines are in running geographical queries.

By a simple variation of the test you can see how tolerant an engine is for spelling errors, and which one you should use when guests visit your city and you’d like to help them in finding their way around. It is also an indication of the general ability of the search engine to find places. You can run your own test on your city fairly quickly – it will be interesting to compare the results!

For me, Microsoft Virtual Earth is, today, the best one for tourists, though it should improve the handling of spelling errors…

As part of the Mapping Change for Sustainable Communities project, we organised the first workshop in the Royal Docks area, at the Sunborn Yacht Hotel last Saturday (27/10). The workshop was very successful and, as I usually do in these workshops, I start with ‘what mapping information can we find on the WWW about your locality’. I’ve been doing it now for about 7 or 8 years, but during the period, the Environment Agency’s Pollution Inventory maps never failed me as an example for technocratic dissemination of information which is not helping the people on the ground.

I find that, in all these workshops with people from many communities across London, very few knew about the Environment Agency information, let alone ever accessed it independently had. As the participants are usually from community or environmental interest groups, they are interested in the information – they just don’t know where to find it. I associate this lack of awareness with the fact that users find the information unfriendly and unhelpful, so there is no ‘word of mouth’ effect that leads to more use of the site. As someone in our Saturday workshop declared, ‘these maps are not written in community language or for community use’ – yet, they tick all the boxes of the Aarhus convention…

To understand what’s wrong, see the image below, which provides a view of the site on an average monitor (1024×768):

The header area is so big that all that is left is a fairly small area for the map.

Furthermore, as the full image of the page shows, the map is supposed to offer several layers of pollution data (on the right-hand side) but, as many of the layers include point data about the same site – all using the same symbols which overlap one another – the user can’t see if, in a given location, there is information from multiple categories.

The area of the map is very small (less than 400×400 pixels), and people find it very difficult to locate where they are or where the postcode is that they have selected in relation to the information on the map. Zooming in to the largest scale, or at any stage during the process, the system will run a query and provide information about the specific location only if the ‘learn more’ option is selected. Even as a more frequent user, I fail to click on this option and find the interaction with the system frustrating.

This site demonstrates that the Aarhus model of access to information, which is ‘we’ll build it and they’ll come’, is not sufficient and that a more user-centred approach is required to achieve public access to environmental information.