The British Museum Test for public mapping websites

Back in 2005, when I worked with Artemis Skarlatidou on an evaluation of public mapping websites, we came up with a simple test to check how well these search sites perform: Can a tourist find a famous landmark easily?

The reasoning behind raising this question was that tourists are an obvious group of users of public mapping sites such as Multimap, MapQuest, Yahoo! Maps, Microsoft’s Virtual Earth or Google Maps. Market research information presented by Vincent Tao from Microsoft in a seminar a year ago confirmed this assumption.

During the usability evaluation, we gave the participants the instruction ‘Locate the following place on the map: British Museum: Great Russell Street, London, WC1B 3DG’. Not surprising, those participants who started with the postcode found the information quickly, but about a third typed ‘British Museum, London. While our participants were London’s residents and were used to postcodes as a means of stating an address precisely, a more realistic expectation from tourists is that they would not use postcodes when searching for a landmark.

In the summer of 2005 when we ran the test, the new generation of public mapping websites (such as Google Maps and Microsoft Virtual Earth) performed especially bad.
The most amusing result came from Google Maps, pointing to Crewe as the location of the British Museum (!).
Google - British Museum in Crewe

The most simple usability test for a public mapping site that came out of this experiment is the ‘British Museum Test’: find the 10 top tourist attractions in a city/country and check if the search engine can find them. Here is how it works for London:

The official Visit London site suggests the following top attractions: Tate Modern, British Museum, National Gallery, Natural History Museum, the British Airways London Eye, Science Museum, the Victoria & Albert Museum (V&A Museum), the Tower of London, St Paul’s Cathedral and the National Portrait Gallery.

Now, we can run the test by typing the name of the attraction in the search box of public mapping sites. As an example, here I’ve used Yahoo! Maps, Google Maps, Microsoft’s Virtual Earth and Multimap. With all these sites I’ve imitated a potential tourist – I’ve accessed the international site (e.g. maps.google.com) and panned the map to the UK, and then typed the query. The results are:

Attraction (search term used) Yahoo! Google Microsoft Multimap
Tate Modern Found and zoomed Found and zoomed Found and zoomed Found and zoomed
British Museum Found and zoomed Found as part of a list Found and zoomed Found and zoomed
National Gallery Found and zoomed Found as part of a list Found and zoomed Found as part of a list (twice!)
Natural History Museum Failed Found as part of a list Found and zoomed Found and zoomed
British Airways London Eye (commonly abbreviated to London Eye) Failed on the full name, found and zoomed on the common abbreviation Found as part of a list, failed on the common abbreviation Failed on the full name, found and zoomed on the common abbreviation Failed on the full name, found and zoomed on the common abbreviation
Science Museum Found and zoomed Found as part of a list Found and zoomed Found and zoomed
The Victoria & Albert Museum (commonly abbreviated to V&A Museum) Found and zoomed on both Found and zoomed, but failed on the common abbreviation Found and zoomed, but failed on the common abbreviation Found and zoomed, but the common abbreviation zoomed on Slough (!)
The Tower of London Found and zoomed Found and zoomed Found and zoomed (failed if ‘the’ included in the search) Found and zoomed
St Paul’s Cathedral Found and zoomed Found and zoomed Found as part of a list Failed
National Portrait Gallery Failed (zoomed to the one in Washington DC) Found and zoomed Found and zoomed Found and zoomed

Notice that none of these search engines managed to pass the test on all the top ten attractions, which are visited by millions every year. There is a good reason for this – geographical search is not a trivial matter and the semantics of place names can be quite tricky (for example, if you look at a map of Ireland and the UK, there are two National Galleries).

On the plus side, I can note that search engines are improving. At the end of 2005 and for most of 2006 the failure rate was much higher. I used the image above in several presentations and have run the ‘British Museum Test’ several times since then, with improved results in every run.

The natural caveat is that I don’t have access to the server logs of the search engines and, therefore, can’t say that the test really reflects the patterns of use. It would be very interesting to have Google Maps Hot Trends or to see it for other search engines. Even without access to the search logs though, the test reveals certain aspects in the way that information is searched and presented and is useful in understanding how good the search engines are in running geographical queries.

By a simple variation of the test you can see how tolerant an engine is for spelling errors, and which one you should use when guests visit your city and you’d like to help them in finding their way around. It is also an indication of the general ability of the search engine to find places. You can run your own test on your city fairly quickly – it will be interesting to compare the results!

For me, Microsoft Virtual Earth is, today, the best one for tourists, though it should improve the handling of spelling errors…

Published by

mukih

Professor of GIScience, University College London

3 thoughts on “The British Museum Test for public mapping websites”

  1. Hi Muki!
    Your post was widely read here at Multimap, and one of my colleagues here, Julian Skeels, took it upon himself to add some common landmarks to geonames.org on Saturday. These landmarks were on Multimap within 24 hours (http://www.multimap.com/maps/?hloc=GB|V~A%20Museum). And of course they are now available to the rest of the Geonames community and to Neogeographers everywhere. If you have a moment to re-test we’d love to know if we do better now!
    And of course, if you or anybody else notices a missing landmark you can just add it at geonames.org and it will be available on Multimap and many other sites within 24 hours.

    Cheers
    Sean

  2. Great to see how quickly companies like Multimap resolve problems like the ones that I have discussed in the post above and improve their public mapping products – a clear advantage of the Web platform where new versions can be deployed quickly and efficiently.
    Muki

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s