The British Museum Test for public mapping websites

Back in 2005, when I worked with Artemis Skarlatidou on an evaluation of public mapping websites, we came up with a simple test to check how well these search sites perform: Can a tourist find a famous landmark easily?

The reasoning behind raising this question was that tourists are an obvious group of users of public mapping sites such as Multimap, MapQuest, Yahoo! Maps, Microsoft’s Virtual Earth or Google Maps. Market research information presented by Vincent Tao from Microsoft in a seminar a year ago confirmed this assumption.

During the usability evaluation, we gave the participants the instruction ‘Locate the following place on the map: British Museum: Great Russell Street, London, WC1B 3DG’. Not surprising, those participants who started with the postcode found the information quickly, but about a third typed ‘British Museum, London’. While our participants were London’s residents and were used to postcodes as a means of stating an address precisely, a more realistic expectation from tourists is that they would not use postcodes when searching for a landmark.

In the summer of 2005 when we ran the test, the new generation of public mapping websites (such as Google Maps and Microsoft Virtual Earth) performed especially bad.
The most amusing result came from Google Maps, pointing to Crewe as the location of the British Museum (!).

The most simple usability test for a public mapping site that came out of this experiment is the ‘British Museum Test’: find the 10 top tourist attractions in a city/country and check if the search engine can find them. Here is how it works for London:

The official Visit London site suggests the following top attractions: Tate Modern, British Museum, National Gallery, Natural History Museum, the British Airways London Eye, Science Museum, the Victoria & Albert Museum (V&A Museum), the Tower of London, St Paul’s Cathedral and the National Portrait Gallery.

Now, we can run the test by typing the name of the attraction in the search box of public mapping sites. As an example, here I’ve used Yahoo! Maps, Google Maps, Microsoft’s Virtual Earth and Multimap. With all these sites I’ve imitated a potential tourist – I’ve accessed the international site (e.g. maps.google.com) and panned the map to the UK, and then typed the query. The results are:

Attraction (search term used)	Yahoo!	Google	Microsoft	Multimap
Tate Modern	Found and zoomed	Found and zoomed	Found and zoomed	Found and zoomed
British Museum	Found and zoomed	Found as part of a list	Found and zoomed	Found and zoomed
National Gallery	Found and zoomed	Found as part of a list	Found and zoomed	Found as part of a list (twice!)
Natural History Museum	Failed	Found as part of a list	Found and zoomed	Found and zoomed
British Airways London Eye (commonly abbreviated to London Eye)	Failed on the full name, found and zoomed on the common abbreviation	Found as part of a list, failed on the common abbreviation	Failed on the full name, found and zoomed on the common abbreviation	Failed on the full name, found and zoomed on the common abbreviation
Science Museum	Found and zoomed	Found as part of a list	Found and zoomed	Found and zoomed
The Victoria & Albert Museum (commonly abbreviated to V&A Museum)	Found and zoomed on both	Found and zoomed, but failed on the common abbreviation	Found and zoomed, but failed on the common abbreviation	Found and zoomed, but the common abbreviation zoomed on Slough (!)
The Tower of London	Found and zoomed	Found and zoomed	Found and zoomed (failed if ‘the’ included in the search)	Found and zoomed
St Paul’s Cathedral	Found and zoomed	Found and zoomed	Found as part of a list	Failed
National Portrait Gallery	Failed (zoomed to the one in Washington DC)	Found and zoomed	Found and zoomed	Found and zoomed

Notice that none of these search engines managed to pass the test on all the top ten attractions, which are visited by millions every year. There is a good reason for this – geographical search is not a trivial matter and the semantics of place names can be quite tricky (for example, if you look at a map of Ireland and the UK, there are two National Galleries).

On the plus side, I can note that search engines are improving. At the end of 2005 and for most of 2006 the failure rate was much higher. I used the image above in several presentations and have run the ‘British Museum Test’ several times since then, with improved results in every run.

The natural caveat is that I don’t have access to the server logs of the search engines and, therefore, can’t say that the test really reflects the patterns of use. It would be very interesting to have Google Maps Hot Trends or to see it for other search engines. Even without access to the search logs though, the test reveals certain aspects in the way that information is searched and presented and is useful in understanding how good the search engines are in running geographical queries.

By a simple variation of the test you can see how tolerant an engine is for spelling errors, and which one you should use when guests visit your city and you’d like to help them in finding their way around. It is also an indication of the general ability of the search engine to find places. You can run your own test on your city fairly quickly – it will be interesting to compare the results!

For me, Microsoft Virtual Earth is, today, the best one for tourists, though it should improve the handling of spelling errors…