Identifying success factors in crowdsourced geographic information use in government

GFDRRA few weeks ago, the Global Facility for Disaster Reduction and Recovery (GFDRR), published an update for the report from 2014 on the use of crowdsourced geographic information in government. The 2014 report was very successful – it has been downloaded almost 1,800 times from 41 countries around the world in about 3 years (with more than 40 academic references) which showed the interests of researchers and policymakers alike and outlined its usability. On the base of it, it was pleasing to be approached by GFDRR about a year ago, with a request to update it.

In preparation for this update, we sought comments and reviews from experts and people who used the report regarding possible improvements and amendments. This feedback helped to surface that the seven key factors highlighted by the first report as the ones that shaped the use of VGI in government (namely: incentives, aims, stakeholders, engagement, technical aspects, success factors, and problems) have developed both independently and in cross-cutting modes and today there is a new reality for the use of VGI in government.

Luckily, in the time between the first report and the beginning of the new project, I learned about Qualitative Comparative Analysis (QCA) in the Giving Time event and therefore we added Matt Ryan to our team to help us with the analysis. QCA allowed us to take 50 cases, have an intensive face to face team workshop in June last year to code all the cases and agree on the way we create the input to QCA. This helped us in creating multiple models that provide us with an analysis of the success factors that help explain the cases that we deemed successful. We have used the fuzzy logic version of QCA, which allowed a more nuanced analysis.

Finally, in order to make the report accessible, we created a short version, which provides a policy brief to the success factors, and then the full report with the description of each case study.

It was pleasure working with the excellent team of researchers that worked on this report: Vyron Antoniou, Hellenic Army Geographic Directorate, Sofia Basiouka, Hellenic Ministry of Culture and Sport, Robert Soden, World Bank, Global Facility for Disaster Reduction & Recovery (GFDRR), Vivien Deparday, World Bank, Global Facility for Disaster Reduction & Recovery (GFDRR). Matthew Ryan, University of Southampton, and Peter Mooney, National University of Ireland, Maynooth. We were especially lucky to be helped by Madeleine Hatfield of Yellowback Publishing who helped us in editing the report and making it better structured and much more readable.

The full report, which is titled “Identifying success factors in crowdsourced geographic information use in government” is available here.

And the Policy Brief is available here. 

Advertisements

Citizen Science & Scientific Crowdsourcing – week 5 – Data quality

This week, in the “Introduction to Citizen Science & Scientific Crowdsourcing“, our focus was on data management, to complete the first part of the course (the second part starts in a week’s time since we have a mid-term “Reading Week” at UCL).

The part that I’ve enjoyed most in developing was the segment that addresses the data quality concerns that are frequently raised about citizen science and geographic crowdsourcing. Here are the slides from this segment, and below them a rationale for the content and detailed notes

I’ve written a lot on this blog about data quality and in many talks that I gave about citizen science and crowdsourced geographic information, the question about data quality is the first one to come up. It is a valid question, and it had led to useful research – for example on OpenStreetMap and I recall the early conversations, 10 years ago, during a journey to the Association for Geographic Information (AGI) conference about the quality and the longevity potential of OSM.

However, when you are being asked the same question again, and again, and again, at some point, you start considering “why am I being asked this question?”. Especially when you know that it’s been over 10 years since it was demonstrated that the quality is beyond “good enough”, and that there are over 50 papers on citizen science quality. So why is the problem so persistent?

Therefore, the purpose of the segment was to explain the concerns about citizen science data quality and their origin, then to explain a core misunderstanding (that the same quality assessment methods that are used in “scarcity” conditions work in “abundance” conditions), and then cover the main approaches to ensure quality (based on my article for the international encyclopedia of geography). The aim is to equip the students with a suitable explanation on why you need to approach citizen science projects differently, and then to inform them of the available methods. Quite a lot for 10 minutes!

So here are the notes from the slides:

[Slide 1] When it comes to citizen science, it is very common to hear suggestions that the data is not good enough and that volunteers cannot collect data at a good quality, because unlike trained researchers, they don’t understand who they are – a perception that we know little about the people that are involved and therefore we don’t know about their ability. There are also perceptions that like Wikipedia, it is all a very loosely coordinate and therefore there are no strict data quality procedures. However, we know that even in the Wikipedia case that when the scientific journal Nature shown over a decade ago (2005) that Wikipedia is resulting with similar quality to Encyclopaedia Britannica, and we will see that OpenStreetMap is producing data of a similar quality to professional services.
In citizen science where sensing and data collection from instruments is included, there are also concerns over the quality of the instruments and their calibration – the ability to compare the results with high-end instruments.
The opening of the Hunter et al. paper (which offers some solutions), summarises the concerned that are raised over data

[Slide 2] Based on conversations with scientists and concerned that are appearing in the literature, there is also a cultural aspect at play which is expressed in many ways – with data quality being used as an outlet to express them. This can be similar to the concerns that were raised in the cult of the amateur (which we’ve seen in week 2 regarding the critique of crowdsourcing) to protect the position of professional scientists and to avoid the need to change practices. There are also special concerns when citizen science is connected to activism, as this seems to “politicise” science or make the data suspicious – we will see next lecture that the story is more complex. Finally, and more kindly, we can also notice that because scientists are used to top-down mechanisms, they find alternative ways of doing data collection and ensuring quality unfamiliar and untested.

[Slide 3] Against this background, it is not surprising to see that checking data quality in citizen science is a popular research topic. Caren Cooper have identified over 50 papers that compare citizen science data with those that were collected by professional – as she points: “To satisfy those who want some nitty gritty about how citizen science projects actually address data quality, here is my medium-length answer, a brief review of the technical aspects of designing and implementing citizen science to ensure the data are fit for intended uses. When it comes to crowd-driven citizen science, it makes sense to assess how those data are handled and used appropriately. Rather than question whether citizen science data quality is low or high, ask whether it is fit or unfit for a given purpose. For example, in studies of species distributions, data on presence-only will fit fewer purposes (like invasive species monitoring) than data on presence and absence, which are more powerful. Designing protocols so that citizen scientists report what they do not see can be challenging which is why some projects place special emphasize on the importance of “zero data.”
It is a misnomer that the quality of each individual data point can be assessed without context. Yet one of the most common way to examine citizen science data quality has been to compare volunteer data to those collected by trained technicians and scientists. Even a few years ago I’d noticed over 50 papers making these types of comparisons and the overwhelming evidence suggested that volunteer data are fine. And in those few instances when volunteer observations did not match those of professionals, that was evidence of poor project design. While these studies can be reassuring, they are not always necessary nor would they ever be sufficient.” (http://blogs.plos.org/citizensci/2016/12/21/quality-and-quantity-with-citizen-science/)

[Slide 4] One way to examine the issue with data quality is to think of the clash between two concepts and systems of thinking on how to address quality issue – we can consider the condition of standard scientific research conditions as ones of scarcity: limited funding, limited number of people with the necessary skills, a limited laboratory space, expensive instruments that need to be used in a very specific way – sometimes unique instruments.
The conditions of citizen science, on the other hand, are of abundance – we have a large number of participants, with multiple skills, but the cost per participant is low, they bring their own instruments, use their own time, and are also distributed in places that we usually don’t get to (backyards, across the country – we talked about it in week 2). Conditions of abundance are different and require different thinking for quality assurance.

[Slide 5] Here some of the differences. Under conditions of scarcity, it is worth investing in long training to ensure that the data collection is as good as possible the first time it is attempted since time is scarce. Also, we would try to maximise the output from each activity that our researcher carried out, and we will put procedures and standards to ensure “once & good” or even “once & best” optimisation. We can also force all the people in the study to use the same equipment and software, as this streamlines the process.
On the other hand, in abundance conditions we need to assume that people are coming with a whole range of skills and that training can be variable – some people will get trained on the activity over a long time, while to start the process we would want people to have light training and join it. We also thinking of activities differently – e.g. conceiving the data collection as micro-tasks. We might also have multiple procedures and even different ways to record information to cater for a different audience. We will also need to expect a whole range of instrumentation, with sometimes limited information about the characteristics of the instruments.
Once we understand the new condition, we can come up with appropriate data collection procedures that ensure data quality that is suitable for this context.

[Slide 6] There are multiple ways of ensuring data quality in citizen science data. Let’s briefly look at each one of these. The first 3 methods were suggested by Mike Goodchild and Lina Li in a paper from 2012.

[Slide 7] The first method for quality assurance is crowdsourcing – the use of multiple people who are carrying out the same work, in fact, doing peer review or replication of the analysis which is desirable across the sciences. As Watson and Floridi argued, using the examine of Zooniverse, the approaches that are being used in crowdsourcing give these methods a stronger claim on accuracy and scientific correct identification because they are comparing multiple observers who work independently.

[Slide 8] The social form of quality assurance is using more and less experienced participants as a way to check the information and ensure that the data is correct. This is fairly common in many areas of biodiversity observations and integrated into iSpot, but also exist in other areas, such as mapping, where some information get moderated (we’ve seen that in Google Local Guides, when a place is deleted).

[Slide 9] The geographical rules are especially relevant to information about mapping and locations. Because we know things about the nature of geography – the most obvious is land and sea in this example – we can use this knowledge to check that the information that is provided makes sense, such as this sample of two bumble bees that are recorded in OPAL in the middle of the sea. While it might be the case that someone seen them while sailing or on some other vessel, we can integrate a rule into our data management system and ask for more details when we get observations in such a location. There are many other such rules – about streams, lakes, slopes and more.

[Slide 10] The ‘domain’ approach is an extension of the geographic one, and in addition to geographical knowledge uses a specific knowledge that is relevant to the domain in which information is collected. For example, in many citizen science projects that involved collecting biological observations, there will be some body of information about species distribution both spatially and temporally. Therefore, a new observation can be tested against this knowledge, again algorithmically, and help in ensuring that new observations are accurate. If we see a monarch butterfly within the marked area, we can assume that it will not harm the dataset even if it was a mistaken identity, while an outlier (temporally, geographically, or in other characteristics) should stand out.

[Slide 11] The ‘instrumental observation’ approach removes some of the subjective aspects of data collection by a human that might make an error, and rely instead on the availability of equipment that the person is using. Because of the increase in availability of accurate-enough equipment, such as the various sensors that are integrated in smartphones, many people keep in their pockets mobile computers with the ability to collect location, direction, imagery and sound. For example, images files that are captured in smartphones include in the file the GPS coordinates and time-stamp, which for a vast majority of people are beyond their ability to manipulate. Thus, the automatic instrumental recording of information provides evidence for the quality and accuracy of the information. This is where the metadata of the information becomes very valuable as it provides the necessary evidence.

[Slide 12] Finally, the ‘process oriented’ approach bring citizen science closer to traditional industrial processes. Under this approach, the participants go through some training before collecting information, and the process of data collection or analysis is highly structured to ensure that the resulting information is of suitable quality. This can include the provision of standardised equipment, online training or instruction sheets and a structured data recording process. For example, volunteers who participate in the US Community Collaborative Rain, Hail & Snow network (CoCoRaHS) receive standardised rain gauge, instructions on how to install it and online resources to learn about data collection and reporting.

[Slide 13]  What is important to be aware of is that methods are not being used alone but in combination. The analysis by Wiggins et al. in 2011 includes a framework that includes 17 different mechanisms for ensuring data quality. It is therefore not surprising that with appropriate design, citizen science projects can provide high-quality data.

 

 

Citizen Science for Observing and Understanding the Earth

Since the end of 2015, I’ve been using the following mapping of citizen science activities in a range of talks:

Range of citizen science activities
Explaining citizen science

The purpose of this way of presentation is to provide a way to guide my audience through the landscape of citizen science (see examples on SlideShare). The reason that I came up with it, is that since 2011 I give talks about citizen science. It started with the understanding that I can’t explain extreme citizen science when my audience doesn’t understand what citizen science is, and that turned into general talks on citizen science.

Similarly to Caren Cooper, I have an inclusive approach to citizen science activities, so in talks, I covered everything – from bird watching to DIY science. I felt that it’s too much information, so this “hierarchy” provides a map to go through the overview (you can look at our online course to see why it’s not a great typology). It is a very useful way to go through the different aspects of citizen science, while also being flexible enough to adapt it – I can switch the “long-running citizen science” fields according to the audience (e.g. marine projects for marine students).

An invitation for Pierre-Philippe Mathieu (European Space Agency) in 2015 was an opportunity to turn this mapping and presentation into a book chapter. The book is dedicated to “Earth Observation Open Science and Innovation and was edited by Pierre-Philippe and Christoph Aubrecht.

When I got to writing the chapter, I contacted two researchers with further knowledge of citizen science and Earth Observation – Suvodeep Mazumdar and Jessica Wardlaw. I was pleased that they were happy to join me in the effort.

Personally, I’m very pleased that we could include in the chapter the story of the International Geophysical Year, (thank Alice Bell for this gem), with Moonwatch and Sputnik monitoring.

The book is finally out, it is open access, and you can read our chapter, “Citizen Science for Observing and Understanding the Earth” for free (as well as all the other chapters). The abstract of the paper is provided below:

Citizen Science, or the participation of non-professional scientists in a scientific project, has a long history—in many ways, the modern scientific revolution is thanks to the effort of citizen scientists. Like science itself, citizen science is influenced by technological and societal advances, such as the rapid increase in levels of education during the latter part of the twentieth century, or the very recent growth of the bidirectional social web (Web 2.0), cloud services and smartphones. These transitions have ushered in, over the past decade, a rapid growth in the involvement of many millions of people in data collection and analysis of information as part of scientific projects. This chapter provides an overview of the field of citizen science and its contribution to the observation of the Earth, often not through remote sensing but a much closer relationship with the local environment. The chapter suggests that, together with remote Earth Observations, citizen science can play a critical role in understanding and addressing local and global challenges.

 

Citizen Science & Scientific Crowdsourcing – week 2 – Google Local Guides

The first week of the “Introduction to Citizen Science and Scientific Crowdsourcing” course was dedicated to an introduction to the field of citizen science using the history, examples and typologies to demonstrate the breadth of the field. The second week was dedicated to the second half of the course name – crowdsourcing in general, and its utilisation in scientific contexts. In the lecture, after a brief introduction to the concepts, I wanted to use a concrete example that shows a maturity in the implementation of commercial crowdsourcing. I also wanted something that is relevant to citizen science and that many parallels can be drawn from, so to learn lessons. This gave me the opportunity to use Google Local Guides as a demonstration.

My interest in Google Local Guides (GLG) come from two core aspects of it. As I pointed in OpenStreetMap studies, I’m increasingly annoyed by claims that OpenStreetMap is the largest Volunteered Geographical Information (VGI) project in the world. It’s not. I guessed that GLG was, and by digging into it, I’m fairly confident that with 50,000,000 contributors (of which most are, as usual, one-timers), Google created the largest VGI project around. The contributions are within my “distributed intelligence” and are voluntary. The second aspect that makes the project is fascinating for me is linked to a talk from 2007 in one of the early OSM conferences about the usability barriers that OSM (or more general VGI) need to cross to reach a wide group of contributors – basically about user-centred design. The design of GLG is outstanding and shows how much was learned by the Google Maps and more generally by Google about crowdsourcing. I had very little information from Google about the project (Ed Parsons gave me several helpful comments on the final slide set), but by experiencing it as a participant who can notice the design decisions and implementation, it is hugely impressive to see how VGI is being implemented professionally.

As a demonstration project, it provides examples for recruitment, nudging participants to contribute, intrinsic and extrinsic motivation, participation inequality, micro-tasks and longer tasks, incentives, basic principles of crowdsourcing such as “open call” that support flexibility, location and context aware alerts, and much more. Below is the segment from the lecture that focuses on Google Local Guides, and I hope to provide a more detailed analysis in a future post.

The rest of the lecture is available on UCLeXtend.

Chapter in Routledge Handbook of Mapping and Cartography – VGI and Beyond: From Data to Mapping

Hot on the heels of the Routledge Handbook of Environmental Justice is thThe Routledge Handbook of Mapping and CartographyRoutledge Handbook of Mapping and Cartography. The handbook was edited by Alex Kent (Canterbury Christ Church University) who is currently the President of the British Cartographic Society and Editor of The Cartographic Journal; and Peter Vujakovic (also from Canterbury Christ Church University) who edited The Cartographic Journal.

Like the other handbooks, this is an extensive collection of 43 chapters and almost 600 page about maps and mapping. The chapters provide a vivid demonstration that cartography and map making is art and science, and that it links to many sciences and practices – from cognitive psychology to geodesy. The list of authors is impressive and includes many of the people that are shaping current cartographic research.

However, with a price tag of £195 for the Book, this collection is expensive and suitable for university libraries and to professional or commercial mapping organisation. The eBook is £35, which makes it much more affordable, though having used the online system, the interface could be better. Luckily the policy of Routledge permits sharing the chapters on personal websites.

My contribution to the book is in a joint paper that was led by Vyron Antoniou titled VGI and Beyond: From Data to Mapping. The chapter is building on a collaboration between Vyron, myself and Cristina Capineri during the COST Action on Volunteered Geographic Information (ENERGIC). In the chapter, we look at the concept of Volunteered Geographic Information (VGI) within practices of mapping and cartography and we attempted to provide an accessible overview of the area. We define what VGI is, provide an overview of the area, look at the advantages and disadvantages of VGI in mapping and cartography, and then look at the impacts of VGI on national mapping agencies, the public, and public bodies. The chapter is available here and we would be very happy to hear comments on it.

 

 

Chapter in ‘Understanding Spatial Media’ on VGI & Citizen Science

77906_9781473949683[1]The book ‘Understanding Spatial Media‘ came out earlier this year. The project is the result of joint effort of the editors Rob Kitchin (NUI Maynooth, Ireland), Tracey P. Lauriault (Carleton University, Canada), and Matthew W. Wilson (University of Kentucky, USA).

The book is filling the need to review and explain what happened in the part 20 years, with the increase use of digital geographic information that then became widespread and can be considered as a media – something that Daniel Sui and Mike Goodchild noted in 2001. The book chapters are covering the underlying technologies, the sources of the data and media that are part of this area, and the implications – from smart cities to surveillance and privacy.

My contribution to this book is in a chapter that belong to the middle section – spatial data and spatial media – and that provides an introduction to Volunteered Geographic Information and Citizen Science. If you’re interested, you can read the chapter here.

Crowdsourced: navigation & location-based services

toyosemite-mapOnce you switch the smartphone off from email and social media network, you can notice better when and how you’re crowdsourced. By this, I mean that use of applications to contribute data is sometimes clearer as the phone becomes less of communication technology and more of information technology (while most of the time it is an information and communication mixed together).

During my last year summer break, while switching off, I was able to notice three types of crowdsourcing that were happening as I was mostly using my phone for tourism:  the set of applications that I used during the period was mostly Google Maps, Waze, and Sygic for navigation (Sygic is an offline satnav app that uses local storage, especially useful as you go away from signal – as in the case of travelling to Yosemite above). I also used TripAdvisor to find restaurants or plan visits, and in a tour of Hollywood, we’ve downloaded a guided tour by “PocketGuide Audio Travel Guide”. There were also the usual searches on Google to find the locations of swimming pools, train schedule, boat tours and all the other things that you do when you’re travelling.

The first type of being crowdsourced that I knew was there but I couldn’t notice was with every action that I’ve done with the phone. Searches, visiting websites and all the other things that I’ve done as long as the phone was on. I know that they are there, but I have no idea who is collecting the traces – I can be quite certain that the phone company and Google were getting information, but I can’t tell for sure – so that’s hidden crowdsourcing.

The second type is being passively crowdsourced – I know that it happened, and I can sotosanjose-lastdayrt of guess what information is being recorded, but I didn’t need any action to make it happen. Checking the map on the way back to the airport at the end of the journey, show how geolocated images and information are being put on the map. Google Maps assume that I visited known locations, mostly commercials, along the way. It was especially funny to stay in a suburban place that happened to be the registered address of a business, and every time we went to it, Google insisted that we’re visiting the business (which doesn’t physically exist in the place). At least with this passive crowdsourcing, I am knowingly contributing information – and since I’m benefiting from the navigation guidance of Google Maps as I drive along, it is a known transaction (regardless of power relationship and fairness). screenshot_20160808-154540

the third and final type of process was active crowdsourcing. This was when I was aware of what I’m contributing, to which system, and more or less how. When I provided an image to Google Local Guides  I knew that it will be shared (I am though, hugely surprised how many times it was viewed, but I’ll write about Local Guides in a different post). I was also actively contributing to TripAdvisor about some place near Venice Beach. A certain surprise also came from Waze, which, in a day of experiencing Los Angeles famous traffic, provided me with a message that ‘I’m one of the top contributors on this route’ after 3 reports. Of course, I can’t tell if this message is real, but if 3 reports are enough to make you a top contributor, the number of reporting participants must be very low.

Few other observations: it was interesting to see how embedded is the consideration that you will be online all the time – Google Maps suggested to download a route when they had information that part of the journey will not be covered by mobile signal, but the application didn’t behave well with offline data (Sygic did). Both Waze and Google Maps behaved very erratically when I passed a blocked slip road and didn’t follow their navigation guidance. For quite a distance, they continue to suggest that I turn around and use the blocked road.

The other aspect that was very apparent is the built in methodological individualism of all these apps – even though I was only with my partner, we found the PocketGuide Audio Travel Guide awkward to use – we wanted to experience the tour (which is interesting) together. The app is just difficult to use when you try to go with other people and discuss things together. This is partially how phones are thought of, but touring is not only a solitary experience…