A citizens observatory is a concept that evolved at EU policy circles, defining the combination of participatory community monitoring, technology and governance structures that are needed to monitor/observe/manage an environmental issue. About two years ago, the EU FP7 funded 5 citizens observatory projects covering areas from water management to biodiversity monitoring. A meeting at Brussels was an opportunities to review their progress and consider the wider implications of citizen science as it stand now. The meeting was organised and coordinated by the group in the Directorate General Research and Innovation that is responsible for Earth Observations (hence the observatory concept!).  The following are my notes from the meeting.

They are very long and I’m sure that they are incoherent at places! 

From Commons Lab The meeting was opened with Kurt Vandenberghe (Director Environment, DG R&I). He suggested that citizens observatories contribute to transparency in governance – for example, ensuring that monitoring is done in the correct place, and not, as done in some member states, where monitoring stations are in the places that will yield ‘useful’ or ‘acceptable’ results but not representative: “Transparency is a driver in intrinsic ethical behaviour”. There is also value in having citizens’ input complementing satellite data. It can help in engaging the public in co-design of environmental applications and addressing environmental challenges. Examples for such participation is provided in Marine LitterWatch and NoiseWatch from EEA and development of apps and technology can lead to new business opportunities. The concept of earth observations is about creating a movement of earth observers who collect information, but also allow citizens to participate in environmental decision-making. This can lead to societal innovation towards sustainable and smart society. From the point of view of the commission DG R&I, they are planning to invest political and financial capital in developing this vision of observatories. The New calls for citizens observatories demonstrators is focusing on citizens’ participation in monitoring land use and land cover in rural and remote areas. Data collected through observatories should be complementing those that are coming from other sources. The commission aim to continue the investment in future years – citizen science is seen as both business opportunities and societal values. A successful set of project that end by showing that citizen observatories are possible is not enough – they want to see the creation of mass movement. Aim to see maximising capital through the citizens observatories. Optimising framework condition to allow citizens observatories to be taken up by member states and extended, implemented and flourish. Some of the open questions include how to provide access to the data to those that collected it? How can we ensure that we reach out across society to new groups that are not currently involved in monitoring activities? How can we deal with citizens observatories security and privacy issues regarding the information? The day is an opportunity for co-creation and considering new ways to explore how to address the issue of citizens observatories from a cross-disciplinary perspective – “Citizen science as a new way to manage the global commons”.

Next, a quick set of presentations of the FP7 projects:

WeSenseIt (Fabio Ciravegna) is a project that focuses on citizens involvement in water resources – citizens have a new role in the information chain of water related decisions. Participants are expected to become part of the decision-making. In this project, citizens observatory is seen as a science method, an environment to implement collaboration and as infrastructure. They are working in Doncaster (UK), Vicenza (Italy) and Delft (The Netherlands). In WeSenseIt, they recognise that different cultures and different ways to do things are part of such systems. A major questions is – who are the citizens? In the UK : normal people and in Italy: civil protection officials and volunteers, while in the Netherlands water and flood management is highly structured and organised activity. They have used a participatory design approach and working on the issue of governance and understanding how the citizen observatories should be embedded in the existing culture and processes. They are creating a citizens’ portal and another one for decision makers. The role of citizens portal is to assist with data acquisition with areas and equipment citizens can deploy – weather, soil moisture,etc. On the decision makers portals, there is the possibility is to provide surveillance information (with low-cost cameras etc), opportunistic sensing and participatory sensing – e.g. smart umbrella while combining all this information to be used together. WeSenseIt created a hybrid network that is aimed to provide information to decision makers and citizens. After two years, they can demonstrate that their approach can work: In Vicenza they used the framework to develop action to deal with flood preparedness. They also started to work with large events to assist in the organisation and support the control room, so in Torino they are also starting to get involved in helping running an event with up to 2m people.

Omniscientis (Philippe Ledent)  – The Omniscientis project (which ended in September) focused on odour monitoring and using different sensors – human and electronic. Odour can be a strong / severe nuisance, in Wallonia and France, and there is concerns about motorways, factories, livestock and waste facilities. Odour is difficult to measure and quantify and complex to identify. Mainly because it is about human perception, not only the measurement of chemicals in the air. In too many regulations and discussions about odour, citizens were considered as passive or victims. The Omniscientis project provided an opportunity to participants be active in the monitoring. The project took a multi-stakeholders  approach (farmers, factory operators, local residents etc.). They created odour management information system with the concept of a living lab. They created a OdoMIS that combines information from sensors, industry, NGOs, experts, and citizens. They created an app OdoMap that provide opportunities for participants to provide observations, but also see what other people measured and access to further information. They created chemical sensor array (e-nose), and the citizens helped in assessing what is the concentration that they sense. This was linked to a computationally intensive dispersion model. They have done a pilot around a pig farm in Austria to validate the model, and another near pulp and paper mill. Evolution of citizens participation was important for the project, and people collected measurements for almost a year, with over 5000 measurements. The results is they would like to link odour sources, citizens and authorities to work on the area. They have used actor netowrk theory to enrol participants in the process with strong UCD element.

COBWEB (Chris Higgins) has been working a generic crowdsourcing infrastructure, with data that can supports policy formation while addressing data quality issues and using open standards. They aimed to encapsulate metadata and OGC standards to ensure that the ifnroamtion is interoperable. They would like to create a toolkit can be used in different contexts and scenarios. They focus on the biospehere reserve network across Europe. They carried out a lot of co-design activities from the start with stakeholders engagement, they are doing co-design with 7 organisations in Wales – Woodlands trusts, RSPB, Snowdonia national park, and others. This lead to different focus and interest from each organisation – from dolphins to birds. They hope to see greater buy-in because of that.

Citi-Sense (Alena Bartonova) focusing on air quality. The objectives of city sense is to explore if people can participate in environmental governance. They are doing empowerment initiatives – urban quality, schools, and public spaces. In the urban context they measure pollution, meteorological observations, noise, health, biomarkers and UV exposure – they looked at technologies from mobile sensors and also static sensors that can be compared to compliance monitoring. In schools, they engage the school children, with looking at sensors that are installed at school and also looking at indoor air quality data. There are co-design activities with students. In Public spaces they focused on acoustic sensing, and discover that phones are not suitable so went to external sensors (we discovered the problem with phones in EveryAware). They explore in 9 cities and focusing from sensors, data and services platform but also explore how to engage people in a meaningful way. The first two years focused on technical aspects. They are now moving to look at the engagement part much more but they need to consider how to put it out. They are developing apps and also considering how to improve air quality apps. They would like a sustainable infrastructure.

Citclops (Luigi Ceccaroni) originally aimed ‘to create a community participatory governance methods aided by social media streams’, but this is an unclear goal that the project partners found confusing! So they are dealing with the issue of marine environment: asking people to take pictures of marine environment and through the app facilitating  visual monitoring of marine environment (available to download by anyone) – they are helping people to assess visually the quality of water bodies. There is an official way of defining the colour of sea waters which they use in the project and also comparing ground observations with satellite information. The project included the design of DIY devices to allow the measurement of water opacity. Finally, exploring water fluorescence. They design and 3D printed a device that can be used with smartphones to measure  fluorescence as this help to understand concentration of chlorophyll and can be associated with remote sensing information. Citizen science is a way to complement official data – such as the data from the water directive.

After a break and demonstration from some of the projects, the first round-table of the day, which include executives from environment protection agencies across Europe started

From @ScotlandEuropa strategic views on Citizens Observatories

[I’ve lost my notes, so below is a summary of the session edited from Valentine Seymour notes]

The chair (Gilles Ollier) of the session highlighted that the following issues as significant for considering the role of citizen science: Are we doing something useful/usable? Valuable? And sustainable?
James Curran (Scottish Environmental Protection Agency) noted that SEPA took citizen science to the core of its business. He highlighted issues of growth, jobs and investments. The need for sustainable growth and that citizen science contributed to these goals very well as the Chinese proverb say “Involve me and I will understand”. SEPA has been promoting mobile applications to detect invasive species and environmental damages. The Riverfly project is an example of engaging people in monitoring to detect water quality and invertebrate sampling and how important it was for the Water Framework Directive (WFD) to include public participation. There is a need to provide accessible information, working with others collaboratively, measuring behavioural changes and the need for public engagement.

Laura Burke (Ireland EPA) main statement was that citizen science do not replace governmental and official scientific monitoring but that citizen science should be seen in complimentary. There are three main issues or areas to consider; terminology (spectrum of the term citizen science), the need for thinking about the long-term sustainable future of citizen science projects, and acknowledge the synergies between projects.

Hugo de Groof (DG Env) noted the importance of access to information and the Open Access Directive that has been passed.  In terms of governance, we need to follow 5 main principles: 1) Accountability, 2) transparency, 3) participation, 4) Effectiveness and efficiency and finally 5) Respect. Raymond Feron from the Dutch ministry for infrastructure and environment emphasised that there is a social change emerging. [End of Valentine’s notes]

The issues of operationalisation received attention – there are different projects, how far are we from large-scale deployment? Colin Chapman (Welsh Government) – maturity across observatory projects vary from case to case and across issues. Technologies are still maturing, there is a need to respond to issues and mobilise resources to address issues that citizens bring up. Systems approach to ecosystem management is also a factor in considering how to integrate observatories. There are too much reliance on macro modelling. A question for policy bodies is “can we incentivise citizens to collect data across policy areas?” for example invasive species, we can use the information in different areas from flood modelling to biodiversity management. David Stanners (EEA) noted that citizens observatories are vulnerable at this point in time and this lack of stability  and there are examples of projects that didn’t last. There are some inter-linkages, but not an ecology of observatories, of interconnectedness and ability to survive. Need better linkage with policy, but not across the board and no direct policy elements. The integration of citizens observatories is a fantastic opportunity at EU level – as issues of the environment suppose to be very visible. Raymond Feron noted that government might have issues in keeping pace with citizens actions. Government organisations need to learn how to integrate citizens observatories, need to learn to reuse parts. Integrate research programme with implementation strategy. James Curran also stated that working with anglers and other stakeholders can increase trust. In terms of quality and relevant, citizen science data is not different to other data. Laura Burke noted that no government have all the answers, and trust issues should be presented as such. Need to move away from concept of one organisation with a solution to any given problem. David Stanners raised the issues of truth seeking. Within the cupernicos programme, there are opportunities to support services with citizen science.

Following the point of views from the panellists, questions about trust, finding ways to include of people without access to technology were raised by the audience. The panellists agreed that from the point of view of policy makes the concept of citizens observatories is obvious but there is a need to make citizens observatories and citizen science activities sustainable and well-integrated in government activities. Interestingly, James Curran noted that the issue of data quality from citizen science is not a major obstacles, inherently because environmental authorities are used to make decisions that are risk based. There was willingness to work with intermediaries to reach out to under-represented groups. David Stanners called for  cross cutting meta-studies to understand citizens observations landscape.

The next series of presentations covered citizen science activities that are not part of the citizens observatories projects.

NoiseWatch/Marine litter watch (David Stanners, EEA) – Noisewatch was developed by the EEA and provie the modelling element, measurement, and citizen rating element. He argued that dB is not good measure, as noise is a perception issue and not about just measurement. NoiseWatch received an award in the Geospatial World Forum. It became global although it wasn’t promoted as such, with uptake in India and China and UNEP are considering to take it over and maintain it. Sustainability of NoiseWatch is a challenge for EEA and it might be more suitable in a global platform such as UNEP Live. NoiseWatch is seen as complementing existing monitoring stations because there as so few of them. When analysing the sources of the measurement, NoiseWatch get a lot of observations from roads, with 21% of industry noise – in total almost 195000 measurements. Another application is Marine LitterWatch which provides a way for people to share information about the state of beaches. The application is more complex as it embedded in protocol of data collection, and David argue that it is ‘more close to citizen science’, EEA got almost 7500 measurements with 144 events to use it, they are developing it further.

LakeWiki (Juhani Kettunen, who was not present) is an initiative that focus on motoring Finnish lakes – was launched by Syke and it is aimed to allow local communities to take care of their lakes, record information and build a long term observations. Simple platform, recording information such as ice break up but it is aimed to allow locals write about the lake, maintain observations sites, upload pictures, announce local events and write in discussion forums, 1400 sites [this project is also noted in COST Energic network]

Raymond Feron presented a programme in Netherlands called  digital Delta Initiative: partnership between research, public and government. IBM, TU Delft and government are involved. Trying to make water data available to everyone. focus of the system allow re-use of information, the government try to do things more efficiently, shorten time to market, improve quality of decisions, while also improving citizen participation. Ideas of increasing export to new places. Involving the public with dyke monitoring because they can do things locally easily.

I gave a talk about Mapping for Change air quality studies, and I hope to discuss them in a different post:

Claudia Goebel followed with a report on ECSA (see my report for ECSA meeting)

Antonoi Parodi from CIMA foundation discussed the DRIHM project. This is mostly a project focused on meteorological information. Issue of meteorology has a very long history of observations, going from 300 BC. There is plenty of reliance of observed patterns of events. Informal folklore was used by early meteorology citizen science. The middle ages, there are examples of getting information to deal with flash flood. Within the project they created a volunteer thinking platform to support classifications of thunderstorms. The Cb-TRAM monitoring observations of thunderstorms. Interestingly, a follow on question explored the link between extreme events (floods last year) and the role of the research project to provide relevant information.

The Socientize project was presented by Francisco Sanz, covering areas of digital science.

There was also a presentation from the SciCafe 2.0 project, including mentioning the European Observatory for Crowd-Sourcing . Another tool from the project is Citizens’ Say tool  

The final panel explored issues on the challenges of citizen science (I was part of this panel). The people on the panel were Jaume Piera (CITCLOPS),;Arne Berre (CITI-SENSE); Bart De Lathouwer (COBWEB); Philippe Valoggia (OMNISCIENTIS); Uta Wehn (WeSenseIt); Susanne Lützenkirchen, City of Oslo and myself.

Susanne noted that the city of Oslo developed some apps, such as safe for schools – people can experience their routes to schools and they are interested in more citizen science and citizen observatories.

Strategy for sustainability of engagement over time – Uta noted that the co-design process is very important and governance analysis to understand the processes and the local needs (in WeSenseIt). The observatories need to consider who are the partners – authorities are critical to see the value of observatories and provide feedback. Jaime suggested – identifying points in the project that give participants feeling that they are part of the process, allowing people to participate in different ways. Making people aware that they are part of the activities and they are valued. Showing the need for long-term observations. Susanne pointed that in Oslo there isn’t any simple answer – the issue of who are the citizens and in others it is a specific groups or more complex design sometime need to think who chose participants and how representative they are.

In WeSenseIT, they have privacy and consent setting – adhering to rules of social media, and it is an issue of data that came from other sources and how it is going to be reused. In general, Uta noted that WeSenseIt would like to try and make the data open as possible.

Data preservation is an issue – how data was handled, if we assume that there are probably 500 projects or more in Europe which is Max Craglia (JRC, who chaired the session) estimation. The issues of citizen observatories, we need to consider the individual data and there is sometime concern about releasing unvalidated data. Bart pointed that Cobweb is taking care of privacy and security of data and they are storing information about observers and there are privacy rules. Privacy legislation are local and need to follow the information. citizens see the benefit in what they collected and the sustainability of commitment. It is important to work with existing social structures and that provide opportunity for empowerment. Views about ownership of data were raised.

In terms of integration and synergy or interoperability of the citizen centred projects – interoperability is critical topic, the data need to be standardised and deal with the metadata (the most boring topic in the world). It should be collected at the right level. There is good foundation in GEOS and OGC, so we can consider how to do it.

What is the role of scientists? the role of scientists – there are partners who focus on dealing with the data and augment it with additional information and there is a role of managing the data. The link to policy also require an understanding of uncertainty. The discourse of science-policy is about what is considered as evidence. There is embracing of citizen science in environment agencies (which was demonstrated in the first panel), but there is a need for honest discussion about what happen to the data, and what degree citizens can participate in decision-making. Relevancy, legitimacy are critical to the understanding.

There was also call for accepting the uncertainty in the data – which is integral part of citizen science data. David Stanners emphasised the need for legitimacy of the information that is coming from citizens observatories as part of the trust that people put in contributing to them.

The final comments came from Andrea Tilche (Head of Unit Climate Actions and Earth Observation, DG R&I). The commission recognise that citizen observatories are not a replacement for institutional monitoring scheme (although he mentioned maybe in the future). The potential of engaging users is tremendous, and the conference demonstrated the energy and scale of activities that can be included in this area . The ownership of information need to be taken into account. We need to link and close the gaps with scientists and policy makers. We need to create market around the observatories – can’t only do it through project that disappear. There is a need for market of citizen observatories and business models. In the new call, they want to see the project generate and credible business processes. Citizens observatories will need demonstrate raising funding from other sources.

The Association of American Geographers is coordinating an effort to create an International Encyclopedia of Geography. Plans started in 2010, with an aim to see the 15 volumes project published in 2015 or 2016. Interestingly, this shows that publishers and scholars are still seeing the value in creating subject-specific encyclopedias. On the other hand, the weird decision by Wikipedians that Geographic Information Science doesn’t exist outside GIS, show that geographers need a place to define their practice by themselves. You can find more information about the AAG International Encyclopedia project in an interview with Doug Richardson from 2012.

As part of this effort, I was asked to write an entry on ‘Volunteered Geographic Information, Quality Assurance‘ as a short piece of about 3000 words. To do this, I have looked around for mechanisms that are used in VGI and in Citizen Science. This are covered in OpenStreetMap studies and similar work in GIScience, and in the area of citizen science, there are reviews such as the one by Andrea Wiggins and colleagues of mechanisms to ensure data quality in citizen science projects, which clearly demonstrated that projects are using multiple methods to ensure data quality.

Below you’ll find an abridged version of the entry (but still long). The citation for this entry will be:

Haklay, M., Forthcoming. Volunteered geographic information, quality assurance. in D. Richardson, N. Castree, M. Goodchild, W. Liu, A. Kobayashi, & R. Marston (Eds.) The International Encyclopedia of Geography: People, the Earth, Environment, and Technology. Hoboken, NJ: Wiley/AAG

In the entry, I have identified 6 types of mechanisms that are used to ensure quality assurance when the data has a geographical component, either VGI or citizen science. If I have missed a type of quality assurance mechanism, please let me know!

Here is the entry:

Volunteered geographic information, quality assurance

Volunteered Geographic Information (VGI) originate outside the realm of professional data collection by scientists, surveyors and geographers. Quality assurance of such information is important for people who want to use it, as they need to identify if it is fit-for-purpose. Goodchild and Li (2012) identified three approaches for VGI quality assurance , ‘crowdsourcing‘ and that rely on the number of people that edited the information, ‘social’ approach that is based on gatekeepers and moderators, and ‘geographic’ approach which uses broader geographic knowledge to verify that the information fit into existing understanding of the natural world. In addition to the approaches that Goodchild and li identified, there are also ‘domain’ approach that relate to the understanding of the knowledge domain of the information, ‘instrumental observation’ that rely on technology, and ‘process oriented’ approach that brings VGI closer to industrialised procedures. First we need to understand the nature of VGI and the source of concern with quality assurance.

While the term volunteered geographic information (VGI) is relatively new (Goodchild 2007), the activities that this term described are not. Another relatively recent term, citizen science (Bonney 1996), which describes the participation of volunteers in collecting, analysing and sharing scientific information, provide the historical context. While the term is relatively new, the collection of accurate information by non-professional participants turn out to be an integral part of scientific activity since the 17th century and likely before (Bonney et al 2013). Therefore, when approaching the question of quality assurance of VGI, it is critical to see it within the wider context of scientific data collection and not to fall to the trap of novelty, and to consider that it is without precedent.

Yet, this integration need to take into account the insights that emerged within geographic information science (GIScience) research over the past decades. Within GIScience, it is the body of research on spatial data quality that provide the framing for VGI quality assurance. Van Oort’s (2006) comprehensive synthesis of various quality standards identifies the following elements of spatial data quality discussions:

  • Lineage – description of the history of the dataset,
  • Positional accuracy – how well the coordinate value of an object in the database relates to the reality on the ground.
  • Attribute accuracy – as objects in a geographical database are represented not only by their geometrical shape but also by additional attributes.
  • Logical consistency – the internal consistency of the dataset,
  • Completeness – how many objects are expected to be found in the database but are missing as well as an assessment of excess data that should not be included.
  • Usage, purpose and constraints – this is a fitness-for-purpose declaration that should help potential users in deciding how the data should be used.
  • Temporal quality – this is a measure of the validity of changes in the database in relation to real-world changes and also the rate of updates.

While some of these quality elements might seem independent of a specific application, in reality they can be only be evaluated within a specific context of use. For example, when carrying out analysis of street-lighting in a specific part of town, the question of completeness become specific about the recording of all street-light objects within the bounds of the area of interest and if the data set includes does not include these features or if it is complete for another part of the settlement is irrelevant for the task at hand. The scrutiny of information quality within a specific application to ensure that it is good enough for the needs is termed ‘fitness for purpose’. As we shall see, fit-for-purpose is a central issue with respect to VGI.

To understand the reason that geographers are concerned with quality assurance of VGI, we need to recall the historical development of geographic information, and especially the historical context of geographic information systems (GIS) and GIScience development since the 1960s. For most of the 20th century, geographic information production became professionalised and institutionalised. The creation, organisation and distribution of geographic information was done by official bodies such as national mapping agencies or national geological bodies who were funded by the state. As a results, the production of geographic information became and industrial scientific process in which the aim is to produce a standardised product – commonly a map. Due to financial, skills and process limitations, products were engineered carefully so they can be used for multiple purposes. Thus, a topographic map can be used for navigation but also for urban planning and for many other purposes. Because the products were standardised, detailed specifications could be drawn, against which the quality elements can be tested and quality assurance procedures could be developed. This was the backdrop to the development of GIS, and to the conceptualisation of spatial data quality.

The practices of centralised, scientific and industrialised geographic information production lend themselves to quality assurance procedures that are deployed through organisational or professional structures, and explains the perceived challenges with VGI. Centralised practices also supported employing people with focus on quality assurance, such as going to the field with a map and testing that it complies with the specification that were used to create it. In contrast, most of the collection of VGI is done outside organisational frameworks. The people who contribute the data are not employees and seemingly cannot be put into training programmes, asked to follow quality assurance procedures, or expected to use standardised equipment that can be calibrated. The lack of coordination and top-down forms of production raise questions about ensuring the quality of the information that emerges from VGI.

To consider quality assurance within VGI require to understand some underlying principles that are common to VGI practices and differentiate it from organised and industrialised geographic information creation. For example, some VGI is collected under conditions of scarcity or abundance in terms of data sources, number of observations or the amount of data that is being used. As noted, the conceptualisation of geographic data collection before the emergence of VGI was one of scarcity where data is expensive and complex to collect. In contrast, many applications of VGI the situation is one of abundance. For example, in applications that are based on micro-volunteering, where the participant invest very little time in a fairly simple task, it is possible to give the same mapping task to several participants and statistically compare their independent outcomes as a way to ensure the quality of the data. Another form of considering abundance as a framework is in the development of software for data collection. While in previous eras, there will be inherently one application that was used for data capture and editing, in VGI there is a need to consider of multiple applications as different designs and workflows can appeal and be suitable for different groups of participants.

Another underlying principle of VGI is that since the people who collect the information are not remunerated or in contractual relationships with the organisation that coordinates data collection, a more complex relationships between the two sides are required, with consideration of incentives, motivations to contribute and the tools that will be used for data collection. Overall, VGI systems need to be understood as socio-technical systems in which the social aspect is as important as the technical part.

In addition, VGI is inherently heterogeneous. In large scale data collection activities such as the census of population, there is a clear attempt to capture all the information about the population over relatively short time and in every part of the country. In contrast, because of its distributed nature, VGI will vary across space and time, with some areas and times receiving more attention than others. An interesting example has been shown in temporal scales, where some citizen science activities exhibit ‘weekend bias’ as these are the days when volunteers are free to collect more information.

Because of the difference in the organisational settings of VGI, a different approaches to quality assurance is required, although as noted, in general such approaches have been used in many citizen science projects. Over the years, several approaches emerged and these include ‘crowdsourcing ‘, ‘social’, ‘geographic’, ‘domain’, ‘instrumental observation’ and ‘process oriented’. We now turn to describe each of these approaches.

Thecrowdsourcing approach is building on the principle of abundance. Since there are is a large number of contributors, quality assurance can emerge from repeated verification by multiple participants. Even in projects where the participants actively collect data in uncoordinated way, such as the OpenStreetMap project, it has been shown that with enough participants actively collecting data in a given area, the quality of the data can be as good as authoritative sources. The limitation of this approach is when local knowledge or verification on the ground (‘ground truth’) is required. In such situations, the ‘crowdsourcing’ approach will work well in central, highly populated or popular sites where there are many visitors and therefore the probability that several of them will be involved in data collection rise. Even so, it is possible to encourage participants to record less popular places through a range of suitable incentives.

Thesocial approach is also building on the principle of abundance in terms of the number of participants, but with a more detailed understanding of their knowledge, skills and experience. In this approach, some participants are asked to monitor and verify the information that was collected by less experienced participants. The social method is well established in citizen science programmes such as bird watching, where some participants who are more experienced in identifying bird species help to verify observations by other participants. To deploy the social approach, there is a need for a structured organisations in which some members are recognised as more experienced, and are given the appropriate tools to check and approve information.

Thegeographic approach uses known geographical knowledge to evaluate the validity of the information that is received by volunteers. For example, by using existing knowledge about the distribution of streams from a river, it is possible to assess if mapping that was contributed by volunteers of a new river is comprehensive or not. A variation of this approach is the use of recorded information, even if it is out-of-date, to verify the information by comparing how much of the information that is already known also appear in a VGI source. Geographic knowledge can be potentially encoded in software algorithms.

Thedomain approach is an extension of the geographic one, and in addition to geographical knowledge uses a specific knowledge that is relevant to the domain in which information is collected. For example, in many citizen science projects that involved collecting biological observations, there will be some body of information about species distribution both spatially and temporally. Therefore, a new observation can be tested against this knowledge, again algorithmically, and help in ensuring that new observations are accurate.

Theinstrumental observation approach remove some of the subjective aspects of data collection by a human that might made an error, and rely instead on the availability of equipment that the person is using. Because of the increased in availability of accurate-enough equipment, such as the various sensors that are integrated in smartphones, many people keep in their pockets mobile computers with ability to collect location, direction, imagery and sound. For example, images files that are captured in smartphones include in the file the GPS coordinates and time-stamp, which for a vast majority of people are beyond their ability to manipulate. Thus, the automatic instrumental recording of information provide evidence for the quality and accuracy of the information.

Finally, the ‘process oriented approach bring VGI closer to traditional industrial processes. Under this approach, the participants go through some training before collecting information, and the process of data collection or analysis is highly structured to ensure that the resulting information is of suitable quality. This can include provision of standardised equipment, online training or instruction sheets and a structured data recording process. For example, volunteers who participate in the US Community Collaborative Rain, Hail & Snow network (CoCoRaHS) receive standardised rain gauge, instructions on how to install it and an online resources to learn about data collection and reporting.

Importantly, these approach are not used in isolation and in any given project it is likely to see a combination of them in operation. Thus, an element of training and guidance to users can appear in a downloadable application that is distributed widely, and therefore the method that will be used in such a project will be a combination of the process oriented with the crowdsourcing approach. Another example is the OpenStreetMap project, which in the general do not follow limited guidance to volunteers in terms of information that they collect or the location in which they collect it. Yet, a subset of the information that is collected in OpenStreetMap database about wheelchair access is done through the highly structured process of the WheelMap application in which the participant is require to select one of four possible settings that indicate accessibility. Another subset of the information that is recorded for humanitarian efforts is following the social model in which the tasks are divided between volunteers using the Humanitarian OpenStreetMap Team (H.O.T) task manager, and the data that is collected is verified by more experienced participants.

The final, and critical point for quality assurance of VGI that was noted above is fitness-for-purpose. In some VGI activities the information has a direct and clear application, in which case it is possible to define specifications for the quality assurance element that were listed above. However, one of the core aspects that was noted above is the heterogeneity of the information that is collected by volunteers. Therefore, before using VGI for a specific application there is a need to check for its fitness for this specific use. While this is true for all geographic information, and even so called ‘authoritative’ data sources can suffer from hidden biases (e.g. luck of update of information in rural areas), the situation with VGI is that variability can change dramatically over short distances – so while the centre of a city will be mapped by many people, a deprived suburb near the centre will not be mapped and updated. There are also limitations that are caused by the instruments in use – for example, the GPS positional accuracy of the smartphones in use. Such aspects should also be taken into account, ensuring that the quality assurance is also fit-for-purpose.

References and Further Readings

Bonney, Rick. 1996. Citizen Science – a lab tradition, Living Bird, Autumn 1996.
Bonney, Rick, Shirk, Jennifer, Phillips, Tina B. 2013. Citizen Science, Encyclopaedia of science education. Berlin: Springer-Verlag.
Goodchild, Michael F. 2007. Citizens as sensors: the world of volunteered geography. GeoJournal, 69(4), 211–221.
Goodchild, Michael F., and Li, Linna. 2012, Assuring the quality of volunteered geographic information. Spatial Statistics, 1 110-120
Haklay, Mordechai. 2010. How Good is volunteered geographical information? a comparative study of OpenStreetMap and ordnance survey datasets. Environment and Planning B: Planning and Design, 37(4), 682–703.
Sui, Daniel, Elwood, Sarah and Goodchild, Michael F. (eds), 2013. Crowdsourcing Geographic Knowledge, Berlin:Springer-Verlag.
Van Oort, Pepjin .A.J. 2006. Spatial data quality: from description to application, PhD Thesis, Wageningen: Wageningen Universiteit, p. 125.

Today marks the publication of the report ‘crowdsourced geographic information in government‘. ReportThe report is the result of a collaboration that started in the autumn of last year, when the World Bank Global Facility for Disaster Reduction and Recovery(GFDRR)  requested to carry out a study of the way crowdsourced geographic information is used by governments. The identification of barriers and success factors were especially needed, since GFDRR invest in projects across the world that use crowdsourced geographic information to help in disaster preparedness, through activities such as the Open Data for Resilience Initiative. By providing an overview of factors that can help those that implement such projects, either in governments or in the World Bank, we can increase the chances of successful implementations. To develop the ideas of the project, Robert Soden (GFDRR) and I run a short workshop during State of the Map 2013 in Birmingham, which helped in shaping the details of project plan as well as some preliminary information gathering. The project team included myself, Vyron Antoniou, Sofia Basiouka, and Robert Soden (GFDRR). Later on, Peter Mooney (NUIM) and Jamal Jokar (Heidelberg) volunteered to help us – demonstrating the value in research networks such as COST ENERGIC which linked us.

The general methodology that we decided to use is the identification of case studies from across the world, at different scales of government (national, regional, local) and domains (emergency, environmental monitoring, education). We expected that with a large group of case studies, it will be possible to analyse common patterns and hopefully reach conclusions that can assist future projects. In addition, this will also be able to identify common barriers and challenges.

We have paid special attention to information flows between the public and the government, looking at cases where the government absorbed information that provided by the public, and also cases where two-way communication happened.

Originally, we were aiming to ‘crowdsource’  the collection of the case studies. We identified the information that is needed for the analysis by using  few case studies that we knew about, and constructing the way in which they will be represented in the final report. After constructing these ‘seed’ case study, we aimed to open the questionnaire to other people who will submit case studies. Unfortunately, the development of a case study proved to be too much effort, and we received only a small number of submissions through the website. However, throughout the study we continued to look out for cases and get all the information so we can compile them. By the end of April 2014 we have identified about 35 cases, but found clear and useful information only for 29 (which are all described in the report).  The cases range from basic mapping to citizen science. The analysis workshop was especially interesting, as it was carried out over a long Skype call, with members of the team in Germany, Greece, UK, Ireland and US (Colorado) while working together using Google Docs collaborative editing functionality. This approach proved successful and allowed us to complete the report.

You can download the full report from UCL Discovery repository

Or download a high resolution copy for printing and find much more information about the project on the Crowdsourcing and government website 

The INSPIRE 2014 conference marks the middle of the implementation process of  the INSPIRE directive (Infrastructure for Spatial Information in the European Community). The directive is aimed at establishing a pan-European Spatial Data Infrastructure (SDI), and that mean lots of blueprints, pipes, machine rooms and protocols for enabling the sharing of geographic information. In GIS jargon,  blueprints translate to metadata which is a standardise way to describe a GIS dataset; pipes and machine rooms translate to data portals and servers, and the protocols translate to web services that use known standards (here you’ll have a real acronym soup of WMS, WCS, WFS and OGC). It is all aimed to allow people across Europe to share data in an efficient way so data can be found and used. In principle, at least!

This is the stuff of governmental organisations that are producing the data (national mapping agencies, government offices, statistical offices etc.) and the whole INSPIRE language and aims are targeted at the producers of the information, encouraging them to publish information about their data and share it with others. A domain of well established bureaucracies (in the positive sense of the word) and organisations that are following internal procedure in producing, quality checking and distributing their information products. At first sight, seem like the opposite world of ‘upscience‘ where sometime there are only ad-hoc structures and activities.

That is why providing a talk in the plenary session that was dedicated to Governance and Information, and aimed to “assess how INSPIRE is contributing to a more effective and participated environmental policy in Europe, and how it provides connectivity with other policies affecting our environment, society, and the economy” was of concern.  So where are the meeting points of INSPIRE and citizen science? 

One option, is to try a top-down approach and force those who collect data to provide it in INSPIRE compliant way. Of course this is destined to fail. So the next option is to force the intermediaries to do the translation – and projects such as COBWEB is doing that, although it remain to be seen what compromises will be needed. Finally, there is an option to adapt and change procedures such as INSPIRE to reflect the change in the way the world works.

To prepare the talk, I teamed with Dr Claire Ellul, who specialises in metadata (among many other things) and knows about INSPIRE more than me.

The talk started with my previous work about the three eras of environmental information, noticing the move from data by experts, and for experts (1969-1992) to by experts & the public, for experts & the public (2012 on)

As the diagrams show, a major challenges of  INSPIRE is that it is a regulation that was created on the basis of the “first era” and “second era” and it inherently assumes stable institutional practices in creating and disseminating and sharing environmental information.

Alas, the world has changed – and one particular moment of change is August 2004 when OpenStreetMap started, so by the time INSPIRE came into force, crowdsourced geographic information and citizen science became legitimate part of the landscape. These data sources are coming from a completely different paradigm of production and management, and now, with 10 years of experience in OSM and growing understanding of citizen science data, we can notice the differences in production, organisation and practices. For example, while being very viable source of geographic information, OSM still doesn’t have an office and ‘someone to call’.

Furthermore, data quality methods also require different framing for these data.  We have metadata standards and quality standards that are assuming the second era, but we need to find ways to integrate into sharing frameworks like INSPIRE the messy, noisy but also rich and important data from citizen science and crowdsourcing.

Claire provided a case study that analyses the challenges in the area of metadata in particular. The case looks at different noise mapping sources and how the can be understood. Her analysis demonstrates how the ‘producer centric’ focus of INSPIRE is challenging when trying to create systems that record and use metadata for crowdsourced information. The case study is based on our own experiences over the past 6 years and in different projects, so there is information that is explicit in the map, some in a documentation – but some that is only hidden (e.g. calibration and quality of smart phone apps).

We conclude with the message that the INSPIRE community need to start noticing these sources of data and consider how they can be integrated in the overall infrastructure.

The slides from the talk are provided below.

 

Following the two previous assertions, namely that:

you can be supported by a huge crowd for a very short time, or by few for a long time, but you can’t have a huge crowd all of the time (unless data collection is passive)’ (original post here)

And

‘All information sources are heterogeneous, but some are more honest about it than others’  (original post here)

The third assertion is about pattern of participation. It is one that I’ve mentioned before and in some way it is a corollary of the two assertions above.

‘When looking at crowdsourced information, always keep participation inequality in mind’ 

Because crowdsourced information, either Volunteered Geographic Information or Citizen Science, is created through a socio-technical process, all too often it is easy to forget the social side – especially when you are looking at the information without the metadata of who collected it and when. So when working with OpenStreetMap data, or viewing the distribution of bird species in eBird (below), even though the data source is expected to be heterogeneous, each observation is treated as similar to other observation and assumed to be produced in a similar way.

Distribution of House Sparrow

Yet, data is not only heterogeneous in terms of consistency and coverage, it is also highly heterogeneous in terms of contribution. One of the most persistence findings from studies of various systems – for example in Wikipedia , OpenStreetMap and even in volunteer computing is that there is a very distinctive heterogeneity in contribution. The phenomena was term Participation Inequality by Jakob Nielsn in 2006 and it is summarised succinctly in the diagram below (from Visual Liberation blog) – very small number of contributors add most of the content, while most of the people that are involved in using the information will not contribute at all. Even when examining only those that actually contribute, in some project over 70% contribute only once, with a tiny minority contributing most of the information.

Participation Inequality Therefore, when looking at sources of information that were created through such process, it is critical to remember the nature of contribution. This has far reaching implications on quality as it is dependent on the expertise of the heavy contributors, on their spatial and temporal engagement, and even on their social interaction and practices (e.g. abrasive behaviour towards other participants).

Because of these factors, it is critical to remember the impact and implications of participation inequality on the analysis of the information. There will be some analysis to which it will have less impact and some where it will have major one. In either cases, it need to be taken into account.

Following the last post, which focused on an assertion about crowdsourced geographic information and citizen science I continue with another observation. As was noted in the previous post, these can be treated as ‘laws’ as they seem to emerge as common patterns from multiple projects in different areas of activity – from citizen science to crowdsourced geographic information. The first assertion was about the relationship between the number of volunteers who can participate in an activity and the amount of time and effort that they are expect to contribute.

This time, I look at one aspect of data quality, which is about consistency and coverage. Here the following assertion applies: 

‘All information sources are heterogeneous, but some are more honest about it than others’

What I mean by that is the on-going argument about authoritative and  crowdsourced  information sources (Flanagin and Metzger 2008 frequently come up in this context), which was also at the root of the Wikipedia vs. Britannica debate, and the mistrust in citizen science observations and the constant questioning if they can do ‘real research’

There are many aspects for these concerns, so the assertion deals with the aspects of comprehensiveness and consistency which are used as a reason to dismiss crowdsourced information when comparing them to authoritative data. However, at a closer look we can see that all these information sources are fundamentally heterogeneous. Despite of all the effort to define precisely standards for data collection in authoritative data, heterogeneity creeps in because of budget and time limitations, decisions about what is worthy to collect and how, and the clash between reality and the specifications. Here are two examples:

Take one of the Ordnance Survey Open Data sources – the map present themselves as consistent and covering the whole country in an orderly way. However, dig in to the details for the mapping, and you discover that the Ordnance Survey uses different standards for mapping urban, rural and remote areas. Yet, the derived products that are generalised and manipulated in various ways, such as Meridian or Vector Map District, do not provide a clear indication which parts originated from which scale – so the heterogeneity of the source disappeared in the final product.

The census is also heterogeneous, and it is a good case of specifications vs. reality. Not everyone fill in the forms and even with the best effort of enumerators it is impossible to collect all the data, and therefore statistical analysis and manipulation of the results are required to produce a well reasoned assessment of the population. This is expected, even though it is not always understood.

Therefore, even the best information sources that we accept as authoritative are heterogeneous, but as I’ve stated, they just not completely honest about it. The ONS doesn’t release the full original set of data before all the manipulations, nor completely disclose all the assumptions that went into reaching the final value. The Ordnance Survey doesn’t tag every line with metadata about the date of collection and scale.

Somewhat counter-intuitively, exactly because crowdsourced information is expected to be inconsistent, we approach it as such and ask questions about its fitness for use. So in that way it is more honest about the inherent heterogeneity.

Importantly, the assertion should not be taken to be dismissive of authoritative sources, or ignoring that the heterogeneity within crowdsources information sources is likely to be much higher than in authoritative ones. Of course all the investment in making things consistent and the effort to get universal coverage is indeed worth it, and it will be foolish and counterproductive to consider that such sources of information can be replaced as is suggest for the census or that it’s not worth investing in the Ordnance Survey to update the authoritative data sets.

Moreover, when commercial interests meet crowdsourced geographic information or citizen science, the ‘honesty’ disappear. For example, even though we know that Google Map Maker is now used in many part

s of the world (see the figure), even in cases when access to vector data is provided by Google, you cannot find out about who contribute, when and where. It is also presented as an authoritative source of information. 

Despite the risk of misinterpretation, the assertion can be useful as a reminder that the differences between authoritative and crowdsourced information are not as big as it may seem.

The Guardian Science Weekly podcast is dedicated to Citizen Science – another example of the growing interest in popular media in Citizen Science. However, the podcast conflate cases were non-professional scientists are involved in scientific project (and Chris Lintott discuss Galaxy Zoo, FoldIt and similar projects) with participation in scientific research through surveys. It is rather interesting that George MacKerron is usually explaining that Mappiness, despite the wide participation in it, is a social survey tool and not a citizen science project. It is also not strictly crowd-sourcing project, so calling the chronotype survey a crowd-sourced science, as the podcast does, is a bit of a hype…

Follow

Get every new post delivered to your Inbox.

Join 2,697 other followers