Crowdsourcing involves breaking large tasks into smaller pieces that can be completed by members of the general public, often online. It provides benefits like lower costs, greater scale and access to citizen data. However, it also faces challenges like ensuring data quality and sustained participation. Well-designed projects and engagement are needed to motivate crowds and address issues. Quality assurance processes are important to build trust in citizen science.

1. What is Crowdsourcing?
Nicola Osborne
nicola.osborne@ed.ac.uk
Digital Education & Service Manager, EDINA
University of Edinburgh Crowd Sourcing & Citizen
Science Network

2. “collaborative production of knowledge and change”
(CSCS 2016)
citizen science n. scientific work undertaken by members of the general
public, often in collaboration with or under the direction of professional
scientists and scientific institutions.
(OED 2014)
• Generally consists of large specialist tasks broken into smaller non-
specialist tasks that anyone can do – especially in “Citizen Science”.
• Usually managed via online and/or mobile platforms but…
• Crowd sourcing is not a new concept…
• The first edition of the Oxford English Dictionary was crowdsourced
(Winchester 1998).
• RSPB Big Garden Bird Watch was conducted on paper for years
before online crowdsourcing emerged.
Crowdsourcing is…

3. • Citizen-led initiatives to gather data not otherwise
available/collected – e.g. OpenStreetMap, FixMyStreet.
• Research or analysis at scale – such as facilitated annotation,
analysis, pattern spotting etc. of large data sets, e.g. Old Weather.
• Research-led initiatives where crowd sourced data and/or analysis
has a clear research purpose, e.g. GalaxyZoo, UK Lamprey Watch.
• Collaborative creative projects, e.g. Wikipedia.
• Paid participation on bit-work basis, e.g. CrowdFlower, Amazon
Mechanical Turk
• Mining and analysis of available crowd sourced data sets–
Twitter, Flickr etc. e.g. WeSenseIt, Reading the London Riots
• Games where research tasks are abstracted, e.g. FoldIt
• On the ground reporting of in-the-moment events, e.g.
COBWEB Flooding, Guardian Witness
Crowdsourcing & Citizen Science can take many forms

4. Why Crowdsource?
• Cost – enables collection of data at scale
and (usually) without payment.
• Access – citizens on the ground are well
placed for incidents, unexpected events, or
to regularly access data about their
location.
• Engagement – makes your presence
visible; enables development of
relationships with communities; route to
engaging citizen in research and policy
making.
• Scale – some tasks are not realistic at
large scale for expert contributors.
• Serendipity – the crowd bring a fresh eye
and can find anomalies and make
unexpected connections

5. Challenges of crowdsourcing
• Time/Cost – to engage with a community takes authentic commitment,
substantial communication channels, significant time, appropriate
resourcing.
• Data Quality – isn’t guaranteed, addressed through training/guidance in
platform, simple interfaces, QA processes, comparison to authoritative data,
sound research design.
• Expectation Management – the crowd need to be motivated and
rewarded for participation, how will you do that but manage expectations of
change resulting from their participation?
• Participation Levels and Locations – to work crowd sourcing projects
have to attract a significantly large group of participants, in the right places,
and sustain their attention.
• Lack of Control – the crowd will not always work as you expect and may
develop its own priorities and etiquette.
• Legal Risk/Liability, Safety – especially in emergency situations. T&Cs
and warnings may be required.
• Data Visibility and Unexpected Implications – such as changes in
insurance premiums due to better understanding of flood area extent/risk;
exposure of location of at-risk species, etc.

6. Project Design and Engagement Matter
• Solid research design matters - hard to change approach part-way
through.
• Identifying audience(s)/community(ies) and level of
engagement desired helps you to tailor engagement.
• Communication is critical: feedback and reward mechanisms are
essential to ensure citizens remain motived.
• Technical considerations need to include; what is possible; what can
citizens access/use.
• Balance priorities: question/problem and data capture/analysis needs
to be useful for project team, but accessible and realistic for non-
specialists.
• Communities/participants require support and guidance.
• Quality assurance processes need to be established, whether through
metadata capture, comparison, manual checking, etc.
• Contingency planning – in case of poor participation, less successful
outcomes, disapointment/expectation management.
• Plan/mechanism to conclude, exit or handover project.

7. Technology and Tool Selection
• Impacts accuracy of data collected (e.g. volunteered vs. GPS
vs. IP based location).
• Interface design, usability, familiarity and language - can
ease or increase complexity of participation.
• Home computer based tasks can exclude participation by e.g.
lower income, older, less able, or more remotely located
communities (see RSE 2014).
• Mobile devices limit to those with appropriate devices, often
also limited to those with iOS/Android/etc. Also require wifi
or 3G signals, or apps/interfaces which can function offline.
• App and data upload size may mean volunteers incur costs
when participating.

8. Quality Assurance
Trust in Citizen Science by researchers and
policy makers varies, quality impacts re/use.
• Quality Assurance method/process :
– Manual/moderation (e.g. Conker Tree
Science);
– Repetition and redundancy (e.g. Galaxy
Zoo);
– Targeted comparison with trusted data
(e.g. sensors);
– Technical measures (e.g. location of
submission);
– Trust/expertise level (e.g. based on
previous submissions).
• Raises social challenges of managing
potential rejection of volunteered
data/effort.
• See Chapman 2015.

9. Questions/Comments?

10. References & Resources
• Chapman, C. 2015. COBWEB D.2.2 Value Adding to Crowdsourced Data for Decision
Making. COBWEB Project. Available from:
https://cobwebproject.eu/news/publications/deliverables
• Dunn, S. and Hedges, M., 2012. Connected Communities: Crowdsourcing in the
humanities, a scoping study. AHRC and Kings College London. Available from:
http://www.ahrc.ac.uk/documents/project-reports-and-reviews/connected-
communities/crowd-sourcing-in-the-humanities/
• Graham, G.G., Cox, J., Simmons, B., Lintott, C., Masters, K., Greenhill, A. and
Holmes, K, 2015. How is success defined and measured in online citizen science: a
case study of Zooniverse projects. Available from:
http://eprints.whiterose.ac.uk/86535/
• Roy, H. et al, 2012. Understanding Citizen Science for Environmental Monitoring.
CEH. Available from:
http://www.ceh.ac.uk/news/news_archive/documents/understandingcitizenscienceen
vironmentalmonitoring_report_final.pdf
• Royal Society of Edinburgh, 2014. Spreading the Benefits of Digital Participation in
Scotland Final Report. Royal Society of Edinburgh. Available from:
http://www.royalsoced.org.uk/1058_SpreadingtheBenefitsofDigitalParticipation.html
• Winchester, S., 1998. The Professor and the Madman: A tale of murder, insanity and
the making of the Oxford English Dictionary. HarperCollins Publishers.

11. Crowd Sourcing & Citizen Science Resources
• European Citizen Science Association:
http://ecsa.biodiv.naturkundemuseum-berlin.de/
• UK-EOF’s Understanding Citizen Science and Environmental
Monitoring:
http://www.ceh.ac.uk/products/publications/understanding-citizen-
science.html
• SWE4Citizen Science – OGC Standards for Citizen Science Projects:
https://cobwebproject.eu/about/swe4citizenscience-github
• University of Edinburgh Crowd Sourcing & Citizen Science Network:
http://citsci.ed.ac.uk/
• COBWEB: Citizen Observatory Web Project – an EU-standards
compliant citizen science infrastructure, including flooding
demonstrators and QA work: https://cobwebproject.eu/
• Crowd Flower: https://www.crowdflower.com/
• Edinburgh CityScope – infrastructure for open creation and sharing of
data, including crowd sourcing projects:
http://www.edinburghcityscope.org/

12. Citizen Science Projects and Tools Mentioned Here
• Amazon Mechanical Turk: https://www.mturk.com/mturk/welcome
• Click to Cure: http://www.clicktocure.net/
• FieldTrip GB – mobile app for small scale crowdsourcing:
http://fieldtripgb.blogs.edina.ac.uk/
• FoldIt: https://fold.it/portal/
• GalaxyZoo: http://www.galaxyzoo.org/
• Guardian Witness:
http://www.theguardian.com/community/series/guardianwitness
• Leafwatch: Conker Tree Science: http://naturelocator.org/leafwatch.html
• Old Weather: https://www.oldweather.org/#/
• OPAL: http://www.opalexplorenature.org/
• OpenStreetMap: http://www.openstreetmap.org/
• RSPB Big Garden Bird Watch: http://www.rspb.org.uk/birdwatch/
• UK Lamprey Watch: https://envscot-csportal.org.uk/lampreywatch/
• WeSenseIt: http://wesenseit.eu/
• Wikipedia: https://en.wikipedia.org/wiki/Main_Page