1. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement
Components and Functions of
Crowdsourcing Systems
A Systematic Literature Review
Lars Hetmank
Dresden, 27.02.2013

2. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Short Vita
• Dipl.-Wirt.-Inf. Lars Hetmank, TU Dresden
• Research interests:
• Information and Knowledge Management
• Semantic Web and Linked Data
• Crowdsourcing
• Affiliations:
• Manifesto for a Standard on Knowledge Exchange in
Social Knowledge Management Environments
• Knowledge Research Center (KRC) e.V. Dresden
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 2 / 13

3. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Motivation
Practical relevance:
• development of a crowdsourcing system (CSS) and
its integration into existing IT infrastructure is
risky, cost-intensive, and time-consuming,
• structured development process required
Theoretical relevance:
• various theoretical contribution in the area of
crowdsourcing
• however, little has been investigated to define a
crowdsourcing system and its technical design
aspects precisely
2/28/2013 Components and Functions of Crowdsourcing Systems Slide 3 / 13

4. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Method: Systematic Literature Review
Research Questions (RQs):
RQ1: How and in which detail are CSSs defined in current research
literature? What design aspects do they cover?
RQ2: What type of components and functions of a CSS can be
conceptualized?
2/28/2013 Components and Functions of Crowdsourcing Systems Slide 4 / 13

5. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Search Strategy #1
• Population: peer-reviewed conference proceedings
and journal papers in English since 2006;
dissertations, newspaper articles, unpublished works
or non-scientific articles were not considered
• Search Resources: ACM Digital Library, Ebscohost
(Academic Search Complete and Business Source
Complete), Emerald, IEEE Xplore Digital Library, Sage
Journals, ScienceDirect, SpringerLink and Wiley
• Search Terms: crowdsourcing system, crowdsourcing
application, and crowdsourcing platform (in singular
and plural)
2/28/2013 Components and Functions of Crowdsourcing Systems Slide 5 / 13

6. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Search Strategy #2
• Inclusion criteria:
• define or at least propose a description of what CSSs are
(RQ 1)
• address design issues of CSSs (RQ 2)
• or classify or give an overview of CSSs (RQ 2).
• Exclusion criteria:
• use CSSs for evaluation purposes but do not address design
issues
2/28/2013 Components and Functions of Crowdsourcing Systems Slide 6 / 13

7. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Conduct Search
Step 1: a) review the abstract, introduction, and conclusion
b) check inclusion and exclusion criteria
c) label publication and research type  Relevant studies = 72
Publication type Research type
Step 2: a) read article, find and record the definitions of CSS
b) set keyword for each article to identify components and
functions of CSSs
2/28/2013 Components and Functions of Crowdsourcing Systems Slide 7 / 13

8. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Results: Crowdsourcing System Definitions
(RQ1)
• 17 definitions or descriptions of the term crowdsourcing
system, application, or platform
• vary in detail and none of them covers all of the four
derived perspectives
“Crowdsourcing Platform is a trusted broker ensuring that providers
successfully complete the task requests and that requestors pay for the
charges. Crowdsourcing Platform issues authentication credentials for
requestors and providers when they join the platform, stores details about
skill-set, history of completed requests, handles charging and
payments, and manages platform misuse. Crowdsourcing platform can
execute crowdsourcing requests in a number of different modes, by advertising
them on the marketplace, allowing providers to bid for them, or in the form of
a competition, where requestor identifies criteria to be used for selection of the
winning submission. Crowdsourcing platform may further allow requestors and
providers to team-up.” (Vukovic, 2009)
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 8 / 13

9. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Perspectives
iv al
c t on
mediator, detailed actions
e
pr spe
pe t i
pe
rs iza
marketplace, performed to data
oc c
crowd
r
pe a n
es tive
interface, or objects or users
g
s
trusted broker or
broker crowdsourcing
process
requester
Crowd-
sourcing
user interface System algorithm
history tracking
standardization
payment mechanism
workflow support
iv ic
ct ntr
te rsp
social
pe
e
authentication
c h ec
pe e
rs n- c
interaction
ni tiv
APIs
ca e
pe a
software human
m
l
components, hu brainpower and
technical functions, collective
or data objects intelligence
Perspectives of crowdsourcing systems (source: own illustration)
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 9 / 13

10. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Results: Components and Functions (RQ 2)
Components and functions of crowdsourcing systems (source: own illustration)
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 10 / 13

11. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Conclusion and Next Steps
• development of an integrated CSS definition
• overview of reusable software components
and functions may be used as a checklist
• dependencies between the elements are not
well represented
• designing a semantic model to support
knowledge-intensive crowdsourcing
activities
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 11 / 13

12. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
References
(1) Kitchenham, B.: Guidelines for performing
Systematic Literature Reviews in Software
Engineering. (2007).
(2) Kitchenham, B., Pearl
Brereton, O., Budgen, D., Turner, M., Bailey, J.,
Linkman, S.: Systematic literature reviews in
software engineering – A systematic literature
review. Information and Software Technology.
51, 7–15 (2009).
(3) Brereton, P., Kitchenham, B.
a., Budgen, D., Turner, M., Khalil, M.: Lessons
from applying the systematic literature review
process within the software engineering
domain. Journal of Systems and Software.
80, 571–583 (2007).
(4) Wieringa, R., Maiden, N., Mead, N., Rolland, C.:
Requirements engineering paper classification
and evaluation criteria: a proposal and a
discussion. Requirements Engineering. 11, 102–
107 (2006).
(5) Bortz, J., Döring, N.: Forschungsmethoden und
Evaluation für Human- und
Sozialwissenschaftler. Springer, Heidelberg
(2009).
http://larshetmank.com/documents/wi2013_css_concept_map.pdf
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 12 / 13

13. Fakultät Wirtschaftswissenschaften | Lehrstuhl für Wirtschaftsinformatik, insbes. Informationsmanagement | Prof. Schoop
Contact
Email: lars@hetmank.de
Twitter: https://twitter.com/larshetmank
28.02.2013 Components and Functions of Crowdsourcing Systems Slide 13 / 13