Introdução sobre Revisão Sistemática da Literatura

1. Systematic Literature
Review
Prof. Dr.Tiago Silva da Silva
(ICT/UNIFESP)
silvadasilva@unifesp.br

2. Literature Review

3. Literature Review
• Experts can be wrong.

4. Literature Review
• Experts can be wrong.
• Researcher’s choice of “related studies” can
be biased.

5. Literature Review
• Experts can be wrong.
• Researcher’s choice of “related studies” can
be biased.
• Informal reviews can miss important
studies.

6. Systematic Literature
Review
• Systematic Review
• Medical domain
• Evidence-based software engineering
(Kitchenham)
• “We cannot have evidence-based software
engineering without a sound methodology for
aggregating evidence from different empirical
studies. SRs provide that methodology.”

7. Systematic Literature
Review
• The major advantage of a SR is that it is based on a
well-defined methodology.
• SRs aim to find, assess, and aggregate all relevant
evidence about some topic of interest in a fair,
repeatable, and auditable manner.
• A researcher conducting a SR selects empirical
studies that are relevant to the particular research
question, assesses the validity of each one, and then
determines the trend shown by those studies.

8. Systematic Literature
Review
• To investigate all available evidence that supports or refutes a
particular “topic of interest”.
• To summarize the existing evidence concerning a treatment or
technology, e.g. to summarize the empirical evidence of the benefits
and limitations of a specific agile method.
• To identify any gaps in current research in order to suggest areas
for further investigation.
• To provide a framework/background in order to appropriately
position new research activities.
• Sometimes a single researcher, such as a PhD student, needs to do
this activity alone, but that limitation raises the risk of missing
relevant papers.

9. Advantages

10. Advantages
• The well-defined methodology makes it
less likely that the results of the literature
are biased, although it does not protect
against publication bias in the primary
studies.

11. Advantages
• The well-defined methodology makes it less likely that
the results of the literature are biased, although it does
not protect against publication bias in the primary
studies.
• They can provide information about the effects of some
phenomenon across a wide range of settings and
empirical methods. If studies give consistent results,
systematic reviews provide evidence that the
phenomenon is robust and transferable. If the studies
give inconsistent results, sources of variation can be
studied.

12. Disadvantage

13. Disadvantage
• Requires considerably more effort than
traditional literature reviews.

14. Systematic Literature
Review
• Planning the review
• Conducting the review
• Reporting the review

15. Systematic Literature
Review
Systematic Review in Software Engineering Technical Report ES 679/05
articles are extracted and synthesized during the result analysis phase. Meanwhile
which one of these phases is executed, their results must be stored. Therefore,
systematic review packaging is performed through the whole process. There are two
checkpoints in the proposed systematic review process. Before executing the systematic
review, it is necessary to guarantee that the planning is suitable. The protocol must be
evaluated and if problems are found, the researcher must return to the planning stage to
review the protocol. Similarly, if problems regarding web search engines are found
during the execution phase, the systematic review may be re-executed.
Figure 2. Systematic Review Process.
The stages listed above may appear to be sequential, but it is important to recognize
that many of the stages involve iteration. In particular, many activities are initiated
during the protocol development stage, and refined when the review proper takes place.
For example [Kitchenham et al., 2002]:
Planning Execution Result Analisys
Packaging
[ protocol plan approved ]
[ protocol plan disapproved ]
[ execution approved ]
[ execution disapproved ]
Planning Conducting Reporting

16. Systematic Literature
Review
e been iden-
d these have
dence-based
nham et al.,
technique,
;
he question;
y (closeness
pplicability
engineering
alues and
n executing
titute a sys-
rder to pro-
relevant to
systematic
d interpret-
ar research
Watson, 2002), they are less common in software engineer-
ing (however, see (Glass et al., 2002) as an example of a sec-
ondary study that samples literature within the software
engineering domain). Indeed, at the present time, outside
of information systems research, reviews in any form, as
well as review journals are really not part of the computing
9. Write Review Report
10. Validate Report
2. Develop Review Protocol
3. Validate Review Protocol
Phase 1:
Plan Review
Phase 2:
Conduct Review
Phase 3:
Document Review
8. Synthesise Data
4. Identify Relevant Research
5. Select Primary Studies
7. Extract Required Data
6. Assess Study Quality
1. Specify Research Questions
Fig. 1. Systematic literature review process.

17. Planning the Review
1. Identification of the need for a review
2. Commissioning a review*
3. Specifying the research question(s)
4. Developing a review protocol
5. Evaluating the review protocol*

18. Conducting the Review
1. Identification of research
2. Selection of primary studies
3. Study quality assessment
4. Data extraction and monitoring
5. Data synthesis

19. Reporting the Review
1. Specifying dissemination mechanisms
2. Formatting the main report
3. Evaluating the report*

20. Planning

21. The need for a
systematic review
• What are the review’s objectives?
• What sources were searched to identify
primary studies? Were there any restrictions?
• What were the inclusion/exclusion criteria
and how were they applied?
• What criteria were used to assess the quality
of primary studies?

22. The need for a
systematic review
• How were quality criteria applied?
• How were the data extracted from the primary
studies?
• How were the data synthesized?
• How were differences between studies investigated?
• How were the data combined?
• Was it reasonable to combine the studies?
• Do the conclusions flow from the evidence?

23. The Research
Question(s)
• The most important part of any systematic review.
The question(s) drive(s) the entire methodology:
• The search process must identify primary
studies that address the research questions.
• The data extraction process must extract the
data items needed to answer the questions.
• The data analysis process must synthesize the
data in such a way that the questions can be
answered.

24. The Research
Question(s)
• Assessing the effect of a technology.
• Assessing the frequency or rate of a project development
factor such as the adoption of a technology, or the
frequency or rate of project success or failure.
• Identifying cost and risk factors associated with a
technology.
• Identifying the impact of technologies on reliability,
performance and cost models.
• Cost benefit analysis of employing specific software
development technologies or software applications.

25. The Research
Question(s) example
• Question 1: What evidence is there that cross-company
estimation models are not significantly different from within-
company estimation models for predicting effort for software/
Web projects?
• Question 2: What characteristics of the study data sets and
the data analysis methods used in the study affect the
outcome of within- and cross-company effort estimation
accuracy studies?
• Question 3: Which experimental procedure is most
appropriate for studies comparing within- and cross-company
estimation models?

26. The Research
Question(s) structure
• Population: software or web-project.
• Intervention: cross-company project
effort estimation model.
• Comparison: single-company project
effort estimation model.
• Outcomes: prediction or estimate
accuracy.

27. Developing a Review
Protocol
• A review protocol specifies the methods that will
be used to undertake a specific systematic review
• Background: the rationale for the survey.
• The research questions that the review is
intended to answer.
• The strategy that will be used to search for
primary studies including search terms and
resources to be searched: digital libraries, specific
journals and proceedings.

28. Developing a Review
Protocol
• Study selection criteria: used to
determine which studies are included in, or
excluded from.
• Study selection procedures: how the
selection criteria will be applied e.g. how
many assessors will evaluate each
prospective primary study, and how
disagreements among assessors will be
resolved.

29. Developing a Review
Protocol
• Study quality assessment checklists and
procedures: quality checklists to assess the
individual studies.
• Data extraction strategy: how the information
required from each primary study will be
obtained.
• Synthesis of the extracted data: defines the
synthesis strategy.
• Dissemination strategy

30. Conducting

31. Identification of
Research
• Generating a search strategy
• Preliminary searches aimed at both identifying
existing systematic reviews and assessing the
volume of potentially relevant studies.
• Trial searches using various combinations of
search terms derived from the research
question.
• Checking trial research strings against lists of
already known primary studies.
• Consultations with experts in the field.

32. Identification of
Research
• Publication bias
• The problem that positive results are
more likely to be published than
negative results.The concept of positive
or negative results sometimes depends
on the viewpoint of the researcher.
• Scanning the grey literature
• Scanning conference proceedings

33. • Bibliography Management and Document Retrieval
• Reference Manager (http://www.refman.com/)
• Endnote (http://endnote.com/)
• Mendeley (http://www.mendeley.com/)
• Papers (http://papersapp.com/mac/)
• Documenting the Search (The process of
performing a systematic literature review must be
transparent and replicable (as far as possible)
Identification of
Research

34. Identification of
Research
literature search.
Once reference lists have been finalised the full articles of potentially useful studies
will need to be obtained. A logging system is needed to make sure all relevant studies
are obtained.
6.1.4 Documenting the Search
The process of performing a systematic literature review must be transparent and
replicable (as far as possible):
The review must be documented in sufficient detail for readers to be able to
assess the thoroughness of the search.
The search should be documented as it occurs and changes noted and justified.
The unfiltered search results should be saved and retained for possible reanalysis.
Procedures for documenting the search process are given in Table 2.
Table 2 Search process documentation
Data Source Documentation
Digital Library Name of database
Search strategy for the database
Date of search
Years covered by search
Journal Hand Searches Name of journal
Years searched
Any issues not searched
Conference proceedings Title of proceedings
Name of conference (if different)
Title translation (if necessary)
Journal name (if published as part of a journal)
Efforts to identify
unpublished studies
Research groups and researchers contacted (Names and contact details)
Research web sites searched (Date and URL)
Other sources Date Searched/Contacted
URL
Any specific conditions pertaining to the search
• ACM Digital Library (http://
dl.acm.org/)
• IEEE Xplore (http://
ieeexplore.ieee.org/)
• Science Direct (http://
www.sciencedirect.com/)
• ISI Web of Science (http://
isiknowledge.com/)
• Scopus (http://www.scopus.com/)
• Google Scholar*

35. Study Selection
• Study selection criteria: to identify those primary studies that provide
direct evidence about the research question. Inclusion and exclusion
criteria should be based on the research question.
• Inclusion:
• any study that compared predictions of cross-company models with
within-company models based on analysis of single company project
data.
• Exclusion:
• studies where projects were only collected from a small number of
different sources (e.g. 2 or 3 companies),
• studies where models derived from a within-company data set were
compared with predictions from a general cost estimation model.

36. Study Selection
• Study selection process:
• Language
• Journal
• Authors
• Setting
• Participants or subjects
• Research Design
• Date of publication
• Reliability of inclusion decisions:
• When two or more researchers assess each paper, agreement
between researchers can be measured using the Cohen Kappa statistic.

37. Identify relevant studies -
automated and manual
search
Exclude studies on the basis
of Title
Exclude studies on the basis
of Abstract
Obtain primary papers and
critically appraise studies
n papers
n papers
n papers
n papers

38. Study Quality
Assessment
• To provide still more detailed inclusion/exclusion
criteria.
• To investigate whether quality differences provide an
explanation for differences in study results.
• As a means of weighting the importance of individual
studies when results are being synthesized.
• To guide the interpretation of findings and
determine the strength of inferences.
• To guide recommendations for further research.

39. • Development of Quality Instruments
• Design
• Conduct
• Analysis
• Conclusions
• Example:
1.Is the data analysis process appropriate?
1.1.Was the data investigated to identify outliers and to assess
distributional properties before analysis?
1.2.Was the result of the investigation used appropriately to transform the
data and select appropriate data points?
Study Quality
Assessment

40. • Less than 10 projects: Poor quality (score=0)
• Between 10 and 20 projects: Fair quality (score=0.33)
• Between 21 and 40 projects: Good quality (score=0.67)
• More than 40 projects: Excellent quality (score=1)
Study Quality
Assessment example

41. Data Extraction
• Design of Data Extraction Forms
• Content of Data Collection Forms
Electronic forms are useful and can facilitate subsequent analysis.
6.4.2 Contents of Data Collection Forms
In addition to including all the questions needed to answer the review question and
quality evaluation criteria, data collection forms should provide standard information
including:
Name of Reviewer
Date of Data extraction
Title, authors, journal, publication details
Space for additional notes
Examples
Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also
included the quality questions).
Table 7 Data Collection form completed for Maxwell et al., 1998
Data item Value Additional notes
Data Extractor
Data Checker
Study Identifier S1
Application domain Space, military and industrial
Name of database European Space Agency (ESA)
Number of projects in
database (including within-
company projects)
108
Number of cross-company
projects
60
Number of projects in within-
company data set
29
Size metric(s):
FP (Yes/No)
Version used:
LOC (Yes/No)
Version used:
FP: No
LOC: Yes (KLOC)
Others: No

42. Data Extraction
• Data extraction procedures
• Whenever feasible, data extraction should be performed
independently by two or more researchers. Data from the
researchers must be compared and disagreements resolved
either by consensus among researchers or arbitration by an
additional independent researcher.
• Multiple publications of the same data
• It is important not to include multiple publications of the
same data in a systematic review synthesis because duplicate
reports would seriously bias any results.When there are
duplicate publications, the most complete should be used.

43. Data Synthesis
• Unpublished data, missing data and data requiring
manipulation
• Collating and summarizing the results of the included
primary studies.The data synthesis activities should
be specified in the review protocol.
• Descriptive (Narrative) Synthesis
• Extracted information about the studies (i.e.
intervention, population, context, sample sizes,
outcomes, study quality) should be tabulated in a
manner consistent with the review question.

44. Data Synthesis
• Quantitative Synthesis: quantitative data should also
be presented in tabular form including:
• Sample size for each intervention.
• Estimates effect size for each intervention with
standard errors for each effect.
• Difference between the mean values for each
intervention, and the confidence
interval for the difference.
• Units used for measuring the effect.

45. Data Synthesis
• Presentation of Quantitative Results
• Charts x Tables
• Qualitative Synthesis
• Tries to integrate studies comprising natural language results and
conclusions, where different researchers may have used terms and
concepts with subtly (or grossly) different meanings
• Synthesis of Qualitative and Quantitative Synthesis
• Synthesize the quantitative and qualitative studies separately.
• Then attempt to integrate the qualitative and quantitative results by
investigating whether the qualitative results can help explain the
quantitative results. For example qualitative studies can suggest reasons
why a treatment does or does not work in specific circumstances.

46. Data Synthesis
• Sensitivity Analysis
• High quality primary studies only.
• Primary studies of particular types.
• Primary studies for which data extraction
presented no difficulties (i.e. excluding any
studies where there was some residual
disagreement about the data extracted).
• The experimental method used by the primary
studies.

47. Reporting

48. Reporting
• Specifying the Dissemination Strategy
• Journals
• Conferences
• Formatting the Main Systematic Review Report
• Technical Report
• Section of a thesis
• Journal or conference format
• Evaluating Systematic Review Reports

49. Problems associated
with conducting a SLR

50. Problems associated
with conducting a SLR
• The protocol may not have identified all the variations in the
designing and reporting of relevant primary studies.
• Selection of the digital libraries to be searched and whether
the search is automated or manual.
• A manual search involves looking at the back issues of a
set of journals (either paper or online) and deciding from
the title and abstract which individual papers are
candidates for inclusion in the review.
• An automated search uses strings, usually based on
complex Boolean formulas, to turn up papers in online
catalogs.

51. (software OR application OR product OR Web OR WWW OR Internet OR
World-Wide Web OR project OR development) AND (method OR process OR
system OR technique OR methodology OR procedure) AND (cross company
OR cross organisation OR cross organization OR cross organizational OR
cross organisational OR cross- company OR cross-organisation OR cross-
organization OR cross-organizational OR cross-organisational OR multi
company OR multi organisation OR multi organization OR multi organizational
OR multi organisational OR multi- company OR multi-organisation OR multi-
organization OR multi-organizational OR multi-organisational OR multiple
company OR multiple organisation OR multiple organization OR multiple
organizational OR multiple organisational OR multiple-company OR multiple-
organisation OR multiple-organization OR multiple-organizational OR multiple-
organisational OR within company OR within organisation OR within
organization OR within organizational OR within organisational OR within-
company OR within-organisation OR within-organization OR within-
organizational OR within-organisational OR single company OR single
organisation OR single organization OR single organizational OR single
organisational OR single-company OR single-organisation OR single-
organization OR single-organizational OR single-organisational OR company-
specific) AND (model OR modeling OR modelling) AND (effort OR cost OR
resource) AND (estimation OR prediction OR assessment)

52. Problems associated
with conducting a SLR
• Digital libraries differ in subtle ways in their implementations of
searches:
• Digital libraries have different interfaces and different
tolerances for the complex Boolean formulas typically used by
SR researchers to turn up relevant papers.
• Digital libraries also have different procedures for searching the
body of the paper, or only the title, abstract, and keywords. In
addition, of course, indexing systems only search titles,
keywords, and abstracts.
• Automated searches from different sources may overlap (i.e.,
the same paper may be found in different digital libraries), and
every search includes a large number of irrelevant papers .

53. Problems associated
with conducting a SLR
• Digital libraries differ in subtle ways in their implementations of searches:
• (TITLE-ABS-KEY(usability) OR TITLE-ABS-KEY("human-computer interaction") OR
TITLE-ABS-KEY("computer-human interaction") OR TITLE-ABS-KEY("human factor") OR
TITLE-ABS-KEY("user experience") OR TITLE-ABS-KEY("user-centered design")) AND
(TITLE-ABS-KEY(agile) OR TITLE-ABS-KEY("agile method") OR TITLE-ABS-KEY("agile
development") OR TITLE-ABS-KEY("agile practice") OR TITLE-ABS-KEY("agile project")
OR TITLE-ABS-KEY("agile lifecycle") OR TITLE-ABS-KEY(scrum) OR TITLE-ABS-
KEY("extreme programming") OR TITLE-ABS-KEY("lean development") OR TITLE-ABS-
KEY("feature driven development") OR TITLE-ABS-KEY("dynamic system development
method") OR TITLE-ABS-KEY("agile unified process")) AND PUBYEAR AFT 2000 AND
(LIMIT-TO(SUBJAREA, "COMP") OR LIMIT-TO(SUBJAREA, "MULT"))
• (usability OR "human-computer interaction" OR "computer-human interaction" OR
"human factor" OR "user experience" OR "user-centered design") AND (agile OR "agile
method" OR "agile development" OR "agile practice" OR "agile project" OR "agile
lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature
driven development" OR "dynamic system development method" OR "agile unified
process")

54. Problems associated
with conducting a SLR
• For manual searches:
• Select the journals and conference proceedings you intend to search.
• Justify your selection of sources. However, rather surprisingly, in one
case study we found that a targeted manual search was much quicker
than a broad automated search.
• In practice, need for a mixed strategy.
• If you do a manual search of some sources (including specialist
conference proceedings), this should give you a baseline set of
candidate primary studies against which you can validate the efficacy
of your automated search strings.
• Alternatively, a domain expert can identify a baseline set of papers.

55. Other types of reviews
• Systematic Mapping Study: a broad review of
primary studies in a specific topic area that
aims to identify what evidence is available
on the topic.
• Tertiary Study: a review of secondary studies
related to the same research question.

56. Lessons Learned
e been iden-
d these have
dence-based
nham et al.,
technique,
;
he question;
y (closeness
pplicability
engineering
alues and
n executing
titute a sys-
rder to pro-
relevant to
systematic
d interpret-
ar research
Watson, 2002), they are less common in software engineer-
ing (however, see (Glass et al., 2002) as an example of a sec-
ondary study that samples literature within the software
engineering domain). Indeed, at the present time, outside
of information systems research, reviews in any form, as
well as review journals are really not part of the computing
9. Write Review Report
10. Validate Report
2. Develop Review Protocol
3. Validate Review Protocol
Phase 1:
Plan Review
Phase 2:
Conduct Review
Phase 3:
Document Review
8. Synthesise Data
4. Identify Relevant Research
5. Select Primary Studies
7. Extract Required Data
6. Assess Study Quality
1. Specify Research Questions
Fig. 1. Systematic literature review process.

57. Lessons Learned
• Stage 1: Specify Research Questions
• L1: Expect to revise your questions during
protocol development, as your understanding
of the problem increases.
• L2: A pre-review mapping study may help in
scoping research questions.

58. Lessons Learned
• Stage 2: Develop Review Protocol
• L3: All systematic review team members need to
take an active part in developing the review
protocol.
• L4: Piloting the research protocol is essential. It
will find mistakes in your data collection and
aggregation procedures. It may also indicate that
you need to change the methodology you intend
to use to address the research questions.

59. Lessons Learned
• Stage 3:Validate Review Protocol
• L5: Data extraction is assisted by having data
definitions and data extraction guidelines from the
protocol recorded in a separate short document.
• L6: There needs to be an agreed validation
process separate from the protocol piloting activity.
Ideally, external reviewers should undertake this
validation process.

60. Lessons Learned
• Stage 4: Identify Relevant Research
• L7: There are alternative search strategies that enable you to
achieve different sort of search completion criteria.You must
select and justify a search strategy that is appropriate for your
research question.
• L8: We need to search many different electronic sources; no
single source finds all of the primary studies.
• L9: Current software engineering search engines are not
designed to support systematic literature reviews. Unlike
medical researchers, software engineering researchers need to
perform resource-dependent searches.

61. Lessons Learned
• Stage 5: Select Primary Studies
• L10: The standard of IT and software
engineering abstracts is too poor to rely on
when selecting primary studies.You should
also review the conclusions.

62. Lessons Learned
• Stage 6:Assess Study Quality
• L11: All the medical standards emphasise that
it is necessary to assess the quality of primary
studies. However, it depends on the type of
systematic literature review you are undertaking.
• L12: It is important to be sure how the quality
assessment will be used in the subsequent data
aggregation and analysis.

63. Lessons Learned
• Stage 7: Extract Required Data
• L13: Having one reader act as data
extractor and one act as data checker may
be helpful when there are a large number of
papers to review.
• L14: Review team members must make
sure they understand the protocol and the
data extraction process.

64. Lessons Learned
• Stage 8: Synthesise Data
• L15: Software engineering systematic reviews are likely
to be qualitative in nature.
• L16: Even when collecting quantitative information it
may not be possible to perform meta-analysis of software
engineering studies because the reporting protocols vary
so much from study to study.
• L17: Tabulating the data is a useful means of
aggregation but it is necessary to explain how the
aggregated data actually answers the research questions.

65. Lessons Learned
• Stage 9: Write Review Report
• L18: Review teams need to keep a detailed record of
decisions made throughout the review process.
• L19: The software engineering community needs to
establish mechanisms for publishing systematic
literature reviews which may result in papers that are
longer than those traditionally accepted by many
software engineering outlets or that have appendices
stored in electronic repositories.

66. Lessons Learned
• Stage 10: Validate Report

67. References
• Kitchenham, B.. (2004). Procedures for Performing Systematic Reviews. Joint Technical Report.
• Biolchini, J.; Mian, P. G.; Natali,A. C. C.;Travassos, G. H.T.. (2005) .Systematic Review in Software
Engineering.Technical Report.
• Kitchneham, B.. (2007). Guidelines for performing Systematic Literature Reviews in Software
Engineering. EBSE Technical Report.
• Brereton, P.; Kitchenham, B.; Budgen, D.;Turner, M.; Khalil, M.. (2008). Lessons from applying the
systematic literature review process within the software engineering domain.The Journal of Systems
and Software.
• Dyba ̊,T.; Dingsøyr,T.. (2008). Empirical studies of agile software development:A systematic review.
Information and Software Technology.
• Kitchenham, B.; Brereton, P.; Budgen, D.;Turner, M.; Bailey, J.; Linkman, S.. (2009). Systematic literature
reviews in software engineering – A systematic literature review. Information and Software
Technology.
• Kitchenham, B.; Pretorius, R.; Budgen, D.; Brereton, P.; Turner, M.; Niazi, M.; Linkman, S.. (2010).
Systematic literature reviews in software engineering – A tertiary study. Information and Software
Technology.
• Kitchenham, B.; Mendes, E.;Travassos, G. H.T..(2007) A Systematic Review of Cross- vs.Within-
Company Cost Estimation Studies, IEEE Trans on SE.

68. Systematic Literature
Review
Prof. Dr.Tiago Silva da Silva
(ICT/UNIFESP)
silvadasilva@unifesp.br

70. User-Centered Design and
Agile Methods:
A Systematic Review
Tiago Silva da Silva,Angela Martin,
Frank Maurer, Milene Silveira
To contact me after my
presentation, text BDL to
INTRO (46876)

71. Defining the terms...

72. Category Keyword
UCD
Usability
Human-Computer Interatcion
Computer-Human Interaction
Human Factor
User Experience
User-Centered Design
User Interface
Agile
Agile
Agile Method
Agile Development
Agile Practice
Agile Project
Agile Lifecycle
Scrum
Extreme Programming
Lean Development
Feature Driven Development
Dynamic System Development
Agile Unified Process

73. Search String
usability OR "human-computer interaction" OR "computer-human
interaction" OR "human factor" OR "user experience" OR "user-
centered design" OR "user interface"
AND
agile OR "agile method" OR "agile development" OR "agile practice"
OR "agile project" OR "agile lifecycle" OR scrum OR "extreme
programming" OR "lean development" OR "feature driven
development" OR "dynamic system development" OR "agile unified
process"

74. Inclusion
Exclusion

75. Digital Library
Amount of
papers
First Classification
Inclusion Exclusion
IEEExplore 59 24 35
Science Direct 4 0 4
CiteSeerX 59 0 59
Scopus 154 24 130
Agile 21 21 0
XP 12 12 0
Total 309 81 228
Percentage 100% 26% 74%

76. Digital Library
Amount
of
papers
Second Classification
Total
Selected
Repeated
Not
Relevant
Total 81 16 7 58
Percentage 100% 20% 8% 72%

77. 309

78. 309
81

79. 309
81
58

82. Descriptive Information
Content-related Information

83. Descriptive Information

85. Theoretical Experience Report Empirical Experimental

86. Focus

89. Approach

92. Results

95. Artefacts, Practices,
Needs...

100. Exploring principles of user-centered agile software development: A
literature review
Manuel Brhel a
, Hendrik Meth b
, Alexander Maedche a,b,⇑
, Karl Werder a
a
Chair of Information Systems IV, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany
b
Institute for Enterprise Systems, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany
a r t i c l e i n f o
Article history:
Received 5 September 2013
Received in revised form 7 January 2015
Accepted 7 January 2015
Available online 17 January 2015
Keywords:
Agile software development
User-centered design
Systematic literature review
a b s t r a c t
Context: In the last decade, software development has been characterized by two major approaches: agile
software development, which aims to achieve increased velocity and flexibility during the development
process, and user-centered design, which places the goals and needs of the system’s end-users at the cen-
ter of software development in order to deliver software with appropriate usability. Hybrid development
models, referred to as user-centered agile software development (UCASD) in this article, propose to com-
bine the merits of both approaches in order to design software that is both useful and usable.
Objective: This paper aims to capture the current state of the art in UCASD approaches and to derive gen-
eric principles from these approaches. More specifically, we investigate the following research question:
Which principles constitute a user-centered agile software development approach?
Method: We conduct a systematic review of the literature on UCASD. Identified works are analyzed using
a coding scheme that differentiates four levels of UCASD: the process, practices, people/social and tech-
nology dimensions. Through subsequent synthesis, we derive generic principles of UCASD.
Results: We identified and analyzed 83 relevant publications. The analysis resulted in a comprehensive
coding system and five principles for UCASD: (1) separate product discovery and product creation, (2)
iterative and incremental design and development, (3) parallel interwoven creation tracks, (4) continuous
stakeholder involvement, and (5) artifact-mediated communication.
Conclusion: Our paper contributes to the software development body of knowledge by (1) providing a
broad overview of existing works in the area of UCASD, (2) deriving an analysis framework (in form a cod-
ing system) for works in this area, going beyond former classifications, and (3) identifying generic prin-
ciples of UCASD and associating them with specific practices and processes.
Ó 2015 Published by Elsevier B.V.
Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
2. Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
2.1. Summary of existing literature reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
2.2. Gap analysis of existing literature reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
3. Research method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Information and Software Technology 61 (2015) 163–181
Contents lists available at ScienceDirect
Information and Software Technology
journal homepage: www.elsevier.com/locate/infsof

101. (4) A data extraction and synthesis method, representing the sys-
tematic approach to consolidate and integrate existing
knowledge.
The review protocol was developed in cooperation with the first
and second authors, and validated by the third author prior to con-
ducting the review. In the following, we describe the implementa-
tion of the review protocol.
in this stage as depicted in Fig. 1. In the first stage, relevant publi-
cations were retrieved by querying the databases mentioned above
with the search string depicted in Table 1. All database queries
were made in the first two weeks of October 2012. This yielded a
total of 1152 initial results, and 1034 publications after the
removal of duplicates, which were included in the second stage.
In the second stage, publications were excluded based on their
titles and abstract. Criteria for inclusion in the following stage were
Table 1
Composition of the search string.
AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,
interface design
OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development
OR Software development, systems development
Fig. 1. Selection process (based on [31]).
166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
(4) A data extraction and synthesis method, representing the sys- in this stage as depicted in Fig. 1. In the first stage, relevant publi-
Table 1
Composition of the search string.
AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,
interface design
OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development
OR Software development, systems development
Fig. 1. Selection process (based on [31]).
166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

102. on these criteria, 74 articles were considered relevant for inclusion
in the next stage. Following the strategy suggested by Webster and
Watson [36], we conducted a forward and backward search based
on these key publications. The retrieved new and potentially rele-
vant articles were fed to the second stage for further processing,
resulting in an iterative extension of the sample. This resulted in
an additional set of nine articles for inclusion in stage four. It has
to be noted that both da Silva et al.’s [20] and Sohaib and Khan’s
[23] reviews were included in the stage three sample. However,
they were excluded from the detailed analysis in stage four as they
do not present the results of primary research. In total, 83 papers
were selected for the final sample.1
In the fourth stage, the final sample passed a first categorization
and a quality assessment. During the categorization, each paper
was assigned to one of the four research foci, which will be intro-
duced in the next section. A quality assessment of each paper was
subsequently conducted to obtain additional information support-
ing the interpretation of the paper’s recommendations. This assess-
ment was based on four questions, which are given in Table 2. Each
of the four criteria was graded on a dichotomous scale as ‘‘yes’’ or
‘‘no,’’ while question 3b was answered by means of the respective
method, which was either explicitly stated in the paper or deter-
mined by the researcher based on the available information.
3.3. Paper analysis
Barksdale and McCrickard [22], we did no
people integration and social integration, a
aspects almost inseparable.
3.3.2. Identification of codes
We used the qualitative data analysis
code the papers. As a universal tool for qu
MAXQDA is usually employed to analyze tex
view transcripts, and supports a variety of qu
ods. Various authors (e.g. [31,37,38]) h
experiences concerning the use of similar s
synthesize data from a corpus of texts eme
literature review. Motivated by these exper
approach to process a large amount of text
transparently. This seems especially usefu
extant literature, which may be used to d
retrieved easily in later stages of the resear
QDA, codes can be assigned to text segme
ments. Besides descriptive statistics on the
documents, MAXQDA provides a convenien
text passages, allowing for easy data synthe
allowed us to conduct a quality analysis of
be able to retrieve papers within a certain c
manner, the categorization was done by ass
the appropriate category to the title and abs
thermore, if a criterion given in Table 2 was
the text segment providing the information
The analysis of the document corpus wa
aspects that have to be taken into account
and ASD, with the overarching aim of der
the analysis of the document corpus, observ
different aspects of integration were assign
with the aim of consolidating the perspectiv
on the same issue later on.
The dimensions introduced above forme
of high-level codes. For simplification reas
‘‘process,’’ ‘‘practices,’’ ‘‘people/social,’’ and
level codes in the following. Based on our a
Table 2
Criteria for quality assessment.
1. Is there a clear statement of the research goals (e.g. in an explicitly
verbalized research question)?
2. Is there an adequate description of the context in which the research was
carried out?
3. (Only applicable to empirical research papers):
a. Is the research method explicitly stated?
b. Which research method was chosen?
4. Are there explicit recommendations, which could be aggregated as
principles?
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
Table 3
Initial coding system.
Level 1 Process Practices People/social Technology
Level 2 Little design up front Prototyping Close collaboration Data exchange
User testing⁄
User testing⁄
Cross-functionality IDE integration
One sprint ahead User stories Knowledge transfer
Cohesive overall design Scenarios
Parallel tracks Personas
Iterative design/development
Incremental design/development
Fig. 2. Number of publications by research type.
168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

103. 3.3.3. Identification of candidate principles
In order to identify candidate principles, each code was investi-
gated concerning two characteristics. First, the frequency of occur-
rence was assessed to gain an impression of the relative
importance of each aspect. As an example, the code ‘‘prototyping’’
was assigned in 40 (48.2%) articles in the literature review, indicat-
ing that the use of prototypes was discussed in this document. In
contrast, the idea of ‘‘remote usability testing’’ was only found in
one (1.2%) article, leading to the conclusion that, while the former
patterns. These patterns represent related ideas or concepts pre-
sented in different publications along the four dimensions, leading
to the merger of independent aspects in the coding system. The
identification of such patterns was based on a qualitative analysis
and the expertise of the researchers in this domain. In this step,
also the findings of the three assessed reviews were drawn upon
to identify themes for potential principles [33].
4. Results
4.1. Number and distribution of publications
When analyzing the result set, it became apparent that, while
the integration of UCD and ASD was first discussed a decade ago,
the topic gained momentum in 2007 and, since then, a constantly
high number of relevant articles have been published every year, as
Fig. 3 illustrates. This reflects that, while the idea of integrating
UCD and ASD, has been around for some time, many integration
issues are still unresolved and research is ongoing. In particular,
at least 17 relevant articles have been published since da Silva
et al.’s [20] systematic literature was conducted in late 2010, justi-
fying an update of their findings.
Moreover, we identified the research type of each paper as pre-
Parallel tracks Personas
Iterative design/development
Incremental design/development
Fig. 2. Number of publications by research type.
Fig. 3. Number of publications by year.
3.3.3. Identification of candidate principles
In order to identify candidate principles, each code was investi-
patterns. These patterns rep
sented in different publicatio
to the merger of independen
identification of such pattern
and the expertise of the rese
also the findings of the three
to identify themes for potent
4. Results
4.1. Number and distribution o
When analyzing the resul
the integration of UCD and A
the topic gained momentum
high number of relevant artic
Fig. 2. Number of publications by research type.
Fig. 3. Number of publications by year.

104. process dimension, many more codes have been identified for
the other two dimensions. Moreover, the codes in the process
dimension are mainly specific to software development, while
the majority of the codes in the other dimensions are rather gen-
eric (e.g. codes like ‘‘collaboration’’ or ‘‘data exchange’’). This might
be due to the overall number of papers assigned to the ‘‘process’’
dimension being larger than in the other dimensions. Apart from
that, it might also be an indicator of the depth and maturity of dis-
course in each of the dimensions. While aspects of process integra-
tion have been intensively discussed and refined, the discussion in
no principles concerning technology integration were derived. A
different challenge emerged for the people/social dimension.
Although we found more sources, the recommendations were con-
tradictory. While ASD suggests the collective accountability of the
team [119], UCD methodologies usually suggest individual respon-
sibility, for example, allocating the responsibility of developing a
usable product to the UCD expert [122–124]. Furthermore, the
team setup led to contradictory evidence. On the one hand, two
dedicated teams, which handle design and development activities
Table 4
Overview of publications in the final sample.
Dimension Included publications Number of
publications
Process Benigni et al. [39], Budwig et al. [40], Felker et al. [41], Ferreira et al. [14], Ferreira et al. [42], Fox et al. [21], Holzinger et al. [43], Hussain
et al. [44], Hussain et al. [45], Kuusinen et al. [46], Lee & McCrickard [47], Lee et al. [48], Lee et al. [49], Losada et al. [50], Losada et al.
[51], Losada et al. [52], Memmel et al. [53], Memmel et al. [54], Miller [55], Najafi & Toyoshiba [56], Paelke & Nebe [57], Paelke & Sester
[58], Wolkerstorfer et al. [59], Zhang et al. [60]
24 (28.9%)
Practices Adikari et al. [61], Beyer et al. [62], Broschinsky & Baker [63], Carvalho [64], Chamberlain et al. [35], Cho [65], Constantine [13],
Constantine & Lockwood [66], da Silva et al. [67], Detweiler [68], Düchting et al. [69], Evnin & Pries [70], Ferré et al. [71], Fisher &
Bankston [72], Haikara [73], Hansson et al. [74], Hellmann et al. [75], Hellmann et al. [76], Hennings [77], Hodgetts [78], Humayoun
et al. [79], Hussain et al. [80], Illmensee & Muff [81], Isomursu et al. [82], Kane [83], Lárusdóttir [84], Lárusdóttir et al. [85], Lee et al.
[86], Medina-Medina et al. [87], Memmel et al. [88], Meszaros & Aston [89], Obendorf & Finck [90], Obendorf et al. [91], Patton [92],
Patton [93], Petrovic & Siegmann [94], Rafla et al. [95], Rittenbruch et al. [96]
38 (45.8%)
People &
Social
Barksdale & McCrickard [22], Barksdale & McCrickard [97], Barksdale et al. [98], Brown et al. [99], Brown et al. [100], Ferreira et al.
[101], Ferreira et al. [102], Kollmann et al. [103], Leszek & Courage [104], Lievesley & Yee [105], Singh [106], Ungar [107], Ungar & White
[108], Williams & Ferguson [109]
14 (16.9%)
Technology Feiner & Andrews [110], Gonçalves & Santos [111], Hosseini-Khayat et al. [112], Humayoun et al. [113], Lee [114], Peixoto [115], Peixoto
[116]
7 (8.4%)
Total 83 (100.0%)

105. Fig. 4. Final coding system for the process, people/social, technology, and practices dimensions.
170 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

106. There is evidence that a limited up-front design effort is
particularly necessary to provide a consistent and cohesive user
interface and navigation structure as it supports designers in find-
ing a suitable design concept from the very beginning [42,61]. Pat-
ton [93] states that user needs and goals have to be clearly defined
before conducting design and development efforts in order to
ensure the usability and usefulness of the software in ASD. Various
authors have confirmed this view, presenting user research as an
essential aspect of up-front design efforts in agile environments
[20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the review
confirm a relationship between LDUF and a cohesive overall design
of the final product, which is challenging to achieve in an agile
Suggestion. A shift from an up-front design to up-front analysis
concept is identified. In order to deliver a product that is valuable
to the customer and end-user in that it is both usable and useful,
product discovery and product creation should be separated in
UCASD. Overall, as listed in Table 5, four codes from our coding sys-
tem support our suggestion to include an additional principle in
this specific context.
Besides making room for sufficient up-front design activities as
a prerequisite for delivering a usable product with a consistent
user interaction concept, it allows for mitigating agile shortcom-
ings with regard to product discovery, fostering the delivery of a
useful product. Thus, we formulate the first principle as:
Fig. 5. Codes and articles related to the process dimension.
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171

107. not part of agile method-
ed by drawing on UCD, in
pectations, and ideas are
ude that the usefulness
sability concerns not only
ring product planning. In
e design throughout soft-
ns should be included in
05,106].
Table 5
Codes related to the separate product discovery and product creation principle.
Included codes Resulting principle
Little design up front Separate product discovery and product creation
Cycle zero
Cohesive overall design
Product vision/innovation
Sohaib and Khan [23]. Specifically, their reviews confirm the signif-
icance of iterative and incremental design and development. There
is general agreement among UCD researchers that an iterative
approach is essential for developing a product with high usability
references to iterative an
Scrum) or UCD (e.g. ISO
papers do not explicitly
approach, none of them ch
principle is articulated as
Principle 2 (Iterative a
ment). User-centered agil
design and development i
manner.
5.1.3. Parallel Interwoven C
Focus Area. Extendin
product creation principle
quent course of action afte
ment activities. As a res
design allows for specifyin
important features of a sys
to be considered in later
Table 6
Codes related to the iterative and incremental design and development principle.
Included codes Resulting principle
Iterative design/development Iterative and incremental design and
developmentIncremental design/
development
Table 7
Codes related to the parallel interwoven creation tracks principle.
Included codes Resulting principle
Parallel tracks Parallel interwoven creation tracks
Design one sprint ahead
Synchronization/integration
172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
Sohaib and Khan [23]. Specifically, their reviews confirm the signif-
icance of iterative and incremental design and development. There
is general agreement among UCD researchers that an iterative
references to iterative a
Scrum) or UCD (e.g. ISO
papers do not explicitly
approach, none of them ch
principle is articulated as
Principle 2 (Iterative a
ment). User-centered agil
design and development i
manner.
5.1.3. Parallel Interwoven C
Focus Area. Extendin
product creation principl
quent course of action afte
ment activities. As a res
design allows for specifyin
important features of a sy
Table 6
Codes related to the iterative and incremental design and development principle.
Included codes Resulting principle
Iterative design/development Iterative and incremental design and
developmentIncremental design/
development
Table 7
Codes related to the parallel interwoven creation tracks principle.
Included codes Resulting principle
Parallel tracks Parallel interwoven creation tracks
Design one sprint ahead
Synchronization/integration
172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

108. need to be adopted to assure the synchronization
n of such tracks.
ods and user-centered development, Chamberlain et
firm the importance of continuous stakeholder in
Fig. 6. Codes and number of articles related to the continuous stakeholder involvement principle.

109. The value of individuals and interactions is documen
agile manifesto [134]. According to UCD, understanding
their tasks should be the focus of software developm
Moreover, UCD and ASD both encourage customer or u
pation in the development of new systems or software
suggest the following principle:
Fig. 7. Codes and number of articles related to the artifact-mediated communication principle.
UCASD.
Description
Separate product discovery and product creation
Iterative and incremental design and development
Parallel interwoven creation tracks
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

110. Fig. 8. Codes and number of articles related to dimension ‘‘Practices’’.