Informal slides from a Birds of feather at ICSE 2020

1. Some Thoughts on Reviewing Papers:
Through a Who, What, How Lens
Margaret-Anne (Peggy) Storey
(based on research with Neil Ernst, Courtney Williams
Emelie Engstrom, Per Runeson….)
July 10th, ICSE 2020 Birds of Feather
@margaretstorey
mstorey@uvic.ca

2. When reviewing papers and summarizing the contributions,
try to identify (and evaluate) the: who, what, how…
2
Based on: “The Who, What, How of Software Engineering Research: A
Socio-Technical Framework”, Margaret-Anne Storey, Neil A. Ernst,
Courtney Williams, Eirini Kalliamvakou. EMSE to appear.
https://arxiv.org/abs/1905.12841

3. 3
Who?
(is the main
beneficiary)
Human System
Researcher

4. For the “what” we use a design science lens…4
What?
(type of research
contribution)
SolutionDescriptive

5. Design Science Lens: what is the contribution / what is being
evaluated (or explored) in the empirical study?
Problem
Instance
Solution
Requirements
Validation
Solution
Evaluation
Problem
Understanding
5
Based on : “A review of software engineering research from a design science
Perspective”, Engström, Storey, Runeson, Höst, Baldassarre, EMSE, 25(4),
2020. http://arxiv.org/abs/1904.12742

6. Technological Rule: To achieve an effect in a
given context use / do intervention.
Design Science Evaluation Criteria
🅒
🅑
🅑🅑
Evaluation Criteria
🅐 Relevance
🅑 Rigor
🅒 Novelty
🅐
Problem
Instance
Solution
Requirements
Validation
Solution
Evaluation
Problem
Understanding

7. 7
Does the “How” match
Research Questions?

8. “A paradigm is a shared world view that
represents the beliefs and values in a
discipline and that guides how problems
are solved.”
– Schwandt, 2001
8 Is your paradigm aligned with the author’s paradigm?

9. When asked to do a review be prepared to…9

10. Qualitative research validity criteria10
https://link.springer.com/article/10.1007/s111
35-006-9000-3

11. “Publish or Perish: Questioning the impact of our research...”, M. Store, ICSE 2019 Keynote
https://www.slideshare.net/mastorey/publish-or-perish-questioning-the-impact-of-our-
research-on-the-software-developer
“Using a visual abstract as a lens for communicating and promoting design science research in
software engineering”, Storey, Engström, Höst, Runeson, Bjarnason, ESEM 2017.
http://chisel.cs.uvic.ca/pubs/storey-ESEM2017.pdf
“Selecting empirical methods for software engineering research,” Easterbrook, Singer, Storey
& Damian, 2008.
http://www.cs.utoronto.ca/~sme/papers/2007/SelectingEmpiricalMethods.pdf
“A review of software engineering research from a design science Perspective”, Engström,
Storey, Runeson, Höst, Baldassarre, EMSE, 25(4), 2020. http://arxiv.org/abs/1904.12742
“The Who, What, How of Software Engineering Research: A Socio-Technical Framework”,
Margaret-Anne Storey, Neil A. Ernst, Courtney Williams, Eirini Kalliamvakou. EMSE to appear.
https://arxiv.org/abs/1905.12841
11