Presentation at the 4th Intl. Workshop on Conducting Empirical Studies in Industry CESI 2016 - An ICSE 2016 Workshop‎

1. What We Have
Technische Universität München
Case Studies in Industry
Daniel Méndez
Technical University of Munich
Germany
CESI 2016
Austin,Texas
@mendezfe
Joint work with
Stefan Wagner (University of Stuttgart)
Based on material from a joint work with:
Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering)
Stefan Wagner (University of Stuttgart)
Feedback from
AntonioVetrò (Technical University of Munich)
Jonas Eckhardt
@binsanoj

2. Compliments
Critique

3. (…)
Background

4. But before we start…

5. Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”

6. Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Selective
perception
“Everything went great!”

7. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Selective
perception
“Everything went great!”

8. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”

9. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”

10. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
“Ok, this was really naive | stupid | biased | not really scientific | …”

11. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial Active learning
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
Learning by doing…it wrong.
“Ok, this was really naive | stupid | biased | not really scientific | …”

12. Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

13. » Integrate studies into ongoing / planned research projects
» Actively approach
– alumni of your university
– companies in your area
– practitioners at (local) events
Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

14. Convincing contacts to participate

15. Convincing contacts to participate
» Find appropriate terminology (technology transfer)
» Find appropriate incentives for project partners
» Respond to their local needs
» Provide early feedback / results
» In funded projects, probability is higher that practitioners collaborate

16. Dealing with uncertainty
• What will the stakeholder characteristics be?
• Availability
• Skills
• Motivation to participate
• Commitment to goals
• Representativeness
• …
• What will the data look like?
• Quality of the data
• Quantity of the data

17. Dealing with uncertainty
• What will the stakeholder characteristics be?
• Availability
• Skills
• Motivation to participate
• Commitment to goals
• Representativeness
• …
• What will the data look like?
• Quality of the data
• Quantity of the data
» Take early samples and test them!
» Test instruments and data quality via pilots
» Get to know your subjects early
» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”

18. Context variables (and phenomena)
• hard to determine (or unavailable)
• hard to measure
» Context often hard to describe
(hence, hard to reproduce by others)
Variables actually
reported
Properly characterising the context
Variables you
should measure
Variables you
can measure

19. Context variables (and phenomena)
• hard to determine (or unavailable)
• hard to measure
» Context often hard to describe
(hence, hard to reproduce by others)
Variables actually
reported
Properly characterising the context
» Orient yourself on
» other (similar) studies or
» on classification schemes from the area of software process models (“Tailoring”)
» Focus on whole context, not only on cases for which you want to draw conclusions
» Ask yourself:“What information would I myself need to understand and replicate the
study under the assumption it would yield same or similar results?”
Variables you
should measure
Variables you
can measure

20. Working with sensitive data

21. Working with sensitive data
» Clarify the rules:
» what you need for the case study
» what can be stored and reported (internally and externally)
» Plan review and acceptance cycles for reports and publications
» The “easier” way (sometimes): Joint publication

22. Possible distortions in data collection:
• biased researchers
(trying to sell own PhD topic)
• weak moderation skills
• hidden agendas of respondents
(missing trust!)
• (…)
Skills matter

23. Possible distortions in data collection:
• biased researchers
(trying to sell own PhD topic)
• weak moderation skills
• hidden agendas of respondents
(missing trust!)
• (…)
Skills matter
» Don’t be biased!
» Involve other researchers in:
» Design of instruments
» Data collection (e.g. interviews)
» Continuously practice your
» Moderation skills
» Rhetoric and listening skills
» Build trust by showing loyalty and respect

24. Possible problem:
Support by management…only by management.
» No backup in projects (organisational culture)
Beware the ivory tower

25. Possible problem:
Support by management…only by management.
» No backup in projects (organisational culture)
» Make clear study plan including needed
subjects & cases (and formally agree on it)
» Talk early to subjects
» Find active supporter (not sponsor) with own
interests in project (= their own incentives)
Beware the ivory tower

26. Case study data should be disclosed
• reliability and trustworthiness?
• reproducibility and replicability?
» The success of empirical SE relies on a
collaborating community
(Data) Openness and transparency

27. Case study data should be disclosed
• reliability and trustworthiness?
• reproducibility and replicability?
» The success of empirical SE relies on a
collaborating community
(Data) Openness and transparency
» If possible, make your (anonymised) data
accessible to others
http://openscience.us/repo/

28. Properly reporting the results
Case study results need to be disseminated
• to practice
• to academia
» How to properly report?

29. Properly reporting the results
Case study results need to be disseminated
• to practice
• to academia
» How to properly report?
» Report results according to target group:
» Small presentations to management w/ focus on results
» Peer-reviewed publication w/ (selected) research scope
» (Technical) Reports with full data (and analysis)
» …
» Rely on established guidelines for reporting on case studies

30. KEY TAKE AWAYS

31. Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult…
getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise
(be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky

32. Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult…
getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise
(be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky
But there is one difference
• Null results matter
• In contrast to your dating experiences, we want to hear about your
case study experiences (especially the failed ones)

33. Thank you!
Daniel Méndez
Daniel.Mendez@tum.de
@mendezfe
• Slides will be made available
(and probably tweeted)
• Approach him if you need
material (studies, templates, …)