This document provides an overview of empirical software engineering. It begins with an introduction to the need for empirical studies in software engineering to systematically evaluate human activities. The document then covers basic concepts like measurement, data collection, and analysis. It discusses empirical strategies like surveys, case studies, and controlled experiments. The document also outlines the process for conducting an empirical study and current tools. It concludes by examining the state of the art in empirical software engineering and opportunities to better integrate experimental results.

1. Rahim Lotfi
AhmadReza Ghasemian
1

2. Table of Contents
Introduction








Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

3. Empirical Software Engineering
•
•
•
•
•
Using other people’s experiences in management and development of softwares
Empirical, included in all areas of science
“I could have told you that from the start” or “That’s obvious.”
Software engineering needs to follow the model of other physical sciences and
develop an experimental paradigm for the field.
Judging empirical work by the standards of statistics texts would lead to
the suppression of many interesting results

4. ESE in the Literature
• “Empirical Software Engineering”
• “Experimentation in Software Engineering”
• “ESEM - Empirical Software Engineering and Measurement “

5. History of Empirical Software engineering

6. History of Empirical Software engineering
• Victor Basili(1996)
• Quotes
Like physics, medicine, manufacturing, and many other disciplines, software engineering
requires the same high level approach for evolving the knowledge of the discipline; the
cycle of model building, experimentation, and learning. We cannot rely solely on
observation followed by logical thought.

7. Key figure
http://academic.research.microsoft.com

8. Key figures in Iran

9. Conferences (64) Empirical Software Engineering

10. Journals (29)
Empirical Software Engineering

11. Table of Contents
Introduction
 Need for empirical studies in the context of software engineering.







Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Goal Question Metric
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

12. Need of Empirical Studies
Why should we conduct empirical studies?
Product evaluation, e.g., prototyping.
Process evaluation
1. Prototypes are not possible (simulation based on models).
2. A process is just a description until it is used by people
 Important for research: experimentation is mandatory in
other disciplines(e.g., medicine, physics, etc.)

13. Need of Empirical Studies
Empirical studies provides a systematic, disciplined,
quantifiable and controlled way of evaluating
human-based activities.
You will learn how to select, plan, conduct and analyse an empirical study.

14. Table of Contents
 Introduction
 Need for empirical studies in the context of software engineering.
 Goals and Benefits






Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

15. Goals and Benefits
• The purpose of a study is
• The purpose of a study is
to explore ...
to find out what’s happening
to seek for new insights
to ask questions and to find answers
Measurement: usually qualitative
• to describe ...
portray accurate profile of situations, events, projects , technologies
•
•
Measurement: quantitative/qualitative
to explain ...
seek explanation of a situation/problem, usual in the form of causal
Relationships
Measurement: quantitative/qualitative
relationships, differences, changes

16. Empirical Studies in SE
• Empirical Studies is research to improve Software Engineering
Practice.
– Apply theoretical concepts in SE practice.
– Add experiences on their appliance to the SE ‘body of knowledge’
– Improve processes, methods and tools (SPPI approach).
– Verify theories and models.

18. Table of Contents
 Introduction
 Need for empirical studies in the context of software engineering.
 Goals and Benefits
 Basic Concepts: Measurement, Collection, Analysis





Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

19. Some Basic Concepts
• Measurement
- is the process of capturing data which are connected to real-world attributes to
describe them.
- Why is measurement important?
• Data Collection
Collection of qualitative / quantitative data according to research questions
• Data Analysis
Analyzing the results according to the research questions ;
statistical tests

20. Measurement
• Quotes
- “You can’t manage what you can’t measure”, Tom DeMarco
– “What is not measurable make measurable”, Galileo Galilei
• Objectives
One objective of science is to find ways to measure attributes of entities
we are interested in.
– Measurement makes concepts more visible and thus more
understandable and controllable.
• Definition
Measurement is the process by which numbers or symbols are mapped
to attributes of entities in the real world in such a way as to describe
them according to clearly defined rules.

25. Table of Contents




Introduction
Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
 Basic Concepts for Empirical Software Engineering




Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

27. Quality Improvement Paradigm
Package &
store experience
Set
goals
Analyze
results
Execute
process
Project
learning
Characterize
& understand
Corporate
learning
Provide process
with feedback
Analyze
results
Choose
processes,
methods,
techniques,
and tools

29. The Experience Factory Organization
A Different Paradigm
Project Organization
Problem Solving
Experience Factory
Experience Packaging
Decomposition of a problem
into simpler ones
Unification of different solutions
and re-definition of the problem
Instantiation
Generalization, Formalization
Design/Implementation process
Analysis/Synthesis process
Validation and Verification
Experimentation
Product Delivery within
Schedule and Cost
Experience / Recommendations
Delivery to Project

34. Table of Contents









Introduction
Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

44. Table of Contents









Introduction
Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
State of the art

46. Table of Contents







Introduction
Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Goal Question Metric
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
 Tools
 State of the art

48. Table of Contents








Introduction
Need for empirical studies in the context of software engineering.
Goals and Benefits
Basic Concepts: Measurement, Collection, Analysis
Basic Concepts for Empirical Software Engineering
Empirical Strategies: survey, case study, controlled experiment
Process of an empirical study
Tools
 State of the art

49. State of the art
• Finding better ways to experiment and integrate the results of
the studies
• Propagate the empirical discipline
•
Build an empirical research engine for software engineering
•
Build test beds for experimentation and evolution of
processes
•
Build product models that allow us to make trade-off
decisions
•
Build decision support systems offering the best empirical
advice for selecting and tailoring the right processes for the
problem

50. • References
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
V. Basili, G. Caldiera, D. Rombach: „The Goal Question Metric Approach“,2000.
B. Boehm, H.D. Rombach, M.V. Zelkowitz: „Foundations of Empirical
Software Engineering – The Legacy of Victor R. Basili“, Springer, 2005.
Freimut et al.: "Empirical Studies in Software Engineering", Tutorial, VISEK
Technical Report, 2002.
IESE Tutorials on Empirical Software Engineering.
A. Jedlitschka and D. Pfahl: "Reporting Guidelines for Controlled
Experiments in Software Engineering", ISESE, 2005.
B. Kitchenham: “Evidence-Based Software Engineering and Systematic
Literature Review”, Profes, 2006.
C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell and A. Wesslen:
"Introduction to Experimentation in Software Engineering", Kluver, 2000.
M. V. Zelkowitz, D. R. Wallace: “Experimental Models for Validating
Technology”, IEEE Computer, 1997.
Site : http:// www.microstrategy.com
Site : http:// www.jamasoftware.com
Site : http://academic.research.microsoft.com

51. Acknowledgments
Bahman Zamani
Thank you for your attention
This research work has been supported by University of Isfahan