When developers perform a software maintenance task, they need to identify artifacts—e.g., classes or more specifically methods—that need to be modified. To this aim, they can browse various kind of artifacts, for example use case descriptions, UML diagrams, or source code. This paper reports the results of a study—conducted with 33 participants— aimed at investigating (i) to what extent developers use different kinds of documentation when identifying artifacts to be changed, and (ii) whether they follow specific navigation patterns among different kinds of artifacts. Results indicate that, although developers spent a conspicuous proportion of the available time by focusing on source code, they browse back and forth between source code and either static (class) or dynamic (sequence) diagrams. Less frequently, developers—especially more experienced ones—follow an “integrated” approach by using different kinds of artifacts.

1. An Empirical Investigation on
Documentation Usage Patterns in
Maintenance Tasks
Gabriele Gerardo Massimiliano Rocco Sebastiano
Bavota Canfora Di Penta Oliveto Panichella

2. Context and Motivations
• Software Development
Case Study
• User Study with 33 Partecipants
Results
•Documentation usage pattern during maintenance evolution
actvity

3. Software Development
Maintenance tasks are generally
facilitated when software
documentation is available

4. Past Work
The use of UML diagram
significantly improves the
functional correctness of changes
E. Arisholm et al. TSE 2006

5. Past Work
Help to reduce the time
needed to understand ho
perform maintenance tasks
by approximately 20 %
Eirik Tryggeseth EMSE 1997

6. How such documentation is
browsed by developers to perform
maintenance activities?

7. Empirical Study
Goal: observe the software documents
browsed by Developers during maintenance
tasks
Quality focus: understand the documentation
usage pattern during maintenance activities
Perspective: researchers interested to develop
tool to improve the program comprehension

8. • RQ1: How much time did participants spend on
different kinds of artifacts?
• RQ2: How do participants navigate
different kinds of artifacts to
identify code to be changed
during the evolution task?
Research Questions

9. Context
• Object: software artifacts from SMOS, a school
automation system developed by graduate
students at the University of Salerno (Italy).
• Subjects: 33 partecipants
11 Bachelor Students 18 Master Students 4 PhD Students

10. Study
Procedure

11. Maintenance tasks
Bug Fixing:
Add a new feature:
Improve existing features:

12. Example: Maintenance task

13. Monitoring Maintenance Activity

14. Monitoring Maintenance Activity
TIMESTAMP

15. Data Collection
XML log
Files

16. Data Collection
XML log
Files

17. 0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Undergraduates Graduates ALL
Subjects Performances

18. 0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Undergraduates Graduates ALL
Subjects Performances
Average
0.75 FM

19. 0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Undergraduates Graduates ALL
Subjects Performances
Average
0.75 FM
+ 5% FM
for Graduate
Students

21. RQ1
How much time did participants
spend on different kinds of artifacts?

22. Javadoc
Use Case
Class Diagram
Sequence Diagram
Source Code
ALL
Graduates
Undergraduates
15%
Use of each artifacts in overall tasks
60%
33%
100%
72%

23. RQ1: How much time did participants
spend on different kinds of artifacts?
72%
13%10%3%2%

24. 72%
13%10%3%2%
RQ1: How much time did participants
spend on different kinds of artifacts?
Undergraduates students used
Source Code and Javadoc significantly
more than Graduate students
Undergraduate
Students

25. 72%
13%10%3%2%
RQ1: How much time did participants
spend on different kinds of artifacts?
Graduate
Students
Graduate students used Class Diagrams
significantly more than Undergraduates

26. Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 2 4 6
Use Case
Class Diagram
Sequence Diagram
Source Code
Javadoc
Graduates
Undergraduates

27. Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 1 2 3 4 5 6
Use Case
Class Diagram
Sequence Diagram
Source Code
Javadoc
Graduates
Undergraduates
Sequence diagram represents
a fair compromise between
use cases (too abstracts) and
class diagrams (providing
useless details about an
entire subsystem)..

28. Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 1 2 3 4 5 6
Use Case
Class Diagram
Sequence Diagram
Source Code
Javadoc
Graduates
Undergraduates
Sequence diagram represents
a fair compromise between
use cases (too abstracts) and
class diagrams (providing
useless details about an
entire subsystem)..
With the other sources of
documentation available
Javadoc became useless to
identify impacted components

29. Summary RQ1

30. Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Participants consider the most important sources of
information sequence and class diagrams
3) Undergraduate students spent a significantly higher
proportion of time on source code than graduate students
that, instead, spent more time on class diagrams

31. Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Participants consider the most important sources of
information sequence and class diagrams
3) Undergraduate students spent a significantly higher
proportion of time on source code than graduate students
that, instead, spent more time on class diagrams

32. Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Participants consider the most important sources of
information sequence and class diagrams
3) Undergraduate students spent a significantly higher
proportion of time on source code than graduate students
that, instead, spent more time on class diagrams

33. RQ2
How do participants navigate different kinds
of artifacts to identify code to be changed
during the evolution task?

34. 3%
Starting Point:
Graduate vs Undergraduate students
12%
17%
25%
42%

35. Starting Point:
Graduate vs Undergraduate
Undergraduate
Graduate
58%
32%
24%
26%
9%
22%
3%
17%
4% 3%

36. 18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Other
Graduate
Undegraduate
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
Most frequents patterns before
reaching source code

37. 18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Other
Graduate
Undegraduate
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
Most frequents patterns before
reaching source code

38. 18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Other
Graduate
Undegraduate
More experienced
participants use a more
“Integrated approach”
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
Most frequents patterns before
reaching source code

39. Source
Code
Sequence
Diagram
Javadoc
Transition Graph between kinds of
Software Artifacts

40. Source
Code
Sequence
Diagram
Javadoc
1) From Source Code
participants in most
cases “go back” to
Sequence and Class
Diagrams
2) From Sequence and
Class Diagrams
participants in most
cases “go back” to
Source Code
3) Starting from a Use
Case, participants go
ahead reading Sequence
Diagrams. Only after,
they reading and writing
Source Code
Transition Graph between kinds of
Software Artifacts

41. 0%
20%
11%
30%
16%
61%
41%
70%
12%
15%
60%
24%
58%
54%
0% 10% 20% 30% 40% 50% 60% 70% 80%
USDC
USD
USC
UDC
SDC
SC
DC
Graduates
Undergraduates
1) Developers never
used Javadoc
Most Frequents Patterns
2) Patterns end always
with Source Code
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
C= Source Code

42. Summary RQ2
• 1) Participants tend to start the assigned task from source
code or from design documents , i.e., class and sequence
diagrams
• 2) More experienced participants tend to follow a more
integrated approach than less experienced Ones
• 3) During tasks, participants tend to go back and forth
repeatedly between source code and to design diagrams
(sequence and class diagrams).

43. …Conclusions…

44. …Conclusions…

45. …Conclusions…

46. …Conclusions…

47. …Conclusions…

48. …Future Directions…