Weitere Àhnliche Inhalte
Ăhnlich wie Characterizing and predicting which bugs get fixed (20)
Mehr von Thomas Zimmermann (20)
KĂŒrzlich hochgeladen (20)
Characterizing and predicting which bugs get fixed
- 1. Characterizing and Predicting
Which Bugs Get Fixed
Philip Guo, Stanford University
Thomas Zimmermann, Microsoft Research
Nachiappan Nagappan, Microsoft Research
Brendan Murphy, Microsoft Research
© Microsoft Corporation
- 8. Not all bugs get fixed
âą Bug #1 occurs rarely and affects only a few users.
â Changes required to fix Bug #1 could be large and
expensive.
âą Fixing Bug #2 could introduce new bugs
â Code changes that fix bugs are up to twice as likely to
introduce new bugs as other kinds of changes.
âą Users are relying on existing behavior
â Fixing the bug could break their systems.
© Microsoft Corporation
- 9. Research question
How do factors related to people and
bug report edits affect whether a bug
is successfully resolved as FIXED?
â Inefficiencies in bug triaging process
â Inform design of tools and policies
© Microsoft Corporation
- 10. Bug handling is pervasive
24,000+ MSFT employees
reported, commented on, or handled
at least one bug in Windows
(vs. 2,000+ Windows developers)
© Microsoft Corporation
- 11. Methodology
Qualitative Quantitative
survey data analysis
Reputation âą Interest âą Reassignments âą Distance âą
Additional qualitative factors from survey
© Microsoft Corporation
- 12. Qualitative survey
âą In your experience, how do each of these
factors affect the chances of whether a bug
will get successfully resolved as FIXED?
â 7-point Likert scale
âą Sent to 1,773 Microsoft employees
â Employees who opened OR were assigned to OR
resolved most Windows Vista bugs
â 358 responded (20%)
© Microsoft Corporation
- 13. SURVEY: How does the bug report's activity affect the
chances of it being successfully fixed?
Severity upgraded
Severity downgraded
Few editors (<10)
Many editors (>=10)
No re-assignments (0)
Few re-assignments (1-5)
Many re-assignments (>30)
At least 1 re-open
-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix
© Microsoft Corporation
- 14. SURVEY: How do the people involved in addressing the
bug affect the chances of it being successfully fixed?
Opener/assignee same person
Same manager
Same building
Same country
Different managers
Different countries
Opened by temp. employee
Assigned to temp. employee
-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix
© Microsoft Corporation
- 15. Other qualitative factors
Textual quality of bug report:
âJust to re-emphasize: The quality of the bug description is very important. Not
necessarily filling in the dozens of fields in the bug database with all sorts of crap
(build numbers, dates, classifications, etc) - but just the plain-text description of
the problem, the implication and maybe even the potential solution.â
Perceived customer/business impact:
âCustomer impact can be a very big impact on a bug if evidence exists to
show the cost of not fixing the bug for one or more customers.â
Rank and seniority of bug opener:
âA bug opened because something went wrong on a VPs [vice presidentâs]
laptop has better chance [of being fixed] than a bug opened because the
same thing happened to an intern.â
Interpersonal skills of bug opener:
âOne other âsoftâ factor is the speaking skill persuasiveness of the developer
(or other representative) when arguing for the bug.â
© Microsoft Corporation
- 16. Quantitative data analysis
âą Correlate likelihood of bug report getting fixed
â Reputation âą Interest âą Reassignments âą Distance
âą Windows Vista and Windows 7 bug reports
â Bug lifetime: open, assign, edit, resolve, close
âą Microsoft employee data
â Manager, Building, Country
© Microsoft Corporation
- 17. #1: Reputation
âA big influence [on bug fixes] is the
reputation of the person opening the bug.
If submitter has a history of submitting
high quality bugs then new bugs from that
person get better attention [...]â
From anonymous survey respondent
© Microsoft Corporation
- 18. Quantifying reputation
For each bug, calculate openerâs reputation by
aggregating over all bugs in the past.
Hooimeijer and Weimer: Modeling bug report quality. ASE 2007.
© Microsoft Corporation
- 21. #2: Interest
Windows Vista Windows 7
© Microsoft Corporation
- 22. #3: Reassignments
Severity upgraded
Severity downgraded
Few editors (<10)
Many editors (>=10)
No re-assignments (0)
Few re-assignments (1-5)
Many re-assignments (>30)
At least 1 re-open
-3 -2 -1 0 1 2 3
© Microsoft Corporation
- 24. The need for reassignments
âBugs many times are exposed in the
UI [user interface], but are not caused
by the team writing the UI code.
These bugs can pass down several
layers of components before landing
on a lower level component owner.â
From anonymous survey respondent
© Microsoft Corporation
- 27. Trust vs. distance
âWhenever I open bugs assigned to people
I know, they are investigated thoroughly as
there is a trust in the report I write.
Often when reporting a bug within areas
where I don't know the owners, there is
inherent distrust in the bug report.â
From anonymous survey respondent
© Microsoft Corporation
- 28. Predictive model
âą At bug opening time, predict probability of fix
âą Logistic regression model with 7 factors
â Bug source
â Reputation of bug opener
â Reputation of first assignee
â Initial severity level
â Opened by temporary employee?
â Opener & initial assignee had same manager?
â Opener & initial assignee in same building?
âą Train on Windows Vista, predict Windows 7 fixes
â 68% precision, 64% recall (0.50 cut-off)
© Microsoft Corporation
- 29. Recommendations
âą Train and incentivize employees to write
higher-quality bug reports
âą Improve awareness of developersâ expertise
to minimize reassignments
âą Improve communication and trust amongst
people in different teams and locations
âą Encourage more objectivity in prioritizing and
handling bugs
© Microsoft Corporation
- 30. Summary
âą Characterization of which bug reports get fixed.
â At least one reassignment increases but too many
reassignments decrease the likelihood
â The higher reputation a bug opener has, the more
likely his/her bugs are to get fixed.
â Bugs handled by multiple teams and across
multiple locations are less likely to get fixed.
âą Survey results provide insights into social factors
that affect the bug triaging process.
â Seniority, reputation, personal relations, and trust.
© Microsoft Corporation