Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012)
•Als KEY, PDF herunterladen•
2 gefällt mir•639 views
The document characterizes the verification process for bug fixes in two open source IDEs by mining their bug repositories. It finds that around 20% of developers perform over 80% of verifications, which tend to occur in bursts after releases. Most verification comments do not specify the technique used. The analysis risks being biased by mass verifications for cleanup and pseudo-verifications where the label does not truly mean the fix was verified. Future work involves modeling the verification process and its effects as a causal network.
Empfohlen
Weitere ähnliche Inhalte
Andere mochten auch
Andere mochten auch (18)
Ähnlich wie Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012)
Ähnlich wie Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012) (20)
Mehr von Rodrigo Rocha
Mehr von Rodrigo Rocha (11)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012)
- 1. Characterizing Verification of Bug Fixes in Two Open Source IDEs Rodrigo Souza* and Christina Chavez Software Engineering Labs Department of Computer Science - IM Universidade Federal da Bahia (UFBA), Brazil {rodrigo, flach}@dcc.ufba.br * speaker June 2, 2012 MSR 2012, Zürich
- 2. 2
- 3. NEW 2
- 4. NEW FI XED 2
- 5. NEW FI XED E D V ERIFI 2
- 6. Characterize the verification process by mining bug repositories VER IFIED 3
- 7. Characterize the verification process by mining bug repositories VER IFIED When? Who? How? 3
- 8. Data MSR 2011 Challenge data set (~ 10 years of bug reports) Platform VersionControl Platform EMF (Eclipse Modeling Framework) 4
- 9. When?
- 10. accumulated # of verifications ~ 800 verifications ~ 2 years 6
- 11. accumulated # of verifications ~ 800 * = release verifications * * * * ~ 2 years 6
- 12. accumulated # of verifications ~ 700 * = release * verifications * * * * * * * ~ 1 year 7
- 13. accumulated # of verifications ~ 700 * = release * verifications * * verification * phase * * * * ~ 1 year 7
- 14. Who?
- 15. QA developer bug fixes x ≥ 10x verifications 9
- 16. QA team 10
- 17. 20% of developers QA team 10
- 18. 20% of developers QA team 80% of the verifications 10
- 19. QA team 11
- 20. How?
- 21. 13
- 22. Most comments just state the obvious. 4% refer to Less than automated testing or code inspection. Further research needed 14
- 23. Pitfalls
- 24. accumulated # of verifications ~ 14k verifications ~ 10 years 16
- 25. accumulated # of verifications ~ 14k verifications ~ 10 years 16
- 26. 17
- 27. 2.6k bugs 17
- 28. 2.6k bugs 17
- 29. 2.6k bugs 17
- 30. Mass verifications: represent repository cleanup, not software verification. They may represent a large part of the verifications and bias your analyses. 18
- 31. Pseudo verifications: In some projects, marking a bug as VERIFIED means something else! (e.g., in Eclipse/EMF, since 2007, it means that the fix is available in a build) 19
- 32. Future Work
- 33. process product (software) 21
- 34. process verification process* product (software) * qa team, reopening verification phase etc. 22
- 35. causal process (bayesian) network verification process* product (software) * qa team, reopening verification phase etc. 22
- 36. Thanks! Verification ✓ ✗ phase QA team ✗ ✓ Comments rarely state the verification technique. Beware of mass verifications and pseudo verifications. 23
Hinweis der Redaktion
- \n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
- We used data from the previous MSR Challenge, containing about 10 years of bug reports from two open source IDEs: Eclipse and NetBeans. We analyzed two subprojects for each IDE.\n
- So, first, when are bug fixes verified?\n
- We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
- We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
- We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
- We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
- We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
- Next, who verifies the bug fixes? In some projects, there is a team dedicated to verifications: Quality Assurance team, or QA team.\n
- We defined that the QA team of a project is formed by all developers that perform at least 10 times more verifications than bug fixes.\n
- Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
- Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
- Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
- Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
- Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
- In Eclipse, we found no evidence of a QA team.\n
- Finally, how are the bug fixes verified?\nThat is, what are the techniques used to verify each bug fix?\n
- We looked at the comments developers write when they mark a bug as VERIFIED.\n
- But it appears that most comments just state the obvious: that the bug fix was verified using some version of the software.\nUsing regular expressions, we found that less than 4% of the comments refer to automated testing or code inspection, although further research is needed.\n
- We’d also like to share some pitfalls we’ve found during this research.\n
- Plotting the number of verifications in NetBeans/Platform over its lifetime, we found what appears to be a huge verification effort, represented by a big rise in the graph.\nHowever, by looking at the data, we discovered that this big rise represents...\n\n
- More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
- More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
- More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
- So, be careful: mass verifications may represent a large part of the verifications in a project, but they are not really software verification and may bias your analyses.\nIn the previous analyses, mass verifications were discarded.\n\n-- Be careful, because such mass verifications may bias your analyses\n
- Also, by reading a few comments, we found that, in some projects, marking a bug as VERIFIED has a special meaning. \nFor example, in Eclipse/EMF it just means that the bug fix was made available in a build).\n\n
- Future work\n
- People say that, if you control your process, you can control the quality of your product. \n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
- Thank you very much!\n