Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Software Systems as Cities: A Controlled Experiment Richard Wettel, Michele Lanza        Romain Robbes    REVEAL @ Faculty...
Software Systems as Cities
City Metaphor                VISSOFT 2007
City Metaphor   class buildingpackage district                    VISSOFT 2007
City Metaphor   class buildingpackage district                    VISSOFT 2007
City Metaphor   class buildingpackage district          nesting level   color                                  VISSOFT 2007
City Metaphor     number of methods (NOM)       height     number of attributes (NOA)    base size  number of lines of cod...
Program Comprehension                  ArgoUML                  LOC     136,325                         ICPC 2007
Program Comprehension
FacadeMDRImplNOA         3                skyscraperNOM       349LOC     3,413
CPPParserNOA        85                office buildingNOM       204LOC     9,111
JavaTokenTypesNOA       173                 parking lotNOM         0LOC         0
house   PropPanelEvent        NOA          2        NOM          3        LOC         37
Program Comprehension                    ICPC 2007
Design Quality Assessmentdisharmony mapArgoUML classes   brain class      8   god class       30   god + brain      6   da...
System Evolution Analysistime traveling                       WCRE 2008
System Evolution Analysis                 time                        ArgoUMLtime traveling          8 major releases     ...
System Evolution Analysis                 time                        ArgoUMLtime traveling          8 major releases     ...
http://codecity.inf.usi.ch                 implemented in Smalltalk                                ICSE 2008 tool demo
Is ituseful  ?
A Controlled Experiment
Design
technical report 2010State of the art?
Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Pr...
Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Pr...
Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
Finding a baseline1. program comprehension2. design quality assessment
Tasks
Tasks                         program comprehension                                    6A1      Identity the convention us...
Tasks                         program comprehension                                    6A1      Identity the convention us...
Tasks                         program comprehension                                    6                                  ...
Tasks                                                     quantitative                 9                                  ...
Main research questions
Main research questions   1       Does the use of CodeCity increase the correctness       of the solutions to program comp...
Main research questions   1       Does the use of CodeCity increase the correctness       of the solutions to program comp...
Variables of the experiment
Variables of the experiment             correctnessdependent             completion time                 CodeCity         ...
Variables of the experiment             correctnessdependent             completion time                 CodeCity         ...
Variables of the experiment             correctnessdependent             completion time                 CodeCity         ...
Variables of the experiment             correctnessdependent                                             FindBugs         ...
Variables of the experiment             correctnessdependent             completion time                 CodeCity         ...
Variables of the experiment             correctnessdependent             completion time                 CodeCity         ...
The experiment’s design                     between-subjects                    randomized-block
The experiment’s design                                             between-subjects                                      ...
The experiment’s designbackground       academia               industryexperience   beginner advanced beginner advanced   ...
The experiment’s designbackground       academia               industryexperience   beginner advanced beginner advanced   ...
Execution
Experimental runs
Experimental runs                   day 1                           timetraining session(1 hour)
Experimental runs                     day 1                             timetraining session(1 hour)                      ...
Experimental runs                     day 1   day 2                                     timetraining session(1 hour)      ...
Experimental runs                     day 1   day 2   day 3                                             timetraining sessi...
Experimental runs                     day 1   day 2   day 3   day 4                                                     ti...
Testing the waters                                       2009         November           December         18 24 25   2   9...
Timeline of the experiment                                         2009 2010           November           December        ...
Timeline of the experiment                                         2009 2010           November           December        ...
Treatments and subjects                               academia        industry                           beginner advanced...
Collecting raw data
Collecting raw data          solution
Collecting raw data        completion time
Controlling time
Controlling timecommon time
Controlling timecommon timeinfo on subjectsName (Task): Remaining time
Assessing correctness       1 T2: Findbugs, analyzed with CodeCity                  A3: Impact Analysis                   ...
Assessing correctnessblinding   1 T2: Findbugs, analyzed with CodeCity                  A3: Impact Analysis               ...
Results
Statistical test   two-way analysis of variance (ANOVA)                            95% confidence interval
Correctness                                                           Ecl+Excl                                            ...
Correctness                                                           Ecl+Excl                                            ...
Correctness                                                           Ecl+Excl                                            ...
Completion time                                                             Ecl+Excl                                      ...
Completion time                                                             Ecl+Excl                                      ...
Completion time                                                             Ecl+Excl                                      ...
after the first roundCodeCityvsEcl+Excl+24% correctness-12% completion time
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Software Systems as Cities: a Controlled Experiment
Nächste SlideShare
Wird geladen in …5
×

Software Systems as Cities: a Controlled Experiment

6.568 Aufrufe

Veröffentlicht am

The slides for the presentation I gave at ICSE 2011, in Honolulu, Hawaii.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Software Systems as Cities: a Controlled Experiment

  1. 1. Software Systems as Cities: A Controlled Experiment Richard Wettel, Michele Lanza Romain Robbes REVEAL @ Faculty of Informatics PLEIAD @ DCC University of Lugano University of Chile Switzerland Chile
  2. 2. Software Systems as Cities
  3. 3. City Metaphor VISSOFT 2007
  4. 4. City Metaphor class buildingpackage district VISSOFT 2007
  5. 5. City Metaphor class buildingpackage district VISSOFT 2007
  6. 6. City Metaphor class buildingpackage district nesting level color VISSOFT 2007
  7. 7. City Metaphor number of methods (NOM) height number of attributes (NOA) base size number of lines of code (LOC) color class buildingpackage district nesting level color VISSOFT 2007
  8. 8. Program Comprehension ArgoUML LOC 136,325 ICPC 2007
  9. 9. Program Comprehension
  10. 10. FacadeMDRImplNOA 3 skyscraperNOM 349LOC 3,413
  11. 11. CPPParserNOA 85 office buildingNOM 204LOC 9,111
  12. 12. JavaTokenTypesNOA 173 parking lotNOM 0LOC 0
  13. 13. house PropPanelEvent NOA 2 NOM 3 LOC 37
  14. 14. Program Comprehension ICPC 2007
  15. 15. Design Quality Assessmentdisharmony mapArgoUML classes brain class 8 god class 30 god + brain 6 data class 17 unaffected 1,715 SoftVis 2008
  16. 16. System Evolution Analysistime traveling WCRE 2008
  17. 17. System Evolution Analysis time ArgoUMLtime traveling 8 major releases 6 years WCRE 2008
  18. 18. System Evolution Analysis time ArgoUMLtime traveling 8 major releases 6 years WCRE 2008
  19. 19. http://codecity.inf.usi.ch implemented in Smalltalk ICSE 2008 tool demo
  20. 20. Is ituseful ?
  21. 21. A Controlled Experiment
  22. 22. Design
  23. 23. technical report 2010State of the art?
  24. 24. Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Provide a not-so-short tutorial of the experimental tool to the participants. 4 Avoid, whenever possible, giving the tutorial right before the experiment. 5 Use the tutorial to cover both the research behind the approach and the tool. 6 Find a set of relevant tasks. 7 Choose real object systems that are relevant for the tasks. 8 Include more than one object system in the design. 9 Provide the same data to all participants.10 Limit the amount of time allowed for solving each task.11 Provide all the details needed to make the experiment replicable.12 Report results on individual tasks.13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.14 Take into account the possible wide range of experience level of the participants.
  25. 25. Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Provide a not-so-short tutorial of the experimental tool to the participants. 4 Avoid, whenever possible, giving the tutorial right before the experiment. 5 Use the tutorial to cover both the research behind the approach and the tool. 6 Find a set of relevant tasks. 7 Choose real object systems that are relevant for the tasks. 8 Include more than one object system in the design. 9 Provide the same data to all participants.10 Limit the amount of time allowed for solving each task.11 Provide all the details needed to make the experiment replicable.12 Report results on individual tasks.13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.14 Take into account the possible wide range of experience level of the participants.
  26. 26. Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
  27. 27. Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
  28. 28. Finding a baseline1. program comprehension2. design quality assessment3. system evolution analysis
  29. 29. Finding a baseline1. program comprehension2. design quality assessment
  30. 30. Tasks
  31. 31. Tasks program comprehension 6A1 Identity the convention used in the system to organize unit tests.A2.1& What is the spread of term T in the name of the classes, their attributes andA2.2 methods?A3 Evaluate the change impact of class C, in terms of intensity and dispersion.A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code perA4.2 method.
  32. 32. Tasks program comprehension 6A1 Identity the convention used in the system to organize unit tests.A2.1& What is the spread of term T in the name of the classes, their attributes andA2.2 methods?A3 Evaluate the change impact of class C, in terms of intensity and dispersion.A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code perA4.2 method.B1.1 Identify the package with the highest percentage of god classes.B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-levelB2.1 design problem.B2.2 Write an overview of the class-level design problems in the system. design quality assessment 4
  33. 33. Tasks program comprehension 6 5A1 Identity the convention used in the system to organize unit tests.A2.1& What is the spread of term T in the name of the classes, their attributes andA2.2 methods?A3 Evaluate the change impact of class C, in terms of intensity and dispersion.A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code perA4.2 method.B1.1 Identify the package with the highest percentage of god classes.B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-levelB2.1 design problem.B2.2 Write an overview of the class-level design problems in the system. design quality assessment 4
  34. 34. Tasks quantitative 9 8A1 Identity the convention used in the system to organize unit tests.A2.1& What is the spread of term T in the name of the classes, their attributes andA2.2 methods?A3 Evaluate the change impact of class C, in terms of intensity and dispersion.A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code perA4.2 method.B1.1 Identify the package with the highest percentage of god classes.B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-levelB2.1 design problem.B2.2 Write an overview of the class-level design problems in the system. qualitative 1
  35. 35. Main research questions
  36. 36. Main research questions 1 Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
  37. 37. Main research questions 1 Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size? 2 Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
  38. 38. Variables of the experiment
  39. 39. Variables of the experiment correctnessdependent completion time CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advancedcontrolled academia background industry
  40. 40. Variables of the experiment correctnessdependent completion time CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advancedcontrolled academia background industry
  41. 41. Variables of the experiment correctnessdependent completion time CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advancedcontrolled academia background industry
  42. 42. Variables of the experiment correctnessdependent FindBugs completion time 1,320 classes 93,310 LOC CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advanced Azureuscontrolled academia 4,656 classes background industry 454,387 LOC
  43. 43. Variables of the experiment correctnessdependent completion time CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advancedcontrolled academia background industry
  44. 44. Variables of the experiment correctnessdependent completion time CodeCity tool Eclipse + Excelindependent medium object system size large beginner experience level advancedcontrolled academia background industry
  45. 45. The experiment’s design between-subjects randomized-block
  46. 46. The experiment’s design between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
  47. 47. The experiment’s designbackground academia industryexperience beginner advanced beginner advanced B1 B2 B3 B4 between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
  48. 48. The experiment’s designbackground academia industryexperience beginner advanced beginner advanced B1 B2 B3 B4 between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
  49. 49. Execution
  50. 50. Experimental runs
  51. 51. Experimental runs day 1 timetraining session(1 hour)
  52. 52. Experimental runs day 1 timetraining session(1 hour) e1experiment session(2 hours) c1
  53. 53. Experimental runs day 1 day 2 timetraining session(1 hour) e1 e2experiment session(2 hours) c1 c2
  54. 54. Experimental runs day 1 day 2 day 3 timetraining session(1 hour) e1 e2 e3experiment session(2 hours) c1 c2
  55. 55. Experimental runs day 1 day 2 day 3 day 4 timetraining session(1 hour) e1 e2 e3experiment session(2 hours) c1 c2 c4
  56. 56. Testing the waters 2009 November December 18 24 25 2 9Lugano 1 3 1 1 1 1 1
  57. 57. Timeline of the experiment 2009 2010 November December January February ... April 18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14Lugano 1 3 1 1 1 1 1 1 1 1 3 Bologna 2 1 6 1 1 1 1 Antwerp 5 6Bern 4 1 6
  58. 58. Timeline of the experiment 2009 2010 November December January February ... April 18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14Lugano 1 3 1 1 1 1 1 1 1 1 3 remote sessions Bologna 2 1 6 1 1 1 1 Antwerp 5 6 remote sessionBern 4 1 6
  59. 59. Treatments and subjects academia industry beginner advanced advanced large 2 2 6 10 CodeCity medium 3 2 7 12 large 2 3 3 8 Ecl+Excl medium 2 5 4 11 9 12 20 41
  60. 60. Collecting raw data
  61. 61. Collecting raw data solution
  62. 62. Collecting raw data completion time
  63. 63. Controlling time
  64. 64. Controlling timecommon time
  65. 65. Controlling timecommon timeinfo on subjectsName (Task): Remaining time
  66. 66. Assessing correctness 1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2 A1 Multiple locations. The god class containing the largest number of methods in the system is There are 40/41 [0.5pts] classes class MainFrame [0.8pts] Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts] which contains 119 [0.1pts] methods. • edu.umd.cs.findbugs A2.1 • edu.umd.cs.findbugs.bcel B2.1 Localized [0.5pts] in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is DataClass [0.5pts] A2.2 A4.1 which affects a number of 67 [0.5pts] classes. Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]: 3 6 in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor • edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba contains 195 methods; • edu.umd.cs.findbugs.anttask 2. class MainFrame • edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2 contains 119 methods; • edu.umd.cs.findbugs.ba.deref 3. class BugInstance • edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs • edu.umd.cs.findbugs.ba.npe contains 118 methods or • edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor defined in package edu.umd.cs.findbugs.ba.type • edu.umd.cs.findbugs.bcel contains 118 methods. • edu.umd.cs.findbugs.classfile A4.2 • edu.umd.cs.findbugs.classfile.analysis The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each • edu.umd.cs.findbugs.classfile.engine 3 6 misplaced]: • edu.umd.cs.findbugs.classfile.impl 1. class DefaultNullnessAnnotations • edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba has an average of 124 lines of code per method; • edu.umd.cs.findbugs.cloud.db 2. class DBCloud.PopulateBugs • edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db has an average of 114.5 lines of code per method; • edu.umd.cs.findbugs.gui 3. class BytecodeScanner • edu.umd.cs.findbugs.gui2 defined in package edu.umd.cs.findbugs.ba • edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method. • edu.umd.cs.findbugs.model B1.1 • edu.umd.cs.findbugs.visitclass oracles The package with the highest percentage of god classes in the system is • edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts] which contains 1 [0.1pts] god classes out of a total of 3 [0.1pts] classes.
  67. 67. Assessing correctnessblinding 1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2 A1 Multiple locations. The god class containing the largest number of methods in the system is There are 40/41 [0.5pts] classes class MainFrame [0.8pts] Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts] which contains 119 [0.1pts] methods. • edu.umd.cs.findbugs A2.1 • edu.umd.cs.findbugs.bcel B2.1 Localized [0.5pts] in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is DataClass [0.5pts] A2.2 A4.1 which affects a number of 67 [0.5pts] classes. Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]: 3 6 in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor • edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba contains 195 methods; • edu.umd.cs.findbugs.anttask 2. class MainFrame • edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2 contains 119 methods; • edu.umd.cs.findbugs.ba.deref 3. class BugInstance • edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs • edu.umd.cs.findbugs.ba.npe contains 118 methods or • edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor defined in package edu.umd.cs.findbugs.ba.type • edu.umd.cs.findbugs.bcel contains 118 methods. • edu.umd.cs.findbugs.classfile A4.2 • edu.umd.cs.findbugs.classfile.analysis The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each • edu.umd.cs.findbugs.classfile.engine 3 6 misplaced]: • edu.umd.cs.findbugs.classfile.impl 1. class DefaultNullnessAnnotations • edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba has an average of 124 lines of code per method; • edu.umd.cs.findbugs.cloud.db 2. class DBCloud.PopulateBugs • edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db has an average of 114.5 lines of code per method; • edu.umd.cs.findbugs.gui 3. class BytecodeScanner • edu.umd.cs.findbugs.gui2 defined in package edu.umd.cs.findbugs.ba • edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method. • edu.umd.cs.findbugs.model B1.1 • edu.umd.cs.findbugs.visitclass oracles The package with the highest percentage of god classes in the system is • edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts] which contains 1 [0.1pts] god classes out of a total of 3 [0.1pts] classes.
  68. 68. Results
  69. 69. Statistical test two-way analysis of variance (ANOVA) 95% confidence interval
  70. 70. Correctness Ecl+Excl CodeCityDoes the use of CodeCity increase the 8correctness of the solutions to programcomprehension tasks, compared to non- 7visual exploration tools, regardless of the 6object system size? 5 4 3 2 1 0 medium large
  71. 71. Correctness Ecl+Excl CodeCityDoes the use of CodeCity increase the 8correctness of the solutions to programcomprehension tasks, compared to non- 7visual exploration tools, regardless of the 6object system size? 5 24.26% 4 3 2 more correct with CodeCity 1 large effect size (d=0.89) 0 medium large
  72. 72. Correctness Ecl+Excl CodeCityDoes the use of CodeCity increase the 8correctness of the solutions to programcomprehension tasks, compared to non- 7visual exploration tools, regardless of theobject system size? 6 24.26% 5 4 more correct with CodeCity 3 large effect size (d=0.89) 2 medium large
  73. 73. Completion time Ecl+Excl CodeCityDoes the use of CodeCity reduce the time 60needed to solve program comprehensiontasks, compared to non-visual exploration 50tools, regardless of the object system size? 40 30 20 10 0 medium large
  74. 74. Completion time Ecl+Excl CodeCityDoes the use of CodeCity reduce the time 60needed to solve program comprehensiontasks, compared to non-visual exploration 50tools, regardless of the object system size? 40 12.01% 30 20 less time with CodeCity 10 moderate effect size (d=0.63) 0 medium large
  75. 75. Completion time Ecl+Excl CodeCityDoes the use of CodeCity reduce the time 60needed to solve program comprehensiontasks, compared to non-visual explorationtools, regardless of the object system size? 50 12.01% 40 30 less time with CodeCity moderate effect size (d=0.63) 20 medium large
  76. 76. after the first roundCodeCityvsEcl+Excl+24% correctness-12% completion time

×