SlideShare ist ein Scribd-Unternehmen logo

Cross-project defect prediction

•
•
Thomas Zimmermann
Thomas Zimmermann

Presented at ESEC/FSE 2009.

TechnologieBusiness
1 von 36
Downloaden Sie, um offline zu lesen
Cross-project defect prediction Thomas Zimmermann, Microsoft Research Nachiappan Nagappan, Microsoft Research Harald Gall, University of Zurich Emanuel Giger, University of Zurich Brendan Murphy, Microsoft Research
Cross-project defect prediction Thomas Zimmermann, Microsoft Research Nachiappan Nagappan, Microsoft Research Harald Gall, University of Zurich Emanuel Giger, University of Zurich Brendan Murphy, Microsoft Research
Bugs are everywhere
Quality assurance is limited... ...by time... ...and by money.
Spent QA resources on the components/files that need it most, i.e., are most likely to fail. © Microsoft Corporation
Defect prediction • Learn a prediction model from historic data • Predict defects for the Predictor Precision Recall same project Pre-Release Bugs 73.80% 62.90% Test Coverage 83.80% 54.40% • Hundreds of prediction Dependencies 74.40% 69.90% models exist Code Complexity 79.30% 66.00% • Models work fairly well Code Churn 78.60% 79.90% with precision and recall Org. Structure 86.20% 84.00% of up to 80%. From: N. Nagappan, B. Murphy, and V. Basili. The influence of organizational structure on software quality. ICSE 2008. © Microsoft Corporation
Why cross-project prediction? • Some projects do have not enough data to train prediction models or the data is of poor quality • New projects do have no data yet • Can such projects use models from other projects? (=cross-project prediction) © Microsoft Corporation
A first experiment: Firefox and IE precision=0.76; recall=0.88 Firefox IE precision=0.54; recall=0.04 Firefox can predict defects in IE. But IE cannot predict Firefox. WHY? © Microsoft Corporation
Comparing Firefox and IE Domain: Browser Features Size in LOC Desktop vs. mobile Usage Profile Dev. Tools and IDE Languages C/CPP) OS and environments Team size Systematic vs. more Open vs. closed source adhoc Local vs. distributed dev. Firefox Internet Explorer © Microsoft Corporation
Research questions • RQ 1. To which extent is it possible to use cross-project data for defect prediction? • RQ 2. Which kinds of systems are good predictors? What is the influence of data, domain, company, and process? © Microsoft Corporation
THE EXPERIMENT © Microsoft Corporation
Experiment outline • 12 systems with 28 datasets – different versions – different levels of analysis (components, files) • Run all 622 cross-project combinations – for example Firefox and IE is one combination – then train model from Firefox data, test on IE – ignore invalid combinations, e.g., do not train from Eclipse 3.0 and test on 2.1 © Microsoft Corporation
Experiment outline • For each combination, we record – similarities between projects – precision, recall, and accuracy values – success, i.e., are all of precision, recall, and accuracy > 0.75 © Microsoft Corporation
Systems studied Project No. of versions Total LOC Total Churn Firefox 2 3.2 – 3.3 MLOC 0.64 – 0.95 MLOC Internet Explorer 1 2.30 MLOC 2.20 MLOC Direct-X 1 1.50 MLOC 1.00 MLOC Internet Information Services (IIS) 1 2.00 MLOC 1.20 MLOC Clustering 1 0.65 MLOC 0.84 MLOC Printing 1 2.40 MLOC 2.20 MLOC File system 1 2.00 MLOC 2.20 MLOC Kernel 1 1.90 MLOC 3.20 MLOC SQL Server 2005 1 4.6 MLOC 7.2 MLOC Eclipse 3 0.79 – 1.3 MLOC 1.0 - 2.1 MLOC Apache Derby 4 0.49 – 0.53 MLOC 4 – 23 KLOC Apache Tomcat 6 0.25 – 0.26 MLOC 8 – 98 KLOC © Microsoft Corporation
Data used in prediction models Relative code measures on churn, complexity and pre-release bugs – Added LOC / Total LOC – Deleted LOC / Total LOC – Modified LOC / Total LOC – (Added + deleted + modified LOC) / (Commits + 1) – Cyclomatic Complexity / Total LOC – Pre-release bugs / Total LOC © Microsoft Corporation
RESULTS © Microsoft Corporation
Success rate Any guesses? 3.4% © Microsoft Corporation
Successful cross-project predictions © Microsoft Corporation
Characterizing projects Domain Domain name Product uses database Product is localized Type of user interface © Microsoft Corporation
Characterizing projects Domain Product Domain name Name Product uses database Company Product is localized Intended audience Type of user interface Operating system Programming language Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
Characterizing projects Domain Product Process Domain name Name Open source Product uses database Company Global development Product is localized Intended audience Code reviews Type of user interface Operating system Static checkers Programming language Number of developers Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
Characterizing projects Domain Product Process Data Domain name Name Open source Level of analysis Product uses database Company Global development Number of observations Product is localized Intended audience Code reviews Median, maxium, and Type of user interface Operating system Static checkers standard deviation of Programming language Number of developers the metrics (18 metrics) Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
Comparing projects • If characteristic is “Domain”, “Product”, “Prog. languages”, or “Level of analysis” – Same, Different • If nominal (for example “Open Source”) – Both X, Both Y, Both Z, …, Different • If numeric (for example “LOC”) – Less, Same, More © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less © Microsoft Corporation
Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less How are these similarities related to precision, recall, and accuracy? © Microsoft Corporation
Influence of individual factors • For each characteristic and level – check with t-tests whether they influence precision, recall, accuracy – in total 375 tests; account for multiple hypothesis testing with Bonferroni correction • Possible effects on precision, recall, accuracy – Increase – Decrease – No significant effect © Microsoft Corporation
Influence of individual factors Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less Accuracy UP Recall UP Precision UP Precision DOWN Accuracy DOWN Recall UP Accuracy DOWN © Microsoft Corporation
Influence of individual factors For full list see Table 2 and 3 in the paper. Table 2. Nominal characteristics and how they influence precision, recall, and accuracy. Factor Both Precision Recall Accuracy Precision Recall Accuracy Domain Same: — — UP Different: — — DOWN Apache: — DOWN — Company Different: DOWN — — Microsoft: UP — DOWN Product Yes: — DOWN UP Open source Different: — UP DOWN No: UP — DOWN Yes: — DOWN UP Global development Different: — UP DOWN No: UP — — Yes: UP UP DOWN Code reviews No: — DOWN UP Yes: UP — DOWN Static checkers Different: — UP DOWN No: — DOWN UP Developer: DOWN DOWN — Intended audience End-user UP UP —
Decision tree for precision Project B Project C Project A
Additional decision trees • Recall – highest observed value 0.728 for – global development (differs or both no), – median of relative number of pre-release bugs (more for test project), and – intended audience (different or both end-user). • Accuracy – highest observed value 0.843 for – median of relative number of pre-release bugs (same), – operating system (both support multiple systems), and – median of relative added LOC (fewer or same in the test project). © Microsoft Corporation
Future work • Replicate: more projects + characteristics • Address further research questions: – Why are cross-project predictions sometimes not transitive? – How does the set of metrics influence the predictions? Does IE predict Firefox when different metrics are used? © Microsoft Corporation
Summary • Out of the box, only 3.4% of cross-project defect predictions worked • But we can do better! Precision + recall > 0.70 – Idenitify factors that influence the success of cross-project predictions – Decision trees help to select the right projects • http://research.microsoft.com/projects/esm • MSR 2010: Mining Software Repositories http://msrconf.org © Microsoft Corporation

Empfohlen

Software defect prediction: do different classifiers find the same defects?
Software defect prediction: do different classifiers find the same defects?Software defect prediction: do different classifiers find the same defects?
Software defect prediction: do different classifiers find the same defects?Jeongwhan Choi
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect PredictionSung Kim
 
MACHINE LEARNING - GENETIC ALGORITHM
MACHINE LEARNING - GENETIC ALGORITHMMACHINE LEARNING - GENETIC ALGORITHM
MACHINE LEARNING - GENETIC ALGORITHMPuneet Kulyana
 
Using SHAP to Understand Black Box Models
Using SHAP to Understand Black Box ModelsUsing SHAP to Understand Black Box Models
Using SHAP to Understand Black Box ModelsJonathan Bechtel
 
Handwritten Digit Recognition(Convolutional Neural Network) PPT
Handwritten Digit Recognition(Convolutional Neural Network) PPTHandwritten Digit Recognition(Convolutional Neural Network) PPT
Handwritten Digit Recognition(Convolutional Neural Network) PPTRishabhTyagi48
 
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEDatabricks
 
Support vector machine-SVM's
Support vector machine-SVM'sSupport vector machine-SVM's
Support vector machine-SVM'sAnudeep Chowdary Kamepalli
 
Genetic algorithms in Data Mining
Genetic algorithms in Data MiningGenetic algorithms in Data Mining
Genetic algorithms in Data MiningAtul Khanna
 

Empfohlen

Software defect prediction: do different classifiers find the same defects?
Software defect prediction: do different classifiers find the same defects?Software defect prediction: do different classifiers find the same defects?
Software defect prediction: do different classifiers find the same defects?Jeongwhan Choi
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect PredictionSung Kim
 
MACHINE LEARNING - GENETIC ALGORITHM
MACHINE LEARNING - GENETIC ALGORITHMMACHINE LEARNING - GENETIC ALGORITHM
MACHINE LEARNING - GENETIC ALGORITHMPuneet Kulyana
 
Using SHAP to Understand Black Box Models
Using SHAP to Understand Black Box ModelsUsing SHAP to Understand Black Box Models
Using SHAP to Understand Black Box ModelsJonathan Bechtel
 
Handwritten Digit Recognition(Convolutional Neural Network) PPT
Handwritten Digit Recognition(Convolutional Neural Network) PPTHandwritten Digit Recognition(Convolutional Neural Network) PPT
Handwritten Digit Recognition(Convolutional Neural Network) PPTRishabhTyagi48
 
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEDatabricks
 
Support vector machine-SVM's
Support vector machine-SVM'sSupport vector machine-SVM's
Support vector machine-SVM'sAnudeep Chowdary Kamepalli
 
Genetic algorithms in Data Mining
Genetic algorithms in Data MiningGenetic algorithms in Data Mining
Genetic algorithms in Data MiningAtul Khanna
 
Fuzzy expert system
Fuzzy expert systemFuzzy expert system
Fuzzy expert systemRohit Srivastava
 
Deep learning and Healthcare
Deep learning and HealthcareDeep learning and Healthcare
Deep learning and HealthcareThomas da Silva Paula
 
Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3SURBHI SAROHA
 
Hidden markov model
Hidden markov modelHidden markov model
Hidden markov modelHaitham Ahmed
 
Software defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithmsSoftware defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithmsVenkat Projects
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
K Nearest NeighborsTilani Gunawardena PhD(UNIBAS), BSc(Pera), FHEA(UK), CEng, MIESL
 
Cloud federation.pptx
Cloud federation.pptxCloud federation.pptx
Cloud federation.pptxYbhh
 
Malware Detection using Machine Learning
Malware Detection using Machine Learning Malware Detection using Machine Learning
Malware Detection using Machine Learning Cysinfo Cyber Security Community
 
Production System in AI
Production System in AIProduction System in AI
Production System in AIBharat Bhushan
 
Autonomic Computing
Autonomic ComputingAutonomic Computing
Autonomic ComputingPrithwishSarkar4
 
Deep Learning - CNN and RNN
Deep Learning - CNN and RNNDeep Learning - CNN and RNN
Deep Learning - CNN and RNNAshray Bhandare
 
Ga
GaGa
Gavenki249
 
Word2Vec
Word2VecWord2Vec
Word2Vecmohammad javad hasani
 
ID3 ALGORITHM
ID3 ALGORITHMID3 ALGORITHM
ID3 ALGORITHMHARDIK SINGH
 
High–Performance Computing
High–Performance ComputingHigh–Performance Computing
High–Performance ComputingBRAC University Computer Club
 
Malware detection-using-machine-learning
Malware detection-using-machine-learningMalware detection-using-machine-learning
Malware detection-using-machine-learningSecurity Bootcamp
 
Conceptual dependency
Conceptual dependencyConceptual dependency
Conceptual dependencyJismy .K.Jose
 
Advantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov modelAdvantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov modeljoshiblog
 
Genetic Algorithm
Genetic AlgorithmGenetic Algorithm
Genetic Algorithmrabidityfactor
 
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.
 
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!goodfriday
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"Stephanie Locke
 

Weitere ähnliche Inhalte

Was ist angesagt?

Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3SURBHI SAROHA
 
Software defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithmsSoftware defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithmsVenkat Projects
 
Cloud federation.pptx
Cloud federation.pptxCloud federation.pptx
Cloud federation.pptxYbhh
 
Production System in AI
Production System in AIProduction System in AI
Production System in AIBharat Bhushan
 
Deep Learning - CNN and RNN
Deep Learning - CNN and RNNDeep Learning - CNN and RNN
Deep Learning - CNN and RNNAshray Bhandare
 
Malware detection-using-machine-learning
Malware detection-using-machine-learningMalware detection-using-machine-learning
Malware detection-using-machine-learningSecurity Bootcamp
 
Conceptual dependency
Conceptual dependencyConceptual dependency
Conceptual dependencyJismy .K.Jose
 
Advantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov modelAdvantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov modeljoshiblog
 
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.
 

Was ist angesagt? (20)

Fuzzy expert system
Fuzzy expert systemFuzzy expert system
Fuzzy expert system
 
Deep learning and Healthcare
Deep learning and HealthcareDeep learning and Healthcare
Deep learning and Healthcare
 
Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3Knowledge Representation & Reasoning AI UNIT 3
Knowledge Representation & Reasoning AI UNIT 3
 
Hidden markov model
Hidden markov modelHidden markov model
Hidden markov model
 
Software defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithmsSoftware defect estimation using machine learning algorithms
Software defect estimation using machine learning algorithms
 
K Nearest Neighbors
K Nearest NeighborsK Nearest Neighbors
K Nearest Neighbors
 
Cloud federation.pptx
Cloud federation.pptxCloud federation.pptx
Cloud federation.pptx
 
Malware Detection using Machine Learning
Malware Detection using Machine Learning Malware Detection using Machine Learning
Malware Detection using Machine Learning
 
Production System in AI
Production System in AIProduction System in AI
Production System in AI
 
Autonomic Computing
Autonomic ComputingAutonomic Computing
Autonomic Computing
 
Deep Learning - CNN and RNN
Deep Learning - CNN and RNNDeep Learning - CNN and RNN
Deep Learning - CNN and RNN
 
Ga
GaGa
Ga
 
Word2Vec
Word2VecWord2Vec
Word2Vec
 
ID3 ALGORITHM
ID3 ALGORITHMID3 ALGORITHM
ID3 ALGORITHM
 
High–Performance Computing
High–Performance ComputingHigh–Performance Computing
High–Performance Computing
 
Malware detection-using-machine-learning
Malware detection-using-machine-learningMalware detection-using-machine-learning
Malware detection-using-machine-learning
 
Conceptual dependency
Conceptual dependencyConceptual dependency
Conceptual dependency
 
Advantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov modelAdvantages and disadvantages of hidden markov model
Advantages and disadvantages of hidden markov model
 
Genetic Algorithm
Genetic AlgorithmGenetic Algorithm
Genetic Algorithm
 
Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18Federated Learning: ML with Privacy on the Edge 11.15.18
Federated Learning: ML with Privacy on the Edge 11.15.18
 

Ähnlich wie Cross-project defect prediction

Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!goodfriday
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"Stephanie Locke
 
1,2,3 … Testing : Is this thing on(line)? with Mike Martin
1,2,3 … Testing : Is this thing on(line)? with Mike Martin1,2,3 … Testing : Is this thing on(line)? with Mike Martin
1,2,3 … Testing : Is this thing on(line)? with Mike MartinNETUserGroupBern
 
Software Security Assurance for DevOps
Software Security Assurance for DevOpsSoftware Security Assurance for DevOps
Software Security Assurance for DevOpsBlack Duck by Synopsys
 
WSO2Con EU 2015: Keynote - The Containerization of the Developer Workspace
WSO2Con EU 2015: Keynote - The Containerization of the Developer WorkspaceWSO2Con EU 2015: Keynote - The Containerization of the Developer Workspace
WSO2Con EU 2015: Keynote - The Containerization of the Developer WorkspaceWSO2
 
Cocoa Programming Certification
Cocoa Programming CertificationCocoa Programming Certification
Cocoa Programming CertificationVskills
 
.NET Fundamentals and Business Application Development
.NET Fundamentals and Business Application Development.NET Fundamentals and Business Application Development
.NET Fundamentals and Business Application Development명신 김
 
Developing apps with techstack wp-dm
Developing apps with techstack wp-dmDeveloping apps with techstack wp-dm
Developing apps with techstack wp-dmActian Corporation
 
Sqp 090508084934 Phpapp02
Sqp 090508084934 Phpapp02Sqp 090508084934 Phpapp02
Sqp 090508084934 Phpapp02sivavis
 
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...rsnarayanan
 
Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Enkitec
 
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity Software Ireland
 
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity Software Ireland
 
Accelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAccelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAmazon Web Services
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicDavid Solivan
 
[AI] ML Operationalization with Microsoft Azure
[AI] ML Operationalization with Microsoft Azure[AI] ML Operationalization with Microsoft Azure
[AI] ML Operationalization with Microsoft AzureKorkrid Akepanidtaworn
 
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～Akira Inoue
 
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョン
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョンIntelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョン
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョンMicrosoft Azure Japan
 

Ähnlich wie Cross-project defect prediction (20)

Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
Build Applications on the Microsoft Platform Using Eclipse, Java, Ruby and PHP!
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"
 
1,2,3 … Testing : Is this thing on(line)? with Mike Martin
1,2,3 … Testing : Is this thing on(line)? with Mike Martin1,2,3 … Testing : Is this thing on(line)? with Mike Martin
1,2,3 … Testing : Is this thing on(line)? with Mike Martin
 
Software Security Assurance for DevOps
Software Security Assurance for DevOpsSoftware Security Assurance for DevOps
Software Security Assurance for DevOps
 
WSO2Con EU 2015: Keynote - The Containerization of the Developer Workspace
WSO2Con EU 2015: Keynote - The Containerization of the Developer WorkspaceWSO2Con EU 2015: Keynote - The Containerization of the Developer Workspace
WSO2Con EU 2015: Keynote - The Containerization of the Developer Workspace
 
Coding Naked
Coding NakedCoding Naked
Coding Naked
 
Cocoa Programming Certification
Cocoa Programming CertificationCocoa Programming Certification
Cocoa Programming Certification
 
Improve Android System Component Performance
Improve Android System Component PerformanceImprove Android System Component Performance
Improve Android System Component Performance
 
.NET Fundamentals and Business Application Development
.NET Fundamentals and Business Application Development.NET Fundamentals and Business Application Development
.NET Fundamentals and Business Application Development
 
Developing apps with techstack wp-dm
Developing apps with techstack wp-dmDeveloping apps with techstack wp-dm
Developing apps with techstack wp-dm
 
Sqp 090508084934 Phpapp02
Sqp 090508084934 Phpapp02Sqp 090508084934 Phpapp02
Sqp 090508084934 Phpapp02
 
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...
Build Mission Critical Applications On The Microsoft Platform Using Eclipse J...
 
Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12
 
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
 
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
 
Accelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAccelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and Microservices
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs Public
 
[AI] ML Operationalization with Microsoft Azure
[AI] ML Operationalization with Microsoft Azure[AI] ML Operationalization with Microsoft Azure
[AI] ML Operationalization with Microsoft Azure
 
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～
Intelligent Mobile App と Cloud Native が創るアプリ開発の未来 ～ これからの時代のアプリケーション開発ビジョン ～
 
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョン
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョンIntelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョン
Intelligent Mobile App と Cloud Native、これからの時代のアプリケーション開発ビジョン
 

Mehr von Thomas Zimmermann

Software Analytics = Sharing Information
Software Analytics = Sharing InformationSoftware Analytics = Sharing Information
Software Analytics = Sharing InformationThomas Zimmermann
 
Predicting Method Crashes with Bytecode Operations
Predicting Method Crashes with Bytecode OperationsPredicting Method Crashes with Bytecode Operations
Predicting Method Crashes with Bytecode OperationsThomas Zimmermann
 
Analytics for smarter software development
Analytics for smarter software development Analytics for smarter software development
Analytics for smarter software development Thomas Zimmermann
 
Characterizing and Predicting Which Bugs Get Reopened
Characterizing and Predicting Which Bugs Get ReopenedCharacterizing and Predicting Which Bugs Get Reopened
Characterizing and Predicting Which Bugs Get ReopenedThomas Zimmermann
 
Data driven games user research
Data driven games user researchData driven games user research
Data driven games user researchThomas Zimmermann
 
Not my bug! Reasons for software bug report reassignments
Not my bug! Reasons for software bug report reassignmentsNot my bug! Reasons for software bug report reassignments
Not my bug! Reasons for software bug report reassignmentsThomas Zimmermann
 
Empirical Software Engineering at Microsoft Research
Empirical Software Engineering at Microsoft ResearchEmpirical Software Engineering at Microsoft Research
Empirical Software Engineering at Microsoft ResearchThomas Zimmermann
 
Security trend analysis with CVE topic models
Security trend analysis with CVE topic modelsSecurity trend analysis with CVE topic models
Security trend analysis with CVE topic modelsThomas Zimmermann
 
Analytics for software development
Analytics for software developmentAnalytics for software development
Analytics for software developmentThomas Zimmermann
 
Characterizing and predicting which bugs get fixed
Characterizing and predicting which bugs get fixedCharacterizing and predicting which bugs get fixed
Characterizing and predicting which bugs get fixedThomas Zimmermann
 
Changes and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesChanges and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesThomas Zimmermann
 
Changes and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesChanges and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesThomas Zimmermann
 
Predicting Defects using Network Analysis on Dependency Graphs
Predicting Defects using Network Analysis on Dependency GraphsPredicting Defects using Network Analysis on Dependency Graphs
Predicting Defects using Network Analysis on Dependency GraphsThomas Zimmermann
 
Quality of Bug Reports in Open Source
Quality of Bug Reports in Open SourceQuality of Bug Reports in Open Source
Quality of Bug Reports in Open SourceThomas Zimmermann
 
Predicting Subsystem Defects using Dependency Graph Complexities
Predicting Subsystem Defects using Dependency Graph Complexities Predicting Subsystem Defects using Dependency Graph Complexities
Predicting Subsystem Defects using Dependency Graph Complexities Thomas Zimmermann
 
Got Myth? Myths in Software Engineering
Got Myth? Myths in Software EngineeringGot Myth? Myths in Software Engineering
Got Myth? Myths in Software EngineeringThomas Zimmermann
 
Mining Workspace Updates in CVS
Mining Workspace Updates in CVSMining Workspace Updates in CVS
Mining Workspace Updates in CVSThomas Zimmermann
 

Mehr von Thomas Zimmermann (20)

Software Analytics = Sharing Information
Software Analytics = Sharing InformationSoftware Analytics = Sharing Information
Software Analytics = Sharing Information
 
MSR 2013 Preview
MSR 2013 PreviewMSR 2013 Preview
MSR 2013 Preview
 
Predicting Method Crashes with Bytecode Operations
Predicting Method Crashes with Bytecode OperationsPredicting Method Crashes with Bytecode Operations
Predicting Method Crashes with Bytecode Operations
 
Analytics for smarter software development
Analytics for smarter software development Analytics for smarter software development
Analytics for smarter software development
 
Characterizing and Predicting Which Bugs Get Reopened
Characterizing and Predicting Which Bugs Get ReopenedCharacterizing and Predicting Which Bugs Get Reopened
Characterizing and Predicting Which Bugs Get Reopened
 
Klingon Countdown Timer
Klingon Countdown TimerKlingon Countdown Timer
Klingon Countdown Timer
 
Data driven games user research
Data driven games user researchData driven games user research
Data driven games user research
 
Not my bug! Reasons for software bug report reassignments
Not my bug! Reasons for software bug report reassignmentsNot my bug! Reasons for software bug report reassignments
Not my bug! Reasons for software bug report reassignments
 
Empirical Software Engineering at Microsoft Research
Empirical Software Engineering at Microsoft ResearchEmpirical Software Engineering at Microsoft Research
Empirical Software Engineering at Microsoft Research
 
Security trend analysis with CVE topic models
Security trend analysis with CVE topic modelsSecurity trend analysis with CVE topic models
Security trend analysis with CVE topic models
 
Analytics for software development
Analytics for software developmentAnalytics for software development
Analytics for software development
 
Characterizing and predicting which bugs get fixed
Characterizing and predicting which bugs get fixedCharacterizing and predicting which bugs get fixed
Characterizing and predicting which bugs get fixed
 
Changes and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesChanges and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development Activities
 
Changes and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development ActivitiesChanges and Bugs: Mining and Predicting Development Activities
Changes and Bugs: Mining and Predicting Development Activities
 
Predicting Defects using Network Analysis on Dependency Graphs
Predicting Defects using Network Analysis on Dependency GraphsPredicting Defects using Network Analysis on Dependency Graphs
Predicting Defects using Network Analysis on Dependency Graphs
 
Quality of Bug Reports in Open Source
Quality of Bug Reports in Open SourceQuality of Bug Reports in Open Source
Quality of Bug Reports in Open Source
 
Meet Tom and his Fish
Meet Tom and his FishMeet Tom and his Fish
Meet Tom and his Fish
 
Predicting Subsystem Defects using Dependency Graph Complexities
Predicting Subsystem Defects using Dependency Graph Complexities Predicting Subsystem Defects using Dependency Graph Complexities
Predicting Subsystem Defects using Dependency Graph Complexities
 
Got Myth? Myths in Software Engineering
Got Myth? Myths in Software EngineeringGot Myth? Myths in Software Engineering
Got Myth? Myths in Software Engineering
 
Mining Workspace Updates in CVS
Mining Workspace Updates in CVSMining Workspace Updates in CVS
Mining Workspace Updates in CVS
 

Kürzlich hochgeladen

GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbuapidays
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusZilliz
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...apidays
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 

Kürzlich hochgeladen (20)

GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 

Cross-project defect prediction

  • 1. Cross-project defect prediction Thomas Zimmermann, Microsoft Research Nachiappan Nagappan, Microsoft Research Harald Gall, University of Zurich Emanuel Giger, University of Zurich Brendan Murphy, Microsoft Research
  • 2. Cross-project defect prediction Thomas Zimmermann, Microsoft Research Nachiappan Nagappan, Microsoft Research Harald Gall, University of Zurich Emanuel Giger, University of Zurich Brendan Murphy, Microsoft Research
  • 4. Quality assurance is limited... ...by time... ...and by money.
  • 5. Spent QA resources on the components/files that need it most, i.e., are most likely to fail. © Microsoft Corporation
  • 6. Defect prediction • Learn a prediction model from historic data • Predict defects for the Predictor Precision Recall same project Pre-Release Bugs 73.80% 62.90% Test Coverage 83.80% 54.40% • Hundreds of prediction Dependencies 74.40% 69.90% models exist Code Complexity 79.30% 66.00% • Models work fairly well Code Churn 78.60% 79.90% with precision and recall Org. Structure 86.20% 84.00% of up to 80%. From: N. Nagappan, B. Murphy, and V. Basili. The influence of organizational structure on software quality. ICSE 2008. © Microsoft Corporation
  • 7. Why cross-project prediction? • Some projects do have not enough data to train prediction models or the data is of poor quality • New projects do have no data yet • Can such projects use models from other projects? (=cross-project prediction) © Microsoft Corporation
  • 8. A first experiment: Firefox and IE precision=0.76; recall=0.88 Firefox IE precision=0.54; recall=0.04 Firefox can predict defects in IE. But IE cannot predict Firefox. WHY? © Microsoft Corporation
  • 9. Comparing Firefox and IE Domain: Browser Features Size in LOC Desktop vs. mobile Usage Profile Dev. Tools and IDE Languages C/CPP) OS and environments Team size Systematic vs. more Open vs. closed source adhoc Local vs. distributed dev. Firefox Internet Explorer © Microsoft Corporation
  • 10. Research questions • RQ 1. To which extent is it possible to use cross-project data for defect prediction? • RQ 2. Which kinds of systems are good predictors? What is the influence of data, domain, company, and process? © Microsoft Corporation
  • 11. THE EXPERIMENT © Microsoft Corporation
  • 12. Experiment outline • 12 systems with 28 datasets – different versions – different levels of analysis (components, files) • Run all 622 cross-project combinations – for example Firefox and IE is one combination – then train model from Firefox data, test on IE – ignore invalid combinations, e.g., do not train from Eclipse 3.0 and test on 2.1 © Microsoft Corporation
  • 13. Experiment outline • For each combination, we record – similarities between projects – precision, recall, and accuracy values – success, i.e., are all of precision, recall, and accuracy > 0.75 © Microsoft Corporation
  • 14. Systems studied Project No. of versions Total LOC Total Churn Firefox 2 3.2 – 3.3 MLOC 0.64 – 0.95 MLOC Internet Explorer 1 2.30 MLOC 2.20 MLOC Direct-X 1 1.50 MLOC 1.00 MLOC Internet Information Services (IIS) 1 2.00 MLOC 1.20 MLOC Clustering 1 0.65 MLOC 0.84 MLOC Printing 1 2.40 MLOC 2.20 MLOC File system 1 2.00 MLOC 2.20 MLOC Kernel 1 1.90 MLOC 3.20 MLOC SQL Server 2005 1 4.6 MLOC 7.2 MLOC Eclipse 3 0.79 – 1.3 MLOC 1.0 - 2.1 MLOC Apache Derby 4 0.49 – 0.53 MLOC 4 – 23 KLOC Apache Tomcat 6 0.25 – 0.26 MLOC 8 – 98 KLOC © Microsoft Corporation
  • 15. Data used in prediction models Relative code measures on churn, complexity and pre-release bugs – Added LOC / Total LOC – Deleted LOC / Total LOC – Modified LOC / Total LOC – (Added + deleted + modified LOC) / (Commits + 1) – Cyclomatic Complexity / Total LOC – Pre-release bugs / Total LOC © Microsoft Corporation
  • 16. RESULTS © Microsoft Corporation
  • 17. Success rate Any guesses? 3.4% © Microsoft Corporation
  • 18. Successful cross-project predictions © Microsoft Corporation
  • 19. Characterizing projects Domain Domain name Product uses database Product is localized Type of user interface © Microsoft Corporation
  • 20. Characterizing projects Domain Product Domain name Name Product uses database Company Product is localized Intended audience Type of user interface Operating system Programming language Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
  • 21. Characterizing projects Domain Product Process Domain name Name Open source Product uses database Company Global development Product is localized Intended audience Code reviews Type of user interface Operating system Static checkers Programming language Number of developers Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
  • 22. Characterizing projects Domain Product Process Data Domain name Name Open source Level of analysis Product uses database Company Global development Number of observations Product is localized Intended audience Code reviews Median, maxium, and Type of user interface Operating system Static checkers standard deviation of Programming language Number of developers the metrics (18 metrics) Single prog. language Project uses C/C++ Project uses C# Project uses Java First version Total lines of code © Microsoft Corporation
  • 23. Comparing projects • If characteristic is “Domain”, “Product”, “Prog. languages”, or “Level of analysis” – Same, Different • If nominal (for example “Open Source”) – Both X, Both Y, Both Z, …, Different • If numeric (for example “LOC”) – Less, Same, More © Microsoft Corporation
  • 24. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity © Microsoft Corporation
  • 25. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same © Microsoft Corporation
  • 26. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different © Microsoft Corporation
  • 27. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes © Microsoft Corporation
  • 28. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less © Microsoft Corporation
  • 29. Comparing projects Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less How are these similarities related to precision, recall, and accuracy? © Microsoft Corporation
  • 30. Influence of individual factors • For each characteristic and level – check with t-tests whether they influence precision, recall, accuracy – in total 375 tests; account for multiple hypothesis testing with Bonferroni correction • Possible effects on precision, recall, accuracy – Increase – Decrease – No significant effect © Microsoft Corporation
  • 31. Influence of individual factors Characteristics Project Domain Open source Code reviews LOC Train: Firefox Browser Yes Yes 3.2M Test: IE Browser No Yes 2.3M Similarity Same Different Both Yes Less Accuracy UP Recall UP Precision UP Precision DOWN Accuracy DOWN Recall UP Accuracy DOWN © Microsoft Corporation
  • 32. Influence of individual factors For full list see Table 2 and 3 in the paper. Table 2. Nominal characteristics and how they influence precision, recall, and accuracy. Factor Both Precision Recall Accuracy Precision Recall Accuracy Domain Same: — — UP Different: — — DOWN Apache: — DOWN — Company Different: DOWN — — Microsoft: UP — DOWN Product Yes: — DOWN UP Open source Different: — UP DOWN No: UP — DOWN Yes: — DOWN UP Global development Different: — UP DOWN No: UP — — Yes: UP UP DOWN Code reviews No: — DOWN UP Yes: UP — DOWN Static checkers Different: — UP DOWN No: — DOWN UP Developer: DOWN DOWN — Intended audience End-user UP UP —
  • 33. Decision tree for precision Project B Project C Project A
  • 34. Additional decision trees • Recall – highest observed value 0.728 for – global development (differs or both no), – median of relative number of pre-release bugs (more for test project), and – intended audience (different or both end-user). • Accuracy – highest observed value 0.843 for – median of relative number of pre-release bugs (same), – operating system (both support multiple systems), and – median of relative added LOC (fewer or same in the test project). © Microsoft Corporation
  • 35. Future work • Replicate: more projects + characteristics • Address further research questions: – Why are cross-project predictions sometimes not transitive? – How does the set of metrics influence the predictions? Does IE predict Firefox when different metrics are used? © Microsoft Corporation
  • 36. Summary • Out of the box, only 3.4% of cross-project defect predictions worked • But we can do better! Precision + recall > 0.70 – Idenitify factors that influence the success of cross-project predictions – Decision trees help to select the right projects • http://research.microsoft.com/projects/esm • MSR 2010: Mining Software Repositories http://msrconf.org © Microsoft Corporation