SlideShare ist ein Scribd-Unternehmen logo
1 von 23
Local Bias and its Impacts on the Performance of Parametric Estimation Models Ye Yang, Lang Xie, Zhimin He (ISCAS) Qi Li, Vu Nguyen, Barry Boehm  (USC) Ricardo Valerdi  (MIT/Univ. of Arizona) Sep. 21, 2011 Promise 2011, Banff, Canada
Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Background ,[object Object],Model user Model maintener Model researcher
Background(Cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Background (Cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object]
Example: COCOMO II model ,[object Object],[object Object],[object Object],[object Object],Ln_effort Ln_Size
Research questions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Local Bias Definition ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Summary of Dataset  CII 2000 Subset After2000 Subset CII 2010 Dataset
Analysis procedure ,[object Object],[object Object],[object Object],CII 2000 Subset After2000 Subset Subset 1 … A, B A 1 ’ , B 1 ’ A 2 ’ , B 2 ’ A n ’ , B n ’ local_bias 1 local_bias 2 local_bias n CII 2010 Dataset Subset 2 Subset n Group by Organization_ID Default Constants: A, B
Measuring local bias - Results ,[object Object],[object Object],[object Object],[object Object],[object Object]
Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Measuring the impacts of local bias ,[object Object],[object Object],[object Object],[object Object],Average MMRE Range of MMRE Average stdMRE Range of stdMRE Repeat the above steps for 2000 times 2000 (MMRE, stdMRE) pairs Spliting data set into training set and test set Tuning model parameters on training set Evaluating model performance on test set MMRE, stdMRE
Analysis procedure ,[object Object],[object Object],[object Object],[object Object],CII 2000 subset I SS1 Performance Local bias CII 2000 subset I SS2 Performance Local bias …… …… …… Correlation analysis
Results ,[object Object],[object Object],Reflecting the uncertainty inherent in model performance when adding just a small group of new data points into the CII 2000 baseline dataset.  CII 2000 CII2010 MMRE 0.3478   0.4063   StdMRE 0.3261   0.3401
Measuring the impacts of local bias(cont.) ,[object Object],[object Object],[object Object]
Discussions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Implications to Parametric Model Calibration   ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Threats to Validity ,[object Object],[object Object],[object Object],[object Object],[object Object]
Ongoing work on handling local bias ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object]
Thank you! Contact: Ye Yang (yangye@nfs.iscas.ac.cn)

Weitere ähnliche Inhalte

Was ist angesagt?

PEMF2_SDM_2012_Ali
PEMF2_SDM_2012_AliPEMF2_SDM_2012_Ali
PEMF2_SDM_2012_Ali
MDO_Lab
 
Data envelopment analysis
Data envelopment analysisData envelopment analysis
Data envelopment analysis
Glory Maker
 

Was ist angesagt? (20)

Feature selection for classification
Feature selection for classificationFeature selection for classification
Feature selection for classification
 
501 183-191
501 183-191501 183-191
501 183-191
 
PEMF2_SDM_2012_Ali
PEMF2_SDM_2012_AliPEMF2_SDM_2012_Ali
PEMF2_SDM_2012_Ali
 
Presentation_1376168115602
Presentation_1376168115602Presentation_1376168115602
Presentation_1376168115602
 
C054
C054C054
C054
 
Models of Operations Research is addressed
Models of Operations Research is addressedModels of Operations Research is addressed
Models of Operations Research is addressed
 
Sensitivity analysis
Sensitivity analysisSensitivity analysis
Sensitivity analysis
 
DSUS_MAO_2012_Jie
DSUS_MAO_2012_JieDSUS_MAO_2012_Jie
DSUS_MAO_2012_Jie
 
Introduction To Taguchi Method
Introduction To Taguchi MethodIntroduction To Taguchi Method
Introduction To Taguchi Method
 
Operations Research - Models
Operations Research - ModelsOperations Research - Models
Operations Research - Models
 
Analytical Hierarchy Process (Decision Making)- Application In Urban Risk Ass...
Analytical Hierarchy Process (Decision Making)-Application In Urban Risk Ass...Analytical Hierarchy Process (Decision Making)-Application In Urban Risk Ass...
Analytical Hierarchy Process (Decision Making)- Application In Urban Risk Ass...
 
Factorial design ,full factorial design, fractional factorial design
Factorial design ,full factorial design, fractional factorial designFactorial design ,full factorial design, fractional factorial design
Factorial design ,full factorial design, fractional factorial design
 
Improving Analogy Software Effort Estimation using Fuzzy Feature Subset Selec...
Improving Analogy Software Effort Estimation using Fuzzy Feature Subset Selec...Improving Analogy Software Effort Estimation using Fuzzy Feature Subset Selec...
Improving Analogy Software Effort Estimation using Fuzzy Feature Subset Selec...
 
Fractional factorial design tutorial
Fractional factorial design tutorialFractional factorial design tutorial
Fractional factorial design tutorial
 
Multi criteria decision making in spatial data analysis
Multi criteria decision making in spatial data  analysisMulti criteria decision making in spatial data  analysis
Multi criteria decision making in spatial data analysis
 
Models of Operational research, Advantages & disadvantages of Operational res...
Models of Operational research, Advantages & disadvantages of Operational res...Models of Operational research, Advantages & disadvantages of Operational res...
Models of Operational research, Advantages & disadvantages of Operational res...
 
Data Envelopment Analysis
Data Envelopment AnalysisData Envelopment Analysis
Data Envelopment Analysis
 
GIS and Decision Making, Literature Review
GIS and Decision Making, Literature ReviewGIS and Decision Making, Literature Review
GIS and Decision Making, Literature Review
 
Data envelopment analysis
Data envelopment analysisData envelopment analysis
Data envelopment analysis
 
Data envelopment analysis
Data envelopment analysisData envelopment analysis
Data envelopment analysis
 

Ähnlich wie Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Estimation Models"

A Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality IndicatorsA Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality Indicators
vie_dels
 
Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...
Chakkrit (Kla) Tantithamthavorn
 
Sarcia idoese08
Sarcia idoese08Sarcia idoese08
Sarcia idoese08
asarcia
 
Thesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
Thesis Defense: Integration of Modeling Methods for Cyber-Physical SystemsThesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
Thesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
Ivan Ruchkin
 

Ähnlich wie Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Estimation Models" (20)

2cee Master Cocomo20071
2cee Master Cocomo200712cee Master Cocomo20071
2cee Master Cocomo20071
 
AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012
 
German credit score shivaram prakash
German credit score shivaram prakashGerman credit score shivaram prakash
German credit score shivaram prakash
 
The Treatment of Uncertainty in Models
The Treatment of Uncertainty in ModelsThe Treatment of Uncertainty in Models
The Treatment of Uncertainty in Models
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Software testing effort estimation with cobb douglas function a practical app...
Software testing effort estimation with cobb douglas function a practical app...Software testing effort estimation with cobb douglas function a practical app...
Software testing effort estimation with cobb douglas function a practical app...
 
Software testing effort estimation with cobb douglas function- a practical ap...
Software testing effort estimation with cobb douglas function- a practical ap...Software testing effort estimation with cobb douglas function- a practical ap...
Software testing effort estimation with cobb douglas function- a practical ap...
 
A value added predictive defect type distribution model
A value added predictive defect type distribution modelA value added predictive defect type distribution model
A value added predictive defect type distribution model
 
Registration & Modeling of Shapes with Uncertainties
Registration & Modeling of Shapes with UncertaintiesRegistration & Modeling of Shapes with Uncertainties
Registration & Modeling of Shapes with Uncertainties
 
[2012] Empirical Evaluation on FBD Model-Based Test Coverage Criteria using M...
[2012] Empirical Evaluation on FBD Model-Based Test Coverage Criteria using M...[2012] Empirical Evaluation on FBD Model-Based Test Coverage Criteria using M...
[2012] Empirical Evaluation on FBD Model-Based Test Coverage Criteria using M...
 
A sensitivity analysis of contribution-based cooperative co-evolutionary algo...
A sensitivity analysis of contribution-based cooperative co-evolutionary algo...A sensitivity analysis of contribution-based cooperative co-evolutionary algo...
A sensitivity analysis of contribution-based cooperative co-evolutionary algo...
 
Kaggle Gold Medal Case Study
Kaggle Gold Medal Case StudyKaggle Gold Medal Case Study
Kaggle Gold Medal Case Study
 
A Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality IndicatorsA Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality Indicators
 
An Automated Tool for MC/DC Test Data Generation
An Automated Tool for MC/DC Test Data GenerationAn Automated Tool for MC/DC Test Data Generation
An Automated Tool for MC/DC Test Data Generation
 
DSUS_SDM2012_Jie
DSUS_SDM2012_JieDSUS_SDM2012_Jie
DSUS_SDM2012_Jie
 
Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...
 
Sarcia idoese08
Sarcia idoese08Sarcia idoese08
Sarcia idoese08
 
Penalized Regressions with Different Tuning Parameter Choosing Criteria and t...
Penalized Regressions with Different Tuning Parameter Choosing Criteria and t...Penalized Regressions with Different Tuning Parameter Choosing Criteria and t...
Penalized Regressions with Different Tuning Parameter Choosing Criteria and t...
 
Analysis and Estimation of Child Mortality and the Influence of Maternal Care...
Analysis and Estimation of Child Mortality and the Influence of Maternal Care...Analysis and Estimation of Child Mortality and the Influence of Maternal Care...
Analysis and Estimation of Child Mortality and the Influence of Maternal Care...
 
Thesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
Thesis Defense: Integration of Modeling Methods for Cyber-Physical SystemsThesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
Thesis Defense: Integration of Modeling Methods for Cyber-Physical Systems
 

Mehr von CS, NcState

Lexisnexis june9
Lexisnexis june9Lexisnexis june9
Lexisnexis june9
CS, NcState
 
Ai4se lab template
Ai4se lab templateAi4se lab template
Ai4se lab template
CS, NcState
 
Automated Software Enging, Fall 2015, NCSU
Automated Software Enging, Fall 2015, NCSUAutomated Software Enging, Fall 2015, NCSU
Automated Software Enging, Fall 2015, NCSU
CS, NcState
 
Dagstuhl14 intro-v1
Dagstuhl14 intro-v1Dagstuhl14 intro-v1
Dagstuhl14 intro-v1
CS, NcState
 

Mehr von CS, NcState (20)

Talks2015 novdec
Talks2015 novdecTalks2015 novdec
Talks2015 novdec
 
Future se oct15
Future se oct15Future se oct15
Future se oct15
 
GALE: Geometric active learning for Search-Based Software Engineering
GALE: Geometric active learning for Search-Based Software EngineeringGALE: Geometric active learning for Search-Based Software Engineering
GALE: Geometric active learning for Search-Based Software Engineering
 
Big Data: the weakest link
Big Data: the weakest linkBig Data: the weakest link
Big Data: the weakest link
 
Three Laws of Trusted Data Sharing: (Building a Better Business Case for Dat...
Three Laws of Trusted Data Sharing:(Building a Better Business Case for Dat...Three Laws of Trusted Data Sharing:(Building a Better Business Case for Dat...
Three Laws of Trusted Data Sharing: (Building a Better Business Case for Dat...
 
Lexisnexis june9
Lexisnexis june9Lexisnexis june9
Lexisnexis june9
 
Welcome to ICSE NIER’15 (new ideas and emerging results).
Welcome to ICSE NIER’15 (new ideas and emerging results).Welcome to ICSE NIER’15 (new ideas and emerging results).
Welcome to ICSE NIER’15 (new ideas and emerging results).
 
Icse15 Tech-briefing Data Science
Icse15 Tech-briefing Data ScienceIcse15 Tech-briefing Data Science
Icse15 Tech-briefing Data Science
 
Kits to Find the Bits that Fits
Kits to Find  the Bits that Fits Kits to Find  the Bits that Fits
Kits to Find the Bits that Fits
 
Ai4se lab template
Ai4se lab templateAi4se lab template
Ai4se lab template
 
Automated Software Enging, Fall 2015, NCSU
Automated Software Enging, Fall 2015, NCSUAutomated Software Enging, Fall 2015, NCSU
Automated Software Enging, Fall 2015, NCSU
 
Requirements Engineering
Requirements EngineeringRequirements Engineering
Requirements Engineering
 
172529main ken and_tim_software_assurance_research_at_west_virginia
172529main ken and_tim_software_assurance_research_at_west_virginia172529main ken and_tim_software_assurance_research_at_west_virginia
172529main ken and_tim_software_assurance_research_at_west_virginia
 
Automated Software Engineering
Automated Software EngineeringAutomated Software Engineering
Automated Software Engineering
 
Next Generation “Treatment Learning” (finding the diamonds in the dust)
Next Generation “Treatment Learning” (finding the diamonds in the dust)Next Generation “Treatment Learning” (finding the diamonds in the dust)
Next Generation “Treatment Learning” (finding the diamonds in the dust)
 
Tim Menzies, directions in Data Science
Tim Menzies, directions in Data ScienceTim Menzies, directions in Data Science
Tim Menzies, directions in Data Science
 
Goldrush
GoldrushGoldrush
Goldrush
 
Dagstuhl14 intro-v1
Dagstuhl14 intro-v1Dagstuhl14 intro-v1
Dagstuhl14 intro-v1
 
Know thy tools
Know thy toolsKnow thy tools
Know thy tools
 
The Art and Science of Analyzing Software Data
The Art and Science of Analyzing Software DataThe Art and Science of Analyzing Software Data
The Art and Science of Analyzing Software Data
 

Kürzlich hochgeladen

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Kürzlich hochgeladen (20)

🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 

Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Estimation Models"

  • 1. Local Bias and its Impacts on the Performance of Parametric Estimation Models Ye Yang, Lang Xie, Zhimin He (ISCAS) Qi Li, Vu Nguyen, Barry Boehm (USC) Ricardo Valerdi (MIT/Univ. of Arizona) Sep. 21, 2011 Promise 2011, Banff, Canada
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10. Summary of Dataset CII 2000 Subset After2000 Subset CII 2010 Dataset
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23. Thank you! Contact: Ye Yang (yangye@nfs.iscas.ac.cn)

Hinweis der Redaktion

  1. These pictures show the stdMRE values and MMRE values in each data group.
  2. This table shows the results of correlation analysis. We can see that the range of stdMRE is significantly positive correlated with local bias and local_bias*num (loca bias times num). Both the average stdMRE and the average MMRE are significantly positive correlated with local_bias*num. Range of stdMRE reflects the uncertainty of model performance. So we argue that the bigger the local bias is, the weaker the model performance is.