Why Do Computational Scientists Trust Their So

•Als ODP, PDF herunterladen•

0 gefällt mir•338 views

A very informal talk I gave to Hausi Muller's group at UVic in June 2009. I have included, without permission, slides from Daniel Hook's excellent presentation at SE-CSE 2009 (http://www.cs.ua.edu/~SECSE09/schedule.htm).

Technologie

Why do climate modellers trust their software? Jon Pipitone Advisor: Steve Easterbrook University of Toronto @Uvic, June 2009

"If we knew what we were doing, it wouldn't be called research, would it?" - attributed to Albert Einstein

What is climate modelling? A kind of computational science

What is computational science? ,[object Object],– Wikipedia, “computational science”

What is computational science? ,[object Object]

Program outputs are the results of the experiment.

Scientific software development my focus

What is climate modelling? ,[object Object]

What is climate modelling? Source: Easterbrook, CUSEC'09 (Source: IPCC AR4, 2007)

General Circulation Models Source: Easterbrook, CUSEC'09 © Crown Copyright

Verification and Validation ,[object Object]

Code review by designated code owners ,[object Object],[object Object],[object Object]

Model-intercomparisons Source: Easterbrook, CUSEC'09

Basic Validation Steps ,[object Object],Source: Easterbrook, CUSEC'09

Validation Notes Source: Easterbrook, CUSEC'09 © Crown Copyright

Validation notes Bit reproducibility Core problems with V&V

What does this mean for software quality?

(note, we're not talking about model quality!)

How do we judge quality in scientific software?

Measuring Quality: Defect Density ,[object Object]

Can we benchmark quality using defect density? ,[object Object],[object Object]

Hadley “defect rates” Some comparisons: NASA Space shuttle: 0.1 failures/KLOC Best military systems: 5 faults/KLOC Worst military systems: 55 faults/KLOC Apache: 0.5 faults/KLOC XP: 1.4 faults/KLOC Hadley’s Unified Model: avg of 24 “bug fixes” per release avg of 50,000 lines edited per release  2 defects / KLOC make it through to released code ,[object Object],Source: Easterbrook, CUSEC'09 ? ?

Few Defects Post-release ,[object Object]

Model runs, but variables drift out of tolerance

Runs don’t bit-compare (when they should) ,[object Object],[object Object]

Weitere ähnliche Inhalte

Andere mochten auch

Hvorfor personlige læringsnettverk (PLN)?Svend Andreas Horgen

Assessments presentetionsjoseceles

Elizabeth rodriguez valencia actividad 1 mapa candresyalex

Usability of Online InstructionMichael Wilder

navegadores webmoooy

Technical Bulletin 0714 Elastomeric insulation versus polyisocyanurate in low...Joe Hughes

L'escol sadako; una xarxa de connexiones educativesAsociación Espiral Educación y Tecnología

Østlandssamarbeidet 5 mai 2010kirsle

Segment 1tspicuzza

Ejercicio de verbos irregulares 2Luisa Estrada

10tips ppGustav Hårtveit

Studieteknikktarzanol

11Sharad Kumar Tripathi

Presentación educacion virtual y distancia jun16yusmeilymieres

2 analysis of foxeshelkin

Exam prepvladgidea

ConclusionSolano YO

Olhos brancosMarcio Lima

Andere mochten auch (18)

Hvorfor personlige læringsnettverk (PLN)?

Assessments presentetions

Elizabeth rodriguez valencia actividad 1 mapa c

Usability of Online Instruction

navegadores web

Technical Bulletin 0714 Elastomeric insulation versus polyisocyanurate in low...

L'escol sadako; una xarxa de connexiones educatives

Østlandssamarbeidet 5 mai 2010

Segment 1

Ejercicio de verbos irregulares 2

10tips pp

Studieteknikk

Presentación educacion virtual y distancia jun16

2 analysis of foxes

Exam prep

Conclusion

Olhos brancos

Ähnlich wie Why Do Computational Scientists Trust Their So

Annotated Bibliography .Guidelines Annotated Bibliograph.docxjustine1simpson78276

2010-03-31 - VU Amsterdam - Experiences testing safety critical systemsJaap van Ekris

On the Malware Detection Problem: Challenges & Novel ApproachesMarcus Botacin

2011-05-02 - VU Amsterdam - Testing safety critical systemsJaap van Ekris

When do software issues get reported in large open source softwareRAKESH RANA

Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...TEST Huddle

SE and AI: a two-way streetCS, NcState

Enhancing Developer Productivity with Code ForensicsTechWell

When do software issues get reported in large open source software - Rakesh RanaIWSM Mensura

Gatling workshop lets test17Gerald Muecke

Test design techniquesGregory Solovey

An Empirical Comparison of the Development History of CloudStack and EucalyptusAhmed Zerouali

Software Engineering Research: Leading a Double-Agent Life.Lionel Briand

How do we detect malware? A step-by-step guideMarcus Botacin

Resilience Engineering: A field of study, a community, and some perspective s...John Allspaw

Software Development and QualityHerwig Habenbacher

Explainable Artificial Intelligence (XAI)  to Predict and Explain Future Soft...Chakkrit (Kla) Tantithamthavorn

Three Interviews About Static Code AnalyzersAndrey Karpov

2cee Master Cocomo20071CS, NcState

Chapter 3.pptxTekle12

Ähnlich wie Why Do Computational Scientists Trust Their So (20)

Annotated Bibliography .Guidelines Annotated Bibliograph.docx

2010-03-31 - VU Amsterdam - Experiences testing safety critical systems

On the Malware Detection Problem: Challenges & Novel Approaches

2011-05-02 - VU Amsterdam - Testing safety critical systems

When do software issues get reported in large open source software

Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...

SE and AI: a two-way street

Enhancing Developer Productivity with Code Forensics

When do software issues get reported in large open source software - Rakesh Rana

Gatling workshop lets test17

Test design techniques

An Empirical Comparison of the Development History of CloudStack and Eucalyptus

Software Engineering Research: Leading a Double-Agent Life.

How do we detect malware? A step-by-step guide

Resilience Engineering: A field of study, a community, and some perspective s...

Software Development and Quality

Explainable Artificial Intelligence (XAI)  to Predict and Explain Future Soft...

Three Interviews About Static Code Analyzers

2cee Master Cocomo20071

Chapter 3.pptx

Kürzlich hochgeladen

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Understanding the Laravel MVC ArchitecturePixlogix Infotech

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

Install Stable Diffusion in windows machinePadma Pradeep

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Slack Application Development 101 Slidespraypatel2

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

How to Remove Document Management Hurdles with X-Docs?XfilesPro

Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Kürzlich hochgeladen (20)

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Understanding the Laravel MVC Architecture

Injustice - Developers Among Us (SciFiDevCon 2024)

How to Troubleshoot Apps for the Modern Connected Worker

Enhancing Worker Digital Experience: A Hands-on Workshop for Partners

08448380779 Call Girls In Friends Colony Women Seeking Men

Benefits Of Flutter Compared To Other Frameworks

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

Install Stable Diffusion in windows machine

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Scaling API-first – The story of a global engineering organization

Handwritten Text Recognition for manuscripts and early printed texts

My Hashitalk Indonesia April 2024 Presentation

08448380779 Call Girls In Civil Lines Women Seeking Men

Slack Application Development 101 Slides

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

How to Remove Document Management Hurdles with X-Docs?

Azure Monitor & Application Insight to monitor Infrastructure & Application

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Why Do Computational Scientists Trust Their So

1. Why do climate modellers trust their software? Jon Pipitone Advisor: Steve Easterbrook University of Toronto @Uvic, June 2009

4. What am I missing?

5. "If we knew what we were doing, it wouldn't be called research, would it?" - attributed to Albert Einstein

6. What is climate modelling? A kind of computational science

9. Program outputs are the results of the experiment.

10. Scientific software development my focus

11.

12. What is climate modelling? Source: Easterbrook, CUSEC'09 (Source: IPCC AR4, 2007)

13. What is climate modelling? Source: Easterbrook, CUSEC'09 (Source: IPCC AR4, 2007)

15. Scientific software development

16. Scientific software development

17.

18.

19. Model-intercomparisons Source: Easterbrook, CUSEC'09

20.

22. Core problems with V&V

23. Validation notes Bit reproducibility Core problems with V&V

24.

25. What does this mean for software quality?

26. (note, we're not talking about model quality!)

27. How do we judge quality in scientific software?

28.

29.

30.

31.

32.

33. Model crashes during a run

34. Model runs, but variables drift out of tolerance

35.

36. The right results for the “wrong reasons” (e.g. over-tuning)

37. Expected improvement not achieved Source: Easterbrook, CUSEC'09

38. Measuring Quality: Defect Density So, Is climate modelling software really that good?

39.

40. When are we counting: pre- or post-release?

41.

42. No standards in the literature

43.

44.

45.

46. What makes a piece of software bad ?

47. How do you know when you're done?

48. How do you train newcomers?

49. or...

50. When have you had to delay releasing due to a bug? Why?

51. Tell me the story behind these and other bugs...

52.

53.

54. Already have connections with CM groups

55.

56.

57.

58. What do climate modellers do when coding to guarantee correctness?

59. What are their notions of quality wrt to code?

60. How can we benchmark computational scientists' code quality?

61.

62.

63. Breakout by defect types (but, what are they?)

64. Maybe: static fault density using automated tool

65. Examine several climate models (>3?)

66.

67. Use questions given previously to guide conversation

68. Investigate the story of a defect, judgement calls that were made.

69. ~5 defect stories per climate modelling centre

70. Cross-case analysis

71.

72.

73.

74. … objective the study?

75.

76.

Why Do Computational Scientists Trust Their So

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (18)

Ähnlich wie Why Do Computational Scientists Trust Their So

Ähnlich wie Why Do Computational Scientists Trust Their So (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Why Do Computational Scientists Trust Their So