Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Sledgehammer to Fine Brush for QA

30 Aufrufe

Veröffentlicht am

Talk given at Global Java Dev Summit hosted by Geekle on August 1, 2020.

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Sledgehammer to Fine Brush for QA

  1. 1. From Sledgehammer to Fine Brush for QA @ShelleyMLambert shelley.lambert@adoptopenjdk.net August 2020
  2. 2. About Me • AdoptOpenJDK TSC • AQA Lead • Eclipse OpenJ9 Committer • IBM Runtimes Global QA Lead • Deep Learning Aficionado • Edible Food Forests • FCF Director (tree planting!) • Yoga teacher
  3. 3. Progressive Refinement Start with brute force End with finesse Both needed in QA • Sledgehammer – ruthless, insensitive, unnecessary force • Fine brush – detail work
  4. 4. The Steps ASSESS WHERE YOU ARE DETERMINE WHAT MATTERS DESIGN PROTOTYPE APPROACH REFINE
  5. 5. Assess Where You Are • What resources you have • Team/people • Existing tests • Dove-tailing into existing solution? Machines / CI • Measure • Code coverage (sledgehammer blow) • Functional coverage (brush stroke) • Other metrics (PR list, change-based testing) • Issues list • What your users tell you
  6. 6. Determine What Matters • Requirements • Users • Prioritize • functionality, security, perf, scalability • Risk map, meh? (related to Issues list)
  7. 7. Design • Design your QA approach like it is a software solution (because it is) • Understand where your tool choices will constrain you • Select tools for prototype, but design to avoid lock-in… (portability) • Online tutorials focus on ‘test with IDE xxxx’ or ‘use framework yyyy’
  8. 8. Prototype Approach • Bare minimum needed to ‘plug in’ testing
  9. 9. Refine • Keep the good, toss the bad • Friction is bad • Testing evolves alongside your code base • Always ask, how can we improve?
  10. 10. Specific Example: AQA • AdoptOpenJDK Quality Assurance (AQA) • Community driven, open-source project • Broadest set of platforms of any OpenJDK distributor • Testing a wide criteria representing actual business requirements to identify binaries ready for production usage
  11. 11. AQA: Assess Current state Post-assessment Roadmap Functional correctness Security OpenJDK regression (open) Passes known vulnerability tests Functional correctness Compliance OpenJDK regression Oracle JCK (closed) Eclipse functional Usability Builder-specific testing (unknown) Application & framework tests Performance Published metrics Achieves minimum throughput scores Scalability & durability Load & stress testing
  12. 12. AQA: Determine What Matters • “Make quality certain to happen” • AQA Manifesto • Open and transparent and community driven • Diverse and robust enough to cover enterprise requirements • Evolving alongside JDK • Portable (run on laptop to support development, run in CI servers) • Tagged/tracked and published to rerun tests and reproduce results
  13. 13. AQA: Design
  14. 14. AQA: Scope OpenJ9 Hotspot SAP Corretto 8 11 13 14 15 + openjdk functional perf system external (3rd party app) Red Hat Branch X JDK Implementations Platforms JDK Versions Test Categories osx aix win xlinux plinux 6 versions aarch 58 impl_platform 250000 tests 87,000,000 Tests Impl_platform x testLevels x testGroups x versions 58 x 2 x 3 x 6 x 10M = 20G+ test output per nightly build Plus PR builds, promotion builds and personal builds & Docker image testing Built in-house Built upstream scala scala scala scala scala camel scala scala scala derby scala scala scala scala elasticsearch scala scala scala scala openliberty scala scala scala scala jenkins scala scala scala scala kafka scala scala scala scala quarkus
  15. 15. AQA: Design AQA AUTOMATION: 3 LAYER CAKE testkitgen (TKG) Test Results Summary Service (TRSS) CI System (Jenkins / Tekton / AzDO / Github Actions, etc.) Responsibilities: - Categorize logically, Generates test targets based on playlist (level/platform/version/impl specific) - Execute tests via common command line / make target patterns - Test addition via auto-discovered directories, Standardize exclusion - Parameters: BUILD_LIST, EXTRA_OPTIONS, JVM_OPTIONS, TEST_FLAG, ITERATION - Generate dynamic test playlists based on input (smart parallelization) Responsibilities: - Schedule regular builds - Multiplex tests across multiple nodes of all platforms - Basic GUI view of test results - Basic forms of parallelization Responsibilities: - Monitor multiple CI servers - Graphical, Aggregate summary, Deep history - Search/sort/filter - Pluggable parsers - Basis for deeper analytics and deep learning services Layer 1: Standalone - Execution - Exclusion - Test target report summary (TAP) - Reproducibility Layer 2: Requires L1 - More nodes - Scheduling - GUI - enhanced reporting for tests that support Junit output Layer 3: Requires L1 & L2 - Database queries - Basis for extras, search /sort /filter /analyze across DB entries - Monitor anything
  16. 16. AQA: Prototype Approach 1st layer of the cake
  17. 17. AQA: Refine • Past 2.5 years have been a shocking amount of refinement • Refer to tests repo history (openjdk-tests, TKG, openjdk-test-tools,etc) • Building blocks for further refinement
  18. 18. AQA 2020 Plans – Keep on Refinin’ • Performance • compare from release to release • Multi-machine • EXTERNAL Test group expansion/clean-up • External_custom, ‘_with_tag’ targets • Smarter/Better/Faster • Smart Parallelism • Automatically test and reinclude excluded tests • Change-based testing • TRSS enhancements • BY ‘started_by’ view, By ‘Version_info’ view • Enhanced Analytics services to aid triage, Bug prediction, Core analytics • Collaboration with research • Deepsmith – deep learning model to generate fuzzers • Feedback to open benchmark teams
  19. 19. The Long List of Brush Strokes In no particular order and definitely not the full list: • Bug prediction • Combinatorial Test Design / Modeling • Improve your Triage tools • Change-based testing (combine codecov with PR list) • Verify your Tests (defect injection) • Apply deep learning to help with certain QA tasks
  20. 20. Functional vs Code Coverage static Integer mathyJunkFunction(int num1, Integer num2, String num3) { return new Integer(num1) / (num2 + new Integer(num3)); } Integer result = mathyJunkFunction(1, new Integer(1), “1”); TESTED, 100% code coverage hazaa! • Defect 1: What if num3 has a value of null (throws NumberFormatException)? • Defect 2: What if num3 is “1” and num2 is -1, or “0” and 0, or any other combination that sum to 0 (throws ArithmeticException)? • Defect 3: What if num2 is null (throws NullPointerException)?
  21. 21. Bug Prediction • SCORES PER FILE BASED ON ‘RECENT’ CHANGES DUE TO DEFECTS (GITHUB PRS/ISSUES), PREDICT BASED ON CHANGE & DEFECT HISTORY, OTHER FEATURES?
  22. 22. Hear from More AdoptOpenJDK Folks https://blog.adoptopenjdk.net/2020/07/ adoptopenjdk-virtual-roadshow
  23. 23. Extra Info • adoptopenjdk.net • trss.adoptopenjdk.net • Bi-weekly AQAvit meetups (open, transparent evolution) • https://github.com/AdoptOpenJDK/openjdk-tests/issues?q=+label%3A%22AQAvit+Meeting%22+
  24. 24. Demo TRSS

×