In this webinar, Amir Rozenberg and Fernando Vidal will discuss the Sauce Labs Continuous Testing Benchmark Report, a new index based on insights gleaned from the more than 2 billion tests run on the Sauce Labs Continuous Testing platform. The index can be used by any organization to evaluate their progress towards continuous testing. Joining Amir and Fernando will be Phil Wells from the New York Times, a Sauce Labs customer that has achieved a nearly perfect 100 against the Continuous Testing Benchmark index. Key takeaways: - The components of the new continuous testing index - What makes those components such strong maturity indicators - How can organizations improve on each of the metrics - What happens organizationally when those metrics are improved - How the NY Times manages their continuous testing pipeline to deliver quality crosswords while driving constant improvements

1. HOW TO MEASURE SUCCESS
IN CONTINUOUS TESTING
MEASURE & OPTIMIZE YOUR WAY TO DELIVER FASTER, BETTER
JULY 10, 2019

2. HELLO
Amir Rozenberg
Director, Product
Fernando Vidal
Sr. Data Scientist
Phil Wells
Software Engineer
July 10, 2019 © Sauce Labs, Inc. 2

3. 3
AGENDA
Introduction to CT Maturity Benchmark
The Composition of the CT Maturity Benchmark
From the Practitioner Perspective
Q&A
July 10, 2019 © Sauce Labs, Inc.

4. 4
DIGITAL TRANSFORMATION: LEADERSHIP OR EXTINCTION
Where in the development cycles you
encounter most delays?
Source: GitLab 2018 Developer Survey
July 10, 2019 © Sauce Labs, Inc.

5. 5
CONTINUOUS TESTING MATURITY
MEASURING AND OPTIMIZING
The Continuous
Testing Journey
LEGACY
TESTING
CONTINUOUS
TESTING
PROOF OF
AUTOMATION VALUE
July 10, 2019 © Sauce Labs, Inc.

6. 6
CONTINUOUS TESTING MATURITY
MEASURING AND OPTIMIZING
The Continuous
Testing Journey
LEGACY
TESTING
CONTINUOUS
TESTING
PROOF OF
AUTOMATION VALUE
July 10, 2019 © Sauce Labs, Inc.

7. 7
2 BILLION+ TESTS RUN TO DATE
SAUCE LABS: THE ONLY VENDOR WITH THE DATA AND EXPERTISE TO CREATE THIS REPORT
● Millions of tests run every day on Sauce Labs
● Thousands of users run tests every day
● Unique position to use this data to
understand what drives success
July 10, 2019 © Sauce Labs, Inc.

8. 8
A SIMPLE 0-100 SCORE FOR AUTOMATED TESTING SUCCESS
COMPONENTS OF THE SAUCE LABS CONTINUOUS TESTING BENCHMARK REPORT
July 10, 2019 © Sauce Labs, Inc.

9. TEST QUALITY
TEST SUITE MAINTENANCE AND BATTLING FLAKY TESTS
July 10, 2019 © Sauce Labs, Inc. 9

10. TEST QUALITY: WHY IT MATTERS
GOAL: Consider a build green if every test passes
● “Quarantine” tests that are flaky
● Test Fix Rate > Test Break Rate
● Quickly repair tests that break due to application changes
High “ambient” levels of test failure due to flaky tests or an ever
increasing number of quarantine tests decrease functional coverage
and reduce confidence in the automated testing process
TEST FAILURE SHOULD LEAD TO HIGH CONFIDENCE THAT APP IS BROKEN, NOT YOUR TEST
July 10, 2019 © Sauce Labs, Inc. 10

11. TEST RUN TIME
THE KEY TO TEST QUALITY AND MAINTAINABILITY
July 10, 2019 © Sauce Labs, Inc. 11

12. TEST RUN TIME: WHY IT MATTERS
LONG TESTS BOTTLENECK A TEST SUITE
Long jobs slow everything down
July 10, 2019 © Sauce Labs, Inc. 12

13. TEST RUN TIME: WHY IT MATTERS
SHORT TESTS FAIL LESS OFTEN
Long jobs slow everything down
July 10, 2019 © Sauce Labs, Inc. 13

14. TEST PLATFORM COVERAGE
QUALITY MEANS CONFIDENCE ACROSS MULTIPLE PLATFORMS
July 10, 2019 © Sauce Labs, Inc. 14

15. TEST PLATFORM COVERAGE: WHY IT MATTERS
THERE IS NO “STANDARD” FOR WEB OR MOBILE
DESKTOP
MOBILE
BROWSERS
July 10, 2019 © Sauce Labs, Inc. 15

16. TEST CONCURRENCY
RUN AS MANY TESTS AS QUICKLY AS POSSIBLE FOR RAPID FEEDBACK
July 10, 2019 © Sauce Labs, Inc. 16

17. TEST CONCURRENCY: WHY IT MATTERS
ACCELERATE THE FEEDBACK LOOP & KNOW QUICKLY IF YOU CAN RELEASE
The Importance of Leveraging Concurrency a
Sample Test Suite with 50 Tests
30 Minute Build 6 Minute Build
July 10, 2019 © Sauce Labs, Inc. 17

18. CT BENCHMARK: AVERAGE OF 4 SCORES
July 10, 2019 © Sauce Labs, Inc. 18

19. Why this is Important to Get Right
Develop
Apps Faster
Improve
Developer
Productivity
Delight
Users
19
Continuous Testing Unlocks Your Ability to Achieve Digital Excellence

20. 20
THE NY TIMES SCORES A PERFECT 100
THOUGHTS AND OBSERVATIONS FROM PHIL WELLS
July 10, 2019 © Sauce Labs, Inc.

21. Q&A