Enhancing Your Test Automation Scenario Coverage with Selenium - QA or the Highway

Perfecto by Perforce © 2020 Perforce Software, Inc.
Enhancing your Test Automation Scenario Coverage With Selenium

2 | Advanced Codeless Testing for Web Apps perfecto.io
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268

perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
3
Continuous Testing & the Need for Scale
Cross Browser Testing & Mobile Landscape Overview
4
How to Scale Testing for Web & Mobile Apps
Demo & Q&A

Continuous Testing and Test Automation at Scale

perfectomobile.com
What Is Continuous Testing?
Continuous testing is the process of executing automated
high value tests as a part of the software delivery pipeline
in order to obtain feedback on business risks associated
with a software release upon every code change.

Measuring Continuous Testing Success
1. Average test case creation time
Total test cases
Total creation time
2. Test update/create ratio per sprint
Test case updated
New test created
3. Average test execution time
Test case executed
Total execution time
4. Test executed %
Test executed
Total test cases
5. False positive %
False failure
Total test cases
6. Tests blocked %
Test blocked
Total test cases
7. Average test case creation time
• Environment & test data preparation
• Test case maintenance, creation &
execution
8. Code coverage %
Lines of code executed
Total lines of code
9. Requirements coverage %
# of requrements
Total requirements
10. Risk coverage %
(frequency, damage)
11. Defect %
Test case failed
Total test cases executed
12. Defect leakage %
Defects slipped
Total defects
13. Defect rejection %
Defects rejected
Total defects found
14. Mean time to detect (MTTD)
Defects found
Total execution time
15. Product downtime %
Product downtime
Total run time
16. Exploratory testing hrs vs planned
17. Average exploratory session time
# of sessions
Total exploratory time
18. Average defects per session
Defects found
# of sessions
19. Defect distribution by
• Priority
• Severity
• Functional area
• Tester type
• Cause (env, architecture, code, design,
requirements, user entry)
• Test type (review, walkthrough, test
execution, exploratory, etc)
x100
x100
x100
x100
x100
x100
x100
x100
20. Cost of hardware
21. Cost of software
22. Cost of maintenance
(hardware, software)
23. Cost of test preparation
24. Cost of test execution
25. Average cost per tester
Total tester cost
# of testers
26. Cost of test maintenance over time
27. Cost of defect by
• Severity
• Stage found (dev, QA,
UAT, production)
S P E E D Q U A L I T Y C O S T

Mature DevOps Organizations Are Aiming Higher
Source: Dora, https://codeclimate.com/blog/state-of-devops-2019/

perfecto.io
World Quality
Report 2019-2020

perfecto.io
World Quality
Report 2019-2020
Maturing DevOps and Agile
Actually Requires More Testing!

• Fast feedback
• > Coverage of platforms
• < Execution of time constraints
The Need for Scale

• Scaling Methods — Build vs. Buy
• DIY (aka Selenium Grid)
• Containerized Computing
• Cloud
• Firefox is switching to a monthly release cycle.
The Need for Scale – Web Testing
W H A T ’ S Y O U R P R I M A R Y F O C U S ?

Mobile Capabilities in Web Browser (Incomplete) — Ramping Up
CameraMic
Device Authentication
BT/BLE
Notification Network

Mixing Technologies
for Maximum Test
Automation Coverage
Selenium
Headless (Puppeteer)
Codeless (Perfecto, Selenium IDE)
BDD

Codeless
Open
Source
Cross-Browser Testing Market Is Highly Fragmented

Cross-Browser Testing Tools (Open-Source) — Download Trends Vary
Protractor
Puppeteer
WebDriver IO
Cypress IO
Selenium WebDriver
Source: npmtrends

Tools that...
• Provide a testing structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).
• Provide assertion functions (Chai, Jasmine, Jest, Unexpected).
• Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).
• Generate code coverage reports (Istanbul, Jest, Blanket).
• Provide a browser or browser-like environment with control of scenario execution, UI testing, and more
(Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe).
• Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).
Additional Categorization of JavaScript Browser Testing Tools
Source: Medium.com

Motivation
• Receive fast feedback (unit and basic functional tests).
• Easy environment setup, no IT dependencies.
• Uses standard JS language.
• Supported and promoted by browser vendors (Google, Mozilla).
• Additional use cases to cover:
• HAR file
• Performance testing
• Basic UI
• Security
Headless Browsers Role

How would you describe your test automation scaling strategy?
• We add test scenarios based on data analysis.
• We leverage cloud and containers to scale testing.
• We divide automation scope all across the pipeline.
• Scaling is not a priority yet — we are still stabilizing automation.
• We scale by using real platforms, virtual platforms, and analyzing test data.
Live Poll

Continuous Testing — Personas Involved
Testing is led by three main personas.
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing.
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.).
Creates unit and build acceptance tests.

22 | Advanced Codeless Testing for Web Apps perfecto.io
Code-Based AutomationCodeless Automation

perfecto.io
High-Level Comparison Between Tools and Practices
Test Framework Supported
Dev
Languages
Supported
Platforms
Supported
Test
Frameworks
Setup and
Execution
Integrations Breadth of
Testing
Options
Maturity,
Documentation,
Support
Required Skillset Cloud and Execution at
Scale
Selenium/Appium
WebDriver
Java, C#,
Java Script,
Python,
Ruby,
Objective-
C
Chrome,
Safari,
Firefox,
Edge,
IE/iOS/Andr
oid
Mocha JS, Jest,
other super
set on top of
Selenium
(Protractor,
WebDriverIO,
etc.)
Download
relevant
driver, set up a
grid, network
and location
impacts
execution
speed
Plenty of
integrations
(CI, CD,
reporting,
visual testing,
cloud vendors)
End-to-
end,
security,
unit,
Robust community,
multiple bindings, best
practices
Coding skills required
(SDET Oriented)
Perfecto fully supports
Selenium and its
WebDriver
configurations. Local
execution requires
setting up a Selenium
grid
XCUITest/Espresso/
Headless/Cypress
Objective
C/Java/Jav
a Script
Chrome,
Electron
NA Embedded
into IDEs,
headless
bundles a
browser in the
FW
CI/CD UI/Unit Good documentation
and code samples
Dev Oriented Built-in Chrome/Firefox
browsers in headless,
Perfecto Cloud supports
scaling
Espresso/XCUITest
Codeless Irrelevant,
based on
record and
playback
All Proprietary UI
with
underlying
Selenium
WebDriver
APIs
Mostly
SaaS/browser
plugin
installation
Limited Functional
/UI
Growing, limited No coding skills
required (Business
Tester Oriented)
Perfecto supports
codeless in the cloud
BDD Java, Ruby,
JS, Kotlin
All Junit,
Selenium,
Appium
Open source,
Maven/Gradle
/TestNG
Plenty + APIs
(e.g. Rest
Assured)
Functional Robust community,
docs, adoption
Step-definition
development in code is
required/scenarios are
no-code (Mix of
Business Tester and
SDETs)
Perfecto Quantum is a
web/mobile BDD
framework

perfecto.io
Scaling Generates Large Test Data – Important Strategy Item

Scaling Best Practices
What to Test
(Scenarios + Platforms)?
Why These (Value)? When in the Cycle,
How Often (CI/Outside)?

• Think continuous value.
• Think maintenance and complexity to create, execute, monitor, and maintain (ROI).
• Test reliability over time.
Test Automation Certification
If { (Test Scenario) == Candidate to add Value add
{ and x– It detect defects}
{ and – It passes across multiple platforms with consistent result}
{ and – Its execution time is < 5 minutes }
Then
{ Include in Test Suite & & CI }
Else {
}
Run such certifications
once in every 2-3
software iterations.

Measuring Test Automation ROI Value
Value of test
automation
•Cycle testing time per iteration
•Test automation types (functional, non-functional, unit)
•Error detection efficiencies
•Product coverage (key business scenarios)
Cross-Team
Collaboration
•3 personas sharing load (resource, smart utilization of platforms)
•Manual testing %
•Backlog of test automation between iterations (in-sprint automation)
•Test debugging and analysis time per cycle
Test
Execution
•Automation within CI success rate
•Execution across platforms success rate
•Test redundancy/duplication/waste
•Coverage across environments (test, production)

• Testing the Boston Globe RWD site at scale with Selenium (Java)
• Smart analytics with CI dashboard and cross-platform comparison
• Technologies Used:
• IntelliJ Idea
• Selenium
• TestNG/Maven
• Perfecto
Live Demo

Perfecto’s Four Key Pillars for Continuous Testing

The 4 Key Pillars of Continuous Testing in DevOps
This is where money is spent (Time, Tools, Resources)
This is where value is being
realized, quality is improved
Unified solution serves DevOps with greater effectiveness.

Thank You!
Visit perfecto.io or follow us @perfectomobile

Enhancing Your Test Automation Scenario Coverage with Selenium - QA or the Highway

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Enhancing Your Test Automation Scenario Coverage with Selenium - QA or the Highway

Similar to Enhancing Your Test Automation Scenario Coverage with Selenium - QA or the Highway (20)

More from Perfecto by Perforce

More from Perfecto by Perforce (17)

Recently uploaded

Recently uploaded (20)

Enhancing Your Test Automation Scenario Coverage with Selenium - QA or the Highway