We all know that automation is one of the key enablers for those on the CI-CD journey.
Most teams are:
• implementing automation
• talking about its benefits
• up-skilling themselves
• talking about tooling
• etc.
In my experience, unfortunately most of the functional automation that is built is:
• not optimal
• not fit-for-purpose
• does not run fast enough
• gives inconsistent feedback, hence unreliable
Hence, for the amount of effort invested in implementing automation, are you really getting the value from this activity?
In this talk, we will discuss these challenges and why it would lead to poor ROI of automation. More importantly, we will discuss the following techniques to make automation valuable:
• know the objective for the automation framework
• establish criteria for tests to be automated
• design your framework with proper abstraction layers
• develop using appropriate design patterns
27. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Test Pyramid
Slow
More integration
Expensive
Fast
More isolation
Cheap
28. Unit (xUnit / JavaScript)
Manual / Exploratory
Web
Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Slow
More integration
Expensive
Fast
More isolation
Cheap
Workflow
API
Contract
Consumer
Provider
Test Pyramid
31. • Page-Object
• Business Layer
• Composition
• Factory Method
• Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE
Commonly used Design Patterns
@BagmarAnand
32. • Page-Object
• Business Layer
• Composition
• Factory Method
• Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE
Commonly used Design Patterns
@BagmarAnand
34. gradle / groovy
Manage Devices
Wifi Management
Optimize Device
Utilization
Appium Server
Management
Test Specifications
Business Rules (cucumber-jvm)
Business Flows
Implement Domain
functionality
(actions, verifications, rules)
Resources
Entities / Models
Pages / Screens
Trigger e2e Tests in
Jenkins Server
- Jenkins File
- Node Management
Archive artefacts
- logs, screenshots, videos, logcat, Appium logs, etc
- Cucumber reports
‘x’ Jenkins Agents on ‘y’ Mac Minis
Helpers / Utilities
(Android / iOS) Driver utilities
Loggers
File utilities
Custom Reporter
Screenshot utilities
ADB utilities Video capture
Analytics
GPU Profiling
Command Executor
Log files from Devices
Enrich Test Execution
Report
Manage Failing Tests
Upload to TTA
35. How do you measure the
quality of your product?
@BagmarAnand
36. • Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100
• Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation)
• Test Coverage = Number of detected faults/number of predicted defects.
• Test Code Coverage = Produce code coverage by e2e / UI tests
• Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100
• Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100
• Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts
planned to be executed) x 100
• %ge of work completed & yet to be completed
• Time to complete the remaining work
@BagmarAnand
(Traditional) Test Metrics
37. USE METRICS THAT ADD VALUE,
NOT JUST BECAUSE SOMEONE IS
ASKING FOR IT
@BagmarAnand
38. • Does your functional automation really add value?
• https://essenceoftesting.blogspot.com/2020/07/does-your-functional-
automation-really.html
• Tracking functional coverage from your api / functional UI (e2e) tests
• https://essenceoftesting.blogspot.com/2020/03/tracking-functional-
coverage.html
• Test Automation in the World of AI & ML
• https://www.infoq.com/articles/test-automation-ai-ml/
Resources
@BagmarAnand