Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users. In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD. To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps. The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below: Functional Automation approach - identify and automate user scenarios, across supported regions Testing approach - what to test, when to test, how to test! Manual Sanity before release - and why it was important! Staged roll-outs via Google’s Play Store and Apple’s App Store Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

1. Testing & Release Strategy for
Native Apps
@BagmarAnand
Anand Bagmar
Software Quality Evangelist, Essence Of Testing

2. @BagmarAnand
https://dragonflytraining.files.wordpress.com/2013/10/man-with-question-01.png

3. Testing & Release Strategy for
Native Apps
@BagmarAnand
Anand Bagmar
Software Quality Evangelist, Essence Of Testing
Quality

4. About Me
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand

5. Web Vs Native Apps
@BagmarAnand
https://www.flickr.com/photos/brendanadkins/505630681/

6. Similarities
in
Web & Native Apps Testing
@BagmarAnand

7. Differences
in
Web & Native Apps Testing
@BagmarAnand

8. Release approach
for
Web Apps
@BagmarAnand

9. @BagmarAnand
Similarities
in
Web & Native Apps Releases

10. Differences
in
Web & Native Apps Releases
@BagmarAnand

11. @BagmarAnand
CI / CD
for
Web Apps & Native Apps

12. @BagmarAnand
Case Study – B2C

13. @BagmarAnand
How would you
Test & Release
such a product?

14. • Big user-base
• Millions of users, across geographies
• 13K+ unique device / OS combinations
@BagmarAnand
Reality

15. • Functionality has to work!
• Releases
• Frequent
• Black-out dates
• Analytics – as critical as some core features
• Partner Integrations & Payments
@BagmarAnand
Non-Negotiable Release Criteria

16. • Slow / inefficient / incomplete Testing
• Poor quality code-base, No visibility / control on impact of
changes
• Limited Test Automation
• Business Metrics
• Analytics – as critical as some core features
• Releases / issues found during black-out dates
@BagmarAnand
Tipping points

17. Quality & Release Strategy
@BagmarAnand

18. Quality Strategy
@BagmarAnand

19. • Setup relevant Agile / XP practices
• Established and enforced Definition-of-Done
• No ad-hoc implementation (by directly talking to devs)!!
@BagmarAnand
Established structured WoW

20. • Build Quality-in by Testing early
• Feature kick-off
• à Business Requirements
• Story kick-off
• à User impact
• Dev-Box testing
• à Implement what was expected
• Thorough Story Testing & Exploratory Testing
• à Test was was implemented
• Product Owner sign-off per story & feature
• à Involve everyone, Quality is Team Responsibility
@BagmarAnand
Established structured WoW

21. • Automate as much as you can
• Unit Tests
• Integration Tests
• Functional Tests (e2e) à focused on user-journeys
• Analytics Tests
@BagmarAnand
Automation is the KEY!

22. Value of Automation
Run the tests on every change
@BagmarAnand

23. @BagmarAnand
Run E2E Tests
App Build
API
Deployment
e2e
CI à CD

24. Solution!
@BagmarAnand

25. MAD-LAB – Mobile Automation Devices LAB

26. MAD-LAB
@BagmarAnand

27. • In-house device lab (> 30 devices – Android, iPhones, Tablets)
• Cucumber-JVM / Appium / Cucumber-Reporting
• Distributed Test Execution
• Maximize device utilization
• Manage devices / Appium
• Analytics Validation
• Performance benchmarking
• Easy to scale (Mac-Mini, powered USB hubs, durable USB cables)
@BagmarAnand
Key Features of MAD-LAB

28. gradle / groovy
Manage Devices
Wifi Management
Optimize Device
Utilization
Appium Server
Management
Test Specifications
Business Rules (cucumber-jvm)
Business Flows
Implement Domain
functionality
(actions, verifications, rules)
Resources
Entities / Models
Pages / Screens
Trigger e2e Tests in
Jenkins Server
- Jenkins File
- Node Management
Archive artefacts
- logs, screenshots, videos, logcat, Appium logs, etc
- Cucumber reports
‘x’ Jenkins Agents on ‘y’ Mac Minis
Helpers / Utilities
(Android / iOS) Driver utilities
Loggers
File utilities
Custom Reporter
Screenshot utilities
ADB utilities Video capture
Analytics
GPU Profiling
Command Executor
Log files from Devices
Enrich Test Execution
Report
Manage Failing Tests
Upload to TTA

29. Only Automation is
NOT SUFFICIENT
@BagmarAnand

30. • Intelligent / Optimized Non-Automated Testing
• Exploratory Testing
• Sanity Testing
• On-Field
• Business
• Pre-release sanity check-list (value / risk / not-automated, …)
• Functionality
• Payments
• Analytics
@BagmarAnand
Functional Testing Approach

31. • On-Field Sanity Testing
• Staged Releases
• Sentiment Analysis
• Review comments (PlayStore / App Store)
• Social Media (Twitter, Instagram, facebook, …)
• Extensive monitoring of staged releases
• Analytics
• Crashes
• Support Tickets
@BagmarAnand
Release Approach

32. Thank you!
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand