Slides from my talk on MAD-LAB delivered in vodQA - The Saga Continues! at Vuclip / ThoughtWorks See blog for more details: https://essenceoftesting.blogspot.com/2017/07/why-i-needed-to-build-my-own-mad-lab.html Description: Building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it! Setting the stage - I am coordinating all Testing activities for VIU - an OTT (over-the-top entertainment) product available on Android, iOS and WAP platforms. This product delivers high quality, popular video content in various different languages for consumers in various different regions. One of the main items in my charter is to implement functional test automation for consumer / user functionalities, and to provide quick feedback to the team and stakeholders on the “true” state of the product on all supported platforms for VIU. In this talk, using the above set context, I will be sharing the following: The automation strategy Chosen tech-stack How (and why) no cloud-based solution worked for me Implementation details - MAD-LAB - which arose from the learnings of the failed experiments - which resulted in setting up my own real-device in-house lab. The core implementation (code) of MAD-LAB (already open-sourced)

1. MAD-LAB
Mobile Automation Devices LAB
Anand Bagmar
Software Quality Evangelist

2. About Me
@BagmarAnand
Blog – essenceoftesting
about.me/anand.bagmar

3. What do you expect
from this session?

4. Context

5. Ecosystem
Platforms
• Android (> v4.4.2)
• iPhones (> v5)
• Desktop / WAP
Regions
• Indonesia
• Malaysia
• India
• Middle-East
• ...

6. Dynamic Content

7. Search

8. Download

9. Offers

10. ADs

11. What this means ... Reality!
• Unique installs (Android) - > 1.75M
• Unique # of devices (Android) - > 13K

12. What this means ... Product!
• One product, One experience
• Unique / Regional / Local content
• User detection & offers based on origin
• Region
• Carrier network
• ...

13. What this means ... Testing!
• Video Quality, Dynamic Content, Download Expiry
• Simulating Carriers / Networks
• Consumer focus!
• Experience
• Device / OS combinations

14. Business Requirements
• Provide visibility into -
• What is Tested
• Overall Quality of Product

15. APIs
Payment
partners
CMS
....
....
Architecture

16. APIs
Payment
partners
CMS
....
....
Where are these?

17. How would you
approach
Automation for such
a product?

18. What Metrics make
sense for
determining Quality?

19. Automation
Strategy

20. Automation Criteria
• E2E tests should focus on Business Rules & User Scenarios
• Reports should reflect the above
• Understood by all Business & Technical stakeholders
• Generated automatically
• Should run frequently
• Every new change in the product-under-test
• On a variety of device / OS combinations

21. What to Automate?
Validate what was working, continues to work!
• Identify user journeys / scenarios
• Prioritize it!

22. Run Automation against...
Usage patterns from Analytics / PlayStore / App Store
• Identify devices that matter
• Identify devices that are “automation-friendly”

23. Automation Tech Stack
• Cucumber-JVM à For specifying Business Rules
• Appium à To allow integration with Android / iOS
• Gradle / Groovy à As build tool scripting language
• Jenkins à Continuous Integration Server
• TTA - Test Trend Analyzer à Trend & Failure Analysis

24. MVP / POC

25. Scale
&
Repeat

26. Experiment, Learn &
Evolve

27. Experiments
• Run tests on Emulators
• Use Cloud-based services (Local Vs Remote)
• SauceLabs
• TestObject
• Amazon Device Farm
• pCloudy
• Devices

28. Some Experiments didn’t work
• Emulators
• Video Player
• Location
• Cloud-based services
• Location
• Tech Stack
• Devices

29. MAD-LAB

30. MAD-LAB – Mobile Automation Devices LAB

31. Innovations in MAD-LAB
• Device management (selection, cleanup, app install and
uninstall)
• Parallel test execution (at Cucumber scenario level) -
maximising device utilisation)
• Appium server management
• Adb utilities (periodic ADB server disconnects)
• Custom reporting using cucumber-reports
• Video recording of each scenario and embedding in the
custom reports
• Analytics Automation

32. Conclusion

33. Learnings (so far)
Blue-print is important!
• Stepping-stone approach
Experiment!
• Know your objective(s)
• Fail-fast!
Make the Product Testable!!

34. References
• Sample code
• https://github.com/anandbagmar/cucumber-jvm-appium-infra
• Blog posts
• https://essenceoftesting.blogspot.com/search/label/madlab
• TTA (Test Trend Analyzer)
• https://essenceoftesting.blogspot.com/search/label/tta

35. Thank you!
@BagmarAnand
Blog – essenceoftesting
about.me/anand.bagmar