Today, mobile app testing expertise is in high demand and offers an exciting career path in test/QA. However, the recent Future of Testing study, sponsored by TechWell, noted that the biggest challenge in mobile―just behind having enough time to test―is expertise. Brad Johnson shares how companies from banking to retail use data from real production users, continuous integration frameworks, cloud-based testing platforms, and real mobile devices to help ensure every user experiences top-rated performance—all the time. Brad shares insight about what to test for mobile, when to first automate, and a metric that will drive real change. Explore how organizations are communicating across teams and improving developer-to-tester collaboration with new approaches. Testers need to develop new skills ranging from software coding requirements to data science. Takeaway tips and ideas to impact your company, enhance your skill set, and propel your career with exciting options and new challenges.

1. © 2015 SOASTA. All rights reserved. June 26, 2015 1
Brad Johnson
VP Product Marketing
@bradjohnsonsv
Vancouver, BC
June 24, 2015

2. TIME

3. EXPERTISE

4. PROCESS

5. Says who

6. you.

7. 250 Testers from 6 Continents

8. Mobile is your chance to make a CHANGE

9. Learn new TECHNIQUES

10. June 26, 2015 10© 2015 SOASTA. All rights reserved.
Seek TIPS from experts

11. Embrace new TECHNOLOGY

12. Learn new TECHNIQUES

13. REALITY always trumps speculation
Static aggregates (ok)

14. REALITY always trumps speculation
Real Time (best)

15. REALITY always trumps speculation
Data science - better than best!

16. June 26, 2015 16© 2015 SOASTA. All rights reserved.
 ASAP
 Simple
 Time Consuming
 Mundane
 Rhythm
 Goal : Continuous Integration execution

17. June 26, 2015 17© 2015 SOASTA. All rights reserved.

18. June 26, 2015 18© 2015 SOASTA. All rights reserved.
o Small Batch Sizes

19. June 26, 2015 19© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control

20. June 26, 2015 20© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies

21. June 26, 2015 21© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation

22. June 26, 2015 22© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback

23. June 26, 2015 23© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds

24. June 26, 2015 24© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments

25. June 26, 2015 25© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration

26. June 26, 2015 26© 2015 SOASTA. All rights reserved.
 Device reaction to app functionality
 Battery, CPU, Memory
 Dependent services and 3rd party apps/ads
 Separate back and front end performance
 User experience under every traffic condition
 Performance to revenue correlation

27. June 26, 2015 27© 2015 SOASTA. All rights reserved.
Seek TIPS from experts

28. June 26, 2015 28© 2015 SOASTA. All rights reserved.

29. June 26, 2015 29© 2015 SOASTA. All rights reserved.
RetestTest Development Infra setup RepeatFixTest Analyze
TIME
Beware Automation Solutions that Hinder Speed
Developers code tests, but not faster.

30. June 26, 2015 30© 2015 SOASTA. All rights reserved.
RetestTest Development Infra setup RepeatFixTest Analyze
TIME
Analysis is still slow & manual
Beware Automation Solutions that Hinder Speed

31. June 26, 2015 31© 2015 SOASTA. All rights reserved.
RetestTest Development Infra setup RepeatAnalyze
Develop plans Post-test Validate
Traditional
Approach
Test Fix
/
Analyze
Fix
Test Dev Test
/
Analyze
Fix
Test Dev Test
/
Analyze
Fix
Test Dev Test
/
Analyze
Fix
Test Dev TestModern
Approach
OR
/
Analyze
Fix
Test Dev Test /
Analyze
Fix
Test Dev Test
AUT/device
setup Manual

32. June 26, 2015 32© 2015 SOASTA. All rights reserved.
250 hours of manual tests
compressed to 1 hour
Regression tests not run until
feature complete now run nightly
4 months to overnight
25% faster Time To Market

33. June 26, 2015 33© 2015 SOASTA. All rights reserved.
“(Mobile test automation) has been critical in
improving both the speed to deliver
and the quality of our mobile apps.”
“We decreased our test cycles from three weeks to a couple of days.”
3 WEEKS 2 Days

34. Embrace new TECHNOLOGY

35. RUM =
“What to
Test”
(real user monitoring)

36. June 26, 2015 36© 2015 SOASTA. All rights reserved.
o Rapid & accurate test development
o Reliable, maintainable test automation
o Stability across mobile OS platforms
o Device performance measurement
o Real-Time feedback (via Continuous Integration is a +)

38. June 26, 2015 38© 2015 SOASTA. All rights reserved.
 Rank active devices
 Device usage
 Weight matrix to device
 Know problem areas per device
 Device Access
 Private
 Public

39. June 26, 2015 39© 2015 SOASTA. All rights reserved.

40. June 26, 2015 40© 2015 SOASTA. All rights reserved.
Source: Mobile Test Coverage Index (Perfecto)

41. June 26, 2015 41© 2015 SOASTA. All rights reserved.
o Behind the firewall
o Always available
o Online access for all your teams
o 24x7, secure, remote access to
devices
o Manual & automated testing
Hands on when you
need them.

42. June 26, 2015 42© 2015 SOASTA. All rights reserved.
o Dedicated
o Latest in-market devices
o Maintained for you
o Secure
o API’s to install apps,
reboot/wake-up, launch tests
Perfect for automated mobile
regression testing

43. June 26, 2015 43© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Database
CDN
Credit
Card
Web
analytics
Other
Svcs…
Performance
Tester(s)
Virtualized services3rd party services
Load Testing
Tool

44. June 26, 2015 44© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Database
CDN
Credit
Card
Web
analytics
Other
Svcs…
Performance
Tester(s)
Virtualized services3rd party services
Cloud
Servers
Load Testing
Tool

45. June 26, 2015 45© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Database
CDN
Credit
Card
Web
analytics
Other
Svcs…
Performance
Tester(s)
Virtualized services3rd party services
Cloud
Servers
Load Testing
Tool

46. June 26, 2015 46© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Database
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
CDN
Credit
Card
Web
analytics
Other
Svcs…
Performance
Tester(s)
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
3rd party services Virtualized services
Cloud
Servers
Real time
analytics
Load Testing
Tool

47. June 26, 2015 47© 2015 SOASTA. All rights reserved.
Utilize Real User data to drive
dev & test specifications

48. June 26, 2015 48© 2015 SOASTA. All rights reserved.
CI runs
unit tests
Utilize Real User data to drive
dev & test specifications

49. June 26, 2015 49© 2015 SOASTA. All rights reserved.
CI runs
unit tests
Utilize Real User data to drive
dev & test specifications
Spin up suitable, consistent
test environments in the cloud
or the lab

50. June 26, 2015 50© 2015 SOASTA. All rights reserved.
Mobile
automation
runs in
remote or on
premise
devices
CI runs
unit tests
Utilize Real User data to drive
dev & test specifications
Spin up suitable, consistent
test environments in the cloud
or the lab

51. June 26, 2015 51© 2015 SOASTA. All rights reserved.
Iterative load
tests run as CI
tasks
Mobile
automation
runs in
remote or on
premise
devices
CI runs
unit tests
Utilize Real User data to drive
dev & test specifications
Spin up suitable, consistent
test environments in the cloud
or the lab

52. June 26, 2015 52© 2015 SOASTA. All rights reserved.
Iterative load
tests run as CI
tasks
Mobile
automation
runs in
remote or on
premise
devices
CI runs
unit tests
Daily (or more frequent) updates keep
teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Utilize Real User data to drive
dev & test specifications
Spin up suitable, consistent
test environments in the cloud
or the lab

53. June 26, 2015 53© 2015 SOASTA. All rights reserved.
Iterative load
tests run as CI
tasks
Mobile
automation
runs in
remote or on
premise
devices
CI runs
unit tests
Manage to a
performance
baseline
Daily (or more frequent) updates keep
teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Utilize Real User data to drive
dev & test specifications
Spin up suitable, consistent
test environments in the cloud
or the lab

54. June 26, 2015 54© 2015 SOASTA. All rights reserved.
EXPERTISE

55. June 26, 2015 55© 2015 SOASTA. All rights reserved.
TIM
E

56. June 26, 2015 56© 2015 SOASTA. All rights reserved.
PROCESS

57. June 26, 2015 57© 2015 SOASTA. All rights reserved.
Mobile Test
Automation

58. June 26, 2015 58© 2015 SOASTA. All rights reserved.
Mobile Test
Automation
Real Test
Devices

59. June 26, 2015 59© 2015 SOASTA. All rights reserved.
Mobile Test
Automation
Performance
Testing
Real Test
Devices

60. June 26, 2015 60© 2015 SOASTA. All rights reserved.
Mobile Test
Automation
Performance
Testing
Real Test
Devices
Solid
Integration

61. Mobile is your chance to make a CHANGE

62. You CAN do it!

63. June 26, 2015 63© 2015 SOASTA. All rights reserved.
SOASTA.com
o Whitepaper: 7 Steps to Pragmatic Mobile Testing
• http://info.soasta.com/Whitepaper-7-Steps-to-Pragmatic-Mobile-Testing.html
o July 7 Webinar “Strategies for Mobile Automation Success”
• http://info.soasta.com/strategies-for-mobile-test-automation-success.html
o Recorded webinars - e.g. “Pragmatic Mobile Testing”
• http://www.soasta.com/webinars/
Others
o “Future of Testing” (search Stickyminds/Techwell)
o “Mobile Test Coverage Index” (search Perfecto)
See “7 Steps” Session @ 3:00pm!

64. June 26, 2015 64© 2015 SOASTA. All rights reserved.
@bradjohnsonsv
bjohnson@SOASTA
Brad Johnson