3. SoLoMo Breakdown
So Lo Mo
Social Local Mobile
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 3
4. SoLoMo Coming Fast
In the past six monthsâŚ
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 4
5. QA Leaders Better Sit Down
⢠Media & analysts arenât the only ones talking about it
â Marketing
â Product
â Engineering
â IT
â Sales
â C-suite
⢠Want a seat at the table?
â Get informed
â Get involved
â Develop an opinion
â Advocate strongly
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 5
7. Not Just For B2C Anymore
⢠The social revolution isnât coming⌠itâs here
â Nothing is disconnected
â No industry is immune
⢠The stats on social media adoption in Fortune 500 firms:
â 77% integrate Twitter
â 84% incorporate LinkedIn
â 93% utilize Facebook
â 57% publish on YouTube
⢠Gartner: By 2016, social
integrated w/ nearly all apps
â Websites & apps
â Mobile apps & mobile web
â Desktop clients
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 7
8. Goes Soooo Much Further
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 8
13. Mobile Apps Boom Market
⢠ABI: mobile revenues will hit $133 billion in 2014
⢠B2B mobility no longer just for email
â Business apps: CRM, ERP, HR systems
â Productivity apps: docs, spreadsheets, presentations
â Collaboration apps: email, IM, publishing
â Medical apps: health records, patient education
⢠B2C mobility growth even steeper
â Retail: location-intelligent m-commerce
â Media: magazines & newspapers going purely digital
â Travel: mobile bookings, check-ins, maps, deals
â Education: tablets in every classroom
â Healthcare: patient records, physician notes
â Social: no explanation required
â Gaming: ditto
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 13
14. Mobileâs Just Getting Started
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 14
15. Did I MentionâŚ
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 15
16. Psst⌠Itâs Not Just Mobile
⢠Smartphones & tablets make QA immeasurably tougher
⢠Now imagine a whole world of apps:
E-Readers
Cars
Household Appliances
Smart Homes
Gaming Consoles
Connected TVs |
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 16
18. Brave New World
⢠Impact on design, dev & IT well-recognized & resourced
⢠Sadly, but not at all surprisingly, not true for testing
â Want proof? Specialized designers & devs⌠but who tests?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 18
19. Social: Impact On Testing
⢠Social presents functional test challenges
â Testing code & integrations that arenât yoursâŚ
â That constantly changesâŚ
â And you get the blame
⢠As well as security & privacy pitfalls
â Internally, a channel for spam, malware and data breaches
â Confidential data may be posted and trigger regulatory penalties
â Externally, increasingly used for user authentication on your products
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 19
20. Social: Impact On Testing
⢠New challenges include how will you:
â Regression test multiple social integrations for web & mobile apps?
â Write (and execute) test cases with zero visibility into socialâs roadmaps?
â Automate anything?
â Test sharing/liking in a QA environment without publishing via social?
â Test your social footprint?
- FB page (and apps)
- LinkedIn page
- Twitter page
â Ensure security/privacy if you enable users to use social sign-on?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 20
21. Local: Impact On Testing
⢠Not just a mobile problem
â Localization testing
â Geo-based personalization
⢠But it is a mobile problem too
â Apps used outside the confines of the
QA lab, under in-the-wild conditions
⢠Impossible to test effectively from
only inside the lab
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 21
22. Local: Impact On Testing
⢠New challenges include, how will you:
â Replicate location (answer: you donât)
â Test in NYC, SF, LA, DFW, CHI?
â Test abroad
- SIM cards
- Carriers
â Test GPS-based intelligence?
â Test IP-based localization?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 22
23. Mobile: Impact On Testing
⢠New challenges include how will you:
â Native apps: Test across ever-changing OS, devices and carriers?
â Mobile web: Test across wide range of OS and mobile browsers?
â Either way, how does your testing replicate real-world conditions
- Specific and varied locales
- Tower-to-tower hopping
- Roaming
- Low battery
- Touch screen actions (pinch, swipe, pan, etc)
- Portrait vs. landscape
- Display definition
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 23
24. Mobile: Impact On Testing
⢠Mobile web vs. native apps
â 2009: 5-horse race
â 2010: 4-horse race
â 2011: 2-horse race (depending upon your industry)
â 2012: 3 horses (including HTML5-driven mobile web)
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 24
25. F|R|A|G|M|E|N|T|E|D
⢠Android alone (and thatâs only by OS version)
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 25
26. Mobile Testing Landscape
⢠Form factors matter again
â Phones (feature & smart)
â Tablets
â Hybrids
⢠Prohibitive HW & SW costs
⢠Here we go again
â Late-90s web, part deux
- Lack of mature tools
- Lack of usability standards
- Lack of security standards
- Extremely fluid landscape
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 26
28. Test Coverage
Web Testing Matrix Is Tough
Features
OS & Browser
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 28
29. Test Coverage
Mobile Testing Matrix Is Insane
Features
Handset Makers
OS & Browser & Models
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 29
30. Test Coverage
Mobile Testing Matrix Is Insane
Features
Wireless
Handset Makers Carriers
OS & Browser & Models
| 30
In-The-Wild Testing for Functional + Security + Load + Usability + Localization
31. Test Coverage
Mobile Testing Matrix Is Insane
Features
Location
Wireless
Handset Makers Carriers
OS & Browser & Models
| 31
In-The-Wild Testing for Functional + Security + Load + Usability + Localization
32. Increasingly Complex Testing
⢠The laughably simplistic notion of a âtest matrixâ
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 32
34. Sufficiently Freaked Out?!?
⢠Donât be⌠the good news is that testing ainât dead!
â Chance for QA leaders to step upâŚ
â To lead the chargeâŚ
â To defend users and app quality
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 34
35. âŚAnd The Bad NewsâŚ
⢠You should be scared⌠need to completely re-think testing
â Canât automate our way out of this
â Canât hire or train our way out
â Canât solve this problem from soley within the lab
⢠More distributed problem than ever before
â Need a new testing mindset
â Need to bridge the growing gap
- Between testing & users
- Between artificial and real
- Between in-the-lab & in-the-wild testing
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 35
36. Lab-Based Testing Innovation
Manual vs.
Automation Managing vs.
Leading Teams
In-House vs.
Outsourced Agile vs.
Waterfall
Exploratory vs. Testers vs.
Test Cases Engineers
Offshore vs.
Nearshore
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 36
37. Users Work & Live In The Wild
I n
The
Wi l
In-The-Wild Testing for Functional + Security + Load + Usability + Localization
|
37
38. To Make Matters Worse
⢠Users are less tolerant of quality failures
⢠Social media & app stores give everyone a megaphone
⢠And their problem⌠is your problem
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 38
39. The Challenge
A Call To Action For
Test Leaders (Two, Actually)
|
39
40. 1. Look Beyond The Lab
⢠Yes, keep testing in-the lab
â Keep training in-house
â Keep outsourcing
â Keep automating
â Keep innovating
⢠Recognize it will never again be sufficient on its own
⢠Build your companyâs plan for testing in-the-wild
â Real-world conditions: Live testers, real devices, imperfect connectivity
â Mirror your user base
- Technologically: OS, browser, anti-virus, device, carrier
- Geographically: Continent, country, city, language
- Demographically: Age, gender, education, employment, industry, hobby
⢠In-house QA leadership is vitally important
â You have the business and product context
â You know what your firm cares about
â You need to be the orchestrator who leads this symphony
- What to test in the wild, how to test in the wild, holistic test planning
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 40
41. 2. Be The Voice For QA
⢠SoLoMo is coming â with or without QAâs buy-in
⢠Want QA to be heard? Want quality to be built in?
â Donât wait: No one else is coming to the rescue
â Get informed: Develop an thoughtful opinion
â Get involved: Find out when/where decisions are being made
â Be proactive: Build a plan for quality from the ground up
â Be protective: Youâre a protector of the user⌠of the brand
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 41
42. The Challenge
SoLoMoWoSto
A Few Stories (if thereâs time)
|
42
43. The Challenge
Questions?
Answers.
Matt Johnston | CMO @ uTest
mattj@utest.com | @matjohnston
|
43