Ensuring quality in cloud and mobile applications

Ensuring Quality in Cloud and
Mobile Applications
Gerard Meszaros
infoq2012hz@gerardm.com

InfoQ Hangzhou 2012 1 Copyright 2012 Gerard Meszaros

My Background

• Software developer
80’s • Development manager
Embedded
----- • Project Manager Telecom
• Software architect
90’s
• OOA/OOD Mentor
• Requirements (Use Case) Mentor
----- I.T.
• XP/TDD Mentor
• Agile PM Mentor
00’s
• Test Automation Consultant & Trainer
Gerard Meszaros
• Lean/Agile Coach/Consultant infoq2012hz@gerardm.com
Product & I.T.

Agenda
•  What’s Special About Cloud and Mobile Apps?
•  What is Quality?
•  A Quality Strategy
•  Testing Cloud-Based Apps
•  Testing Mobile Apps
•  No DOH!
•  Ensuring Design for Testability
•  Tools, Frameworks and Patterns


Current Challenges
The Good Old Days Current Reality
•  Captive Audience •  Users Have Real Choice
–  Have no real choice –  Many alternatives
•  Stable requirements •  Rapid change
–  Can have a single release –  Highly iterative; multi releases
•  Long lead times •  Rapid change
–  Can have long manual test –  No time for extensive manual
cycle testing
•  Relatively simple •  Highly Complex
environment Environment
–  Just a few inputs (user, DB) –  Many sources of inputs
–  Mostly synchronous –  Sync and Async


Typical Desktop Application
•  UI Logic and
Business Logic
run on desktop Other System

•  Server hosts Other System
the database.
Desktop DB Server

User Business
Database
Interface Logic

OS Services

Other System


Typical Cloud-Based Application
•  UI Logic runs in (A.K.A. Software-as-a-Service or SaaS)
browser
•  Business Logic is split Other System
between Server and Desktop
Browser
Browser
•  Both use browser Browser Server
services
–  Widgets, Cookies,
SSL,DOM rendering UI Business Business
Logic Logic Logic Database
•  Business logic may
also interact with
other systems.
Browser Services
–  E.g. IP-based location Container Services
•  Browser-based BL
may receive data OS Services
from server: Other System
–  synchronous
(request-reply)
–  asynchronous Dynamic scripting languages
(event-based) •  won’t report compile errors until run.


Typical Web-Based Mobile Application
•  UI Logic runs in (A.K.A. Software-as-a-Service or SaaS)
browser
between Server and Device
Browser
Browser
•  Both use browser Browser Server
services
–  Widgets, Cookies,
SSL,DOM rendering UI Business Business
also interact with
other systems.
Browser Services
may receive data Platform Services
–  synchronous Sensors OthrApps
(request-reply) GPS Acc Contacts
–  asynchronous Msg’ing
(event-based) Gyro Batt Gyro


Typical Mobile Application
•  UI Logic runs in
device
between Device and Device
Servers
•  Both use Platform Server
services
–  Config, Prefs,
Widgets,SSL, UI Business Business
also interact with
other systems.
Platform Services
Config Time
may receive data Prefs Prefs
–  synchronous Sensors OthrApps
(request-reply) GPS Acc Contacts
–  asynchronous Msg’ing
(event-based) Gyro Batt Gyro


Outputs / Controlables
•  Screen (LCD) •  Other Applications
•  LED indicators •  Add event to
•  Vibration motor Calendar
•  Camera •  Add/edit/read contact
•  Wifi, Bluetooth, 3G,
NFC Radios (on/off)


Other static inputs
Physical Configuration User Preferences
•  Screen size •  Language / locale
•  Screen density •  Keyboard prefs
•  Roles of Physical •  Application prefs
buttons
•  Camera(s)
capabilities


Sources of Events
Sensors Services
•  Proximity •  Send message (e-mail,
SMS, MMS, IAM)
•  Accelerometer
•  Voice interpretation
•  GPS •  Output Audio
•  Signal Strength (Wifi, •  IP Location
3g, BT, NFC) •  Wifi Location
•  Compass •  GPS Location
•  Get/Show Advertising
•  In-App purchasing


Events & Reponses
•  Low Battery •  Valid Responses
•  Lost Data •  Data returned sync
Connection •  Data returned async
•  Change in data •  Error responses
capabilities
•  Low resources
•  Geofence crossed
•  Can’t communicate


Example – Screen Layout
•  Large number of available screen sizes
•  Need to verify app uses screen space well
•  Four different orientations
•  For each screen size and orientation:
– need to verify how various screen elements are sized
and placed
•  May be possible to do manually for IOS
–  iPhone 1, iPhone 4, iPhone 5, iPad1/2, iPad3, iPod Touch
•  Not possible to do well manually for Android
– Automation is essential; best done at component level


Example – Screen Layout
•  What about changes to orientation of the
device?
•  That’s 4P2 or 12 transitions we need to test
North ->South South -> North East -> North West -> North
North -> East South -> East East -> West West -> East
North -> West South -> West East -> South West -> South
•  And we have to run this test at various states
of the app
•  So, 12 tests in 10’s or 100’s of states
•  On 10’s to 100’s of device configurations


Agenda
•  No DOH!


Google Quality Checklists
Core App Quality Tablet App Quality
Quality Criteria 1. Test for Core App Quality
•  Design and Interaction 2. Optimize your layouts
•  Functionality 3. Use the extra screen area
•  Performance and Stability 4. Use assets designed for
•  Google Play tablets
Testing 5. Adjust fonts and touch targets
•  Setting Up a Test 6. Adjust HomeScreen widgets
Environment 7. Offer the app's full feature set
•  Test Procedures 8. Don’t require hardware
You Should Also Read features
•  Tablet App Quality Checklist 9. Declare tablet screen support
•  Improving App Quality 10. Follow best practices for
publishing in Google Play
http://developer.android.com/distribute/googleplay/quality/core.html
http://developer.android.com/distribute/googleplay/quality/tablet.html


What Exactly is Quality?
Building the Right Thing Building the Thing Right
•  Fitness-for-purpose •  Good architecture
•  Understand the users •  Well implemented in
•  And their usage env’ts code
•  Well tested at all levels

Practices: Practices:
•  UxD •  Automated Unit Testing
•  Usability testing •  Test-Driven
Development
•  Lean Startup stuff
•  ATDD
•  Exploratory Testing

Traditional Test Cycle

e
S oftwar

Requirements Requirements

Verification &
Development Acceptance

Agenda
•  No DOH!


How to Achieve Quality?


What Should We Test?
•  Product X consists of components A, B, and C
•  Components A and B are well tested but have
never been used together.
•  Component C is not well tested

•  We need to Component-test C,
•  Interaction test A vs. B
•  Interaction test A vs C


A Quality Strategy
•  Use automated tests at unit and component
level
– to maximize confidence in quality of components.
•  Use system-level tests focusing on the logic
not already tested
– mostly integration tests
– automated if possible, manual otherwise
•  Use Exploratory testing to find out what else
could possibly go wrong.
– Let testers follow their hunches to find problem areas


A word about Terminology
•  Testing vs. Checking
– Checking is verifying that a behaviour occurs as
expected
– Testing is seeing what happens when I do something
– Automated Testing is therefor an oxymoron; it should be
automated checking.
•  Direct vs Indirect Inputs (and outputs) Depended-on
Client Objects (Users) System Under Test Components
Direct
Indirect DOC
Client Input
Output
Input SUT
methodA( ) DOC
Client
Output methodB( )
Client Direct Indirect
DOC
Output Input

Test Automation Pyramid
•  Tools to support effective
exploratory testing Exploratory Tests

•  A small number of tests
for the entire application System
& workflow Tests
– Ensure application(s) support Component
users’ requirements Tests
•  Medium number of
functional tests for major
components Unit Tests

– Verify integration of units
•  Large numbers of very
small unit tests
– Ensures integrity of code
InfoQ Hangzhou 2012 Pyramid originally proposed by Mike Cohn
24 Copyright 2012 Gerard Meszaros

Agenda
•  No DOH!


Testing Cloud-Based Application
•  Natural interface (A.K.A. Software-as-a-Service or SaaS)
between Browser &
Server facilitates Other System
testing
–  Server testing
–  Browser testing with Browser Server
Server stubbed out
UI Business Business
•  Component tests for Logic Logic Logic Database
Browser Business
Logic bypassing UI
–  Requires API on BL Browser Services
Container Services
•  Also need unit tests
for Business Logic OS Services
Other System
code


between Browser &
testing
Server stubbed out Server
UI Business Business
•  Component tests for Logic LogicTest Logic Database
Browser Business
Logic bypassing UI
Container Services
Other System
code


between Browser &
testing
Whole Server
Server stubbed out
Client UI Business Test
Business
•  Component tests for Logic
Test Logic StubLogic Database
Browser Business
Logic bypassing UI
Container Services
Other System
code


between Browser &
testing
Server stubbed out Unit Unit Test Unit Unit Test
UI Test Stub Test
Business Stub
Business
•  Component tests for Logic Unit Logic Test Unit Database
Logic Test
Browser Business Unit Unit
Test Stub Test Stub
Logic bypassing UI
Container Services
Other System
code


between Browser &
testing
Server
Server stubbed out Business Test
UI Business Stub Business
•  Component tests for Logic Logic Database
Logic Logic
Browser Business Test
Logic bypassing UI
Platform Services
Browser Services
–  Requires API on BL Container Services
Test Stub
Other System
code


between Browser &
testing
–  Browser testing with Browser Server DB
Server stubbed out Server Test
UI Business Business Stub
•  Component tests for Logic Database
Logic Logic Logic
Browser Business Test
Logic bypassing UI
–  Requires API on BL Browser Services Platform Services
Container Services
•  Also need unit tests Test Stub
Other System
code


between Browser &
testing
Server stubbed out
Whole
Business
System for UI
•  Component tests
Business
Database
Logic Logic Logic
Test
Browser Business
Logic bypassing UI
Container Services
Other System
code


Special Considerations
•  Different Browsers and Browser-Versions
– Eg. Rendering behaviour, widget behavior, etc.
•  Different Host-OSes
– For OS-based services such as file choosers, etc.
•  Different languages & locales
•  Browser resizing


Testing Browser-Based Mobile
Similar to testing Application
Cloud-based apps
but: Other System
•  Different browsers
& Screen sizes
Device Server
•  Need to test with
events from UI Business Business
relevant sensors Logic Logic Logic Database
–  E.g. Orientations
and changes
Platform Services
Container Services
Config Time
Prefs Prefs
Other System
Sensors OthrApps
GPS Acc Contacts
Msg’ing
Gyro Batt Gyro


Testing Browser-Based Mobile
Similar to testing Application
Cloud-based apps
but: Other System
•  Different browsers
& Screen sizes
Device Server
Server
•  Need to test with Business Test
events from UI Business Stub Business
Logic Database
relevant sensors Logic Logic Logic
Test
–  E.g. Orientations
and changes
Platform Services
Platform Services
Container Services
Test
Config Stub
Time
Prefs Prefs
Other System
Sensors OthrApps
GPS Acc Contacts
Msg’ing
Gyro Batt Gyro


Testing Native Mobile Application
•  To test device BL,
need to:
–  Provide API Other System
bypassing UI
–  Trap outgoing
reqests to sensors
and services Device Server
–  Inject responses
from same UI Business Business
•  Need ability to:
–  Stub out platform
services, sensors
and other apps Platform Services
Container Services
Config Time
•  Need unit tests for Prefs Prefs
all code
Other System
Sensors OthrApps
GPS Acc Contacts
Msg’ing
Gyro Batt Gyro


need to:
bypassing UI
reqests to sensors Server
and services Device Server
Business Test
–  Inject responses
from same UI Business Stub Business
Logic Database
Logic Logic Logic
•  Need ability to: Test
services, sensors Platform Services
Platform Services
and other apps Container Services
Test
Config Stub
Time
all code
Other System
Sensors OthrApps
GPS Acc Contacts
Msg’ing
Gyro Batt Gyro


need to:
bypassing UI
reqests to sensors Dependency
and services Device
Characterization Server
–  Inject responses Tests
from same UI Business Business
•  Need ability to:
services, sensors
and other apps Platform Services
Container Services
Config Time
all code
Other System
•  Characterization Sensors OthrApps
Tests to detect GPS Acc Contacts
changes in Msg’ing
dependencies Gyro Batt Gyro


Agenda
•  What’s Special About Cloud and Mobile
Apps?
•  No DOH!


DOH!
•  DOH! = Debugging on Hardware
•  Should be avoided because
•  Slow, tedious, non-deterministic
•  Complete code coverage is impossible
because we cannot cause on-demand:
– Hardware errors
– “Impossible” return codes or exceptions
– All but most common events from sensors

You have to ask yourself the following question:

Do I feel lucky?

James Grenning on DOH!
•  Reasons why DOH is not feasible:
– There is no hardware, it’s being concurrently developed.
– The hardware is expensive, so team members have to
share it.
– DOH is slow, with tools that may be less capable than
the development system native tools.
– The hardware has bugs, as if DOH! isn’t slow enough;
buggy hardware makes it extra difficult.
– Building code for the target is slow.
– Downloading to the target is slow.
– The embedded processor is probably slow too.


How No DOH?
•  How do we avoid Debugging on Hardware?

Test in IDE Test in Test on Test in
(Automated) Emulator Hardware Acceptance
(Automated) (Automated) (manual)

When:
Day 1 As soon as Available Weekly

Which tests:
Unit Tests
Component Tests
System Tests


No DOH How
•  How do we avoid Debugging on Hardware?

Test in IDE Test in Test on Test in
(Automated) Emulator Hardware Acceptance
(Automated) (Automated) (manual)

Frequency:
Every few minutes Daily Weekly+

Each code change Per component test Per user story


Agenda
Apps?
•  No DOH!


How Do We Ensure Testability?
“Design for Testability”, then code
•  Write tests last
•  And hope it’s easy

Or
•  Code for Testability
•  Via Test-Driven Development (TDD)


TDD Rhythm
Test Code Product Code
•  First Test •  Hard-coded method
•  Second Test •  Introduce variable
•  Third Test •  Introduce conditional
•  Fourth Test •  Surround with a loop

Just Like Double-Entry Booking:
•  An entry on the test side for each entry on
the Prod side


Test - After Architecture
•  Must test through User Interface

System Under Test

Configuration Configure
User Notification Notification
Workflow
Interface Threshold Rules
Test
Should we
Transaction Process Notify?
Interface Transaction Do
Notification.

Notification
Log


Test-Driven Architecture
•  Need to provide API’s to invoke functionality
directly
System Under Test

Workflow
Test
Should we Test Stub
Notification.

Notification
•  And ways to stub out Log
dependencies

•  “What kind of tests do I want to be able to automate?”
•  “Which component would be responsible for that part
of the behavior?”
Configuration
TX Test
Should we
Notification.

Notification
Log


•  “What other components would that component
depend on?”
•  “How can I break that dependency when component
testing this component?”
Notification
Notification
User Notification
Rules Fake
Rules
Interface Threshold
Notification Should we
Rule Test
Notification
Method Test Notification.

Notification
Log

•  With the right architecture, automating these
tests is trivial

Behavior Specification at Right Level
•  Specify broad scope at minimum detail
– E.g. Use least detail when specifying workflow
•  Specify most detailed req’ts at narrowest scope
– E.g. Don’t use workflow when specifying business rules

Too much detail
Workflow
Low

Unmaintainable
System
Tests
Transactions Make examples /
Detail

tests easy to
Component
Business understand and
Tests
Rules easy to write
High

Too vague
Unit Tests Unit
Tests
Broad Narrow
InfoQ Hangzhou 2012 Scope 51 Copyright 2012 Gerard Meszaros

Agenda
Apps?
•  No DOH!


Tools, Frameworks & Patterns
•  Tools come & go
– Different ecosystems (Android, IOS, BBOS, WP, etc.)
– New, better tools appear regularly
•  Strategies persist
– Need to define strategy first
•  Then choose tools to support the strategy
•  And Patterns for testable designs

Avoids “Impedance Mismatch” between
tools and strategy

Framework Issues
•  Many frameworks not designed for testability
•  May need to introduce level of indirection to
API (via Proxy or Adapter)
•  May need to extract Plain Ordinary Testable
Object (POTO) leaving a Humble Object that
subclasses framework base class


Extract POTO - Before
ActionBase ActionContext
getRules()
Framework

Plug-in
+rules
MyActionTest tests MyAction
Rules
testDoAction() doAction()


Extract POTO - After
Runs in device; ActionBase ActionContext
Tests “wrapping” getRules()
(Optional)
Framework

Plug-in
+rules
MyActionHOTest tests MyActionHO
Rules
testDoAction() doAction()

Runs in IDE; Tests Humble
“business logic” Rules Object
Testable
MyActionPotoTest
tests
MyActionPoto Object
testDoAction()
Fake doAction(rules)
Rules

Simple design change; Huge testability improvement!

Conclusions
•  Mobile and Cloud-Based Applications come
with special challenges
– While business logic tends to be simpler, constraints
and variations make design and testing more
challenging
•  Testing-in Quality is not an option
– There are too many potentially interacting dimensions
– Many are not controllable from outside the black box
•  Need to avoid Debugging on Hardware
– Test as much as possible “off device”
– Only deploy code you know works correctly


Conclusions
•  Design-for-Testability is essential for success
– Otherwise, tests will be too expensive to execute and/or
automate
•  Beware tools/frameworks that make DfT
difficult
– May need to decouple from framework to ensure
testability
– But extra effort is compensated for by better quality and
productivity


What Can You Do?
•  Product Owner:
– Break work into manageble chunks
– Provide examples/tests for each chunk to drive
development
– Insist that team automates the tests
•  Tester/BA:
– Provide tests to developers Before they start dev
– Insist on seeing the results of the automate tests before
you start testing
– Focus manual testing on areas not covered by
automated tests


Developer: What Can You Do?
•  Ask for tests from PO/Testers before agreeing to build
•  Discuss the kinds of tests that they will provide
•  Include the work to automate the tests in any
estimates you provide
•  Implement the automation before you develop the
code, not after!
•  Use the automated tests to drive development
•  Don’t recreate provided tests at unit test level; find
ways of reusing business rule tests as unit tests
•  Think/Do Design-for-Testability
•  If you find yourself in an interactive debugger, ask
yourself: “What unit tests would tell me the same
information and prevent this from happening again?”


References
•  James Grenning “Don’t Let Embedded Tool Chain
Slow You Down”
–  http://www.renaissancesoftware.net/blog/archives/41
•  Michael Bolton “Testing vs. Checking”
–  http://www.developsense.com/blog/2009/08/testing-vs-checking/
•  Dion Johnson “Cyber-dueling Over ‘Check’ vs. ‘Test’
and Other Semantics”
–  http://www.automatedtestinginstitute.com/home/index.php?
option=com_content&task=view&id=1373
•  Gerard Meszaros “Humble Object”
–  http://xunitpatterns.com/Humble%20Object.html
•  Homer Simpson “Doh!”
–  http://en.wikipedia.org/wiki/D%27oh!


Resources
•  http://www.mobileapptesting.com/
•  For Success, Build Record/Playback into Your
Application - StarEast 2008
– http://builtinrecordandplayback.xunitpatterns.com

– http://mobile.testAutomationPatterns.com


Thank You!
Gerard Meszaros
infoq2012hz@gerardm.com
http://www.xunitpatterns.com

Slides:
http://infoQ2012hz.testautomationpatterns.com Jolt Productivity Award
winner - Technical Books
Call me when you:
Coming Soon:
•  Want to transition to Agile or Lean
•  Want to do Agile or Lean better
•  Want to teach developers how to test
•  Need help with test automation strategy
•  Want to improve your test automation


Ensuring quality in cloud and mobile applications

Ensuring quality in cloud and mobile applications

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Ensuring quality in cloud and mobile applications

Similar to Ensuring quality in cloud and mobile applications (20)

Ensuring quality in cloud and mobile applications