Building a useful set of devices for testing apps requires significant knowledge of the Android ecosystem. Once assembled, the device matrix provides broad, efficient coverage with minimal investment.
Nell’iperspazio con Rocket: il Framework Web di Rust!
Building the Ultimate Device Matrix
1. Picking
The
Right
Set
of
Mobile
Devices
By
Brian
Kitchener
So;ware
Quality
Architect
bkitchener@prototest.com
2. Overview
• About
me
• Some
Background
• The
Problem
• Understanding
Android
• How
Apps
Work
• Building
a
Device
Matrix
• Example
Matrices
• Conclusion
3. About
Me
• So;ware
Quality
Architect
at
ProtoTest
• We're
a
mobile
test
lab
that
combines
usability
tesMng
with
quality
assurance
to
culMvate
a
great
user
experience
• Project
Architect,
Technical
Lead,
Trainer.
• Started
in
QA
in
2001
• BA
in
Applied
CompuMng
from
University
of
Denver
• TesMng
background
:
FuncMonal,
Performance,
UAT,
Security,
API,
Database.
• AutomaMon
:
Selenium,
WebDriver,
WaMN,
MonkeyTalk,
SOASTA,
Fitnesse,
QTP,
EggPlant,
Squish
• Languages
:
C#,
Java,
Ruby,
Javascript
5. Some
Stats
for
2012
• Mobile
Apps
achieved
$17
billion
in
sales
• 5.2
Mobile
Subscribers
– 1.2
Billion
PC’s
– 4.2
Billion
people
use
a
toothbrush
– 1
Billion
Smartphones
• 722
Million
Smartphones
sold
• 1.4
Million
iOS
+
Android
Apps
• 25
developers
=
half
of
app
revenue
7. About
the
iPhone
• Steve
Ballmer
:
Microso;
CEO
– “There’s
no
chance
the
iPhone
is
going
to
gain
significant
market
share.
No
chance.”
• Patrick
Stewart:
– “Last
Wednesday,
I
stupidly
dropped
my
iPhone
in
the
bath,
and
my
life
has
sort
of
spiraled
almost
out
of
control.”
• Jon
Rubinstein
–
Palm
CEO
– Is
there
a
toaster
that
also
knows
how
to
brew
coffee?
There
is
no
such
combined
device,
because
it
would
not
make
anything
be;er
than
an
individual
toaster
or
coffee
machine.
It
works
the
same
way
with
the
iPod,
the
digital
camera
or
mobile
phone:
it
is
important
to
have
specialized
devices.
• Mike
Lazaridis
–
Blackberry
CEO
– And
so
what
[the
iPhone]
has
actually
done
is
increased
our
sales.
9. And
Then
There
Were
Two…
• Android
unveiled
November
2007
• First
device
was
sold
in
October
2008.
• Over
11,000
models
have
been
released.
• 48
Billion
app
installs
• Over
1
Billion
Android
devices
acMvated
• 8
OS
Revisions
12. Let’s
do
some
math!
• 16
device
display
categories
• 20
different
common
resoluMons
• 8
OS
versions
• 6
Hardware
Manufacturers
• 4
Major
cellular
networks
• 16
x
20
x
8
x
6
x
4
=
76,800
permutaMons
• Pairwise
approach
=
over
30
permutaMons
• Who
can
afford
to
increase
tesMng
by
30X?
13. Our
Approach
• Efficiency,
not
coverage
• Flexible:
support
small
or
large
number
of
devices
• Understand
how
apps
work
to
logically
select
criteria
• Use
Market
research
to
pick
most
common
configuraMons
• Pick
minimum
and
maximum
boundary
values
for
each
criteria
• Choose
a
value
that
matches
an
edge
case
or
abnormal
configuraMon.
• Pick
values
that
stress
or
tax
the
system
15. Android
• Built
and
Maintained
by
Google
• Open
Source
• Built
on
Linux
kernel
• ARM
• X86
Ports
• Built
to
support
almost
any
type
of
device
– Phones,
tablets,
phablets,
media
players,
tv’s,
watches,
etc.
• Device
Manufacturers
customize
code.
16. Example:
Kindle
Fire
• Forked
Android
2.3
– Not
updateable
• Customized
UI
• Separate
App
store
• Not
all
android
apps
work
• Custom
web
browser
17. The
OperaMng
System
• Google
Releases
“stock”
versions
• 10
Major
Releases
since
2008
– API
Level,
not
Version
• Device
manufacturers
like
to
customize
the
OS
– Drivers,
libraries,
UI
• “Stock”
OS
available
in
Nexus
devices
or
an
Emulator
18. Simulators
/
Emulators
• Simulator
imitates
the
so;ware
layer
– OS
and
Libraries
– Apple
provides
a
simulator
in
xCode
IDE
• Emulator
duplicates
the
hardware
and
so;ware
– Processor
and
Memory
– Cannot
mimic
GPU,
GPS,
accelerometer
• Always
run
stock
OS
• Can
be
used
to
test
some
funcMonality
• Should
always
test
on
a
physical
device
too
19. The
Processor
• ARM
RISC-‐based
instrucMon
set
• SpecificaMon
defined
by
ARM
holdings
• 32
bit
• Same
as
iOS
• X86
patches
and
ports
• It’s
only
a
spec,
can
be
modified.
20. SoC
–
System
On
A
Chip
• Main
Board
• Processor
• RAM
Bus
• GPU
• May
include
:
– Cellular
– WiFi
– NFC
– GPS
– Bluetooth
22. Common
SoC
Manufacturer
Device
Name
Cores
Processor
GPU
Qualcomm
Snapdragon
S4
2
or
4
ARM
Cortex-‐
A15
Adreno
Nvidia
Tegra
3
4+1
ARM
Cortex-‐
A9
Geforce
Samsung
Exynos
4
2
or
4
ARM
Cortex-‐
A9
Mali
Intel
Medfeld
1
Intel
x86
PowerVR
Texas
Instruments
OMAP
4
2+2
ARM
Cortex
A9
PowerVR
ST-‐Ericcson
NovaThor
2
ARM
Cortex-‐
A9
PowerVR
23. ResoluMon
is
not
enough
• Unlimited
number
of
screen
sizes
available
• Screens
range
from
3”
to
11”
• Each
screen
has
a
resoluMon,
same
as
a
monitor
– If
you
increase
the
resoluMon
everything
shrinks!
• Pixels
per
Inch
=
Density
• Screen
Size
+
Density
=
Display
Bucket
• ResoluMon
is
not
enough!
24. The
Display
Buckets
Size
:
• Xlarge
:
8”
-‐
10”
tablet.
• Large
:
5”
-‐
7”
tablet.
• Normal
:
3.5”
-‐
5”
phones.
• Small
:
3”
-‐
3.5”
phones.
Density
:
• ldpi
=
Low
DPI
(~120)
• mdpi
=
Medium
DPI
(~160)
• hdpi
=
High
DPI
(~240)
• xhdpi
=
Extra
High
DPI
(~320)
Low
density
(120),
ldpi
Medium
density
(160),
mdpi
High
density
(240),
hdpi
Extra
high
density
(320),
xhdpi
Small
screen
QVGA
(240x320)
480x640
Normal
screen
WQVGA400
(240x400)
WQVGA432
(240x432)
HVGA
(320x480)
WVGA800
(480x800)
WVGA854
(480x854)
600x1024
640x960
Large
screen
WVGA800*
*
(480x800)
WVGA854*
*
(480x854)
WVGA800*
(480x800)
WVGA854*
(480x854)
600x1024
Extra
Large
screen
1024x600
WXGA
(1280x800)
†
1024x768
1280x768
1536x1152
1920x1152
1920x1200
2048x1536
2560x1536
2560x1600
25. Display
Buckets
• Galaxy
S3
– 1280
x
720
– Xhdpi
density
(331ppi)
– Normal
screen
(4.7”)
• Galaxy
Tab
10.1
– 1280
x
800
– ldpi
density
(149ppi)
– Xlarge
sceren
(10.1”)
• Galaxy
Note
LTE
– 1280
x
800
– hdpi
density
(285ppi)
– Large
Screen
(5.5”)
27. Aspect
RaMo
• UI
is
manipulated
from
code
• Density
Pixels
adjust
for
screen
size
– But
can
use
regular
pixels!
• Need
to
take
both
X
and
Y
into
account!
– Easy
to
overlap
or
hide
things
• Includes
orientaMon
• Some
devices
include
an
aspect
raMo
changer!
(LG
OpMmus
Vu)
28. Cellular
Carrier
• Four
Major
US
Networks
– Verizon,
Sprint,
AT&T,
T-‐Mobile
– Some
phone
interoperability
– 2
protocols
• GSM
–
T-‐Mobile
AT&T
• CDMA
–
Verizon
and
Sprint
– Carriers
assigned
specific
frequency
bands
– LTE
will
be
new
standard
-‐
But
spectrum
issues
will
prevent
cross-‐network
phones
• So
if
the
phone
supports
the
carrier’s
protocol
and
band
it
can
theoreMcally
connect.
30. How
Apps
work
• Apps
need
to
work
on
all
screen
sizes
– May
not
be
funcMonal
– May
be
wasted
space
– May
not
make
sense
• Apps
define
XML
layouts
similar
to
HTML
– Node
structure
– StaMc
Content
–
Images,
etc
– Dynamic
Content
–
Color,
Text,
etc.
31. Layouts
and
Fragments
• XML
Fragments
are
reusable
components
• Layouts
sMtch
together
fragments
for
a
specific
sized
device
• App
may
need
different
flow
for
tablet
vs
phone
33. Our
Criteria
• OperaMng
System
– OS
customizaMons,
missing
libraries,
driver
issues,
• Screen
Size
– Rendering
issues,
usability,
missing
layouts
• Pixel
Density
– Density
Independence,
missing
layouts.
• Aspect
RaMo
– X,Y
calculaMons,
overlapping
panels,
display
issues
• SoC
– Hardware
performance,
InstrucMon
set,
bavery,
signal
• Carrier
– Network
protocol,
speed,
responsiveness,
packet
loss
34. The
Goal
• Efficiency,
not
coverage!
• Build
a
set
of
devices
to
be
used
for
app
and
website
tesMng.
• Know
when
to
update
them
• Define
a
list
of
simple
categories
of
devices
• Pick
devices
that
offer
broad
coverage
• Adjust
the
number
of
devices
based
upon
needed
coverage
35. Categorical
Approach
• Define
scope
–
Android,
iOS,
phone,
tablet,
etc.
• Understand
TesMng
requirements
• Self-‐descripMve
Names
• Help
to
broaden
coverage
• Will
adjust
devices
chosen
to
cover
our
criteria
• Should
be
apparent
when
to
update
a
device
• Spread
coverage
:
– Usage
-‐>
Edge
Cases
-‐>
Strange
-‐>
Stress
36. Example
Categories
• Common
– Matches
most
common
display
configuraMon
• Newest
– Latest
OS
version,
largest
screen,
highest
resoluMon
• Oldest
– Oldest,
slowest,
smallest
device.
• Abnormal
– Non-‐standard
OS,
aspect
raMo,
orientaMon,
size
• Popular
– Most
popular
device
in
terms
of
sales
• Budget
– Low-‐priced
new
model.
Tend
to
have
strange
specs
• Flagship
– Nexus
device
running
stock
Android
OS
• Catch-‐All
– Cover
any
missing
criteria
37. Android
Phone
Matrix
March
2012
Device
Name
OS
Display
Aspe
ct
SoC
Carrier
Newest
HTC
Droid
DNA
4.2
Normal-‐xhdpi
9:16
Snapdragon
S4
Verizon
Oldest
HTC
Tavoo
1.6
Small-‐ldpi
3:4
Snapdragon
S1
AT&T
Common
Motorola
Droid
3
2.3
Normal-‐hdpi
9:16
TI
OMAP
4
Verizon
Popular
Samsung
Galaxy
S3
4.1
Normal-‐xhdpi
9:16
Exynos
4
Sprint
Abnormal
LG
OpMmus
VU
4
Large-‐hdpi
3:4
Nvidia
Tegra
3
Tmobile
Flagship
LG
Nexus
4
4.2
Normal-‐xhdpi
3:5
Snapdragon
S4
TMobile
Budget
Dell
Venue
2.2
Normal-‐mdpi
3:5
Snapdragon
S3
AT&T
Catch-‐All
Sony
Xperia
P
2.3
Normal-‐hdpi
9:16
Sony
NovaThor
AT&T
38. iOS
Matrix
March
2012
Device
Name
OS
Display
Aspect
SoC
Carrier
Newest
iPhone
5S
7
4”
1136
x
640
326ppi
9:16
Apple
64bit
A7
T-‐Mobile
Oldest
iPhone
3g
6
3.5”
320
x
480
165ppi
2:3
Apple
A3
AT&T
Common
iPhone
5
6
4”
1136
x
640
326ppi
9:16
Apple
A5
Verizon
Popular
iPhone
4
6
3.5”
640x960
330ppi
2:3
Apple
A4
Sprint
iPad
(ReZna)
iPad
3
7
10”
1536x2048
264ppi
3:4
Apple
A5X
Verizon
iPod
iPod
Touch
(4th
gen)
5
3.5”
640x960
326ppi
2:3
Apple
A4
WiFi
Mini
iPad
Mini
6
7”
1024
x
768
162ppi
3:4
Apple
A5
AT&T
39. Conclusion
• Understanding
how
everything
works
allows
us
to
logically
select
devices.
• A
large
number
of
permutaMons
can
be
covered
in
few
devices
• If
addiMonal
coverage
is
needed
addiMonal
devices
can
be
added
• White
Paper
:
– hvp://www.prototest.com
:
– Building
the
UlMmate
Device
Matrix