Our special guest embedded and IOT testing expert Jon Hagar shares his thoughts on mobile testing, embedded testing, and IOT testing and how they intersect to form the perfect storm in the coming 4th Industrial Revolution where all businesses become software businesses and security/privacy reach new heights of concern.

1. Top IoT Tes(ng Challenges Webinar
Jon D. Hagar, Consultant, Grand
So4ware Tes8ng
embedded@ecentral.com

2. XBOSoft
Dedicated to Software Quality Improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco & Beijing
“Thorough, accurate and fast”
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
2

3. House Rules
§ Participants other than the speakers are muted
§ Ask questions in the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
§ Questions may be asked throughout the webinar -
we’ll try to answer them at the end
§ You’ll receive info on recording after the webinar
Webinar Hashtag: #IOTTesting
Copyright 2017 Jon D. Hagar – "So4ware
Test AHacks to Break Mobile and
Embedded Devices" 3

4. Meet Our Speakers
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
4
Philip Lew
CEO and Founder,
XBOSoft
• Software quality
processes
• UX design and evaluation
• Mobile UX
• Agile testing methods
• Risk management

5. Lots of opportuni8es (challenges) for Testers
• All the problems of hardware, embedded, IT and mobile so4ware
• Worse than that. . . Headlines:
The Devices Kill
4-10 trillion USD in next 10+ years
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"
5
IoT is the Part of the
Industrial Revolu4on 4.0
Internet of Things To Make Up Almost Half
of IT Budgets By 2020

6. • Specialized IoT hardware
• Crea(ng and tes(ng mul(ple user interfaces — the UX challenge
• Third-party soAware func(onality
• Security and privacy of IoT
• Not cover everything : hardware, ops, systems,…..
Today’s Challenges
6
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
6
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Cyber-Physical
Systems (today)
Table: IoT Challenges 2
Software
factors Opportunities Potential Solutions include
Mapping to
Attacks**
Security and privacy Hacking tests, crowd source testing chapter 9
Data usage and processing
AI, deep learning, stats use in testing IoT book 1, 15
Ubiquitous usability UI V&V chapter 7
Third party vendor impacts Functional testing, regression testing,
set based design, V&V 27
Software specialized to hardware
Test at the edge, Field testing, V&V 5,6, 7,8
Short life cycles with interfaces to
hardware life cycle Test planning and strategy IoT book 1
Developer who do not test (or do
not test well) Independent code coverage testing chapter 2
Hard to find code errors Static code analysis, V&V 1
Portable operating system
confusion Compatibility, combinatorial testing 32, IoT book 1
Allocation of processing to edge,
app, network, and/or cloud System V&V IoT book 2

7. Technology Space
Physical
Systems
(circa 100,000 BC)
7 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
Cyber
Systems
(1950s)
Embedded
IoT
Mobile-Smart
Personal
Computers
Big Iron
(Cloud)
Many Options
Huge
Numbers of
Devices
(billions)
Numbers of
Devices
(millions)
Cyber-Physical
Systems (today)

8. Today’s Test Pa<erns Will Con4nue
• Requirements verifica8on checking
- So4ware Coverage
• Risk–based analysis and tes8ng
• Exploratory Tes8ng (and Ad-hoc)
But you need more
Copyright 2017 Jon D. Hagar – "So4ware
Test AHacks to Break Mobile and Embedded
Devices" Copyright 2017 Jon D. Hagar - “So4ware Test AHacks to Break Mobile and Embedded Devices” 8

9. Challenge:
IoT Has Unique Specialized Hardware
• Different on Web or IT where hardware is
“generic”
• IoT
= Sensors
= Controllers
= Communica8on (Comm) lines and vendors
9 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

10. IoT Hardware: We Will Need
Modelling and Device Management
• Address systems,
so4ware, and hardware
test
• Developer and
independent test models
• Improves understanding
in teams
• Interfaces to CAD/CAM
systems?
10 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

11. • Model-based tes8ng offers a solu8on (and use is growing)
• IoT “high integrity” areas will need it
• Model-based tes8ng can support:
• Genera8on of test cases
• Early analysis valida8on
• Simula8on
• Verifica8on
• Oracles or judges
Model-based Tes,ng for High Risk Devices
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices" 11

12. • Configura8on Management and Control
• Recalls and Updates
• So4ware Updates: Push vs Pull
• Ownership
• Product Evolu8on
• Data Management and Analy8cs
Device Management
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
12

13. • Mobile Devices
Challenge: Hardware and Ux Challenge
Crea4ng Tests to Deal with Complexity
13
• User Interfaces
• IoT
Devices
• Stakeholder Data
• Comm
Channels
• IoT Home
Protocol
How many Tests?
to address, data, configurations, devices
Comm, UI, integration, etc…..
10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"

14. Complexity of Combina4ons -
Using the ACTS Combinatorial Tool
14
Parameters:
Mobile Devices [Device 1, Device 2, Device 3, Device 4, Device 6,
Device 7, Device 8, Device 9, Device 10]
IoTHomeProtocol [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door,
Garage door, Home gaurd, Stereo, Temp Control,
Lights, Drapes, Water Heater, window openers]
Ux Interfaces [0, 1, 2, 3, 4, 5]
Comm providers
[Cell1, Broadband, cable, Cell 2, Space based,
Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlaqorm IoTsHome IoTDevices Routers
Comm
providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space
based -99999
4 Device 6 false Refrig 5
Vendor
godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Test
Sample
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

15. Challenge: Third Party Hardware
and SoAware
Data, Op,ons, Func,ons

16. IoT Will Be Made Up of
Piece Parts From Many Vendors
• Off-the-shelf
– Hardware
– So4ware
• Many providers of “services”
• Piece parts may change during dev and ops
• Temp8ng to trust BUT
Verifica8on evalua8on is a MUST (using the
data)
16 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

17. • How many vendors of components and ops
will be in a CAR?
• Will Government policy save us?
– Defines SAE 5 levels of autonomy
– Dev and test “requirements”
– Will be at least as strong as aircra4 requirements
– Who “owns” the third party tes8ng?
• Cars = Life and Death
Third Party items must be TESTED
Example: USA DOT Self-Driving Car Policy
(and standards?)
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"
17

18. • ISO Dev system and so4ware standards
• ISO 15288
• ISO 12207
• ISO 26262
• ……………………………………………
• IEEE 1012 V&V plan
• Product standards
– Comm, low power, interfaces, networks, processors,
plaqorms, “eco systems”,………..
System, SoUware and Product Standards
18Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"
Love and Hate
Too Many to Name

19. FYI: ISO 29119 SoGware Test Standard
19Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"

20. Challenge: Third Party Vendors
Will Bury Us in
Data =>
“Standards” => Dev/Tes,ng
Op,ons

21. IoT to Generate Huge Amounts of Data
(Petabyte, Exabyte, ZeHabyte, Or a YoHabyte)
Current analy8cs focus is on marke8ng/sales
If user is a tester genera8ng data…….
Dev – Test will need to use data analy8cs
But for what?
21 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

22. As of Last Year:
Testers and Data Did NOT mix
SODA – Self Organizing Data Analy8cs
AI Neural Nets
Deep Learning Stat’s
• Tools
• Tester I talked to DID NOT CARE about using data!!!!
22
Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

23. Steps in Tes4ng Third Party
• Risk evalua8on of vendors/op8ons
• Selec8on of vendor op8ons
– Assess (test) the decision up front
– Set based design evalua8ons
• Select more than one???
– Data analy8cs
• Expect regression tests on changes
• Verify requirements and explora8on as an
ongoing effort
23 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

24. What keeps me up at night
(and living in a bunker)
Challenge: IoT Security and
Privacy
(Many experts think these are top priority)

25. Safety-Security/Privacy Bugs
• Leyng users be the Testers
• Common programming errors (developer
level)
• Forgeyng about or losing the hardware
• System environment
• Holes in stress and unusual cases
• IoT denial of service hack 2016
25 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

26. A<ack-based Exploratory Tes4ng
What is an a<ack?
• A paHern (for tes8ng) based on a common mode of failure
or informa8on need seen over and over
– Maybe seen as a nega8ve, when it is really a posi,ve
– Goes a4er the “bugs” that may be in the so4ware
– May include or use classic test techniques and test concepts
• See Lee Copeland’s book on test design and many other good test
books
• A PaHern (more than a process) which must be modified
for the context at hand to do the tes8ng
• Testers learn mental aOack paHerns when
working over the years in a specific domain
Copyright 2017 Jon D. Hagar – "So4ware
Test AHacks to Break Mobile and Embedded
Devices" Copyright 2017 Jon D. Hagar excerpted from “So4ware Test AHacks to Break Mobile and Embedded Devices” 26

27. • AOack 28 Penetra8on AOack Test
• AOack 28.1 Penetra8on Sub–AOacks: Authen8ca8on — Password
• AOack 28.2 Sub–AOack Fuzz Test
• AOack 29: Informa8on The4—Stealing Device Data
• AOack 29.1 Sub AOack –Iden8ty Social Engineering
• AOack 30: Spoofing AOacks
• AOack 30.1 Loca8on and/or User Profile Spoof Sub–AOack
• AOack 30.2 GPS Spoof Sub–AOack
Security A<acks (from “So4ware Test AHacks to Break Mobile and Embedded Devices”)
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices" 27

28. Internal Security Tes4ng Hacks Are Only a Start
• External Crowd Source Security Hacking -
GOOD
• More internals aOacks – BETTER
• Bad guys find “zero” day using aOacks
• Data analy8cs and scien8fic explora8on -
BEST
28 Copyright 2017 Jon D. Hagar – "So4ware Test
AHacks to Break Mobile and Embedded Devices"

29. More Examples So4ware AOacks for Exploratory Tes8ng
Excerpted from “So4ware Test AHacks to Break Mobile and Embedded Devices”
29
SoAware Test AOack Type AOack Finds Notes on the AOack
Developer level aOacks
Code and data structure
problems
Almost a quarter of errors in mobile and embedded can be found by
structural tes(ng
Control system aOacks
Hardware and soAware control
system errors
Many cri(cal errors in mobile and embedded are centered in the
control logic, for example analog-to-digital and digital-to-analog
computa(on problems
Hardware-soAware aOacks
Hardware and soAware
interface issues
The soAware should be tested to work with any unique hardware
Communica(on aOacks
Digital communica(ons
problems
SoAware communicates with hardware, network, and other
soAware with complex interfaces that should be tested
Time aOacks
Time, performance, sequence,
and scenario errors
System soAware can have cri(cal (ming and performance factors
that tes(ng can provide valuable informa(on about
User interface aOacks
Problems between man and
machine
The usability of devices and soAware are cri(cal to success
Smart/Mobile/Hardware aOacks
Issues specific to smart device
configura(ons including cloud
issues
Cloud-hybrid compu(ng comprises a majority of the new soAware
systems being deployed
Security test hacking aOacks
SoAware errors that can
expose devices to security
threats
Security of devices or systems is increasing in importance and
aOacks include, for example, GPS and iden(ty spoofing
Generic func(onal verifica(on
aOacks
Requirements and
interoperability errors
Basic checks that testers should conduct on systems and soAware
Sta(c code analysis aOacks
Hard to find errors that classic
tes(ng oAen misses
Can oAen be done by the development group but some(mes the
test group must run this analysis
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"

30. Mobile-IoT Challenge Summary
• To defeat an enemy, you must know the bug
• The IoT test data is limited,
– What exists has implica8ons to testers
– Do NOT hide from the DATA
• Improve our tes8ng with PaOerns
– IoT Dev-Test eBook = search: LeanPub.com – Hagar (free)
– More Challenges at Mobile/IoT Conference San Diego April 23-28
– “SoAware Test AOacks to Break Mobile and Embedded Devices”
• So4ware will be in very nearly everything
– Good tes8ng may be a limi8ng factor
30
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices"

31. References (my’s favorites)
• “SoGware Test AOacks to Break Mobile and Embedded Devices”
– Jon Hagar
– IoT Tests Book in 2017 - LeanPub.com - Hagar
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books
• “A Practitioner’s Guide to Software Test Design” Copeland, 2004
• “Computer Related Risks”, Neumann, 1995
• “Safeware: System Safety and Computers” Leveson, 1995
• Honorable mentions:
– James Bach
– Cem Kaner
– Many others
Copyright 2017 Jon D. Hagar – "So4ware
Test AHacks to Break Mobile and Embedded
Devices"
Copyright 2017 Jon D. Hagar excerpted from “So4ware Test AHacks to Break Mobile and Embedded Devices” 31

32. More Resources
• www.stickyminds.com – Collection of test info
• www.embedded.com – info on attacks
• www.sqaforums.com - Mobile Devices, Mobile Apps - Embedded
Systems Testing forum
• Association of Software Testing
– BBST Classes http://www.testingeducation.org/BBST/
• Your favorite search engine
• My web sites and blogs
Copyright 2017 Jon D. Hagar – "So4ware Test AHacks to Break Mobile and Embedded Devices" 32

33. Post your webinar questions on Twitter @XBOSoft
Registrants will receive an email with information on where to view the recording
and slides from today’s webinar.
Join us to keep updated on all our webinars, reports and white papers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
Check out our blog: http://xbosoft.com/software-quality-blog/
Download our free white papers: http://xbosoft.com/knowledge-center/
Email us with ideas for future webinars or questions regarding our services!
services@xbosoft.com
Thank you!
Q+A
www.xbosoft.com