Automated GUI-Testing of Android Apps: From Research to Practice

ICSME16
Raleigh, NC
Wednesday, October 5th, 2016
Kevin Moran,
Mario Linares-Vásquez,
& Denys Poshyvanyk
College of William & Mary - SEMERU - Department of Computer Science
Automated GUITesting of Android
Apps: From Research to Practice
&

UI Events
GUITesting: The Main Idea

UI Events
Oracle

Output, layout, exceptions,
presentation logic, quality attributes, …
UI Events
Oracle

GUITesting: Core Concepts
Test
Result
Oracle

Monkey Testing !!
Automated Testing !!

Automated GUI-Testing
Output, layout, exceptions,
presentation logic, quality attributes, …
UI Events
Monkey

GUI-Based Testing for Mobile Apps

The App Economy
http://de.my-walls.net/android-os-robot/

Mobile App Dev Challenges
• Rapid platform (and third-party lib) evolution
• API-instability and fault-proneness
• Inﬂux of unstructured (“low quality”) user reviews
• Platform-fragmentation at device and OS level
• Quality attributes (performance, energy) and
inputs (sensors, camera, WiFi etc.)

Android (andThird Party library) Instability

Study: mining of code changes
history (5,848 apps); survey with 45
developers
Method: descriptive statistics,
statistical hypothesis testing, and
open coding
Main ﬁndings:
- Successful apps use more stable and bug
free APIs
- APIs issues are perceived by users as app
programming issues
- Lack of tools for identifying API issues and
their impact on a new API release
Android (andThird Party library) Instability

LargeVolumed of Crowdsourced
Requirements and Ratings

Mobile AppTesting Challenges
• Highly event-driven, gesture based nature of apps
• Lack of effective automated input generation
• Lack of robust (semi) automated testing and
monitoring tools
• Lack of support for debugging apps and bug
reporting
Joorabchi, M.E.; Mesbah, A.; Kruchten, P., "Real Challenges in Mobile App Development," in Empirical Software
Engineering and Measurement, 2013 ACM / IEEE International Symposium on , vol., no., pp.15-24, 10-11 Oct. 2013

Manual Testing is Still Preferred!!

Automated Mobile Testing Approaches

Record & Replay
AUT/SUT
UI Events

Record & Replay
AUT/SUT
UI Events
Recorder Script

Record & Replay
AUT/SUT
UI Events
Recorder Script
Scripts

Record & Replay
AUT/SUT
UI Events
Recorder Script
Scripts
UI Events
Monkey AUT/SUT
UI Events

Pros and Cons
R&R ✓ Easy reproduction - Expensive collection
and maintenance
- Coupled to locations

Model-BasedTesting
UI Events
Model Monkey AUT/SUT

Model-BasedTesting
UI Events
Model Monkey AUT/SUT
- Manually generated
- Automatically generated (source code)
- Ripped at runtime (upfront)
- Ripped at runtime (interactive)

GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey

GUI-Rippers
Snapshot 1

GUI-Rippers
Snapshot 1
Snapshot 1

GUI-Rippers
Snapshot 1
Snapshot 1
GUI State 1

GUI-Rippers
Snapshot 1
Snapshot 1
GUI State 1
A or B ?

GUI-Rippers
Snapshot 1
Snapshot 1
GUI State 1
Event 1
A or B ?

GUI-Rippers
Snapshot 1
Snapshot 1
GUI State 1
Event 1
Snapshot 2
Snapshot 2
GUI State 2
A or B ?

GUI-Rippers
Monkey
A or B ?
Breadth-FirstDepth-First

GUI-Rippers
Monkey
A or B ?
Breadth-FirstDepth-First
Random (Uniform) Random (A-priori distr.)
Other options (online decision)

GUI-Rippers
Ripper/Extractor
Computer/Mobile device
OS
- Framework
- API
- Utilities
GUI State
Events
Monkey

Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
MBT ✓ Event sequences
✓ Automatic exploration
- Invalid sequences (a-
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models

Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
- No history awareness

Random/FuzzTesting (R/FT)
Monkey
X or Y ?
AUT/SUT

Monkey
X or Y ?
AUT/SUT
Event x
Invalid

Monkey
X or Y ?
AUT/SUT
Event x
Invalid
Event Y
Valid

Pros and Cons
R&R ✓ Easy reproduction - Expensive collection
and maintenance
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
R/FT ✓ Fast execution
✓ Good at ﬁnding crashes
- Invalid events
- Lack of expressiveness

Automation Frameworks/APIs (AF/A)
TESTS
JUnit, Espresso, UI Automator, Robotium
Monkey

Automation Frameworks/APIs (AF/A)
UI Automator

Pros and Cons
R&R ✓ Easy reproduction
- Expensive collection
and maintenance
- Invalid sequences (a-priori
model)
- Few corner cases
- Use cases are not considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
R/FT ✓ Fast execution
✓ Good at ﬁnding crashes
- Invalid events
- Lack of expressiveness
AF/A ✓ Easy reproduction
✓ High level syntax
✓ Black box testing
- Learning curve
- User-deﬁned oracles
- Expensive maintenance

RESEARCH PROTOTYPES
Tool Name Instrumentation GUI Exploration Types of Events
Replayable Test
Cases
Emulators, Devices
Dynodroid Yes Guided/Random System, GUI, Text No No
EvoDroid No System/Evo GUI No N/A
AndroidRipper Yes Systematic GUI, Text No N/A
MobiGUItar Yes Model-Based GUI, Text Yes N/A
A3E Depth-First Yes Systematic GUI No Yes
A3E Targeted Yes Model-Based GUI No Yes
Swifthand Yes Model-Based GUI, Text No Yes
PUMA Yes Programmable System, GUI, Text No Yes
ACTEve Yes Systematic GUI No Yes
VANARSena Yes Random System, GUI, Text Yes N/A
Thor Yes Test Cases Test Case Events N/A No
QUANTUM Yes Model-Based System, GUI Yes N/A
AppDoctor Yes Multiple System, GUI2, Text Yes N/A
ORBIT No Model-Based GUI No N/A
SPAG-C No Record/Replay GUI N/A No
JPF-Android No Scripting GUI Yes N/A
MonkeyLab No Model-based GUI, Text Yes Yes
CrashDroid No Manual Rec/Replay GUI, Text Yes Yes
SIG-Droid No Symbolic GUI, Text Yes N/A
CrashScope No Systematic GUI, Text, System Yes Yes
Sapienz Yes Evolutionary GUI, Text,System Yes Yes

OTHER APPROACHES
FOR MOBILE GUI TESTING

CrowdsourcedTesting
AUT/SUTThird Party Service

Remote ManualTesting (Device Farm)
Third Party Service

Google Alpha/BetaTesting
https://support.google.com/googleplay/android-developer/answer/3131213

Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
Monkey (Fuzz testing)

Tools: Summary
Record and Replay
Monkey (Fuzz testing) ✓ Unexpected scenarios
- Reproducibility/Expressiveness

Tools: Summary
Record and Replay
✓ Easy reproduction
✓ Automation on different devices
- Test scripts maintenance
- Learning curve

Tools: Summary
Record and Replay
✓ Similar to manual testing
- Learning curve

Tools: Summary
Record and Replay
- Open Source vs Paid
✓ Large set of devices
✓ Remote data collection
✓ Support automation frameworks
✓ Similar to manual testing
- Learning curve

Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories

maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13][TSE’15]
Performance
bottlenecks [ICSME’15]
[MSR’16]
User reviews [ICSME’15]
Energy [MSR’14]

maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13][TSE’15]
Performance
bottlenecks [ICSME’15]
[MSR’16]
User reviews [ICSME’15]
Energy [MSR’14]
Crowdsource-based
requirements
[ICSME’15]
GUI - based testing
[MSR’15]
Bug reporting [FSE’15]
Optimizing energy
consumption [FSE’15]

SEMERUTools: Monkey Lab
• Testing approach based on natural usages of
an application.
• Models usage using n-gram language models to
produce unseen yet natural event sequences.
• Capable of generating orthogonal test cases
to those already modeled.

SEMERUTools: FUSION
www.fusion-android.com
• Novel Bug Reporting System for GUI-based
and mobile apps
• Uses a combination of static and dynamic
analysis to help inform the reporting process
• Uses auto-completion to ease the burden on
reporters and ensure high-quality reports.

SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug reports and repayable
scripts
• A practical tool, requiring no instrumentation framework, or
modiﬁcation to the OS or applications
• Capable of running on both physical devices and emulators
• Differing execution strategies able to test contextual features

Enabling Practical Automated Mobile GUI-Testing

Essential Challenges in Mobile GUITesting
Essential Challenges
Test Oracles: How does one design an extensible GUI- based
oracle that can be used to verify expected application behavior
across different device configurations?
Event Coordination: In automated GUI-based testing, deciding the
proper timing of firing events on a device is challenging.
External Features: How can external features such as notifications
be properly incorporated into large-scale testing of mobile apps?
External Dependencies: Many mobile apps use external servers or
databases in order to send, store, and receive information. How can
these dependencies be appropriately incorporated into large-scale
testing?

Accidental Challenges in Mobile GUITesting
Accidental Challenges
Application Data and Cold Starts: How can one ensure a sterile or
desired testing environment within the context of large scale
testing?
Bugs in Framework Utilities: There exist certain limitations with
core Android Frameworks popular with GUI-testing.
Bugs and Limitations in the SDK Tools: There are limitations
inherent to Android developer tools, such as adb, that make
large-scale parallel execution and testing difﬁcult.
Challenges Scaling Concurrent Virtual Devices: Android emulators
were not designed to scale well on modest hardware.

Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
Drawbacks: May miss some ﬁned grained information about the
GUI (e.g. colors, images)

Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
adb shell shell /system/bin/uiautomator
dump /sdcard/ui_dump.xml
Drawbacks: May miss some ﬁned grained information about the
GUI (e.g. colors, images)

Essential Challenges: Event Coordination

Essential Challenges: Event Coordination
adb shell dumpsys window -a
mAppTransitionState
APP_STATE_READY
APP_STATE_IDLE
APP_STATE_TIMEOUT
APP_STATE_RUNNING

Essential Challenges: External Features and Dependencies
Potential Solution: Use real usage data from the ﬁeld?

Accidental Challenges:Application Data and Cold Starts

Accidental Challenges:Application Data and Cold Starts
adb shell rm <app-data-folder>
adb shell pm clear <app-package-name>

Accidental Challenges: Bugs in Framework Utilities

Solution: Modify Framework Components for the AOSP

Accidental Challenges: Bugs and Limitations in the SDKTools
adb Server 1 adb Server 2

Accidental Challenges: Bugs and Limitations in the SDKTools
adb Server 1 adb Server 2
No more than 15 devices per adb server
instance!

Accidental Challenges: Scaling ConcurrentVirtual Devices

Record>Mine>Generate>Validate Framework

Enabling Large Scale Analysis: Execution Engine
✓Support for multiple
maintenance tasks
✓Open/Closed principle
✓Horizontal/Vertical scalability
✓Asynchronous communication
between clients and execution
engine
✓Support for fragmentation
Message bus
Execution Engine
Virtual devices (Pool)

Enabling Large Scale Analysis: Execution Engine
Dispatcher
Worker
Task
Device/VM/
Thread
token
Task, Device/VM/
Thread token
Task status,
response
Task status
- Task-specific
frameworks
- Third party libraries
<<invokes>>
Client i
Message Bus
Local DB
Client-specific
technologies
Request
(i.e., new Task)
Web folder or
Content Delivery
Network
Images,
files, etc.Images,
files, etc.
Response (i.e.,
Task results
and status)
Execution Engine (EE)
EE Settings
Devices/VMs
pool
Background
Threads pool
Clients
Client n

THANK YOU!
Kevin Moran
kpmoran@cs.wm.edu
Mario Linares Vasquez
m.linaresv@uniandes.edu.co
Denys Poshyvanyk
denys@cs.wm.edu
http://www.research-appendix.com/android-testing

Automated GUI-Testing of Android Apps: From Research to Practice

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Automated GUI-Testing of Android Apps: From Research to Practice

Ähnlich wie Automated GUI-Testing of Android Apps: From Research to Practice (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Automated GUI-Testing of Android Apps: From Research to Practice