This document summarizes a presentation on cross-platform mobile test automation using Xamarin and SpecFlow. It discusses using Xamarin to build native mobile apps with shared C# code, SpecFlow for writing acceptance criteria in Gherkin, and testing the user interface of mobile apps using Appium. It also describes testing at the controller layer for faster test execution. Automating tests is recommended to improve quality and provide faster feedback, but manual testing still has value for exploring apps and hard-to-automate checks.
1. COPYRIGHT, TECHTALK - WWW.TECHTALK.AT
ANDREAS WILLICH (@SABOTAGEANDI)
CHRISTIAN HASSA (@CHRISHASSA)
Mobile Quality Crew Meetup, Vienna, April 23rd 2015
Cross Mobile Testautomation with
Xamarin & SpecFlow
2. 2
Topics
• Xamarin
• Test automation in agile projects
• Specification-By-Example
• SpecFlow
• UI automation on mobile devices
• Mobile app automation with SpecFlow
11. 11
Xamarin.Android
twitterListView.setOnItemClickListener (new OnItemClickListener () {
public void onItemClick (AdapterView<?> parent, View view, int position, long id) {
Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class);
tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID);
startActivity (tweetDetails);
}
};
twitterListView.ItemClick += (s, e) => {
var tweetDetails = new Intent (this, typeof (TweetDetailsScreen));
tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID);
StartActivity (tweetDetails);
};
C#
12. 12
Development environment
OS Mac OSX, Windows Windows
UI Designer Android, iOS
Build Android, iOS (via Xamarin Build Host on Mac OSX)
Debug Android (Device and ARM/x86 emulator)
iOS (Device and emulator on Mac)
14. 14
Xamarin.Forms
One UI definition for 3 platforms (iOS, Android, WP)
in XAML (data binding)
translates to native UI controls on each platform
can be mixed with custom UI
16. 16
Test automation becomes expensive when …
• automating manual tests
• automation makes tests
unreadable
• automating only after
implementation
structure
readability
right moment
17. 17
Structure
Manual tests
Asserts Multiple combined
features
Structure ACT-ASSERT-
ACT-ASSERT-
ACT-ASSERT-
…
Dependent features
Long test path with
high chance to break
Cause and impact of
error hard to trace
Automated Check
Single aspect of a
single feature
ARRANGE –
ACT –
ASSERT
Independent features
Short test path with
lower chance to break
Cause and impact of
error easy to relate
19. 19
// Go to web page 'http://localhost:40001/' using new browser instance
BrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(44, 14));
int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' link
Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));
// Click 'Save' button
Mouse.Click(uISpeichernButton, new Point(34, 11));
int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(fundNr1 + 1 == fundNr2);
// Click 'Close' button
Mouse.Click(uICloseButton, new Point(26, 11));
Readability
20. 20
A readable test case
Scenario: New found items should receive a
consecutive number for the current year
Given the previous found item of the
current year had the number 145
When I register a new found item
Then the last found item of the
current year should have the number 146
21. 21
When to test (right moment)
Acceptance criteria checks
(ATDD, BDD)
Unit Tests
(TDD)
business view
technical view
Exploratory tests
User Journey tests
Performance, Scalability,
Usability,Security, …
productdefinition
productcritique
New dimension: defining the product
Synergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick
22. 22
Automated checks don’t replace manual testing
User
journeys
Acceptance-
criteria
Units
exploratory
testing
manual check
in definition of done
for stories
Main success paths
Undiscovered
acceptance criteria
No/(fewer) manual
regression checks
Fewer paths
are enough
More time
for exploration
25. 25
Agile specifications
define upfront define just-in-time
Code
Reminder
for a
conversation
Goals
Impacts
Epics
User Stories
Acceptance
criteria
Why?
Outcome
How?
Specification
26. 26
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We’d like to encourage new customers to buy in our shop.
Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
28. 28
Specification with examples
Examples …
make abstract descriptions better understandable
are usually not formally documented or exchanged
Examples Tests
Requirements
describe validate
fulfillment of
consist of
29. 29
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
We’d like to encourage new customers to buy in our shop.
Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
30. 30
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
New customers should receive 10% discount:
Given the customer hasn‘t ordered yet
When the customer adds a book for EUR 37.50 to the shopping cart
Then the shopping cart shows a total price of EUR 33.75.
31. 31
Actually, this is not quite right:
Books on sale should be excluded!
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
33. 33
Collecting Acceptance Criteria
“I would try to put a book into the
shopping cart …”
“I would try to remove a book
from the shopping cart…”
“I’d check whether the shopping cart
is empty, when I enter the shop …”
Books can be added to
shopping cart.
Books can be removed from
shopping cart.
Shopping cart should be empty
when entering the shop.
... ? …
As a potential customer
I want to collect books in a shopping cart
So that I can order several books at once.
“Imagine this story is
already implemented:
How would you verify it?”
“I would try to add 1000 books to
the shopping cart …”
34. 34
Exploration through examples
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
Context
Action
Assertion
35. 35
Key examples: Breaking the model
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
What happens, if I search for
“Explore Specification”?
Can I search for single
characters, e.g. “e”?
Is the number of search results
limited, or paged?
Is the search also performed in the
sub-title of a book?
36. 36
Purpose of the examples
• Shared understanding:
acceptance criteria
• Documentation:
specification details
• Regression-tests:
violated specifications
38. 38
SpecFlow – BDD for .NET
#62 in Visual Studio Gallery based on popularity (Apr 23 2015)
#303 most downloaded package on NuGet
in the last 6 weeks (Apr 23 2015)
~1‘400 unique daily visitors on specflow.org
> 30’000 active users
> 40 contributors
39. 39
Feature: Description of feature or user story
Szenariogrundriss: Beschreibung des Akzeptanzkriteriums
Szenario: Beschreibung des Akzeptanzkriteriums
Gherkin Feature Files
Background: context for all scenarios in the feature file
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Given/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>Examples: table with examples for <place holders>
Given: automated scenario steps
@tagname
@tagname
@tagname
@tagname
40. 40
Abstract acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping
basket
41. 41
Scenarios: Examples in Gherkin
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”
42. 42
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Scenarios: Examples in Gherkin
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
The same book can be added multiple times to the shopping basket
43. 43
The same book can be added multiple times to the shopping basket
Structure of Scenarios
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Title: Describes intention/abstract acceptance criterion
Arrange: Context, describes state of the system
Act: Execution of the feature
Assert: Assertion of observable behaviour
And I should see the warning: “Book already existed in basket”
Triple-A
constraint
“Checks”
Chaining
up steps
44. 44
Automated continuous validation
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
System
„Step Definitions“ are binding individual steps to an automatable interface of the application.
Automatable
interface
UI
Automation
Automation does not necessarily have to bind to the UI.
Automatability of system is supported/evolving with development.
48. 48
Appium (www.appium.io)
• Provides WebDriver API (JSON Wire Protocol)
• Uses native UI Automation library
• Can test native & hybrid Apps, mobile web apps
• Clients for C#, Java, Ruby, Python, PHP, JavaScript
• Works with devices and emulators
51. 51
SpecFlow & Appium
• Write acceptance criteria independent of the platform
• Write bindings against a Selenium- like- API
• Run it on a devices or emulator
54. 54
Different levels of automation
Controller
Business Layer
Data Layer
Model
View
Device
automation
Trigger behaviour
through controller
Assert behaviour
on model, db, ..
Setup pre-conditions
through service
interfaces
Out-of-process
In-process
55. 55
How?
• Reuse the same feature files
• Write new bindings against the
Controller/ViewModel/…
58. 58
Why is it faster?
• No overhead for deploying and starting the app
• No need to wait for UI animations to finish
• No delays for searching UI controls
60. 60
Controller tests cannot fully replace UI tests
• UI tests are still needed
• Diversity of mobile devices
• Changes of manufacturer
• Different Hardware
Screen sizes
Densities
…
• Automated UI tests are still faster than manual testing
62. 62
Excuses for avoiding mobile test automation
• “This is just a small app”
• “Only complex end-end tests would make sense”
• “Maintaining automated tests is too costly”
• Diversity of devices
• Small apps also run into regressions
• Bad feedback on app store cannot be undone
63. 63
Automated testing in mobile App development
• Business readable automated checks
• specification of acceptance criteria
• test first development
• Testability is an ongoing architectural concern
• every new feature needs to be testable
• not every acceptance criterion needs to be tested through UI
• Automated tests don’t replace manual testing entirely
• exploratory testing
• assertions that are hard to automate
• …
64. 64
Questions and further information
Andreas Willich (awi@techtalk.at)
Christian Hassa (ch@techtalk.at)