View webinar: http://www.eurostarconferences.com/community/member/webinar-archive/webinar-87-experiences-of-test-automation-at-spotify- At Spotify, we want the manual testing effort to be focused as much as possible at feature testing, less on regression tests. But we still have to do regression. So, we tried to automate a big chunk of that. Regression tests are run on our Desktop, Android, iOS and WebPlayer clients, and also some backend services. I will share with you how far we have come. What techniques, tools and methodologies we have tried. What experiences has been good, and what has been not that good.

1. Experiences Of Test Automation
@ Spotify
Kristian Karl
December 2, 2013

3. Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson

5. Why automation?
•
•
•
•
We want to move faster
Quick dev feed-back
Continous Delivery
Continously monitor the
state of the product
• Bug hunting
From: ”Intelligent Test Automation” by Harry Robinson

6. What to automate?
Graphical user interface testing
Usability testing
Software performance testing
System testing
Functional testing
Load testing
Volume testing
Stress testing
Security testing
Scalability testing
Sanity testing
Unit testing
Smoke testing
Component testing
API testing
Regression testing
Installation testing
Maintenance testing
Recovery and failover testing.
Accessibility testing
Monkey testing
Integration testing

7. What to automate?
Graphical user interface testing

8. Our goals
•
Create automated end-user regression tests on 4 major
platforms
1) Desktop – Windows and OSX
2) iOS – iPhone and iPad
3) Android
4) Webplayer
•
•
•
Help testers, not replacing them
Deliver automated regression tests for a feature as a part of
definition of done
Deliver short feedback loops to teams using Dashboards

9. Our challenges
• Hard-to-test SUT
(Experiences of test automation: Case Study 1, An Agile Team’s Test
Automation Journey: The First Year), Dorothy Graham & Mark Fewster)
• Maintenance of automation
• Expectations
• Flaky tests... or is it flaky SUT?
• Testability
• Test data, test environments
• Supporting services

11. Model-based testing
• Models are the abstraction layer
• Testers design the automation using
models
• Developers implements the automation
using the models as drivers

12. App not
running
Exit app
Start app
Toggle
’Remember Me’
Close
Invalid credentials
Login view
displayed
Log out
Valid credentials
Start app
Main view displayed

14. public interface SimpleLogin {
public void e_Close();
public void e_Exit();
public void e_Init();
public void e_InvalidCredentials();
public void e_Logout();
public void e_StartClient();
public void e_ToggleRememberMe();
public void e_ValidPremiumCredentials();
public void v_ClientNotRunning();
public void v_LoginPrompted();
public void v_WhatsNew();
}

15. Demo
http://graphwalker.org/demo

16. Test automator
•
•
•
•
Professional Java developer.
Test experience is not mandatory.
Embedded in the squad (team).
Test automators form their own Guild

17. Developers and developers
• Why not use developers for TA?
• Why use developers for TA?
• Test API’s
- Defined by TA
- Implemented by developers

18. 18
Section name
Before
Test model
Test model
After
Test model
Test model
Test interface
+ wrappers
Test model
Test model
QA/TA
Java test interface
JSON
Strings
ObjC test interface
Client
Devs
Client

19. Before
String cmd = "android.view.View seekBarView =
solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);";
cmd += "int[] xy = new int[2];";
cmd += "seekBarView.getLocationOnScreen(xy);";
cmd += "solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * " +
position + "f, xy[1] + seekBarView.getHeight() / 2.0f)";
BeanRemoteClient.sendToServer(cmd);
After
PlayerAuto player = Pages.remote(PlayerAuto.class);
player.seekTrack(position);

20. Supporting services
•
•
•
•
•
Continuous testing
QA Lab
TDS – Test Data Service
TRS – Test Result Service
Virtualization

22. Open source (and freeware) tools that we use
• yEd – editing FSM models [www.yworks.com]
• GraphWalker – generate test sequences from models
[graphwalker.org github.com/spotify/python-graphwalker]
• Sikuli – image recognition tool [www.sikuli.org]
• NuRemote - communicating with iOS client
[github.com/nevyn/NuRemoting]
• TestNG – group tests into suites and and run them [testng.org]
• Java – code implementation
• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]

23. Demo

24. Q&A