Most of the people think that quality in software development is limited to manual testing on the latest stage before releasing a product. That might be true 20 years ago in the industrial era. But current world is much more dynamic than before. Time to market became the most crucial metric nowadays. Releasing code to production need to be done faster and faster. How to maintain quality on a sufficient level in this fast paced environment? How to find a time to work on quality improvements? Those are two main questions I want to answer during this talk. Do not expect a silver bullet or even receipt to success. But definitely expect a lot of information about continuous delivery/deployment/improvements with a case studies and lessons we learned at Spotify. Spotify Engineering Culture: https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/ Scaling Agile @ Spotify http://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify Scaled Agile @ Spotify http://vimeo.com/111131934

1. December 6, 2014
Andrii Dzynia
Road Manager to Continuous Delivery
@adzynia
Quality Built In

2. 2
Spotify brings you the right music for
every moment!
Started in 2006 (in Sweden)
Now 1500+ employees, 500+ engineers
Over 30 million songs available
Over 20,000 songs added every day
5 development centres across the globe

3. What is Quality?
3

4. 4
Quality is a “state” ….
when expectations matches the reality.
Not just software defects, but stakeholders expectations as well

5. Why Quality is Important?
5

6. Cost of Bug/Change
6

7. What is a way to build high
quality product?
7

8. 8 Do not release until it’s perfect

9. 9
No one is perfect.

10. Example
Implementing internal back office system

11. 11
Step 1 - Prototyping phase
Technical spike investigation
Hacking different Back-End solutions
Ad-hoc design discussions
Stub implementation of Front End
Getting early user’s feedback on Front End

12. 12 WebApp
with
stubs
Sketches
on
whiteboard
Whatever works
to get end-user
understanding

13. 13
Step 2 - Setting delivery pipeline
Add more unit tests
Add more integration tests
Add more functional tests
Automate deployment configuration
Add dashboards and system monitoring
Clarifying End User Needs

14. 14
http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment

15. 15
Step 3 - Connecting the dots
Connect back-end and front-end parts
into
End-2-End system
Get user’s feedback and iterate

16. 16
Keep releasing small, usable, incremental pieces

17. 17
What is under the hood ?

18. 18

19. Organic Structure
19
@joakimsunden Scaled Agile @ Spotify

20. 20
Test Automation Pyramid

21. 21
How to avoid Ice-Cream effect?

22. 22
Be strict with this rule.

23. Testable micro-services architecture 23
Unit Tests
Component Tests
Functional
Tests
Integration Tests
System/End-2-End Tests
Unit Tests
Component Tests
Functional
Tests
JS Unit Tests
JS Component Tests
UI
Functional
Tests

24. 24
Unit Tests
Check logic of minimal code snippet
with no or mocked dependencies
https://github.com/mockito/mockito
https://github.com/junit-team/junit

25. 25
Component Tests
Very much like unit tests but emulating
external components
e.g. cassandra database
https://github.com/jsevellec/cassandra-unit

26. 26
Fake back-end for client apps
Client App
Fake Back-End Server
Spotify Back End
https://github.com/azagniotov/stubby4j
https://github.com/dreamhead/moco
http://jsonstub.com

27. 27
Integration Tests
Check an integration between components
You do not need whole system to check the integration
https://github.com/spotify/helios/blob/master/docs/testing_framework.md

28. 28
Create Test Environment ‘on the fly’
@docker containers
https://www.docker.com

29. 29
Functional Tests
Check services API behaviour
or
End user use cases
http://hc.apache.org
http://seleniumhq.org

30. 30
Model Based Testing
http://graphwalker.org

32. 32
Code Review
&
Human Testing
on
Pull Request

33. Delivery Pipeline
33

34. 34
Was not covered during this talk …
Load Testing
Gradual Rollouts
A/B Testing
Feature Toggles
Monitoring/Alerting
Information Radiators
… but stay tuned

35. 35
Test Engineering Roles and Responsibilities
Technical Test Engineer Test Engineer
! = Test Automator ! = Manual Tester
~ Software Engineer in Test
~ Context Driven Tester

36. 36
Technical Test Engineer
Work as a software developer
Advocate testability of the product
Argue on software design with software engineers
Help with building new/injecting existed development tools

37. 37
Development Productivity Tools
e.g. cli control over amazon cloud http://dashing.io
Docker orchestration
https://github.com/spotify/helios
Cassandra Unit
https://github.com/jsevellec/cassandra-unit
Spoticloud
Jenkins job-dsl-plugin
https://github.com/mikaellanger/job-dsl-plugin
Dashboards

38. 38
Test Engineer
Knows business domain
Focused on exploring the product
Free to use any programming language to test specific use case
Free to use any programming language to automate his work

39. 39
Mind Map as a Tool
https://www.mindmup.com
Product tree
Scenarios
Playbooks
Checklists
Session notes

40. Quality Engineers
40
@jamesmarcusbach calls us Test Jumpers
TEST JUMPERS: ONE VISION OF AGILE TESTING
http://www.satisfice.com/blog/archives/1372
“RESPONSIBLE TESTER”
http://www.satisfice.com/blog/archives/1364
OMEGA TESTER
http://www.satisfice.com/articles/omega_tester.pdf

41. 41
Any other ways
to improve
quality?

42. 42
Break Uncertainty
test ideas during healthy
discussions
test requirements via end
users collaboration
discuss system design and
conner cases earlier in the
planing/design meetings
define definition of done

43. 43
Use Google Docs collaboration for finding group consensus
But have responsible person to figure out what consensus means in each case

44. Social Programming
44
write “checks” during implementation
shape your thoughts via pair discussions
peer review before merging to master

45. 45
Not mentioned Engineering Practices
User Stories
Planning meeting
Standup meeting
Open workspace
TDD
Refactoring
etc……
http://www.extremeprogramming.org/rules.html

46. Employee Early Builds Testing
46

47. 47
Hackathons
One week to do whatever
you want
Do it alone or with a team
Crazy ideas comes out

48. 48
QA engineer at Spotify
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

49. How to check you are on
a right path ?
49

50. That is easy to spot 50

51. 51 Ready for a journey?
Make Quality explicit
Find Quality promoters
Define your way of Quality
improvements
Set right Quality constraints
Share results as early as possible
Keep looking further quality improvements
Probably you will never end :)

52. 52
@adzynia
adzynia@spotify.com
Spotify Engineering Culture
part 1
part 2

53. and …
Stop saying QA
when you mean Testing
Want to join the band?
Check out http://www.spotify.com/jobs or @Spotifyjobs for more information