By David Drake, Lead Software Development Engineer in Test at Dynacron Group.

1. WORKS IN (X, Y, Z)
Parallel Combination Testing With Selenium,
JUnit, and Sauce Labs
David Ross Drake

2. 4/12/2013 2
Dynacron Group
• Surprisingly, not the evil corporation from that
80’s dystopian sci-fi movie
• Actually a Seattle consultancy that helps
companies implement continuous delivery…
• Integrated Testing
• Custom Development
• Big Data Strategies
• Deployment Automation

3. 4/12/2013 3
A Common Selenium Problem
• Tests are slow
• Lots of combinations need to be tested
• Browsers
• Versions
• Operating Systems
• Languages
• Skins
• Combination interfaces are very similar

4. 4/12/2013 4
Parameter Description
Browser
- Application
- Type
- Version
- OS
Brand
- Code
- URL Component
Locale
- Language
- Country

5. 4/12/2013 5
Extant Solution: CSV
• Simple for Manual • Incapable of
Testers to Use containing
• Allows some nested complicated data
structures (PSV in • Hard to Read
CSV…) • Ugly

6. 4/12/2013 6
TYPESAFE CONFIG
Demo

7. But the Problem is Actually Worse…
IC CP IN HI EX SB CW 6C PCR
US En X X X X X X X X
UK En X X X X X X
French X X X X X X X X
German X X X X X X
Japanese X X X
Chinese X X X X X X X
Korean X X X
Italian X X X X
Arabic X
Hebrew X
Portuguese X X
Spanish X X X X X X X X
Russian X
Turkish X

8. 4/12/2013 8
How Many Combinations?
• Browsers: A, B, C, D, E
• Locales: 1, 2, 3, 4
• Brands: X, Y, Z
Let S1 be the size of the largest parameter array,
S2 be the next largest, etc…

9. 4/12/2013 9
Each Once?
• Browsers: A, B, C, D, E 1. (A, 1, X)
• Locales: 1, 2, 3, 4 2. (B, 2, Y)
• Brands: X, Y, Z 3. (C, 3, Z)
4. (D, 4, X)
Number of 5. (E, 1, Y)
combinations: S1

10. 4/12/2013 10
All Pairs?
• Browsers: A, B, C, D, E 1. (A, 1, X) 11. (C, 3, Y)
2. (A, 2, Y) 12. (C, 4, Z)
• Locales: 1, 2, 3, 4
3. (A, 3, Z) 13. (D, 1, X)
• Brands: X, Y, Z 4. (A, 4, X) 14. (D, 2, Y)
5. (B, 1, Y) 15. (D, 3, Z)
6. (B, 2, Z) 16. (D, 4, X)
Number of 7. (B, 3, X) 17. (E, 1, Y)
combinations: (S1S2) 8. (B, 4, Y) 18. (E, 2, Z)
9. (C, 1, Z) 19. (E, 3, X)
10. (C, 2, X) 20. (E, 4, Y)

11. 4/12/2013 11
All?
• Browsers: A, B, C, D, E 1. (A, 1, X) 11. (A, 4, Y)
2. (A, 1, Y) 12. (A, 4, Z)
• Locales: 1, 2, 3, 4
3. (A, 1, Z) 13. (B, 1, X)
• Brands: X, Y, Z 4. (A, 2, X) 14. (B, 1, Y)
5. (A, 2, Y) 15. (B, 1, Z)
6. (A, 2, Z) 16. (B, 2, X)
Number of 7. (A, 3, X) 17. (B, 2, Y)
combinations: (S1S2S3) 8. (A, 3, Y) 18. (B, 2, Z)
9. (A, 3, Z) 19. (B, 3, X)
10. (A, 4, X) 20. …

12. 4/12/2013 12
Risk vs Rewards
Risk Reward
x = probability a given pair y = average time it takes to
has a failure only found test a combination
when together P = Fraction of testing time
R = probability that “each- taken testing “each-once”
once” fails to reveal a as compared to “all-pairs”
defect that could be seen
in “all-pairs” M = S1 / S1S2
R = (1 - x) ^ (S1S2 – S1)

13. 4/12/2013 13
Converting Combinations into Tests
Read Parameters
• Add overrides for class, profiles, and system properties
• Read as TypeSafe Config objects
Form list of config combinations
• Apply combination strategy from config
Convert configurations back into json
• Simple in TypeSafe Config – prevents writing custom
object mapping
Deserialize json combination in
ParameterCombination object
• Uses Jackson and @FromJson annotations as necessary

14. 4/12/2013 14
Parallelization
• Easy to Implement
• See Sauce Labs blogs!
• Very hard to do right
• See demo

15. 4/12/2013 15
Class Browser Reuse
b = time to start and close browser
t = average time to run a test method
m = average number of methods in a class
F = fraction of time spent testing when using
one browser per class instead of per method
F = (b/m + t) / (b + t)

16. 4/12/2013 16
Class Browser Reuse (small tests)
15 = time to start and close browser
6 = average time to run a test method
5 = average number of methods in a class
F = fraction of time spent testing when using
one browser per class instead of per method
F = (15/5 + 6) / (15 + 6) = 3/7 = 42.9%

17. 4/12/2013 17
Class Browser Reuse (long tests)
15 = time to start and close browser
60 = average time to run a test method
5 = average number of methods in a class
F = fraction of time spent testing when using
one browser per class instead of per method
F = (15/5 + 60) / (15 + 60) = 21/25 = 84%

18. 4/12/2013 18
Tracking Parameterized and
Parallelized Tests
• Descriptive method naming
• Descriptive logging
• Demo!

19. 4/12/2013 19
Future Work
• Remembering Tested Combinations in
Coverage Reports
• Effectively displaying tested combinations
• Converting to using Junit 4.11 style
parameterized runner.

20. 4/12/2013 20
Questions?
• Parallel Webtest: https://github.com/dynacron-
group/parallel-webtest
• TypeSafe Config:
https://github.com/typesafehub/config
• Sauce post that provided the seed:
http://sauceio.com/index.php/2010/10/paralle
l-junit-4-and-selenium-part-three-parallelism-
and-ondemand/