Slides from my talk "Selenium Based Visual Test Automation" at the Selenium Israel Meetup on Dec 1st, 2014.

1. SELENIUM BASED
VISUAL TEST
AUTOMATION
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com

2. YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

4. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

5. WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users

6. A VISUAL BUG

7. AND ANOTHER…

8. WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO
APPROXIMATE MANUALLY
 Web browsers
 Operating systems
 Screen resolutions
 Responsive designs
 L10n

9. WHY SHOULD IT BE AUTOMATED?

10. WHY SHOULD IT BE AUTOMATED?
MANUAL
VISUAL
TESTING
IS
ERROR
PRONE

11. WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSS Critc
Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
Eyes
Huxley
FBSnapshotTestCase
Selenium Visual Diff Gemini
VisualCeption
Specter
Snap And Compare

12. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

13. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

14. DEMO https://github.com/webdriverio/webdrivercss

15. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

16. DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java)
Wraith (URLs) CSS Critic (URLs)
Needle (Python)
dpdxt (URLs)
Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCase
Selenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
Gemini (JS)

17. SCREENSHOTS
 Real browsers?
 Full page?
 Frames?
 Regions?
 Viewport size?
 Page stabilization?
 Page preparation?

18. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

20. WHY NOT?
FALSE POSITIVES

21. ANTI-ALIASING?

22. ANTI ALIASING 1/2

23. ANTI ALIASING 2/2

24. BRIGHTNESS 1/2

25. BRIGHTNESS 2/2

26. SUB PIXEL SHIFTS 1/2

27. SUB PIXEL SHIFTS 2/2

28. SUB PIXEL SHIFTS 1/2

29. SUB PIXEL SHIFTS 2/2

30. ANDMORE…
 1 pixel offsets in element positioning
 Dynamic content
Moving elements
 Images of different size

32. Image Comparison APIs

33. ImageMagick
 A powerful command line tool for image processing.
 APIs are available for most programming languages.
 Fuzzing is used to eliminate slight color differences
 An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

34. Resemble.js
 An image comparison tool implemented in Javascript
 Used by PhantomCSS and other tools.
 Good antialiasing support
 An error ratio is usually used to determine a match
 http://huddle.github.io/Resemble.js/

35. Applitools Eyes
 A specialized image processing stack designed to compare
computer generated UI images
 Anti-aliasing
 Partial and full pixel offsets
 Images of different size
 Dynamic content
 Moving elements
 Layout matching
 Available as a cloud service

36. DEMO

37. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

38. REPORT DIFFERENCES
As files on the file system (combined with source control)

39. REPORT DIFFERENCES
As a Gallery (example from Selenium Visual Diff)

40. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

41. UPDATE THE BASELINE
 Rename or commit individual image files
 Overwrite mode

42. UPDATE THE BASELINE
 GUI (Gemini GUI)
 Automatic maintenance (demo)

43. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

44. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA

45. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others

46. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA

47. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA

48. QUESTIONS?

49. THANK YOU
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com