Slides from my presentation on visual testing at http://www.meetup.com/GDG-Rashlatz/events/216316352/ and http://www.meetup.com/SeleniumSanJose/events/209029032/

1. AUTOMATED VISUAL
TESTING THAT DOES
NOT SUCK
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com

2. YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

4. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

5. WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users

6. A VISUAL BUG

7. WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO
APPROXIMATE MANUALLY
 Web browsers
 Operating systems
 Screen resolutions
 Responsive designs
 L10n

8. WHY SHOULD IT BE AUTOMATED?

9. WHY SHOULD IT BE AUTOMATED?
MANUAL
VISUAL
TESTING
IS
ERROR
PRONE

10. WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSSCritique
Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
Eyes
Huxley
FBSnapshotTestCase
Selenium Visual Diff Gemini
VisualCeption
Specter
Snap And Compare

11. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

12. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

13. DEMO https://github.com/webdriverio/webdrivercss

14. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

15. DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java)
Wraith (URLs) CSSCritique (URLs)
Needle (Python)
dpdxt (URLs)
Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCase
Selenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
Gemini (JS)

16. SCREENSHOTS
 Real browsers?
 Full page?
 Frames?
 Regions?
 Viewport size?
 Page stabilization?
 Page preparation?

17. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

19. WHY NOT?
FALSE POSITIVES

20. ANTI-ALIASING?

21. ANTI ALIASING 1/2

22. ANTI ALIASING 2/2

23. BRIGHTNESS 1/2

24. BRIGHTNESS 2/2

25. SUB PIXEL SHIFTS 1/2

26. SUB PIXEL SHIFTS 2/2

27. SUB PIXEL SHIFTS 1/2

28. SUB PIXEL SHIFTS 2/2

29. ANDMORE…
 1 pixel offsets in element positioning
 Dynamic content
Moving elements
 Images of different size

31. Image Comparison APIs

32. ImageMagick
 A powerful command line tool for image processing.
 APIs are available for most programming languages.
 Fuzzing is used to eliminate slight color differences
 An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

33. Resemble.js
 An image comparison tool implemented in Javascript
 Used by PhantomCSS and other tools.
 Good de-antialiasing
 An error ratio is usually used to determine a match
 http://huddle.github.io/Resemble.js/

34. Applitools Eyes
 A specialized image processing stack designed to compare
computer generated UI images
 Anti-aliasing
 Partial and full pixel offsets
 Images of different size
 Dynamic content
 Moving elements
 Layout matching
 Available as a cloud service

35. DEMO

36. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

37. REPORT DIFFERENCES
As files on the file system (combined with source control)

38. REPORT DIFFERENCES
As a Gallery (example from Selenium Visual Diff)

39. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

40. UPDATE THE BASELINE
 Rename or commit individual image files
 Overwrite mode

41. UPDATE THE BASELINE
 GUI (Gemini GUI)
 Automatic maintenance (demo)

42. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?

43. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA

44. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others

45. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA

46. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA

47. QUESTIONS?

48. THANK YOU
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com