1. SIKULI
USING SCREENSHOTS FOR GUI
AUTOMATION AND TESTING
Tsung-Hsiang (Sean) Chang, Rob Miller
MIT CSAIL
Collaborator: Tom Yeh
1

2. How to
tell computers what to do?
2

3. case $rtval in
1) rm -f /tmp/dirip.$$ ; return ;;
255) rm -f /tmp/dirip.$$ ; return ;;
esac
mfile=`cat /tmp/dirip.$$`
if [ -z $mfile ]
then
mfile=`pwd`/*
SHELL SCRIPT
else
grep "*" /tmp/dirip.$$
if [ $? -eq 1 ]
then
! mfile=$mfile/*
fi
fi
for i in $mfile
do
if [ -f $i ]
then
! echo "$i Delete?" >> /tmp/finallist.$$
fi!
done
3

4. 4

5. GUI AUTOMATION
5

6. 2
3
1
6

7. 7

8. click click
7

9. SIKULI SCRIPT
8

10. SIKULI TEST
9

11. OUTLINE
• Problem - How to refer to a GUI widget?
• Sikuli Script - examples and demos
• Sikuli Test - demo
• Test By Demonstration
• Support of testing practices
• Conclusions
10

12. HOW TO REFER TO A GUI WIDGET?
11

13. HOW TO REFER TO A GUI WIDGET?
<AWTTestScript> Abbot
<launch classpath="src/demo" args="[]"
class="CelsiusConverter" method="main" />
<action args="45" method="actionKeyString"/>
<action class="javax.swing.AbstractButton"
args="Convert..." method="actionClick"/>
<assert value="113 Fahrenheit"
component="113 Fahrenheit"
method="getText"/>
by pre-programmed references
<terminate/>
</AWTTestScript>
11

14. HOW TO REFER TO A GUI WIDGET?
<AWTTestScript> Abbot
<launch classpath="src/demo" args="[]"
Autoit
class="CelsiusConverter" method="main" />
<action args="45" method="actionKeyString"/>
<action class="javax.swing.AbstractButton"
args="Convert..." method="actionClick"/>
<assert value="113 Fahrenheit"
component="113 Fahrenheit"
method="getText"/>
by pre-programmed references
<terminate/> by absolute locations
</AWTTestScript>
11

15. Use Screenshots!
12

16. TEMPLATE MATCHING
click(
)
13

17. TEMPLATE MATCHING
click(
)
13

18. EXAMPLE 1:
EMPTY TRASH CAN
2
3
1
14

19. EX2: FACEBOOK STALKER
15

20. EX2: FACEBOOK STALKER
15

21. 16

22. RESTRICT SEARCH REGION
17

23. EX3: MUTE SPEAKERS
18

24. EX3: MUTE SPEAKERS
18

25. SPATIAL OPERATORS
19

26. SPATIAL OPERATORS
19

27. SPATIAL OPERATORS (2)
20

28. EX4: SET IP ADDRESS
21

29. VISUAL-EVENT-DRIVEN
PROGRAMMING
22

30. VISUAL-EVENT-DRIVEN
PROGRAMMING
22

31. SIKULI TEST
23

32. VISUAL ASSERTION
• Verify appearance
24

33. VISUAL ASSERTION
• Verify replacement
25

34. TEST BY DEMONSTRATION
detect changes
click
26

35. Demo
27

36. GENERATE ACTION
COMMANDS
mouse mouse
‘A’ ‘A’ ‘b’ ‘b’
down up
down up down up
28

37. GENERATE ACTION
COMMANDS
low-level events
mouse mouse
‘A’ ‘A’ ‘b’ ‘b’
down up
down up down up
28

38. GENERATE ACTION
COMMANDS
low-level events
mouse mouse
‘A’ ‘A’ ‘b’ ‘b’
down up
down up down up
high-level
commands
click type “Ab”
28

39. GENERATE ACTION
COMMANDS
low-level events
mouse mouse
down up
‘A’ ‘A’ ‘b’ ‘b’ click
down up down up
high-level
commands
click type “Ab”
combine with
screenshots
28

40. GENERATE ASSERTIONS
click
29

41. GENERATE ASSERTIONS
click
1. find differences / manually hint
29

42. GENERATE ASSERTIONS
click
1. find differences / manually hint
2. group together
29

43. GENERATE ASSERTIONS
click
1. find differences / manually hint
2. group together
3. combine with assertExist
29

44. GENERATE ASSERTIONS
click
1. find differences / manually hint
2. group together
3. combine with assertExist
29

45. jUNIT INTEGRATION
30

46. TEST-DRIVEN DEVELOPMENT
31

47. READABILITY OF CODE
32

48. PLATFORM INDEPENDENCE
• Works
on ANY GUI that can be displayed on
Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
33

49. 34

50. 34

51. LIMITATIONS
Theme Variations
Visibility Constraints
35

52. DISCUSSION
Sikuli programs are written against the user interface
instead of an API
UI: visible, familiar, always exists
API: faster, probably more stable
36

53. FUTURE WORK
• fast and accurate OCR on screen
• more robust against different skins
37

54. Thanks!
http://sikuli.csail.mit.edu
or, just google “sikuli”
New Sikuli 0.10 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
38