Presentation given at ICPRAM 2013, International Conference on Pattern Recognition, Barcelona, Spain, February 2013 ABSTRACT: We propose a novel gesture spotting approach that offers a comprehensible representation of automatically inferred spatiotemporal constraints. These constraints can be defined between a number of characteristic control points which are automatically inferred from a single gesture sample. In contrast to existing solutions which are limited in time, our gesture spotting approach offers automated reasoning over a complete motion trajectory. Last but not least, we offer gesture developers full control over the gesture spotting task and enable them to refine the spotting process without major programming efforts. Paper:http://www.academia.edu/2157798/Declarative_Gesture_Spotting_Using_Inferred_and_Refined_Control_Points

1. Declarative Gesture Spotting Using
Inferred and Refined Control Points
Department of Computer Science, Vrije Universiteit Brussel (VUB)
Lode Hoste - lhoste@vub.ac.be
Brecht De Rooms - bderooms@vub.ac.be
Beat Signer - bsigner@vub.ac.be
1

2. Gesture Classification
template
sample segment
e.g. Dynamic Time Warping
2

3. Gesture Classification
template
sample segment
e.g. Dynamic Time Warping
3

4. Gesture Classification
4

5. Gesture Classification
5

6. Gesture Classification
6

7. Gesture Classification
7

8. Gesture Classification
8

9. Gesture Classification
Too Complex
9

10. Gesture Classification
Start?
We need gesture spotting
End?
10

11. Gesture Classification
Start?
We need gesture spotting
reduce amount of work for gesture
classification
End?
11

12. Gesture Spotting
Goal:
- High Recall
- High Precision
Popular techniques:
- Hidden Markov Models
- Continuous Dynamic Programming
12

13. Our Declarative Gesture Spotting Approach
1. Automatically infer control points
- from one representative sample
- one shot learning
2. Generate declarative code
Spotted? - understandable and extensible spotting definitions
3. Call a gesture classification algorithm
13

14. Automatically Infer Control Points
One representative sample
Z gesture
14

15. Automatically Infer Control Points
One representative sample
- start with the first point
Z gesture
15

16. Automatically Infer Control Points
One representative sample
- start with the first point
- search for the smallest angle
- within a given time-frame
Z gesture
16

17. Automatically Infer Control Points
One representative sample
- start with the first point
- search for the smallest angle
- within a given time-frame
Z gesture
17

18. Automatically Infer Control Points
One representative sample
- start with the first point
- search for the smallest angle
- within a given time-frame
- refine control points
(optionally)
Z gesture
18

19. Comprehensible Code Generation
Based on the inferred control points:
19

20. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D) there is a start point
?p1
20

21. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) there is a second point ?p1 ?p2
21

22. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
temporal constraint:
that comes after the start point
22

23. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
spatial constraint:
23

24. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
4
185
spatial constraint:
.. a circular area with centerpoint
x: 185 y:4 (pixels)
24

25. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
4
185
spatial constraint:
.. a circular area with centerpoint
x: 185 y:4 (pixels)
and radious 76
25

26. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
4
185
26

27. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
4
6 ?p3 (Point2D) 185
7 (test (< ?p2.time ?p3.time))
8 (test (inside_control_point ?p1 ?p3 15 178 76))
9 ?p4 (Point2D)
10 (test (< ?p3.time ?p4.time))
11 (test (inside_control_point ?p1 ?p4 197 175 76))
?p3 ?p4
27

28. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
6 ?p3 (Point2D)
7 (test (< ?p2.time ?p3.time))
8 (test (inside_control_point ?p1 ?p3 15 178 76))
9 ?p4 (Point2D)
10 (test (< ?p3.time ?p4.time))
11 (test (inside_control_point ?p1 ?p4 197 175 76))
each point expressed relative to ?p1
?p3 ?p4
28

29. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
6 ?p3 (Point2D)
7 (test (< ?p2.time ?p3.time))
8 (test (inside_control_point ?p1 ?p3 15 178 76))
9 ?p4 (Point2D)
10 (test (< ?p3.time ?p4.time))
11 (test (inside_control_point ?p1 ?p4 197 175 76))
temporal constraints:
the points should be ordered ?p3 ?p4
29

30. Comprehensible Code Generation
Based on the inferred control points:
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D) ?p1 ?p2
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
6 ?p3 (Point2D)
7 (test (< ?p2.time ?p3.time))
8 (test (inside_control_point ?p1 ?p3 15 178 76))
9 ?p4 (Point2D)
10 (test (< ?p3.time ?p4.time))
11 (test (inside_control_point ?p1 ?p4 197 175 76))
12 =>
13 (call DynamicTimeWarping
14 (select-between ?p1.time ?p4.time)
15 (gesture-set “CharacterZ”)) ?p3 ?p4
call an external algorithm for classification
30

31. A State Machine is Not Enough
template continuous stream
State 1
State 1 State 2
State 3 State 4
31

32. A State Machine is Not Enough
template continuous stream
State 2
State 1 State 2
State 3 State 4
32

33. A State Machine is Not Enough
template continuous stream
State 2
State 1 State 2
reset state machine?
State 3 State 4
33

34. A State Machine is Not Enough
template continuous stream
State 1
State 1 State 2
reset state machine?
State 3 State 4
34

35. A State Machine is Not Enough
template continuous stream
State 1
State 1 State 2
State 3 State 4
35

36. A State Machine is Not Enough
template continuous stream
State 1
State 1 State 2
State 3 State 4
36

37. A State Machine is Not Enough
template continuous stream
State 1 State 2
gesture overshot
 missed gesture
State 3 State 4
37

38. A State Machine is Not Enough
template continuous stream
State 1 State 2
State 3 State 4
38

39. A State Machine is Not Enough
template continuous stream
State 1 State 2
refused by gesture
classification
State 3 State 4
39

40. A State Machine is Not Enough
template continuous stream
State 1 State 2
refused by gesture
classification
State 3 State 4
overlapping gesture
 missed gesture
40

41. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
41

42. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
42

43. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
43

44. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
44

45. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
45

46. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z 3 ?p2 (Point2D)
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
46

47. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z 3 ?p2 (Point2D)
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
47

48. Incremental Evaluation of Every Possible Combination
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
3 ?p2 (Point2D)
1 (defrule symbol_z
2 ?p1 (Point2D)
1 (defrule symbol_z 3 ?p2 (Point2D)
2 ?p1 (Point2D)
3 ?p2 (Point2D)
4 (test (< ?p1.time ?p2.time))
5 (test (inside_control_point ?p1 ?p2 185 4 76))
RETE engine takes care
of caching intermediate calculations
48

49. Incremental Evaluation of Every Possible Combination
?p1
49

50. Incremental Evaluation of Every Possible Combination
?p1
?p1
50

51. Incremental Evaluation of Every Possible Combination
?p1
?p1
?p1
51

52. Incremental Evaluation of Every Possible Combination
?p1
?p1
?p1
52

53. Incremental Evaluation of Every Possible Combination
?p1
?p1
?p1
53

54. Incremental Evaluation of Every Possible Combination
?p1
?p1
?p1
?p1
54

55. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
55

56. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
?p1
56

57. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
?p3
?p1
57

58. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
?p3
?p1
58

59. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
?p3
?p1
59

60. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p1
?p1
?p3
?p1
60

61. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
?p1
?p1
?p3
?p1
61

62. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
?p1
?p1
?p3
?p3
?p1
62

63. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p1
?p1
?p3
?p3
?p1
63

64. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p1
?p1
?p3
?p3
64

65. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p1
?p3
?p3
65

66. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p1
?p3 ?p1
?p3
66

67. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p1
?p3 ?p1 ?p4
?p3
67

68. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p3 ?p1 ?p4
?p3
68

69. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p3 ?p1 ?p4
?p3
69

70. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p3 ?p1 ?p4
?p3
70

71. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p2
?p3 ?p1 ?p4
?p3
71

72. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p2
?p3 ?p1 ?p4
?p3
72

73. Incremental Evaluation of Every Possible Combination
?p2
?p1
?p1
?p1
?p2
Limit calculations to
a sliding window
?p2
?p3 ?p1 ?p4
?p3
73

74. Extending the ‘Flick Right’ Definition
Reducing false positives using negation
?p1 ?p2 ?p3
74

75. Extending the ‘Flick Right’ Definition
Reducing false positives using negation
?p1 ?p2 ?p3
75

76. Extending the ‘Flick Right’ Definition
?p1 ?p2 ?p3
76

77. Extending the ‘Flick Right’ Definition
?p1 ?p2 ?p3
(not
(and
(Point2D (y ?b y) (time ?b time)) … there should not be a point where the …
(test (< ?b time ?p3.time))
(test (> ?b time ?p1.time))
(test (> (abs (- ?p1.y ?b y)) 245))))
77

78. Extending the ‘Flick Right’ Definition
?p1 ?p2 ?p3
(not
(and
(Point2D (y ?b y) (time ?b time)) … there should not be a point where the …
(test (< ?b time ?p3.time))
… time lies in between ?p1 and ?p3 …
(test (> ?b time ?p1.time))
(test (> (abs (- ?p1.y ?b y)) 245))))
78

79. Extending the ‘Flick Right’ Definition
y
?p1 ?p2 ?p3
y
(not
(and
(Point2D (y ?b y) (time ?b time)) … there should not be a point where the …
(test (< ?b time ?p3.time))
… time lies in between ?p1 and ?p3 …
(test (> ?b time ?p1.time))
(test (> (abs (- ?p1.y ?b y)) 245)))) … for which y is greater than 245.
79

80. Evaluation
22 77.50 52.10 78.75 56.50
24 83.13 47.16 84.38 52.53
26 90.63 42.40 91.25 46.79
28 93.75 39.47 94.38 43.26
30 97.50 35.37 97.50 39.29
32 98.75 32.78 98.75 36.41
• 16 gestures
• 1760 gesture samples
• 10 subjects
80

81. Future Work
Scale / rotational Visual 3D gesture
invariant spotting definitions
81

82. Summary
lhoste@vub.ac.be bderooms@vub.ac.be bsigner@vub.ac.be
Auto infer control points Generate Spotting Code Call classification
• Extensive spotting: support for real-time continuous streams.
• No gesture overshooting: support for overlapping submatches.
• Comprehensible: easy to understand and to correct.
• Extensible: allows for expert adjustments.
82

83. Lode Hoste, Brecht De Rooms
and Beat Signer, Declarative
Gesture Spotting Using
Inferred and Refined Control
Points, Proceedings of
ICPRAM 2013, International
Conference on Pattern
Recognition, Barcelona, Spain,
February 2013
83