This is the slide deck from my senior project presentation. More information is available at http://compsci.snc.edu/cs460/pavlra/

1. The Tactus Project
Novel Multi-Touch Task Management
Ryan Pavlik

2. The Project Definition

3. The Project Definition
Overall objective: Do something neat and new.

4. The Project Definition
Overall objective: Do something neat and new.
Build hardware to enable multi-touch (MT) input to a
computer system.

5. The Project Definition
Overall objective: Do something neat and new.
Build hardware to enable multi-touch (MT) input to a
computer system.
Develop software to use this hardware in a novel way.

6. The Project Definition
Overall objective: Do something neat and new.
Build hardware to enable multi-touch (MT) input to a
computer system.
Develop software to use this hardware in a novel way.
Refinement: Create task-management software to
use multi-touch

7. Getting it done...

8. Getting it done...
Lots of sawdust, plenty of error messages

9. Hardware Deliverables

10. Hardware Deliverables
Multi-touch displays capable of also supporting a
“standard” workstation

11. Hardware Deliverables
Multi-touch displays capable of also supporting a
“standard” workstation
“MT Mini” temporary test device

12. Hardware Deliverables
Multi-touch displays capable of also supporting a
“standard” workstation
“MT Mini” temporary test device
Desk #1 - Projected Rear DI - complete

13. Hardware Deliverables
Multi-touch displays capable of also supporting a
“standard” workstation
“MT Mini” temporary test device
Desk #1 - Projected Rear DI - complete
Desk #2 - LCD Laser Light Plane - lasers arrived on
Monday...

14. Software Deliverables

15. Software Deliverables
rp-mt-scripts - Multi-touch software system

16. Software Deliverables
rp-mt-scripts - Multi-touch software system
Linux-based (Ubuntu 9.04) system for automatic
setup and management of multi-touch software

17. Software Deliverables
rp-mt-scripts - Multi-touch software system
Linux-based (Ubuntu 9.04) system for automatic
setup and management of multi-touch software
Now installing required software a one-step process

18. Software Deliverables
rp-mt-scripts - Multi-touch software system
Linux-based (Ubuntu 9.04) system for automatic
setup and management of multi-touch software
Now installing required software a one-step process
tactus-navigator - “Nav display” task manager

19. Software Deliverables
rp-mt-scripts - Multi-touch software system
Linux-based (Ubuntu 9.04) system for automatic
setup and management of multi-touch software
Now installing required software a one-step process
tactus-navigator - “Nav display” task manager
Replacement for task bar

20. Software Deliverables
rp-mt-scripts - Multi-touch software system
Linux-based (Ubuntu 9.04) system for automatic
setup and management of multi-touch software
Now installing required software a one-step process
tactus-navigator - “Nav display” task manager
Replacement for task bar
Places icons for current applications on MT display

21. What’s up with multi-touch?

22. What’s up with multi-touch?
Keyboard

23. What’s up with multi-touch?
Keyboard
Mouse

24. What’s up with multi-touch?
Keyboard
Mouse
Touchscreen

25. What’s up with multi-touch?
Keyboard
Mouse
Touchscreen
Multi-touch tablets/screens

26. Optical multi-touch sensing

27. Optical multi-touch sensing
Blob tracking through computer vision

28. Infrared
sensing

29. Infrared
sensing
Permits multi-touch
displays, not just
tablets

30. Table design principles

31. Table design principles
1001 ways to make a multi-touch screen

32. Revision 0: “MT Mini”

33. Revision 0: “MT Mini”
Software test platform, works by “front DI”

34. Rear diffused illumination

35. Rear diffused illumination
“Projected rear DI” chosen for first hardware
revision

38. Laser light plane (LLP)

39. Laser light plane (LLP)
LCD LLP: Slow shipping is disappointing...

40. Software

41. Software
Doing work with multi-touch, without throwing
out your keyboard or mouse

42. Navigation display

43. Navigation display
Multi-touch surface as secondary display

44. Navigation display
Multi-touch surface as secondary display
Enhance interaction with existing software

45. Navigation display
Multi-touch surface as secondary display
Enhance interaction with existing software
Requires a custom desk for both multi-touch I/O and
normal I/O

46. Normal
Display
Physical Multi-touch
Nav. Display
Keyb.
Layout LCD and
Laser Light Plane
LCD and light
plane sandwiched
in there
Mouse
IR camera
Wheels to be installed
for portability

47. Normal
Display
Physical Multi-touch
Nav. Display
Keyb.
Layout LCD and
Laser Light Plane
LCD and light
plane sandwiched
in there
“Normal” desk with Mouse
IR camera
the multitouch
surface to the left of
Wheels to be installed
the keyboard for portability

48. Navigation display

49. Navigation display
Replaces your taskbar,
dock, etc.

50. Navigation display
Replaces your taskbar,
dock, etc.
Horizontal multi-touch
screen

51. Navigation display
Replaces your taskbar,
dock, etc.
Horizontal multi-touch
screen
Supports a variety of
gestures

52. Minimize all current windows

53. Minimize all current windows
Drag from “current display” bar into “pile canvas”

54. Minimize all current windows
Drag from “current display” bar into “pile canvas”

55. Restore a “window pile”

56. Restore a “window pile”
Drag from “pile canvas” into “current display” bar

57. Restore a “window pile”
Drag from “pile canvas” into “current display” bar

58. Move window to new “pile”

59. Move window to new “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful

60. Move window to new “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful

61. on ...
t en si
Ex
Spread out a “pile”

62. on ...
t en si
Ex
Spread out a “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful

63. on ...
t en si
Ex
Spread out a “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful

64. Demo time

65. Demo time
Please feel free to ask questions, etc...

66. How it got done

67. How it got done
What I used, what I learned

68. Process

69. Process
Research/Comparison

70. Process
Research/Comparison
Interaction Design

71. Process
Research/Comparison
Interaction Design
Parallel Platform Development:

72. Process
Research/Comparison
Interaction Design
Parallel Platform Development:
Hardware Construction

73. Process
Research/Comparison
Interaction Design
Parallel Platform Development:
Hardware Construction
Software platforms (task nav. with no interface, rp-
mt-scripts)

74. Process
Research/Comparison
Interaction Design
Parallel Platform Development:
Hardware Construction
Software platforms (task nav. with no interface, rp-
mt-scripts)
Integration

75. Strategies

76. Strategies
Internet resources: nuigroup.com forums, etc

77. Strategies
Internet resources: nuigroup.com forums, etc
DIY MT is popular right now

78. Strategies
Internet resources: nuigroup.com forums, etc
DIY MT is popular right now
Use existing components where possible

79. Strategies
Internet resources: nuigroup.com forums, etc
DIY MT is popular right now
Use existing components where possible
Avoid re-inventing the wheel when you can do just as
well without doing so.

80. Strategies
Internet resources: nuigroup.com forums, etc
DIY MT is popular right now
Use existing components where possible
Avoid re-inventing the wheel when you can do just as
well without doing so.
Careful design consideration and comparison of
alternatives

81. Knowledge

82. Knowledge
Object-oriented programming - CS core

83. Knowledge
Object-oriented programming - CS core
Lots of polymorphism and multiple inheritance

84. Knowledge
Object-oriented programming - CS core
Lots of polymorphism and multiple inheritance
Linux skills, shell scripting - CS370 (Operating Systems)

85. Knowledge
Object-oriented programming - CS core
Lots of polymorphism and multiple inheritance
Linux skills, shell scripting - CS370 (Operating Systems)
Python programming language

86. Knowledge
Object-oriented programming - CS core
Lots of polymorphism and multiple inheritance
Linux skills, shell scripting - CS370 (Operating Systems)
Python programming language
Event programming - PyMT/Pyglet and GTK+ (to build
test interface to window manager) event systems

87. Knowledge
Object-oriented programming - CS core
Lots of polymorphism and multiple inheritance
Linux skills, shell scripting - CS370 (Operating Systems)
Python programming language
Event programming - PyMT/Pyglet and GTK+ (to build
test interface to window manager) event systems
Open-source dev techniques - version control, etc.

88. Advice

89. Advice
Hardware always takes time

90. Advice
Hardware always takes time
Make bold decisions - be wary of the paradox of
choice

91. Advice
Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - git and github.com

92. Advice
Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - git and github.com
Build on open platforms when possible

93. Advice
Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - git and github.com
Build on open platforms when possible
the “shoulders of giants” principle

94. Difficulties and scope
control...

95. Difficulties and scope
control...
Pile interaction there but disabled for demo

96. Difficulties and scope
control...
Pile interaction there but disabled for demo
Auto-clustering algorithms designed but not coded

97. Difficulties and scope
control...
Pile interaction there but disabled for demo
Auto-clustering algorithms designed but not coded
Ordering and shipping delays slowed construction

98. Difficulties and scope
control...
Pile interaction there but disabled for demo
Auto-clustering algorithms designed but not coded
Ordering and shipping delays slowed construction
Transportation to demos led to hardware damage

99. Extensions

100. Extensions
Implement other gestures planned and more

101. Extensions
Implement other gestures planned and more
User interaction/efficiency studies

102. Extensions
Implement other gestures planned and more
User interaction/efficiency studies
Add launcher functionality

103. Acknowledgements

104. Acknowledgements
It takes a college, a city, and an Internet to
create this technology...

105. Acknowledgements:
Presentation

106. Acknowledgements:
Presentation
3d images of multi-touch techniques: Nolan, from PeauProductions.org
and NUI Group Forums
Schematics and samples for techniques: Seth (cerupcat) from NUI
Group Forums
Touch gesture images, from the Designing Gestural Interfaces book by
Dan Saffer, drawings by Rachel Glaves:
DesigningGesturalInterfaces.com

107. Acknowledgements:
Project Support

108. Acknowledgements:
Project Support
SNC IT and Tech Support
Mr. John Friedrich, Division Manager at Midland Plastics Inc. -
Contributed acrylic sheets and materials expertise
Dr. Michael Olson (Physics) - Laser information, lasers and safety
equipment, laser assembly space
SNC Computer Science Department - hard drive for Linux on test
machine, camera for IR sensing conversion, construction and
management advice, assembly space
My family - construction assistance

109. Acknowledgements -
Software

110. Acknowledgements -
Software
Ubuntu Linux and GNOME

111. Acknowledgements -
Software
Ubuntu Linux and GNOME
Python

112. Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management

113. Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam adjustments

114. Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam adjustments
NUI Group’s tBeta (using OpenCV) for blob tracking

115. Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam adjustments
NUI Group’s tBeta (using OpenCV) for blob tracking
PyMT for interface development

116. The Tactus Project
Hope you enjoyed this walkthrough!

117. The Tactus Project
Hope you enjoyed this walkthrough!