Build Your Own Multi-Touch Interface with Java and JavaFX Technology Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127

1. Build Your Own Multi-Touch Interface with
Java and JavaFX Technology
Simon Ritter, Sun Microsystems
Angela Caicedo, Sun Microsystems
TS-6127

2. TM
Learn how to build a Java technology-powered
touch screen that recognises multiple touch
TM
points. Also see how easy it is to use JavaFX
technology to build user interfaces that work
with multi-touch input.
2008 JavaOneSM Conference | java.sun.com/javaone | 2

3. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 3

4. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 4

5. Touch Screen Basics
Several different approaches
• Special surface coating - capacitive, resistive
• Surface acoustic wave, acoustic pulse
• Optical – Frustrated total internal reflection
Most screens only respond to single touch point
• Touch replaces use of mouse
Multi-touch is becoming more popular
• Apple iPhone
• Samsung multi-touch display
• Microsoft table
2008 JavaOneSM Conference | java.sun.com/javaone | 5

6. Frustrated Total Internal Reflection
Similar concept to fibre optic cable
Touching screen causes IR to appear where touched
Finger
Acrylic
Sheet
Infra-Red
Source (LED)
Screen
Web-cam Projector
2008 JavaOneSM Conference | java.sun.com/javaone | 6

7. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 7

8. Touch Surface
Thick acrylic sheet
• We used 6mm thick, 50x40cm
Edges need to be polished
We used a three stage process
• Back of a hacksaw blade
• Very fine sandpaper
• Car polish
Screen for display needs to be separate
• Coating the underside of the touch surface was a complete disaster
• Used a separate 3mm thick acrylic sheet covered in tracing paper
2008 JavaOneSM Conference | java.sun.com/javaone | 8

9. Infra-Red Source/Detection
Infra-red LEDs mounted along edge of screen
• Initially mounted one LED every 7mm
• This was reduced to one LED every 3cm (lower power consumption)
Webcam mounted under screen to observe touch points
• Webcams can see infra-red, unlike humans
• Most webcams have an infra-red filter which must be removed
• This can be difficult (we destroyed several webcams!)
• Replace this with a filter that only lets IR through
• Use the filter from a remote control
• Tricky bit is getting the webcam far enough away to see whole screen
• Wide angle lens makes this easier
2008 JavaOneSM Conference | java.sun.com/javaone | 9

10. Infra-Red LED Wiring
LEDs wired in parallel
100 Ohm resistor for each LED
• Good, bright LED
• Not too much power consumption
• Not too hot
2008 JavaOneSM Conference | java.sun.com/javaone | 10

11. Infra-Red LED Mounting
Each LED machined flat
Better optical connectivity to touch surface
2008 JavaOneSM Conference | java.sun.com/javaone | 11

12. Touch Surface
IR LEDs
2008 JavaOneSM Conference | java.sun.com/javaone | 12

13. Touch Surface with Screen
IR LEDs
Display screen
Touch surface
2008 JavaOneSM Conference | java.sun.com/javaone | 13

14. Webcam
Without filter With IR bandpass filter
2008 JavaOneSM Conference | java.sun.com/javaone | 14

15. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 15

16. Getting an Image From a Webcam
Java Media Framework (JMF) API
• The forgotten Java API, last update 2003
Version 1 only for playback, version 2 introduced capture
Still works well for SolarisTM (SPARC®) technology, Linux and
Windows
• Pure Java technology version available
• Performance packs for specific OS
• Decision was to use Ubuntu Linux for driver support
Performance was not an issue
• Able to grab and process images at suitably high frame rate
2008 JavaOneSM Conference | java.sun.com/javaone | 16

17. JMF API Frame Grabbing
Locate device with CaptureDataManager
Get DataSource through MediaLocator
Set Format
Create and Realize Processor
Create a PushBufferDataSource
• We really want a PullBufferDataSource, but can't have one
Get BufferStream through DataSource
Read Buffer
Convert Buffer to BufferedImage with
BufferToImage
Convert BufferedImage to RGB array with
PixelGrabber
2008 JavaOneSM Conference | java.sun.com/javaone | 17

18. Image Processing: Stage 1
Image from camera was not good for processing
• Camera had automatic adjustment of white balance, contrast, etc.
Use webcam device driver ioctls
• Turn off automatic adjustment
• Retrieve current settings
• Manually change settings
Driver needed modification
• Yay open source!
2008 JavaOneSM Conference | java.sun.com/javaone | 18

19. Image Processing: Stage 2
Java 2DTM and Java Advanced Imaging API
Convert to 8-bit greyscale for simplified processing
• ColorConvolveOp
BufferedImage is really useful for this kind of processing
BufferedImage
ColorModel Raster
SampleModel
ColorSpace
DataBuffer
2008 JavaOneSM Conference | java.sun.com/javaone | 19

20. Brightness and Contrast Adjustment
Filtered Pixel Values
Window / 2
Window
Level
Image Pixel Values
Create a ByteLookupTable
Apply LookupOp filter
LookupOp lop = new LookupOp(byteTable, null);
filteredImage = lop.filter(oldImage, null);
2008 JavaOneSM Conference | java.sun.com/javaone | 20

21. Image Processing: Stage 3
Identify where bright spots are in image
Algorithm is basically simple
• Although certain cases make life a bit harder
Ensure image is mono-chrome, each pixel is white or black
• 8-bit greyscale -> 1 bit mono-chrome
• Tunable level for change from black to white (0-255)
Sum pixels in each row and column (white = 1, black = 0)
Rows or columns with high values indicate touch point
Generate a set of x, y co-ordinates for all touch points
2008 JavaOneSM Conference | java.sun.com/javaone | 21

22. Image Processing: Stage 3
2
3
3 y point 1
3
3
y point 2
3
x point 1 242 333 x point 2
Points can be reversed
Test possible locations to get accurate result
2008 JavaOneSM Conference | java.sun.com/javaone | 22

23. Generating Touch Events
We now have a set of touch point co-ordinates
Simple events are somewhat like mouse events
• Change of position
• New touch point
• Touch point disappeared
Touch point code is more complex than mouse
• Must track points and figure out when a new one appears
• Or an existing one disappears
Gesture recognition is even more complex
• Swipe movement
• expand/shrink/rotate using two points
• Requires time-based analysis
We defined an extendable touch event interface
2008 JavaOneSM Conference | java.sun.com/javaone | 23

24. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 24

25. JavaFX Technology Basics
Programming Language for the Java Platform
Object-oriented
Declarative Syntax
Statically-typed with type-inference
Automatic data binding
Extensive Widget library encompassing Swing and Java 2D API
Development tools including NetBeansTM and Eclipse IDE
plugins
2008 JavaOneSM Conference | java.sun.com/javaone | 25

26. Java/JavaFX Technology and Multi-Touch
Software Integration
Why JavaFX technology:
• Great for image manipulation
• Data binding simplifies the software implementation
• Simple and easy to design and implement the user interface
Why Java technology:
• Heavy multi-threading implemented in Java technology
• Java based model bound to the interface with JavaFX technology
capabilities
• Model change <-> view change
Why Multi-touch:
• Intuitive user interaction
• Multiple point of interactions
• Allows multiple users interacting with the software
• Cool and interesting technology
2008 JavaOneSM Conference | java.sun.com/javaone | 26

27. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 27

28. Multi-Touch Events
They capture:
• Initial position (finger down)
• Direction (finger trajectory)
• Speed (finger speed)
• Final position (finger up)
Dynamically associated with SmartObject
Dynamically released from SmartObjects
Universe vs. individual interaction
Direction = 0 and speed = 0 then locking events
Speed > threshold then animation triggered
2008 JavaOneSM Conference | java.sun.com/javaone | 28

29. Smart Objects
Register multi-touch events
Object's behavior based on multi-touch events and areas of
interactions
rotating & rotating, rotating &
stretching stretching & stretching
flipping
rotating, rotating,
stretching & rotating, stretching &
flipping flipping
rotating & rotating, rotating &
stretching stretching & stretching
flipping
2008 JavaOneSM Conference | java.sun.com/javaone | 29

30. Stretching with Locking Action
2008 JavaOneSM Conference | java.sun.com/javaone | 30

31. Both Sides Stretching
2008 JavaOneSM Conference | java.sun.com/javaone | 31

32. Flipping
2008 JavaOneSM Conference | java.sun.com/javaone | 32

33. Locking and Rotating
2008 JavaOneSM Conference | java.sun.com/javaone | 33

34. Free Rotation
2008 JavaOneSM Conference | java.sun.com/javaone | 34

35. Putting Things Together
2008 JavaOneSM Conference | java.sun.com/javaone | 35

36. Universe Interaction (1 of 2)
2008 JavaOneSM Conference | java.sun.com/javaone | 36

37. Universe Interaction (2 of 2)
2008 JavaOneSM Conference | java.sun.com/javaone | 37

38. Smart Objects Still Active (1 of 2)
2008 JavaOneSM Conference | java.sun.com/javaone | 38

39. Smart Objects Still Active (2 of 2)
2008 JavaOneSM Conference | java.sun.com/javaone | 39

40. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 40

41. Where Next?
Infra-red is very useful as we can't see it
• No interference with rest of display
Wiimote from Nintendo has IR detector
• Can track up to four points simultaneously
• Reports positions via bluetooth interface
• Used to figure out where the Wiimote is relative to the display
Why not keep the Wiimote stationary, move the IR?
Great examples of this
• Johnny Chung Lee of CMU
• 3D position interpolation
Working on combining this with touch screen
• Using wiiremotej open source project
2008 JavaOneSM Conference | java.sun.com/javaone | 41

42. Agenda
Basic ideas of the multi-touch display
Construction of the display
Using Java technology to recognise touch points
JavaFX technology integration with multi-touch events
JavaFX technology multi-touch user interfaces
Future directions
Summary and further information
2008 JavaOneSM Conference | java.sun.com/javaone | 42

43. Summary
Multi-touch screens are simple and cheap to build
• Most expensive bit is the projector
Java technology makes the software easy
JavaFX technology is easy to integrate with new
types of event
JavaFX technology makes building multi-touch user
interfaces simple
This is just the beginning...
2008 JavaOneSM Conference | java.sun.com/javaone | 43

44. For More Information
Java Media Framework
• java.sun.com/products/java-media/jmf/
Java Advanced Imaging
• java.sun.com/javase/technologies/desktop/media/jai/
JavaFX technology
• openjfx.org
Jeff Hahn (FTIR multi-touch screen)
• cs.nyu.edu/~jhan/ftirtouch/
• www.perceptivepixel.com/
Building Imaging Applications With Java
Technology
• Lawrence H. Rodrigues
2008 JavaOneSM Conference | java.sun.com/javaone | 44

45. Multi-touch screen in action with
JavaFX technology
2008 JavaOneSM Conference | java.sun.com/javaone | 45

46. Simon Ritter, Sun Microsystems
Angela Caicedo, Sun Microsystems
TS-6127
2008 JavaOneSM Conference | java.sun.com/javaone | 46