Workshop Winds/UFSCar July,2010 Master Degree from Maximiliano Marques

1. Project - Multi-touch surface with tangible objects and augmented reality support Student:MaximilianoSelmi Marques WINDIS Lab( former LRVNet Group ) - UFSCar July/ 2010

2. Objectives Hardware and Software Prototype Development Table-Based Interface Multimedia Collaborative Multiuser Tangible Augmented Reality Ready Case study: Emergency Management and Unmanned Aerial Vehicle (UAV) Other applications: entertainment, games, monitoring systems, control panel March 2010 Maximiliano Selmi Marques 2

3. State of the Art The ReacTable( MTG @ UPF – Barcelona, 2005 ) Tangible Electro-Acoustic Music Instrument Multi-touch & Multiuser (Collaborative) Computer Vision Based Solution March 2010 Maximiliano Selmi Marques 3

4. Surface Architeture Requisites Multiuser Intuitive Agile Precise Multimedia Mobile (somewhat) Paradigms Shareable Tangible Multimedia Gestural Augmented Reality Mobile March 2010 Maximiliano Selmi Marques 4

6. Plastic Film for protection

7. Paper Film for projection and light diffuser surface

8. Silicone Rubber for total internal reflection

9. Plexiglas with IR led frame

10. Video Camera

11. With IR Bandpass Filter

12. Without IR Cut-off Filter

13. Rear Video Projector

14. IR Spotlight for pattern illuminationMarch 2010 Maximiliano Selmi Marques 5

15. Hardware Prototype March 2010 Maximiliano Selmi Marques 6

16. Software Prototype Computer Vision System Touch pattern detection Pattern detection for tangible objects with orientation and position Calibration between camera and projector Background and noise subtraction Multimedia & Graphic User Interface (GUI) Gesture Capture Use of augmented reality to surpass tangible interface common limitations Expansible and modular architecture Open source libraries and toolkits to pattern detection and tracking TUIO Standard approach Protocol API Extended XML Like Standard to describe visual and actions components (ie. XUL, XAML and MXML) March 2010 Maximiliano Selmi Marques 7

17. Application Features Security and permissions polices Handle of external events Dispatch user actions Prioritize external events and user actions Resources loading and management Network connection management March 2010 Maximiliano Selmi Marques 8

18. Framework Features Tangible object registry Handle internal events Tracking user touch and tangible objects Controller Definition Graphic Component Definition Let application easily assign controllers to graphics objects Bind user and action (if possible) User workspace delimitation Visual Layers implementation Replace real tangible objects to AR Virtual Objects Render AR point of view and layers per user Handle of multimedia content Link with another device to expand surface area March 2010 Maximiliano Selmi Marques 9

19. March 2010 Maximiliano Selmi Marques 10 System Architecture

20. Case Study– Mobile Map Collaborative Control Panel Control of Unmanned Aerial Vehicle Send commands to UAVs Recreate cockpit information Show acquired data Draw routes, position and estimated time of arrival (ETA) Check user permission and polices March 2010 Maximiliano Selmi Marques 11

21. Reduced ScopeWhat is IN Single Map One large image with geo reference Waypoints Creation; Edition; Deletion Position & altitude Path Fixed to two paths One objective: Go from first Waypoint to last one Audio Feedback Simple sounds UAV simple movement to follow the path Fixed to two UAV Minimum HUD Info (altitude, speed, heading) AR with Wiimote & iPhone WLAN & Bluetooth iPhone render only Few Tangible Tokens PAN & Zoon Waypoint manipulation (x2) UAV command (with menu) March 2010 Maximiliano Selmi Marques 12

22. Reduced ScopeWhat is Out Online Map Integration Server Simulation Unlimited paths GUI manager Security User control Gesture recognition March 2010 Maximiliano Selmi Marques 13