The slides from my keynote address to the International Symposium on Ubiquitous Virtual Reality, 2010. Makes the case for image-based modelling as a tool for 3D user-created content, and argues that user-created content is critical to AR and ubiquitous AR particularly.

1. Image-based modelling for augmented reality Anton van den Hengel Director, Australian Centre for Visual technologies Professor, Adelaide University, South Australia Director, PunchCard Visual Technologies

2. 3D Modelling for AR AR needs models AR is about the interaction between the real and the synthetic 3D modelling isn’t much fun Even with the best interfaces invented 3D Studio Max? Blender?

3. User-created content 2D UCC has changed the face of the web Blogs, Wikis, Social networking sites, Advertising, Fanfiction, News Sites, Trip planners, Mobile Photos & Videos, Customer review sites, Forums, Experience and photo sharing sites, Audio, Video games, Maps and location systems and such, but more Associated Content, Atom.com, BatchBuzz.com, Brickfish, CreateDebate, Dailymotion, Deviant Art, Demotix, Digg, eBay, Eventful, Fark, Epinions, Facebook, Filemobile, Flickr, Forelinksters, Friends Reunited, GiantBomb, Helium.com, HubPages, InfoBarrel, iStockphoto, Justin.tv, JayCut, Mahalo, Metacafe, Mouthshut.com, MySpace, Newgrounds, Orkut, OpenStreetMap, Picasa, Photobucket, PhoneZoo, Revver, Scribd, Second Life, Shutterstock, Shvoong, Skyrock, Squidoo, TripAdvisor, The Politicus, TypePad, Twitter, Urban Dictionary, Veoh, Vimeo, Widgetbox, Wigix, Wikia, WikiMapia, Wikinvest, Wikipedia, Wix.com, WordPress, Yelp, YouTube, YoYoGames, Zooppa

4. User-created content for AR

5. Google-created content for AR

6. UCC for AR Just using images is a good start But limits interactions to 2D Flexible AR requires 3D models Ubiquitous AR requires UCC Flexible ubiquitous AR requires 3D UCC

7. 3D UCC 3D has been limited by the lack of good UCC tools This is true for AR But also VR, 3D TV, Second Life, Google Earth, Little Big Planet, 3D PDF, Adobe Premier, Unreal Tournament, Playstation, SGML, ...

8. 3D UCC AR particularly needs to model the real world Images are a good source of 3D information Easily accessible They’re typically captured anyway Almost everything has a camera attached Humans are very good at interpreting them Can AR be ubiquitous without UCC?

9. Image-based 3D UCC The image is the interface People can’t help but see images in 3D Most image sets embody 3D Powerful way to model real objects Varying levels of interaction Varying types of models Helps even in modelling imaginary objects

10. Image-based modelling for AR AR is largely about interactive images Any other mode of interaction adds complexity The majority of the content is real 3D modelling from images seems a natural fit with AR

11. Image-based 3D modelling Automatic Very detailed models of everything But it’s getting better Interactive Means you can specify What you want to model What kind of model you want

12. Videotrace Interactive image-based modelling A familiar interface Image-based interactions The image is the interface Generates low polygon count models with textures

13. Videotrace

14. Input

15. Modelling

16. Results

17. Another example

18. Modeling

19. Results

20. Interactive 3D modelling 3D modelling is critical to all sorts of application Special effects, but also mining, architecture, defence, urban planning, … People are getting more visually sophisticated More 3D data is being generated More cameras, but also scanners etc The interfaces of modelling programs are usually very hard to fathom

21. Low polygon-count models Insert your own objects into a game Model an environment for AR Put your house into Google Earth Video editing Cut and paste between sequences Remove someone from your home videos

22. Put your truck into a game

23. Put your truck into a game

24. Modelling for special effects

25. Video editing requires models

26. Video editing requires models

27. Modelling architecture

28. Modeling for virtual environments

29. Modeling for virtual environments

30. Modeling for virtual environments

31. Modeling for virtual environments

32. The process Capture and import the video Run video through the camera tracker Performs structure and motion analysis Interact with the system to generate and edit the model Export to your application

33. The approach Pre-compute where possible Structure from motion (camera tracking) Superpixels Then interact Interactions allow user to exploit precomputed results

34. Structure from motion Camera tracking Calculates Reconstructed point cloud Camera parameters Location Orientation Intrinsics (eg. Focal length) Informs interaction interpretation process

35. Structure from motion

36. Interactions Straight lines Closed sets of lines define planar polygons Curves For planar shapes with curved edges For NURBS surfaces Mirroring Duplicates existing geometry Extrusion Dense meshing

37. Fitting planar faces User specifies boundary Boundary specifies infinitely many planes Fitting similar to pre-emptive RANSAC Generate bounded plane hypotheses from point cloud Eliminate hypotheses that fail a series of tests Run simplest / most robust tests first Generally 3d tests before 2d tests

38. Line of sight Object points Fitting planar faces Image plane

39. Hierarchical RANSAC Generate bounded plane hypotheses Tests Support from point cloud Reprojects within new image boundaries Constraints on relative edge length and face size Colour histogram matching on faces Colour matching on edge projections Reprojection is not self-occluding

40. 2D Curves

41. 3D Curves

42. Mirroring

43. Extrusion

44. Dense surface reconstruction

45. Live modelling

46. Live modelling Most geometry cannot be modelled beforehand You can’t tell where it will be Modelling the whole world won’t work Need to generate models in-situ While you’re there

47. Live modelling in AR Using VideoTrace to model geometry from live video To insert elsewhere in the world So real objects can occlude synthetic geometry

48. Live modelling for AR The camera tracking is performed live using SLAM Simultaneous Localisation and mapping Markerless video tracking No prior model of the space Using PTAM Parallel Tracking and Mapping Klein and Murray

49. PTAM

50. Videotrace - Live

51. Occlusion Low polygon count models? Needed for efficiency Not accurate enough for occlusion calculations SLAM errors also prevent direct occlusion modelling

52. Occlusion boundary refinement The model of the foreground object is projected into the image Using the PTAM-estimated camera parameters But there is always some misalignment Solve using a live segmentation of the real object from the video

53. Occlusion boundary refinement Lay out nodes of a graph around the projected boundary Set foreground and background probabilities per node from colour model Set link weights from edge strength Segment using max-flow algorithm At frame rate

54. Occlusion boundary refinement

55. Occlusion boundary refinement Graph cut means that model doesn’t need to be accurate Very low polygon counts Very simple modelling process More complex objects possible

56. Occlusion boundary refinement Graph cut gives a hard segmentation Fix with an alpha matte Blends between foreground and synthetic object Fixes some holes in the cut

57. Live modelling for AR

58. AR modelling for other purposes

59. Minimal interaction AR modelling Use the camera as the modelling tool The user only specifies the object, the rest is done with the camera Projective texturing Some compensation for Visual Hull

60. Silhouette modelling

61. Minimal interaction modelling

62. How to get Videotrace It’s available on free beta test Just register at www.punchcard.com.au They will email you a link It’s a real beta Hopefully the final version will be free too

63. What’s next? New interactions, applications and data sources Interactive SFM, Better SLAM Videoshop