A talk from the Develop Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017. Ryan Damm (Visby): Light Fields & Polygons: Making Sense of New Image Formats Augmented and Virtual Reality displays have tremendous potential, but to unlock their true power they require a revolution in imaging and visualization. Going beyond 2D images, these displays have to potential to leverage 3D 'volumetric' representations, as well as 4D light fields, to display visually richer scenes. This session will examine the range of imaging options today, including their strengths and weaknesses. We'll also cover the likely developments in the near future, and how these developments will impact different verticals. http://AugmentedWorldExpo.com

1. LIGHT FIELDS
& POLYGONS
Making Sense of New Image
Formats

2. WHAT WE’LL COVER: IMAGES
• Major image formats of the future
• Compare/contrast each
• Look at where they fit in
Join the conversation @ARealityEvent & #AWE2017

3. • No current standards
• Display hardware ahead of software /
imaging stack
• Standards bodies lagging
• Market will determine winner
• Quality / performance will (probably) win
WHY IMAGE FORMATS?
Join the conversation @ARealityEvent & #AWE2017

4. WHY IMAGE FORMATS?
Join the conversation @ARealityEvent & #AWE2017
• No current standards
• Display hardware ahead of software /
imaging stack
• Standards bodies lagging
• Market will determine winner
• Quality / performance will (probably) win

5. WHY IMAGE FORMATS?
Join the conversation @ARealityEvent & #AWE2017

6. WHY IMAGE FORMATS?
Join the conversation @ARealityEvent & #AWE2017
The Software Stack Needs Work

7. FORMATS
Let’s start with the
fundamentals.
Join the conversation @ARealityEvent & #AWE2017

8. • 360 Video (for VR)
• Volumetric (polygons)
FORMATS TODAY, TOMORROW
Join the conversation @ARealityEvent & #AWE2017
Today Tomorrow

9. • 360 Video
• Volumetric (polygons)
FORMATS TODAY, TOMORROW
Join the conversation @ARealityEvent & #AWE2017
• RGB+D
Today Tomorrow

10. • 360 Video
• Volumetric (polygons)
FORMATS TODAY, TOMORROW
Join the conversation @ARealityEvent & #AWE2017
• RGB+D
• Volumetric
Today Tomorrow

11. • 360 Video
• Volumetric (polygons)
FORMATS TODAY, TOMORROW
Join the conversation @ARealityEvent & #AWE2017
• RGB+D
• Volumetric
• Light Fields
Today Tomorrow

12. • As an image format: R, G, B,
Depth
• Depth carried along with pixel
value
• Often seen as acquisition /
intermediate format
RGB + DEPTH
Join the conversation @ARealityEvent & #AWE2017
x,y raster of
R,G,B,D values:
<255,128,16,128>

13. • Polygons + pixels
• Typically plays back in game
engine
• Lots of different capture /
translation techniques being
developed
VOLUMETRIC
Join the conversation @ARealityEvent & #AWE2017

14. • Polygons + pixels
• Typically plays back in game
engine
• Lots of different capture /
translation techniques being
developed
VOLUMETRIC
Join the conversation @ARealityEvent & #AWE2017
Mesh (vertex lists, etc) for shape
Textures (u,v maps) for color

15. • Represent light directly: location and
direction
• Subsample / shuffle rays for view synthesis
• Capture: lots of cameras
LIGHT FIELD
Join the conversation @ARealityEvent & #AWE2017

16. Join the conversation @ARealityEvent & #AWE2017
BUT WHAT ARE
THEY GOOD FOR?
Good, Bad, and Ugly.

17. • It Works!
• Extends existing standard
• Re-uses robust video tech stack
• Compression, delivery, decode
• Hardware accelerated, etc
RGB + DEPTH: Good
Join the conversation @ARealityEvent & #AWE2017

18. • Very small parallax
• No roundness*, specularity
Join the conversation @ARealityEvent & #AWE2017
*high-resolution, stereo 3D can fake this for small ranges
RGB + DEPTH: Bad

19. • Discrete depths
• Infill?
RGB + DEPTH: Ugly
Join the conversation @ARealityEvent & #AWE2017

20. • Discrete depths
• Infill?
RGB + DEPTH: Ugly
Join the conversation @ARealityEvent & #AWE2017

21. • It works!
• Re-uses robust tech stack
• Large movement space
• Allows for rich interactivity
VOLUMETRIC: The Good
Join the conversation @ARealityEvent & #AWE2017

22. • Capture is an open question
• Compression is tough
VOLUMETRIC: The Bad
Join the conversation @ARealityEvent & #AWE2017

23. VOLUMETRIC: The Ugly
Join the conversation @ARealityEvent & #AWE2017

24. VOLUMETRIC: The Ugly
Join the conversation @ARealityEvent & #AWE2017
(Also true of RGB+D, but masked by limited movement
range.)

25. • Photorealistic with 6DOF
• Potentially most accurate: light in, light out
LIGHT FIELD: Good
Join the conversation @ARealityEvent & #AWE2017

26. • Data!
• Movement range
defined by
cameras
• Aliasing, edges
LIGHT FIELD: Bad
Join the conversation @ARealityEvent & #AWE2017

27. • Doesn’t actually exist
LIGHT FIELD: Ugly
Join the conversation @ARealityEvent & #AWE2017

28. • Doesn’t actually exist (yet)
LIGHT FIELD: Ugly
Join the conversation @ARealityEvent & #AWE2017

29. SYNTHESIS: Polygons +
Light
Join the conversation @ARealityEvent & #AWE2017
• Hybrid approach: Polygons + light
• Light field + polygon clipping masks
• Solves interpolation problem
• ‘BRDF’ / “dynamic pixel map”

30. • “Virtual objects / real environment”
• Games / hybrid experiences
• Interactive experiences
Volumetric
SYNTHESIS: A Mix of
Formats?
Join the conversation @ARealityEvent & #AWE2017
• “Dynamic posters”
• Near-term format
• User-generated content, live-
streaming
RGB+D
• “Windows and crystal balls”
• Realistic experiences
• Transportation and immersion
Light Field

31. THANK YOU
For keeping it real.
Join the conversation @ARealityEvent & #AWE2017