Massive volumetric models on light field displays

www.crs4.it/vic/

Technologies for improving real-time
exploration of massive (volumetric)
volumetric)
models
Marco Agus
[magus@crs4.it]
CRS4
Visual Computing

Cagliari, October 2012

M. Agus, Technologies for improving massive models understanding, October 2012

We focus on…….
• Advanced technologies for massive models
rendering on light field displays
– Joint work with J. A. Iglesias Guitian, E. Gobbetti, F.
Marton, F. Bettio, G. Pintore, R. Pintus, A. Zorcolo


Technology pillars

• Software: Scalable
techniques for interactive
rendering of massive
models
– State of the art methods able
to handle potentially infinite
static surface and volume
models
• Hardware: Autostereoscopic
light field displays
– They are now able to render
the perceptual aura of real 3D
objects
– They provide real compelling
3D experience without glasses


Research challenges
• Light field display driving
– How to project geometry on such kind of systems?
• Image representation
– How to create effective light fields from 3D massive data?
– How to deal with volumes, surfaces, video streams?
• Evaluation
– Do light field displays improve perceptive cues and immersiveness?
– How to improve visual confort and depth discrimination?
• Interaction
– How to exploit light field displays to create natural interaction
systems?
– How to exploit display characteristics to provide meaningful
information?
• Applications
– How these kind of displays can be useful?


Main Contributions
Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, Fabio Marton, and Giovanni Pintore.
GPU Accelerated Direct Volume Rendering on an Interactive Light Field Display.
Computer Graphics Forum, 27(3): 231-240, 2008. Proc. Eurographics 2008.

Marco Agus, Fabio Bettio, Andrea Giachetti, Enrico Gobbetti, José Guitián, Fabio Marton, Jonas Nilsson, and Giovanni Pintore.
An interactive 3D medical visualization system based on a light field display.
The Visual Computer, 25(9): 883-893, 2009

José Antonio Iglesias Guitián, Enrico Gobbetti, and Fabio Marton.
View-dependent Exploration of Massive Volumetric Models on Large Scale Light Field Displays.
The Visual Computer, 26(6--8): 1037-1047, 2010

Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and Fabio Marton.
Evaluating layout discrimination capabilities of continuous and discrete automultiscopic displays.
In Proc. Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2010.

Fabio Marton, Enrico Gobbetti, Fabio Bettio, José Antonio Iglesias Guitián, and Ruggero Pintus.
A Real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field display.
In Proc. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 2010

Marco Agus, Giovanni Pintore, Fabio Marton, Enrico Gobbetti, and Antonio Zorcolo.
Visual enhancements for improved interactive rendering on light field displays.
In Eurographics Italian Chapter Conference, November 2011.

Fabio Marton, Marco Agus, Enrico Gobbetti, Giovanni Pintore, and Marcos Balsa.
Natural exploration of 3D massive models on large-scale light field displays using the FOX proximal navigation technique
Computers & Graphics, 2012.


DATA
Main Contributions

VOLUMES


A Real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field display.
VIDEO
In Proc. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 2010 STREAMS

SURFACES


Main Contributions APPLICATIONS

The Visual Computer, 25(9): 883-893, 2009 MEDICAL


A Real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field display. 3D - TV
In Proc. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 2010


VIRTUAL
Computers & Graphics, 2012. MUSEUMS


Main Contributions HUMAN FACTORS

USER
PERFORMANCE
+
PERCEPTION
Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and Fabio Marton.
Evaluating layout discrimination capabilities of continuous and discrete automultiscopic displays.
In Proc. Fourth International Symposium on 3D Data Processing, Visualization and Transmission, 2010.


INTERACTION


Light-field display: overview
• The key feature characterizing
3D light field displays is
direction-selective light
emission

• 3D display hardware based on
commercially available
technology developed by
Holografika
http://www.holografika.com
– Specially arranged projector
array and a holographic screen
– Side mirrors increase the
available light beams count
– Each projector emits light
beams toward a subset of the
points of the holographic screen


Physical behavior
• In the horizontal
direction, selective light
transmission
Projector
• Vertically, the screen
scatters widely (diffuse
behavior)
• Results in homogeneous
Screen
light distribution and
continuous 3D view

Light field


Projection technique
• In practice, at any moment in time,
a given screen pixel has the same
color when viewed from all vertical
viewing angles

• In order to provide a full
perspective effect, the vertical
viewing angle must thus be shown.
We thus introduce a “virtual
observer”, fixing the viewer height
and distance from screen

• The resulting MCOP technique is
exact for all viewers at the same
distance from the screen and
height as the virtual observer. It
proves in practice to be a good
approximation for all viewing
positions in the display workspace.


Depth-dependent resolution

• The display design has
consequences not only on
the projection equation but
also imposes limits on the
spatial resolution that
depends on depth

• In general, the size of the
smallest feature that can be
reproduced depends on the
distance of its center from
the screen and from the
beam angular size

M. Agus, E. Gobbetti, J.A. Iglesias Guitián, F. Marton, and G. Pintore.
Computer Graphics Forum, 27(3), 2008. Proc. Eurographics 2008.


Rendering System overview
• Full-system architecture overview


Parallel rendering
• We employ a GPU cluster for rendering.
• Sort first parallel rendering approach
– Adaptive out-of-core GPU rendering vs Replicating data
– Static assignment: rendering process images.
• Good load balancing. Caused by the geometry of the display,
with all projectors looking at the same portion of the model.


Light field displays considered
• Two light field displays are considered:
– Large-scale model capable of visualizing
35Mpixels by composing images generated
by 72 SVGA LED projectors.
• The screen has 160x90cm, 50◦ horiz. field-of
view with 0.8◦ angular accuracy.
• Pixel size on the screen surface is 1.5mm.
• Rendering cluster: 18 Athlon64 3300 + Linux
PCs equipped with two NVidia 8800GTS 640MB.
• GPUs are several generations older. Slowdown
of 2x−3x
– Small-scale model capable of visualizing
7Mpixels by composing images generated by
96 320x240 small CCDs
• The screen is 26 inches, 50 degrees horiz. field-
of view with 0.8◦ angular accuracy.
• The display is driven by one linux PC equipped
with one NVidia 8800GTS 640MB.


Massive volume rendering


Massive volume rendering

APPLICATION:
Massive volume
rendering

View-dependent Exploration of Massive Volumetric Models
on Large Scale Light Field Displays.


Technique overview
• Use CPU for …
– Creation & loading
– Octree refinement
– Encode current cut using an
spatial index

• Use GPU for …
– Stackless octree traversal

Enrico Gobbetti, Fabio Marton, and José Antonio Iglesias Guitián.
A single-pass GPU ray casting framework for interactive
out-of-core rendering of massive volumetric datasets.
– Rendering The Visual Computer, 24, 2008. Proc. CGI 2008.


Taking advantage of GPGPU

• Improved multi-resolution
CUDA ray caster:
– Flexible ray traversal and
compositing strategies
– Improved visibility feedback (e.g.
scatter writes)
– Pre-integrated transfer-functions
– Integrate an adaptive frame
reconstruction scheme


Method overview
[ creation and maintainance ] [ rendering ]

preprocessing
adaptive loader
visibility
offline
octree refinement
feedback
has current working set
no
enough accuracy?
storage
yes
octree node volume
database prepare to render
render

CPU GPU


Preview: massive volumes rendering


Massive surface rendering


Massive surface rendering

APPLICATION:
Virtual exploration
of massive
surface models

Fabio Marton, Marco Agus, Enrico Gobbetti, Giovanni Pintore,
and Marcos Balsa Rodriguez.
Natural exploration of 3D massive models on large-scale light
field displays using the FOX proximal navigation technique.


Handling massive models: overview

• Multiresolution structure based on
a modification of Adaptive Tetra
Puzzles
– Few Ktri Patches
– Optimized GPU/CPU communication
– Per-patch spatial index to organize
patches in triangle strip
Paolo Cignoni, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton,
Federico Ponchio, and Roberto Scopigno.
Adaptive TetraPuzzles - Efficient Out-of-core Construction
and Visualization of Gigantic Polygonal Models.
ACM Transactions on Graphics, 23(3): 796-803, August 2004.
Proc. SIGGRAPH 2004.

Giovanni Pintore, Enrico Gobbetti, Fabio Marton, Russell Turner,
and Roberto Combet.
An Application of Multiresolution Massive Surface
Representations to the Simulation of Asteroid Missions.
In Eurographics Italian Chapter Conference. Pages 9-16.
Eurographics Association, November 2010.


Handling massive models
• Offline construction
– spatial partition by longest
edge bisection of tetrahedra
– fine-to-coarse parallel out-
of-core simplification of the
surface contained in
diamonds
• Run-time rendering
– selective refinement queries
based on projected space
error estimation on
tetrahedron hierarchy
– adaptive loader rapidly
produces LOD by combining
precomputed patches


Results: massive models rendering


Multiview capture and rendering


Multiview capture and rendering

APPLICATION:
Multiview capture
and rendering
system

Fabio Marton, Enrico Gobbetti, Fabio Bettio, José Guitián, and
Ruggero Pintus.
A Real-time coarse-to-fine multiview capture system for
all-in-focus rendering on a light-field display.
In Proc. 3DTV Conference: The True Vision - Capture,
Transmission and Display of 3D Video (3DTV-CON). 2011


Capture

• Simple front-end: a single PC acquires
M-PEG video stream from low cost USB
camera array.

• Each node need to receive all camera
images since each projectors sees a
large portion of the display workspace

• Solution: front-end sends single-frame
JPEG images through UDP multicast
protocol to display cluster nodes

29


Upload to GPU

• Minimization of CPU work
– Moreover, old PCs not fast enough for JPG decoding.
• Pipelined CPU-GPU parallel decoding, interleaving:
– CPU:entropy decoding using libjpeg-turbo
– GPU: CUDA
CPU
• dequantization, Img 0
• inverse-DCT,
CPU GPU
• YCbCr to RGB conversion Img 1 Img 0
– 6X speed-up wrt standard CPU solution
– Fast but still dominates the whole processing
time using old G80 GPUs!
CPU GPU
Img N-1 Img N-2

GPU
Img N-1

30


Depth estimation (CUDA)
• For each pixel of each projector
– Find depth and color
– identify a ray using equations which
consider MCOP geometry of the display.
– identify the closest cameras to the ray
using the camera viewing transforms.
• Plane sweeping to find best depth
– Evaluate pixel similarities at different
depths and keep the best one

31


Coarse to fine depth estimation & color
sample
• Start at half depth with
coarser resolution
• Upsample and refine
• Filter each level
• Up to final resolution
• Final color gathering

32


Results: Multiview Capture System


Results: “First teleport experiment”


Illustrative methods


Illustrative methods

FOCUS:
Improve volume
understanding

View-dependent Exploration of Massive Volumetric Models
on Large Scale Light Field Displays.


Novel view-dependent illustrative
tools

• The view-dependent characteristics of the display can
be exploited to develop specialized interactive
illustrative techniques designed to improve spatial
understanding
• Simple head motions can reveal new aspects of the
inspected data


Clip-plane with view-dependent
context

• Traditional cut away visualization, when our view
direction is orthogonal to the clip plane, while
offering more helpful contextual information in
other situations


context
• Compute distance from plane

• If distance is positive
• modify the opacity of samples by multiplying it by a
view-dependent correction factor:

• Otherwise
• vary the opacity of the plane and shading parameters
from the original ones at to full opacity
and ambient plus emission shading at


context


Other view-dependent illustrative
tools
• Context-preserving
probe

• Band-picker

• More info in...

José Iglesias Guitián, Enrico Gobbetti and Fabio Marton.
View-dependent exploration of massive volumetric
models on large-scale light field displays.
The Visual Computer, 26, 2010. Proc. CGI 2010.


System overview


System overview

FOCUS:
Perceptual and
performance
evaluation

Marco Agus, Enrico Gobbetti, José Antonio Iglesias Guitián, and
Fabio Marton. Evaluating layout discrimination capabilities of
continuous and discrete automultiscopic displays.
In Proc. Fourth International Symposium on 3D Data Processing,
Visualization and Transmission, 2010.


Evaluation

• The main goal of tests
performed in the evaluation
process is …

– … to elucidate if light field displays
could provide visual information not
available with traditional volume
rendering systems

• The main focus will be set on
psychophysical tests.


Evaluation

• Stereopsis evaluation
– Random dot spiral ramp
– Depth discrimination
(overlapping disks)
• Spatial understanding
evaluation
– Path tracing
performance evaluation


Evaluation results
Users rapidly recover all
depth cues to
instantaneously recognize
complex structures
– Very useful for analysis of
angiography datasets

• More details about the evaluation tests can be found in …
M. Agus, F. Bettio, A. Giachetti, E. Gobbetti, J. A. Iglesias Guitián, F. Marton, J. Nilsson, and G. Pintore.
The Visual Computer, Vol. 25, No. 9, pp. 883–893, 2009.

M. Agus, E. Gobbetti, J. A. Iglesias Guitián, F. Marton. Evaluating layout discrimination capabilities of
continuous and discrete automultiscopic displays. 3DPVT, 2010


System overview

FOCUS:
Display adaptation

Marco Agus, Giovanni Pintore, Fabio Marton, Enrico Gobbetti,
and Antonio Zorcolo.
Visual enhancements for improved interactive rendering on
light field displays.
In Eurographics Italian Chapter Conference. Pages 1-7.
Eurographics Association, November 2011.


Problem: visual discomfort
• Aliasing artifacts occur
when objects are too far
with respect to the screen
• Causes
– Geometry errors due to the
discrete characteristics of
projectors and display
– Image-based calibration
error
• subpixel errors on the
display screen
• bigger errors as the distance
from screen increases


Visual discomfort: our solution
• Non-linear depth
remapping method
– constrains most of the scene
to stay inside CVR
– drives users to focus on
parts inside CVR
• Depth-of-field simulation
method
– blurs parts of the scene in
background
• Frame fade-out method
– reduces clipping artifacts due
to the borders of the light
field display


Non-linear depth remapping method
• Implemented on a vertex
shader according to the
equation

• B, F are the comfort
thresholds
• DF, DB are the asymptotic
output ranges


Depth of field simulation
• Depth-dependent blur to adapt the frequency
content to display resolution
– Focused objects are sharp within CVR, around focal plane
– Background objects are instead blurred
– Objects close to the viewer cannot be blurred
• Implemented as image-based two-pass DOF
simulation [Nguyen 2007]
– CoC proportional to display spatial resolution
– Post-processing pixel shader blending between hi-res and
low-res images according to CoC and blurriness by stochastic
Poisson sampling


Frame fade-out
• Limited angular workspace
– Objects can abruptly disappear while
viewer moves horizontally
• Simple but effective solution
– Color blending which fades to
background in boundary areas
– Objects smoothly fade out
– Implemented in a fragment shader


Qualitative results
• Non-linear depth-mapping (front positions)

Without Depth mapping Depth mapping
adaptation DF = 1500 mm DF = 500 mm


Qualitative results
• Non-linear depth-mapping + DOF blur

Depth mapping
Without Depth mapping DB = 1000 mm +
adaptation DB = 1000 mm DOF


Qualitative results

Depth mapping
Without
DF = 1000 mm
adaptation


System overview

FOCUS:
natural interaction

Fabio Marton, Marco Agus, Giovanni Pintore, and Enrico Gobbetti.
FOX: The Focus Sliding Surface Metaphor for Natural
Exploration of Massive Models on Large-scale Light Field
Displays.
In Proc. VRCAI. Pages 83-90, December 2011.


Natural interaction: requirements

• Light field display
constraints
– Depth-dependent
spatial resolution,
calibration errors,
angular bounds
• Interaction metaphor
should be simple
– Museum context
– Reduced number of
DOFs
– Short learning time


FOX interface: key ideas
• Guided motion
– Scene is kept centered wrt
display workspace Focus
• DOF reduction +
– Interaction consists of Visual confort
changing a contact point
– Translation, rotation and
zoom are coupled together
• Natural and intuitive
– Simple to learn: only one Ease of use
button +
– It can be implemented with Efficiency
various devices


FOX interface: components
• Translation and
rotation
• Automatic
zooming
• Automatic hotspot
placement


Translation and rotation

• Surface slides on display hotspot
– (dx,dy) defines a displacement on current plane
– Closest point and normal is forced to display hotspot
– Up direction is kept
– Smoothing filters


Automatic zooming
• Speed-dependent automatic zooming
– Vector length dS = |dx,dy| defines rate of motion
– Pan-speed function with three states
– Steady (FOCUS) vs fast motion (EXPLORATION)


Hot-spot automatic placement
• Coarse sampling for computing average
plane
• Hotspot depth corrected to put scene in
focus


Resulting interface
• From device (2D pos + button state) to displacement
vector dS
• Model matrix M function of dS and current state (model
surface)
• Easy to implement with all pointing devices (2D mouse,
3D mouse, free-hand motion-tracking, touch-screens)
• Cursor glyphs indicating dS and pan-zoom state


Kinect implementation
• Hand tracking with gesture recognition
(open/close)
– Covariance analysis in order to recognize singular
points


Results
• Real-time exploration of David 0.25 mm model,
composed of 970M triangles
• User evaluation:
– Metaphor comparison ( 5DOF ObjectInHand vs FOX)
– Device comparison (IS 9000 vs Kinect)


Interaction with IS device


Free-hand interaction (Kinect)


User evaluation
• 33 participants (26 novices, 7 experts)
• Quantitative evaluation
– FOX vs 5DOF, IS vs KINECT
– Guided and explorative tasks
– Task completion time and image quality
• Qualitative evaluation
– Metaphor comparison (ease of learning, ease of reaching
positions, perceived 3D image quality, preferred one)
– Device comparison (ease of learning, ease of reaching
positions, preferred one)


User evaluation: samples


Results: 3D image quality


Results: task completion time


Results: qualitative evaluation


Results: user preferences


Discussion
• Tasks are easier with FOX (especially for novice
users)
• Overall image quality is sensibly better with
FOX
• With respect to devices IS 900 performs better
(especially for novice users)
• FOX provides better comfort (even difference in
ease of use is not significant)


Summary of contributions
• Interactive system for virtual exploration of
massive models on light field displays
• Light field display adaptation techniques for
improving visual perception
• Natural interaction metaphor for light field
displays
• Perceptual and performance evaluation of light
field displays for interactive exploration


Current and future work
• But there is still a lot of work to do ...
– Improve light field representations
• Complex scenes (hybrid scenes, point-based
representations, image-based representations)
• Model retargeting within the range of 3d displays
– Improve the interaction techniques
• e.g. natural interfaces for the manipulation of
large models (gesture interfaces)
• Exploration of more complex virtual
environments (jumps, fly-through, change of
metaphors)


Take-home messages
• Data explosion Impossible to explore all data
you acquire
• Displays are evolving Many ways to improve
exploration of data and provide useful
information
• Complexity reduction is needed
– Interface Constrained but natural exploration
– Visual representation Extraction of meaningful information
+ Hints on exploratio + Integration and fusion


Take-home message
• Simple scene simple
exploration
– Interface can be simple
– User can explore all the
scene to reach her target


Take-home message
• Huge scene Simple or complex interface?
– Where is the Pac-man food?
– How much time would Pac-Man need to complete the maze?


That’s all, folks…
Thank you
Marco Agus
[magus@crs4.it]
CRS4
Visual Computing
http://www.crs4.it/vic
vic@crs4.it

Massive volumetric models on light field displays

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (16)

Similar to Massive volumetric models on light field displays

Similar to Massive volumetric models on light field displays (20)

More from CRS4 Research Center in Sardinia

More from CRS4 Research Center in Sardinia (20)

Recently uploaded

Recently uploaded (20)

Massive volumetric models on light field displays