Requirements and Directions for AR Visualization (Keynote for PacificViz 2017)

1. REQUIREMENTS AND DIRECTIONS IN
AUGMENTED REALITY VISUALIZATION
GERARD JOUNGHYUN KIM
DIGITAL EXPERIENCE LABORATORY
KOREA UNIVERSITY

2. IN THIS TALK
• ARGUE AR AS AN INFORMATION VISUALIZATION TECHNIQUE
• DISCUSS SPECIAL REQUIREMENTS FOR AR VISUALIZATION
• SURVEY OF NOTABLE AR VISUALIZATION
• ADDRESS STANDING ISSUES AND ASSESS FUTURE DIRECTIONS

3. AUGMENTED REALITY (AR)?
• “AUGMENTED REALITY (AR) IS A FIELD OF COMPUTER RESEARCH
WHICH DEALS WITH THE COMBINATION OF REAL-WORLD AND
COMPUTER-GENERATED DATA.” – WIKIPEDIA.ORG
• AZUMA, R. T. (1997)
• COMBINES REAL AND VIRTUAL IMAGES
• INTERACTIVE IN REAL-TIME
• REGISTERED IN 3D

4. MOTIVATION
• VISUALIZATION OF DATA IS IMPORTANT
• FACILITATE DISCOVERY OF KNOWLEDGE
• AMPLIFY INTELLIGENCE
• AR VISUALIZATION HAS SPECIAL NEEDS
• DATA = REAL OBJECTS + AUGMENTATION

5. AR HAS SPECIAL FEATURES
• INSERTION OF SYNTHETIC AUGMENTATION INTO 3D REAL WORLDS
• 3D REAL WORLD IS DYNAMIC
• 3D REAL WORLD OBJECTS CANNOT BE EASILY REARRANGED
• TARGET REAL WORLD OBJECTS OFTEN HAVE SPATIAL CONTEXTS

6. GENERAL REQUIREMENTS FOR INFO. VIZ.
[MOERE AND PURCHASE]
• UTILITY – EFFECTIVE CONVEYING OF INFORMATION
• EASILY PERCEPTIBLE  MATCHING CAPABILITY OF HUMAN
• SOUNDNESS – RELIABILITY AND QUALITY OF THE VISUALIZATION
• ATTRACTIVENESS – APPEAL AND BEAUTY

7. VIZ. ISSUES IN AR
Characteristics of AR How it might affect the “utility” of visualization
Data insertion into 3D world  Resulting augmented imagery might not be natural
 Does not look 3D
 Does not look to be in the right place
 Does not have the right affordance or metaphor
 Registration error is too large
 Target data may be occluded
 Augmentation may be occluded
Dynamic background  Augmentation may not be clearly visible
 Augmentation may disappear
Unchangeable target data
(i.e. real objects fixed in place)
 Difficult to find proper place for the augmentation
 Difficult to represent structure of the collective target data
 Difficult to determine the proper size of augmentation
Target data with spatial contexts  Augmentation may occlude the context (neighboring objects)

8. PROPOSED REQUIREMENT 1: NATURALNESS
• THE USER IS NOT PUZZLED ABOUT THE IMAGERY - IT IS NOT TOO MUCH DEVIANT
FROM THE MENTAL MODEL OF THE USER’S UNDERSTANDING OF THE WORLD
• 3D ALIGNMENT: AUGMENTATION SHOULD BE REGISTERED INTO THE 3D SPACE IN A
“PHYSICALLY PLAUSIBLE” WAY AND/OR PROVIDE DEPTH CUES
• REALISTIC – THE AUGMENTATION SHOULD LOOK AS REALISTIC AND HARMONIOUS
WITH THE REAL WORLD AS POSSIBLE
Assumption: Spatially Registered AR

9. Transparent overlay of 3D
structures
[Azuma et al.]
3D cutting planes [Navab et
al.]
Phantom objects for physical plausibility/occlusion
(3D model of real objects)
[Kalkofen et al.]
(consider shadows, size, motion, ground, …)
3D DEPTH CUES

10. Supan et al.
VISUAL REALISM / LIGHTING
Rhee et al.

11. PHYSICAL PLAUSIBILITY
vs.
vs.vs.
movieAvatar adaptation to remote environment [Jo and Kim et al.] Floating Pokemons
Conspicuous but unnatural? Natural but not so outstanding

12. PROPOSED REQUIREMENT 2: VISIBILITY
• THE VISIBILITY OF INFORMATION SHOULD BE MAXIMIZED
• THERE ARE THREE TYPES OF INFORMATION TO CONSIDER
• THERE CAN BE TRADE-OFFS BETWEEN MAXIMIZING VISIBILITIES AMONG THE THREE
Augmentation
Target object
Neighboring
context
Assumption: Spatially Registered AR

13. TARGET OBJECT (ENHANCEMENT)
Emphasis with edges
[Avery et al.]
Motion profile
[Milgram et al.]
Filtering
[Tanzgem et al.]

14. CLEAR AUGMENTATION
SALIENCY BASED METHOD [AHN AND KIM]
Saliency map Saliency map
Adjustment
Simplified/Real time
and Mobile

15. SALIENCY BASED METHOD [AHN AND KIM]
No adjustment Simplified real time
Ground truth - local Ground truth - global

16. MULTIMODALITY [KIM AND KIM]
Visual, Aural, Visual + Aural ?
Fast paced guidance
movie

17. AUGMENTATION AND TARGET OBJECT
Saliency of occlude
and occludee
[Sandor et al.]
Transparency adjusted
by saliency
[Zollman et al.]
Is it possible to make both target and augmentation clearly visible at the same time?

18. CONTEXT / NEIGHBORHOOD
Warp to show more
Sandor et al.
Remove to show more
Inami et al.

19. CONTEXT / NEIGHBORHOOD
Organize to show more
[Choi and Kim]
Crowded tag problem
Associate Geo-tags to a group
By location/direction similarity

20. PROPOSED REQUIREMENT 3: STABILITY AND PERSISTENCE
• AUGMENTATION SHOULD BE STABLE AND COMFORTABLE TO VIEW/INTERACT
• ONCE AUGMENTED IN CERTAIN FORM, IT SHOULD NOT DISAPPEAR NOR
CHANGE WITHOUT ANY MEANINGFUL REASON

21. Motion stabilization
[Azuma et al.]
Handle blur
[Tyoura et al.]
Handle occlusion
[Percheim et al.]

22. Handle surface variations
[Bimber et al.]

23. GOOGLE GLASS STUDY [YU AND KIM]
Torsion, Dual focus and Fatigue
movie

24. OTHER: WHO DELIVERS WHERE? [JO AND KIM]
2D video
Character avatar in VR Character avatar in AR
Photorealistic character in AR
Photorealistic character in VR
movie

25. HUMAN ROBOT INTERACTION [AHN AND KIM]
Robot vs. Avatar
Near vs. Away
from the target

26. SUMMARY AND DIRECTIONS
• SPECIAL NEEDS FOR AR VISUALIZATION
• THREE REQUIREMENTS: NATURALNESS / VISIBILITY / STABILITY (COMFORT)
• REVIEWED SOME NOTABLE WORKS
• REMAINING ISSUES
• OCCLUDER (AUGMENTATION) AND/OR OCCLUDE (TARGET)
• WHERE TO PUT THE AUGMENTATION WITH RESPECT TO TARGET
• VISIBILITY VS. NATURALNESS
• MAYBE WE CAN REARRANGE THE REAL? (DIMINISHED REALITY)
• METRICS FOR THE THREE REQUIREMENTS
• DISPLAY ERGONOMICS
• COMBINE TRADITIONAL INFO VIZ. TECHNIQUES

27. THANK YOU
DXP.KOREA.AC.KR
GJKIM@KOREA.AC.KR