2019年1月の合同研究会にて招待講演をした際のスライドを公開用に修正したものです。 Talk slides adapted from my invited talk slides at a domestic workshop: http://sigmr.vrsj.org/events/2019Jan.html https://www.ieice.org/ken/program/index.php?tgs_regid=32fc07ec511f6ac834c1588222cd2fe889910e6d32c48bce6730e4b59885b16d&lang= 20190114_SIG-MR_PRMU_CVIM_MVE_PoTS

1. Augmenting Your Vision
視覚拡張に向けた最近のnear-eyeディスプレイ研究の発展に関して
伊藤勇太
2019/1/17@PRMU-MVE-IPSJ-CVIM合同研究会
Augmented Vision Lab, Tokyo Institute of Technology

2. 伊藤勇太
研究テーマ：「世界を計算により賢く上書きする」
専門：光学透過型HMD・視覚拡張・拡張現実感（AR）
（主）東京工業大学 情報理工学院 テニュアトラック助教
（兼） JSTさきがけ研究者
（兼）理研AIPセンター客員研究員

3. ARは既にオワコンなのか？

4. AR
MR
? VR
VR(2017)

5. Augmented Reality (AR)
Feiner, Steven, Blair Macintyre, and Dorée Seligmann. "Knowledge-based
augmented reality." Communications of the ACM 36.7 (1993): 53-62.

6. VR HMDs Video
See-Through
Optical
See-Through
Opaque
display
Scene
camera
See-through
Head-Mounted Displays (HMD)

7. StereoScope (1838)
Wheatstone, Charles. "Contributions to the physiology of vision.--Part the first. On some remarkable,
and hitherto unobserved, phenomena of binocular vision." Philosophical transactions of the Royal
Society of London (1838): 371-394.

8. The Sword of Damocles (1968)
The ultimate display
IE Sutherland - Multimedia: From Wagner to virtual reality, 1965

9. The Sword of Damocles (1968)

10. At Computer History Museum in Mountain View, Mar. 2017

11. At Computer History Museum in Mountain View, Mar. 2017

12. Optical See-Through Head-Mounted Display
EPSON BT-300, 2016Microsoft HoloLens,
2016
ODG R-9, 2016 Meta Meta2, 2016 Magic Leap One, 2018
Vuzix Blade, 2018 Lumus DK Vision, 2018
Lumus DK52, 2018

13. (OST-)HMDの研究動向（2017の時）
• ガチガチの自作ハード
• Computational Photography

14. OST-HMDの研究動向（2018の時）
• Varifocal系
Stevens et al.
P Chakravarthula et al.
• 眼を矯正したり、映像を眼に合わせる
もの
K. Rathinavel et al.

15. HMDの研究動向（2018の時）
Deep Learningの波も。（RGBD→多画像をビデオレートで）
Xiao, L., Kaplanyan, A., Fix, A., Chapman, M., & Lanman, D. (2018, August). DeepFocus:
learned image synthesis for computational display. In ACM SIGGRAPH 2018 Talks (p. 4). ACM.

16. (Free-form)
prism
Point
source
Waveguide
Laser
source
Half-mirror
Eye
Prism-based
Waveguide-based
Half-mirror (or cubic prism)
Classification from: Hainich and Bimber, Displays Fundamentals and Applications, 2011
Retinal Scanning
Vuzix
EPSON
Lumus Microvision
HoloLens
NVIS
QD Laser
OST-HMD: Optics Designs (Commercial)

17. Coherent
light
Eye
Pinhole light array
(Pin-light)
Light field
(Microlens array)
Light field
(Stuck LCD)
Holography
UNC-CH &
NVIDIA
2013
Moon et al. 2014
Backlight
Multiple
LCDs
Eye
LCD
Pinlight
LCD
Back
light
Takaki & Yamaguchi
2015
Micro
Lens
Array
Spatial light
modulator
UNC-CH
& NVIDIA, 2014 Pinhole effect:
“focus free”
Maimone et al. 2017
OST-HMD: Optics Designs (Research)

18. Optical See-Through HMDs
Display virtual images into the real view

19. Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Perceptual
[Kiyokawa ’01] [Liu ’08]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
[Maimone ’14][Yamaguchi ’16][Fukiage ’14] [Kalkofen ’13]

20. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-kalibrierung
fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).
Spatial Realism in OST-HMDs

21. Spatial Realism in OST-HMDs

22. Eye-HMD Calibration
3D2D

23. Eye-HMD Calibration
3D

24. 眼球位置追跡による自動位置合わせ
HMDカメラで眼球を３D計測し、AR像を視界に自動位置合わせする
Eye tracker
Yuta Itoh, Gudrun Klinker: Interaction-free calibration for optical see-through head-mounted displays
based on 3D Eye localization. 3DUI 2014: 75-82

25. 視線追跡はDe facto Standardに？
HTC Vive Pro Eye @CES2019
https://japanese.engadget.com/2019/01/08/
htc-vive-pro-vr-vive-pro-eye/
Magic Leap One, 2018
https://jp.ifixit.com/Teardown/Magic+Le
ap+One+Teardown/112245

26. HMD角膜反射による眼球位置推定
Alexander Plopski, Yuta Itoh, Christian Nitschke, Kiyoshi Kiyokawa, Gudrun Klinker, Haruo Takemura:
Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays. IEEE Trans. Vis.
Comput. Graph. 21(4): 481-490 (2015)

27. 点光源による焦点非依存の広画角HMD
Pinlight Displays
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources” Maimone et al., TOG’14
Pinhole effect
 images are always in focus in theory

28. 点光源による焦点非依存の広画角HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources” Maimone et al., TOG’14

29. CES2019で似た光学系が
https://av.watch.impress.co.jp/img/avw/docs/1164/584/html/10letinar_o.jpg.html
LetinAR。ただし、光源はWaveguideで持ってくる。
https://letinar.com/product/

30. 輻輳（両眼視）と調節（目のピント）の矛盾をなくす
奥行きを自然に再現したディスプレイ
Dunn, D., Chakravarthula, P., Dong, Q., & Fuchs, H. (2018, May). Mitigating vergence-accommodation
conflict for near-eye displays via deformable beamsplitters. In Digital Optics for Immersive Displays (Vol.
10676, p. 106760U). International Society for Optics and Photonics.

31. 輻輳（両眼視）と調節（目のピント）の矛盾をなくす
奥行きを自然に再現したディスプレイ
Dunn, D., Chakravarthula, P., Dong, Q., & Fuchs, H. (2018, May). Mitigating vergence-accommodation
conflict for near-eye displays via deformable beamsplitters. In Digital Optics for Immersive Displays (Vol.
10676, p. 106760U). International Society for Optics and Photonics.

32. 視線に合わせて視力を矯正するメガネ
視力（近視や遠視）に合わせてメガネの焦点距離を自動調整
Padmanaban, N., Konrad, R., & Wetzstein, G. (2018, August). Autofocals: gaze-contingent
eyeglasses for presbyopes. In ACM SIGGRAPH 2018 Emerging Technologies (p. 3). ACM.

33. 視力矯正不要のARディスプレイ
視力（近視や遠視）に合わせて映像の可焦点距離を自動調整
Chakravarthula, P., Dunn, D., Akşit, K., & Fuchs, H. (2018). FocusAR: Auto-focus Augmented
Reality Eyeglasses for both Real World and Virtual Imagery. IEEE transactions on visualization
and computer graphics, 24(11), 2906-2916.

34. 眼の位置に投影像を動的に合わせる
Steerable Application-Adaptive Near Eye Displays
Rathinavel, K., Chakravarthula, P., Akşit, K., Spjut, J., Boudaoud, B., Whitted, T., ... & Fuchs, H. (2018,
August). Steerable application-adaptive near eye displays. In ACM SIGGRAPH 2018 Emerging
Technologies (p. 17). ACM.

35. 網膜での色収差を使う奥行き再現
Longitudinal chromatic aberration (LCA,縦色収差)を利用
Cholewiak, S. A., Love, G. D., Srinivasan, P. P., Ng, R., & Banks, M. S. (2017).
ChromaBlur: Rendering chromatic eye aberration improves accommodation and
realism. ACM Transactions on Graphics (TOG), 36(6), 210.

36. 網膜での色収差を使う奥行き再現
Longitudinal chromatic aberration (LCA,縦色収差)を利用

37. 視線追跡＋網膜投影
“Retinal 3D: Augmented Reality Near-Eye Display Via Pupil-Tracked Light Field
Projection on Retina,” Changwon Jang et al., SIGGRAPH Asia 2017

38. 視線追跡＋網膜投影

39. 最終手段→ホログラフィを使う
“Holographic Near-Eye Displays for Virtual and Augmented Reality,” Microsoft Research, Andrew
Maimone, Andreas Georgiou, Joel Kollin, Established: May 19, 2017
位相限定-反射型液晶素“Computational Lens”

40. VRディスプレイ版
Oculus Research , “Focal Surface Display Discovery”
Nathan Matsuda, Alexander Fix, Douglas Lanman, SIGGRAPH 2017

41. 簡単＆きれいな光学遮蔽
光学シースルーHMDにて、背景映像をうまく重畳して、遮蔽マス
クのぼやける輪郭をくっきり見せる。
Yuta Itoh, Takumi Hamasaki, Maki Sugimoto:
Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive
Spatial Light Modulator. IEEE Trans. Vis. Comput. Graph. 23(11): 2463-2473 (2017)

42. Occlusion
Additive（＋）image construction

43. 簡単＆きれいな光学遮蔽

44. ARは既にオワコンなのか？

45. 人間拡張

46. Vision
Augmentation/Modulation

47. Hiroi, Y., Itoh, Y., Hamasaki, T., & Sugimoto, M. (2017, March). AdaptiVisor: assisting eye adaptation via
occlusive optical see-through head-mounted displays. In Proceedings of the 8th Augmented Human
International Conference (p. 9). ACM.
スマートなサングラス

48. スマートなサングラス
Hiroi, Y., Itoh, Y., Hamasaki, T., & Sugimoto, M. (2017, March). AdaptiVisor: assisting eye adaptation via
occlusive optical see-through head-mounted displays. In Proceedings of the 8th Augmented Human
International Conference (p. 9). ACM.

49. 短未来予測＋AR映像表示
Yuta Itoh, Yuichi Hiroi, Jiu Otsuka, Maki Sugimoto, Jason Orlosky, Kiyoshi Kiyokawa, Gudrun Klinker:
Laplacian vision: augmenting motion prediction via optical see-through head-mounted displays and projectors. SIGGRAPH Emerging Technologies 2016: 13:1
実物体の予測軌道をHMDでリアルタイムに可視化し判断を助ける
Yuta Itoh, Jason Orlosky, Kiyoshi Kiyokawa, Gudrun Klinker:
Laplacian Vision: Augmenting Motion Prediction via Optical See-Through Head-Mounted Displays. AH 2016: 16:1-16:8, (Best paper award)
物理学で言うところの
「ラプラスの悪魔
(Laplace’s demon)」
がコンセプト

50. プロジェクションマッピングで色補正
Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance
Enhancement for Visually Impaired with Projector Camera Feedback.“,
TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009
E.g., assisting color-blinded people

51. 視覚拡張による色覚補助
光学シースルーHMDで補正色を視界に重畳し、色覚異常者を補助する
Tobias Langlotz, Jonathan Sutton, Stefanie Zollmann, Yuta Itoh, Holger Regenbrecht:
ChromaGlasses: Computational Glasses for Compensating Colour Blindness. CHI 2018: 390
(Best paper honorable mention, top 5%)

52. 視覚拡張による色覚補助
光学シースルーHMDで補正色を視界に重畳し、色覚異常者を補助する
Tobias Langlotz, Jonathan Sutton, Stefanie Zollmann, Yuta Itoh, Holger Regenbrecht:
ChromaGlasses: Computational Glasses for Compensating Colour Blindness. CHI 2018: 390
(Best paper honorable mention, top 5%)

53. Blurred
Enhanced
Defocus Cancelation
Itoh, Y., & Klinker, G. (2015, March). Vision enhancement: defocus correction via optical see-through
head-mounted displays. In Proceedings of the 6th Augmented Human International Conference (pp.
1-8). ACM.
Compensation
layer

54. De-blurring OST-HMD screen
Oshima, K., Rompapas, D. C., Moser, K., Swan, E., Ikeda, S., Yamamoto, G., ... & Kato, H. (2015). Sharpview:
Improved legibility of defocussed content on optical see-through head-mounted displays. In Demo under
submission for: ACM International Symposium on Mixed and Augmented Reality.

55. まとめ
• ARは“幻滅のくぼ地 (trough of disillusionment)”から脱する段階にある。
• 視覚の拡張(augmentation)/変調(modulation)が先のテーマになるかも
• （視覚に関する）AR要素技術として欲しいもの
• 薄くて軽い光源や光学部品
• （屋内外を問わない、6DoFトラッキングインフラ）
• （キラーコンテンツ・・・）

56. Appendix

57. https://jp.rs-online.com/web/p/mil-connector/0466652/