SlideShare ist ein Scribd-Unternehmen logo
1 von 44
June 25, 2020
Tomoyuki Mukasa
Rakuten Institute of Technology
Rakuten, Inc.
2
2015Ph.D. Student Engineer Researcher2012
3D Reconstruction
& Motion Analysis
Tomoyuki MUKASA, Ph.D. 3D Vision Researcher
VR for
Exhibition
AR for Tourism
AR/VR/HCI for
e-commerce
4
Contributing to
existing businesses
Exploring
new ideas
Increasing
tech-brand awareness
Using Computer Vision & Human Computer Interaction
5
Woman
Red
Blouse
Category Attributes
6
Commoditization of
AR/SLAM
Prototypes in Rakuten
Impact of ARKit/ARCore
Deep learning for
AR/SLAM
Dense 3D reconstruction on SLAM
SLAM w/o Camera
Sensors on AR glasses
AR/SLAM for Web WebAR/SLAM
5G + MEC
Web for AR/SLAM Web as Stock footage
Beyond the field of view
Learning from IoT data
7
Augmented Reality (AR) and Simultaneous Localization and Mapping (SLAM) has been commoditize.
• Prototypes in Rakuten
• AR furniture app
• Impact of ARKit/ARCore
• Scale estimation solved by IMU fusion
• Research on the shoulder of giants
9
10
Need to be tracked in 3D!
11
Need to be tracked in 3D!
Almost solved in ARKit/ARCore…
12
Merchants’ pages
3D models SLAM w/ scale estimation
Advanced visualization w/
inpainting & relighting
AR app for everyone
E. Zhang, M. F. Cohen, and B. Curless.
"Emptying, Refurnishing, and Relighting Indoor Spaces”, SIGGRAPH Asia, 2016.
13
ARKit /
ARCore
Merchants’ pages
3D models SLAM w/ scale estimation
Advanced visualization w/
inpainting & relighting
AR app for everyone
E. Zhang, M. F. Cohen, and B. Curless.
"Emptying, Refurnishing, and Relighting Indoor Spaces”, SIGGRAPH Asia, 2016.
14
• Dense 3D reconstruction on SLAM
• Depth prediction by CNN
• SLAM + Depth prediction
• SLAM w/o Camera
• Sensors on AR glasses
• Google Glass’s return
• Revival of UWB
15
• Direct method based on photo consistency
• Multi-baseline stereo using GPU
• Getting easier to run on the latest mobile
device, but still unwanted from the end-user
point of view because of energy consumption,
etc.
R. A. Newcombe, S. J. Lovegrove and A. J. Davison,
"DTAM: Dense tracking and mapping in real-time," ICCV, 2011
16
D. Eigen, C. Puhrsch, and R. Fergus.
“Depth map prediction from a single image using a multi-scale deep network.”
NIPS, 2014.
M. Kaneko, K. Sakurada and K. Aizawa.
“MeshDepth: Disconnected Mesh-based Deep Depth Prediction.”
ArXiv, 2019.
Global Coarse-Scale Network +
Local Fine-Scale Network
Disconnected mesh representation
17
Semi-dense SLAM + Prediction Compact and optimizable representation of
dense geometry
K. Tateno, F. Tombari, I. Laina and N. Navab, "CNN-SLAM: Real-Time Dense
Monocular SLAM with Learned Depth Prediction," CVPR, 2017.
M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger and A. J. Davison.
“CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM.”
CVPR, 2018.
18
Image capturing
& Visualization thread
2D tracking thread
3D Mapping thread
Depth prediction thread
Depth fusion thread
CLIENT-SIDE
SERVER-SIDE
Monocular visual SLAM
Depth prediction by CNN
3D reconstruction
Depth fusion
by surface mesh deformation
t t+1 t+2 t+3 t+4
Key-frame
ARAP deformation
19
Figure 4. (Top) Distribution of weightswi for thedeformation and
(bottom) thecorresponding textured mesh. Larger intensity values
in thetop figureindicate thehigher weights.
4. Experiments
frames detected by ORB-SLAM because these are selected
based on visual changes. We filter out those key-frames us-
ing a spatio-temporal distance criterion similar to the other
feature-based approaches, e.g., PTAM , and send them to the
server.
The key-frames are processed on the server and the depth
image for each frame is estimated by the CNN architecture.
In the fusion process, we convert the depth images to a re-
fined mesh sequence as shown at the bottom of Figure 5.We
also make the ground truth mesh sequence correspond to the
refined one from the raw depth maps captured by the depth
sensor on the other hand. We compute residual errors be-
tween the refined mesh and the ground truth as shown in Ta-
ble 2 and Figure 6. We can observe that our framework ef-
ficiently reduces the residual errors for all sequences. Both
the average and the median of the residual errors fall within
the range from about two thirds to a half.
We also evaluate the absolute scale estimated from depth
prediction as shown in the rightmost column in the Table 2.
The average error of the estimated scales for our six office
scenes is 20% of the ground truth scale.
5. Conclusion
In this paper, we proposed a framework fusing the re-
20
Sofa area 1 Sofa area 2 Sofa area 3 Desk area 1 Desk area 2 Meeting room
Figure 5. Input data for our depth fusion and the reconstructed scenes. From top to bottom row: color images, feature tracking result
of SLAM, corresponding ground truth depth images, depth images estimated by DNN, and 3D reconstruction results on six office scenes,
respectively.
Scene M esh from CNN depth map Refined mesh by our method
Mean Median Std dev Mean Median Std dev Scale
21
22
• DeepFactors: Real-Time Probabilistic Dense Monocular SLAM. Jan Czarnowski, Tristan Laidlow,
Ronald Clark, Andrew J. Davison. IEEE Robotics and Automation Letters (RA-L), 2020
23
RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods
Hang Yan, Sachini Herath, Yasutaka Furukawa
• Now SLAM is possible only w/ IMU
24
• The 1st Google Glass raised privacy concerns (cf. driving recorder / cameras on connected cars)
• Google Glass returned as enterprise edition
• U1 chip uses Ultra Wideband (UWB) technology
• UWB devices can detect locations within 10 cm
• Wide indoor area localization
• Apple glasses w/o camera, but w/ U1?
• Potential application: SLAM w/o camera
25
• WebAR/SLAM
• Marker-based WebAR for events
• WebAR/SLAM w/ IMU fusion
• 5G + MEC
• The future of 5G-enabled Augmented Reality
26
概要
Pros:
• アプリインストール無しでARが可能
• HTML(+Javascript)のみでコンテンツ制作可能
Cons:
• 現状では要専用マーカー
• 対応環境に制限(iOS11以降のSafari, Android5以降のChrome)
実装
• AR.js: マーカー位置推定
• A-frame: コンテンツ制作
Future work
• 任意画像マーカー
• マーカーレスAR (cf. ARKit, ARCore)
• GeolocationとARマップの統合
27
Pros:
• No need to install native app
• Easy to create only w/ HTML(+Javascript)
Cons:
• Marker-based
• Need newer environment
(Later than iOS11Safari, Android5 Chrome)
Implementation
• AR.js + A-frame
28
• AR photo booth: 240 groups
• AR lottery: 510 people
29
Trial in Mother’s day &
Father’s day
Trial
@Tokyo Dome
R-mobile campaign
30
8th Wall © 2019
8th Wall built their own highly-optimized SLAM engine, and then re-architected it for the mobile web.
AUGMENTED REALITY FOR THE WEB
Javascript
WebGL
WebAssembly
Six-Degrees-of-Freedom (6DoF)
Tracking
Point-Cloud
Lighting
Surface Estimation
Image Detection
31
Light-
weight
Web AR
SOTA
SLAM
Schneider, Thomas et al.
“Maplab: An Open Framework for Research
in Visual-Inertial Mapping and Localization.”
IEEE Robotics and Automation Letters, 2018.
32
Offline loop
closure and
optimization
Online
recording
StartEnd
Office Space Mapping and Optimization
Back end visualization of the location map
Objects
Object detection
& recognition
Input image
Surface orientation Partial view alignment
3D pose estimation
Plane fitting 3D scene initialization
Room Geometry
Objects in 3D scene
walls initialized
with unknown scale
Output:
3D model
reconstruction
Potential 3D applications on server in future
• Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image, Yinyu Nie,
Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, Jian Jun Zhang, CVPR, 2020
35
The future of 5G-enabled Augmented Reality
• Powered by Mobile Edge Computing (MEC)
• Big data processing w/ultra low latency
• Example by Scape + Samsung
Scape Technologies © 2019
36
• Web as Stock footage
• “Understanding Media”: "Hot" and "cool" media
• Stock footage for narrative
• Google street view time-lapse
• Beyond the field of view
• Photo uncrop
• Neural rendering in the wild
• Learning from web
• Learning human depth
• Learning from IoT data
37
Understanding Media: The Extensions of Man by Marshall McLuhan (1964)
• "Hot" and "cool" media
• Hot: "high definition” like film
• Cool: require more active participation on the part of the user like TV
• Content of every medium is always another (previous) medium.
The birth of virtual reality as an art form by Chris Milk (TEDTalks, 2016)
• Is VR the last medium?
• What about AR?
38
Edwin S. Porter:
Life of an American Fireman
(1903)
Sameer Agarwal, et al.:
Building a Rome in a Day
(ICCV 2009)
39
• Photo uncrop, Qi Shan, Brian Curless, Yasutaka Furukawa, Carlos Hernandez, and Steven M. Seitz,
ECCV ‘14.
40
M. Meshry, D. B. Goldman, S. Khamis, H. Hoppe, R. Pandey, N. Snavely and R. M-Brualla.
“Neural Rendering in the Wild.” CVPR, 2019
Total Scene Capture
• Encode the 3D structure of the scene, enabling rendering from an arbitrary viewpoint,
• Capture all possible appearances of the scene and allow rendering the scene under any of them.
• Understand the location and appearance of transient objects in the scene
and allow for reproducing or omitting them.
41
Z. Li, T. Dekel, F. Cole, R. Tucker,
N. Snavely, C. Liu and W. T. Freeman.
“Learning the Depths of Moving People
by Watching Frozen People.”
CVPR, 2019.
42
Kinect returns as
Azure Kinect
• Higher resolution, more accurate depth
• Multimodal sensing
• Integrated to Azure Cognitive Services, Azure IoT
Azure Kinect DK Kinect for Windows v2
Audio Details 7-mic circular array 4-mic linear phased
array
Motion sensor Details 3-axis accelerometer
3-axis gyro
3-axis accelerometer
RGB Camera Details 3840 x 2160 px @30
fps
1920 x 1080 px @30
fps
Depth Camera Method Time-of-Flight Time-of-Flight
Resolution 640 x 576 px @30 fps 512 x 424 px @ 30 fps
512 x 512 px @30 fps
1024x1024 px @15
fps
Connectivity Data USB3.1 Gen 1 with
type USB-C
USB 3.1 gen 1
Power External PSU or USB-
C
External PSU
Synchronization RGB & Depth internal,
external device-to-
device
RGB & Depth internal
only
Mechanical Dimensions 103 x 39 x 126 mm 249 x 66 x 67 mm
Mass 440 g 970 g
Mounting One ¼-20 UNC. Four
internal screw points
One ¼-20 UNC
Microsoft Azure IoT reference architecture
43
• AR/SLAM technologies are
commoditized,
but still nice-to-have,
not must-have.
• Deep learning is pushing
the boundaries of AR/SLAM
• Web-based AR/SLAM has
huge potential in 5G era
• AR/SLAM can be improved by
learning from Web
including IoT data
Designed by
macrovector / Freepik
AR/SLAM and IoT

Weitere ähnliche Inhalte

Was ist angesagt?

3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理
3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理
3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理Toru Tamaki
 
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術SSII
 
Direct Sparse Odometryの解説
Direct Sparse Odometryの解説Direct Sparse Odometryの解説
Direct Sparse Odometryの解説Masaya Kaneko
 
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentationTakuya Minagawa
 
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成MobileRoboticsResear
 
カルマンフィルタ入門
カルマンフィルタ入門カルマンフィルタ入門
カルマンフィルタ入門Yasunori Nihei
 
20190307 visualslam summary
20190307 visualslam summary20190307 visualslam summary
20190307 visualslam summaryTakuya Minagawa
 
ORB-SLAMを動かしてみた
ORB-SLAMを動かしてみたORB-SLAMを動かしてみた
ORB-SLAMを動かしてみたTakuya Minagawa
 
サーベイ論文:画像からの歩行者属性認識
サーベイ論文:画像からの歩行者属性認識サーベイ論文:画像からの歩行者属性認識
サーベイ論文:画像からの歩行者属性認識Yasutomo Kawanishi
 
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装についてMobileRoboticsResear
 
LiDAR点群と画像とのマッピング
LiDAR点群と画像とのマッピングLiDAR点群と画像とのマッピング
LiDAR点群と画像とのマッピングTakuya Minagawa
 
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)Masaya Kaneko
 
Immersal を活用した AR クラウドなシステム開発とハンズオン!
Immersal を活用した AR クラウドなシステム開発とハンズオン!Immersal を活用した AR クラウドなシステム開発とハンズオン!
Immersal を活用した AR クラウドなシステム開発とハンズオン!NishoMatsusita
 
SSII2020 [O3-01] Extreme 3D センシング
SSII2020 [O3-01]  Extreme 3D センシングSSII2020 [O3-01]  Extreme 3D センシング
SSII2020 [O3-01] Extreme 3D センシングSSII
 
三次元点群を取り扱うニューラルネットワークのサーベイ
三次元点群を取り扱うニューラルネットワークのサーベイ三次元点群を取り扱うニューラルネットワークのサーベイ
三次元点群を取り扱うニューラルネットワークのサーベイNaoya Chiba
 
Cosine Based Softmax による Metric Learning が上手くいく理由
Cosine Based Softmax による Metric Learning が上手くいく理由Cosine Based Softmax による Metric Learning が上手くいく理由
Cosine Based Softmax による Metric Learning が上手くいく理由tancoro
 
点群深層学習 Meta-study
点群深層学習 Meta-study点群深層学習 Meta-study
点群深層学習 Meta-studyNaoya Chiba
 
SLAM入門 第2章 SLAMの基礎
SLAM入門 第2章 SLAMの基礎SLAM入門 第2章 SLAMの基礎
SLAM入門 第2章 SLAMの基礎yohei okawa
 
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use ToolsDeep Learning JP
 

Was ist angesagt? (20)

3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理
3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理
3次元レジストレーションの基礎とOpen3Dを用いた3次元点群処理
 
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術
SSII2021 [OS3-01] 設備や環境の高品質計測点群取得と自動モデル化技術
 
Direct Sparse Odometryの解説
Direct Sparse Odometryの解説Direct Sparse Odometryの解説
Direct Sparse Odometryの解説
 
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation
2018/12/28 LiDARで取得した道路上点群に対するsemantic segmentation
 
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成
 
カルマンフィルタ入門
カルマンフィルタ入門カルマンフィルタ入門
カルマンフィルタ入門
 
20190307 visualslam summary
20190307 visualslam summary20190307 visualslam summary
20190307 visualslam summary
 
ORB-SLAMを動かしてみた
ORB-SLAMを動かしてみたORB-SLAMを動かしてみた
ORB-SLAMを動かしてみた
 
サーベイ論文:画像からの歩行者属性認識
サーベイ論文:画像からの歩行者属性認識サーベイ論文:画像からの歩行者属性認識
サーベイ論文:画像からの歩行者属性認識
 
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について
【RSJ2021】LiDAR SLAMにおける高信頼なループ閉合の実装について
 
LiDAR点群と画像とのマッピング
LiDAR点群と画像とのマッピングLiDAR点群と画像とのマッピング
LiDAR点群と画像とのマッピング
 
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)
Visual SLAM: Why Bundle Adjust?の解説(第4回3D勉強会@関東)
 
Immersal を活用した AR クラウドなシステム開発とハンズオン!
Immersal を活用した AR クラウドなシステム開発とハンズオン!Immersal を活用した AR クラウドなシステム開発とハンズオン!
Immersal を活用した AR クラウドなシステム開発とハンズオン!
 
SSII2020 [O3-01] Extreme 3D センシング
SSII2020 [O3-01]  Extreme 3D センシングSSII2020 [O3-01]  Extreme 3D センシング
SSII2020 [O3-01] Extreme 3D センシング
 
Lucas kanade法について
Lucas kanade法についてLucas kanade法について
Lucas kanade法について
 
三次元点群を取り扱うニューラルネットワークのサーベイ
三次元点群を取り扱うニューラルネットワークのサーベイ三次元点群を取り扱うニューラルネットワークのサーベイ
三次元点群を取り扱うニューラルネットワークのサーベイ
 
Cosine Based Softmax による Metric Learning が上手くいく理由
Cosine Based Softmax による Metric Learning が上手くいく理由Cosine Based Softmax による Metric Learning が上手くいく理由
Cosine Based Softmax による Metric Learning が上手くいく理由
 
点群深層学習 Meta-study
点群深層学習 Meta-study点群深層学習 Meta-study
点群深層学習 Meta-study
 
SLAM入門 第2章 SLAMの基礎
SLAM入門 第2章 SLAMの基礎SLAM入門 第2章 SLAMの基礎
SLAM入門 第2章 SLAMの基礎
 
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools
【DL輪読会】Toolformer: Language Models Can Teach Themselves to Use Tools
 

Ähnlich wie AR/SLAM and IoT

Visual geometry with deep learning
Visual geometry with deep learningVisual geometry with deep learning
Visual geometry with deep learningNAVER Engineering
 
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro..."High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...Edge AI and Vision Alliance
 
Dario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringDario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringAdvanced-Concepts-Team
 
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...Tomohiro Fukuda
 
Building rome in_a_day_yas-nyan
Building rome in_a_day_yas-nyan Building rome in_a_day_yas-nyan
Building rome in_a_day_yas-nyan YasunobuToyota
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in VisionSangmin Woo
 
Using Deep Learning to Derive 3D Cities from Satellite Imagery
Using Deep Learning to Derive 3D Cities from Satellite ImageryUsing Deep Learning to Derive 3D Cities from Satellite Imagery
Using Deep Learning to Derive 3D Cities from Satellite ImageryAstraea, Inc.
 
Large Scale Image Retrieval 2022.pdf
Large Scale Image Retrieval 2022.pdfLarge Scale Image Retrieval 2022.pdf
Large Scale Image Retrieval 2022.pdfSamuCerezo
 
AR, the TODAY
AR, the TODAYAR, the TODAY
AR, the TODAYJongHyoun
 
Understanding the world in 3D with AI.pdf
Understanding the world in 3D with AI.pdfUnderstanding the world in 3D with AI.pdf
Understanding the world in 3D with AI.pdfQualcomm Research
 
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...Universitat Politècnica de Catalunya
 
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...Virtual Tourism
 
=iros16tutorial_2.pdf
=iros16tutorial_2.pdf=iros16tutorial_2.pdf
=iros16tutorial_2.pdfusmanarif88
 
fyp presentation of group 43011 final.pptx
fyp presentation of group 43011 final.pptxfyp presentation of group 43011 final.pptx
fyp presentation of group 43011 final.pptxIIEE - NEDUET
 
Integration of a Structure from Motion into Virtual and Augmented Reality for...
Integration of a Structure from Motion into Virtual and Augmented Reality for...Integration of a Structure from Motion into Virtual and Augmented Reality for...
Integration of a Structure from Motion into Virtual and Augmented Reality for...Tomohiro Fukuda
 
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...Pirouz Nourian
 
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...Christian Kehl
 

Ähnlich wie AR/SLAM and IoT (20)

Visual geometry with deep learning
Visual geometry with deep learningVisual geometry with deep learning
Visual geometry with deep learning
 
Introduction of slam
Introduction of slamIntroduction of slam
Introduction of slam
 
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro..."High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...
"High-resolution 3D Reconstruction on a Mobile Processor," a Presentation fro...
 
Dario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineeringDario izzo - Machine Learning methods and space engineering
Dario izzo - Machine Learning methods and space engineering
 
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...
Point Cloud Stream on Spatial Mixed Reality: Toward Telepresence in Architect...
 
Building rome in_a_day_yas-nyan
Building rome in_a_day_yas-nyan Building rome in_a_day_yas-nyan
Building rome in_a_day_yas-nyan
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in Vision
 
Using Deep Learning to Derive 3D Cities from Satellite Imagery
Using Deep Learning to Derive 3D Cities from Satellite ImageryUsing Deep Learning to Derive 3D Cities from Satellite Imagery
Using Deep Learning to Derive 3D Cities from Satellite Imagery
 
Large Scale Image Retrieval 2022.pdf
Large Scale Image Retrieval 2022.pdfLarge Scale Image Retrieval 2022.pdf
Large Scale Image Retrieval 2022.pdf
 
AR, the TODAY
AR, the TODAYAR, the TODAY
AR, the TODAY
 
Understanding the world in 3D with AI.pdf
Understanding the world in 3D with AI.pdfUnderstanding the world in 3D with AI.pdf
Understanding the world in 3D with AI.pdf
 
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
 
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...
Evaluation of the Acceptance of Virtual Worlds in the Tourism Sector: An Ext...
 
=iros16tutorial_2.pdf
=iros16tutorial_2.pdf=iros16tutorial_2.pdf
=iros16tutorial_2.pdf
 
fyp presentation of group 43011 final.pptx
fyp presentation of group 43011 final.pptxfyp presentation of group 43011 final.pptx
fyp presentation of group 43011 final.pptx
 
Sccg Many Projects Layout03
Sccg Many Projects Layout03Sccg Many Projects Layout03
Sccg Many Projects Layout03
 
Integration of a Structure from Motion into Virtual and Augmented Reality for...
Integration of a Structure from Motion into Virtual and Augmented Reality for...Integration of a Structure from Motion into Virtual and Augmented Reality for...
Integration of a Structure from Motion into Virtual and Augmented Reality for...
 
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...
Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators us...
 
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...
Interactive Simulation and Visualization of Large-Scale Flooding Scenarios (J...
 
Portfolio
PortfolioPortfolio
Portfolio
 

Mehr von Rakuten Group, Inc.

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話Rakuten Group, Inc.
 
楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のりRakuten Group, Inc.
 
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Rakuten Group, Inc.
 
DataSkillCultureを浸透させる楽天の取り組み
DataSkillCultureを浸透させる楽天の取り組みDataSkillCultureを浸透させる楽天の取り組み
DataSkillCultureを浸透させる楽天の取り組みRakuten Group, Inc.
 
大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開Rakuten Group, Inc.
 
楽天における大規模データベースの運用
楽天における大規模データベースの運用楽天における大規模データベースの運用
楽天における大規模データベースの運用Rakuten Group, Inc.
 
楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャーRakuten Group, Inc.
 
楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割Rakuten Group, Inc.
 
Rakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Group, Inc.
 
The Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfThe Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfRakuten Group, Inc.
 
Supporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfSupporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfRakuten Group, Inc.
 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfRakuten Group, Inc.
 
How We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfHow We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfRakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoRakuten Group, Inc.
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoRakuten Group, Inc.
 
Introduction of GORA API Group technology
Introduction of GORA API Group technologyIntroduction of GORA API Group technology
Introduction of GORA API Group technologyRakuten Group, Inc.
 
100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情Rakuten Group, Inc.
 
社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャーRakuten Group, Inc.
 

Mehr von Rakuten Group, Inc. (20)

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話
 
楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり楽天における安全な秘匿情報管理への道のり
楽天における安全な秘匿情報管理への道のり
 
What Makes Software Green?
What Makes Software Green?What Makes Software Green?
What Makes Software Green?
 
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...
 
DataSkillCultureを浸透させる楽天の取り組み
DataSkillCultureを浸透させる楽天の取り組みDataSkillCultureを浸透させる楽天の取り組み
DataSkillCultureを浸透させる楽天の取り組み
 
大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開大規模なリアルタイム監視の導入と展開
大規模なリアルタイム監視の導入と展開
 
楽天における大規模データベースの運用
楽天における大規模データベースの運用楽天における大規模データベースの運用
楽天における大規模データベースの運用
 
楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー楽天サービスを支えるネットワークインフラストラクチャー
楽天サービスを支えるネットワークインフラストラクチャー
 
楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割楽天の規模とクラウドプラットフォーム統括部の役割
楽天の規模とクラウドプラットフォーム統括部の役割
 
Rakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdfRakuten Services and Infrastructure Team.pdf
Rakuten Services and Infrastructure Team.pdf
 
The Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdfThe Data Platform Administration Handling the 100 PB.pdf
The Data Platform Administration Handling the 100 PB.pdf
 
Supporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdfSupporting Internal Customers as Technical Account Managers.pdf
Supporting Internal Customers as Technical Account Managers.pdf
 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdf
 
How We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdfHow We Defined Our Own Cloud.pdf
How We Defined Our Own Cloud.pdf
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech info
 
Travel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech infoTravel & Leisure Platform Department's tech info
Travel & Leisure Platform Department's tech info
 
OWASPTop10_Introduction
OWASPTop10_IntroductionOWASPTop10_Introduction
OWASPTop10_Introduction
 
Introduction of GORA API Group technology
Introduction of GORA API Group technologyIntroduction of GORA API Group technology
Introduction of GORA API Group technology
 
100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情100PBを越えるデータプラットフォームの実情
100PBを越えるデータプラットフォームの実情
 
社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー社内エンジニアを支えるテクニカルアカウントマネージャー
社内エンジニアを支えるテクニカルアカウントマネージャー
 

Kürzlich hochgeladen

Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 

Kürzlich hochgeladen (20)

Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 

AR/SLAM and IoT

  • 1. June 25, 2020 Tomoyuki Mukasa Rakuten Institute of Technology Rakuten, Inc.
  • 2. 2 2015Ph.D. Student Engineer Researcher2012 3D Reconstruction & Motion Analysis Tomoyuki MUKASA, Ph.D. 3D Vision Researcher VR for Exhibition AR for Tourism AR/VR/HCI for e-commerce
  • 3.
  • 4. 4 Contributing to existing businesses Exploring new ideas Increasing tech-brand awareness Using Computer Vision & Human Computer Interaction
  • 6. 6 Commoditization of AR/SLAM Prototypes in Rakuten Impact of ARKit/ARCore Deep learning for AR/SLAM Dense 3D reconstruction on SLAM SLAM w/o Camera Sensors on AR glasses AR/SLAM for Web WebAR/SLAM 5G + MEC Web for AR/SLAM Web as Stock footage Beyond the field of view Learning from IoT data
  • 7. 7 Augmented Reality (AR) and Simultaneous Localization and Mapping (SLAM) has been commoditize. • Prototypes in Rakuten • AR furniture app • Impact of ARKit/ARCore • Scale estimation solved by IMU fusion • Research on the shoulder of giants
  • 8.
  • 9. 9
  • 10. 10 Need to be tracked in 3D!
  • 11. 11 Need to be tracked in 3D! Almost solved in ARKit/ARCore…
  • 12. 12 Merchants’ pages 3D models SLAM w/ scale estimation Advanced visualization w/ inpainting & relighting AR app for everyone E. Zhang, M. F. Cohen, and B. Curless. "Emptying, Refurnishing, and Relighting Indoor Spaces”, SIGGRAPH Asia, 2016.
  • 13. 13 ARKit / ARCore Merchants’ pages 3D models SLAM w/ scale estimation Advanced visualization w/ inpainting & relighting AR app for everyone E. Zhang, M. F. Cohen, and B. Curless. "Emptying, Refurnishing, and Relighting Indoor Spaces”, SIGGRAPH Asia, 2016.
  • 14. 14 • Dense 3D reconstruction on SLAM • Depth prediction by CNN • SLAM + Depth prediction • SLAM w/o Camera • Sensors on AR glasses • Google Glass’s return • Revival of UWB
  • 15. 15 • Direct method based on photo consistency • Multi-baseline stereo using GPU • Getting easier to run on the latest mobile device, but still unwanted from the end-user point of view because of energy consumption, etc. R. A. Newcombe, S. J. Lovegrove and A. J. Davison, "DTAM: Dense tracking and mapping in real-time," ICCV, 2011
  • 16. 16 D. Eigen, C. Puhrsch, and R. Fergus. “Depth map prediction from a single image using a multi-scale deep network.” NIPS, 2014. M. Kaneko, K. Sakurada and K. Aizawa. “MeshDepth: Disconnected Mesh-based Deep Depth Prediction.” ArXiv, 2019. Global Coarse-Scale Network + Local Fine-Scale Network Disconnected mesh representation
  • 17. 17 Semi-dense SLAM + Prediction Compact and optimizable representation of dense geometry K. Tateno, F. Tombari, I. Laina and N. Navab, "CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction," CVPR, 2017. M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger and A. J. Davison. “CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM.” CVPR, 2018.
  • 18. 18 Image capturing & Visualization thread 2D tracking thread 3D Mapping thread Depth prediction thread Depth fusion thread CLIENT-SIDE SERVER-SIDE Monocular visual SLAM Depth prediction by CNN 3D reconstruction Depth fusion by surface mesh deformation t t+1 t+2 t+3 t+4 Key-frame ARAP deformation
  • 19. 19 Figure 4. (Top) Distribution of weightswi for thedeformation and (bottom) thecorresponding textured mesh. Larger intensity values in thetop figureindicate thehigher weights. 4. Experiments frames detected by ORB-SLAM because these are selected based on visual changes. We filter out those key-frames us- ing a spatio-temporal distance criterion similar to the other feature-based approaches, e.g., PTAM , and send them to the server. The key-frames are processed on the server and the depth image for each frame is estimated by the CNN architecture. In the fusion process, we convert the depth images to a re- fined mesh sequence as shown at the bottom of Figure 5.We also make the ground truth mesh sequence correspond to the refined one from the raw depth maps captured by the depth sensor on the other hand. We compute residual errors be- tween the refined mesh and the ground truth as shown in Ta- ble 2 and Figure 6. We can observe that our framework ef- ficiently reduces the residual errors for all sequences. Both the average and the median of the residual errors fall within the range from about two thirds to a half. We also evaluate the absolute scale estimated from depth prediction as shown in the rightmost column in the Table 2. The average error of the estimated scales for our six office scenes is 20% of the ground truth scale. 5. Conclusion In this paper, we proposed a framework fusing the re-
  • 20. 20 Sofa area 1 Sofa area 2 Sofa area 3 Desk area 1 Desk area 2 Meeting room Figure 5. Input data for our depth fusion and the reconstructed scenes. From top to bottom row: color images, feature tracking result of SLAM, corresponding ground truth depth images, depth images estimated by DNN, and 3D reconstruction results on six office scenes, respectively. Scene M esh from CNN depth map Refined mesh by our method Mean Median Std dev Mean Median Std dev Scale
  • 21. 21
  • 22. 22 • DeepFactors: Real-Time Probabilistic Dense Monocular SLAM. Jan Czarnowski, Tristan Laidlow, Ronald Clark, Andrew J. Davison. IEEE Robotics and Automation Letters (RA-L), 2020
  • 23. 23 RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods Hang Yan, Sachini Herath, Yasutaka Furukawa • Now SLAM is possible only w/ IMU
  • 24. 24 • The 1st Google Glass raised privacy concerns (cf. driving recorder / cameras on connected cars) • Google Glass returned as enterprise edition • U1 chip uses Ultra Wideband (UWB) technology • UWB devices can detect locations within 10 cm • Wide indoor area localization • Apple glasses w/o camera, but w/ U1? • Potential application: SLAM w/o camera
  • 25. 25 • WebAR/SLAM • Marker-based WebAR for events • WebAR/SLAM w/ IMU fusion • 5G + MEC • The future of 5G-enabled Augmented Reality
  • 26. 26 概要 Pros: • アプリインストール無しでARが可能 • HTML(+Javascript)のみでコンテンツ制作可能 Cons: • 現状では要専用マーカー • 対応環境に制限(iOS11以降のSafari, Android5以降のChrome) 実装 • AR.js: マーカー位置推定 • A-frame: コンテンツ制作 Future work • 任意画像マーカー • マーカーレスAR (cf. ARKit, ARCore) • GeolocationとARマップの統合
  • 27. 27 Pros: • No need to install native app • Easy to create only w/ HTML(+Javascript) Cons: • Marker-based • Need newer environment (Later than iOS11Safari, Android5 Chrome) Implementation • AR.js + A-frame
  • 28. 28 • AR photo booth: 240 groups • AR lottery: 510 people
  • 29. 29 Trial in Mother’s day & Father’s day Trial @Tokyo Dome R-mobile campaign
  • 30. 30 8th Wall © 2019 8th Wall built their own highly-optimized SLAM engine, and then re-architected it for the mobile web. AUGMENTED REALITY FOR THE WEB Javascript WebGL WebAssembly Six-Degrees-of-Freedom (6DoF) Tracking Point-Cloud Lighting Surface Estimation Image Detection
  • 31. 31 Light- weight Web AR SOTA SLAM Schneider, Thomas et al. “Maplab: An Open Framework for Research in Visual-Inertial Mapping and Localization.” IEEE Robotics and Automation Letters, 2018.
  • 32. 32 Offline loop closure and optimization Online recording StartEnd Office Space Mapping and Optimization Back end visualization of the location map
  • 33. Objects Object detection & recognition Input image Surface orientation Partial view alignment 3D pose estimation Plane fitting 3D scene initialization Room Geometry Objects in 3D scene walls initialized with unknown scale
  • 34. Output: 3D model reconstruction Potential 3D applications on server in future • Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image, Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, Jian Jun Zhang, CVPR, 2020
  • 35. 35 The future of 5G-enabled Augmented Reality • Powered by Mobile Edge Computing (MEC) • Big data processing w/ultra low latency • Example by Scape + Samsung Scape Technologies © 2019
  • 36. 36 • Web as Stock footage • “Understanding Media”: "Hot" and "cool" media • Stock footage for narrative • Google street view time-lapse • Beyond the field of view • Photo uncrop • Neural rendering in the wild • Learning from web • Learning human depth • Learning from IoT data
  • 37. 37 Understanding Media: The Extensions of Man by Marshall McLuhan (1964) • "Hot" and "cool" media • Hot: "high definition” like film • Cool: require more active participation on the part of the user like TV • Content of every medium is always another (previous) medium. The birth of virtual reality as an art form by Chris Milk (TEDTalks, 2016) • Is VR the last medium? • What about AR?
  • 38. 38 Edwin S. Porter: Life of an American Fireman (1903) Sameer Agarwal, et al.: Building a Rome in a Day (ICCV 2009)
  • 39. 39 • Photo uncrop, Qi Shan, Brian Curless, Yasutaka Furukawa, Carlos Hernandez, and Steven M. Seitz, ECCV ‘14.
  • 40. 40 M. Meshry, D. B. Goldman, S. Khamis, H. Hoppe, R. Pandey, N. Snavely and R. M-Brualla. “Neural Rendering in the Wild.” CVPR, 2019 Total Scene Capture • Encode the 3D structure of the scene, enabling rendering from an arbitrary viewpoint, • Capture all possible appearances of the scene and allow rendering the scene under any of them. • Understand the location and appearance of transient objects in the scene and allow for reproducing or omitting them.
  • 41. 41 Z. Li, T. Dekel, F. Cole, R. Tucker, N. Snavely, C. Liu and W. T. Freeman. “Learning the Depths of Moving People by Watching Frozen People.” CVPR, 2019.
  • 42. 42 Kinect returns as Azure Kinect • Higher resolution, more accurate depth • Multimodal sensing • Integrated to Azure Cognitive Services, Azure IoT Azure Kinect DK Kinect for Windows v2 Audio Details 7-mic circular array 4-mic linear phased array Motion sensor Details 3-axis accelerometer 3-axis gyro 3-axis accelerometer RGB Camera Details 3840 x 2160 px @30 fps 1920 x 1080 px @30 fps Depth Camera Method Time-of-Flight Time-of-Flight Resolution 640 x 576 px @30 fps 512 x 424 px @ 30 fps 512 x 512 px @30 fps 1024x1024 px @15 fps Connectivity Data USB3.1 Gen 1 with type USB-C USB 3.1 gen 1 Power External PSU or USB- C External PSU Synchronization RGB & Depth internal, external device-to- device RGB & Depth internal only Mechanical Dimensions 103 x 39 x 126 mm 249 x 66 x 67 mm Mass 440 g 970 g Mounting One ¼-20 UNC. Four internal screw points One ¼-20 UNC Microsoft Azure IoT reference architecture
  • 43. 43 • AR/SLAM technologies are commoditized, but still nice-to-have, not must-have. • Deep learning is pushing the boundaries of AR/SLAM • Web-based AR/SLAM has huge potential in 5G era • AR/SLAM can be improved by learning from Web including IoT data Designed by macrovector / Freepik