New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Robotic Spatial AR (로봇 공간증강현실 기술 소개)
1. 한국CAD/CAM학회 2014년 정기 학술발표회
세션:Human-Machine Interaction
프로젝터-카메라 시스템을 활용한 사용자 상호작용 기술
공간증강현실 및 로봇으로의 응용을 중심으로
!
User Interaction Techniques based on
Projector-Camera System
Applications in Spatial AR and Integration with Robotics
이주행
ETRI 융합기술연구부문 책임연구원
2. Agenda for SCCE 2014
• Projector-Camera System in SAR / RSAR
-
Related works including ETRI FRC
-
RSAR: path tracing, augmentation, IR Pen
-
Kinematics, Projection, Hybrid
• Real Examples
SAR: curve design, FTIR canvas, tracking performance
• Control Issues in RSAR
• Q&A
2
Joo-Haeng Lee (joohaeng at etri.re.kr)
3. AR, SAR, RSAR
• AR
-
Augmented Reality
• SAR
-
Spatial Augmented Reality
• RSAR
-
Robotic Spatial Augmented Reality
3
Joo-Haeng Lee (joohaeng at etri.re.kr)
4. AR, SAR, RSAR
AR
SAR
RSAR
User views
Monitor
Real world
Real world
Augmentation over
Camera image
Real world
Real world
HW Components
Camera,
Monitor
Camera,
Projector
Camera,
Projector,
Manipulator
Mobility
Static,
Handheld,
Wearable
Static,
Handheld,
Wearable
Dynamic,
Autonomous
4
Joo-Haeng Lee (joohaeng at etri.re.kr)
5. AR
• Camera for Real World
• Computer Graphics for Virtual World
• Augmentation of the virtual objects (such as
information, UI, ...) on the captured image of the real
world
• A user mainly watches a monitor screen.
• Need to recognize objects in the camera image
• Need to register the two imageries seamlessly
5
Joo-Haeng Lee (joohaeng at etri.re.kr)
6. Examples of AR
• Vuforia (Qualcomm AR SDK)
6
Joo-Haeng Lee (joohaeng at etri.re.kr)
7. Examples of AR
• Vuforia (Qualcomm AR SDK)
7
Joo-Haeng Lee (joohaeng at etri.re.kr)
8. Examples of AR
• World Lens (Quest Visual)
8
Joo-Haeng Lee (joohaeng at etri.re.kr)
9. SAR
• Projector + AR
-
mobility through pico projectors
-
A user mainly watches a real world rather than a display
device.
-
Usually, projectors and cameras are not motion-controlled:
static, hand-held, wearable
-
Images can be pre-warped before projection
Similar with AR in recognition and registration
are directly
• Virtual objectsthe surface ofaugmented (or
projected) on
real objects.
9
Joo-Haeng Lee (joohaeng at etri.re.kr)
10. Examples of SAR
• Early works in SAR by Ramesh Raska (MIT)
-
iLamps: Geometrically Aware and Self-Configuring
Projectors (SIGGRAPH 2003)
-
RFIG Lamps: Interacting with a Self-describing World via
Photosensing Wireless Tags and Projectors (SIGGRAPH
2004)
10
Joo-Haeng Lee (joohaeng at etri.re.kr)
25. Examples of SAR
• MirageTable (ACM SIGCHI 2012; Microsoft)
25
Joo-Haeng Lee (joohaeng at etri.re.kr)
26. Examples of SAR
• MirageTable (ACM CHI 2012; Microsoft)
26
Joo-Haeng Lee (joohaeng at etri.re.kr)
27. Examples of SAR
• MirageTable (ACM CHI 2012; Microsoft)
27
Joo-Haeng Lee (joohaeng at etri.re.kr)
28. Examples of SAR
• Augmented Interactive Cake (Patent filed; 2012;
Disney Research)
28
Joo-Haeng Lee (joohaeng at etri.re.kr)
29. Examples of SAR
• HideOut (ACM TEI 2013; Disney Research)
29
Joo-Haeng Lee (joohaeng at etri.re.kr)
30. Examples of SAR
• HideOut (ACM TEI 2013; Disney Research)
30
Joo-Haeng Lee (joohaeng at etri.re.kr)
31. Examples of SAR
• SAR in Automotive Manufacturing (Virtual Reality
2012; WCL, U South Australia)
31
Joo-Haeng Lee (joohaeng at etri.re.kr)
32. Examples of SAR
• “Projected Augmented Reality: Keeping Pace with
Innovation” (Intel; 2012)
32
Joo-Haeng Lee (joohaeng at etri.re.kr)
33. Examples of SAR
• IllumiRoom (ACM SIGCHI 2013; Microsoft)
33
Joo-Haeng Lee (joohaeng at etri.re.kr)
34. Examples of SAR
• IllumiRoom (ACM SIGCHI 2013; Microsoft)
34
Joo-Haeng Lee (joohaeng at etri.re.kr)
35. References
• Book
-
O. Bimber and R. Raskar, Spatial Augmented Reality Merging Real and Virtual Worlds, 2005. A.K. Peters
• Tutorial
-
SIGGRAPH Course 2008
35
Joo-Haeng Lee (joohaeng at etri.re.kr)
36. RSAR
• Robotics + SAR
-
Projectors and cameras can be mounted on robotic
manipulators and/or mobile robot.
-
More DOFs: Kinematics + Image Warping
New type of end effectors and constraints
•
•
direction, position, shape, illumination intensity, ...
soft collision = visibility and occlusion
• Collaboration of multiple autonomous units.
36
Joo-Haeng Lee (joohaeng at etri.re.kr)
37. Examples of RSAR
• PixelFlex (IEEE Vis. 2001; MIT)
37
Joo-Haeng Lee (joohaeng at etri.re.kr)
38. Examples of RSAR
• Ubiquitous Display (HCII 2009; Ritsumeikan U Korea U)
38
Joo-Haeng Lee (joohaeng at etri.re.kr)
44. Applications of RSAR
• Interaction Techniques
-
as a component of multi-modal interaction
• Precision Applications with Interactive Performance
-
3D measurement
3D Imaging
•
Stereoscopic Photography
44
Joo-Haeng Lee (joohaeng at etri.re.kr)
45. RSAR in ETRI
• FRC: Future Robotic Computer
• Multimodal UI (User Interface)
-
Speech, Gesture, TTS, Touch, ...
• Intelligence
-
Situation awareness, Autonomous behavior, ...
• Bridging the real and virtual worlds
-
Spatial AR + Robotics + Sensors
45
Joo-Haeng Lee (joohaeng at etri.re.kr)
48. ETRI FRC 2012
• Major components for RSAR
• RSAR = Robotic Spatial Augmented Reality
Robotis
Dynamixel
MX-28
Optoma
PK-320
Logitech
HD Pro Webcam
C920
49. Pico Projectors
• Microvision SHOWWX
-
Laser-based
No need to control focus
10 lumen
• Optoma PK-301
-
DLP
Manual focusing required
20 lumens; 50 on AC
49
Joo-Haeng Lee (joohaeng at etri.re.kr)
50. Pico Projectors
• Microvision SHOWWX+
-
Laser-based
No need to control focus
15 lumens
• Optoma PK-320
-
DLP
Manual focusing required
25~50 lumens; 100 on AC
50
Joo-Haeng Lee (joohaeng at etri.re.kr)
56. FRC Kinematic Features
• 5 Motors: (tilt + pan) x 2 + (center pan)
-
Redundancy in Pan Rotation
56
Joo-Haeng Lee (joohaeng at etri.re.kr)
57. FRC Kinematic Features
• 5 Motors: (tilt + pan) x 2 + (center pan)
-
Redundancy in Pan Rotation
57
Joo-Haeng Lee (joohaeng at etri.re.kr)
58. RSAR Real Experiments
•
RSAR (with Ritsumeikan Univ.)
- IK path following
- Augmenting details
- Interaction with IR Pen
•
SAR
- Planar curve design (ETRI)
- FTIR Canvas (ETRI)
- 3D Tangram (with Kookmin Univ.)
58
Joo-Haeng Lee (joohaeng at etri.re.kr)
59. •
RSAR IK Path Following
Jacobian IK to follow a path (with Ritsumeikan U)
59
Joo-Haeng Lee (joohaeng at etri.re.kr)
60. RSAR Augmenting Details
Image in the world:
Body outline
Image from R Prj:
Skeleton
60
Image from L Prj:
Vessels
Joo-Haeng Lee (joohaeng at etri.re.kr)
82. RSAR Control Issues
• Kinematic Control
-
Inverse Kinematics
• Projection Control
-
Inverse Projection + Image Pre-warping
• Hybrid Control
-
Current: Kinematics + Projection
Next: Kinematics + Projection + Object Recognition
82
Joo-Haeng Lee (joohaeng at etri.re.kr)
83. Forward Control
Kinematics
Projection
K+P
Input
Parameters
- angles
- displacements
Image
+
Parameters
- internal, external
Image
+
Parameters
- internal, external
Process
Transformation
- rigid
Transformation
- perspective
Transformation
- rigid
- perspective
Output
End Effector
in a position
Projected Image
on a surface
Projected Image
on a surface
Exception
Collision
- physical objects
Occlusion
- light
- physical objects
Occlusion
+
Collision
83
Joo-Haeng Lee (joohaeng at etri.re.kr)
84. Backward Control
Kinematics
Input
Projection
K+P
End Effector
in a specific position
Projected Image
on a specific surface
with a specific
condition
Projected Image
on a specific surface
with a specific
condition
Process
Inverse Kinematics
Inverse Projection
IK
+
IP
Output
Parameters
- angles
- displacements
Image
+
Parameters
- internal, external
Image
+
Parameters
- internal, external
Constraint
Collision
- physical objects
Occlusion
- light
- physical objects
Occlusion
+
Collision
84
Joo-Haeng Lee (joohaeng at etri.re.kr)
85. More Constraints
• Geometry
-
correct geometry without distortion
-
uniform intensity over the surface
-
clear image with minimum blurring
-
maximum visibility and minimum shadowing
• Photometry
• Image
• Visibility
85
Joo-Haeng Lee (joohaeng at etri.re.kr)
93. Inverse Kinematics
•
Input
- q: position / orientation of the end-effector
•
Output
- k: angles of each joint
•
Processing
- A-1: inverse of A; generally, no analytic form.
- k = A-1(q)
93
Joo-Haeng Lee (joohaeng at etri.re.kr)
95. •
Inverse Kinematics
Partially Analytic IK
- To satisfy that the ray intersects at the base surface, the zcoordinate of the end-effector should be zero: z = 0.
- The length of the ray can be parameterized with tilt angle
100
80
60
40
20
-1.5
95
-1.0
-0.5
Joo-Haeng Lee (joohaeng at etri.re.kr)
96. •
Inverse Kinematics
Partially Analytic IK
- To satisfy that the ray intersects at the base surface, the zcoordinate of the end-effector should be zero: z = 0.
- The length of the ray can be parameterized with tilt angle
96
Joo-Haeng Lee (joohaeng at etri.re.kr)
100. •
Inverse Kinematics
Incremental Differential IK
- Jacobian of transformation matrix: J = dA/dt
- Inverse of Jacobian: J-1
- Jacobian Transpose: JT
100
Joo-Haeng Lee (joohaeng at etri.re.kr)
110. Example of
Inverse Projection (1)
• How to project the largest undistorted rectangle
to a viewer with a single projector?
-
Inscribed Rectangle
Anamorphic Illusion
Joo-Haeng Lee (joohaeng at etri.re.kr)
114. Artwork 1
Piet Mondrian, Composition A, 1923
Oil on canvas (Galleria Nazionale d'Arte Moderna e Contemporanea, Rome)
Joo-Haeng Lee (joohaeng at etri.re.kr)
123. Demo
• Simulation in Mathematica
-
Maximum Inscribing Rectangle inside a Projected
Quadrilateral
123
Joo-Haeng Lee (joohaeng at etri.re.kr)
124. Example of
Inverse Projection (2)
• Projector Pose Estimation
- Given an arbitrary convex quadrilateral, find external
parameters of a projector to project it from a
certain rectangle:
-
Which convex quadrilateral cannot be projected
from a rectangle?
-
How to modify a quadrilateral to be projectable?
Joo-Haeng Lee (joohaeng at etri.re.kr)
125. Quiz #1
Assume that you have a hand-held projector which can
accept an rectangular source image. Which of the
following convex quadrilaterals is projectable from your
projector?
(a)
(c)
Rhombus
Trapezoid___
(b)
Parallelogram
(d)
Isosceles
Trapezoid
125
Joo-Haeng Lee (joohaeng at etri.re.kr)
128. Quad: Configuration
• Ext Params
-
center of projection
• Int Params
-
pc
Qs
filed-of-view
v2
• Source Quad
-
rectangle (cf) aspect ratio
-
convex quadrilateral
v3
ms
v1
m
• Projected Quad
v0
• Length-related coefficients: α , β and γ for each diagonal
i
128
i
i
Joo-Haeng Lee (joohaeng at etri.re.kr)
129. Quad: Analytic Solution
• Constraint 1: cop
!
d0 =
!
cosθ 0
α0
=
cosθ1
= d1
α1
pc
y1 y0
• Constraint 2: fov
!
tanψ 0 =
!
tanθ 0
β0
=
tanθ1
β1
Qs
ms
v2
y0y1
q1
= tanψ 1
v3
m
q0
v1
• 2 Eqs and 2 Unknowns
-
Existence of solution?
v0
Geometric meaning?
129
Joo-Haeng Lee (joohaeng at etri.re.kr)
130. Quad: Analytic Solution
Equation
• Constraint 1: cop
!
d0 =
!
cosθ 0
α0
=
cosθ1
cosθ 0 = ±
= d1
α1
A=
• Constraint 2: fov
!
tanθ 0
!
β0
tanψ 0 =
=
tanθ1
β1
B=
= tanψ 1
• 2 Eqs and 2 Unknowns
-
C=
Existence of solution?
C 2 −1
A B −1
2
α1
α0
β1
β0
γ1
γ0
Coefficients
Geometric meaning?
130
Joo-Haeng Lee (joohaeng at etri.re.kr)
143. Summary
• Projector-Camera System in SAR / RSAR
-
Related works including ETRI FRC
• Real Examples
- RSAR path tracing, augmentation, IR Pen
- curve design, FTIR canvas
• Control Issues in RSAR
-
Kinematics, Projection, Hybrid
143
Joo-Haeng Lee (joohaeng at etri.re.kr)