Presentation slides for URAI 2012. http://www.kros.org/urai2012/ http://www.kros.org/urai2012/04program07.php

1. URAI 2012, Session TD4: Robot & Future Info Device
Calibration Issues in FRC:
Camera, Projector, Kinematics based
Hybrid Approach
Joo-Haeng Lee ETRI
Kosuke Maegawa Ritsumeikan University
Jong-Seung Park Ritsumeikan University
Joo-Ho Lee Ritsumeikan University

2. Agenda
• Introduction: FRC
• Example Application: Robotic Spatial AR (RSAR)
• Calibration Issues: Camera, Projector, Kinematics
• Summary
• Q&A
2 Joo-Haeng Lee (joohaeng at etri.re.kr)

3. ETRI FRC 2010
3 Joo-Haeng Lee (joohaeng at etri.re.kr)

4. ETRI FRC 2011
4 Joo-Haeng Lee (joohaeng at etri.re.kr)

5. ETRI FRC 2012
5 Joo-Haeng Lee (joohaeng at etri.re.kr)

6. ETRI FRC 2012
• Major components for RSAR
• RSAR = Robotic Spatial Augmented Reality

7. ETRI FRC 2012
• Major components for RSAR
• RSAR = Robotic Spatial Augmented Reality
Robotis Logitech
Optoma
Dynamixel HD Pro Webcam
PK-320
MX-28 C920

8. Examples of RSAR
PixelFlex Ubiquitous Display
(MIT, 2001) (Ritsumeikan Univ. , 2006)
LuminAR Beamatron
(MIT; 2010) (Microsoft Research, 2012)

9. Examples of RSAR
PixelFlex Ubiquitous Display
(MIT, 2001) (Ritsumeikan Univ. , 2006)
LuminAR Beamatron
(MIT; 2010) (Microsoft Research, 2012)

10. FRC RSAR
Image in the world:
Body outline
8 Joo-Haeng Lee (joohaeng at etri.re.kr)

11. FRC RSAR
Image in the world: Image from R Prj:
Body outline Skeleton
8 Joo-Haeng Lee (joohaeng at etri.re.kr)

12. FRC RSAR
Image in the world: Image from R Prj: Image from L Prj:
Body outline Skeleton Vessels
8 Joo-Haeng Lee (joohaeng at etri.re.kr)

13. FRC RSAR

14. FRC RSAR

15. Motivation
• Calibration really matters in RSAR!
• camera to capture the geometry of the world
• projector to display on the real-world surface
• kinematics to control and sense the motion

16. Calibration: Camera

17. Calibration: Camera
• Camera Model
• Qc = Mc Xwc G
• Qc: image in the camera
• Mc: camera internal
• Xwc: camera external
• G: geometry in the world

18. Calibration: Camera
• Chang’s method in OpenCV
• internal and external parameters + lens distortion

19. Calibration: Camera
• Chang’s method in OpenCV
• internal and external parameters + lens distortion
• Issues
• geometric constraints should be considered for
precise calibration of other components such as
kinematics

20. Calibration: Projector

21. Calibration: Projector
• Projection Model
• Qp = Mp Xwp Gp
• Gp = Xwp-1 Mp-1 Qp
• Qp: image to be projected
• Mp: projector internal
• Xwp: projector external
• Gp: projected area in the world

22. Calibration: Projector
• Chang’s method in OpenCV
• If a projector is not moving or well aligned, we can
apply Chang’s method as in the camera case

23. Calibration: Projector
• Chang’s method in OpenCV
• If a projector is not moving or well aligned, we can
apply Chang’s method as in the camera case
• Issues
• However, for a moving projector, we need to handle
lens shift, which cannot be solved using Chang’s.

24. Calibration: Projector
• Tsai’s method with custom implementation
• Can handle lens shift: no need to specify the image
size

25. Calibration: Projector
• Tsai’s method with custom implementation
• Can handle lens shift: no need to specify the image
size
• Constrained concave programming based on
Lagrangian multiplier method: Qp = P G

26. Calibration: Projector
• Tsai’s method with custom implementation
• Can handle lens shift: no need to specify the image
size
• Constrained concave programming based on
Lagrangian multiplier method: Qp = P G
• RQ decomposition: P = Mp Xwp

27. Calibration: Projector
• Tsai’s method with custom implementation
Fig. An optical center of a projector (in green) that is approximated from camera frustums
from data set #2. Each frustum is aligned in the common coordinate frame of a camera.
The optical center of a camera (in white) is the origin of the frame. Optical centers of a
projector computed using the previous method (assuming no-lens shift) are in gray.
A pair of red and blue points is the
closest points between two frustum
Extended rectangles (in orange) from the partial rectangles (in blue).(in gray) of a
Fig. edges. The average is the Six optical centers
approximate optical center of a projector assuming the no lens-shift.
projector (in green). Computed using the fixed internal

28. Calibration: Projector-Camera
• Projector-camera system
• Calibrated camera: Mc and Xwc
• Calibrated projector: Mp and Xwp

29. Calibration: Projector-Camera
• Projector-camera system
• Calibrated camera: Mc and Xwc
• Calibrated projector: Mp and Xwp
• Transformation between projector and camera

30. Calibration: Projector-Camera
• Projector-camera system
• Calibrated camera: Mc and Xwc
• Calibrated projector: Mp and Xwp
• Transformation between projector and camera
• Xcp = Xwp Xcw = Xwp Xwc -1

31. Calibration: Projector-Camera
• Projector-camera system
• Calibrated camera: Mc and Xwc
• Calibrated projector: Mp and Xwp
• Transformation between projector and camera
• Xcp = Xwp Xcw = Xwp Xwc -1
• Transformation from the world to the projector

32. Calibration: Projector-Camera
• Projector-camera system
• Calibrated camera: Mc and Xwc
• Calibrated projector: Mp and Xwp
• Transformation between projector and camera
• Xcp = Xwp Xcw = Xwp Xwc -1
• Transformation from the world to the projector
• Xwp(t) = Xcp Xwc(t)

33. Calibration: Kinematics
• Precise calibration of kinematics is required for
the quality of RSAR application in FRC
• (ex) inverse kinematics

34. Calibration: Kinematics
• Need to consider the difference between the
CAD model and the actual assembly

35. Calibration: Kinematics
• Vision-based kinematics calibration (in progress)

36. Calibration: Kinematics
• Vision-based kinematics calibration (in progress)

37. Calibration: Kinematics
• Vision-based kinematics calibration (in progress)

38. Calibration: Kinematics
• Vision-based kinematics calibration (in progress)

39. Calibration: Kinematics
• Vision-based kinematics calibration (in progress)

40. Summary

41. Summary
• Calibration Process
• Camera: Chang’s method
• Projector: Tsai’s method
• Projector-Camera: Xcp
• Kinematics: vision-based approach

42. Summary
• Calibration Process
• Camera: Chang’s method
• Projector: Tsai’s method
• Projector-Camera: Xcp
• Kinematics: vision-based approach
• Future Works
• Better precision from hybrid calibration approach

43. Q &A
joohaeng at etri dot re dot kr

45. Memo