1. Camera Calibration

2. What is Camera Calibration?
• Primarily, finding the quantities internal to
the camera that affect the imaging process
§ Position of image center in the image
• It is typically not at (width/2, height/2) of image
§ Focal length
§ Different scaling factors for row pixels
and column pixels
§ Skew factor
§ Lens distortion (pin-cushion effect)

3. Motivation
• Good calibration is important when we need
to
§ Reconstruct a world model: Virtual L.A. project
§ Interact with the world
• Robot, hand-eye coordination
Image plane Evaluation of position of a square for 2 focal
lengths (red and blue projection geometry)
We see a square of
known size

4. Scaling of Rows and Columns
in Image
• Camera pixels are not necessarily square
• Camera output may be analog (NTSC)
• Image may be obtained by digitizing card
§ A/D converter samples NTSC signal
Monitor
Camera NTSC signal Digitizing display
CCD/
CMOS

5. Compound Lens Imaging
• Inexpensive single lens systems distort image at
its periphery
• Compound lenses may be used to reduce
chromatic effects and pin-cushion effects
Camera
Image plane
Nodal Point
Principal planes
f
Image plane for equivalent
pinhole camera is f C
not camera image plane Center of Projection

6. Variety of Techniques
• VERY large literature on the subject
• Work of Roger Tsai influential
• Linear algebra method described here
§ Can be used as initialization for iterative non
linear methods.
• Some interesting methods use vanishing
points

7. Camera and Calibration Target

8. Calibration Procedure
• Calibration target: 2 planes at right angle with
checkerboard patterns (Tsai grid)
§ We know positions of pattern corners only with
respect to a coordinate system of the target
§ We position camera in front of target and find
images of corners
§ We obtain equations that describe imaging and
contain internal parameters of camera
• As a side benefit, we find position and orientation of
camera with respect to target (camera pose)

9. Image Processing of
Image of Target
• Canny edge detection
• Straight line fitting to detected linked edges
• Intersecting the lines to obtain the image
corners
• Matching image corners and 3D target
checkerboard corners
§ By counting if whole target is visible in image
• We get pairs (image point)--(world point)
( xi , yi ) → ( X i , Yi , Z i )

10. Central Projection
If world and image points are represented by homogeneous
vectors, central projection is a linear transformation:
xs
xi = f u   f 0
 xs 
0 0  
zs v  =  0   ys  xi = u / w, yi = v / w
   f 0 0
 zs 
ys  w  0
   0 1 0  
 1
yi = f   Image plane Scene point
zs y Image point
(xs , ys , zs )
center of (xi , yi , f)
x
projection
z
C
f

11. Transformation From Lengths to Pixels
Transformation uses: y Image point
ypix
(xi , yi , f)
• image center (x0, y0)
y0 Image center x
• scaling factors kx and ky
x0 xpix
xs xs + zs x0
xi = f x pix = k x xi + x0 = f k x
zs zs
ys y s + z s y0
yi = f y pix = k y yi + y0 = f k y
zs zs
 xs 
 u '  α x 0 x0 0   α x = f kx x pix = u ' / w'
 v'  =  0 α y0   ys 
0
   with then
y
 zs  α y = f ky y pix = v' / w'
w'  0
   0 1 0  
 1
 

12. Internal Camera Parameters
 xs 
 u '  α x s x0 0    α x = f kx x pix = u ' / w'
 v'  =  0 α y y0 0   ys 
   with
 zs  α y = − f ky y pix = v' / w'
w'  0
   0 1 0  
 1
 
α x s x0 0 α x s x0  1 0 0 0
0 α y0 0 =  0 α y y0  0 1 0 0 = K [I 3 | 03 ]
 y    
0
 0 1 0  0
  0 1  0 0 1 0
 
• αx and αy “focal lengths” in pixels
• x0 and y0 coordinates of image center in pixels
•Added parameter s is skew parameter
• K is called calibration matrix. Five degrees of freedom.
•K is a 3x3 upper triangular matrix

13. From Camera Coordinates
y to World Coordinates
C x
Image plane
Image point Z
z
(xi , yi , f)
Y
(xs , ys , zs )
M O
(Xs , Ys , Z s ) X

14. From Camera Coordinates
to World Coordinates 2
x
X
i k z M
j (Xs , Ys , Z s ) in world cordinate system
C
I (xs , ys , zs ) in image cordinate system
CO = T translation vector J
O K Z
CM = CO + OM
xS i + y S j + zS k = Tx i + Ty j + Tz k + X S I + YS J + Z S K
xS = Tx + X S I.i + YS J.i + Z S K.i
 xS  Tx   I.i J.iK.i   X S 
 y  = T  +  I.j J.j K.j   YS 
 S   y   
 z S  Tz  I.k
     J.k K.k   Z S 
 

15. Homogeneous Coordinates
 xS  Tx   I.i J.i K.i   X S 
 y  = T  +  I.j J.j K.j   Y 
 S  y   S 
 z S  Tz  I.k J.k K.k   Z S 
      
 xS   I.i J.i Tx   X S 
K.i  xS  X S 
 y   I.j y  T  YS 
 S =  J.j Ty   YS 
K.j    S = R  
 z S  I.k J.k K.k Tz   Z S   z S  0 T
 3
 Z 
1 S
        
1  0 0 0 1  1  1  1 

16. From Camera Coordinates
to World Coordinates 3
x
X
i k z M
j (Xs , Ys , Z s )
C
I (xs , ys , zs )
J
O K Z
CM = OM - OC
xS i + yS j + z S k = ( X S - X C ) I + (YS - YC ) J + ( Z S - Z C ) K
xS = ( X S - X C ) I.i + (YS - YC ) J.i + ( Z S - Z C ) K.i
~ ~
x cam = R (X - C) (T = -R C)
~
C is vector OC expressed in world coordinate system

17. Homogeneous Coordinates 2
~
• Here we use - R C instead of T
 xS  XS 
y   ~  
 S = R - R C  YS 

 z S  0 T 1  ZS 
   
3
1  1 

18. Linear Transformation from
World Coordinates to Pixels
• Combine camera projection and coordinate
transformation matrices into a single matrix P
 xs 
u' y 
 v '  = K [I | 0 3 ] s 
  3
 zs  XS 
w'
    u'  ~  
1  v'  = K [I | 0 ] R
⇒ 
- R C  YS 
3  T 
1  Z S 
3
 xS  XS   w' 0 3
   
y   ~    1 
 S = R - R C  YS 
 XS 
 z S  0 T 1  ZS  u' Y 
    ⇒  v'  = P  S 
3
  x = PX
1  1  ZS 
w'
   
1 

19. Properties of Matrix P
• Further simplification of P:
R ~
- R C
x = K [I 3 | 0 3 ] T X
0 3 1 
~
[I 3
R
| 0 3 ] T
- R C
 [ ~
] [ ~
= R - R C = R I3 | -C ]
0 3 1 
[ ~
x = K R I3 | -C X ]
[
~
P = K R I3 | - C ]
• P has 11 degrees of freedom: ~
• 5 from triangular calibration matrix K, 3 from R and 3 from C
• P is a fairly general 3 x 4 matrix
•left 3x3 submatrix KR is non-singular

20. Calibration
• 1. Estimate matrix P using scene points and
their images
• 2. Estimate the interior parameters and the
exterior parameters
[ ~
P = K R I3 | - C ]
§ Left 3x3 submatrix of P is product of upper-
triangular matrix and orthogonal matrix

21. Finding Camera Translation
• Find homogeneous coordinates of C in the
scene
• C is the null vector of matrix P
§ P C = 0:  X c  0
1 0 0 X c    
P = K R I3[ ~
| -C ] 0 1 0
 Yc   Yc  = 0
 Z c  0
0 0 1
 Zc    
 1
  0
• Find null vector C of P using SVD
§ C is the unit singular vector of P corresponding to the
smallest singular value (the last column of V, where P
= U D VT is the SVD of P)

22. Finding Camera Orientation and
Internal Parameters
• Left 3x3 submatrix M of P is of form M=K R
§ K is an upper triangular matrix
§ R is an orthogonal matrix
• Any non-singular square matrix M can be
decomposed into the product of an upper-
triangular matrix K and an orthogonal matrix R
using the RQ factorization
§ Similar to QR factorization but order of 2 matrices is
reversed

23. RQ Factorization of M
1 0 0   c' 0 s ' c' ' − s' ' 0
R x = 0 c − s  , R y =  0 1 0  , R z = s ' ' c' ' 0
     
0 s c 
  − s ' 0 c'
  0
 0 1 
m33 m32
• Compute c = − , s= 2
( m32 + m33 )
2 2 1/ 2
(m32 + m33 )1/ 2
2
• Multiply M by Rx. The resulting term at (3,2) is zero
because of the values selected for c and s
• Multiply the resulting matrix by Ry, after selecting c’ and
s’ so that the resulting term at position (3,1) is set to zero
• Multiply the resulting matrix by Rz, after selecting c’’ and
s’’ so that the resulting term at position (2,1) is set to zero
M RxR yR z = K ⇒ M = K R TRTRT = K R
z y x

24. Computing Matrix P
• Use corresponding image and scene points
§ 3D points Xi in world coordinate system
§ Images xi of Xi in image
• Write xi = P Xi for all i
• Similar problem to finding projectivity
matrix H (i.e. homography) in homework

25. Improved Computation of P
• xi = P Xi involves homogeneous
coordinates, thus xi and P Xi just have to be
proportional: x i × P Xi = 0
• Let p1T, p2T, p3T be the 3 row vectors of P
p 1 X i 
T
 v'i p T Xi − w'i p T Xi 
3 2
 T   
P X i = p 2 X i  xi × P X i =  w'i p1 Xi − u 'i p 3 Xi 
T T
p T X i   u'i p T X i − v'i p1 X i 
T
 3   2 
 0T − w'i X T v 'i X T   p 1  p1 

4 i i
T    
⇒  w 'i X Ti 0T4 − u'i X i  p 2  = 0 p2  is a 12 × 1 vector
 − v 'i X T u 'i X T 0T  p 3  p3 
 i i 4    

26. Improved Computation of P,
cont’d
• Third row can be obtained from sum of u’i
times first row - v’i times second row
 0T 4 − w'i X T
i v 'i X T   p 1 
i
  
 w'i X T i 0T4 − u 'i X T  p 2  = 0
i
 − v 'i X T u 'i X T 0T  p 3 
 i i 4  
• So we get 2 independent equations in 11
unknowns (ignoring scale)
• With 6 point correspondences, we get
enough equations to compute matrix P
Ap=0

27. Solving A p = 0
• Linear system A p = 0
• When possible, have at least 5 times as
many equations as unknowns (28 points)
• Minimize || A p || with the constraint
|| p || = 1
§ P is the unit singular vector of A corresponding
to the smallest singular value (the last column
of V, where A = U D V T is the SVD of A)
• Called Direct Linear Transformation (DLT)

28. Improving P Solution with
Nonlinear Minimization
• Find p using DLT
• Use as initialization for nonlinear
minimization of ∑ d ( x i , PX i ) 2
i
§ Use Levenberg-Marquardt iterative minimization

29. Radial Distortion
• We have assumed that lines are imaged as
lines
• Not quite true for real lenses
§ Significant error for cheap optics and for short
focal lengths
Scene Image Corrected image

30. Radial Distortion Modeling
• In pixel cordinates the correction is written
xc − x0 = L(r ) ( x − x0 )
(x, y) yc − y0 = L ( r ) ( y − y0 )
(xc, yc)
(x0, y0) with
Distorted line
r 2 = ( x − x0 ) 2 + ( y − y0 ) 2
And ideal line:
L(r ) = 1 + κ 1 r + κ 2 r 2 + ...
(x, y)
(x’, y’)
• Minimize f (κ 1 ,κ 2 ) = ∑ ( x'i − xci ) 2 + ( y 'i − yci ) 2
i
using lines known to be straight
(x’,y’) is radial projection of (x,y) on straight line

31. References
• Multiple View Geometry in Computer Vision, R.
Hartley and A. Zisserman, Cambridge University
Press, 2000, pp. 138-183
• Three-Dimensional Computer Vision: A
Geometric Approach, O. Faugeras, MIT Press,
1996, pp. 33-68
• “A Versatile Camera Calibration Technique for
3D Machine Vision”, R. Y. Tsai, IEEE J. Robotics
& Automation, RA-3, No. 4, August 1987, pp.
323-344