1. A Physical Approach to Moving Cast Shadow
Detection
Jia-Bin Huang and Chu-Song Chen
jbhuang0604@gmail.com, song@iis.sinica.edu.tw
Institute of Information Science
Academia Sinica, Taipei, Taiwan
April 23, 2009
1 / 26

2. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
2 / 26

3. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
3 / 26

4. Introduction
Motivation
Moving object detection is one of the most important task
in low-level vision.
Detecting moving cast shadows is one of the most
challenging problems for accurate object detection in video
streams since shadow points are often misclassified as
object points.
Without careful consideration, cast shadows may introduce
significant error in segmentation, tracking, and recognition.
4 / 26

5. Introduction
The Cause of Cast Shadows
Light sources are partially or totally blocked by the foreground
objects.
Why Detecting Cast Shadows Is Difficult?
1 Shadow points are detectable as foreground points and
typically differ significantly from the background.
2 Cast shadows have the same motion as the objects
casting them.
3 Shaded regions are usually connected with the foreground
objects.
5 / 26

6. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
6 / 26

7. Related Works (1/2)
Previous Works (before 2003)
A Survey paper: [Prati et al. PAMI 2003]
Statistical parametric: [Mikic et al. ICPR 2000]
Statistical nonparametric: [Horprasert et al. ICCV
Workshop 1999]
Deterministic model-based: [Onoguchi ICPR 1998]
Deterministic nonmodel-based: [Cucchiara et al. PAMI
2001]
Major Drawbacks
Need to explicitly tune the parameters for each scene.
Hard to adapt to the illumination conditions and
environment changes.
7 / 26

8. Related Works (2/2)
Learning-based Approaches
Basic Idea: Learn cast shadow model from video sequences.
Shadow Flow, [Porikli et al. ICCV 2005]
Gaussian Mixture Shadow Modeling, [Martel-Brisson et al.
PAMI 2007]
Combining Local and Global Features, [Liu et al. CVPR 07]
Learning Physical Model of Light Sources and Surfaces
[Martel-Brisson et al. CVPR 2008]
Drawbacks
Most of them assume shadow values will attenuate linearly
along the line between the value of the corresponding
background and the origin.
Pixel-based models may suffer from slow learning due to
the lack of sufficient samples.
8 / 26

9. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
9 / 26

10. Main Idea
A general physics-based shadow model
Decompose light incident at the background surface into
two classes
Direct light sources (e.g., sun)
Ambient illumination (e.g., light scattered by the sky,
colored light from nearby surfaces (color bleeding))
Suppose we have N light sources and M ambient
illumination, then the intensity function of light:
N M
E(λ) = Eincident,n(λ) + Eambient,m(λ).
n m
10 / 26

11. Ambient Illuminations and Direct Light Sources
Lambertian model: camera
sensor response gk(p) at point p
gk(p) = E(λ, p)ρ(λ, p)Sk(λ)dλ.
E(λ, p) Intensity function of light
sources
ρ(λ, p) The reflectance of an
object surface
Sk(λ) Sensor spectral sensitivity
function
11 / 26

12. Appearance Variation Under Cast Shadow
Part or total light sources are blocked by foreground
objects
Ambient illumination may be slightly changed (from BGA to
′
BGA)
12 / 26

13. Color Feature Vector
We encode the difference vector between background and
shadow value as our color feature.
xs,t(p) = [αt(p), θt(p), φt(p)]T (in spherical coordinate
system)
Illumination attenuation
||vt(p)||
αt(p) =
||BGt(p)||
Angle information
vG(p)
t
θt(p) = arctan( )
vR(p)
t
vB(p)
t
φt(p) = arccos( )
||vt(p)||
13 / 26

14. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
14 / 26

15. Overview
1 Perform background subtraction to obtain foreground
candidates (i.e., including real foreground and cast
shadows)
2 Apply weak shadow detector as a pre-filter to obtain
shadow candidates (e.g., filter out those pixels whose
illumination values are larger than the corresponding
background values)
3 For these shadow candidates, learn the color feature
vector xs,t(p) using GMM over time
4 Detecting cast shadow using the learned cast shadow
model
15 / 26

16. Weak Shadow Detector
Criterion for shadow candidates
BGt(p)
rl(p) =
xt(p) cos(ψ(p))
xt(p), BGt(p)
ψ(p) = arccos
xt(p) BGt(p)
rmin < rl(p) < rmax, ψ(p) < ψmax
16 / 26

17. Incorporating Spatial Information
Prior Knowledge of Cast Shadows
Cast shadows would not enhance the spatial gradient intensity
Introduce ωt(p) as a confidence value of cast shadows
ε + |∇(Bt(p))|
ωt(p) = ,
ε + max{|∇(It(p))|, |∇(Bt(p))|}
where ε is a smooth term.
To accelerate the learning speed of the pixel-based
shadow model, take ωt(p) as confidence value to update
shadow model at pixel p
Penalize samples having larger gradient intensity than
background by lessening the learning rate
17 / 26

18. Detecting Shadows at Light/Shadow Border
Shadows at light/shadow border show different behavior
from shadows inside the shaded region
Solution: Detecting cast shadows only with angle
information
(a)αt(p)
(b)θt(p)
(c)φt(p)
18 / 26

19. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
19 / 26

20. Qualitative Evaluation
(a) (b) (c) (d)
Figure: (a) Original images, (b) Background posterior probability, (c)
Shadow posterior probability, and (d) Forground posterior probability
20 / 26

21. Quantitative Evaluation
Performance Evaluation Metrics [Prati et al. PAMI 2003]
Shadow Detection Rate η
TPS
η=
TPS + FNS
Shadow Discriminative Rate ξ
TPF
ξ=
TPF + FNF
Sequence Highway I Highway II Hallway
Method η% ξ% η% ξ% η% ξ%
Proposed 72.34 84.98 72.70 79.89 71.69 88.25
Kernel 70.50 84.40 68.40 71.20 72.40 86.70
LGf 72.10 79.70 - - - -
GMSM 63.30 71.30 58.51 44.40 60.50 87.00
21 / 26

22. Effect of Shadows at Shadow/Light border
(a) (b) (c)
Figure: Effect of shadows at shadow/light border (a) Original frame of
sequence “Highway I". (b)(c) Foreground posterior without/with
considering shadows at shadow/light border.
22 / 26

23. Outline
1 Introduction
2 Related Works
3 Physical Model for Cast Shadows
4 Learning and Detecting Cast Shadows
5 Experimental Results
6 Conclusion and Future Work
23 / 26

24. Conclusion
Provide a better description for background surface value
variation under cast shadow
Incorporate spatial information to accelerate the learning of
pixel-based shadow model
Take shadows at light/shadow border into consideration
24 / 26

25. Future Work
Derive physics-based features for building a global shadow
model in a scene
Jia-Bin Huang and Chu-Song Chen, “Moving Cast Shadow
Detection using Physics-based Features", CVPR 2009
Extend the physical model to handle more general cases
(e.g., surface with specular reflection, spatial varing
ambient illumination, etc.)
25 / 26

26. The End
Thank you
26 / 26