1. Course Calendar
Class DATE Contents
1 Sep. 26 Course information & Course overview
2 Oct. 4 Bayes Estimation
3 〃 11 Classical Bayes Estimation - Kalman Filter -
4 〃 18 Simulation-based Bayesian Methods
5 〃 25 Modern Bayesian Estimation ：Particle Filter
6 Nov. 1 HMM(Hidden Markov Model)
Nov. 8 No Class
7 〃 15 Bayesian Decision
8 〃 29 Non parametric Approaches
9 Dec. 6 PCA(Principal Component Analysis)
10 〃 13 ICA(Independent Component Analysis)
11 〃 20 Applications of PCA and ICA
12 〃 27 Clustering, k-means et al.
13 Jan. 17 Other Topics 1 Kernel machine.
14 〃 22(Tue) Other Topics 2
2. Lecture Plan
Nonparametric Approaches
1. Introduction
1.1 An Example
1.2 Nonparametric Density Estimation Problems
1.3 Histogram density estimation
2. Kernel Density Estimation
3. K-Nearest Neighbor Density Estimation
4. Cross-validation
3. 1. Introduction
3
Automatic Fish-Sorting Process
action １
belt conveyer
Classification/ Decision Theory
p ," sea bass" ,
p ," salmon"
x
x
x1
x2
Decision
Boundary
R2
R1
(Duda, Hart, & Stork 2004)
Training phase
Test phase
4. 1.1 An Example -Fish sorting problem-
4
The first step -Training process-
The first task is a supervised learning process, thus each observed
sample has its own label either states of nature 𝜔1 𝑜𝑟 𝜔2.
Training(Learning) data
For a given set of N data samples, suppose the following labeled data
(the lightness of a fish) are observed.
1 21 2 1 2
1 2
( ) ( )
1
2
( ) ( )
: , , , , : , , ,
where , are the lightness of i-th samples of (sea bass)
and j-th sample of (salmon) respectively.
We assume , are discrete data as illustrated in Fig.1
N N
i j
i j
x x x y y y
x y
x y
(a)
This joint probability distribution gives the histograms of other
probabilities and densities as shown in Fig.1.
5. 5
These histograms can be used for Bayes decision classification.
(a) Samples drawn from a joint probability
over x and ω
Fig.1
iN
N
p x
1P
2P
1p x 2p x
x
x
x
x
(b)
(c)
(d)
(e)
1
2
6. 6
Density Estimation
The approach attempts to estimate the density directly from the
observed data.
The Bayes decision rules discussed in the last lecture have been
developed on the assumption that the relevant probability density
functions and prior probabilities are known. But this is not the case in
practice, we need to estimate the PDFs from a given se of observed data.
given data density distribution
Modeling of
density
http://www.unt.edu/benchmarks/archives/2003/february03/rss.htm
7. 1.2 Nonparametric approaches
Two approaches for density estimation
Parametric approach :
Assume that the forms of the density functions are known, and the
parameters of its are to be estimated.
Non-parametric approach:
This can be used with arbitrary distributions, and does not assume the
form of density function.
Why nonparametric?*
/ Classical parametric densities are unimodal ones whereas practical
problems involve multimodal densities. Non parametric approaches are
applicable to arbitrary density with few assumptions
*Another idea: a mixed (hybrid) with parametric and non-parametric densities.
8. 1.3 Histogram density estimation
A single variable (x) case
into distinct intervals (called ) of width (often chosen as
uniform bins ), and the number of data falling in -th bin.
For turning to a normalized probability density w
i
i
i
x
n i
n
Partition bins
count
e put
= at over i-th bin
The density ( ) is approximated by a stepwise function like bar graph.
In multi-dimensional case,
=
where V is
i
i
i
i
n
p x
N
p x
n
p x
NV
the volume of the bin.
Fig. 2 (RIGHT) Histogram density estimation
(from Bishop [3] web site) 50 data points are
generated from the distribution shown by the
green curve
(1)
9. 9
The feature of the histogram estimation depends on the width (Δi
) of bins as shown in Fig.2.
Small Δ → density tends to have many spikes
Large Δ → density tends to be over-smoothed
Merit: Convenient visualization tool
Problems:
Discontinuities at the bin’s edges
Computational burden in high dimensional space (MD)
10. 2. Kernel Density Estimation
10
-Basic idea of density estimation-
Unknown Density
D
p x
x R
1 2
A set of observations
, , N
N
x x x
p xEstimate
Consider a small regiron surrounding ( ) and define a probability
which means the probability of falling into
The number falling within of overall observe
x x
P p d
x
K N
R
R R
R
R.
d data would be
Suppose can be approximated by a constant over we have
K PN
p x R
generates
(2)
(3)
11. 11
where means the volume of .
Eqs. (2) (3) give the following density estimation form.
Two wayes of exploitation of this:
1) Fix then estimate
P p x V
V
K
p x
NV
V K
R
Kernel Density Estimation
2) Fix then estimate k-Nearest Neighbor EstimationK V
(4)
(5)
12. 12
Kernel Density Estimation
A point : wish to determine the density at this point
Region : small hypercube centered on
the volume of
Find the number of samples K that fall within the region .
For
x
x
V
R
R
R
counting we introduce the kernek function.
[one dimensional kernel function]
1
1
2
0
K
u
K u
elsewhere
For a given observation , considern
n
x
x x
K
h
(6)
(7)
13. 13
1
1
2
0
where is called the bandwidth or smoothing parameter.
For a set of observations : 1, ,
gives the number of data which are located
nn
n
N
n
n
h
x xx x
K
h
elsewhere
h
x n N
x x
K K
h
1
within ,
2 2
Substituting (9) into (5)
1
An example graph of is illustrated in Fig.3
N
n
n
h h
x x
x x
p x K
Nh h
p x
(8)
(9)
(10)
14. 14
Example : Data set {xn} n=1~4
x
x1 x2 x3 x4
xx1 x2 x3 x4
11 x x
K
h h
41 x x
K
h h
4
1
1 i
i
x x
K
h h
1
4
p x
x1
x2
x3
x4
Fig. 3
15. 15
Discontinuity and Smooth Kernel Function
Kernel Density Estimator will suffer from discontinuities in
estimated density. Smooth kernel function such as Gaussian is used.
This general method is referred to as the kernel density estimator or
Parzen estimator.
Example: Gaussian and 1-D case,
where h is the standard deviation.
Determination of bandwidth h
Small h → spiky p(x)
Large h → over-smoothed p(x)
Defects:
High computational cost
Because of fixed V there may be too few samples in some regions.
2
2
1
( )1 1
exp
22
N
n
n
x x
p x
N hh
(11)
17. 17
Example: Bayesian decision by the Parzen kernel estimation
Example: Kernel density estimation
Fig 4. Kernel density estimation
(from Bishop [3] web site)
Apply KDE method to the same 50
data used in Fig. 2
Fig. 5
The decision
boundaries
LEFT : small h
RIGHT: large h
Duda et al [1]
18. 3. K-Nearest Neighbors density estimation
18
KDE approaches use fixed h throughout the data space. But we want
to apply small h for highly dense data region, on the other hand, to set
a larger h for sparse data region.
We come up with the idea of K-Nearest Neighbor (K-NN) approaches.
Expand the region (radius) surrounding the estimation point x until it
encloses K data points.
K
p x
NV
Fixed
Determine the minimum
volume V containing K
points in R
Volume of
hypersphere
with radius ( )
D
D
r x
K K
p x
N V N c r x
(12)
19. 19
x
K-th closest neighbor point
1 2 3
4
( 2, , , )
3
Dc c c c
Fig. 7 K-nearest neighbor density
estimation
(from Bishop [3] web site)
Apply to the same 50 data points
used in Fig. 2
K is the free parameter in K-NN
method
r(x)
Problems: 1) integration of p(x) is not bounded, 2) discontinuities, 3)huge
computation time and storage
Fig. 6 K-NN algorithm
20. 20
K-NN estimation as a Bayesian classifier
A method to generate decision boundary directly based on a set of
data.
− N training data with class labels (ω1 ~ ωc)
Nl points for l-th class, such that
− Classify a test sample x
− Get a sphere with minimum radius r(x) which encircles K samples.
− The volume of the sphere：
− Kl points for l-th class (ωl)
− Class-conditioned density at x:
− Evidence:
− Prior probabilities:
− Posterior Probabilities (Bayes’ theorem)
1
c
l
l
N N
D
DV c r x
l
l
l
K
p x
N V
l
l
N
p
N
K
p x
NV
l l l
l
p x P K
P x
p x K
(14)
21. 21
K-NN classifier
− Find the class maximizing the posterior probability (Bayes decision)
− The point x is classified into 𝜔𝑙0
Summary: (Fig.8)
1) Select K data surrounding the estimation point x
2) The point x is assigned the major class of K points in the neighbor.
0 : argmax argmax l
l
l l
K
l P x
K
Nearest Neighbor classifier
Let consider K=1 for K-NN classification.
The point x is classified into the class of the nearest point to x.
→ Nearest Neighbor Classifier
Classification boundary of the K-NN with respect to K
small K → tends to make many small class regions (Fig. 9 (b))
large K → few class regions (Fig.9(a))
(14)
23. 23
4. Cross-Validation
Parameter determination problem:
- Since the classifier have free parameters such as K in K-NN
classifier and h in Kernel density based classifiers, we need to
select the optimal parameters by evaluating the classification
performances.
- Over-fitting problem
- The classifier’s parameter (decision boundary) obtained from
using overall training data will overfit to them
→ Need an appropriate new test data
Cross-Validation
- Given data are split into S parts.
S=4
- Fig. 10
- Use (S-1) parts for training and a rest part is used for testing
- Apply all different parts for the test as shown in Fig. 11 (S=4)
total data
24. 24
training datatest data
Experiment 1
Experiment 2
Experiment 3
Experiment 4
Score * 1
Score 2
Score 3
Score 4
×(1/4)＝Averaged score
If we want to determine the best K for K-NN classifier, we
choose the K providing the highest averaged score by the
cross-validation procedure.
*Score= error rate or conditioned risk et al.
Fig. 11
＋）
25. 25
References:
[1] R.O. Duda, P.E. Hart, and D. G. Stork, “Pattern Classification”,
John Wiley & Sons, 2nd edition, 2004
[2] C. M. Bishop, “Pattern Recognition and Machine Learning”,
Springer, 2006
[3] All data files of Bishop’s book are available at the
“http://research.microsoft.com/~cmbishop/PRML”
