1. Randomized Algorithms
CS648
Lecture 2
• Randomized Algorithm for Approximate Median
• Elementary Probability theory
1

2. Randomized Monte Carlo Algorithm for
approximate median
This lecture was delivered at slow pace and its flavor was that of a
tutorial.
Reason: To show that designing and analyzing a randomized
algorithm demands right insight and just elementary probability.
2

3. A simple probability exercise
•
There is a coin which gives HEADS with probability ¼ and TAILS with
probability ¾. The coin is tossed times. What is the probability that we get at
least HEADS ?
[Stirling’s approximation for Factorial: ]
3

4. Probability of getting
“at least HEADS in tosses”
Probability of getting at least heads:
•
Using Stirling’s approximation
Since , so …
Inverse exponential in .
4

5. Approximate median
Definition: Given an array A[] storing n numbers and ϵ > 0, compute an
element whose rank is in the range [(1- ϵ)n/2, (1+ ϵ)n/2].
Best Deterministic Algorithm:
• “Median of Medians” algorithm for finding exact median
• Running time: O(n)
• No faster algorithm possible for approximate median
Can you give a short proof ?
5

6. ½ - Approximate median
A Randomized Algorithm
Rand-Approx-Median(A)
1. Let k  c log n;
2. S  ∅;
3. For i=1 to k
4.
x  an element selected randomly uniformly from A;
5.
S  S U {x};
6. Sort S.
7. Report the median of S.
Running time: O(log n loglog n)
6

7. Analyzing the error probability of
Rand-approx-median
n/4
Left Quarter
Elements of A arranged in
Increasing order of values
3n/4
Right Quarter
When does the algorithm err ?
To answer this question, try to characterize what
will be a bad sample S ?
7

8. Analyzing the error probability of
Rand-approx-median
n/4
Elements of A arranged in
Increasing order of values
Left Quarter
Median of S
3n/4
Right Quarter
Observation: Algorithm makes an error only if k/2 or more elements
sampled from the Right Quarter (or Left Quarter).
8

9. Analyzing the error probability of
Rand-approx-median
•
n/4
Elements of A arranged in
Increasing order of values
3n/4
Right Quarter
Left Quarter
Pr[ An element selected randomly from A is from Right quarter] = ¼
??
Pr[ Out of k elements sampled from A, at least k/2 are from Right quarter] = ??
for
Exactly the same as the coin
tossing exercise we did !
9

10. Main result we discussed
•
Theorem: The Rand-approx-median algorithm fails to report ½
-approximate median from array A[1.. ] with probability at most.
Homework: Design a randomized Monte Carlo algorithm for
computing ϵ-approximate median of array A[1.. ] with running
time O(log n loglog n) and error probability for any given
constants ϵ and .
[Do this homework sincerely without any friend’s help.]
10

11. Elementary probability theory
(It is so simple that you underestimate its elegance and power)
11

12. Elementary probability theory
(Relevant for CS648)
•
•
We shall mainly deal with discrete probability theory in this course.
We shall take the set theoretic approach to explain probability theory.
Consider any random experiment :
o Tossing a coin 5 times.
o Throwing a dice 2 times.
o Selecting a number randomly uniformly from [1..n].
How to capture the following facts in the theory of probability ?
1. Outcome will always be from a specified set.
2. Likelihood of each possible outcome is non-negative.
3. We may be interested in a collection of outcomes.
12

13. Probability Space
Definition: Probability space associated with a random experiment is an
ordered pair (Ω,P), where
• Ω is the set of all possible outcomes of the random experiment
• P : Ω R such that
•
–
P(ω) ≥ 0 for each ωϵ Ω
Ω
Elements of Ω are called elementary events or sample points.
13

14. Event in a Probability Space
Definition: An event A in a probability space (Ω,P) is a subset of Ω. The
probability of event A is defined as
•
A
Ω
For sake of compact notation, we extend P for events as described above.
14

15. Exercises
A randomized algorithm can also be viewed as a random experiment.
1. What is the sample space associated with Randomized Quick sort ?
2. What is the sample space associated with Rand-approx-median
algorithm ?
15

16. An Important Advice
In the following slides, we shall state well known equations
(highlighted in yellow boxes) from probability theory.
• You should internalize them fully.
• We shall use them crucially in this course.
• Make sincere attempts to solve exercises that follow.
16

17. Union of two Events
Given two events A and B defined over a probability space (,P), what is
P(AUB) ?
•
A
B
Ω
P(AUB) = P(A) + P(B)
P(A∩B)
Try to prove it by showing the following:
Each ω ϵ AUB contributes exactly P(ω) in the right hand side.
17

18. Union of three Events
Given three events A₁, A₂, A₃, defined over a probability space (,P), what is
P(A₁ U A₂ U A₃) ?
•
A
B
C
Ω
P(A₁ U A₂UA₃) = P(A₁) + P(A₂) + P( A₃)
P(A₁∩A₂) P(A₂∩A₃) P(A₁∩A₃)
+ P(A₁∩A₂∩A₃)
Try to prove this equation as well by showing the following:
Each ω ϵ A₁ U A₂UA₃ contributes exactly P(ω) in the right hand side.
18

19. Exercises
•
•
For events ,…, defined over a probability space (,P), prove that P() =
…
)
•
There are letters envelopes. For each letter, there is a unique envelope in
which it should be placed. A careless postman places the letters randomly
into envelopes (one letter in each envelope). What is the probability that
no letter is placed correctly (into the envelope meant for it) ?
19

20. Conditional Probability
Happening of some event influences the likelihood of happening of other events. This
notion is formally captured by conditional probability as follows.
•
Probability of event A conditioned on event B, compactly represented as P[A|B],
means the following.
Given that event B has happened, what is the probability that event A has also
happened ?
You might have seen and used the following equation for conditional probability.
P[A|B] =
Can you give suitable reason to justify the validity of the above equation ?
In particular, give justification for ] in numerator and ] in denominator in this
equation.
20

21. Exercises
•
A man possesses five coins, two of which are double-headed, one is
double-tailed, and two are normal. He shuts his eyes, picks a coin at
random, and tosses it. What is the probability that the lower face of the
coin is a head ? He opens his eyes and sees that the coin is showing heads;
what it the probability that the lower face is a head ? He shuts his eyes
again, and tosses the coin again. What is the probability that the lower
face is a head ? He opens his eyes and sees that the coin is showing heads;
what is the probability that the lower face is a head ? He discards this
coin, picks another at random, and tosses it. What is the probability that it
shows heads ?
21

22. Partition of sample space and
an “important Equation”
A set of events ,…, defined over a probability space (,P) is said to induce a
partition of if
• =
•
•
=∅ for all
B
Ω
Given an event B, how can we express P(B) in terms of a given partition ?
P(B) = )
22

23. Exercises
•
•
There are sticks each of different heights. There are vacant slots arranged
along a line and numbered from 1 to as we move from left to right. The
sticks are placed into the slots according to a uniformly random
permutation. A stick placed at th slot is said to be a dominating stick if its
height is largest among all sticks placed in slots 1 to . Find the probability
that th slot contains a dominating stick.
23

24. Independent Events
Two events A and B defined over a probability space (,P) are said to be
independent if happening of one of them has no influence on the probability
of the another event. Mathematically, it means that
P(A|B)= P(A) and P(B|A)=P(B)
•
The following equation also compactly captures independence of two events.
P(A ∩ B) = P(A) · P(B)
Question: Can two independent events ever be disjoint ?
24

25. Exercises
•
1.
Two fair dice are rolled. Show that the event that their sum is 7 is
independent of the score shown by the first die.
2.
Let (,P) be a probability space where = {1,2,…,p} for a given prime
number p, and each elementary event has probability 1/p. Show that if
two events A and B defined over (,P) are independent, then at least one
of A and B is either ∅ or .
25