2. Syllabus
Notion of an Algorithm
Fundamentals of the Analysis of Algorithm Efficiency
Asymptotic Notations and its properties
Performance measurements of Algorithm, Time and space trade-offs
Analysis of recursive algorithms through recurrence relations
Substitution method
Recursion tree method
Masters’ theorem
3/7/2022
Design and Analysis of Algorithm - UNIT 1
2
3. Introduction to Algorithm with 2 examples GCD and Prime
3/7/2022
Design and Analysis of Algorithm - UNIT 1
3
4. ALGORITHM
An algorithm is an exact specification of how to solve a computational problem
An algorithm must specify every step completely, so a computer can implement
it without any further “understanding”
An algorithm must work for all possible inputs of the problem.
Algorithms must be:
• Correct: For each input produce an appropriate output
• Efficient: run as quickly as possible, and use as little memory as possible – more
about this later
There can be many different algorithms for each computational problem.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
4
5. 1-5
Notion of an algorithm
An algorithm is a sequence of unambiguous
instructions for solving a problem, i.e., for obtaining
a required output for any legitimate input in a finite
amount of time.
“computer”
problem
algorithm
input output
3/7/2022
Design and Analysis of Algorithm - UNIT 1
6. 1-6
Properties of an algorithm
1. Finiteness
terminates after a finite number of steps
2. Definiteness
rigorously and unambiguously specified
3. Input
valid inputs are clearly specified
4. Output
can be proved to produce the correct output given a valid input
5. Effectiveness
steps are sufficiently simple and basic
3/7/2022
Design and Analysis of Algorithm - UNIT 1
7. 1-7
Some Well-known Computational Problems
Sorting
Searching
Shortest paths in a graph
Minimum spanning tree
Traveling salesman problem
Knapsack problem
Chess
Towers of Hanoi
3/7/2022
Design and Analysis of Algorithm - UNIT 1
8. 1-8
Basic Issues Related to Algorithms
How to design algorithms
How to express algorithms
Proving correctness
Efficiency
Optimality
3/7/2022
Design and Analysis of Algorithm - UNIT 1
9. 1-9
Algorithm design strategies
Brute force
Greedy approach
Divide and conquer
Dynamic programming
Backtracking and Branch and bound
Decrease and conquer
Transform and conquer
3/7/2022
Design and Analysis of Algorithm - UNIT 1
10. 1-10
Analysis of Algorithms
How good is the algorithm?
Correctness
Time efficiency
Space efficiency
3/7/2022
Design and Analysis of Algorithm - UNIT 1
13. 1-13
Euclid’s Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?
Euclid’s algorithm is based on repeated application of equality
gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the problem
trivial.
Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12
3/7/2022
Design and Analysis of Algorithm - UNIT 1
14. 1-14
Two descriptions of Euclid’s algorithm
Step 1 If n = 0, return m and stop; otherwise go to Step 2
Step 2 Divide m by n and assign the value fo the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to
Step 1.
while n ≠ 0 do
r ← m mod n
m← n
n ← r
return m
3/7/2022
Design and Analysis of Algorithm - UNIT 1
15. 1-15
Other methods for computing gcd(m,n)
Consecutive integer checking algorithm
Step 1 Assign the value of min{m,n} to t
Step 2 Divide m by t. If the remainder is 0, go to Step 3;
otherwise, go to Step 4
Step 3 Divide n by t. If the remainder is 0, return t and stop;
otherwise, go to Step 4
Step 4 Decrease t by 1 and go to Step 2
3/7/2022
Design and Analysis of Algorithm - UNIT 1
16. 3/7/2022
Design and Analysis of Algorithm - UNIT 1
16
Step 1 Assign the value of min{m,n} to t
Step 2 Divide m by t. If the remainder is 0, go to
Step 3;
otherwise, go to Step 4
Step 3 Divide n by t. If the remainder is 0, return
t and stop;
otherwise, go to Step 4
Step 4 Decrease t by 1 and go to Step 2
17. 1-17
Other methods for gcd(m,n) [cont.]
Middle-school procedure
Step 1 Find the prime factorization of m
Step 2 Find the prime factorization of n
Step 3 Find all the common prime factors
Step 4 Compute the product of all the common prime factors
and return it as gcd(m,n)
60 = 2 . 2 . 3 . 5
24 = 2 . 2 . 2 . 3
gcd(60, 24) = 2 . 2 . 3 = 12.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
18. Prime No
Sieve of Eratosthenes
Standard method based on divisors
3/7/2022
Design and Analysis of Algorithm - UNIT 1
18
19. Sieve of Eratosthenes - To find out all
primes under n
1. Generate a list of all integers from 2 to n. (Note: 1 is not prime)
2. Start with a smallest prime number, ie p = 2.
3. Mark all the multiples of p which are less than n as composite.
mark the value of the numbers (multiples of p) in the generated list
as 0.
Do not mark p itself as composite.
4. Assign the value of p to the next prime. The next prime is the next non-
zero number in the list which is greater than p.
5. Repeat the process until p≤ sqrt (n).
3/7/2022
Design and Analysis of Algorithm - UNIT 1
19
20. Let’s start with 2. 2 is a prime number but all of the multiples of 2 will be
composite numbers since they will be divisible by 2. We cross out all of the
multiples of 2 on the table.
The next prime number is 3, so we can cross out all the multiples of 3 since
they will be composite numbers.
After 3, is the next prime number 5, so we cross out all of the multiples of 5.
Then we have the prime number 7 and we cross out all of the multiples of
7.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
20
24. 1-24
Sieve of Eratosthenes
Input: Integer n ≥ 2
Output: List of primes less than or equal to n
for p ← 2 to n do A[p] ← p
for p ← 2 to n do
if A[p] 0 //p hasn’t been previously eliminated from the list
j ← p* p
while j ≤ n do
A[j] ← 0 //mark element as eliminated
j ← j + p
3/7/2022
Design and Analysis of Algorithm - UNIT 1
25. Algorithm
Design &
Analysis
Steps
Prove that gcd(m, n) = gcd(n, m mod n)
Design Technique,
Data Structures
Time Efficiency & Space
Efficiency, simplicity, generality
25
3/7/2022
Design and Analysis of Algorithm - UNIT 1
25
26. 1-26
Fundamental data structures
list
array
linked list
string
stack
queue
priority queue
Graph
tree
set and dictionary 3/7/2022
Design and Analysis of Algorithm - UNIT 1
27. Fundamentals of the Analysis of Algorithm
Efficiency
3/7/2022
Design and Analysis of Algorithm - UNIT 1
27
28. 1-28
Example of computational problem: sorting
Statement of problem:
Input: A sequence of n numbers <a1, a2, …, an>
Output: A reordering of the input sequence <a´
1, a´
2, …, a´
n> so that a´
i ≤ a´
j whenever
i < j
Instance: The sequence <5, 3, 2, 8, 3>
Algorithms:
Selection sort
Insertion sort
Merge sort
(many others)
• There may exist several algorithms for solving the same problem.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
29. Efficiency
Analysis of algorithms means to investigate an algorithm’s efficiency with respect to
resources: running time and memory space.
Time efficiency
a measure of amount of time for an algorithm to execute.
how fast an algorithm runs
Space efficiency
a measure of the amount of memory needed for an algorithm to execute.
Typically, algorithms run longer as the size of its input increases
3/7/2022
Design and Analysis of Algorithm - UNIT 1
29
30. Which algorithm is better?
The algorithms are correct, but
which is the best?
Measure the running time
(number of operations
needed).
Measure the amount of
memory used.
Note that the running time of
the algorithms increase as the
size of the input increases.
3/7/2022
30
Design and Analysis of Algorithm - UNIT 1
31. HOW DO WE COMPARE ALGORITHMS?
We need to define a number of objective measures.
(1) Compare execution times?
Not good: times are specific to a particular computer !!
(2) Count the number of statements executed?
Not good: number of statements vary with the programming language
as well as the style of the individual programmer.
Engineered for Tomorrow
3/7/2022
Design and Analysis of Algorithm - UNIT 1
31
32. Method – I OLD
One possible approach is to count the number of times each of the algorithm’s operations is
executed.
Difficult and usually unnecessary
Method- II
Count the number of times an algorithm’s basic operation is executed.
Basic operation: the operation that contributes the most to the total running time.
For example, the basic operation is usually the most time-consuming operation in the
algorithm’s innermost loop.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
32
33. Example - Old method
• Associate a "cost" with each statement.
• Find the "total cost“ by finding the total number of times each statement is executed.
Algorithm 1 Cost Algorithm 2 Cost
arr[0] = 0; c1 for(i=0; i<N; i++) c2
arr[1] = 0; c1 arr[i] = 0; c1
arr[2] = 0; c1
... ...
arr[N-1] = 0; c1
----------- -------------
c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 = (c2 + c1) x N + c2
Engineered for Tomorrow
3/7/2022
Design and Analysis of Algorithm - UNIT 1
33
34. 34
Another Example - Old method
• Algorithm 3 Cost
sum = 0; c1
for(i=0; i<N; i++) c2
for(j=0; j<N; j++) c2
sum += arr[i][j]; c3
------------
c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N2
2N2 + 2N + 2
Engineered for Tomorrow
3/7/2022
Design and Analysis of Algorithm - UNIT 1
34
35. Method – II – based on basic operation
Almost all algorithms run longer on larger inputs.
Therefore, it is logical to investigate an algorithm’s efficiency as a function of some parameter n
indicating the algorithm’s input size.
Input size depends on the problem.
Example1: what is the input size of the problem of sorting n numbers?
Example2: what is the input size of adding two n by n matrices?
3/7/2022
Design and Analysis of Algorithm - UNIT 1
35
36. Method II contd..
Time efficiency is analyzed by determining the number of repetitions of the basic operation
as a function of input size
Basic operation: the operation that contributes most towards the running time of the
algorithm
T(n) ≈ copC(n)
running time execution time
for basic operation
Number of times
basic operation is
executed
input size
3/7/2022
Design and Analysis of Algorithm - UNIT 1
36
37. Example – Method - II
• Associate a "cost" with each statement.
• Find the "total cost“ by finding the total number of times each statement is executed.
Algorithm 1 Cost
for(i=0; i<N; i++)
arr[i] = 0; c2
N x c2
Engineered for Tomorrow
3/7/2022
VIT INTERVIEW
37
38. 38
Another Example – Method-II
• Algorithm 2 Cost
sum = 0;
for(i=0; i<N; i++)
for(j=0; j<N; j++)
sum += arr[i][j]; c3
------------
c3 x N2
Engineered for Tomorrow
3/7/2022
VIT INTERVIEW
38
39. Input size and basic operation examples
Problem Input size measure Basic operation
Searching for key in
a list of n items
Number of list’s items,
i.e. n
Key comparison
Multiplication of two
matrices
Matrix dimensions or
total number of
elements
Multiplication of
two numbers
Checking primality
of a given integer n
n’size = number of
digits (in binary
representation)
Division
Typical graph
problem
#vertices and/or edges
Visiting a vertex or
traversing an edge
Spell checking
No of chars/ No of
words
comparison
Polynomial Equation
addition
Degree Addition
3/7/2022
40. Types of formulas for basic operation’s count
Exact formula
e.g., C(n) = n(n-1)/2
Formula indicating order of growth with specific multiplicative constant
e.g., C(n) ≈ 0.5 n2
Formula indicating order of growth with unknown multiplicative constant
e.g., C(n) ≈ cn2
3/7/2022
Design and Analysis of Algorithm - UNIT 1
40
41. Best-case, average-case, worst-case
For some algorithms efficiency depends on form of input:
Worst case: Cworst(n) – maximum over inputs of size n
for the worst case input of size n
The algorithm runs the longest among all possible inputs of size n
Best case: Cbest(n) – minimum over inputs of size n
best case input of size n
The algorithm runs the fastest among all possible inputs of size n
Average case: Cavg(n) – “average” over inputs of size n
NOT the average of worst and best case
typical/random input of size n.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
41
42. Example: Sequential search
Input Size : n
Basic Operation: Comparison with key value (not equal to)
Best case
Worst case
Average case
3/7/2022
43. • Best Case: Case 1: The key matches with the first element
• Determine the kind of inputs for which C(n) count will be smallest among all possible inputs of size n
• our array was [1, 2, 3, 4, 5] and we are finding if "1" is present in the array
• T(n) = 1
• Worst Case: Case 2: Key does not exist
• Determine the kind of inputs for which C(n) count will be largest among all possible inputs of size n
• For any instance of size n, the running time of algorithm will not exceed Cworst(n)
• if the given array is [1, 2, 3, 4, 5] and we try to find if element "6" is present in the array
• Cworst(n) = n
T(n) ≈ cop C(n)
Ignore Cop
So
T(n) = n
3/7/2022
44. Avg Case: Case 3: The key is present at any location
in the array
running time for all possible inputs,
3/7/2022
46. Complexity Analysis : Summary
Design and Analysis of Algorithm - UNIT 1
46
3/7/2022
Case
Number of key
comparisons
Asymptotic
complexity
Remark
Case 1 T(n) = 1 T(n) = O(1) Best case
Case 2 T(n) = n T(n) = O(n) Worst case
Case 3 T(n) = O(n) Average case
2
1
)
(
n
n
T
48. 3/7/2022
Design and Analysis of Algorithm - UNIT 1
48 Asymptotic Notations
• To compare two algorithms with running times f(n) and g(n), we need a rough measure that
characterizes how fast each function grows.
• Asymptotic notations are the mathematical notations used to describe the running time
of an algorithm
• The asymptotic notation of an algorithm is classified into 3 types:
• Big Oh notation(O): (Asymptotic Upper bound)
• Big Omega notation(Ω): (Asymptotic Lower bound)
• Big Theta notation(θ): (Asymptotic Tight bound)
50. Basic asymptotic efficiency classes g(n)
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
51. Big-Oh
A function t(n) is said to be in O(g(n)), denoted t(n) O(g(n)),
if t(n) is bounded above by some constant multiple of g(n) for
all large n, i.e., if there exist some positive constant c and
some nonnegative integer n0 such that
t(n) ≤ cg(n) for all n n0
Examples:
10n is O(n2)
5n+20 is O(n)
57. -notation Big-omega
Formal definition
A function t(n) is said to be in (g(n)), denoted t(n) (g(n)), if
t(n) is bounded below by some constant multiple of g(n) for all
large n, i.e., if there exist some positive constant c and some
nonnegative integer n0 such that
t(n) cg(n) for all n n0
Exercises: prove the following using the above definition
10n2 (n2)
0.3n2 - 2n (n2)
0.1n3 (n2)
60. -notation Big-theta
Formal definition
A function t(n) is said to be in (g(n)), denoted t(n)
(g(n)), if t(n) is bounded both above and below
by some positive constant multiples of g(n) for all
large n, i.e., if there exist some positive constant c1
and c2 and some nonnegative integer n0 such that
c2 g(n) t(n) c1 g(n) for all n n0
Exercises: prove the following using the above definition
10n2 (n2)
0.3n2 - 2n (n2)
(1/2)n(n+1) (n2)
64. Order of growth
Most important: Order of growth within a constant multiple as n→∞
Example:
How much faster will algorithm run on computer that is
twice as fast?
How much longer does it take to solve problem of
double input size?
66. Asymptotic order of growth
A way of comparing functions that ignores constant factors and small input sizes
O(g(n)): class of functions f(n) that grow no faster than g(n)
Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)
67. Establishing order of growth using limits
lim T(n)/g(n) =
0 order of growth of T(n) < order of growth of g(n)
c > 0 order of growth of T(n) = order of growth of g(n)
∞ order of growth of T(n) > order of growth of g(n)
n→∞
68. L’Hôpital’s rule and Stirling’s formula
L’Hôpital’s rule:
Stirling’s formula: n! (2n)1/2 (n/e)n
f(n)
g(n)
lim
n
=
f ´(n)
g ´(n)
lim
n
72. Some properties of asymptotic order of growth
f(n) O(f(n))
f(n) O(g(n)) iff g(n) (f(n))
If f (n) O(g (n)) and g(n) O(h(n)) , then f(n) O(h(n))
Note similarity with a ≤ b
If f1(n) O(g1(n)) and f2(n) O(g2(n)) , then
f1(n) + f2(n) O(max{g1(n), g2(n)})
73. Time efficiency of nonrecursive algorithms
General Plan for Analysis
Decide on parameter n indicating input size
Identify algorithm’s basic operation
Determine worst, average, and best cases for input of size n
Set up a sum for the number of times the basic operation is executed
Simplify the sum using standard formulas and rules (see Appendix A of book)
74. Useful summation formulas and rules
liu1 = 1+1+…+1 = u - l + 1
In particular, liu1 = n - 1 + 1 = n (n)
1in i = 1+2+…+n = n(n+1)/2 n2/2 (n2)
1in i2 = 12+22+…+n2 = n(n+1)(2n+1)/6 n3/3 (n3)
0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a 1
In particular, 0in 2i = 20 + 21 +…+ 2n = 2n+1 - 1 (2n )
(ai ± bi ) = ai ± bi cai = cai liuai = limai + m+1iuai
78. Space Efficiency
Space Complexity of an algorithm denotes the total space used or needed by the
algorithm for its working, for various input sizes
#include<stdio.h>
int main()
{
int a = 5, b = 5, c;
c = a + b;
printf("%d", c);
}
3 integer variables are used. The size of the integer data type is 2 or 4 bytes which depends
on the compiler. Now, lets assume the size as 4 bytes. So, the total space occupied by the
above-given program is 4 * 3 = 12 bytes
Hence, space complexity for the above-given program is O(1), or constant.
3/7/2022
Design and Analysis of Algorithm - UNIT 1
78
79. Example
#include <stdio.h>
int main()
{
int n, i, sum = 0;
scanf("%d", &n);
int arr[n];
for(i = 0; i < n; i++)
{
scanf("%d", &arr[i]);
sum = sum + arr[i];
}
printf("%d", sum);
}
3/7/2022
Design and Analysis of Algorithm - UNIT 1
79
In the above-given code, the array consists of n integer
elements.
So, the space occupied by the array is 4 * n.
Also we have integer variables such as n, i and sum.
Assuming 4 bytes for each variable, the total space occupied
by the program is 4n + 12 bytes.
Since the highest order of n in the equation 4n + 12 is n,
so the space complexity is O(n) or linear.
80. Time Space Trade Off
3/7/2022
Design and Analysis of Algorithm - UNIT 1
80
81. Time Space Trade off
In computer science, a space-time or time-memory tradeoff is a way of solving a
problem in
Less time by using more memory
very little space by spending a long time
Types of Trade Off
Compressed / Uncompressed Data
Smaller Code / Loop Unrolling
Lookup Table / Recalculation
3/7/2022
Design and Analysis of Algorithm - UNIT 1
81
82. Compressed / Uncompressed Data
A space -time trade off can be applied to the problem of data storage.
If data is stored uncompressed, it takes more space but less time.
If the data is stored compressed, it takes less space but more time to run the
decompression algorithm.
Smaller Code(with loop) / Larger Code (without loop)
Smaller code occupies less space in memory but it requires high computation time which is
required for jumping back to the beginning of the loop at the end of each iteration.
Larger code or loop unrolling can be traded for higher program speed.
It occupies more space in memory but requires less computation time. (as we need not
perform jumps back and forth since there is no loop.)
Lookup Table / Recalculation
In lookup table, an implementation can include the entire table which reduces computing
time but increases the amount of memory needed.
It can recalculate i.e. compute table entries as needed, increasing computing time but
reducing memory requirements
3/7/2022
Design and Analysis of Algorithm - UNIT 1
82
85. MATHEMATICALANALYSIS OF RECURSIVE
ALGORITHMS
Decide on a parameter indicating an input’s size.
Identify the algorithm’s basic operation.
Check whether the number of times the basic op. is executed may
vary on different inputs of the same size. (If it may, the worst, average,
and best cases must be investigated separately.)
Set up a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
Solve the recurrence (or, at the very least, establish its solution’s order
of growth) by backward substitutions or another method.
86. Example 1: Recursive evaluation of n!
Definition: n ! = 1 2 … (n-1) n for n ≥ 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1) n for n ≥ 1 and
F(0) = 1
87. Recursion
• To see how the recursion works, let’s break down the
factorial function to solve factorial(3)
Engineered for Tomorrow
91. Analysis of Counting # of bits
Recursive relation including initial conditions
A(n) = A(floor(n/2)) + 1
IC A(1) = 0
substitute n = 2k (also k = lg(n))
A(2k) = A(2k-1) + 1 and IC A(20) = 0
A(2k) = [A(2k-2) + 1] + 1 = A(2k-2) + 2
= [A(2k-3) + 1] + 2 = A(2k-3) + 3
...
= A(2k-i) + i
...
= A(2k-k) + k
=A(20) +k
= k
Substitute back k = lg(n)
A(n) = lg(n)
A(n) ε Θ(lg n) 3/7/2022
92. Given: Three Pegs A, B and C
Peg A initially has n disks, different size, stacked up,
larger disks are below smaller disks
Problem: to move the n disks to Peg C, subject to
1. Can move only one disk at a time
2. Smaller disk should be above larger disk
3. Can use other peg as intermediate
Example 3: Tower of Hanoi
A B C A B C
93.
94. Tower of Hanoi
How to Solve: Strategy…
Generalize first: Consider n disks for all n 1
Our example is only the case when n=4
Look at small instances…
How about n=1
Of course, just “Move disk 1 from A to C”
How about n=2?
1. “Move disk 1 from A to B”
2. “Move disk 2 from A to C”
3. “Move disk 1 from B to C”
95. Tower of Hanoi (Solution!)
General Method:
First, move first (n-1) disks from A to B
Now, can move largest disk from A to C
Then, move first (n-1) disks from B to C
Try this method for n=3
1. “Move disk 1 from A to C”
2. “Move disk 2 from A to B”
3. “Move disk 1 from C to B”
4. “Move disk 3 from A to C”
5. “Move disk 1 from B to A”
6. “Move disk 2 from B to C”
7. “Move disk 1 from A to C”
96. Algorithm for Towel of Hanoi (recursive)
Recursive Algorithm
when (n=1), we have simple case
Else (decompose problem and make recursive-calls)
Hanoi(n, A, B, C);
(* Move n disks from A to C via B *)
begin
if (n=1) then “Move top disk from A to C”
else (* when n>1 *)
Hanoi (n-1, A, C, B);
“Move top disk from A to C”
Hanoi (n-1, B, C, A);
endif
end;
98. Fibonacci numbers
The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, …
The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2)
F(0) = 0
F(1) = 1
General 2nd order linear homogeneous recurrence with
constant coefficients:
aX(n) + bX(n-1) + cX(n-2) = 0
99. Algorithm Fibonacci (Recursive)
Input: A positive integer n
Output: The sequence of Fibonacci numbers
procedure Fibonacci(int n)
if (n == 0 or n == 1)
return n;
else
return Fibonacci(n-1) + Fibonacci(n-2)