This document provides an introduction to algorithms and algorithm analysis. It defines what an algorithm is, provides examples, and discusses analyzing algorithms to determine their efficiency. Insertion sort and merge sort are presented as examples and their time complexities are analyzed. Asymptotic notation is introduced to describe an algorithm's order of growth and provide bounds on its running time. Key points covered include analyzing best-case and worst-case time complexities, using recurrences to model algorithms, and the properties of asymptotic notation. Homework problems are assigned from the textbook chapters.
2. Introduction to Algorithms & Algorithm
Analysis 2
What is an Algorithm?
Any well-defined computational procedure that takes
some value or set of values as input, and produces
some value or values as output; or
A sequence of computational steps that transforms
the input into the output; or
Tool for solving a well-specified computational
problem
The problem specifies the relationship between input and
output
The algorithm describes a procedure for achieving that
relationship
3. Introduction to Algorithms & Algorithm
Analysis 3
Algorithms: An Example
The Sorting Problem
The need to sort data arises frequently in practice
Formally, the sorting problem is specified as:
Input: A sequence of n numbers {a1, a2, …, an}
Output: A permutation (reordering) {a1’, a2’, …, an’} of
the input sequence such that a1’ <= a2’ <= … <= an’
That is, given the input sequence { 5, 7, 3, 2, 9
}, a sorting algorithm returns as output the
sequence { 2, 3, 5, 7, 9}
This is one instance of the sorting problem
4. Introduction to Algorithms & Algorithm
Analysis 4
Correctness of Algorithms
An algorithm is correct if for every input
instance, it halts with the correct output
i.e., it solves the given computational problem
Correctness isn't everything
Incorrectness can be useful, if the error rate can
be controlled
5. Introduction to Algorithms & Algorithm
Analysis 5
Analyzing Algorithms
To analyze an algorithm means to predict the
resources that an algorithm will require
Why do we care about an algorithm’s
resource requirements?
Resources are bounded – there isn’t an infinite
amount of time or space for an algorithm to
execute in
What resources do we care about?
Time (how long does it take)
Space (how much memory does it use)
6. Introduction to Algorithms & Algorithm
Analysis 6
Analyzing Algorithms
Analysis usually measures two forms of
complexity:
Spatial Complexity: memory, communications
bandwidth
Temporal Complexity: computational time
7. Introduction to Algorithms & Algorithm
Analysis 7
Analyzing Algorithms: An Example
Two algorithms for solving the same problem
often differ dramatically in their efficiency
Consider two sorting algorithms:
Insertion sort takes time roughly equal to c1n2
n is the number of items being sorted;
c1 is a constant that does not depend on n
Merge sort takes time roughly equal to c2nlog2n
8. Introduction to Algorithms & Algorithm
Analysis 8
Analyzing Algorithms: An Example
What does this mean? Consider sorting
1,000,000 numbers on two different
computers
Computer A uses insertion sort, and executes
1,000,000,000 instructions per second; we’ll
assume c1 = 2
Computer B uses merge sort, and executes
10,000,000 instructions per second; we’ll assume
c2 = 50
How long does it take?
9. Introduction to Algorithms & Algorithm
Analysis 9
Analyzing Algorithms: An Example
Computer A:
Computer B:
seconds2000
ns/secinstructio10
nsinstructio102
9
26
seconds100
ns/secinstructio10
nsinstructio10log1050
7
6
2
6
10. Introduction to Algorithms & Algorithm
Analysis 10
Insertion Sort
Insertion sort is one method of solving the
sorting problem
Conceptually, insertion sort works the way
many people sort playing cards:
Start with one hand empty and the cards face
down
Remove one card at a time from the table and
insert it into the correct position in the left hand
11. Introduction to Algorithms & Algorithm
Analysis 11
Insertion Sort: Algorithm
void InsertionSort(ArrayType A[], unsigned size)
{
for ( unsigned j = 1 ; j < size ; ++j )
{
ArrayType key = A[j];
int i = j-1;
while ( i >= 0 && A[i] > key )
{
A[i+1] = A[i];
--i;
}
A[i+1] = key;
}
}
14. Introduction to Algorithms & Algorithm
Analysis 14
Analysis of Insertion Sort
How long does insertion sort take?
Time varies based on:
The size of the input
How well sorted the input is to begin with
What is “the size of the input”?
May represent the number of items in the input, or
the number of bits being calculated
May be represented by more than one number
15. Introduction to Algorithms & Algorithm
Analysis 15
Analysis of Insertion Sort
What is “running time”?
Running time is based on the number of steps
performed by the processor
Different processors run at different speeds, so
time per step will vary between processors
We will express times in terms of a constant cost
per step, ci
16. Introduction to Algorithms & Algorithm
Analysis 16
Analysis of Insertion Sort
What is the “cost” of Insertion Sort?
Total cost = the sum of the cost of each statement *
number of times statement is executed
Statem ent Cost Tim es
for ( unsigned j = 1 ; j < size ; ++j ) c1 n
ArrayType key = A[j]; c2 n-1
int i = j-1; c3 n-1
while ( i >= 0 && A[i] > key ) c4
n
j j
t
1
A[i+1] = A[i]; c5
n
j
t j1
)1(
--i; c6
n
j
t j1
)1(
A[i+1] = key; c7 n-1
17. Introduction to Algorithms & Algorithm
Analysis 17
Analysis of Insertion Sort
In the best case, all tj = 1
The inner loop only executes once, because the array is
already sorted
In this case, T(n)=(c1+c2+c3+c4+c7)n-(c2+c3+c4+c7)
Since all constants are unknown, we can as easily say that
T(n)=an+b, where a and b depend on the cost of each
statement
So in the best case, the running time is a linear function of n
)1()1()()1()1()( 716514321
nctcctcncncncnT
n
j j
n
j j
Total cost for insertion sort is:
18. Introduction to Algorithms & Algorithm
Analysis 18
Analysis of Insertion Sort
In the worst case, tj=j, since every element must be
compared
This occurs when the array is reverse sorted
To solve this, we need the solve the summations:
)1()1()()1()1()( 716514321
nctcctcncncncnT
n
j j
n
j j
n
j
nn
j
1
1
2
)1(
n
j
nn
j
1
2
)1(
)1(
19. Introduction to Algorithms & Algorithm
Analysis 19
Analysis of Insertion Sort
So, in the worst case:
)1(
2
)1(
)(1
2
)1(
)1()1()( 7654321
nc
nn
cc
nn
cncncncnT
)()(
2
1
2
)()( 74326547321
2
654
ccccnccccccc
n
cccnT
We can say that T(n)=ax2+bx+c
The running time in the worst case is therefore a
quadratic function of n
In general, we are most interested in worst case
running time
20. Introduction to Algorithms & Algorithm
Analysis 20
Analysis of Insertion Sort
Order of Growth
Actual running time is not nearly as important as
the “order of growth” of the running time
Order of growth of the running time is how the running
time changes as the size of the input changes
Provides a concrete method of comparing alternative
algorithms
21. Introduction to Algorithms & Algorithm
Analysis 21
Analysis of Insertion Sort
Order of Growth
Order of growth is typically easier to compute than
exact running time
In general, we are most interested in the highest order
terms of the running time
Lower order terms become insignificant as the input size
grows larger
We can also ignore leading coefficients
constant factors are not as significant as growth rate in
determining computational efficiency
22. Introduction to Algorithms & Algorithm
Analysis 22
Analysis of Insertion Sort
Example of Order of Growth:
Best case:
Linear running time: T(n)=an+b
The order of growth is O(n)
This representation (O(n)) is called asymptotic notation
Worst case:
Quadratic running time: T(n)=an2+bn+c
The order of growth is (O(n2))
Order of growth provides us a means of comparing
the efficiency of algorithms
Algorithms with lower worst-case order of growth are
usually considered to be more efficient
24. Introduction to Algorithms & Algorithm
Analysis 24
Designing Algorithms
Insertion sort is typical of an incremental
approach to algorithm design
The input is processed in equally sized
increments
An alternative approach is divide-and-
conquer
The input is recursively divided and processed in
smaller, similar chunks
25. Introduction to Algorithms & Algorithm
Analysis 25
Divide and Conquer
Divide and Conquer is a recursive approach
to algorithm design
Consists of three steps at each level of
recursion:
Divide the problem into smaller, similar
subproblems
Conquer the subproblems by solving them
recursively
Combine the solutions to the subproblems into
the solution to the original problem
27. Introduction to Algorithms & Algorithm
Analysis 27
The Merge Sort
void MergeSort(ArrayType A[], int p, int r)
{
if ( p < r )
{
int q = (p+r)/2; // Divide
MergeSort(A, p, q); // Conquer left
MergeSort(A, q+1, r); // Conquer right
Merge(A, p, q, r); // Combine
}
}
28. Introduction to Algorithms & Algorithm
Analysis 28
Analysis of Merge Sort
How do we describe the running time of
Merge Sort?
We must derive a recurrence
A recurrence describes the overall running time of
an algorithm on a problem of size n in terms of the
running time on smaller inputs
29. Introduction to Algorithms & Algorithm
Analysis 29
Analysis of Merge Sort
What is the recurrence for Merge Sort?
Dividing always takes the same time; it is constant
(represented as O(1))
Conquering occurs twice, on an input half the original size
Conquering therefore takes 2T(n/2)
Merge() is not presented, but should be linear (O(n))
This leads to a total running time of T(n) =
2T(n/2)+O(n)+O(1)
O(1) is constant, and can be dropped
This is an infinite recurrence - when does it stop?
When the input size reaches 1
30. Introduction to Algorithms & Algorithm
Analysis 30
Analysis of Merge Sort
So, the total running time is shown as the
following recurrence:
We shall show later in the course that the
order of growth for this recurrence is
O(nlog2n)
1if)()2/(2
1if)1(
)(
nnOnT
nO
nT
31. Introduction to Algorithms & Algorithm
Analysis 31
Asymptotic Notation
Asymptotic notation provides a means of
bounding the running time of an algorithm
Upper bounds signify worst-case running times
E.g., if T(n) has an upper bound of g(n), this means that
the running time of T(n) is at most cg(n)
Lower bounds signify best-case running times
32. Introduction to Algorithms & Algorithm
Analysis 32
Asymptotic Notation
There are three basic asymptotic notations:
O-notation: denotes the upper bound of an
expression
-notation: denotes the lower bound of an
expression
-notation: denotes both upper and lower
bounds of an expression
34. Introduction to Algorithms & Algorithm
Analysis 34
Asymptotic Notation
Why do we use asymptotic notation?
Reduces clutter by eliminating constant factors
and lower-order terms
E.g., 2n2 + 3n + 1 = 2n2 + O(n) = O(n2)
We are interested in comparing algorithms
Actual running times are very hard to compute, and
compare
Bounds are easier to compute and provide a more
realistic basis for comparison
We can always go back and compute actual running
times if we need them
35. Introduction to Algorithms & Algorithm
Analysis 35
Asymptotic Notation
Asymptotic notation of running times exhibits
some important mathematical properties:
Transitivity: If f(n) = O(g(n)) and g(n) = O(h(n))
then f(n) = O(h(n))
Reflexivity: f(n) = O(f(n))