Lecture 3

Asymptotic Notation

The result of the analysis of an algorithm is usually a formula giving the amount of time, in terms of seconds, number of memory accesses, number of comparisons or some other metric, that the algorithm takes. This formula often contains unimportant details that don't really tell us anything about the running time. For instance, when we analyzed selection sort, we found that it took T(n) = n² + 3n - 4 array accesses. For large values of n, the 3n - 4 part is insignificant compared to the n² part.

When comparing the running times of two algorithms, these lower order terms are unimportant when the higher order terms are different. Also unimportant are the constant coefficients of higher order terms; an algorithm that takes a time of 100n² will still be faster than an algorithm that takes n³ for any value of n larger than 100. Since we're interested in the asymptotic behavior of the growth of the function, the constant factor can be ignored.

Upper Bounds: Big-O

We need a formal way of expressing these intuitive notions. One popular way is "big-Oh" notation. It tells us that a certain function will never exceed another, simpler function beyond a constant multiple and for large enough values of n. For example, we can simplify 3n² + 4n - 10 to O(n²). We write "3n² + 4n - 10 = O(n²)" and say "three n squared plus four n minus ten is big-Oh of n squared." We might also say "...is in big-Oh..," but we don't say "...is equal to big-Oh..," the equal sign in this case is more like a set-membership sign.

Let's define big-Oh more formally:

O(g(n)) = { the set of all f such that there exist positive constants c and n₀ satisfying 0 <= f(n) <= cg(n) for all n >= n₀ }.

This means that, for example, O(n²) is a set, or family, of functions like n² + n, 4n² - n log n + 12, n²/5 - 100n, n log n, 50n, and so forth. Every function f(n) bounded above by some constant multiple g(n) for all values of n greater than a certain value is in O(g(n)).

Examples:

Show 3n² + 4n - 2 = O(n²).
We need to find c and n₀ such that:
3n² + 4n - 2 <= cn² for all n >= n₀ .
Divide both sides by n², getting:
3 + 4/n - 2/n² <= c for all n >= n₀ .
If we choose n₀ equal to 1, then we need a value of c such that:
3 + 4 - 2 <= c
We can set c equal to 6. Now we have:
3n² + 4n - 2 <= 6n² for all n >= 1 .
Show n³ != O(n²). Let's assume to the contrary that
n³ = O(n²)
Then there must exist constants c and n₀ such that
n³ <= cn² for all n >= n₀.
Dividing by n², we get:
n <= c for all n >= n₀.
But this is not possible; we can never choose a constant c large enough that n will never exceed it, since n can grow without bound. Thus, the original assumption, that n³ = O(n²), must be wrong so n³ != O(n²).

Big-Oh gives us a formal way of expressing asymptotic upper bounds, a way of bounding from above the growth of a function. Knowing where a function falls within the big-Oh hierarchy allows us to compare it quickly with other functions and gives us an idea of which algorithm has the best time performance.

Lower Bounds: Omega

Another way of grouping functions, like big-Oh, is to give an asymptotic lower bound. Given a complicated function f, we find a simple function g that, within a constant multiple and for large enough n, bounds f from below. Define

(g(n)) = { the set of all f such that there exist positive constants c and n₀ satisfying 0 <= cg(n) < f(n) for all n >= n₀ }.

This gives us a somewhat different family of functions; now i any function f that grows strictly faster than g is in

(g). So, for example, n³ =

(n²).

Tight Bounds: Theta

Neither big-Oh or Omega are completely satisfying; we would like a tight bound on how quickly our function grows. To say it doesn't grow any faster than something doesn't help us know how slowly it grows, and vice-versa. So we need something to give us a tigher bound; something that bounds a function from both above and below. We can combine big-Oh and Omega to give us a new set of functions, Theta:

(g(n)) = { the set of functions f(n) such that f(n) = O(g(n)) and f(n) = (g(n)) } .

It is equivalent to say:

(g(n)) = { the set of all f such that there exist positive constants c₁, c₂ and n₀ satisfying 0 <= c₁g(n) <= f(n) <= c₂g(n) for all n >= n₀ }.

Whenever possible, we try to bound the running time of an algorithm from both above and below with Theta notation.

Convenient Theorems

For polynomials, and indeed a large class of functions that are the sums of monotonically increasing terms, we can simplify the process of O etc. notation by noticing these rules. Below, p and q are arbitrary polynomials:

c p(n) = (p(n)).
p(n) + q(n) = (max (p(n), q(n)) .
n^k = O((1 + )ⁿ) for any constant k and any positive constant (i.e., any polynomial is bounded by any exponential).
Readings: Read Chapters 4, 6 and 7 in the book.