Data Analysis with R (combined slides)

R Programming
Guy Lebanon
September 22, 2015

Goals
I Understand when to use R and when not to use it
I Understand basic syntax and be able to write short programs
I Understand scalability issues in R and di↵erent ways to resolve
them
I Prepare for the next module: visualizing data with R
Module will be separated to 4 parts: (a) getting started, (b) data
types, (c) control ﬂow and functions, and (c) scalability and
interfaces.

R, Matlab, and Python
R is similar to Matlab and Python:
I They run inside an interactive shell or graphical user interface
I They emphasize storing and manipulating data as
multidimensional arrays.
I They include many general purpose and specialized packages
(linear algebra, statistics, ML, etc.)
I They are typically slower than C, C++, and Fortran (though
vectorization can help)
I They can interface with native C++ code for speeding up
bottlenecks

R, Matlab, and Python
The three languages di↵er:
I R and Python are open-source and free. Matlab is not.
I It is easier to contribute packages to R
I R has a large group of motivated contributors who contribute
high quality packages
I R syntax is more suitable for statistics and data
I R has better graphics capabilities
I R is popular in statistics, biostatistics, and social sciences.
Matlab is popular in engineering and applied math. Python is
popular in web development and scripting.

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
100
1000
10000
2002 2004 2006 2008 2010 2012 2014
Near exponential growth in contributed packages

Running R
I Interactively:
I Type R in prompt (type q() to quit)
I R graphic application
I R-Studio
I Within Emacs
I Non-Interactively:
I call script from R: source("foo.R")
I call script from shell: R CMD BATCH foo.R
I call script from shell: Rscript foo.R
I executable script, preﬁxed by #!/usr/bin/Rscript, followed
by ./foo.R < inFile > outFile

R Language
I drops whitespace, semi-colons optional but are needed for
multiple commands in the same line
I comments: #
I case sensitive
I functional and object oriented programming (a=b rephrased as
’=’(a,b))
I interpreted but with lazy evaluation
I not strongly typed
I help() displays help on a function, dataset, etc.
a = 3.2
a = "a string"; b = 2 # no strong typing
print(a)
## [1] "a string"

ls() # list variable names in workspace memory
# save all variables to a file
save.image(file = "R_workspace")
# save specified variables
save(new.var, legal.var.name, file = "R_workspace")
# load variables saved in file
load("R_workspace")
help("load")
install.packages("ggplot2")
library(ggplot2)
system("ls -al")

Scalars
Major scalar types: numeric, integer, logical, string, dates, and
factors (NA: not available)
a = 3.2; b = 3 # double types
c = as.integer(b) # cast to integer type
d = TRUE
e = as.numeric(d) # casting to numeric
f = "this is a string" # string
ls.str() # show variables and their types
## a: num 3.2
## b: num 3
## c: int 3
## d: logi TRUE
## e: num 1
## f: chr "this is a string"

Factors can be ordered or unordered
# ordered factor
current.season = factor("summer",
levels = c("summer", "fall", "winter", "spring"),
ordered = TRUE)
# unordered factor
my.eye.color = factor("brown",
levels = c("brown", "blue", "green"), ordered = FALSE)

Vectors and Arrays
x = c(4, 3, 3, 4, 3, 1) # c for concatenate
length(x) # return length
2*x+1 # element-wise arithmetic
# Boolean vector (default is FALSE)
y = vector(mode = "logical", length = 4)
# numeric vector (default is 0)
z = vector(length = 3, mode = "numeric")

q = rep(3.2, times = 10) # repeat value multiple times
w = seq(0, 1, by = 0.1) # values in [0,1] in 0.1 increments
w = seq(0, 1, length.out = 11) # equally spaced values
w <= 0.5 # boolean vector
any(w <= 0.5) # is it true for some elements?
all(w <= 0.5) # is it true for all elements?
which(w <= 0.5) # for which elements is it true?
w[w <= 0.5] # extracting from w entries for which w<=0.5
subset(w, w <= 0.5) # an alternative with the subset function
w[w <= 0.5] = 0 # zero out all components <= 0.5

Arrays are multidimensional generalization of vectors.
z = seq(1, 20,length.out = 20) # create a vector 1,2,..,20
x = array(data = z, dim = c(4, 5)) # create a 2-d array
x[2,3] # refer to the second row and third column
x[2,] # refer to the entire second row
x[-1,] # all but the first row - same as x[c(2,3,4),]
y = x[c(1,2),c(1,2)] # 2x2 top left sub-matrix
2 * y + 1 # element-wise operation
y %*% y # matrix product (both arguments are matrices)
x[1,] %*% x[1,] # inner product
t(x) # matrix transpose
outer(x[,1], x[,1]) # outer product
rbind(x[1,], x[1,]) # vertical concatenation
cbind(x[1,], x[1,]) # horizontal concatenation

Lists
Lists are ordered collections of possibly di↵erent types. Named
positions allow creating self-describing data.
L=list(name = 'John', age = 55,
no.children = 2, children.ages = c(15, 18))
names(L) # displays all position names
L[[2]] # second element
L[2] # list containing second element
L$name # value in list corresponding to name
L['name'] # same thing
L$children.ages[2] # same as L[[4]][2]

Dataframes
Dataframe are ordered sequence of lists sharing the same
signature. A popular usecase is a table where rows correspond to
data examples and columns correspond to dimensions or features.
vecn = c("John Smith","Jane Doe")
veca = c(42, 45)
vecs = c(50000, 55000)
R = data.frame(name = vecn, age = veca, salary = vecs)
R
## name age salary
## 1 John Smith 42 50000
## 2 Jane Doe 45 55000
names(R) = c("NAME", "AGE", "SALARY") # modify column names
R
## NAME AGE SALARY
## 1 John Smith 42 50000
## 2 Jane Doe 45 55000

Datasets
Example: Iris dataset (in datasets package)
names(iris) # lists the dimension (column) names
head(iris, 4) # show first four rows
iris[1,] # first row
iris$Sepal.Length[1:10] # sepal length of first ten samples
# allow replacing iris£Sepal.Length with shorter Sepal.Length
attach(iris, warn.conflicts = FALSE)
mean(Sepal.Length) # average of Sepal.Length across all rows
colMeans(iris[,1:4]) # means of all four numeric columns
subset(iris, Sepal.Length < 5 & Species != "setosa")
# count number of rows corresponding to setosa species
dim(subset(iris, Species == "setosa"))[1]
summary(iris)

If-Else
a = 10; b = 5; c = 1
if (a < b) {
d = 1
} else if (a == b) {
d = 2
} else {
d = 3
}
print(d)
## [1] 3
AND: &&, OR: ||, equality: ==, inequality: !=

Loops
For, repeat, and while loops:
sm=0
# repeat for 100 iteration, with num taking values 1:100
for (num in seq(1, 100, by = 1)) {
sm = sm + num
}
repeat {
sm = sm - num
num = num - 1
if (sm == 0) break # if sm == 0 then stop the loop
}
a = 1; b = 10
while (b>a) {
sm = sm + 1
a = a + 1
b = b - 1
}

Functions
By default, arguments ﬂow into the parameters according to their
order at the call site. Providing parameter names allow out of
order binding.
foo(10, 20, 30) # parameter bindings by order
foo(y = 20, x = 10, z = 30) # out of order parameter bindings
foo(z = 30) # missing parameters assigned default values

# myPower(.,.) raises the first argument to the power of the
# second. The first argument is named bas and has default value
# The second parameter is named pow and has default value 2.
myPower = function(bas = 10, pow = 2) {
res = bas^pow # raise base to a power
return(res)
}
myPower(2, 3) # 2 is bound to bas and 3 to pow (in-order)
# same binding as above (out-of-order parameter names)
myPower(pow = 3, bas = 2)
myPower(bas = 3) # default value of pow is used

Vectorized Code
Vectorized code runs much faster than loops due to R interpreter
overhead.
a = 1:10000000; res = 0
system.time(for (e in a) res = res + e^2)
## user system elapsed
## 3.742 0.029 3.800
system.time(sum(a^2))
## user system elapsed
## 0.180 0.032 0.250

External/Native API
Often, 10% percent of the code is responsible for 90% of
computing time. Implementing bottlenecks in C/C++ allows
staying mostly within the R environment.
dyn.load("fooC2.so") # load compiled C code
A = seq(0, 1, length = 10)
B = seq(0, 1, length = 10)
.Call("fooC2", A, B)
Newer packages: Rcpp, RcppArmadillo, RcppEigen

## [1] 13.34 17.48 21.21 24.71 28.03 31.24 34.34 37.37 40.33 43
## [1] 13.34 17.48 21.21 24.71 28.03 31.24 34.34 37.37 40.33 43
0.0
0.5
1.0
1.5
0 250 500 750 1000
array size
computationtime(sec)
language
C
R

Graphing Data with R
Guy Lebanon
September 22, 2015

Goals
I Learn how to use base graphics
I Learn how to use base ggplot2
I Understand basic graph types and when to use them
Module will be separated to 4 parts: (a) base graphics, (b)
ggplot2, (c) datasets, (d) basic graph types and case studies.

Base Graphics
Base graphics syntax: plot function followed by helper functions
for annotating the graph.
plot(x = dataframe$col_1, y = dataframe$col_2)
title(main = "figure title") # add title
Examples of low-level functions in the graphics package are:
I title adds or modiﬁes labels of title and axes,
I grid adds a grid to the current ﬁgure,
I legend displays a legend connecting symbols, colors, and
line-types to descriptive strings, and
I lines adds a line plot to an existing graph.

GGPLOT2
Philosophy: (a) Grammar of graphics, (b) logical separation of
graphics and data, (c) concise and maintainable code.
Option 1: Use the qplot function. Pass dataframe column names,
dataframe name, geometry, and graphing options.
qplot(x = x1,
y = x2,
data = DF,
main = "figure title",
geom = "point")
Remember to install and load package using
install.packages('ggplot2')
library(ggplot2)

Option 2: Use the ggplot function. Pass dataframe, column
names through aes function. Compose function output with
additional layers using + operator.
ggplot(dataframe, aes(x = x, y = y)) +
geom_line() + geom_point()
Function (and addition operator) returns an object that can be
printed (using the print function) or saved for later.

Datasets
We will use the three datasets below.
I faithful: eruption time and waiting time to next eruption
(both in minutes) of the Old Faithful geyser in Yellowstone
National Park, Wyoming, USA.
I mtcars: model name, weight, horsepower, fuel e ciency, and
transmission type of cars from 1974 Motor Trend magazine.
I mpg: fuel economy and other car attributes from
http://fueleconomy.gov (similar to mtcars but larger and
newer).

names(faithful)
## [1] "eruptions" "waiting"
names(mtcars)
## [1] "mpg" "cyl" "disp" "hp" "drat" "wt" "qsec" "vs"
## [11] "carb"
names(mpg)
## [1] "manufacturer" "model" "displ" "year"
## [5] "cyl" "trans" "drv" "cty"
## [9] "hwy" "fl" "class"

Strip Plot
Strip plots graph one-dimensional numeric data as points in a
two-dimensional space, with one coordinate corresponding to the
index of the data point, and the other coordinate corresponding to
its value.
plot(faithful$eruptions, xlab = "sample number",
ylab = "eruption times (min)",
main = "Old Faithful Eruption Times")

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0 50 100 150 200 250
1.52.53.54.5
Old Faithful Eruption Times
sample number
eruptiontimes(min)

I We conclude from the ﬁgure above that Old Faithful has two
typical eruption times — a long eruption time around 4.5
minutes, and a short eruption time around 1.5 minutes.
I It also appears that the order in which the dataframe rows are
stored is not related to the eruption variable.

Histograms
Histograms graph one-dimensional numeric data by dividing the
range into bins and counting number of occurrences in each bin. It
is critical to set the bin width value correctly.
qplot(x = waiting,
data = faithful,
binwidth = 3,
main = "Waiting time to next eruption (min)")
ggplot(faithful ,aes(x = waiting)) +
geom_histogram(binwidth = 1)

0
10
20
30
40
40 60 80 100
waiting
count
Waiting time to next eruption (min)

There are clearly two typical eruption times – one around 2
minutes and one around 4.5 minutes.
y values can be replaced with probability/frequency using the
following syntax.
ggplot(faithful, aes(x = waiting, y = ..density..)) +
geom_histogram(binwidth = 4)
Selecting the best bandwidth to use when graphing a speciﬁc
dataset is di cult and usually requires some trial and error.

0
5
10
15
40 60 80
waiting
count

0
20
40
60
80
50 75 100
waiting
count

Line Plot
Line plot: a graph displaying a relation between x and y as a line
in a Cartesian coordinate system. The relation may correspond to
an abstract mathematical function or to a relation between two
samples (for example, dataframe columns)
x = seq(-2, 2, length.out = 30)
y = x^2
qplot(x, y, geom = "line") # line plot
qplot(x, y, geom = c("point", "line")) # line and point plot
dataframe = data.frame(x = x, y = y)
ggplot(dataframe, aes(x = x, y = y)) +
geom_line() + geom_point() # same as above but with ggplot

S = sort.int(mpg$cty, index.return = T)
# x: city mpg
# ix: indices of sorted values of city mpg
plot(S$x, # plot sorted city mpg values with a line plot
type = "l",
lty = 2,
xlab = "sample number (sorted by city mpg)",
ylab = "mpg")
lines(mpg$hwy[S$ix] ,lty = 1) # add dashed line of hwy mpg
legend("topleft", c("highway mpg", "city mpg"),
lty = c(1, 2))

0 50 100 150 200
101520253035
mpg
highway mpg
city mpg

Smoothed Histograms
Denoting n values by x(1), . . . , x(n), the smoothed histogram is the
following function fh : R ! R+
fh(x) =
1
n
nX
i=1
Kh(x x(i)
)
where the kernel function Kh : R ! R typically achieves its
maximum at 0, and decreases as |x x(i)| increases. We also
assume that the kernel function integrates to one
R
Kh(x) dx = 1
and satisﬁes the relation
Kh(r) = h 1
K1(r/h).
We refer to K1 as the base form of the kernel and denote it as K.

Four popular kernel choices are the tricube, triangular, uniform,
and Gaussian kernels, deﬁned as Kh(r) = h 1K(r/h) where the
K(·) functions are respectively
K(r) = (1 |r|3
)3
· 1{|r|<1} (Tricube)
K(r) = (1 |r|) · 1{|r|<1} (Triangular)
K(r) = 2 1
· 1{|r|<1} (Uniform)
K(r) = exp( x2
/2)/
p
2⇡ (Gaussian).
As h increases the kernel functions Kh become wider.

h=1 h=2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
gaussiantriangulartricubeuniform
−2 0 2 −2 0 2
x
K_h(x)

−2 0 2 4 6 8
0.000.100.200.30
x
f_h(x)
Smoothed histogram (h=1/6)

−2 0 2 4 6 8
0.000.050.100.150.200.25
x
f_h(x)
Smoothed histogram (h=1/3)

−2 0 2 4 6 8
0.050.100.15
x
f_h(x)
Smoothed histogram (h=1)

In ggplot2:
ggplot(faithful, aes(x = waiting, y = ..density..)) +
geom_histogram(alpha = 0.3) +
geom_density(size = 1.5, color = "red")

0.00
0.02
0.04
40 60 80 100
waiting
density

Scatter Plot
A scatter plot graphs the relationships between two numeric
variables. It graphs each pair of variables as a point in a two
dimensional space whose coordinates are the corresponding x, y
values.
plot(faithful$waiting,
faithful$eruptions,
pch = 17,
col = 2,
cex = 1.2,
xlab = "waiting times (min)",
ylab = "eruption time (min)")

50 60 70 80 90
1.52.53.54.5
waiting times (min)
eruptiontime(min)

I We conclude from the two clusters in the scatter plot above
that there are two distinct cases: short eruptions and long
eruptions.
I Furthermore, the waiting times for short eruptions are
typically short, while the waiting times for the long eruptions
are typically long.
I This is consistent with our intuition: it takes longer to build
the pressure for a long eruption than it does for a short
eruption.

The relationship between two numeric variables and a categorical
variable can be graphed using a scatter plot where the categorical
variable controls the size, color, or shape of the markers.
plot(mtcars$hp,
mtcars$mpg,
pch = mtcars$am,
xlab = "horsepower",
cex = 1.2,
ylab = "miles per gallon",
main = "mpg vs. hp by transmission")
legend("topright", c("automatic", "manual"), pch = c(0, 1))

●●
●
●
●
●
●
●
●
●
●
●
●
50 100 150 200 250 300
1015202530
mpg vs. hp by transmission
horsepower
milespergallon
●
automatic
manual

We draw several conclusions from this graph.
I There is an inverse relationship between horsepower and mpg.
I For a given horsepower amount, manual transmission cars are
generally more fuel e cient.
I Cars with the highest horsepower tend to be manual (the two
highest horsepower cars in the dataset are Maserati Bora and
Ford Pantera, both sports cars with manual transmissions).

Changing marker size in a scatter plot
qplot(x = wt,
y = mpg,
data = mtcars,
size = cyl,
main = "MPG vs. weight (x1000 lbs) by cylinder")

● ●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
10
15
20
25
30
35
2 3 4 5
wt
mpg
cyl
●
●
●
●
●
4
5
6
7
8
MPG vs. weight (x1000 lbs) by cylinder

I When data is noisy, it is useful to add a smoothed line curve
to visualize median trends
I One technique to address this issue is to add a smoothed line
curve yS , which is a weighted average of the original data
(y(i), x(i)) i = 1, . . . , n:
yS (x) =
nX
i=1
Kh(x x(i))
Pn
i=1 Kh(x x(i))
y(i)
.
where the Kh functions above are the kernel functions
described earlier
I yS (x) is an average the y(i) values, weighted in a way that
emphasizes y(i) values whose corresponding x(i) values are
close to x.
I The denominator in the deﬁnition of yS ensures that the
weights deﬁning the weighted average sum to 1.

qplot(disp,
mpg,
data = mtcars,
main = "MPG vs Eng. Displacement") +
stat_smooth(method = "loess",
degree = 0,
span = 0.2,
se = TRUE)
span parameter inﬂuences the value of h in the slide before and
can make the line more or less smooth. Optional argument se
adds standard errors as shaded region.

●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
10
15
20
25
30
35
100 200 300 400
disp
mpg
MPG vs Eng. Displacement

Facets
I Facets are a way to display multiple graphs next to each other
in the same scale with shared axes.
I This is an e↵ective way to visualize data that has higher
dimensionality than 2 (mixed numeric-categorical).
I The argument facets in qplot or ggplot takes a formula
a ⇠ b where a, b specify the variables according to which the
rows and columns are organized.
qplot(x = wt,
y = mpg,
facets = .~amf,
data = mtcars,
main = "MPG vs. weight by transmission")

automatic manual
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
10
15
20
25
30
35
2 3 4 5 2 3 4 5
wt
mpg
MPG vs. weight by transmission

● ●
●
●
●●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10
15
20
25
30
35
10
15
20
25
30
35
flatV−shape
2 3 4 5
wt
mpg
MPG vs. weight by engine

automatic manual
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
10
15
20
25
30
35
10
15
20
25
30
35
flatV−shape
2 3 4 5 2 3 4 5
wt
mpg
MPG vs. weight by transmission and engine

I Manual transmission cars tend to have lower weights and be
more fuel e cient
I Cars with V-shape engines tend to weigh less and be more fuel
e cient
I Manual transmission and V-engine cars tend to be lighter and
more fuel e cient. Automatic transmission and non V-engine
are heavier and less fuel e cient.
“All pairs” plot:
DF = mpg[, c("cty", "hwy", "displ")]
library(GGally)
ggpairs(DF)

ctyhwydispl
cty hwy displ
10
15
20
25
30
35
Corr:
0.956
Corr:
−0.799
20
30
40
● ●
●●
●●●●●
●●
●●●●●●
●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●●●●
●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●
●●●●
●
●
●●
●●●●●●
●
●
●●●
●
●●●●
●
●●●
●
●
●
●●
●
●●
●●
●●
●
●
●●●
●●●
●
●●
●
●
●●
●●
●●
●
●●
●●
●●
●
●
●
●●
●●●●●
●●
●
●
●●●●
●●●
●
●●
●
●●●●●
●
●
●
●●●
●
●
●
●
●
●●
●●
●●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●●●●
●●
●
●
●
●
●●●●●●
●●●
Corr:
−0.766
2
3
4
5
6
7
10 15 20 25 30 35
● ●●●
●●
●
●● ●●
●●
●●
●
●
●
●● ●
●
●
●●
●●
●
●●
●
●
● ●
●
●●
●
●
●●●●●
●●●●
●●●●
●●●
●●
●
●● ●
●
●●
●● ●●●●
●●
●●
●
●●
●●●●
●
●
●●
●●●
●●
●●●●
●●●●
●
●●●●●●●●●
●● ●●●●
●
●●●●
●●●
●
●
●
●● ●
●
●
●●●●
●●●
●●
●
●
●● ●●
●●
●●
●●●
●
●
●
●●●
●
●●●●●●
●●
●●●●●●●●
●●
●
●
●●●●
●●
●
●●●●
●●
●
●●●●●
●
●
●●●
●●
●●
●●●●
●
●●● ●●
●●
●●
●●●●
●●
●●●●
●●
●
20 30 40
●●●●
●●
●
●●●●
●●
●●
●
●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●●●●●
●●●●
●●●●
●●●
●●
●
●● ●
●
●●
●● ●●●●
●●
●●
●
●●
●●●●
●
●
●●
●●●
●●
●●●●
●●●●
●
●●●●●●●●●
●●●●●●
●
●●●●
●●●
●
●
●
●● ●
●
●
●●●●
●●●
●●
●
●
●● ●●
●●
●●
●●●
●
●
●
●●●
●
●●●●●●
●●
●●●●●●●●
●●
●
●
●● ●●
●●
●
●●●●
●●
●
●●●●●
●
●
●●●
●●
●●
●●●●
●
●●●●●
●●
●●
●●●●
●●
●●●●
●●
●
2 3 4 5 6 7

Contour Plots
Contour plots graph relationship between three numeric variables:
z as a fuction of x, y. Steps: (a) create a grid for x values, (b)
create a grid for y values, (c) create an expanded x ⇥ y grid, (d)
compute values of z on the expanded grid, (e) graph data.
x_grid = seq(-1, 1, length.out = 100)
y_grid = x_grid
R = expand.grid(x_grid, y_grid)
names(R) = c('x', 'y')
R$z = R$x^2 + R$y^2
ggplot(R, aes(x = x,y = y, z = z)) + stat_contour()

−1.0
−0.5
0.0
0.5
1.0
−1.0 −0.5 0.0 0.5 1.0
x
y

Quantiles and Box-Plots
Box plots are an alternative to histograms that are usually more
“lossy” but emphasize quantiles and outliers in a way that a
histogram cannot.
I The r-percentile of a numeric dataset is the point at which
approximately r percent of the data lie underneath, and
approximately 100 r percent lie above.
I Another name for the r percentile is the 0.r quantile.
I The median or 50-percentile is the point at which half of the
data lies underneath and half above.
I The 25-percentile and 75 percentile are the values below which
25% and 75% of the data lie. These points are also called the
ﬁrst and third quartiles (the second quartile is the median).
I The interval between the ﬁrst and third quartiles is called the
inter-quartile range (IQR) (region covering the central 50% of
data).

The box plot is composed of;
I box denoting the IQR,
I an inner line bisecting the box denoting the median,
I whiskers extending to the most extreme point no further than
1.5 times IQR length away from the edges of the box,
I points outside the box and whiskers marked as outliers.
ggplot(mpg, aes("",hwy)) +
geom_boxplot() +
coord_flip() +
scale_x_discrete("")

ggplot(mpg, aes(reorder(class, -hwy, median), hwy)) +
geom_boxplot() +
coord_flip() +
scale_x_discrete("class")

● ●● ●
●●
●
●● ●● ●●●●
●●● ●
compact
midsize
subcompact
2seater
minivan
suv
pickup
20 30 40
hwy
class

I The graph suggests the following fuel e ciency order among
vehicle classes: pickups, SUV, minivans, 2-seaters,
sub-compacts, midsizes, and compacts.
I The compact and midsize categories have almost identical box
and whiskers but the compact category has a few high outliers.
I The spread of subcompact cars is substantially higher than
the spread in all other categories.
I We also note that SUVs and two-seaters have almost disjoint
values (the box and whisker ranges are completely disjoint)
leading to the observation that almost all 2-seater cars in the
survey have a higher highway mpg than SUVs.

QQ-Plots
I Quantile-quantile plots are useful for comparing two datasets,
one of which may be sampled from a certain distribution.
ggplot(R, aes(sample = samples)) +
stat_qq(distribution = qt, dparams = pm)
I They are essentially scatter plots of the quantiles of one
dataset vs. the quantiles of another dataset.
I The shape of the scatter plot implies the following conclusions
(the proofs are straightforward applications of probability
theory).

I A straight line with slope 1 that passes through the origin
implies that the two datasets have identical quantiles, and
therefore that they are sampled from the same distribution.
I A straight line with slope 1 that does not pass through the
origin implies that the two datasets have distributions of
similar shape and spread, but that one is shifted with respect
to the other.
I A straight line with slope di↵erent from 1 that does not pass
through the origin implies that the two datasets have
distributions possessing similar shapes but that one is
translated and scaled with respect to the other.
I A non-linear S shape implies that the dataset corresponding
to the x-axis is sampled from a distribution with heavier tails
than the other dataset.
I A non-linear reﬂected S shape implies that the dataset whose
quantiles correspond to the y-axis is drawn from a distribution
having heavier tails than the other dataset.

D = data.frame(samples = c(rnorm(200, 1, 1),
rnorm(200, 0, 1),
rnorm(200, 0, 2)))
D$parameter[1:200] = 'N(1,1)';
D$parameter[201:400] = 'N(0,1)';
D$parameter[401:600] = 'N(0,2)';
qplot(samples,
facets = parameter~.,
geom = 'histogram',
data = D)

0
10
20
30
40
50
0
10
20
30
40
50
0
10
20
30
40
50
N(0,1)N(0,2)N(1,1)
−4 0 4 8
samples
count

D = data.frame(samples = c(rnorm(200, 1, 1),
rnorm(200, 0, 1),
rnorm(200, 0, 2)));
D$parameter[1:200] = 'N(1,1)';
D$parameter[201:400] = 'N(0,1)';
D$parameter[401:600] = 'N(0,2)';
ggplot(D, aes(sample = samples)) +
stat_qq() +
facet_grid(.~parameter)

N(0,1) N(0,2) N(1,1)
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
−5.0
−2.5
0.0
2.5
5.0
−3 −2 −1 0 1 2 3−3 −2 −1 0 1 2 3−3 −2 −1 0 1 2 3
theoretical
sample

R = data.frame(density = dnorm(x_grid, 0, 1))
R$tdensity = dt(x_grid, 1.5)
R$x = x_grid
ggplot(R, aes(x = x, y = density)) +
geom_area(fill = I('grey')) +
geom_line(aes(x = x, y = tdensity)) +
labs(title = "N(0,1) (shaded) and t-distribution (1.5 dof)")

0.0
0.1
0.2
0.3
0.4
−6 −3 0 3 6
x
density
N(0,1) (shaded) and t−distribution (1.5 dof)

R = data.frame(density = dnorm(x_grid, 0, 1))
R$samples = rnorm(200, 0, 1)
pm = list(df = 1.5)
ggplot(R, aes(sample = samples)) +
stat_qq(distribution = qt, dparams = pm)

● ●
● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●
●
●
●
−2
0
2
−30 −20 −10 0 10 20 30
theoretical
sample

Preprocessing Data
Guy Lebanon
September 30, 2015

Goals
I Learn how to handle missing data
I Learn how to handle outliers
I Learn when and how to transform data
I Learn standard data manipulations techniques
Module will be separated to 4 parts based on the four goals above.

Missing Data
Data may be missing for a variety of reasons.
I corrupted during its transfer or storage
I some instances in the data collection process were skipped
due to di culty or price associated with obtaining the data
Di↵erent features in di↵erent samples may be missing: ﬁrst sample
(row) may have third feature (column) missing while the second
sample may have the ﬁfth feature missing.

Examples of Missing Data
I Recommendation systems recommend to users items from a
catalog based on historical user rating. Often, there are a lot
of items in the catalog and each user typically indicates their
star ratings for only a small subset of them.
I In longitudinal studies some of the subjects may not be able
to attend each of the surveys throughout the study period.
The study organizers may also have lost contact with some of
the subjects, in which case all measurements beyond a certain
time point are missing.
I In sensor data, some of the measurements may be missing due
to sensor failure, battery discharge, or electrical interference.
I In user surveys, users may choose to not respond to some of
the questions for privacy reasons.

Missing Completely at Random
I If a variable (dataframe column) is as likely to be missing as
all other variables, we say that it is MCAR.
I For example, in the case of users rating movies using 1-5
stars, we consider ratings of specific movies as dataframe
columns and ratings associated with specific users as
dataframe rows. Since some movies are more popular than
others, some columns are more likely to be missing than
others, violateing the MCAR definition.

Missing at Random (MAR)
I MAR occurs when the probability that a variable is missing
depends only on the other information available in the dataset.
I For example, in a survey recording gender, race, and income,
gender and race are not very objectionable questions, so we
assume for now that the survey respondents answer these
questions fully. The income question is more sensitive and
users may choose to not respond for privacy reasons.
I The tendency to report income or to not report income
typically varies from person to person. If it only depends on
gender and race, then the data is MAR.
I If the decision whether to report or not report income depends
also on other variables that are not in the dataframe (such as
age or profession), the data is not MAR.

Handling Missing Data
Most methods are designed to work with fully observed data.
Below are some general ways to convert missing data to
non-missing data.
I Remove all data instances (for example dataframe rows)
containing missing values.
I Replace all missing entries with a substitute value, for example
the mean of the observed instances of the missing variable.
I Estimate a probability model for the missing variable and
replace the missing value with one or more samples from that
probability model.
In the case of MCAR, all three techniques above are reasonable in
that they may not introduce systematic errors. In the more likely
case of MAR or non-MAR data the methods above may introduce
systematic bias into the data analysis process.

Missing Data and R
I R represents missing data using the NA symbol.
I The function is.na returns a data structure having TRUE
values where the corresponding data is missing and FALSE
otherwise.
I complete.cases() returns a vector whose components are
FALSE for all samples (dataframe rows) containing missing
values and TRUE otherwise.
I na.omit() returns a new dataframe omitting all samples
(dataframe rows) containing missing values.
I Some functions have an na.rm argument, which if set to TRUE
changes the function behavior so that it proceeds to operate
on the supplied data after removing all dataframe rows with
missing values.

The code below analyzes the dataframe movies in the ggplot2
package, which contains 24 attributes (genre, year, budget, user
ratings, etc.) for 58788 movies obtained from the website
http://www.imdb.com with some missing values.
mean(movies$length) # average length
## [1] 82.34
mean(movies$budget) # average budget
## [1] NA
# average budget (removing missing values)
mean(movies$budget, na.rm = TRUE)
## [1] 13412513
mean(is.na(movies$budget)) # frequency of non-missing budget
## [1] 0.9113

moviesNoNA = na.omit(movies)
qplot(rating, budget, data = moviesNoNA, size = I(1.2)) +
stat_smooth(color = "red", size = I(2), se = F)
●●
●
●
●
●● ●●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
● ●
●
●
●
●●●●
●
●
●● ●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
● ●
● ●●
●
●●
●
●●
●
●
●
●
●
● ● ●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
● ● ●
●
●
●
●
●
●●●
●
●
●
● ●●●●
●
●●
●
●
●
● ●● ●
●
●
●
●●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
● ●●● ●●
●
●
●
●
● ●
●
●
●
● ● ●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●●● ● ●● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●● ●
●
● ●
●
●●
●
●
● ●
●
●
●
● ●
●
●
●●
●
●
●
●
●●●
●
●
●●
●
● ●
●
● ●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●● ●● ●● ●
●
● ● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●● ● ●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●● ●●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●●
●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●●
●
●● ●● ●●
●
●
● ●●●●
●
● ●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
● ●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●● ●
●
●
● ●●●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●●
●
●
●
● ● ●
●
●● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●● ●
●
●
●
●● ●●
●
●
●
●●●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●●● ●●●● ●●
●
●
●
● ●
● ● ●
●
●
●
●
●●
●
● ●● ●●
●
●
● ●● ●●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
●
●● ●● ●● ● ●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
● ●●
●
●●
●
● ●
●
●●
●
●
●
●
●
●
● ● ● ●●
●
● ●
●
●●
●
●
●
●●
●● ●●●●
●
●
● ●● ●●●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●●●
●
●
●
●
● ● ●
● ● ●
●
● ●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●● ●
●
●
●
● ●
●
●
●
●
● ●
●●
●●
●
● ●
●
●
●
●
●
●
● ● ●●
●
● ●●●
●
●●
●
●
●
●●
●
●
●
● ● ● ●
●
●●●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
● ●
●
●
●
●●
●
● ● ●
●
●
●
●
● ●
●
●
● ●
●
● ●
●
●
●
● ● ●
●
●
●
●
●
●
● ●
●●
●
●
●●
●
●●
●
●
●●
●
●
●●●●
●
● ●
●
●● ●
●
● ● ●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
● ●● ●●
●
●●
●●
●●● ●
●
●
●
● ●●
●
●
●
●●
● ● ●●
●
●
●
●
●● ● ● ●● ●
●
●
●
●●● ● ●
●
● ●● ●
●
●
●● ●●●
●
●
●●
●
●●
●●
●●
●
●●
● ● ●●
●
●●
●
●
●
●●
●
● ●● ●
●
● ●
●
●
●
●
●
●
●
●
●
● ●● ●● ●●
●
●● ●
●
● ●
●
●●
●
●
●
● ●
●
● ● ●●●
●
● ●●
●
●● ●
●
● ●
●
●
●
● ●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
● ● ● ●
●
●●
●
● ●
●
●●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
● ●●
●
●
●
●
●
●
●●
●
● ● ●
●
●
●
●
●
●
●
● ●● ●
●
●
● ●●
●
●
●
●● ●●● ●●
●
●●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●●● ●●●● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
● ●
●
●
●
●
●●
●
● ●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●● ●●●
●
●
● ● ●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●●● ● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●● ●●●
●
●● ●
●
●
●
● ●●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
● ●● ●●●
●
●
●
● ●●
●
●
●
● ●
●
● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●● ●
●
●
●
●
●
● ●●
●
●
●
●
●
●●
●
●● ●● ●● ● ●
●●
●● ●●
●
● ●
●
●● ●●
●
●
●
●
●
●
●
●●
●● ●
●
●
● ●
●
●
●● ●
● ● ●
●
●● ●
● ●
●
●
●
●
●
●
●
● ●
●● ●●● ●● ●
●
●
●
●
●
●
●
●●
●●●
● ●
●
●
●●
●
●
●
●
●
●●
●● ●
●
●
● ●
●
●● ●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
●
●
●
●●
●
● ●
●●● ●●
●
●
●
●
●
●
●
●
●
● ●
●
● ●●● ●
●
●
●●
●●
●
●● ● ●● ●● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
● ●
●
●
●
● ●
●
● ●
● ●
●
●● ●●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
● ●
●
●
●
●
●● ●
●
● ●
●
●
●●
●
●●
●
●
●
● ●● ●
●
● ●
●
●
● ●●
●
● ●
●
●
●
●
●
●
●● ●
●
●●●
●
●
●
●●
●
●
●
●●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
● ●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
●
●
● ●● ●● ●● ● ●
●
● ●
●
●
●
●
● ●
●
●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●● ●
●
●
●
●
●●
●
●
●
●
●● ● ●
●
●
●
●● ● ●●● ●
●
●
●●
●●
●
●
●
●
●
●●●●
●
●
●
● ●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●● ●
●
●
●
●
●
●
● ●● ● ●●
●
●●
●
● ●●
●
●
●
● ●
●
● ●
●
● ●
●
●
●
●
●
●
● ●●●● ●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
● ● ●●
●
●● ●●
●
●
●
● ●●
●
● ●
●
●
●
●
●
●
●
●● ●●●●● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
●
●
● ●●
●
●
● ●●●
●
●●● ●●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
● ●
●
●
● ●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●● ●●
●
●
●
● ●
●
● ●●
●
●
●
● ●● ●● ●●●● ● ●
●
● ● ●● ●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●●●
●
●
● ●●●
●
●
●●● ●
●
●
●
●
●● ●●
●
●●
●●
●
●
●
● ●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●● ● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ● ●
●
●●● ●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●
●
●
●
● ● ●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●
●
●
●
●●
●
●
●
●
●
● ●
●
● ●●●●
●
● ●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●● ●
●
● ●●
●
●
●
●
● ●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
● ●● ● ●
● ● ●
● ●
●
●●● ●●
●
●
●
● ●●
●
● ●● ●
● ●
●
●
●
● ●●
●
● ●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●● ●● ●
●
●●
●
●
●
●
●
●
●
●
● ● ●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●●●
●
●
●
● ●
●
● ●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
● ●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●● ●
●
●
●
● ●
●
● ●●
●
●
● ●
●
●
● ●
● ●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
●● ●
●
●
●
●
●
●●
● ●
●
●
● ●
●
● ●
●
●
●●●
●
●
●
●
●
●● ●
●
●
●● ●● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●●
●
● ●●
●
●
●
●
●
● ●●● ●
●
● ●
●
●●
●
● ●
●
●
●
●
●●
●
● ●
●
● ●
●
●
● ●●
●
●
●
●
●●
●
●●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●●●
●
● ●● ● ●●
●
●● ●●
●
●
● ●
●
●●
●
●
●●●●
●
●
●
●
●
●● ●
●
● ●
●
● ●●●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
● ● ●
●
●
●
●
●
●
●
● ●● ● ●● ●●
●
●● ● ●
●
●
●
● ●
●
● ●●
●
● ●
●
●
●
●
●
●
●
● ●
●
● ●● ●
●
● ●● ●
●
●
●
●●
●
●●
●
●
●
●
●●● ●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●●
●●
● ●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
● ●
●
●
●●●
●
● ●
●
●● ●
●● ●●●
●
●
●
● ● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ● ●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ●●
●
●
●
●
●●
●
●
● ●
●
●
●● ●●
●●● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●●● ●● ●
●
●
●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
● ● ●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●● ●● ●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
● ●
●
●
●
● ● ●
●
●●
●
●
●
● ● ●●
●
●
● ●●●● ●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
● ● ●● ●● ●
●
●● ● ●●
●
●●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ● ● ●● ●
● ●●
●● ●
●
●
●
●●
●
●
●
●
●
●
●● ●
●
●
● ● ●● ● ●
●
●●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●● ●●
●
●
●
●
●
● ●●● ●
●
●
●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
● ●
●
● ●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
● ● ●
●
●● ●●
●
●
●
●●
●
●
●
●●
●
●
●● ●
●
●
●
●● ●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
● ●
●●
●
●
●
●● ●● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
● ●● ●
●
●
●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●●● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●●
●
● ●
●
●● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
● ●
●●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●●● ●
●
●● ●
●
● ●●● ● ●●
●
●●
●
●
● ●●●●
●
●
●
●
●
●
●
●
● ●● ● ●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●● ●
●
●
●
●●
● ●
●●
●● ●●
●
●
●
●
● ●● ● ●● ●●
●
●
●
●
●
●
●●●
●
● ●
●
●
●
●●●
●
●
●
● ●
● ●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
● ● ● ●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●
●●
● ● ●●
●
●
●
●●
●
●● ●●●
● ● ●● ●
●
● ● ●● ● ●
●●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●● ●
●
●
● ●
●
●
●
● ●● ●
●
●
●● ●
●
●
●
●
● ●●
●
●● ●●
●
●
●
●● ●● ●●●
●● ●
●
●
●
●
●
●
●
●
● ● ●●
●
●
●●● ● ●
●
●
●
●
●
●
●
●
●
●● ●
●
● ● ● ●●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●
● ● ● ●●
●
●
●
●
●
●●
●
●
● ●
● ●●●
●
●
● ●
●
●● ● ●
●
● ●●
●
●
●
● ●● ●
●
●
● ●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●● ● ●●
●
●
●
●
●
●
●●
● ●
●
●
● ●● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●● ●● ●
●
●
●
●● ●● ● ●
●
●
●
●
●
●
●
●
●
●
● ● ●
● ●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
● ●● ●
●
●
●
●
●●
●●●
●
●
●●●●
●
●● ●
●
●
●
●
●
●
● ●
●
●
● ●
●
●●● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●● ●●
● ●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
● ●●
●
●
●●
●● ●
●●
●
●●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
● ● ●●
●
●● ●●●
●
●●
●
●
●
●
●
●
●
●
●● ●●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
● ●●●
●
●
● ●
●
●●
●
●
● ●●●
● ●
● ●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●● ●
●
●
●●
●
● ●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●●●●● ● ●●
●● ●
●●
●
● ●●
●
●
●
●●
●
● ● ●
●
●
●
●
● ●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
● ●
●
●●
●
●●●●●
●
●
●
●
●
●● ●● ●●●
●
●●
●
●
●● ●
●
●
●●
●
●
●
●
● ●
●
●
●● ●●
●
●
●●
●
●
●● ●●
●
●
●
●
● ●
●
●
● ●
●
●
●● ●●
●
●
●
●
●
●
●
● ● ●●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
● ●
●
●●
● ●
●●
●●● ●
●
●
●
●
●●
●●
●
●
● ●●●
●
●
●●
●
●●
●
●●
●
●
● ●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●● ●●
●
●
●
●
●
●
● ●
●
●
●●
●
● ●
●
●
●
●
●● ● ●
●●
●
●
●● ●
●
●
● ●
●●
0.0e+00
5.0e+07
1.0e+08
1.5e+08
2.0e+08
2.5 5.0 7.5 10.0
rating
budget

moviesNoNA = na.omit(movies)
qplot(rating, votes, data = moviesNoNA, size = I(1.2))
●
●
●●
●
●● ●●
● ●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
● ●
●
● ●
●
●
●
●● ●
●●
●●● ●
●
●●
●
●
● ●
●
●
●
●●●●
●
●
●● ●● ●● ●
●
●
●●
●
●
●
●●
●
●
●● ●● ● ●
●
●
●●
●
●
● ● ●● ●●
●
●
●
● ●
●
●
● ●●
●
●●
●
●● ●● ●●
●
● ● ●
●
●● ●●
●
●●● ●
●
●
●
●●
●
●
● ● ●
●
●
●
● ● ●
●
●
●
●
●
●
●●
●
●
● ● ●●●●
●
●●
●
● ●
● ●● ●
●
●
●
●●
●
●
● ●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●●
●
●
●
●
● ●
●
●
●● ● ●
●
●
●
●
● ●● ●● ●
●
●●
●
●
●
●
● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●● ●● ●●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●●● ●●
●
●
●●
●●
●
●● ●●
●●
●
●
●
●● ●●●● ● ●● ● ●● ●
●
●
●
●
●
●
● ●
●●●
●
●
● ●●
●
●
● ●●●● ●
●
●● ●●
●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●● ●
●
● ●●
●
●
●
●●
●
●
● ●
●
● ●●● ●
●
● ●●●● ●
●
●
●
●
●
●● ●●
●
●
● ●
●
●
●●● ●●
●
●
●●
●
●
●● ●● ●● ●● ● ● ●● ●
●
●
●
● ● ●●
●
●
●
●
● ●
●
●
●●
●
●●● ●● ● ●● ●● ●
●●
●
●
● ●●
●●
●
●
●
●
● ●●
●
●
●
●
●
●●● ●
● ●● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●● ●
●
●
● ●●●
●
●
●
●
●● ●● ●
●
●
● ●
●
●
● ●●●
●●
●
●
●●
●
●
● ●● ●● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●● ●● ●
●●
●● ●● ●●
●
●
● ●●●●●● ●
●
● ●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●●
●
●●
●
●
● ●
●
●
● ●●●
●
● ●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●
●
●
●
● ● ●● ●●
●
● ●
● ●
● ●
●
●
●
●● ●
●
● ●●
●
●
●
●●
●
●
●
●● ● ●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●● ●
●
●
●
●● ●●
●
●
●
●●
● ● ●
●
● ●●
●
●
●● ●●●
●
●
●
● ●
●
●
●
●● ●
● ●●
●
●
●
●●● ●●●● ●
●
●
●
● ●
●●
● ●
●
●
●
● ●●● ● ●● ●●●● ● ●● ●●
●
● ●
● ●
●
●
● ●
●
● ● ● ●● ● ●● ●● ●● ●● ● ●●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●●
●
●
●
●
●
●●●
●●
●
●●
●
● ●
●
●●
●
●●● ●
●
● ● ● ●●●● ●
●
●●
●
●●
●
● ●● ●●●●
●
●
● ●● ●
●
●
●
●●
●
● ●● ● ●●
●● ●●
●
●
● ●
●
●
● ●● ●
●
● ●●
●
●
●
● ●●●
●
● ●
●
● ●
●
● ●
●
●
●
●
●
● ●● ● ●
●
●●●
●
●
●
●
● ●●●
●
●●
●
●● ●● ●
●
●
●
● ●
●
●
●
●● ●
●●
●
●
●
● ●
●
● ●
●
●
●● ● ●●
●
● ●●●●●●
●
●
●
●●
●●
●
● ● ● ●
●
●●● ●● ●● ●
●
●
● ●●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
●
●● ●●
●
●●●
●
●● ●
●
● ●● ●●
●
● ●
● ●
●
●● ●● ● ●
●
●
●
●● ●
●
●● ●●● ● ● ●
●
● ● ●
●
●●●● ● ● ● ●●●
●
●●
●
●
●
●
●
●●
●●
●●●●
●
● ●
●
●● ● ● ● ● ●●
●● ●●
● ●●
●
●●● ●●● ● ●● ●●
●
●●
●
●
●●● ●
●
●
●
● ●●
●
●
●
●
●● ●
●
●
●●
●● ●● ● ● ●●
●
●
●
●
●●● ●
●
● ●
●
● ●
●
●
●● ●●● ● ●
●
●
● ●●
●
●
●●
●
●●● ● ●●
●
●
●
● ●
●
●● ●● ●● ●
●
● ●
●
● ●
●
●
● ●
●
●● ●● ●●
●
●
●
●● ●●● ●
●●● ●●● ● ●
●
● ● ●●●
●
● ●●
●
●● ●● ● ●
●
●
● ● ●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●● ● ● ● ●
●
●●
●
● ●● ●●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●●
●
●●
●
●
● ●●
●
●● ●● ●
●
●
●
● ● ●●●● ●
●●●
●
● ●
●● ●
●
●
●
●
●
● ●●
●
●
● ●● ●●● ●●
●
●●
●
●● ● ●
●
●
● ●
●●
●
●
● ●
●
●
● ●
●
● ● ● ● ●●
●
●
●
●
●
●
● ● ●●
●
●
●
●
●
● ●
●
● ●●●● ●
●
●● ● ●● ●
●
●
● ● ●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
● ●●●● ●
●
●●●
●
●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●●● ●●●
●
●
●
● ●
●
●
●
●
●
●●● ●
●
●
● ●● ●
●
●●
●
● ●●● ●
●●
●
●
●●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●●● ●
●
●●
●
●
●
● ●● ●● ●●● ● ●● ●
●
●
●
● ●
●
●
●
● ●
●●
●
● ● ●
●
●
●
●
●
●●
●● ● ●
●
●
● ●
●● ●● ● ●
●
●
●
●● ●
●
●
● ●
●
●
● ●● ●●●
●
●
●
● ●●
●
●
●
● ●● ● ● ●●
●●
●
●
●
●
●
●●
●
● ●
●
●
●
●●● ●● ●
●
●
●
● ●
● ●●
●●
●
●
●
●
●
●
●● ●
●
●● ●
●
●● ●● ●●
●
● ●
● ●
● ●●● ●●
●
●
● ●●● ●● ●
●
●
●
●
●
●
●● ●● ●
●●
●● ● ● ●●
●
●●
●
●
●● ●
●
● ●●●
●
●
●
●
● ●
●●
●
●
●
●●●● ●
● ●
●
●● ●
●
●
●●
●
● ●● ●
●
●●
●
●
●
● ●●● ●
●
●●● ●
●
● ●● ●
●
●
●
●
● ●●
●
●
●
●
●
●●
● ● ●
●
●●●●
●
●●●
● ●
●
● ● ●
●
●
●
●
●
●
●
●●
●
● ●● ●
●
●● ●
●
●
●
●
●●
● ● ●●● ●●
●
● ●
●
●
●
●●
●
● ●
●
● ●●● ●
●
●
●
●
●
●
●
●● ● ●● ●●
●
● ●
●
●
● ●●
●
●
●
●●● ●
●
●
●
● ●
●
● ●
●
● ●
●
●
●
● ●
●
●
●
● ●●●● ●● ●●● ● ●
●
●
●
●
●
●
● ●●●● ●
●
●●●
●
● ●
●
●
●● ●● ●●● ●●
●
●
●
●
●
●
●
●
●●●● ●
●
●
●
●
● ●
● ●● ● ●● ●
●
●
●●
●
●●
●
●● ●●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●●
●
●●
●●
●● ●
● ●
●● ● ●
●
●
●
●
●● ●●
●
●
●
●
● ● ●
●
●●
●●●●●
● ●
●
●
●
●
● ●●●●
●
●
●
●
●●●● ●● ●● ●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
● ●●
●
● ●●
●
●
●
●
●● ●●
●
●
● ●
●● ●● ●● ●● ● ●
●
● ●
●
●
●
●
● ●
●
●●
●
●
●
● ●●
●
●
●
● ●●●
●
●
●
●
● ●●● ●●
●
●●
●
● ●
●
●
●
●
● ●●
●
●
●
●●●
● ●
●
●
●
●
●
●
●●● ●●
●
● ●
●
● ● ●
●
●
●●
●
●● ●●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
● ●● ● ●
●
●
●
●● ●
●
●● ●
●
●●●●● ●●
●
●
●
●●●●
●
●
● ● ●● ●●● ● ●●
● ●
● ●
●
●
●
●
●
● ●● ●● ●
●●
●
●●
●
● ●● ● ●●●
●
●●
●
●
●
●
●
●● ●● ● ●
●
● ●
●
●
●
●
●
● ● ●●●● ●●
●
● ●
●
●
●
●
●
● ● ●●●
●
●
●
●
●
●
●●
●
●● ●●●● ●
●
●
●
●● ●●●
●
●
● ●●
●
● ●●
●
● ●●● ● ●●
●
●●●● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●● ●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●●
●
● ●● ● ●
● ●
●●
●
●●● ●●●
●
●●
● ● ●
●●
●
●
●● ●● ●●
●
●● ●
●
●
●
●
●
●
●
●
● ●● ● ●
●●
●
●
●
●
● ● ●●●●
●
●
● ●
●
●
●
● ●
●
●●●● ●●
●
●
●
●● ● ●● ●
●
●
●
●●
●
●
●● ●●
●
● ●●●● ● ●●●
●
●
● ●
●
●
●
●●
●
● ● ●●●
●
● ●
●
●
●
●●● ●
●
● ●●●
●
● ●●●
●
● ●
●
●
●● ●●● ●● ●●
●●
●
● ●● ●
●
● ●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
●● ●●
●
●● ●
●
●
● ●●●
●
● ●
●
●
●●
●
●
● ● ●
●
●●● ●
●
●
●
●
●
●
● ●
● ●● ●
●
●●
●
●
● ● ●
● ● ●● ●●
●
●
● ● ●
●
●
●● ●
●
●
● ●● ●
●
● ●● ●
●
●
●
●
●●
●
●
●
●
●
●
●● ●●● ●●● ●
●
●
●
●
●
●
● ●● ●
●
● ●●●●
●
● ●
●
●● ●●
●
●
●
●● ● ●●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
● ● ●● ●
●
●● ●
● ●
●
●
●
●
●● ●●
●
●● ● ● ●● ● ●● ● ●
● ●● ●●● ●● ●●
●
● ●● ●● ●●
●
● ●●●
●
● ●
●
●● ●
●
●●● ●
●
● ●●
●
●
●
●
●
●
●
●● ●● ●
●
●●
●
●
●
● ●● ●
●
● ● ●
●
●
●
●
●
●
●● ●
●
●●●
●
●
● ●
●
●● ●
●
●
● ●●● ●●● ●●●
●
●● ● ●
●
● ●● ●
●
● ●● ●
●
●● ●
●
●
●
●
●
●
●
●
●
● ●
● ●●
●
●
●
●
●
●●
● ●
● ●● ●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
● ● ●
●
●
●
●
●
●
● ●
●
● ●
●
●● ●
●
●● ●●
●
●●
●●
● ●●
●
●
● ●●
●
●
●
●
●●
●
●
●
●
● ●
●● ●
●
●
●
●
●
●● ●
●
● ●
●
● ●● ●
●●
● ● ●
●
● ●●
●
●●●
●
●
●
●
●
●● ●●● ●●
●
● ● ●● ● ● ●●●
●
● ●
●
●
●
●
●
● ●●
●
● ●●
●
● ●
●
● ●
●
●● ●
●
● ●
●
●● ● ● ●
●
●
●
●
●● ●● ●
●
● ●
●
●
● ●● ●
● ●
● ●●
●
●● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●
● ●●
●
●● ●●
●● ● ●●●
●
●
●
●
●
●
●
●
● ●
● ●
●
●●
●
●
●
●
●● ● ●
●
●
●
●
●
●
●
● ●●●●
●
● ●● ● ●●
●
●● ●
●
●
●
● ●
●
●●
●
●
●●●●
●
● ●
●
● ●● ●
●
● ●
●
● ●●●
●
●● ● ●
● ●
●
● ● ●
●●● ●
● ● ● ●● ●● ●●
●
● ●● ● ●●
●
●
●
●
● ●
●
●
●●●
●
●
● ●● ● ●
●
●
●
●
●●
●●● ● ●● ●● ●● ● ●● ●
●
●
●●●
●
●●
●
●
●● ●●●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●●
●
● ● ● ●●
●
●
●●
●
● ●● ●
●
● ● ●●●
●●● ●
●
● ●
●
●
●
●
●
●
● ●●● ● ●
●
●
●
●
● ●●●
●
● ●
●
●
●
●
●
● ●● ●●
●
●● ●
●●● ●
●●
● ●
●
●
●●
● ●
●
● ●
● ●●
●
● ●●
●●
●
● ●●
●
● ●
●
●
●●●
●
● ●
●
●
● ●
●● ●●●
●
●
●
● ● ●
●
●
●
●● ●
●
●
●●
●
●
●●● ● ● ●● ●●
●
●
●●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●●● ●●
●
●● ●●
●
● ●● ●●●
●
● ●
●●
●● ●●●●● ● ●
●
●●
● ●
●
●
●
● ●
●
●
● ●
● ●
●
●●
●
●
●
● ●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●
●
●● ●
●
●● ●
●
●
●
●
● ●
●
●
●
●● ●● ●●●
●
●
●
●
●● ●●
●
● ●
●●
●
●
●● ●●
●
●● ● ●
●
●●
●
●
●● ●●
●
●
●
●
●
●
●● ●● ●● ● ●
●
●
●
●●●●
●
●
●
●● ● ●●
●
●
●●
● ● ●
●
●●
●
●
●
● ● ●●
●
●
● ●●●● ●
●
●●
●
●
● ●●●
●
●● ●
●
●
●●
●
●
●
●
●
●●
●
●●●● ●
●
● ●●
●
●
●
●
●
●
● ●
●
●● ● ●
●
●
●
● ●
● ●●
●
●●
●
●
●
● ●● ● ●●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●●● ● ● ●● ●●
●●
●
●
●
●
● ●
●●●
●
●● ●● ●● ●● ● ● ● ●● ● ●● ●●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●
●
●●●
●
●● ●● ●
●
●
●●●
●
●
●●
●
● ●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
● ●
● ● ●● ●
●
● ●●
●
●
●
● ●
●
● ●● ● ● ●
●●
●
●
● ●●
●
●
●
●
●
●●
●●
●
● ●● ●
●
● ●
●●
●● ●● ●
●
●
●●
●
●
●●●●● ●● ●●● ● ●● ●●●
●●●
● ●
●
●
●
●
●
●
●● ● ●
●
● ● ●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
● ●
●
●
●
●●
●
●● ●● ●● ●
●
● ●
●
●● ●
●
●
● ● ●
●
●
●
●
●●●
● ●
●● ●●
●
●
●
●
●
● ●●●
●
●
● ●
● ●
●
● ●
●
● ●● ●
●
●
●
● ● ●
●
●
● ● ●
●
●
●
●
●
● ●●● ●
●
●
●● ●
● ● ●●
●
●
●
●
● ●● ● ●●
●
● ●● ●● ●●
●
●
●
●
● ●● ●●●
●
●
●
●
●
●●●
●●● ●● ● ●●●
●●
●
●
●
●
●
●
● ●●●
●
● ●● ● ●● ●●● ● ●
●
●
●●
●
●
●
●●●● ●
●
●
●
●●
●● ● ●● ●
●
●
●
●
●
● ●●
●
●●
●
● ●
●
●
●
●
● ●
●
●
●●● ●
●
●
● ●● ●● ●●
●
●
● ●● ● ●● ●●●●
●
●
● ●●●●
●
● ●● ●● ●●● ●●
●
● ●● ●● ●
●
● ●
●
●● ●
●
●
●●
●
●● ● ●
●
● ● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●● ●●
●
●
●
● ●
●● ● ● ●
●
●
●
●
●●
●
●● ●●●● ●
●
●
●
●
● ● ●● ● ● ●●●
●
●
● ●
●
●
●● ●●
●
●
●
●
●
●
●● ●
●
●
●●
●
●
●
● ●●●●
●
● ●● ●●
●
● ●●●● ● ●● ●
● ●
●● ●
●
●
●
●
● ●●
●
●● ●●
●
●
●
●● ●
● ●
●
●
●●
●
● ●
●
●
●
●
●
●
● ● ●●● ● ●●● ● ●●
● ●●
●
●
●
●
●
●● ●
●
● ● ●
●
●●
●
●●
●
●
●
●
●
●
● ●● ●●●● ● ● ●●●● ● ●●
●
●
●
●
● ● ●
●
●
● ●●● ●
●
●● ● ●
●
●
●● ●●
●
● ●●
●
●
●
● ●
● ● ● ●●
●
●
●● ● ●●
●
●● ●● ●
●
●
●
●●
●
●
●
●●
● ●
●
●
● ●● ● ●● ●● ●
●
●● ● ●
●
●
●
●
● ●● ●● ●● ●●
●
●
●
● ●● ●● ● ●
●
●
●
● ●●●
●
●
●● ● ●
●
●● ●●● ●
●
●
●
●
● ●
●
●
●
● ●● ●● ●●
●
●
● ●● ●●● ●
●
●●●●
●
●● ●
●
● ●● ● ●● ●●
●
●
●
●
●●● ●
●
●
● ●
●● ●
●
●
●
●●
●
●
●
● ●●● ●
● ●
●
●●●● ●●
●
●● ●
●
●
●●
●
●
●
● ● ●●
●
●●
●
●
●● ●
●
● ●●●
●
●
● ●● ● ●●
●
●● ● ●
●
●
●●
●
●● ●● ●●
●
●● ● ●●
●
●● ●
●
●
●
●●
●
●
●
●
● ●
●● ●● ●● ●●●● ●●
●
●
●●●
●
● ●
●
●
●
●
●
●●
●●●
●
●● ●
●
●
●
●
●
● ●●
●● ● ● ●
●
●● ●● ●●
●
●
●
●● ●
●●
●● ● ●● ●
●
●
● ● ●●●
●
●●●●●
●
●
●
● ●●●●
●
●
●● ●
●
●
●● ●● ●●
●
●
●●●
●
●
●
●
●
●
●
●● ●● ●●● ●
●
●●
●
●
●
●
● ●● ●
●
●
● ●●
●
●
●
●
●
●
●●●●● ● ●● ●● ● ●●
●
● ●● ●●
●
●
●
● ●
●
●
●
●●
●
● ●●●
●
●
●
●●● ●●
●
●
●●● ●●● ●
●
●
●
●
●
●●
●
● ● ●
●
● ●●
● ●● ●●
●
●●●●●● ●
●● ●
●
● ●● ●●●
●
●● ●
●
●
● ●
●
●
●●
●
●
●
● ●
●
●● ●● ●●
●
●●●
●
● ●● ●●●
●
●
●
● ●
●
●
● ●
●
●
●● ●●
●
●●●
●
● ●● ● ●●
●
●
●
●
●
●●
●
●
●
●●
● ●
●
●
● ●● ●●● ●●
●
●●● ● ●
●
●
●
●●
●
●
●
●
● ●●●
●
● ●●
●
●● ● ●
● ● ●● ●
●
● ●● ● ●●
●
●
●
●
●
●
●
●
●● ●
●
● ●● ●
●
●● ●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
● ●
●
●
●● ●
●
● ●
●
●● ●●● ● ●
●
● ●
●
●
0
50000
100000
150000
2.5 5.0 7.5 10.0
rating
votes

I Number of votes (which can be used as a surrogate for
popularity) tend to increase as the average rating increase.
I Spread in the number of votes increases with the average
rating.
I Movies featuring the highest average ratings have a very small
number of votes.
Note that users tend to see movies that they think they will like,
and thus the observed ratings tend to be higher than ratings
gathered after showing users random movies.

Outliers
Two types of outliers
I Corrupted values, for example, human errors during a manual
process of entering measurements in a spreadsheet.
I Substantially unlikely values given our modeling assumptions,
for example Black Monday stock crash on October 19, 1987,
when the Dow Jones Industrial Average lost 22% in one day.
In both cases, data analysis based on outliers may result in
drastically wrong conclusions.

library(Ecdat)
data(SP500, package = 'Ecdat')
qplot(r500,
main = "Histogram of log(P(t)/P(t-1)) for SP500 (1981-91)",
xlab = "log returns",
data = SP500)
0
300
600
900
−0.2 −0.1 0.0 0.1
log returns
count
Histogram of log(P(t)/P(t−1)) for SP500 (1981−91)

qplot(seq(along = r500),
r500,
data = SP500,
geom = "line",
xlab = "trading days since January 1981",
ylab = "log returns",
main = "log(P(t)/P(t-1)) for SP500 (1981-91)")
−0.2
−0.1
0.0
0.1
0 1000 2000
logreturns
log(P(t)/P(t−1)) for SP500 (1981−91)

Robustness
Robustness describes a lack of sensitivity of data analysis
procedures to outliers.
I The mean of n numbers is a non-robust procedure while the
median is a robust procedure.
I Assuming a symmetric distribution of samples around 0, we
expect the mean to be zero, or at least close to it. But, the
presence of a single outlier (very positive value or very
negative value) may substantially a↵ect the mean calculation
and drive it far away from zero, even for large n.
I In contrast the median will not change its value.

Dealing with Outliers
Truncating. Remove all values deemed as outliers.
Winsorization. Replace outliers with the most extreme of the
remaining values.
Robustness. Analyze the data using a robust procedure.

Removing Outliers
To remove outliers we need to ﬁrst detect them.
I Values below the ↵ percentile or above the 100 ↵ percentile
for some small ↵ > 0.
I Values more than c standard deviations away from the mean.
I Chicken-and-egg problem since standard deviation and mean
calculations above will be corrupted by outliers. One solution
is computing the mean and standard deviation after removing
the most extreme values (see next slide). Alternatively
percentile (that are more robust) can be used.

originalData = rnorm(20)
originalData[1] = 1000
sortedData = sort(originalData)
originalData = originalData[3:18]
lowerLimit = mean(sortedData) - 5 * sd(sortedData)
upperLimit = mean(sortedData) + 5 * sd(sortedData)
noOutlierInd = (lowerLimit < originalData) &
(originalData < upperLimit)
dataWithoutOutliers = originalData[noOutlierInd]

library(robustHD)
originalData = c(1000, rnorm(10))
print(originalData[1:5])
## [1] 1000.0000 -0.6265 0.1836 -0.8356 1.5953
print(winsorize(originalData[1:5]))
## [1] 3.2060 -0.6265 0.1836 -0.8356 1.5953

Data Transformations: Skewness and Power
Transformation
I In many cases, data is drawn from a highly-skewed
distribution that is not well described by one of the common
statistical distributions.
I A simple transformation may map the data to a form that is
well described by common distributions, such as the Gaussian
or Gamma distributions
I A suitable model can then be ﬁtted to the transformed data
(if necessary, predictions can be made on the original scale by
inverting the transformation).
Power Transformation Family: replace non-negative data x by
f (x) =
8
><
>:
(x 1)/ > 0
log x = 0
(x 1)/ < 0
x > 0, 2 R.

I Intuitively, the power transform maps x to x , up to
multiplication by a constant and addition of a constant.
I This mapping is convex for > 1 and concave for < 1.
I A choice of < 1 removes right-skewness (data has a heavy
tail to the right) with smaller values of resulting in a more
aggressive removal of skewness. Similarly, a choice of > 1
removes left-skewness.
I Subtracting 1 and dividing by makes f (x) continuous in
as well as in x.
I One way to select is to try di↵erent values, graph the
resulting histograms, and select one of them. There are also
more sophisticated methods for selecting based on the
maximum likelihood method.

print(diamonds[1:10,1:8])
## carat cut color clarity depth table price x
## 1 0.23 Ideal E SI2 61.5 55 326 3.95
## 2 0.21 Premium E SI1 59.8 61 326 3.89
## 3 0.23 Good E VS1 56.9 65 327 4.05
## 4 0.29 Premium I VS2 62.4 58 334 4.20
## 5 0.31 Good J SI2 63.3 58 335 4.34
## 6 0.24 Very Good J VVS2 62.8 57 336 3.94
## 7 0.24 Very Good I VVS1 62.3 57 336 3.95
## 8 0.26 Very Good H SI1 61.9 55 337 4.07
## 9 0.22 Fair E VS2 65.1 61 337 3.87
## 10 0.23 Very Good H VS1 59.4 61 338 4.00

diamondsSubset = diamonds[sample(dim(diamonds)[1], 1000),]
qplot(price, data = diamondsSubset)
0
50
100
150
200
0 5000 10000 15000 20000
price
count

qplot(log(price), size = I(1), data = diamondsSubset)
0
20
40
6 7 8 9 10
log(price)
count

I Power transformations are useful also for examining the
relationship between two or more data variables.
I The following plot shows the relationship between diamond
price and diamond carat. It is hard to draw much information
from that plot beyond the fact that there is a non-linear
increasing trend.
I Transforming both variables using a logarithm shows a striking
linear relationship on a log-log scale.

qplot(carat,
price,
size = I(1),
data = diamondsSubset)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
0
5000
10000
15000
1 2 3 4
carat
price

qplot(carat,
log(price),
size = I(1),
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
6
7
8
9
10
1 2 3 4
carat
log(price)

qplot(log(carat),
price,
size = I(1),
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
● ● ●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
0
5000
10000
15000
−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5
log(carat)
price

qplot(log(carat),
log(price),
size = I(1),
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
6
7
8
9
10
−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5
log(carat)
log(price)

Data Analysis with R (combined slides)

Data Analysis with R (combined slides)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Analysis with R (combined slides)

Similar to Data Analysis with R (combined slides) (20)

More from Guy Lebanon

More from Guy Lebanon (6)

Recently uploaded

Recently uploaded (20)

Data Analysis with R (combined slides)