R workshop xx -- Parallel Computing with R

PPaarraalllleellccoommppuuttiinnggiinnRR
Vivian Zhang, Yuan Huang, Tong He
SupStat Inc
Parallel computing in R http://nycdatascience.com/slides/parallel_R/index.html#1
1 of 28 6/12/14, 5:26 PM

OOuuttlliinnee
Introduction to Parallel computing
Implementation in R
Examples
·
·
Overview
Package: Foreach
-
-
·
2/28
2 of 28 6/12/14, 5:26 PM

Introduction to Parallel computing
3/28
3 of 28 6/12/14, 5:26 PM

SSeerriiaallvvssPPaarraalllleellCCoommppuuttaattiioonn
Serial Computation
Traditionally, software is written for serial computation, where tasks must be performed in sequence
on a single processor. Only one instruction may execute at any moment in time.
Illustration of Serial Computation
4/28
4 of 28 6/12/14, 5:26 PM

SSeerriiaallvvssPPaarraalllleellCCoommppuuttaattiioonn
Parallel Computation
Parallel computing aims to speed up the computation. In parallel computing,
Illustration of Parallel Computation
The problem is broken apart into discrete pieces of work.
Instructions from each part execute simultaneously on different processors.
·
·
5/28
5 of 28 6/12/14, 5:26 PM

PPaarraalllleellppaarraaddiiggmm
Master-worker paradigm
Master submits jobs to workers and collect results from workers.
No communications between workers
·
·
6/28
6 of 28 6/12/14, 5:26 PM

SStteeppssttoowwaarrddssPPaarraalllleellccoommppuuttiinngg
Hardware platforms1.
Whether the problem is parallelable ?2.
Tips to improve the parallel computing's effeciency.3.
Implementation in R.4.
7/28
7 of 28 6/12/14, 5:26 PM

HHaarrddwwaarreeppllaattffoorrmmss
Two representative Hardware platforms are multicore and cluster.
Multicore: Most of the PCs have multiple processors on a single chip, which enables the parallel
computing. CPU is subdivided into multiple "cores", each being a unique execution unit.
Cluster: A set of independent nodes that are connected by a certain network.
·
·
8/28
8 of 28 6/12/14, 5:26 PM

WWhheetthheerrtthheepprroobblleemmiissppaarraalllleellaabbllee??
Recap: parallel computing requires that
Example:
The problem can be broken apart into discrete pieces of work.
Instructions from each part can execute simultaneously on different processors.
·
·
vec <- runif(10)
# sum(vec): i-th itervation uses the result from (i-1)-th iteration.
sum.vec <- 0
for (i in seq(vec)) sum.vec <- sum.vec+vec[i]
# cumsum(vec): i-th itervation are independent from (i-1)-th iteration.
cumsum.vec <- 0*seq(vec)
for (i in seq(vec)) cumsum.vec[i] <- sum(vec[1:i])
9/28
9 of 28 6/12/14, 5:26 PM

IIsstthhiissaaggooooddppaarraalllleelliizzaattiioonnsscchheemmee??
For this cumsum example,
Scheme: Prepare 10 cores and let each core implement one sum(vec[1:i]).
Question: Is this a good parallel scheme?
vec <- runif(10)
# cumsum(vec): i-th itervation are independent from (i-1)-th iteration.
cumsum.vec <- 0*seq(vec)
for (i in seq(vec)) cumsum.vec[i] <- sum(vec[1:i])
10/28
10 of 28 6/12/14, 5:26 PM

IIsstthhiissaaggooooddppaarraalllleelliizzaattiioonnsscchheemmee??
No, because different i may have result in quite widely different computation time, which may bring in
a serious load balance issue.
Consider load balancing!
11/28
11 of 28 6/12/14, 5:26 PM

LLooaaddbbaallaanncciinngg
Load balancing aims to spread tasks evenly across processors.
When tasks and processors are not load balanced:
Some processes ﬁnish early and sit idle waiting
Global computation is ﬁnished when the slowest processor(s) completes its task.
·
·
12/28
12 of 28 6/12/14, 5:26 PM

PPaarraalllleelloovveerrhheeaadd
The amount of time required to coordinate parallel tasks, as opposed to doing useful work. Parallel
overhead can include factors such as:
Communication is much slower than computation; care should be taken to minimize unnecessary
data transfer to and from workers.
Assign the tasks in chunck help parallel overhead.
Task start-up time
Synchronizations
Data communications
Software overhead imposed by parallel languages, libraries, operating system, etc.
·
·
·
·
13/28
13 of 28 6/12/14, 5:26 PM

RRaannddoommnnuummbbeerrggeenneerraattoorrss
Random number generators require extra care. Random number streams on different nodes need to
be independent. It is important to avoid producing the same random number stream on each worker
and at the same time be able to facilitate reproducible research.
Special-purpose packages (rsprng, rlecuyer) are available; the snow package provides an integrated
interface to these packages.
In snow package, use the following code before the simulations.
clusterSetupSPRNG(cl)
14/28
14 of 28 6/12/14, 5:26 PM

AApppplliiccaattiioonnssoonnssttaattiissttiiccaallmmooddeelliinngg
Model selection
Data mining
Monte Carlo simulations
Boostrap (see examples)
Web Scrapper (see examples)
Subset selection
Tuning parameter selection (eg. tuning in regularized regression)
K-fold crossvalidation(see examples)
·
·
·
Random forest (see examples)
Clustering (see examples)
Principle component Analysis
·
·
·
15/28
15 of 28 6/12/14, 5:26 PM

Implementation in R
16/28
16 of 28 6/12/14, 5:26 PM

OOvveerrvviieeww::PPaacckkaaggeess
Use foreach + doSNOW.
Rmpi ( R interface to MPI; ﬂexible; powerful, but more complex.)
Snow (will be used for backends with foreach package today)
multicore (work only on a single node and Linux-like machine)
parallel (hybrid package containing snow and multicore)
foreach (parallel backends doSNOW / doMPI / doMC)
·
·
·
·
·
17/28
17 of 28 6/12/14, 5:26 PM

ffoorreeaacchh::OOnneeRRiinnggttooRRuulleeTThheemmAAllll
foreach was written by Steve Weston (was in Revolution, now at Yale)
an elegant framework for parallel computing: loop construct + parallel execution
allows the user to specify the parallel environment. The parallel backends include:
·
·
·
doMC (multicore),
doSNOW (snow)
doMPI (Rmpi)
doParallel (parallel)
...
-
-
-
-
-
18/28
18 of 28 6/12/14, 5:26 PM

RReeggiisstteerrtthheeppaarraalllleellbbaacckkeennddss::
The doMC package acts as an interface between foreach and the multicore functionality.
The doSNOW package acts as an interface between foreach and the snow functionality.
# Register multicore as backend.
library(doMC)
registerDoMC(2)
foreach code
# Register snow as backend.
library(doSNOW)
cl <- makeCluster(2, type="SOCK")
registerDoSNOW(cl)
foreach code
stopCluster(cl)
19/28
19 of 28 6/12/14, 5:26 PM

PPaacckkaaggee::ssnnooww
Interfaces provided by snow package include:
MPI: Message Passing Interface, via Rmpi1.
NWS: NetWork Spaces via nws2.
PVM: Parallel Virtual Machine3.
Sockets: via the operating system4.
20/28
20 of 28 6/12/14, 5:26 PM

RReeggiisstteerrtthheeppaarraalllleellbbaacckkeennddss
Get the name of the currently registered backend:
Get the version of the currently registered backend:
Check how many workers foreach is going to use
getDoParName()
getDoParVersion()
getDoParWorkers()
21/28
21 of 28 6/12/14, 5:26 PM

ffoorreeaacchhccooddee::
Syntax
foreach object %do% expression1.
foreach object %dopar% expression2.
# here for(i=1:4) is the foreach object. we call i the iterator.
x <- foreach(i=1:4) %dopar% {exp(i)}
foreach object: Specify Looping sturcture, similar to for loop. (for -> foreach)
%do%, %dopar% : Specify excecution method.
expression: Specify how to excecute in each process.
·
·
%do% Execute the R expression sequentially
%dopar% Execute the R expression using the currently registered backend
-
-
·
22/28
22 of 28 6/12/14, 5:26 PM

Return is a list by default. Use .combine argument in foreach object to change.
library(doSNOW) # Register snow as backend.
registerDoSNOW(cl)
foreach(i=1:3) %dopar% {i^2} # foreach code
[[1]]
[1] 1
[[2]]
[1] 4
[[3]]
[1] 9
stopCluster(cl)
23/28
23 of 28 6/12/14, 5:26 PM

Set ".combine='c'" to obtain the return in vector.
library(doSNOW)
registerDoSNOW(cl)
# foreach code
foreach(i=1:3, .combine='c') %dopar% {i^2}
[1] 1 4 9
stopCluster(cl)
24/28
24 of 28 6/12/14, 5:26 PM

ffoorreeaacchhccooddee::ttwwooiitteerraattoorrss..
library(doSNOW)
registerDoSNOW(cl)
# foreach code: here we have two itrators, i and j.
foreach(i=1:3,j=4:6, .combine='c') %dopar% {i+j}
[1] 5 7 9
foreach(i=1:3,j=4:9, .combine='c') %dopar% {i+j}
[1] 5 7 9
stopCluster(cl)
25/28
25 of 28 6/12/14, 5:26 PM

NNeessttiinnggtthheellooooppss
Syntax
foreach object 1 %:%
foreach object 2 %dopar%
{ expression }
Example
# foreach code
bvec <- c(1,2,3)
avec <- c(-1,-2,-3)
x <-
foreach(b=bvec, .combine='c') %:%
foreach(a=avec, .combine='c') %dopar% {
a + b
}
26/28
26 of 28 6/12/14, 5:26 PM

NNeessttiinnggtthheellooooppss
library(doSNOW)
registerDoSNOW(cl)
# foreach code
bvec <- c(1,2,3)
avec <- c(-1,-2,-3)
x <-
foreach(b=bvec, .combine='c') %:%
foreach(a=avec, .combine='c') %dopar% {
a + b
}
stopCluster(cl)
27/28
27 of 28 6/12/14, 5:26 PM

RReeffeerreennccee
Good Overview: State of the Art in Parallel Computing with R, Journal of Statistical Software.1.
Hand-on tutorial:2.
Comprehensive textbook:3.
Package foreach, Steve Weston
Using The foreach Package,Steve Weston
Nesting Foreach Loops,Steve Weston
Getting Started with doMC and foreach,Steve Weston
·
·
·
·
Programming on Parallel Machines, Norm Matlof
Introduction to Parallel Computing, Blaise Barney
·
·
28/28
28 of 28 6/12/14, 5:26 PM

R workshop xx -- Parallel Computing with R

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (8)

Ähnlich wie R workshop xx -- Parallel Computing with R

Ähnlich wie R workshop xx -- Parallel Computing with R (20)

Mehr von Vivian S. Zhang

Mehr von Vivian S. Zhang (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

R workshop xx -- Parallel Computing with R