Slides from my talk at IEEE BigData 2013.

1. Direct QR factorizations for tall-and-skinny
matrices in MapReduce architectures
Austin Benson
ICME, Stanford University
David Gleich (Purdue) and Jim Demmel (UC-Berkeley)
A Q
R
m
n
m
n
n
n
IEEEDATA
October 8, 2013

2. Contributions 2
I Numerically stable and scalable algorithm for QR and SVD of
tall-and-skinny matrices in MapReduce
I Performance and stability tradeos of several methods
I Performance model: prediction within a factor of two
I Code: https://github.com/arbenson/mrtsqr

3. MapReduce overview 3
Two functions that operate on key value pairs:
(key; value)
map
! (key; value)
(key; hvalue1; : : : ; valueni) reduce
! (key; value)
shue stage between map and reduce to sort values by key.

4. MapReduce overview 4
The programmer implements:
I map(key, value)
I reduce(key, h value1, : : :, valuen i)
Handled by MapReduce framework, e.g., Hadoop:
I shue
I load balancing
I reading and writing data
I data serialization
I fault tolerance
I ...

5. MapReduce Example: ColorCount 5
(key, value) input is (image id, image)
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
shuffle
Map 1 Reduce
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
5
2
1
1
4
2
def ColorCountMap(key , val ) :
for pixel in val :
yield ( pixel , 1)
def ColorCountReduce(key , vals ) :
total = sum( vals )
yield (key , total )

6. Why MapReduce? (for scientists) 6
MapReduce is restrictive! Why Bother?
I Easy
I load balancing
I structured data I/O
I fault tolerance
I cheap clusters with large data storage
Hadoop may not be the best option...
Generate lots of
data on
supercomputer
Post-process and
analyze on
MapReduce cluster

7. Tall-and-skinny matrices 7
What are tall-and-skinny matrices? m n
n
m A
Examples: rows are data samples; blocks of A are images from a
video; Krylov subspaces; unrolled tensors

8. Matrix representation 8
We have matrices, so what are the key-value pairs?
A =
2
1:0 0:0
2:4 3:7
0:8 4:2
9:0 9:0
664
3
775
!
2
(1; [1:0; 0:0])
(2; [2:4; 3:7])
(3; [0:8; 4:2])
(4; [9:0; 9:0])
664
3
775
(key, value) ! (row index, row)

9. Matrix representation: an example 9
Scienti

10. c example: (x, y, z) coordinates and model number:
((47570,103.429767811242,0,-16.525510963787,iDV7), [0.00019924
-4.706066e-05 2.875293979e-05 2.456653e-05 -8.436627e-06 -1.508808e-05
3.731976e-06 -1.048795e-05 5.229153e-06 6.323812e-06])
Figure: Aircraft simulation data. Aero/Astro Department, Stanford

11. Tall-and-skinny matrices 10
Tall-and-skinny: m n
n
m A
Slightly more rigorous de

12. nition:
It is cheap to pass O(n2) data to all processors.

13. Quick QR and SVD review 11
A Q
R
VT
n
n
n
n
n
n
n
n A U
n
n
n
n
Σ
n
n
Figure: Q, U, and V are orthogonal matrices. R is upper triangular and
is diagonal with decreasing, nonnegative entries.

14. Tall-and-skinny QR 12
A Q
R
m
n
m
n
n
n
Tall-and-skinny (TS): m n. QTQ = I .

15. TS-QR ! TS-SVD 13
A Q
R Σ
VT
Q
UR
U
R is small, so computing its SVD is cheap.

16. Why Tall-and-skinny QR and SVD? 14
1. Regression with many samples
2. Principle Component Analysis (PCA)
3. Model Reduction
Pressure, Dilation, Jet Engine
Figure: Dynamic mode decomposition of the screech of a jet. Joe
Nichols, University of Minnesota.

17. Cholesky QR 15
Cholesky QR
ATA = (QR)T (QR) = RTQTQR = RTR
I Computing ATA in MapReduce is easy and well-studied.
I We call this Cholesky QR.

18. Cholesky QR: Getting Q 16
Q = AR1.
A
A1
R-1 map
R
Q1
A2
R-1 map
Q2
A3
R-1 map
Q3
A4
R-1 map
Q4
emit
emit
emit
emit
distribute
Local MatMul

19. Stability problems 17
I Can get Q = AR1
I Problem: Columns can be far from orthogonal and ATA
squares condition number (data later)
I Idea: Use a more advanced algorithm.

20. Communication-avoiding TSQR 18
A =
2
A1
A2
A3
A4
664
3
775
| {z }
8n4n
=
2
Q1
664
Q2
Q3
Q4
3
775
| {z }
8n4n
2
R1
R2
R3
R4
664
3
775
| {z }
4nn
=
=Q z2 }| {
Q1
664
Q2
Q3
Q4
3
775
| {z }
8n4n
~Q
|{z}
4nn
|{Rz}
nn
Demmel et al. 2008

21. Communication-avoiding TSQR 19
A =
2
A1
A2
A3
A4
664
3
775 | {z }
8n4n
=
2
Q1
664
Q2
Q3
Q4
3
775
| {z }
8n4n
2
R1
R2
R3
R4
664
3
775
| {z }
4nn
Ai = QiRi can be computed in parallel. If we only need R, then
we can throw out the intermediate Qi factors.

22. MapReduce TSQR 20
S(1)
A
A1
A3
A3
R1 map
A2
R emit 2 map
R emit 3 map
A4
R emit 4 map
shuffle
S1
A2
reduce
S2
R2,1 emit
reduce R2,2
emit
emit
shuffle
AS23
reduce R2,3 emit
Local TSQR
identity map
SA(22 )
reduce R emit
Local TSQR Local TSQR
Figure: S(1) is the matrix consisting of the rows of all of the Ri factors.
Similarly, S(2) consists of all of the rows of the R2;j factors.

23. MapReduce TSQR: Getting Q 21
I Again: have R, want Q
A = QR ! Q = AR1
I We call this method Indirect TSQR.
I Problem: Q can be far from orthogonal (again).

24. Indirect TSQR: Iterative Re

25. nement 22
Iterative re

26. nement: repeat TSQR for a more orthogonal Q
A
A1
R-1 map
R
Q1
A2
R-1 map
Q2
A3
R-1 map
Q3
A4
R-1 map
Q4
emit
emit
emit
emit
distribute
TSQR
Q
Q1
R1
-1 map
R1
Q1
Q2
R1
-1 map
Q2
Q3
R1
-1 map
Q3
Q4
R1
-1 map
Q4
emit
emit
emit
emit
distribute
Local MatMul Local MatMul
Iterative Refinement step

27. Indirect TSQR: a randomized approach 23
I Idea: Take a small sample of rows of A and form Rs
s , Qs ! R1, Q = QsR1
I Re

28. nement step by Qs = AR1
1
I R = R1Rs , QTQ I for ill-conditioned A
I Theory on why this works, need 100n log n rows
[Mahoney 2011], [Avron, Maymounkov, and Toledo 2010],
[Ipsen and Wentworth 2012]
We call this Pseudo-Iterative Re

29. nement

30. Pseudo-Iterative Re

31. nement 24
A
A1
Rs
-1 map
Rs
Q1
A2
Rs
-1 map
Q2
A3
Rs
-1 map
Q3
A4
Rs
-1 map
Q4
emit
emit
emit
emit
distribute
TSQR
Q
Q1
R1
-1 map
R1
Q1
Q2
R1
-1 map
Q2
Q3
R1
-1 map
Q3
Q4
R1
-1 map
Q4
emit
emit
emit
emit
distribute
Local MatMul Local MatMul
Iterative Form Qs Refinement step
A1
TSQR
Form Rs
(In the implementation, combine AR1
s and TSQR in one pass)

32. Direct TSQR 25
Why is computing truly orthogonal Q dicult in MapReduce?
I Orthogonality is a global property, but we compute locally.
I Can only label data via keys and

33. le names.

34. Communication-avoiding TSQR 26
A =
2
A1
A2
A3
A4
664
3
775
| {z }
8n4n
=
2
Q1
664
Q2
Q3
Q4
3
775
| {z }
8n4n
2
R1
R2
R3
R4
664
3
775
| {z }
4nn
=
2
664 Q1
Q2
Q3
Q4
3
775
| {z }
8n4n
2
Q1;2
Q2;2
Q3;2
Q4;2
664
3
775
| {z }
4nn
|{Rz}
nn
=
2
Q1Q1;2
Q2Q2;2
Q3Q1;2
Q4Q1;2
664
3
775
| {z }
8nn
|{Rz}
nn
= QR

35. Gathering Q 27
2
R1
R2
R3
R4
664
3
775 | {z }
n#(mappers)n
=
2
Q1;2
Q2;2
Q3;2
Q4;2
664
3
775
| {z }
n#(mappers)n
|{Rz}
nn
I Idea: Compute QR (n #(mappers) rows) in serial.
I Idea: Pass Qi ;2 (n rows each) in second pass to reconstruct Q.
I We call this Direct TSQR

36. Direct TSQR: Steps 1 and 2 28
A
A1
map R1
Q1 emit
emit
A2
map R2
Q2 emit
emit
A3
map R3
Q3 emit
emit
A4
map R4
Q4 emit
emit
First step
R1
R2
R3
R4
Q1,2
Q2,2
Q3,2
Q4,2
R
reduce
emit
emit
emit
emit
emit
Second step
shuffle

37. Direct TSQR: Step 3 29
Q1
Q1,2
emit map Q
Q2
Q2,2
emit map Q
Q3
Q3,2
emit map Q
Q41
Q4,2
emit map Q
Q12 Q22
Q32 Q42
distribute
Third step

38. Stability 30
0
2
10
0
10
−2
10
−4
10
−6
10
−8
10
−10
10
−12
10
−14
10
−16
10
2
10
4
10
6
10
8
10
10
10
12
10
14
10
16
10
10
(A)
k
2
2
||QTQ − I||
Numerical stability: 10,000x10 matrices
Indir. TSQR + PIR
Dir. TSQR
Indir. TSQR
Indir. TSQR + IR
Chol.
Chol. + IR

39. Performance model 31
I Only count reads and writes
I Streaming benchmark for read and write bandwidth of system
I Within a factor of two of experimental data for all algorithms
I I/O dominates runtime
I Algorithms take same time as a few passes over data

40. Performance 32
4B x 4
(134.6 GB)
2.5B x 10
(193.1 GB)
600M x 25
(112.0 GB)
500M x 50
(183.6 GB)
150M x 100
(109 GB)
Matrix size
7000
6000
5000
4000
3000
2000
1000
0
Time to solution (seconds)
Performance of QR algorithms on MapReduce
Chol
Indir TSQR
Chol + PIR
Indir TSQR + PIR
Chol + IR
Indir TSQR + IR
Direct TSQR

41. Direct TSQR: recursive extension 33
2
R1
R2
R3
R4
664
3
775
| {z }
n#(mappers)n
TSQR
!
2
Q1;2
Q2;2
Q3;2
Q4;2
664
3
775
| {z }
n#(mappers)n
|{Rz}
nn
I n #(mappers) rows is too large ! recurse

42. Direct TSQR: recursive performance 34
0 50 100 150 200
6000
4000
2000
0
number of columns
running time (s)
150M rows
0 50 100 150 200 250
8000
6000
4000
2000
0
number of columns
running time (s)
100M rows
0 50 100 150 200 250 300
15000
10000
5000
0
number of columns
running time (s)
50M rows
no recursion
recursion
no recursion
recursion
no recursion
recursion

43. End 35
Contributions:
I Numerically stable and scalable QR
I Performance and stability tradeos
I Performance model
I Code: https://github.com/arbenson/mrtsqr
Contact:
I arbenson@stanford.edu