fast-matmul-cse15
•Download as PPTX, PDF•
0 likes•248 views
This document presents a framework for practically implementing parallel fast matrix multiplication algorithms. It compares the performance of several Strassen-like algorithms to the Intel MKL library on sequential and parallel systems with up to 24 cores. Certain Strassen-like algorithms like <4,2,4> and <3,2,3> generally outperform MKL for sequential and small parallel problems, while MKL performs best for large parallel problems. Different parallelization strategies like depth-first, breadth-first, and hybrid are explored, with hybrid providing better load balancing.
![Fast matrix multiplication:
bridging theory and practice
• There are a number of Strassen-like algorithms for matrix
multiplication that have only been “discovered” recently.
[Smirnov13], [Benson&Ballard14]
• How well do they work in practice?
2
32 2.81
[Strassen79]
2.37
[Le Gall14]
xxx xx xx xx](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to fast-matmul-cse15
Similar to fast-matmul-cse15 (20)
More from Austin Benson
More from Austin Benson (20)
Recently uploaded
Recently uploaded (20)
fast-matmul-cse15
- 1. A FRAMEWORK FOR PRACTICAL PARALLEL FAST MATRIX MULTIPLICATION code and paper: github.com/arbenson/fast-matmul 1 Austin Benson arbenson@stanford.edu Stanford University Joint work with Grey Ballard, Sandia SIAM CSE 2015 Salt Lake City, UT 9000 10000 11000 12000 13000 14 16 18 20 22 Dimension (N) EffectiveGFLOPS/core Performance (24 cores) on N x N x N MKL STRASSEN S<4,3,3> <4,2,2> <3,2,3> <3,3,2> <5,2,2> <2,5,2>
- 2. Fast matrix multiplication: bridging theory and practice • There are a number of Strassen-like algorithms for matrix multiplication that have only been “discovered” recently. [Smirnov13], [Benson&Ballard14] • How well do they work in practice? 2 32 2.81 [Strassen79] 2.37 [Le Gall14] xxx xx xx xx
- 4. 4 [Smirnov13] [Strassen69] All implemented with code generation
- 5. Sequential performance = 5 Effective GFLOPS for M x K x N multiplies = 1e-9 * 2 * MKN / time in seconds Classical peak 0 2000 4000 6000 8000 16 18 20 22 24 26 28 Dimension (N) EffectiveGFLOPS Sequential performance on N x N x N MKL STRASSEN <4,2,2> <3,2,3> <3,3,2> <5,2,2> S<4,3,3>
- 6. 2000 4000 6000 8000 10000 12000 20 22 24 26 28 dimension (N) EffectiveGFLOPS Sequential performance on N x 1600 x N MKL <4,2,4> <4,3,3> <3,2,3> <4,2,3> STRASSEN Sequential performance = • Almost all algorithms beat MKL • <4, 2, 4> and <3, 2, 3> tend to perform the best 6
- 7. DFS Parallelization C M1 M7 + M2 … M1 M7 + M2 … All threads Use parallel MKL + Easy to implement - Need large base cases for high performance 7
- 8. BFS Parallelization C M1 M7 + M2 … M1 M7 + M2 … omp taskwait omp taskwait 1 thread + High performance for smaller base cases - Sometimes harder to load balance: 24 threads, 49 subproblems - More memory 1 thread 1 thread 8
- 9. HYBRID parallelization C M1 M7 + M2 … M1 M7 + M2 … omp taskwait omp taskwait 1 thread 1 thread all threads + Better load balancing - Explicit synchronization or else we can over-subscribe threads 9
- 10. Parallel performance = 10 9000 10000 11000 12000 13000 18 20 22 24 26 28 Dimension (N) EffectiveGFLOPS/core Performance (6 cores) on N x N x N MKL STRASSEN S<4,3,3> <4,2,2> <3,2,3> <3,3,2> <5,2,2> <2,5,2> 9000 10000 11000 12000 13000 14 16 18 20 22 Dimension (N) EffectiveGFLOPS/core Performance (24 cores) on N x N x N MKL STRASSEN S<4,3,3> <4,2,2> <3,2,3> <3,3,2> <5,2,2> <2,5,2> • 6 cores: similar performance to sequential • 24 cores: can sometimes edge out MKL
- 11. 10000 15000 20000 10000 15000 18 19 20 21 22 23 24 dimension (N) EffectiveGFLOPS/core Performance (6 cores) on N x 2800 x N MKL <4,2,4> <4,3,3> <3,2,3> <4,2,3> STRASSEN 5000 10000 15000 20000 12 14 16 18 20 dimension (N) EffectiveGFLOPS/core Performance (24 cores) on N x 2800 x N MKL <4,2,4> <4,3,3> <3,2,3> <4,2,3> STRASSEN Parallel performance = • 6 cores: similar performance to sequential • 24 cores: MKL best for large problems 11
- 12. A FRAMEWORK FOR PRACTICAL PARALLEL FAST MATRIX MULTIPLICATION code and paper: github.com/arbenson/fast-matmul 12 Austin Benson arbenson@stanford.edu Stanford University Joint work with Grey Ballard, Sandia SIAM CSE 2015 Salt Lake City, UT 9000 10000 11000 12000 13000 14 16 18 20 22 Dimension (N) EffectiveGFLOPS/core Performance (24 cores) on N x N x N MKL STRASSEN S<4,3,3> <4,2,2> <3,2,3> <3,3,2> <5,2,2> <2,5,2>
Editor's Notes
- \omega
- \begin{bmatrix} C_{11} & C_{12} \\ C_{21} & C_{22} \end{bmatrix} = \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix} \cdot \begin{bmatrix} B_{11} & B_{12} \\ B_{21} & B_{22} \end{bmatrix} S_1 &= A_{11} + A_{22} \\ S_2 &= A_{21} + A_{22} \\ S_3 &= A_{11} \\ S_4 &= A_{22} \\ S_5 &= A_{11} + A_{12} \\ S_6 &= A_{21} - A_{11} \\ S_7 &= A_{12} - A_{22} \\ T_1 &= B_{11} + B_{22} \\ T_2 &= B_{11} \\ T_3 &= B_{12} - B_{22} \\ T_4 &= B_{21} - B_{11} \\ T_5 &= B_{22} \\ T_6 &= B_{11} + B_{12} \\ T_7 &= B_{21} + B_{22} \\
- \begin{table} \begin{tabular}{l c c c c c} Algorithm & Multiples & Multiplies & Speedup per & \\ base case & (fast) & (classical) & recursive step & Exponent\\ $\langle 2,2,3\rangle $ & 11 & 12 & 9\% & 2.89 \\ $\langle 2,2,5\rangle $ & 18 & 20 & 11\% & 2.89\\ $\langle 2,2,2\rangle $ & 7 & 8 & 14\% & 2.81 \\ $\langle 2,2,4\rangle $ & 14 & 16 & 14\% & 2.85\\ $\langle 3,3,3\rangle $ & 23 & 27 & 17\% & 2.85 \\ $\langle 2,3,3\rangle $ & 15 & 18 & 20\% & 2.81 \\ $\langle 2,3,4\rangle $ & 20 & 24 & 20\% & 2.83\\ $\langle 2,4,4\rangle $ & 26 & 32 & 23\% & 2.82 \\ $\langle 3,3,4\rangle $ & 29 & 36 & 24\% & 2.82 \\ $\langle 3,4,4\rangle $ & 38 & 48 & 26\% & 2.82 \\ $\langle 3,3,6\rangle $ & 40 & 54 & 35\% & 2.77 \\ \end{tabular} \end{table}
- \begin{eqnarray*} S_7 &=& A_{12} - A_{22} \\ T_7 &=& B_{21} + B_{22} \\ M_7 &=& S_7 \cdot T_7 \end{eqnarray*}
- \begin{eqnarray*} S_7 &=& A_{12} - A_{22} \\ T_7 &=& B_{21} + B_{22} \\ M_7 &=& S_7 \cdot T_7 \end{eqnarray*}