Talk given at ODSC East 2022

1. Understanding and optimizing
parallelism in NumPy-based programs
Ralf Gommers
21 April 2022

2. First make it work, then make it fast
>>> %timeit main()
50.1 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> # ... perform some optimizations
>>> %timeit main()
9.58 ms ± 22.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> # break out your profiler (e.g., py-spy), optimize some more
>>> %timeit main()
2.83 ms ± 30.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

3. `htop` output

4. Approaches for performant numerical
code (single-threaded)
Vectorization Use compiled code
Python compilers Python interpreters
Pythran
CPython
Plus Cinder, Pyston, and more -- very experimental,
and limited gains for numerical code

5. multiprocessing & multithreading

6. A key issue: oversubscription
Package A sees N CPU cores, and decides to use them all:

7. A key issue: oversubscription
Package B, which uses package A, or the end user decides to use
multiprocessing, 1 process per core:
The more CPU cores a machine has, the worse the effect is!

8. Parallel APIs & behavior: NumPy
NumPy is single-threaded, no code in NumPy is written for parallel execution.
However, most numpy.linalg functions (those using BLAS or LAPACK) execute in
parallel. They use all available cores on a machine.
NumPy does release the GIL wherever it can.
numpy.random has specific APIs to allow users to:
(a) Obtain independent streams for random number generation across
processes (local or distributed)
(b) Perform multithreaded random number generation

9. Parallel APIs & behavior: SciPy
SciPy is single-threaded by default (same as NumPy)
Calls to functionality using BLAS or LAPACK is again multithreaded:
● primarily in scipy.linalg and scipy.sparse.linalg,
● also higher-level functionality using linear algebra under the hood:
kernel density estimation, multivariate distributions etc. in scipy.stats,
vector quantization in scipy.cluster, interpolators in scipy.interpolate,
optimizers in scipy.optimize, and more
Some APIs have a workers=1 keyword, which allows the user to control the
number of processes or threads. Or pass in a custom Pool.
scipy.fft provides a context manager:

10. Parallel APIs & behavior: SciPy
An example using workers=:

11. Parallel APIs & behavior: scikit-learn
Scikit-learn is mostly single-threaded by default.
However, more and more functionality uses OpenMP for automatic
parallelization. This defaults to the number of virtual (not physical) CPU cores.
Many scikit-learn APIs offer a n_jobs= keyword to let user enable multiple
threads or processes via joblib.
Scikit-learn implements fairly complex control of NumPy/SciPy’s BLAS and
LAPACK libraries to prevent oversubscription in the presence of
multiprocessing on top of multi-threading. This is done via the threadpoolctl
package.

12. Controlling parallelism - packages
Dependencies (Conda)

13. Controlling parallelism - packages
Dependencies (PyPI)

14. Controlling parallelism - packages
Conda PyPI

15. Tuning the default behavior
Default behavior is inconsistent: too aggressive for linear algebra, and too
conservative for workers (SciPy) and n_jobs (scikit-learn)!
OpenBLAS, MKL and OpenMP don’t have a nice API, only environment variables:
For scikit-learn you can explicitly choose a backend (but defaults are usually fine):

16. Tuning the default behavior
NumPy, SciPy and scikit-learn all recommend using threadpoolctl in case you
want more granular control over threading behavior of BLAS, LAPACK and
OpenMP libraries (or cannot set environment variables):

17. A pitfall on multi-tenant machines
Multi-tenant machines: N “vCPU” (virtual CPU) cores for you, M in total.
CircleCI gives you 2 cores for a CI job, on a 64 core machine (and
os.cpu_count() reports 64). Set OPENBLAS_NUM_THREADS=2 to avoid problems!
GitHub Actions, Azure DevOps and other services are better behaved.
The impact can be severe:

18. Parallel random number generation

19. Parallel random number generation
First what not to do – simply drawing random numbers in different
subprocesses will give you the same numbers in each process:

20. Parallel random number generation
Use SeedSequence to obtain independent streams easily:

21. Parallel random number generation
Second option: use the .jumped() method of BitGenerator instances to obtain
independent streams easily:

22. Parallel random number generation

23. Where is NumPy going - technical
Interoperability
Array API standard support
Extensibility
Easier custom dtypes
Performance
SIMD acceleration on:
x86, arm64, PPC, …?
C++
Just dipping our toes in the
water here - so far it was just
Python and C
Platform support
PPC, AIX, s390x,
cross-compiling to embedded
ARM systems, ...
Type annotations
Main namespace annotations
just completed
Note what is not on this list: auto-parallelization

24. Resources to learn more
Scikit-learn:
https://scikit-learn.org/stable/computing/parallelism.html
https://joblib.readthedocs.io/en/latest/parallel.html
SciPy:
http://scipy.github.io/devdocs/dev/toolchain.html#openmp-support
http://scipy.github.io/devdocs/search.html?q=workers
NumPy:
https://numpy.org/doc/stable/reference/random/parallel.html
Relevant paper: Composable Multi-Threading and Multi-Processing for Numeric Libraries

25. Find me at: ralf.gommers@gmail.com, rgommers, ralfgommers
Thank you!