The document summarizes Parsl, a Python library for pervasive parallel programming. Parsl allows users to naturally express parallelism in Python programs and execute tasks concurrently across different computing platforms while respecting data dependencies. It supports various use cases from small machine learning workloads to extreme-scale simulations involving millions of tasks and thousands of nodes. Parsl provides simple, scalable, and flexible parallel programming while hiding complexity of parallel execution.

1. Parsl:
Pervasive Parallel Programming in Python
18 October 2019
Daniel S. Katz
(d.katz@ieee.org, http://danielskatz.org, @danielskatz)
Assistant Director for Scientific
Software & Applications, NCSA
Research Associate Professor,
CS, ECE, iSchool
Parsl Team: Y. Babuji, A. Woodard,
Z. Li, D. S. Katz, B. Clifford, R. Kumar,
L. Lacinski, R. Chard, J. M. Wozniak,
I. Foster, M. Wilde, K. Chard

2. Supporting composition and parallelism in Python
Software is increasingly assembled rather than written
• High-level language (e.g., Python) to integrate and wrap components from
many sources
Parallel and distributed computing is no longer a niche area
• Increasing data sizes combined with plateauing sequential processing power
• Parallel hardware (e.g., accelerators) and distributed computing systems
Parsl allows for the natural expression of parallelism in such a way that
programs can express opportunities for parallelism that can then be
realized, at execution time, using different execution models on different
parallel platforms

3. Traditional workflow
• A set of tasks and dependencies between them
• Perhaps expressed as data structure, e.g. graph (DAG or cyclic)
• How is this different than a procedural computer program?
• At the level of a set of instructions:
• Dependencies are often explicit, perhaps like a compiled intermediate
representation
• Tasks are much longer (running time O(sec) – O(hr))
• At level of set of functions:
• Tasks as more well-defined (inputs, outputs)
• Tasks are often longer (running time O(sec) – O(hr))
https://danielskatzblog.wordpress.com/2018/01/08/expressing-workflows-as-code-vs-data/
https://danielskatzblog.wordpress.com/2019/02/05/using-workflows-expressed-as-code-and-workflows-expressed-as-data-together/

4. Traditional workflow expression
• Why express a workflow differently than a program?
• Program (script) is a natural way of expressing a workflow
• Easy to understand, easy to change
• Examples: shell scripts, programs in Parsl
• Parsl: functions used to identify components
• Expressing it as data corresponds to the compiled (assembly)
version of the workflow
• Maybe easier to execute?
• Perhaps easier to reproduce?
https://danielskatzblog.wordpress.com/2018/01/08/expressing-workflows-as-code-vs-data/
https://danielskatzblog.wordpress.com/2019/02/05/using-workflows-expressed-as-code-and-workflows-expressed-as-data-together/

5. Workflow as code vs workflow as data
• Goes back to workflow lifecycle concept
• Workflows follow a cycle:
• Experimentation/exploration phase (scientific hacking): workflow is extension of
thought processes of workflow maker
• Productization/ dissemination phase: developer (or someone else) prepares for
wider and repeated use by documentation and optimization then disseminates
• This use by others can simply be use, or it can be further development.
• Different types of users have different needs
• Experts want to be able to do as much as possible
• Other users trade away complex features for simpler user interface
C. Wroe, C. Goble, et al., “Recycling workflows and services through discovery and reuse,”
CCPE v19, pp. 181-194, 2007. doi:10.1002/cpe.1050

6. Representative Parsl Use Cases
Input
Output
DLHub SwiftSeq LSST-DESC

7. DESC image simulation
Catalog
Simulator
Image
Simulator
Atmosphere
, Telescope,
Camera...
LSST Data
Management
Stack
Fake
Observations!SCIENCE!
NASA/JPL-
Caltech/ESO
/R. Hurt
HSC Project /
NAOJ
LSST Project
Credit: Antonio Villarreal

8. ImSim workflow
Bundler
189 sensors
x ~10,000s
of instance
catalogs
Node-sized bundles
(64 tasks each)
JSON description
Catalog 1 189
tasks
x 4000 nodes
Parsl
Extreme
Scale
Executor
x 189 x ~10,000s
256K cores
for 3 days
Catalog 2
Catalog 3
128K cores
for 3.5 days
x 2000 nodes

9. Representative Parsl use cases
DLHub
Machine Learning
Inference
SwiftSeq
DNA Sequence
Analysis
LSST-DESC
Simulated Sky
Survey
O(Tasks) Thousands Thousands Millions
O(Nodes) Tens Hundreds Thousands
O(Duration) Milliseconds-Seconds Hours-Days Hours-Day
Pattern Bag-of-tasks Dataflow Dataflow
Requirements Low latency bounds High throughput Extreme scale

10. Parsl Basics

11. Parsl: Interactive parallel programming in Python
Apps define opportunities for parallelism
• Python apps call Python functions
• Bash apps call external applications
Apps return “futures”: a proxy for a result
that might not yet be available
Apps run concurrently respecting data
dependencies. Natural parallel
programming!
Parsl scripts are independent of where
they run. Write once run anywhere!
pip install parsl
Try parsl via binder at bottom left of http://parsl-project.org

12. Expressing a many task workflow in Parsl
1) Wrap the science applications as Parsl Apps:
@bash_app
def simulate(outputs=[]):
return './simulation_app.exe {outputs[0]}’
@bash_app
def merge(inputs=[], outputs=[]):
i = inputs; o = outputs
return './merge {1} {0}'.format(' '.join(i), o[0])
@python_app
def analyze(inputs=[]):
return analysis_package(inputs)

13. Expressing a many task workflow in Parsl
2) Execute the parallel workflow by calling Apps:
sims = []
for i in range (nsims):
sims.append(simulate(outputs=['sim-%s.txt' % i]))
all = merge(inputs=[i.outputs[0] for i in sims],
outputs=['all.txt'])
result = analyze(inputs=[all.outputs[0]])
simulate simulate simulate
…
merge
analyze
sim-1.txt sim-2.txt sim-N.txt
all.txt

14. Decomposing dynamic parallel execution into a task-dependency
graph
Parsl

15. Parsl scripts are execution provider independent
The same script can be run locally, on grids, clouds,
or supercomputers
Growing support for various schedulers and cloud
vendors
From Parsl docs

16. Separation of code and execution
Choose execution environment
at runtime. Parsl will direct
tasks to the configured
execution environment(s).

17. Authentication and authorization
Authn/z is hard…
• 2FA, X509, GSISSH, etc.
Integration with Globus Auth
to support native app
integration for accessing
Globus (and other) services
Using scoped access tokens,
refresh tokens, delegation
support

18. Transparent (wide area) data management
Implicit data movement to/from
repositories, laptops,
supercomputers
Globus for third-party, high
performance and reliable data
transfer
• Support for site-specific DTNs
HTTP/FTP direct data staging
parsl_file =
File(globus://EP/path/file)
www.globus.org

19. Parsl Performance

20. Different types of scientific workloads
High-throughput workloads
• Protein docking, image processing, materials reconstructions
• Requirements: 1000s of tasks, 100s of nodes, reliability, usability,
monitoring, elasticity, etc.
Extreme-scale workloads
• Cosmology simulations, imaging the arctic, genomics analysis
• Requirements: millions of tasks, 1000s of nodes (100,000s cores)
Interactive and real-time workloads
• Materials science, cosmic ray shower analysis, machine learning inference
• Requirements: 10s of nodes, rapid response, pipelining

21. Different types of execution
High-throughput executor (HTEX)
• Pilot job-based model with multi-threaded manager deployed on workers
• Designed for ease of use, fault-tolerance, etc.
• <2000 nodes (~60K workers), ms tasks, task duration/nodes > 0.01
Extreme-scale executor (EXEX)
• Distributed MPI job manages execution. Manager rank communicates
workload to other worker ranks directly
• Designed for extreme scale execution on supercomputers
• >1000 nodes (>30K workers), ms tasks, >1 m task duration
Low-latency Executor (LLEX)
• Direct socket communication to workers, fixed resource pool, limited features
• 10s nodes, <1M tasks, <1m tasks

22. Short tasks scale to thousands of workers
Strong scaling: 50,000 tasks submitted
with increasing number of workers
* Fireworks only 5,000 tasks
HTEX and EXEX outperform other
Python-based approaches when >256
workers
Other approaches are limited to fewer
than 128 nodes; HTEX and EXEX
continue to scale
0s tasks
1s tasks

23. Executors scale to 2M tasks/256K workers
0s tasks
1s tasks
Weak scaling: 10 tasks per worker
HTEX and EXEX again outperform
other Python-based approaches up
to ~2M tasks
HTEX and EXEX scale to 2K nodes
(~65k workers) and 8K nodes
(~262K workers), respectively, with
>1K tasks/s

24. Parsl executors can provide low latency
• LLEX achieves low
(3.47ms) and
consistent latency
• HTEX (6.87ms) and
EXEX (9.83) are less
consistent

25. Scalability summary
• EXEX scales to over 250,000 workers across 8,000 nodes
• Both EXEX and HTEX deliver ~1200 tasks/s
• LLEX achieves an average latency of 3.47 ms with tight bounds
Framework Max. number of
workers
Max. number of
nodes
Max tasks/sec
Parsl-IPP 2048 64 330
Parsl-HTEX 65 536 2048 1181
Parsl-EXEX 262 144 8192 1176
FireWorks 1024 32 4
Dask distributed 4096 128 2617

26. More Parsl Functionality

27. Interactive supercomputing in Jupyter notebooks

28. Monitoring
and
visualization
Workflow view Task view

29. DOE Distributed Computing & Data Ecosystem
(DCDE)
• A DOE group is identifying best practices and research challenges to create
and operate a DOE/SC wide federated Distributed Computing & Data
Ecosystem (DCDE)
• Future Lab Computing Working Group (FLC-WG)
• Initially working towards a pilot
• Using OAuth, working with Globus
• Test deployment at BNL
• Parsl is part of this effort, via initial work in linking ORNL and BNL
• We’ve added support for an OAuthSSHChannel
• Now being tested on test deployment

30. Multi-site execution
1.Loading Parsl
configuration triggers:
a) Creation of SSH
channels
b) Deployment of an
interchange process
onto login nodes
c) Submission of pilot jobs
that will connect to the
interchange
2.Parsl submits tasks
directly to interchange
3.Parsl uses Globus to
stage data
Interchange Interchange
Parsl

31. Multi-site execution
Too much small code
See demo instead
https://bit.ly/2Wsjlep
(code in https://github.com/Parsl/demo_multifacility)

32. Other functionality provided by Parsl
Globus. Delegated authentication
and wide area data management
Fault tolerance. Support for retries,
checkpointing, and memoization
Containers. Sandboxed execution
environments for workers and tasks
Data management. Automated
staging with HTTP, FTP, and Globus
Multi site. Combining
executors/providers for execution
across different resources
Elasticity. Automated resource
expansion/retraction based on
workload
Monitoring. Workflow and resource
monitoring and visualization
Reproducibility. Capture of workflow
provenance in the task graph
Jupyter integration. Seamless
description and management of
workflows
Resource abstraction. Block-based
model overlaying different providers
and resources

33. Summary

34. Parsl’s parallelism in Python
• Simple: minimal new constructs
• Safe: deterministic parallel programs through immutable
input/output objects, dependency task graph, etc.
• Scalable: efficient execution from laptops to the largest
supercomputers
• Flexible: programs composed from existing components and
then applied to different resources/workloads

35. Open source
https://github.com/Parsl/parsl

36. Questions?
http://parsl-project.org
U . S . D E P A R T M E N T O F
ENERGY