Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze Research Breakthroughs

© 2020 Pittsburgh Supercomputing Center© 2020 Pittsburgh Supercomputing Center
A Journey in High Performance AI:
Evolving Cyberinfrastructure, Democratizing Data,
and Scaling AI to Catalyze Breakthroughs in Research
Nicholas A. Nystrom
Chief Scientist
PSC
nystrom@psc.edu
Paola A. Buitrago
Director, AI & Big Data
PSC
paola@psc.edu
2020 Stanford Conference · April 22, 2020

Outline
• The evolving demands of research
• Evolution of cyberinfrastructure @ PSC
• Exemplars of success
• Bridges-2 Overview
• Summary
2

3
Prior to 2012, AI results closely tracked
Moore’s Law, with compute doubling
every two years. Post-2012, compute
has been doubling every 3.4 months.”
“

4
Network Published Parameters
BERT Large October 11, 2018 340M
PEGASUS Large December 18, 2019 568M
GPT-2 (48 layers) February 2019 1.5B
Megatron-LM August 13, 2019 8.3B
Convolutional Neural Networks (CNNs) Some Recent Transformer-type Networks
Figure from S. Bianco, R. Cadene, L. Celona, and P. Napoletano, Benchmark Analysis of
Representative Deep Neural Network Architectures, IEEE Access, vol. 6, pp. 64270– 64277,
2018. arXiv:1810.00736v2.
Sources of Additional Complexity
Generative Adversarial Networks (GANs)
Domain Adaptation
Reinforcement Learning (RL)

Outline
• Summary
5

Brain Image Library
PSC+CMU+Pitt:
Data+HPDA/HPAI
15PB
Campus Clusters
CMU, Pitt, WVU:
Dedicated + cloud use
of Bridges & data
PghBio Filesystem
Pitt, CMU, UPMC:
BD4BH, CAIIMI
2.2PB
Human BioMolecular
Atlas Program
International Consortium;
Hybrid on-prem + cloud
The Expanding Data + HPDA Ecosystem of Bridges
11/2019 – designed
10/2020* – production
5/2014 – designed
1/2016 – production
7/2018 – designed

The Expanding Data + HPDA Ecosystem of Bridges
11/2019 – designed
10/2020* – production
5/2014 – designed
Brain Image Library
PSC+CMU+Pitt:
Data+HPDA/HPAI
15PB
Campus Clusters
CMU, Pitt, WVU:
Dedicated + cloud use
of Bridges & data
PghBio Filesystem
Pitt, CMU, UPMC:
BD4BH, CAIIMI
2.2PB
Human BioMolecular
Atlas Program
International Consortium;
Hybrid on-prem + cloud
7/2018 – designed

Bridges converges HPC, AI, and Big Data to empower new research communities, bring desktop convenience
to advanced computing, expand remote access, and help researchers to work more intuitively.
• Funded by NSF award #OAC-1445606 ($20.9M), Bridges emphasizes usability, flexibility, and interactivity
• Available at no charge for open research and coursework and by arrangement to industry
• Popular programming languages and applications: Python, Jupyter, R, MATLAB, Java, Spark, Hadoop, …
• 856 compute nodes containing Intel Xeon CPUs and 128GB (800), 3TB (42), and 12TB (4) of RAM each
• 216 NVIDIA Tesla GPUs: 64 K80, 64 P100, (new) 88 V100 configured to balance capability & capacity
• Dedicated nodes for persistent databases, gateways, and distributed services
• The world’s first deployment of the Intel Omni-Path Architecture fabric
• Available at no cost for open research and
courses and by arrangement to industry
• Easier access for CMU and Pitt faculty through the
Pittsburgh Research Computing Initiative
• 29,036 Intel Xeon CPU cores
• 216 NVIDIA GPUs: 64 K80, 64 P100, 88 V100
• 17 PB storage (10 PB persistent, 7.3 PB local)
• 277TB memory (RAM), up to 12 TB per node
• 44M core-hours, 173k GPU-AI-hours,
442k GPU-hours, and 343k TB-hours allocated
quarterly
• Serving 2,110 projects and >20,000 users at
>800 institutions, spanning 121 fields of study
• Bridges-AI: NVIDIA DGX-2 Enterprise AI
system + 9 HPE 8-Volta Apollo 6500 Gen10
servers (aggregate 88 V100 GPUs)
8

Conceptual Architecture
9
Intel Omni-Path
Architecture
fabric
Management
nodes
Parallel File
System
Web Server
nodes
Database
nodes
Data Transfer
nodes
Login
nodes
Users,
XSEDE,
campuses,
instruments
ESM Nodes
12TB RAM
4 nodes
LSM Nodes
3TB RAM
42 nodes
RSM Nodes
128GB RAM
800 nodes,
48 with GPUs
Bridges-AI
NVIDIA DGX-2
(16 V100 GPUs)
9x HPE A6500
(9x 8 V100 GPUs)
Introduced in
Operations Year 3

32 HPE Apollo 2000 nodes, each with
2 NVIDIA Tesla P100 GPUs
748 HPE Apollo 2000 (128 GB)
compute nodes
20 “leaf” Intel® OPA edge switches
6 “core” Intel® OPA edge switches:
fully interconnected, 2 links per switch
42 HPE ProLiant DL580 (3 TB)
compute nodes
20 Storage Building Blocks,
implementing the parallel Pylon storage
system (10 PB usable)
4 HPE Integrity Superdome X
(12 TB) compute nodes …
12 HPE ProLiant DL380
database nodes
6 HPE ProLiant DL360 web
server nodes
4 MDS nodes
2 front-end nodes
2 boot nodes
8 management nodes
Intel® OPA cables
… each with 2 gateway nodes
Purpose-built Intel® Omni-Path
Architecture topology for
data-intensive HPC
16 HPE Apollo 2000 (128 GB) GPU nodes with
2 NVIDIA Tesla K80 GPUs each
Simulation,
including AI-enabled
ML, inferencing, DL development,
Spark, HPC AI (Libratus, Pluribus)
Distributed AI/ML, Spark, etc.
Representative
uses for AI
Robust paths to
parallel storage
Project &
community
datasets
Large-memory
Python, R,
MATLAB,
and Java
User interfaces for
AIaaS, BDaaS
https://psc.edu/bvt
Bridges Virtual Tour:
Maximum-Scale Deep Learning
NVIDIA DGX-2 and 9
HPE Apollo 6500
Gen10 nodes:
88 NVIDIA Tesla V100
GPUs
Bridges-AI
Simulation & AI
10

Elements not found in traditional supercomputers
Bridges Makes Advanced Computing Accessible
Make HPC accessible to all research communities
Converge HPC, AI, and Big Data
Support the widest range of science with an extremely rich
computing environment
• 3 tiers of memory: 12 TB, 3 TB, and 128 GB
• Powerful, flexible CPUs and GPUs
• Familiar, easy-to-use user environment:
– Interactivity
– Popular languages and frameworks:
Python, Anaconda, R, MATLAB, Java, Spark, Hadoop
– AI frameworks: TensorFlow, Caffe2, PyTorch, etc.
– Containers and virtual machines (VMs)
– Databases
– Gateways and distributed (web) services
– Large collection of applications and libraries
11
J. Saltz and R. Gupta, Stony Brook Medicine
A. Khan and E. A. Huerta, U. of Illinois

Community Datasets
12
• Hosting mature corpus of data and data tools
for an open science community
– Accessible by multiple users, multiple groups.
– Provision of reusable data management tools
– Facilitate collaboration
– Offload data management
• Interoperable with HPC capabilities
– High speed data transfer
– High performance compute capabilities
• Support copies, maintenance, guarantee
integrity
• Data resource not subject to project
limitations
C4

Bridges-AI
• Bridges-AI is integrated with Bridges and allocated through XSEDE as resource “Bridges GPU-AI”, analogous to Bridges GPU, RM, LM, and Pylon
• Bridges-AI adds 11 Pf/s of mixed-precision tensor, 1.24 Pf/s of fp32, and 0.62 Pf/s of fp64
• The $1.786M supplement includes additional staffing to support solutions and scaling
13
NVIDIA DGX-2
HPE Apollo 6500 Gen10
Server
1 NVIDIA DGX-2 Enterprise AI System for the most demanding AI challenges
16 NVIDIA Tesla V100-32GB SXM2 GPUs: 81,920 CUDA cores and 10,240 tensor cores
Performance: 2Pf/s mixed-precision tensor, 251Tf/s 32b, 125Tf/s 64b
Memory: 512GB HBM2, 14.4TB/s aggregate memory bandwidth
2×Intel Xeon Platinum 8168 CPUs (24c, 2.7–3.7 GHz) and 1.5TB of DDR4-2666 RAM
8×3.84 TB NVMe SSDs (aggregate ~30 TB) for user data + 2×960 TB NVMe SSDs to host the OS
8×Mellanox ConnectX adapters for EDR InfiniBand & 100 Gb/s Ethernet
NVSwitch: 50 GB/s/port, 900 GB/s/chip bidirectional bandwidth, 2.4TB/s system bisection bandwidth
9 HPE Apollo 6500 Gen10 servers to balance great AI capability and capacity
8 NVIDIA Tesla V100-16GB SXM2 GPUs: 40,960 CUDA cores and 5,120 tensor cores
Performance: 1Pf/s mixed-precision tensor, 125Tf/s 32b, 64Tf/s 64b
Memory: 128GB HBM2, 7.2TB/s aggregate memory bandwidth
2×Intel Xeon Gold 6148 CPUs (20c, 2.4–3.7 GHz) and 192GB of DDR4-2666 RAM
4×2TB NVMe SSDs for user and system data
NVLink 2.0 hybrid cube-mesh topology connecting the 8 V100 GPUs

Deep Learning and ML Frameworks on Bridges-AI
14

Outline
– From Fundamental Research to National Defense
– Deep Dive: AI Quantification of Tumor-Infiltrating Lymphocytes
– Additional examples
• Summary
15

AI @ PSC: The Evolution of No-Limit Texas Hold’em Poker
16
Blacklight
TartanianN
Claudico
Prof. Tuomas
Sandholm
Noam Brown
2010
2015
• Imperfect information,
representative of real-
world challenges
• Heads-Up: 10161 situations

2019
August 30, 2019
Surpassing Human Expertise
17
2017
Made possible with:
19M core-hours of computing
2.6PB knowledge base

International “Outstanding Achievement in AI”
18
2019
Prof. Tuomas
Sandholm
Noam Brown 2019 Marvin Minsky
Medal for Outstanding
Achievements in AI
“Poker is an important challenge for AI because any poker
player has to deal with incomplete information. Incomplete
information makes the computational challenge orders of
magnitude harder. Libratus used fundamentally new
techniques for dealing with incomplete information, which
have exciting potential applications far beyond games.”
— Professor Michael Wooldridge,
— Chair of the IJCAI Awards Committee

Prof. Sandholm launched two startups on
Libratus’ algorithms:
Strategic Machine Inc. and Strategy Robot.
Strategy Robot received a 2-year contract
for up to $10M from the Pentagon’s
Defense Innovation Unit.
https://www.wired.com/story/poker-playing-robot-goes-to-pentagon/
From Poker to Real-World Applications
19

20
https://vis.sciencemag.org/breakthrough2019/finalists/#intelligence · December 19, 2019

Outline
• Summary
21

Toward a Deeper Understanding of Cancer
Spatial maps of populations of tumor and immune cells provide a snapshot
of the interaction between a tumor and the patient’s immune system.
Dr. Joel Saltz Dr. Raj Gupta
DEEP DIVE: AI QUANTIFICATION OF TUMOR-INFILTRATING LYMPHOCYTES
22

Pathomics Using Whole-Slide Images (WSIs)
Whole-slide images (WSIs) are
generated from glass slides that are
used by pathologists to detect and
classify cancer and other diseases to
guide patient care and treatment.
WSIs are large: up to 100,000 ×
100,000 pixels, and 1–4 GB each.
Pathomics Data in Digital Pathology
generated by extracting and
quantifying image-based phenotypic
features of tissues, cells, nuclei, and
other structures and objects in whole-
slide images (WSIs) of tissue samples
used for diagnostic testing of cancer. H. Le et al., arXiv:1905.10841v2
23

Quantifying TILS
Quantify:
• The quantity and pattern of immune infiltrate.
• Division of immune infiltrate between different
compartments.
• Does it surround the tumor region? Is it present in
tumor, invasive margin?
• https://www.tilsinbreastcancer.org/
24

Tumor Infiltrating Lymphocytes (TILs)
Tumor infiltrating lymphocytes (TILs) are
particularly important in predicting
response to cancer immune therapies.
Spatial maps of tumors and TILs provide a
snapshot of the interaction and are clinically
valuable, particularly for immunotherapy.
– High densities of TILs correlate with favorable
clinical outcomes including longer disease-free
survival or improved overall survival in multiple
cancer types.
– The spatial context and the nature of cellular
heterogeneity are important in cancer
prognosis.
A: A multi-gigapixel whole slide breast cancer image
obtained from the public Cancer Genome Atlas
collection.
B, C: Predicted probability maps corresponding to tumor and
infiltrating lymphocytes.
D: A map depicting the spatial distribution of lymphocytes
in and near tumor regions.
H. Le et al., arXiv:1905.10841v2
25

Deep Learning for Cancer and Lymphocyte Detection
• Independent networks were trained for cancer detection (C) and
lymphocyte classification (L)
• Evaluated VGG16, ResNet-34, and Inception-v4
– ResNet-34 and Inception-v4: dimension of the output layer from
1,000 classes to 2 classes for binary classification
– VGG16: size of the intermediate features of the classification layer
reduced from 4,096 to 1,024 and kept only kept the first
4 classification layers
• CNNs pretrained on ImageNet and implemented in PyTorch
26
Le, Gupta, Hou, Abousamra, Fassler, Kurc, Samaras, Batiste, Zhao, Van Dyke, Sharma, Bremer, Almeida, and Joel Saltz,
Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor Infiltrating Lymphocytes in Invasive Breast Cancer,
arXiv:1905.10841v2.

Cancer Detection Network
• Cancer training dataset: 350×350-pixel patches at 400×
magnification, resized to 224×224 for VGG16 and ResNet-34, and
299×299 for Inception-v4
• → C-VGG16, C-Resnet34 and C-Incepv4
From Le et al., arXiv:1905.10841v2
27

Lymphocyte Detection Network
• Lymphocyte training dataset: 100×100-pixel patches at 200×
magnification, resized to 200×200
• → L-VGG16, L-Resnet34 and L-Incepv4
(2018)
From Le et al., arXiv:1905.10841v2
28

AI Detection of Complex Heterogeneous Forms of Breast Cancer
29

AI-Driven Analyses of Multi-Gigapixel WSIs of Breast Cancer
Tissue that is diffusely infiltrated by TILs
TILs that are primarily outside and adjacent
to the tumor at the invasive leading edge
but unable to infiltrate the tumor
Limited scattered TILs in the tumor
Scant TILs in a focal area of the tumor
H&E stained WSIs of tissue sections
Probability of TILs mapped to the tissue
Combined tumor-TIL map
30

Tumor-Immune Interaction in Multiple Disease Sites: Pancreatic Cancer
31

How Bridges-AI Helps
• Bridges-AI is playing a crucial role in the development of methods to extract
image-based biomarkers to use pathomics to steer patient treatment.
• Computationally intensive AI algorithms are indispensable to this work in creating
highly detailed, quantitative, and nuanced characterization of the dynamic and
complex immune responses within the tumor microenvironment.
• Algorithm development, evaluation, and validation are all critical.
• Both training and inference (predictions) are highly computationally intensive.
1. Le, Gupta, Hou, Abousamra, Fassler, Kurc, Samaras, Batiste, Zhao, Van Dyke, Sharma, Bremer, Almeida, and Joel Saltz, Utilizing Automated Breast
Cancer Detection to Identify Spatial Distributions of Tumor Infiltrating Lymphocytes in Invasive Breast Cancer, https://arxiv.org/abs/1905.10841 (2019).
2. Buitrago, P. A., Nystrom, N. A., Gupta, R., & Saltz, J. (2020). Delivering Scalable Deep Learning to Research with Bridges-AI. In High Performance
Computing: 6th Latin American Conference, CARLA 2019, Turrialba, Costa Rica, September 25–27, 2019 (Vol. 1087, pp. 200–214).
Gewerbestrasse, Switzerland: Springer Nature. doi: 10.1007/978-3-030-41005-6_14.
32

Outline
• Summary
33

Deep Learning for Large Volume Cancer Imaging Analysis
Shandong Wu, University of Pittsburgh
Bridges GPU and AI enables the Intelligent Computing
for Clinical Imaging (ICCI) lab to perform deep
learning-based breast imaging analysis for:
– Breast cancer risk prediction (including risk of
developing breast cancer in screening populations and
risk of recurrence or new cancer development for
patients with a breast cancer diagnosis);
– Breast tumor proliferation rate estimation;
– Reducing unnecessary call-backs for screening;
– Pre-reading of digital breast tomosynthesis images to
aid radiologists for more efficient and accurate breast
cancer diagnosis;
– Enhancing deep learning interpretability for different
breast image classification;
– Incorporating physician expert knowledge into deep
learning modeling.
34
Top: Deep learning modeling pipeline.
Bottom-Left: Model prediction performance.
Bottom-right: Visualization of deep learning-identified imaging features in
relation to the risk prediction.
AI on XSEDE Systems Promises Early Prediction of Breast Cancer,
insideHPC, September 9, 2019. https://insidehpc.com/2019/09/ai-
on-xsede-systems-promises-early-prediction-of-breast-cancer/
“Bridges has enabled us to process large volume, both 2D and
3D, and different modalities of breast images including digital
mammography, breast magnetic resonance imaging (MRI),
and digital breast tomosynthesis.” —Shandong Wu, UPitt

Scanning for Zebrafish Neurons using a 3D CNN
Joel Welling, Liyunshu Qian*, Richard Zhao*, Minyue Fan*, and Brian Leonard*, PSC
Bridges-AI is being used to scan a representative fraction
of a larval zebrafish electron microscopy (EM) dataset for
neurons using a novel deep neural network (DNN).
– Fish neurons within the dataset have been identified using
a computer-assisted manual method. These known
neurons were used to build a training set for the DNN, but
only a tiny fraction of the fish can be included in the
training set. The goals are to test the DNN in a more
representative sample of the zebrafish volume and to add
well-chosen cases to the training set.
– The DNN was developed by PI Welling, with assistance from
several PSC interns. The DNN’s unique feature is that the
calculation is being done in a spherically symmetric way,
rather than slice-wise or in terms of a rectangular array of
data voxels. It is a double convolutional neural network (CNN),
whose first half performs spherically symmetric filtering on the
input and then feeds into a conventional CNN.
– Results so far are promising. They are expanding their
procedure to the edge of the available zebrafish data.
35
The grayscale surfaces show cuts through the 3D zebrafish EM data.
Human-labeled neuron locations are blue; the orange and green
surfaces form a double wall around regions identified by the DNN.
The green inner wall appears where the volume boundary cuts the
surface. Human-labeled neurons generally fall within the boundary.

Predictive Modeling for Climate Resilient Phenotypes in Mega-Size Plant Genomes
Charles Chen, Translational Genomics Laboratory, Oklahoma State University
36
Genomic prediction that estimates
phenotypic variation based on whole-
genome polymorphism:
– Factors affecting predictability are
comprehensiveness and the learning capacity
of prediction algorithms.
– Convolutional deep learning models for
bacterial antimicrobial-resistance phenotypes
require high-dimensional whole bacterial
genomes (median number of variables: 2.84
million base-pairs) to produce intermediate
tensors and eventually classify the antibiotic
resistance of a given strain.
“With the new Volta V100 GPUs (32GB Graphics RAM), we are
able to fit tensors with millions of dimensions into the GPU-
RAM and have it easily train a model in under a day using
TensorFlow.” —Charles Chen, Oklahoma State University
The predictability of antibiotic resistance can be reached at the
comparable level of the traditional MIC (minimum inhibitory
concentration) methods, suggesting a possible paradigm shift
for antibiotic resistance detection to a predictive analytics.

Deep Learning for Large-Scale Astronomical Surveys
Asad Khan, E. A. Huerta, et al., University of Illinois
Khan, Huerta, et al. demonstrate that knowledge from deep
learning algorithms, pre-trained with real-object images, can
be transferred to classify galaxies that overlap Sloan Digital
Sky Survey (SDSS) and Dark Energy Survey (DES) images,
achieving state-of-the-art accuracy around 99.6%.
– They used their neural network classifier to label over
10,000 unlabeled DES galaxies which do not overlap
previous surveys.
– They also used their neural network model as a feature
extractor for unsupervised clustering, finding that
unlabeled DES images can be grouped in two distinct
galaxy classes based on their morphology, providing a
heuristic check that the learning is successfully
transferred to the classification of unlabeled DES images.
– These newly labeled datasets can be combined with
unsupervised recursive training to create large-scale DES
galaxy catalogs in preparation for the Large Synoptic
Survey Telescope era.
37
To expose the neural network to a variety of potential scenarios
for classification, we augment original galaxy images with
random vertical and horizontal flips, random rotations, height
and width shifts and zooms.
A. Khan, E. A. Huerta, S. Wang, R. Gruendl, E. Jennings, and H. Zheng, “Deep learning at scale for the
construction of galaxy catalogs in the Dark Energy Survey,” Phys. Lett. B, vol. 795, pp. 248–258, 2019.
http://www.sciencedirect.com/science/article/pii/S0370269319303879

38
Human Biomolecular Atlas Program (HuBMAP)
The HuBMAP Infrastructure and Engagement Component is supported by NIH project 3OT2OD026675.
“The Human BioMolecular Atlas Program (HuBMAP) aims
to facilitate research on single cells within tissues by
supporting data generation and technology development
to explore the relationship between cellular organization
and function, as well as variability in normal tissue
organization at the level of individual cells.” —NIH
HuBMAP Initial Data Release planned for summer 2020

• Research Goal: Facilitate research on single cells within tissues by supporting data generation and technology
development to explore the relationship between cellular organization and function, as well as variability in
normal tissue organization at the level of individual cells
• Approach: Flexible hybrid cloud infrastructure: on-prem storage and HPC/HPAI for cost-effectiveness and
capability + public cloud for high availability core services, interoperation, and burst
• Data Generation Community:
Currently 5 Tissue Mapping
Centers (TMCs) + 4 Rapid
Technology Implementation
Centers (RTIs); more expected
• Overall Complexity: High
– Currently 12 distinct assays
and 8 tissue types
– Over time: ~30 assay types,
more tissue types
• Data Volume: TBD
• Other Issues Impacting
Complexity: Building HuBMAP
Portal coupling tools, mapping,
and data: democratization
HuBMAP Community
Michael Snyder,
Stanford University
Small bowel and colon
Long Cai,
Caltech
Endothelium
Mark Atkinson,
University Of Florida
Lymphatic system
Jeffrey Spraggins,
Vanderbilt University
Kidney, pancreas, bone
Kun Zhang,
UCSD
Kidney, urinary tract, lung
Tissue Mapping
Centers (TMCs)
Pehr Harbury,
Stanford University
Next-generation genomic
imaging
Long Cai,
Caltech
In situ transcriptome
profiling in single cells
Julia Laskin,
Purdue University
Sub-cellular resolution
mass spectrometry
Peng Yin,
Harvard University
High-throughput, highly
multiplexed in situ
proteomic imaging
Transformative
Technology
Development (TTDs)
Nick Nystrom, PSC
Jonathan Silverstein, Pitt
Flexible Hybrid Cloud
Infrastructure for Seamless
Management of HuBMAP
Resources
Infrastructure and
Engagement
Component (IEC)
Tools Components
(TCs)
Ziv Bar-Joseph,
Carnegie Mellon
University
Nils Gehlenborg,
Harvard University
Mapping
Components (MCs)
Katy Börner,
Indiana University
Bloomington
Rahul Satija,
New York Genome
Center
Rapid Technology
Implementation
(RTIs)
Yousef Al-Kofahi,
GE
Multi-Scale 3-D Image Analytics
for High Dimensional Spatial
Mapping of Normal Tissues
Mike Angelo, Sean
Bendall, Stanford
A robust platform for
multiplexed, subcellular
proteomic imaging in human
tissue
Neil Kelleher,
Northwestern Univ.
Renewable and specific affinity
reagents for mapping
proteoforms in human tissues
Evan Macosko,
Broad Institute
Implementation of Slide-seq for
high-resolution, whole-
transcriptome human tissue
maps
18 Components, 19 Lead Institutions

HuBMAP Assets
Institution Current Assays Future Assays
Florida
CODEX, 10X (scRNAseq),
IMC
+Lightsheet, Immuno-phenotyping
(FACS)
Stanford CODEX
+TMT-LC-MS, HILIC & RPMC, SCIEX,
snRNAseq, Bulk RNA-seq, Bulk ATAC-seq,
scATAC-seq, Bulk WGS, MIBI
UCSD SNARE-Seq2, SPLiT-seq +DARTFISH, Bulk RNA-seq
Cal Tech
sci-RNA-seq, sci-ATAC-
seq, seqFISH
+SABERFISH
Vanderbilt/
Purdue
LC-MS, MALDI-IMS,
Autoflorescence &
Stained microscopy
+nanoDESI IMS, nanoPOTS, CODEX
GE Research n/a CellDive
Broad n/a Slide-seq
Total 12 distinct 29 distinct
Institution Donors Organs Tissue Samples Current Assays
Florida 4
44: Lymph,
Spleen,
Thymus
168
CODEX, 10X
(scRNAseq), IMC
Stanford 1
4: S&L
Bowel
53 CODEX
UCSD 11 10: Kidney 48
SNARE-Seq2, SPLiT-
seq
Cal Tech 2
3: Heart,
Spleen,
Colon
14
sci-RNA-seq, sci-
ATAC-seq, seqFISH
Vanderbilt/
Purdue
25 25: Kidney 1816
LC-MS, MALDI-IMS,
Autoflorescence &
Stained microscopy
Total 45 86 2099 12 distinct
• File Types: 3 modalities (genomic, proteomic, imaging), ~30 assay types
• Mapping: Registration User Interface (a) (Katy Börner, IU),
Lung CCF (Rahul Satija, NYGC)
• Tools: Vitessce (b) (Nils Gehlenborg, HMS), Pipelines (Ziv Bar-Joseph, CMU)
• HuBMAP Portal: Integration of data, mapping, and tools; pipeline orchestration
across on-prem (PSC HPC+AI+data Bridges & Bridges-2) and cloud infrastructure;
federated identity management and data security
a
b
CURRENTSNAPSHOT

Outline
• Summary
41

Driving Rapidly Evolving Science and Engineering
Ease of use,
familiar software,
interactivity,
productivity
HPC + AI + data,
workflows,
heterogeneous,
cloud
HPAI and
AI-enhanced
simulation and
modeling
Community Data,
Big Data as a
Service
HPC & HTC
Simulation and
Modeling

An Ecosystem for Rapidly Evolving, Data-Intensive Science & Engineering
2,100 projects
20,000 users
800 institutions
121 fields of study
130 education allocations
Award OAC-1445606
Pioneered HPC+AI+Big Data
Bridges-AI expansion
Intel OPA first installation
Has become an ecosystem
Award OAC-1928147
Full-system HPAI
Fast flash array
HDR-200 InfiniBand
Tiered data management
Cloud interoperability
Coming Summer 2020
43

Bridges-2 builds on the flexible architecture of Bridges and introduces innovations to scale AI and HPDA.
• Bridges core concepts:
– Converged HPC + AI + Data
– Custom fat tree Clos topology optimized for
data-centric HPC, AI, and HPDA
– Heterogeneous node types for different
aspects of workflows
– CPUs and AI-targeted GPUs
– 3 tiers of RAM (256GB, 512GB, 4TB) per node
• Innovations beyond Bridges:
– AMD EPYC 7742 CPUs: 64-core, 2.25–3.4GHz
– AI scaling to 192 V100 SXM2 32GB HBM2 GPUs
– 100TB flash array accelerates deep learning training,
genomics, and other applications
– Mellanox HDR-200 InfiniBand doubles bandwidth &
supports RDMA, virtualization, and new optimizations
– Cray ClusterStor E1000 Storage System
– HPE DMF single namespace across disk and tape
for data security and expandable archiving
Bridges-2: Scalable Converged Computing, Data, and Analytics for
Rapidly Evolving Science and Engineering Research
Bridges-2 will provide transformative capability for rapidly-evolving, computationally-intensive and data-intensive
research, supporting new and existing opportunities for collaboration and convergence research.
It will support traditional and nontraditional research communities and applications, integrate new technologies for
converged, scalable HPC, AI, and data; prioritize researcher productivity and ease of use; and provide an
extensible architecture for interoperation with complementary data-intensive projects, campuses, and clouds.
Bridges-2 is supported by NSF Award OAC-1928147
Version: September 16, 2019
Bridges-2 will broadly enable research, including:
• Understanding learning and memory (a)
• Characterizing bacterial DNA with ML
• ML-driven design of elastic smart materials
• Mapping the brain at full synaptic detail
• Simulation, data analysis, and machine
learning for LHC CMS
• Metagenomics & Metadesign of Subways
and Urban Biomes (MetaSUB consortium) (b)
• Protein structure with AI-enabled cryo-EM
• Accurate forecasting of severe storms and
tornadoes
• Multi-messenger astrophysics with
IceCube (c)
• Assembling metagenomes; application to
industrial biomolecules
Paola Buitrago, Co-PI &
AD for AI & Data Science
Robin Scibek, Co-PI &
AD for Broader Impacts
Nick Nystrom, PI/PD &
AD for Acquisition & Ops
Sergiu Sanielevici, Co-PI &
AD for User Support
Broader Impacts
Innovating for
the Future
Improving Our
Society
Reaching
Beyond Borders
Engaging a
Wider Audience
Building Stem
Talent
a. Long-term potentiation
hemisphere, 9.60 μm.
b. Sites for discovery of new
urban biology.
c. Event view of the real-time
alert IceCube-170922A.
10/19
Acquisition Operations → …
10/20 10/21
8/20–: Early User Program
7/20: Installation and testing
Timeline
*
*DMF
*new elements
*
*
*
*
*
*
100TB

Bridges-2 GPU Infrastructure
12 × HPE Apollo 6500 Gen10 Server
Each: 1Pf/s tensor, 125 Tf/s fp32, 64 Tf/s fp64
... ...
...
12 Mellanox Quantum Spine Switches
12 × HPE Apollo 6500 Gen10 Server
Each: 1Pf/s tensor, 125 Tf/s fp32, 64 Tf/s fp64
2 × HDR-200 IB per server

AI and Data @ Bridges-2
Bridges-2 HPE DMF
Multimodal Data Non-traditional
users
Community
datasets
BigData aaS

Bridges-2

Bridges-2
BIL

Bridges-2 Application Areas: Examples
F. E. Garrett-Bakelman et al., The NASA
Twins Study: A Multidimensional Analysis of
a Year-Long Human Spaceflight, Science,
2019. DOI: 10.3847/1538-4357/aac329.
Gene Expression
S. Abousamra et al., Learning from Thresholds: Fully Automated
Classification of Tumor Infiltrating Lymphocytes for Multiple Cancer
Types, 2019. ArXiv 1907.03960v1.
Advancing Digital Pathology with AI
Mapping the Human Body
at Cellular Resolution
M. Snyder et al., Mapping the Human Body at Cellular
Resolution -- The NIH Common Fund Human
BioMolecular Atlas Program, Nature, to appear.
Developing Smart Cities
J. Argota and D. Cardoso Llach, Using Computation to
Understand How Pedestrians Use Market Square, 2018.
https://soa.cmu.edu/news-archive/2018/9/5/using-
computation-to-understand-how-pedestrians-use-
market-square.
Workflows for CMS @ HL-LHC
Event display of heavy-ion collision registered at the CMS
detector on Nov. 8, 2018 (image: Thomas McCauley).
From https://cms.cern/news/2018-heavy-ion-collision-run-has-
started.
X. Zheng et al., Detecting Comma-shaped Clouds for Severe Weather
Forecasting using Shape and Motion, IEEE Transactions on
Geosciences and Remote Sensing, 2019.
DOI: 10.1109/TGRS.2018.2887206.
Improving Severe Storm Prediction Understanding Immunity
C. M. Quinn et al., Dynamic regulation of HIV-1 capsid
interaction with the restriction factor TRIM5α identified
by magic-angle spinning NMR and molecular dynamics
simulations, PNAS, 2018.
DOI: 10.1073/pnas.1800796115.
M. Amsler et al., Cubine, A Quasi
Two-Dimensional Copper-
Bismuth Nanosheet, Chem.
Mater. 2017. DOI:
10.1021/acs.chemmater.7b03997
New Materials

Outline
• Summary
51

Summary
• AI is an integral and expanding component of research across a great number of fields. By
combining the strengths of HPC and AI, PSC provides the national research community and
their collaborators uniquely scalable resources at no cost for open research and on a cost-
recovery basis for industry.
• Bridges pioneered AI, HPC, and Big Data, and through its heterogeneous, flexible architecture,
created a large community of nontraditional users and ecosystem of interoperating
cyberinfrastructure. Bridges-AI adds the greatest capability for deep learning and increased
aggregate capacity across NSF cyberinfrastructure by 283%.
• Bridges-2 will greatly extend those proven concepts with full-system HPAI, a new all-flash
component for fast data, tiered data management, powerful new CPUs, and enhanced cloud
interoperability.
– The Bridges-2 Early User Program is planned for August 2020. Contact me if you’d like to participate!
– Production is planned to begin in October 2020.
• “Startup” and “Education” proposals are accepted at any time.
• Larger “Research” proposals are reviewed quarterly. The next submission window for research
proposals is June 15 – July 15.
52

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze Research Breakthroughs

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze Research Breakthroughs

Ähnlich wie Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze Research Breakthroughs (20)

Mehr von inside-BigData.com

Mehr von inside-BigData.com (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze Research Breakthroughs