Won 1st place at ACM SIGMOD's research competition.

1. Hydra: Efficient Training for Larger-than-Memory Deep Learning Models
Kabir Nagrecha, Arun Kumar
GPU
GPU
GPU

2. Deep Learning and Scale…A Natural Pairing
GPT-3: 175B parameters
BERT-Large: 345M parameters
Megatron-LM: 1 Trillion Parameters

3. But how do we train them?
No GPU in the world can train a
trillion-parameter model…
But perhaps 1000 GPUs together?
Model Parallelism – Combining
GPUs into a “super-device”

4. Model Parallelism - Overview

5. Model Parallelism…isn’t Parallel

6. “Processors” or Repositories?
DRAM
GPU

7. Model Parallelism VS Task Parallelism
Model
Parallelism
Task/Data
Parallelism
Larger
Models
Faster
Training
AN IMPOSSIBLE
CHOICE!

8. Shard Alternator Parallelism (SHARP)

9. Scheduling Shard Training: Sharded-LRTF
A
B B C A
C
Device 1
Device 2
A B
B
C A
C
Schedule 1
Schedule 2 Device 1
Device 2

10. But is this enough?
Latency
Throughput
Hydra
Model Parallelism
Ideal

11. Shard Prefetching – Double Buffering for ML
GPU
Model A Model B

12. Evaluations
Hydra produces near-optimal speedups!
82% Average GPU Utilization >7.4X Speedups with 8
Devices
Pretraining of 1B parameter BERT architectures
12 models

13. What’s Next?
• Mixed Task-Data parallelism – a new tradeoff space?
• Multi-query Optimization – shared layers?
• Multi-node training for large-scale operations
GPU
GPU
GPU