PyTorch, the popular open-source ML framework, has continued to evolve rapidly since the introduction of PyTorch 1.0, which brought an accelerated workflow from research to production.

1. SCALING UP RESEARCH
TO PRODUCTION WITH
PYTORCH & MLFLOW
JOE SPISAK
PYTORCH PRODUCT LEAD

2. AGENDA 01
WHAT IS PYTORCH?
02
THE AMAZING COMMUNITY
03
HOW FACEBOOK USES PYTORCH
04
FOUR CHALLENGES TO SCALING AI
05
GET STARTED

4. SIMPLICITY
OVER
COMPLEXITY
HARDWARE
ACCELERATION
DISTRIBUTED
TRAINING
DYNAMIC
NEURAL
NETWORKS
EAGER &
GRAPH-BASED
EXECUTION
WHAT IS PYTORCH?

5. P Y T O R C H
R E S E A R C H
P R O T O T Y P I N G
P R O D U C T I O N
D E P L O Y M E N T
+

6. to rc h . o p tim
O P T I M I Z E R S
to rc h . n n
N E U R A L N E T W O R K S
to rc h . d ata
D A T A S E T S &
D A T A L O A D E R S
to rc h .visio n
M O D E L S , D A T A
to rc h . au to g rad
A U T O
D I F F E R E N T I A T I O N
to rc h . j it
T O R C H S C R I P T
D E P L O Y M E N T

7. C O M M U N I T Y

10. ~1,400C O N T R I B U T O R S
50%+Y O Y G R O W T H
29K+P Y T O R C H F O R U M U S E R S

11. G R O W I N G U S A G E I N O P E N S O U R C E
Source: https://paperswithcode.com/trends

12. P Y T O R C H | A T F A C E B O O K

13. F A C E B O O K P R O D U C T S L E V E R A G I N G M L
Search
Translation
Ads
News Feed
Face Tagging

15. S C A L I N G A I
F O U R C H A L L E N G E S

16. C H A L L E N G E # 1
D E V E L O P E R E X P E R I E N C E

17. A RICH ECOSYSTEM OF TOOLS AND LIBRARIES
FULL FREEDOM TO ITERATE AND EXPLORE THE DESIGN SPACE
CLEAN AND INTUITIVE APIS
DEVELOPER EXPERIENCE

18. DEVELOPER EFFICIENCY
ENABLING A HIGH VELOCITY OF MODEL
ITERATION AND INNOVATION
P Y T H O N C + +
import torch
class Net(torch.nn.Module):
def __init__(self):
self.fc1 = torch.nn.Linear(8, 64)
self.fc2 = torch.nn.Linear(64, 1)
def forward(self, x):
x = torch.relu(self.fc1.forward(x))
x = torch.dropout(x, p=0.5)
x = torch.sigmoid(self.fc2.forward(x))
return x
#include <torch/torch.h>
struct Net : torch::nn::Module {
Net() : fc1(8, 64), fc2(64, 1) {
register_module("fc1", fc1);
register_module("fc2", fc2);
}
torch::Tensor forward(torch::Tensor x) {
x = torch::relu(fc1->forward(x));
x = torch::dropout(x, /*p=*/0.5);
x = torch::sigmoid(fc2->forward(x));
return x;
}
torch::nn::Linear fc1, fc2;
};

19. BUILDING FOR
SCALE
HIGH PERFORMANCE EXECUTION FOR
MODEL TRAINING AND INFERENCE
Net net;
auto data_loader = torch::data::data_loader(
torch::data::datasets::MNIST("./data"));
torch::optim::SGD optimizer(net.parameters());
for (size_t epoch = 1; epoch <= 10; ++epoch) {
for (auto batch : data_loader) {
optimizer.zero_grad();
auto prediction = net.forward(batch.data);
auto loss = torch::nll_loss(prediction,
batch.label);
loss.backward();
optimizer.step();
}
if (epoch % 2 == 0) {
torch::save(net, "net.pt");
}
}
P Y T H O N C + +
net = Net()
data_loader = torch.utils.data.DataLoader(
torchvision.datasets.MNIST('./data'))
optimizer = torch.optim.SGD(net.parameters())
for epoch in range(1, 11):
for data, target in data_loader:
optimizer.zero_grad()
prediction = net.forward(data)
loss = F.nll_loss(prediction, target)
loss.backward()
optimizer.step()
if epoch % 2 == 0:
torch.save(net, "net.pt")

20. T O O L S &
L I B R A R I E S

21. C H A L L E N G E # 2
M O D E L S I Z E A N D C O M P U T E
N E E D S

22. PARAMETER COUNTS CONTINUE TO GROW

23. PRUNING
State-of-the-art deep learning techniques rely on over-
parametrized models that are hard to deploy.
Identify optimal techniques to compress models by reducing
the number of parameters without sacrificing accuracy.
new_model = LeNet()
for name, module in new_model.named_modules():
# prune 20% of connections in all 2D-conv layers
if isinstance(module, torch.nn.Conv2d):
prune.l1_unstructured(module, name='weight', amount=0.2)
# prune 40% of connections in all linear layers
elif isinstance(module, torch.nn.Linear):
prune.l1_unstructured(module, name='weight', amount=0.4)
# to verify that all masks exist
print(dict(new_model.named_buffers()).keys())

24. QUANTIZATION
model = ResNet50()
model.load_state_dict(torch.load("model.pt"))
qmodel = quantization.prepare(
model, {"": quantization.default_qconfig})
qmodel.eval()
for batch, target in data_loader:
model(batch)
qmodel = quantization.convert(qmodel)
EXPERIMENTAL
Neural networks inference is expensive
IoT and mobile devices have limited resources
Quantizing models enables efficient inference at scale

25. fp32 accuracy int8 accuracy change Technique CPU inference speed up
ResNet50 76.1
Top-1, Imagenet
-0.2
75.9
Post Training
2x
214ms ➙102ms,
Intel Skylake-DE
MobileNetV2 71.9
Top-1, Imagenet
-0.3
71.6
Quantization-Aware
Training
4x
75ms ➙18ms
OnePlus 5, Snapdragon 835
Translate / FairSeq 32.78
BLEU, IWSLT 2014 de-en
0.0
32.78
Dynamic
(weights only)
4x
for encoder
Intel Skylake-SE
These models and more available on TorchHub - https://pytorch.org/hub/
EXAMPLE MODELS | W/ QUANTIZATION

26. C H A L L E N G E # 3
T R A I N I N G M O D E L S A T S C A L E

27. T H E P A T H T O B E T T E R E X P E R I M E N T A T I O N
IDEAS ARE GETTING
MORE COMPLEX
1
MODELS ARE GETTING
MORE COMPLEX
2
Complex Models
GEN 4
SPARSE
NN
GEN 3
NNS: DEEP
LEARNING
GEN 2
SPARSE LINEAR
REGRESSION
GEN 1
DECISION
TREES
x-3
= 2a
x-a

28. T H E P A T H T O B E T T E R E X P E R I M E N T A T I O N
IDEAS ARE GETTING
MORE COMPLEX
1
MODELS ARE GETTING
MORE COMPLEX
2 ResNeXt-101-32x4d
ResNet-101
Training Time (days)
Months?
7
4
2
14
ResNet-50
Internet-Scale Data / Videos
IN5K-ResNeXt-101-64x4d

29. HETEROGENOUS HARDWARE
• HPC workloads sensitive to hardware types, generations
• Scheduling more complex than just {x GB RAM, y CPU} per task
• Multiple gen GPUs, CPUs, ASICs
Heterogeneous Hardware
HPC workloads are sensitive to hardware types, generations
Scheduling is more complex than just {x GB RAM, Y CPU} per task
Multiple gen GPUs, CPUs, ASICs
• HPC workloads sensitive to hardware types, generations
• Scheduling more complex than just {x GB RAM, y CPU} per task
• Multiple gen GPUs, CPUs, ASICs
Heterogeneous Hardware

30. EXPENSIVE &
EXPERIMENTAL
Unlike data pipelines, majority of ML jobs are ad hoc
Long running jobs complicate demand prediction & control
Cost & Efficiency
ROI for jobs hard to estimate
Sub-linear scaling + huge scale﹦ an easy way to waste
resources

31. PY TORCH EL ASTIC
Enables users to write fault tolerant and elastic distributed PyTorch jobs
Use cases
Fault tolerance
Run on non-HPC cluster (e.g. cloud)
Mission critical production job
Dynamic Capacity Management:
Run on leased capacity that can be preempted (e.g. AWS spot
instances)
Shared pools where the pool size can change dynamically based
on demand (e.g. autoscaling)
Open sourced 0.1.0.rc - https://github.com/pytorch/elastic
Integrated with Classy Vision - https://classyvision.ai/tutorials/pet_aws

32. REALLY
L ARGE
SCALE
Tens of billions of training examples
Some models larger than size of machine
Hundreds of experimental runs competing for
resources
More than 1000+ different production models

33. PY TORCH RPC (REMOTE
PROCEDURE CALL)
Enables applications to run functions remotely, and
automatically handles autograd if necessary.
Use cases
Scale out applications
Parameter Server Framework
In RL, multiple observers streaming states and rewards to
the agent for policy training
Distributed model parallelism
Allow large models to span multiple machines
Hogwild! training

34. PY TORCH RPC COMPONENTS
RPC
Run user code with given args on the specified
destination
Remote Reference (RRef)
Tracks and maintains objects owned by a remote worker.
Distributed Autograd
Connects autograd graphs on different workers into one
global graph and provides a similar backward API.
Distributed Optimizer
Automatically handles RRef of subnets and expose a
similar API as local optimizers.

35. C H A L L E N G E # 4
D E P L O Y M E N T A T S C A L E

36. INFERENCE AT SCALE
Deploying and managing models in production is
difficult.
Some of the pain points include:
Loading and managing multiple models, on multiple
servers or end devices
Running pre-processing and post-processing code on
prediction requests.
How to log, monitor and secure predictions
What happens when you hit scale?

37. TORCHSERVE
• Default handlers for common use cases (e.g., image segmentation, text classification) along with custom handlers support
for other use cases
• Model versioning and ability to roll back to an earlier version
• Automatic batching of individual inferences across HTTP requests
• Logging including common metrics, and the ability to incorporate custom metrics
• Robust HTTP APIS - Management and Inference
EXPERIMENTAL

38. L O G G I N G A N D L O A D I N G
MLFLOW.PY TORCH
Integration with
TorchServe Coming Soon!

39. T H E L A T E S T | G E T S T A R T E D

40. • PyTorch 1.5 released on April 21st. Over 2,200 commits
• (Experimental) TorchServe model serving solution in partnership with AWS
• (Experimental) TorchElastic Kubernetes operator
• Stable Distributed Model Parallel Training - large models spanning across GPUs
• (Experimental) New high level autograd API including jacobian, hessian, jvp, vjp, hvp and vhp to
torch.autograd.functional
• Stable C++ Front End API
• (Experimental) New Custom C++ API, torch::class_, for binding custom C++ classes into TorchScript and Python
• Deprecated Python 2 support
NEW (CORE) FEATURES

41. • torchvision 0.6
- Faster R-CNN now supports negative samples which allows the feeding of images without annotations at training time
- Added a aligned flag to RoIAlign to match Detectron2
- Refactored abstractions for C++ video decoder
• torchtext 0.6
- Fixed issue related to the SentencePiece dependency in conda package
- Added support for the experimental IMDB dataset to allow a custom vocab
- A number of documentation updates including - code of conduct, deduplication of the docs
- Feedback requested for Experimental Datasets (#664) and Text classification dataset with new abstractions to
decouple data and vocab/tokenizer (PR#701)
• torchaudio 0.5
- Added the Griffin-Lim functional and transform, InverseMelScale and Vol transforms, and DB_to_amplitude
- Added support for allpass, fade, bandpass, band reject, band, treble, deep mph and riaa filters and transformations
- New datasets added including LJSpeech and SpeechCommands datasets.
DOMAIN LIBRARIES - NEW IN 1.5

42. G E T S TA R T E D
PY TORCH.ORG

43. EDUCATIONAL
RESOURCES

45. EDUCATIONAL
RESOURCES

46. https://pytorch.apachecn.org/
ALSO IN CHINESE :)

47. JOIN THE PYTORCH
DEVELOPER COMMUNITY
PyTorch.org
Youtube.com/pytorch
Twitter.com/pytorch
Facebook.com/pytorch
Medium.com/pytorch