AWS Summit Milano 2019 - Amazon SageMaker sviluppa, addestra e distribuisci modelli di Machine Learning su larga scala - Giuseppe Porcelli, Solutions Architect, AWS | Cliente: Contentwise

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker: sviluppa, addestra
e distribuisci modelli di Machine
Learning su larga scala
Giuseppe A. Porcelli
Sr. Solutions Architect
Amazon Web Services EMEA
A W S S U M M I T M I L A N 2 0 1 9
Mario Scriminaci
Head of Product Management and Research
ContentWise

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine Learning at Amazon
Voice driven
interactions
Fulfillment automation
& inventory management
Personalized
recommendations
Drones
Inventing entirely new
customer experiences

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS MISSION
Put Machine Learning in the
hands of every developer
A long history of successfully
running large-scale
Machine Learning
Democratizing Artificial Intelligence
AMAZON’S EXPERIENCE

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The AWS Machine Learning Stack
M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
M L P L A T F O R M S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
Vision Speech Language Chatbots
F O R E C A S T
Forecasting
T E X T R A C T P E R S O N A L I Z E
Recommendations

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The AWS Machine Learning Stack
M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
Vision Speech Language Chatbots
F O R E C A S T
Forecasting
T E X T R A C T P E R S O N A L I Z E
Recommendations
M L P L A T F O R M S
A M A Z O N
S A G E M A K E R
B U I L D T R A I N D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization (N E O )
One-click deployment & hosting
Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The AWS Machine Learning Stack
M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
Vision Speech Language Chatbots
F O R E C A S T
Forecasting
T E X T R A C T P E R S O N A L I Z E
Recommendations
M L P L A T F O R M S
F r a m e w o r k s I n t e r f a c e s
I n f r a s t r u c t u r e
E C 2 P 3
& P 3 D N
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
A M A Z O N
S A G E M A K E R
B U I L D T R A I N D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization (N E O )
One-click deployment & hosting
Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
A W S D E E P L E A R N I N G A M I

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Tens of Thousands of Customers Running ML on AWS

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine Learning Process is Hard, and Time Consuming
Experimentation
• Setup and manage
Notebook environments
• Get data to notebooks securely
Training
• Setup and manage clusters
• Scale/distribute ML algorithms
Deployment
• Setup and manage
inference clusters
• Manage and auto scale
inference APIs
• Testing, versioning,
and monitoring
Fetch data
Clean & format
data
Prepare &
transform data
Train modelEvaluate model
Integrate with
prod
Monitor/
debug/refresh
6–18
months
Data Wrangling
• Manage data ingestion
• Execute ETL

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker
A managed service that provides the quickest and
easiest way for developers and data scientists to get
ML models from idea to production

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AMAZON SAGEMAKER
Pre-built
notebooks for
common problems
Built-in, high
performance
algorithms
BUI L D
T R AI N & T UNE D E P L O Y
BUILD, TRAIN, TUNE AND HOST YOUR OWN MODELS

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AMAZON SAGEMAKER
Pre-built
notebooks for
common problems
Built-in, high
performance
algorithms
One-click
training
BUI L D T R AI N & T UNE
D E P L O Y
Hyperparameter
optimization
BUILD, TRAIN, TUNE AND HOST YOUR OWN MODELS

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AMAZON SAGEMAKER
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common problems
Built-in, high
performance
algorithms
One-click
training
BUI L D T R AI N & T UNE D E P L O Y
Hyperparameter
optimization
BUILD, TRAIN, TUNE AND HOST YOUR OWN MODELS

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AMAZON SAGEMAKER
Fully managed
hosting with auto-
scaling
One-click
deployment
Pre-built
notebooks for
common problems
Built-in, high
performance
algorithms
One-click
training
BUI L D T R AI N & T UNE D E P L O Y
End-to-end encryption with KMS
End-to-end VPC support
Compliance and audit capabilities Pay as you go
Hyperparameter
optimization
BUILD, TRAIN, TUNE AND HOST YOUR OWN MODELS
BUILT-IN BYO

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Is Framework Agnostic
Choose from popular frameworks
Run them fully managed Or run them yourself

15. © 2019 ContentWise
Speed-Up algorithms from prototype to
production with Amazon SageMaker
Mario Scriminaci
Head of product
management and research
March 12th, 2019 AWS Summit Milan

16. © 2019 ContentWise© 2019 ContentWise
the driverless user experience

17. © 2019 ContentWise17
personalization
is the art and technology to serve a
customer
1-to-1 tailored experience

18. © 2019 ContentWise18
Give superpowers to editorial teams
Deliver a personalized,
self-tuning experience
Automate the
digital content
storefront
ContentWise helps
customers manage the UX

19. © 2019 ContentWise© 2019 ContentWise
from prototype to production

20. © 2019 ContentWise20
Prototype
Implementation
Trying/evolving an
algorithm has to be quick
and standardized
Production
deployment
Once an algorithm is
evaluated offline it has to
be ready for production
Performance &
quality evaluation
We have 30+ algorithms
but only few are used in
production

21. © 2019 ContentWise21
AWS Cloud
Amazon VPC
Next evolution
Amazon Simple Storage Service (S3)
Amazon SageMaker
CW Data
Scientist
Algorithm Git
repository
Amazon EC2
Container
Registry
Container Git
repository
Dataset Model
Notebook Train Model Endpoint
Amazon EC2
Production

22. © 2019 ContentWise22
Data scientists implement
and test with small instances
and then we evaluate quality
and performance with the
scaled ones.
Creating, testing, and
evaluating a new algorithm is
now time effective.
Containerization on both
training and delivery with
standard interfaces.
benefits time
cost
agility

23. © 2019 ContentWise23
evolution
complete the
integration with the
product
pipelines
apply the same cycle
to other ML algorithms

24. © 2019 ContentWise
Headquarters
Via Schiaffino 11
20158 Milan Italy
T +39-02-4951-7001
USA Boston
211 Congress Street
Boston, MA 02110
T: +1-617-936-0212
USA Los Angeles
12655 W Jefferson Blvd
Los Angeles, CA 90066
T: +1-323-524-0524
@contentwisetv moviri-
contentwise
contentwise.tv

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
90+ New Enhancements to SageMaker last Year
MXNet 1.3 container | CloudTrail integration for audit logs | TensorFlow 1.7 Containers | Automatic Model Tuning—Add/Delete tags | Jupyter Notebooks IP Filtering
Region expansion to SFO | Image Classification Multi-label Support | TensorFlow and MXNet Containers—Open Sourcing and Local Mode | PyTorch pre-built container
Region expansion to PDT | Batch customer VPC | PCI DSS Compliance | XGBoost Instance Weights | NTM—vocab, metrics, and subsampling
Anomaly Detection (Random Cut Forest) Algorithm | Deep AR algorithm | SageMaker region expansion to ICN | Hyperparameter tuning job cloning on the console
Autoscaling console | PyTorch 1.0 container | Customer VPC support for training and hosting | PrivateLink support for SageMaker inferencing APIs
Horovod support in TensorFlow Container | Variable sizes for notebook EBS volumes |nbexample support in SageMaker notebook instances | Tag-based access control
Automatic Model Tuning—early stopping | IP Insights algorithm | Chainer 4.1 Container | Region expansion to SIN Built-in Algorithms Pipe Mode Support
TensorFlow 1.8 Container | Region expansion to FRA | Training job cloning in console | Algorithm Pipe mode enhancements | Pipe mode support for text, recordIO, and images
TensorFlow 1.5, MXNet 1.0, and CUDA 9 Support | DeepAR Algorithm Enhancements | Linear Learner Multi-class Classification | TensorFlow 1.10 Container
Region expansion to YUL | BlazingText Algorithm | Batch KMS | k-nearest neighbors | Object detection |Chainer pre-built container | Apache Airflow integration
Region expansion to BOM | GDPR compliance | BlazingText Enhancements | TensorFlow 1.9 Container | Notebook bootstrap script
Amazon SageMaker Hosting custom header attribute | Metrics Support in Training Jobs | Object2vec | TensorFlow container enhancements | CloudFormation support
PrivateLink support for SageMaker Control Plane | MXNet 1.2 Container | HIPAA compliance | Ground Truth | Python SDK Marketplace support
Git integration for SageMaker notebooks | Pipe mode support for TensorFlow | ml.p3.2xlarge notebook instances | Internet-free notebook instances
Semantic segmentation algorithm | SageMaker Reinforcement Learning support | Linear Learner Improvements | SageMaker Batch Transform
Region expansion to NRT | High Performance I/O streaming in PIPE Mode | Pause/resume for active learning algorithms | Pre-built scikit-learn container
Step Functions for SageMaker | KMS support for training and hosting | Incremental learning algorithm enhancements | TensorFlow 1.11 container | NTM feature release
Deep Learning Compiler | ONNX Support for Frameworks and Algorithms |Full instance type support | Pipe mode CSV support | Region expansion to LHR
Incremental training platform support | Login anomaly detection algorithm | Serial inference pipeline | Experiment Management | Region expansion to SYD
MXNet container enhancements | Automatic Model Tuning | Automatic Model Tuning—incremental tuning | Spark MLeap 1P container
TensorFlow 1.6 and MXNet 1.1 Containers | Region expansion to SIN | Mead Notebook PrivateLink Support | Linear Learner sparsity support

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Three areas we are improving for ML developers
CostData
Cost
We’re improving both training and
inference speed & cost
Data
Preparing data for ML is major
expensive, complex, and time consuming
Ease of use
We continue to want to reduce the
barrier of entry to ML for all developers

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Improving Training & Inference Cost

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The largest P3 instance,
optimized for distributed training

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon EC2 P3dn instance
The largest P3 instance, optimized for distributed training
Reduce machine
learning training time
Better GPU
utilization
Support larger, more
complex models
K E Y F E A T U R E S
100Gbps of networking
bandwidth
(4x more P3)
8 NVIDIA Tesla
V100 GPUs
32GB of
memory per GPU
(2x more P3)
96 Intel
Skylake vCPUs
(50% more than P3)
with AVX-512

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Training gets a lot of attention,
but what about inference?

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Prediction Training
Inference
(Prediction)
90%
Training
10%
Predictions drive
complexity and
cost in production

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The challenges of prediction in production
One size does not fit all Elasticity is important

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Add GPU acceleration to any Amazon EC2 instance or Amazon SageMaker
for faster inference at much lower cost (up to 75% savings)

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Elastic Inference
Reduce deep learning inference costs up to 75%
Match capacity
to demand
Available between 1 to 32
TFLOPS
K E Y F E A T U R E S
Integrated with
Amazon EC2,
Amazon SageMaker, and
Amazon DL AMIs
Support for TensorFlow,
Apache MXNet, and ONNX
with PyTorch coming soon
Single and
mixed-precision
operations
Lower inference costs

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Making it easier to obtain high
quality labeled data

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build highly accurate training datasets and reduce
data labeling costs by up to 70% using machine
learning

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
Label machine learning training data easily and accurately

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Driving Ease of Use

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Marketplace for machine learning
ML algorithms and models available instantly

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Optimization is extremely complex

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
A deep learning model compiler that lets customers train models once,
and run them anywhere with up to 2X improvement in performance

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo
Train once, run anywhere with 2x the performance
K E Y F E A T U R E S
Compiler & run-time are open source 1/10th the size of original models

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’s next for
machine learning?

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Supervised learning
Unsupervised learning
Types of Machine LearningSOPHISTICATIONOFMLMODELS
AMOUNT OF TRAINING DATA REQUIRED

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Types of Machine Learning
AMOUNT OF TRAINING DATA REQUIRED
Supervised learning
Unsupervised learning
SOPHISTICATIONOFMLMODELS

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Types of Machine Learning
Reinforcement learning
(RL)
Supervised learning
Unsupervised learning
AMOUNT OF TRAINING DATA REQUIRED
SOPHISTICATIONOFMLMODELS

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does RL work?
USE CASES

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New machine learning capabilities in Amazon SageMaker to
build, train and deploy with reinforcement learning

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker RL
Reinforcement learning for every developer and data scientist
Broad support
for frameworks
Broad support for simulation
environments including
SimuLink and MatLab
K E Y F E A T U R E S
TensorFlow, Apache
MXNet, Intel RL Coach,
and Ray RL support
2D & 3D physics
environments and
OpenAI Gym support
Supports Amazon Sumerian and
AWS RoboMaker
Fully
managed
Example notebooks
and tutorials

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Other ways to
getting started…

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine Learning University
Uses the same
materials used to train
Amazon developers
Foundational
knowledge with
real-world application
Structured
courses and
specialist certification

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon ML Solutions Lab
Brainstorming Modeling Teaching
Leverage Amazon experts with decades of ML
experience with technologies like Amazon Echo,
Amazon Alexa, Prime Air and Amazon Go
Amazon ML Solutions
Lab provides ML
expertise
Quick turnaround
Demonstration of
functionality
Static data
Limited live data-sets
Core functionality
Limited integration with
production systems
Initial Architecture
Full live data-sets
Enhanced functionality
Link to production systems
Scaled Architecture
Iteration on MVP2
Fully operationally
Fully supported
Live security-approved
account

53. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Giuseppe A. Porcelli
Sr. Solutions Architect
Amazon Web Services EMEA
Mario Scriminaci
Head of Product Management and Research
ContentWise