Deploy PyTorch models in Production on AWS with TorchServe

6. 6© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | But deploying PyTorch models in production is a challenge • No official model server • Need to write custom code to deploy and predict with the trained models • For production workloads, need to build your own systems for scaling, monitoring, security, etc.

7. 7© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | Easily deploy PyTorch models in production at scale. Introducing TorchServe An open-source model serving library for PyTorch, built and maintained by AWS in collaboration with Facebook.

8. 8© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | Benefits with TorchServe No need to write custom code for common model types Low latency model serving Works with any ML environment

9. 9© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | How it works Client Application Model Training Train models with PyTorch TorchServe Start TorchServe model server and load trained models Prediction One or more models hosted on SageMaker, EC2, EKS, Kubernetes, or any other ML environment Prediction API

12. 12© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | Key features • Low latency prediction API provided automatically • Default handlers for most common applications like object detection, text classification, etc. • Multi-model serving • Model versioning for A/B testing • Monitoring/logging • RESTful end points that can be accessed via web requests (HTTP)

13. 13© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | “We continuously optimize and improve our computer vision models, which are critical to TRI- AD’s mission of achieving safe mobility for all with autonomous driving. Our models are trained with PyTorch on AWS, but until now PyTorch lacked a model serving framework. As a result, we spent significant engineering effort in creating and maintaining software for deploying PyTorch models to our fleet of vehicles and cloud servers. With TorchServe, we now have a performant and lightweight model server that is officially supported and maintained by AWS and the PyTorch community.“ – Yusuke Yachide, Lead of ML Tools at TRI-AD Toyota Research Institute - Advanced Development

14. 14© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | “Matroid, maker of computer vision software that detects objects and events in video footage, develops a rapidly growing number of machine learning models using PyTorch on AWS and on-premise environments. The models are deployed using a custom model server that requires converting the models to a different format, which is time-consuming and burdensome. TorchServe allows Matroid to simplify model deployment using a single servable file that also serves as the single source of truth, and is easy to share and manage.” – Reza Zadeh, Founder CEO, Matroid

16. /suman-d Stay Connected…

17. Thank You Suman Debnath Principal Developer Advocate, India ml.aws

Deploy PyTorch models in Production on AWS with TorchServe

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Deploy PyTorch models in Production on AWS with TorchServe

Ähnlich wie Deploy PyTorch models in Production on AWS with TorchServe (20)

Mehr von Suman Debnath

Mehr von Suman Debnath (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Deploy PyTorch models in Production on AWS with TorchServe