“Deploying PyTorch Models for Real-time Inference On the Edge,” a Presentation from Nomitri
•
0 gefällt mir•1,034 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/10/deploying-pytorch-models-for-real-time-inference-on-the-edge-a-presentation-from-nomitri/ Moritz August, CDO at Nomitri GmbH, presents the “Deploying PyTorch Models for Real-time Inference On the Edge” tutorial at the May 2021 Embedded Vision Summit. In this presentation, August provides an overview of workflows for deploying compressed deep learning models, starting with PyTorch and creating native C++ application code running in real-time on embedded hardware platforms. He illustrates these workflows on smartphones with real-world examples targeting ARM-based CPU, GPUs, and NPUs as well as embedded chips and modules like the NXP i.MX8+ and NVIDIA Jetson Nano. August examines TorchScript, architecture-side optimizations, quantization and common pitfalls. Additionally, he shows how the PyTorch deployment workflow can be extended to conversion to ONNX and quantization of ONNX models using an ONNX Runtime. On the application side, he demonstrates how deployed models can be integrated efficiently into a C++ library that runs natively on mobile and embedded devices and highlights known limitations.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie “Deploying PyTorch Models for Real-time Inference On the Edge,” a Presentation from Nomitri
Ähnlich wie “Deploying PyTorch Models for Real-time Inference On the Edge,” a Presentation from Nomitri (20)
Mehr von Edge AI and Vision Alliance
Mehr von Edge AI and Vision Alliance (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
“Deploying PyTorch Models for Real-time Inference On the Edge,” a Presentation from Nomitri
- 1. © 2021 Nomitri GmbH Deploying PyTorch Models for Real-time Inference On the Edge Moritz August CDO & Co-Founder Nomitri
- 2. Intro
- 3. © 2021 Nomitri GmbH • Start-Up based in Berlin, Germany • Deep Learning vision applications for mobile/edge • First product in retail • From edge application over ML to backend service • Use PyTorch and C++ library to deploy our models About us
- 4. © 2021 Nomitri GmbH • Imperative, simple API • Dynamic computation graphs at its core • Great ecosystem • Debug with Python debugger • Caught up with TensorFlow Why PyTorch? Google Trends: PyTorch (Blue) vs TensorFlow (Red)
- 5. © 2021 Nomitri GmbH PyTorch Mobile • Available for Android, Linux and iOS • Provides several ready-to-use models • Simple deployment workflows • Support for Arm CPU and accelerators Torchvision • Provides many pre-trained vision architectures • Tools for augmentation, IO and bounding boxes • Contains models optimized for mobile/edge PyTorch Mobile & Torchvision
- 6. © 2021 Nomitri GmbH x86 CPU NVIDIA GPU ARM CPU Mobile GPU NPU/DSP torch.nn.Module TorchScript ONNX torch.jit torch.onnx LibTorch ONNX Runtime Python C++ Deployment Workflows
- 8. © 2021 Nomitri GmbH Example Model: Classification
- 9. © 2021 Nomitri GmbH Example Model: Classification
- 10. © 2021 Nomitri GmbH Example Model: Classification
- 11. © 2021 Nomitri GmbH • Use depthwise-separable convolutions • Fuse operations like Conv, BatchNorm, ReLU • Make channels divisible by 8 • Use efficient and fuseable activations like ReLU • Use channels-last memory format Architecture Optimization
- 12. © 2021 Nomitri GmbH Architecture Optimization
- 13. © 2021 Nomitri GmbH Torchvision offers utility function to adapt channel numbers Architecture Optimization
- 14. © 2021 Nomitri GmbH • PyTorch has built-in support for model compression • Pruning • Quantization • Pruning is implemented via weight masks • Quantization for weights and activations • Advanced techniques have open-source implementations Model compression
- 15. © 2021 Nomitri GmbH • Quantization supports different modes • Dynamic quantization • Static quantization • Quantization-aware training • Has two backends for execution on x86 and Arm CPUs • Quantization can be customized via configuration Workflow static/post-training quantization Quantization Add Stubs Train Fuse Prepare Calibrate Quantize
- 16. © 2021 Nomitri GmbH Quantization
- 17. © 2021 Nomitri GmbH Quantization
- 18. © 2021 Nomitri GmbH Quantization
- 19. © 2021 Nomitri GmbH • Performing the optimizations yields drastic improvements • Measurements for 8 layers with 18 to 1154 channels Model Comparison
- 20. Deployment
- 21. © 2021 Nomitri GmbH • Statically typed intermediate representation • Serialize and ship to production environments • Multiple ways to convert models to TorchScript • Convert entire module • Trace graph with example input • Write in TorchScript TorchScript
- 22. © 2021 Nomitri GmbH TorchScript: Conversion, Tracing & Serialization
- 23. © 2021 Nomitri GmbH TorchScript: Example output
- 24. © 2021 Nomitri GmbH • PyTorch has C++ 14 API • Closely follows the Python API • Can be used by simple inclusion of torch header • Typically used for inference only • Simply load TorchScript models for inference General workflow for using LibTorch LibTorch Define model Train model Convert to TorchScript Load TorchScript model Do Inference Python C++
- 25. © 2021 Nomitri GmbH FBGEMM CUDA QNNPACK Vulkan/ Metal NNAPI LibTorch x86 CPU NVIDIA GPU ARM CPU Mobile GPU NPU/DSP Backends
- 26. © 2021 Nomitri GmbH • Simple interface to perform graph optimizations • Allows optimization for different backends • XNNPACK for floating point on Arm CPU • QNNPACK for quantized 8-bit on Arm CPU • Vulkan for GPU on Android • Metal for GPU on iOS Mobile CPU and GPU
- 27. © 2021 Nomitri GmbH • TorchScript model can be optimized to run via NNAPI • Model should be fused and quantized beforehand • Channels-last memory format is mandatory • NNAPI model can be wrapped to provide float interface Workflow NNAPI optimization NNAPI Quantize Remove Stubs Prepare NNAPI input Trace Convert
- 28. © 2021 Nomitri GmbH NNAPI
- 29. © 2021 Nomitri GmbH • Open Neural Network Exchange (ONNX) • Open-Source ecosystem for switching frameworks • ONNX Runtime ML acceleration framework by Microsoft • Supports variety of accelerators for inference and training • PyTorch has native support for export to ONNX ONNX
- 30. © 2021 Nomitri GmbH • Only unquantized models can be exported to ONNX • ONNX Runtime supports quantization • Dynamic • Static • Quantization-aware training • PyTorch dataloader can be re-used with a small wrapper Static/post-training quantization workflow ONNX Quantization Define model Train model Convert to ONNX Load ONNX model Calibrate & Quantize Python/PyTorch C++/ONNX Runtime
- 31. © 2021 Nomitri GmbH ONNX Quantization
- 32. © 2021 Nomitri GmbH • Different backends support different operations • Safe choices for architectures are • Convolution (including grouping) • Add • ReLU • Nearest neighbor upsampling • Adaptive average pooling • Strictly sequential graphs • Memory layout requirements vary Pitfalls & Limitations
- 33. © 2021 Nomitri GmbH • Efficient models are not enough • Inefficient handling of camera input can cost a lot • Some useful optimizations • Resize frames early • Merge float conversion and normalization • Merge reordering and splitting of channels • Use buffers to hold frames C++: Efficient input pipeline
- 35. © 2021 Nomitri GmbH • Has images with latest PyTorch versions available • Supports CUDA • Can run TorchScript float models on GPU Jetson Nano
- 36. © 2021 Nomitri GmbH • PyTorch Mobile can be used directly in Java/Kotlin • Recommended: native C++ library using LibTorch • Available accelerators can be used with little overhead • Quantized models on CPU are good fallback Android Phone
- 37. © 2021 Nomitri GmbH Android Phone
- 38. © 2021 Nomitri GmbH • Yocto images include PyTorch and ONNX Runtime • PyTorch version can only access CPU • Execution provider for ONNX Runtime to use NPU/GPU • Convert PyTorch models to quantized ONNX models i.MX 8M Plus
- 39. © 2021 Nomitri GmbH Runtime Comparison • Classifier with full MobileNetV2 backbone • Hardware: i.MX 8M Plus vs. Jetson Nano vs. OnePlus 6
- 40. Conclusion
- 41. © 2021 Nomitri GmbH Summary • PyTorch facilitates model development & prototyping • Easy model architecture optimization • Provides workflows for Arm CPU, GPU and NPU/DSP • ONNX is a powerful alternative for inference • Deployment to all popular hardware platforms
- 42. © 2021 Nomitri GmbH ● PyTorch website ● TorchVision ● PyTorch Mobile ● ONNX ● ONNX Runtime ● ONNX Runtime Quantization ● PyTorch ONNX Export ● PyTorch Quantization ● PyTorch Mobile Optimizer ● PyTorch IOS GPU Workflow ● PyTorch Android GPU Workflow ● PyTorch NNAPI Workflow ● QNNPACK blog post Resources Example code of this talk Talk is based on PyTorch 1.8.1 and ONNX Runtime 1.7