ONNX: Past, Present, and
Future
… with use cases
Jim Spohrer (IBM), Prasanth Pulavarthi (Microsoft)
Monday June 29, 2020 11:30am - 12:20pm Central Time

Today’s Speakers
6/23/2020 2
Introduction to ONNX: Past, Present, Future
Jim Spohrer (IBM) – Director, Cognitive Opentech Group/CODAIT
Linux Foundation AI – Technical Advisory Council Chairperson
ONNX Steering Committee Member
https://www.ibm.com/opensource/centers/codait/
https://www.linkedin.com/in/spohrer/
@JimSpohrer
ONNX in Practice: Why we use it and how you can too
Prasanth Pulavarthi (Microsoft) – Principal Program Manager, AI Platform
ONNX Steering Committee Member, ONNX Co-Founder
https://www.linkedin.com/in/prasanthpulavarthi/

Introduction to ONNX: Past, Present, and Future
6/23/202
0
3
› Past: Why ONNX?
› Quick Review: ONNX Website Tour
› https://onnx.ai/
› Present: Growth of Community and Tools
› Quick Review: April 2020 ONNX Community Meeting
› https://tinyurl.com/onnx-april2020
› Future: You, AI Landscape and ONNX!
› Quick Review: Call to Action
› https://landscape.lfai.foundation/

46/23/2020
Nick Pentreath

56/23/2020
Jagreet Kaur Gil

ONNX Website Tour
https://onnx.ai/
6/23/2020 6
› Home Page
› News
› About
› Getting Started
› Supported Tools
› GitHub
› Gitter

April 2020 Community Meeting
https://tinyurl.com/onnx-april2020
6/23/2020 7
› Welcome and Updates
› Partners
› IBM Chief Data Office
› Huawei Mindspore
› Microsoft Runtime Optimizations
› Xilinx FINN
› UC Santa Cruz Genomics
› Microsoft Azure OCR
› Mathworks
› SIGs and Working Groups
› Some Highlights

Registered
86/23/2020

One Year Growth
96/23/2020
From March 2019 – April 2020 As of June 22, 2020
1548 162 8.5K
1.5K 148 35

Tools
106/23/2020

You, AI Landscape, and ONNX
https://landscape.lfai.foundation/
6/23/2020 11
› > 250 GitHub Projects
› > 1.4M stars
› > $12T Market Cap of
companies
› > $5B Investment in startups
› ONNX key to interoperability
and optimizations

ONNX in Practice:
Why we use it and how you can too
126/23/2020

Microsoft 365
AI/ML at Microsoft
| Research
Edge

Common problems impacting ML productivity
• Inference latency is too high to put into production
• Training in Python but need to deploy into a C#/C++/Java app
• Model needs to run on edge/IoT devices
• Same model needs to run on different hardware and operating systems
• Need to support running models created in several different frameworks
• (more recently) Training very large models takes too long

ONNX Runtime
https://onnx.ai
@onnxai
ONNX
https://onnxruntime.ai
@onnxruntime
Compatible with PyTorch, TensorFlow, Keras, SciKit-Learn, and more

Training framework Deployment target
CPU
GPU
FPGA
NPU
Improving ML deployment productivity

Training framework Deployment target
CPU
GPU
FPGA
NPU
Improving ML deployment productivity
ONNX
Runtime
Freedom to use tool of
choice
Strong performance and
compatibility with platforms
and accelerators

Speech with ONNX Runtime
Speech Services
power a vast landscape of products and
services at MSFT. From Office, Cortana to
Xbox and Bing, the ML models service
hundreds of millions of requests a month.
ONNX Runtime powers inferencing for
Speech at high scale in production
environments including Cognitive Services,
on-premise solutions, and micro services
Agility:
10x reduction in time to
productize new models
Performance:
10% accuracy and latency improvements

Azure Cognitive Services using ONNX Runtime

Azure Kinect with ONNX Runtime
Desktop scenario
Body Tracking SDK is installed on your PC
Reduced the First Frame Processing Time by
7.8x on the GTX 1070 with ONNX Runtime with
CUDA execution provider.
Body Tracking SDK
Azure Kinect developer kit for tracks bodies in 3D
with advanced AI sensors that use sophisticated
computer vision and speech models.
IoT scenario (in progress)
Body Tracking SDK is installed on Jetson TX2
(ARM CPU + Nvidia GPU)

ONNX model portability with ONNX Runtime
model.onnx
TensorRTOpenVINO
model.onnx
CUDA
model.onnx

WindowsML with ONNX Runtime
• Windows API for machine learning
inference
• Input models are ONNX models for
broad framework support
• ONNX Runtime as engine with
DirectML acceleration

Azure Customers with ONNX Runtime
Financial ISV
• Uses AI for economic scenario modeling
• Train models in Python with SciKit-Learn and PyTorch but production environment is pure
C#
• ONNX Runtime with C# API was a good fit (bonus 2x speedup)

Transformer Inferencing
› Hugging Face provides popular transformer models, like
BERT, GPT2, etc.
› Can be trained with either PyTorch or TensorFlow
› Hugging Face module transformers.convert_graph_to_onnx
exports ONNX models
› ONNX Runtime does inferencing with speedup whether you
are using CPU or GPU

New: Transformer Training
› Integrates with PyTorch (and TensorFlow) to accelerate training and fine-
tuning of large transformer models
› Incorporates latest algorithms and techniques such as DeepSpeed/ZeRO and
Parasail/Adasum
› Used by Office, Visual Studio, and others at Microsoft
› Available as preview now
256/23/2020

Getting Started

Pre-trained ONNX models: https://onnx.ai/models

ML.NET
Creating an ONNX model

Examples: Model Conversion
python -m tf2onnx.convert
--input frozen_model.pb
--inputs input_batch:0, lengths:0
--outputs top_k:1
--fold_const
--opset 8
--output deepcc.onnx
import sklearn
import skl2onnx
initial_type = [('float_input', FloatTensorType([1, 4]))]
onnx_model = skl2onnx.convert_sklearn(pipe,
initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
f.write(onnx_model.SerializeToString())

ONNX Runtime: https://onnxruntime.ai

Using ONNX Runtime
C#
…… also available for C, C++, Java, and JavaScript (Node.js)

Get involved

ONNX has open governance
› Annual Steering Committee election
› Technical decisions made by SIGs and Working Groups
› All meetings open to everyone
› Calendar: https://onnx.ai/calendar
› GitHub: https://github.com/onnx
› Gitter: https://gitter.com/onnx
› Mailing List: https://lists.lfai.foundation/g/onnx-announce
336/23/2020

Thank You!
Q & A
346/23/2020
Resources
https://onnx.ai
https://onnxruntime.ai

Legal Notices
› The Linux Foundation, The Linux Foundation logos, and other marks that may be used herein are owned by The Linux Foundation or its
affiliated entities, and are subject to The Linux Foundation’s Trademark Usage Policy at https://www.linuxfoundation.org/trademark-usage, as
may be modified from time to time.
› Linux is a registered trademark of Linus Torvalds. Please see the Linux Mark Institute’s trademark usage page at https://lmi.linuxfoundation.org
for details regarding use of this trademark.
› Some marks that may be used herein are owned by projects operating as separately incorporated entities managed by The Linux Foundation,
and have their own trademarks, policies and usage guidelines.
› TWITTER, TWEET, RETWEET and the Twitter logo are trademarks of Twitter, Inc. or its affiliates.
› Facebook and the “f” logo are trademarks of Facebook or its affiliates.
› LinkedIn, the LinkedIn logo, the IN logo and InMail are registered trademarks or trademarks of LinkedIn Corporation and its affiliates in the
United States and/or other countries.
› YouTube and the YouTube icon are trademarks of YouTube or its affiliates.
› All other trademarks are the property of their respective owners. Use of such marks herein does not represent affiliation with or authorization,
sponsorship or approval by such owners unless otherwise expressly specified.
› The Linux Foundation is subject to other policies, including without limitation its Privacy Policy at https://www.linuxfoundation.org/privacy and its
Antitrust Policy at https://www.linuxfoundation.org/antitrust-policy. each as may be modified from time to time. More information about The Linux
Foundation’s policies is available at https://www.linuxfoundation.org.
› Please email legal@linuxfoundation.org with any questions about The Linux Foundation’s policies or the notices set forth on this slide.
6/23/2020 35