For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/06/seamless-deployment-of-multimedia-and-machine-learning-applications-at-the-edge-a-presentation-from-qualcomm/ Megha Daga, Senior Director of Product Management for AIoT at Qualcomm, presents the “Seamless Deployment of Multimedia and Machine Learning Applications at the Edge” tutorial at the May 2022 Embedded Vision Summit. There has been an explosion of opportunities for edge compute solutions across the internet of things. This growth in opportunities and the diversity of applications is leading to fragmentation in the IoT space both in hardware and software, which creates challenges for developers. In addition, customers and developers are facing challenges in efficient data management and optimized application deployment on embedded edge platforms. In this session, Daga introduces the Qualcomm Intelligent Multimedia SDK, which empowers developers to tackle these challenges and deploy edge compute applications in a scalable, flexible and optimized way. The Qualcomm Intelligent Multimedia SDK easily decodes and organizes sensor data and executes applications efficiently on edge platforms.

1. Seamless Deployment
of Multimedia & Machine
Learning Applications
at the Edge
Megha Daga
Sr Director, Product Management, AIoT
Qualcomm Technologies, Inc.

2. Intelligent Edge use cases
growing at exponential scale
2
Retail
Intelligent Inventory
Management:
Latency driving productivity
Secured & Bandwidth
driven last mile delivery
Manufacturing
Robot Navigation
& Management:
Latency driven real-time
defect inspection on
conveyor belts
Healthcare
Latency critical
Remote Robotics:
Assisted Surgery,
Immersive training AR/VR,
Image diagnostics,
OR monitoring
Smart City
Smart AI Cameras:
Bandwidth & Latency
critical traffic flow across
connected cameras
Secured building management
across connected sensors
Warehouse & Logistics
People Navigation:
Latency driven quickest route
Real-time package sorting
Latency/
Speed
Bandwidth/
Load Balancing/
Scale
Reliability Cost
Privacy/
Security

3. Exponential increase and diversity in use cases
leading to fragmentation across Hardware & Software
HARDWARE
Fragmentation
NPU CV Audio
SOFTWARE
Fragmentation
Framework Tools
Profiler Debugger
Deployment

4. Iterative process on embedded:
Developer challenge around efficient
end-to-end MM/ML pipeline deployment
4
Planning/Prototyping:
Feature and Product
Segmentation selection
Deploy/Scale:
Feature update
Pipeline creation (end to end application development) | Memory management | Debugging/Profiling/Optimization
Development:
Feature Integration

5. Qualcomm IM SDK
Hardware
CV ISP/Camera Video AI/ML Audio
Streaming Apps w/ AI-inference
Platform Libs, HALs, Drivers
Qualcomm®
Intelligent Multimedia SDK for end to end
MM/ML pipeline deployment
5
Modular | Scalable | Optimized
Qualcomm® Intelligent Multimedia (IM) SDK
Multimedia
Plugins
AI/ML
Plugins
Display
Plugins
Networking
Plugins
CV
Plugins
QualcommIM SDK is a productof QualcommTechnologies, Inc. and/or its
subsidiaries.

6. Qualcomm IM SDK Architecture
6
Customer Applications
HAL/Platform libs
LE Camera Service OMX Video/Decode
Camx/Camera fastCV
Fast RPC Pulse/Audio
Binder Weston
GLES/Open CL GBM
Drivers
Camx/Camera Video
DSP ION/DMA
WLAN ALSA
Display Display
GPU AIC100
HW
Camera/ISP CV Sensors Audio
Video AI100
Camera Source Video Transform
Video Encode/Decode Video Compose
Audio Encode/Decode Wayland Sink
Socket Src/Sink Frame Overlay
Pulse Src/Sink File Parsers/Muxers
Multimedia and Display Subsystem
Post Process
AIC100 Tensor Convertor Batching
TFLite ROI Muxer Stream/Tensor Serializer
SNPE CROP ROI Tensor Demuxer
Detection Classification
Segmentation Pose
AI/ML Subsystem
Optical Flow
DFS
CV Subsystem
RTSP
HLS in/out
Web-RTC
Network Subsystem
IM SDK Plugins

7. Qualcomm IM SDK is
a unified SW SDK across our IoT chipsets
7
Unified across our IoT chipsets
LOW
MID
HIGH
PREMIUM
• 1-2 camera stream
• <1TOPs/stream
• Upto 4 camera streams
• 1-2 TOPs/stream
• Upto 16 camera streams
• 2-4 TOPs/stream
• >16 streams
• >4 TOPs/stream

8. • Runs Inference on live stream to find objects
in frame and highlight with bounding boxes.
• Returns multiple bounding boxes (bbox) with
score and label name
• Inference results - draw bbox around objects
with text per frame
MACHINE LEARNING – EXAMPLE USE CASE
Live Camera Inference: object detection
8
Pipeline
filesrc qtdemux queue h264parse qtivdec tee
qtimlvconverter qtimlaic qtimlvclassification
qtivcomposer waylandsink

9. 9
Needs to be formatted
Creating a reference application

10. 10
Creating a reference application
Launch GStreamer Identify all required plugins & configure parameters Connect plugins & Launch

11. 11
Creating a reference application
Launch GStreamer Identify all required plugins & configure parameters Connect plugins & Launch

12. 12
Creating a reference application
Launch GStreamer Identify all required plugins & configure parameters Connect plugins & Launch

13. • Runs multiple inferences on 16 live streams
to find objects in frame and highlight with
bounding boxes.
• Inference results - draw bounding boxes
around objects with different colors for
different objects
• Runs optical flow concurrently on a stream
& display motion vector drawing
13
MULTIMEDIA & MACHINE LEARNING – EXAMPLE USE CASE
Multi-Inferencing

14. 14
MULTIMEDIA & MACHINE LEARNING – EXAMPLE USE CASE
Multi-Inferencing
Demux Decode Frame skip
Demux Decode Frame skip
Demux Decode Frame skip
17 Instances 17 Instances 17 Instances
CPU Batch
(Aggregator)
4 Channels
CPU Batch
(Aggregator)
4 Channels
CPU Batch
(Aggregator)
4 Channels
Optical flow
(CVP)
1 Channel
Processing
(GLES/SL)
Processing
(GLES/SL)
Processing
(GLES/SL)
Motion Vector
Overlay
4 Channel IN 1 channel OUT
YoloV5
Contigous
Memory
4x480x640
Contigous
Memory
4x480x640
Contigous
Memory
4x480x640
Tensor
Demux
Tensor
Demux
Tensor
Demux
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
NMS Sub Module
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
NMS Sub Module
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
Post Process
NMS Sub Module
3 instances 16 instances 16 instances
NMS post
processing
and offline
rendering
(RGB)
Video Encode
Wayland sink
Parse /
MP4 mux.
Display
OR
34 instances 34 input channels =
17 YUV channels +
17 RGB channels
Tee
Tee
Tee
Tee
Tee
Video
Composer

15. To resolve fragmentation problem,
need for “IoT as a simple software
stack” is eminent.
Conclusion
15
Qualcomm IM SDK
Hardware
CV ISP/Camera Video AI/ML Audio
Streaming Apps w/ AI-inference
Platform Libs, HALs, Drivers
Qualcomm IM SDK
Multimedia
Plugins
AI/ML
Plugins
Display
Plugins
Networking
Plugins
CV
Plugins
Qualcomm IM SDK act as the major
building block of the software stack for
seamless end to end application
deployment.

16. Thank you
16
Nothing in these materials is an offer to sell any of the components
or devices referenced herein.
©2018-2022 Qualcomm Technologies, Inc. and/or its affiliated
companies. All Rights Reserved.
Qualcomm is a trademark or registered trademark of Qualcomm
Incorporated. Other products and brand names may be trademarks or
registered trademarks of their respective owners.
References in this presentation to “Qualcomm” may mean Qualcomm
Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or
business units within the Qualcomm corporate structure, as applicable.
Qualcomm Incorporated includes our licensing business, QTL, and the vast
majority of our patent portfolio. Qualcomm Technologies, Inc., a subsidiary
of Qualcomm Incorporated, operates, along with its subsidiaries,
substantially all of our engineering, research and development functions,
and substantially all of our products and services businesses, including our
QCT semiconductor business.