AIoT and edge computing solutions on Azure

1. AIoT 與邊緣運算
Tommy Wu 吳志忠
tommy.wu@microsoft.com
微軟大中華區物聯網解決方案架構師

2. Agenda
• The Internet of Things
• IoT Solutions
• IoT Solutions with Machine Learning
• IoT Edge Computing
• AI Platform – Deep Learning and Cognitive Services
• Computer Vision and Mix Reality(AR)
• AIoT Use Cases Sharing
2

3. Intelligent Cloud
Intelligent Edge

4. 68

5. Connectivity Data AnalyticsThings Action

6. Things Insights Actions
Cloud
Gateway

7. Solution PortalProvisioning API
Identity & Registry Stores
Stream Event Processor
Analytics/
Machine
Learning
Data
Visualization &
Presentation
Device State Store
Gateway/
Edge
Storage
IP capable
devices
Existing IoT
devices
Low power
devices
PresentationDevice and Event Processing
Data
Transport
Devices and
Data Sources
Cloud
Gate-
way
Agent
Libs
Agent
Libs
Control System Worker Role
Agent
Libs

8. Azure Time Series
Insights
Azure Machine
Learning
Azure Stream
Analytics
Cosmos DB Azure Data Lake
Azure Data Lake
Analytics
Azure HD Insight
Spark, Storm,
Kafka
Azure Event Hubs
Microsoft Flow
Azure Logic Apps
Notification Hubs
Azure Websites
Microsoft Power
BI
Azure Active
Directory
Azure IoT Hub
Azure IoT Hub
Device Provisioning
Service
Azure IoT Edge
Azure Monitor
PaaSServices&
DeviceSupport
Edge Support
Device Support
Azure IoT Device
SDK
Certified Devices
Azure Certified for
IoT
Security Program
for Azure IoT
IoT Services Data & Analytics Services Visualization & Integration Services
IoTSolutions
(PaaS)
IoTSolutions
(SaaS)
Microsoft IoT Central
IoT SaaS
Microsoft Connected Field Service
Field Service SaaS
Remote Monitoring Predictive Maintenance Connected factory
Windows 10 IoT
Core
Azure IoT Suite

9.  使用 Azure IoT 裝置 SDK 來實作用戶端應用程式，
以便在裝置硬體平臺與作業系統上執行。
 裝置 SDK 包含程式庫，可協助將遙測傳送至 IoT
中樞，並接收雲端到裝置訊息。 當您使用裝置
SDK 時，您可以從數種網路通訊協定中選擇，以便
與 IoT 中樞通訊
ConnectivityThings
•Azure IoT SDK for C
•Azure IoT SDK for Python
•Azure IoT SDK for Node.js
•Azure IoT SDK for Java
•Azure IoT SDK for .NET
 Windows 10 IoT Core
 Windows 10 IoT Enterprise
 Linux
 Android
 RTOS’s (Real Time Operating System)
(ex. VxWorks)
 Many others-BLE ,Modbus RTU ,LoRa…
 IP connectivity
 Built in security

11. 從設備端到資料洞察到採取行動，改變整個企業，走向全世界
架設在行業領先的雲平臺之上
安全
端到端
覆蓋設備端、網路連接、資料加
密以及雲端
開放
連接任意物件
相容任意設備、作業系統、資料
來源、軟體和服務
快速
數分鐘內可開展
預置的方案範本適用於大多數的
物聯網場景
擁有全球領先的商業智慧和分析平臺*
彈性
對高增⾧應付自如
支援數百萬的設備、TB級別的資
料、本地或雲端部署，輕易覆蓋
全球30多個地區
人數據 洞察 行動閘道設備
*February 2015. The Gartner Magic Quadrant for Business Intelligence and Analytics Platforms is the property of Gartner, Inc. and available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and
does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all
warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. The above graphics were published by Gartner, Inc. as part of a larger research document and should be evaluated in the
context of the entire document.

12. • 數分鐘內就可以完成部署並開始收集資料
• 規則和警示可以基於現有的來修改
• 按需要添加設備和客制
常見的IoT場景可迅速開展
遠程監控
• 對特定的資產和流程進行有針對性的調整優化
• 對即時運營的資料㇐目了然
• 整合現有的後臺系統
完整實現物聯網的應用
預測性維護

13. 流式即時處理和預測分析
設備連接與管理
採集資料，發送命令和控制
工作流自動化與整合
報表與視覺化展示
Azure Solution Accelerator: 遠程監控
設備
後臺
系統
和業
務流
程
Cosmos DB
Web App
Logic AppsIoT Hub
模擬器
Active
Directory
Orche
strato
r
Micros
ervices
VM
Micros
ervices
VM
微服務
VM
• IoT 遠端監控方案架構 - 基於微服務架構建置
• 提供 .NET 和 Java 版本的開源實現

14. Azure IoT Central -完整的SaaS 物聯網解決方案
相較於典型的 IoT 專案，Azure IoT 中心藉由下列方
法，讓 IoT 解決方案管理變輕鬆：
•降低管理負擔。
•降低營運成本和額外負荷。
•高度可擴展性
•讓應用程式自訂變容易，同時利用：
• 領先業界的技術，例如 Azure IoT 中
樞 和 Azure 時間序列深入解析。
• 企業級安全性功能，例如端對端加密

16. IoT Button 物聯網應用Demo-PlayyourideaanywherethroughAzureIoTtechnology
Azure IoT Central
Short Press – Button Power On
Long Press – AP mode & Settings
Azure Web App
Trigger Event

17. 提供客戶在裝置上分析資料的服務，也稱為 「在 Edge 上分析」，而不是在雲端進行分析。 將部份工作負載移至
Edge，您的裝置就能以較少的時間將訊息傳送至雲端，並更快速地變更狀態
Azure IoT Edge 是由三個元件組成：
 IoT Edge 模組是執行 Azure 服務、第三方服務或自有程式碼的容器(Docker)。 這類模組會
部署到 IoT Edge 裝置，並在這些裝置本機上執行。
 IoT Edge 執行階段會在每個 IoT Edge 裝置上執行，並管理部署到每個裝置的模組。
 以雲端為基礎的介面可讓您在遠端監視及管理 IoT Edge 裝置。
若要使用 Windows 容器，您必須執行：
Windows 10 Fall Creators Update，或
Windows Server 1709 (組建 16299)，或
x64 型裝置上的 Windows IoT 核心版 (組建 16299)
pip install -U azure-iot-edge-runtime-ctl

18. Things
IoT Pattern + Edge
Insights Actions
Azure IoT Edge
Cloud
Gateway
Azure IoT Hub
ActionsInsights

19. Azure IoT Edge Deployment
Edge Brings AI to Reality
Azure
IoT Hub
Azure
Machine
Learning
IoT Edge
Device
Azure Container Registry
Azure
Cognitive
Services
Azure
Event Grid
Azure
Functions
Deployment
Manifest
Azure
Stream
Analytics
SQL
Server
Docker
Container
Docker
Container
Docker
Container
Docker
Container
Docker
Container
Docker
Container

20. https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/category/internet-of-
things?page=1&subcategories=iot-edge-modules

22. ARTIFICIAL INTELLIGENCE
MACHINE LEARNING
DEEP LEARNING
Deep neural networks revolutionize AI
The idea of “intelligent machines”
Development of algorithms to learn from data

23. Bringing the best of AI to Azure and the best of Azure to AI
Microsoft AI Platform
AI Services
AI Infrastructure
AI Tools
PRE-BUILT AI CONVERSATIONAL AI CUSTOM AI
Cognitive Services Bot Framework Azure Machine Learning
AI ON DATA AI COMPUTE
Data
Lake
SQL
Server
Cosmos
DB
Spark DSVM Batch AI AkS
Azure ML
Studio
Azure
Notebooks
VS Tools for
AI/AML
DEEP LEARNING FRAMEWORKS
Cognitive
Toolkit
TensorFlow Caffe2
Others (Azure Workbench, Pycharm…)
Others (Scikit-learn, Keras, PyTorch, MxNet, Chainer…)
CODING AND MANAGEMENT TOOLS
IoT
AI SILICON

25. A variety of real-world applications
Vision Speech
Intent: PlayCall
Language Knowledge Search

27. Bringing the best of AI to Azure and the best of Azure to AI
Microsoft AI Platform
AI Services
AI Infrastructure
AI Tools
PRE-BUILT AI CONVERSATIONAL AI CUSTOM AI
Cognitive Services Bot Framework Azure Machine Learning
AI ON DATA AI COMPUTE
Data
Lake
SQL
Server
Cosmos
DB
Spark DSVM Batch AI AkS
Azure ML
Studio
Azure
Notebooks
VS Tools for
AI/AML
DEEP LEARNING FRAMEWORKS
Cognitive
Toolkit
TensorFlow Caffe2
Others (Azure Workbench, Pycharm…)
Others (Scikit-learn, Keras, PyTorch, MxNet, Chainer…)
CODING AND MANAGEMENT TOOLS
IoT
AI SILICON

28. Building your own AI models for Transforming Data into Intelligence
Prepare Data Build & Train Deploy

29. Lifecycle 1. define problem
2. acquire + process data
3. design model architecture
4. train model5. test/evaluate
a. initialize
b. feed in minibatch of data
c. calculate lossd. optimize: minimize loss
e. update weights
y =Wx + b
loss = |desired – actual outcome|δ
6. deploy

30. Prepare
Data
Register and
Manage Model
Train & Test
Model
Build
Image
Build model
Azure Notebooks
Deploy
Service
Monitor
Model
Prepare Experiment Deploy
Azure Machine Learning Process

31. Machine Learning Studio 互動式工作區
提供互動式的視覺化工作區，讓您輕鬆建置、測試和反覆運算預測分析模型。 您可以將「資料集」和分析「模組」拖放到
互動式畫布，將它們連接在㇐起以構成「實驗」，然後在 Machine Learning Studio 中執行

32. Anomaly detection models in Azure Stream Analytics
Built-in ML models for anomaly detection in Azure Stream Analytics significantly
reduces the complexity and costs associated with building and training machine
learning models.
•AnomalyDetection_SpikeAndDip function to detect
temporary or short-lasting anomalies such as spike or
dips.
•AnomalyDetection_ChangePoint function to detect
persistent or long-lasting anomalies such as bi-level
changes, slow increasing and slow decreasing trends.
SELECT sensorid, System.Timestamp as time, temperature as temp,
AnomalyDetection_SpikeAndDip(temperature, 95, 120, 'spikesanddips’)
OVER PARTITION BY sensorid
LIMIT DURATION(second, 120) as SpikeAndDipScores
FROM input

37. Model

38. https://aka.ms/cvsexport
https://github.com/Azure-
Samples/cognitive-services-ios-
customvision-sample
https://github.com/Xamarin/ios-
samples/tree/master/ios11/Core
MLAzureModel

39. 




40. You own
Prediction Endpoint
(Docker container)
Local or on-prem
Azure App Service

41. The vision AI developer kit
- Qualcomm® Technologies, Inc. and Microsoft
collaboration
- Run AI models on the edge without additional
computers or web connection or leverage the cloud
- Create, deploy and manage all your models in the cloud
and the edge with Azure ML and Azure IoT Edge
- Register for early access preview today:
http://www.visionaidevkit.com

43. 視覺邊緣運算展示#2: 工安視覺辨識
Things Insights
• Machine Learning on the Edge
• Vision Artificial Intelligence
Action
• Operational & compliance
reporting

45. Azure Spatial Anchors overview
Azure Spatial Anchors empowers developers with essential capabilities to build spatially aware mixed reality
applications. It enables developers to work with mixed reality platforms to perceive spaces, designate precise
points of interest, and to recall those points of interest from supported devices. These precise points of
interest are referred to as Spatial Anchors. It is composed of a managed service and client SDKs for
supported device platforms
• Support Microsoft HoloLens, iOS-based devices
supporting ARKit, and Android-based devices
supporting ARCore.
• Support Unity for creating and deploying mixed reality
applications
• Multi-user experiences. Spatial Anchors makes it easy for
people in the same place to participate in multi-user
mixed reality applications.
• Way-finding. Developers can also connect Spatial
Anchors together creating relationships between them.
• Persisting virtual content in the real-world. An app can
let a user place a virtual calendar on a conference room
wall, that people can see using a phone app or a
HoloLens device.

46. 46
With Azure Spatial Anchors , You can
• Create and locate anchors using Azure Spatial Anchors
• Anchor relationships and way-finding in Azure Spatial Anchors
• Persisting virtual content in the real-world.
• Logging and diagnostics in Azure Spatial Anchors
• Android SDK reference

47. Solution Architecture
Devices Azure
Spatial Anchors with Speech Service Demo
1. 搜尋附近可用錨點
2. 上傳語音並轉換為文字(Cognitive Service)
3. 判別所選物件並產生錨點(上傳至Spatial Anchors)
4. 將產生的錨點和物件儲存至其 Web服務(API)
使用者掃描現場環境, 找到真實世界的可用錨點, 並且
使用Android App透過語音輸入所要物件
Spatial Anchors
Azure App Service
Speech to Text(Cognitive Service)
1
2
Android
Mobile Device
3
4

49. AIoT Use Cases Sharing&Demo
• Sketch2Code
• GAN Network
• Industrial AOI for Laser-cut machinery
• Vision-Guided AGV/Drone
• OpenPose/Posenet on IoT Edge

51. Transform Your handwritten design from a picture to valid HTML markup
https://sketch2code.azurewebsites.net/

52. •A Microsoft Custom Vision Model:
This model has been trained with images of
different HTML elements like buttons, text
box, and images.
•A Microsoft Computer Vision Service:
To identify the text written into a design
element a Computer Vision Service is used.
•An Azure Blob Storage:
All steps involved in the HTML generation
process are stored, including the original
image, prediction results and layout
grouping information.
•An Azure Function:
Serves as the backend entry point that
coordinates the generation process by
interacting with all the services.
•An Azure website:
User font-end to enable uploading a new
design and see the generated HTML results.

53. Confirm

54. https://drawingbot.azurewebsites.net/
At the core of Microsoft’s drawing bot is a
technology known as a Generative Adversarial
Network, or GAN. The network consists of two
machine learning models, one that generates
images from text descriptions and another,
known as a discriminator, that uses text
descriptions to judge the authenticity of
generated images

55. 雷射加工機檢測與參數優化挑戰
• 金屬加工扣件產業
• 人工檢測費時
• 開發機器學習模型時間較長
• 參數調整目前憑人工經驗
• 檢測Fail 種類繁多

56. 解決方案- Azure Customvision AI 模型訓練
• 快速開發模型，僅需1天時間
• 檢測分類快速，無需人工介入
• Object Detection Model Compact model offline for IoT Edge.
• 後續根據錯誤分類，進行雷射參數調整訓練

57. 資料集標籤截圖畫面

58. 資料集標籤截圖畫面
雷射焊道標記

59. Custom Vision 訓練結果
未來會再增加資料集提升準確度

60. 應用程式產出

62. • Vision-guided AGVs to
follow a trained object
without human
assistance
• Training on the fly ,
Execution on the edge.
• Dynamic change
object model without
prebuilt cost
• Multi-object detection
for path routing
decision
• Companion Edge

63. • Quick Dashboard for
Drone Flight Statistics
with IoT Central
• Training on the fly ,
Execution on the edge.
• Multi-object detection
for drone control
decision
• Intelligent Edge

64. Bring your own models

65. OpenPose/Posenet on IoT Edge
• Edge Inference
• Detect Position
• Calculate Angle
• Virtual Assistant
Advisor
https://a9nodered.azurewebsites.net/posenet2

66. System Architecture
AML with Posenet/OpenPose