The number of IoT devices which stream data to the cloud increases daily. In this practical session, we will build an end-to-end architecture for real-time analytics using the latest IoT technologies like IoT edge and data bricks.

1. Real Time Data Analytics in IoT
From edge to cloud
Hi, I am Marcel, technical director of Codit Switzerland
1
marcel.lattmann@codit.swiss
Live from Zürich !

2. Connecting end to end
Connect things to the cloud
Cloud strategy & architecture
Automate business processes
Expose API’s
2

3. About Codit
3
2000 Belgium
2004 France
2013 Portugal
2016 Switzerland
2016 UK
2016 The Netherlands
2017 Malta
180
worldwide
Largest Microsoft
partner in Europe for
integration, API
management, IoT
and Azure Solutions

4. 4
The IoT Value Chain

5. Traffic camera scenario
5

6. Traffic camera scenario
6

7. Traffic camera scenario
7
Edge
| License plate recognition
| Speeding alert (display)
| Display traffic info
Cloud
Speeding tickets
Detect suspicious cars
Detect and predict incidents
|
|
|

8. 1. Azure IoT Edge : Transmit camera events
8
Azure IoT Edge IoT Hub
Devices
Local Storage
Nebulus SerialPort
(Container)
Functions
RuntimeContainer
Management
Device
Twin
Device
Twin
Azure Stream
Analytics
(Container)
Azure Functions
(Container)
Cognitive Services
(Container)
Azure Machine
Learning
(Container)
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin
Module
Twin

9. Architecture
9
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

10. 2. Time Series Insights : Data exploration
10
• Get near real-time
insights in seconds
• Start in seconds, scale in
minutes
• Create a global view of
your IoT-scale data
• Leverage the power of
Time Series Insights in
your Apps and Solutions

11. Architecture
11
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

12. Architecture
12
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

13. 3. Stream Analytics: in the cloud & on the edge
13
Presentation &
Action
Storage &
Batch Analysis
Stream
Analytics
Event Queuing
& Stream
Ingestion
Event
production
IoT Hubs
Applications
Archiving for long
term storage/
batch analytics
Real-time dashboard
Stream
Analytics
Automation to
kick-off workflows
Machine Learning
Reference Data
Event Hubs
Blobs
Devices &
Gateways PowerBI

14. Stream analytics on the edge
14
Camera simulator
Stream Analytics Display simulator
Cloud hub
Edge hub
"routes": {
"edge": "FROM /messages/modules/camera/outputs/*
INTO BrokeredEndpoint("/modules/traffic-speeding-detection-edge/inputs/iot-speed-events")",
"camera": "FROM /messages/modules/traffic-speeding-detection-edge/outputs/*
INTO BrokeredEndpoint("/modules/display/inputs/camera")",
"cloud": "FROM /messages/modules/camera/outputs/*
INTO $upstream"
}

15. Architecture
15
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

16. Architecture
16
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

17. 4. Structured data streaming: Azure Data Bricks
17
Optimized Databricks Runtime Engine
DATABRICKS I/O SERVERLESS
Collaborative Workspace
Cloud storage
Data warehouses
Hadoop storage
IoT / streaming data
Rest APIs
Machine learning models
BI tools
Data exports
Data warehouses
Azure Databricks
Deploy Production Jobs & Workflows
APACHE SPARK
MULTI-STAGE PIPELINES
DATA ENGINEER
JOB SCHEDULER NOTIFICATION & LOGS
DATA SCIENTIST BUSINESS ANALYST

18. Streaming pipeline
| readStream…load() creates
a streaming DataFrame, does
not start any computation
val input = spark.readStream
.format("json")
.load("source-path")
val output = input
.select(“clientid“, “querytime”)
.where(“querytime > 100")
output.writeStream
.format("json")
.start(“dest-path")

19. Structured streaming model

20. Structured streaming model
±
±

21. Continuous aggregations
| Continuously compute average query time
across all clients
input.avg(“querytime")
input.groupBy(“devicemake")
.avg(“querytime")

22. Continuous windowed aggregations
• Simplifies event-time stream processing (not possible in DStreams)
• Works in both, streaming and batch jobs
input.groupBy(
$“devicemake”,
window($“event-time”, “10 min”))
.avg(“querytime”)

23. Joining streams with static data
| Join streaming data from Kafka with static
dataset from JDBC source to enrich
streaming data
val kafkaDataset = spark.readStream
.kafka(“device-updates”)
.load()
val staticDataset = spark.read
.jdbc(“jdbc://”, “device-info”)
val joinedDataset =
kafkaDataset.join(
staticDataset, “devicemake”)

24. Architecture
25
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

25. Architecture
26
IoT Hub
Time Series
Stream Analytics
Blob Storage
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge

26. Takeaways
27
| Azure IoT Edge for connectivity & AI
close to the devices
| Azure IoT Hub as secure , high
performant service that connects it
all
| Data analytics options available
| Stream Analytics for quick starting and
easy query logic
| Azure Data Bricks as 1st class citizen for
streaming, machine learning and
translation
| Multiple data integration options
available
The value of IoT
is defined by the
data and
integration

27. Thank you. Let’s connect!
28