This session shows you how you can use Microsoft Azure to build a high-scalable solution for event-processing. You can use this approach for classical IoT-scenarios or if you want for example to capture telemetry-data of a widely distributed application. Then each application-instance will send data to Azure’s Event Hub. In this session you will not only get some insights into the Event Hub, but also into Stream Analytics. Stream Analytics is used to aggregate the millions of events coming from the Event Hub by using a SQL-like syntax. From Stream Analytics the data can be pushed into a database or for example into a Live Dashboard in Microsoft’s Power BI.

1. BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE
HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH
#SDF16
Building High-scalable Enterprise
Solutions with Microsoft Azure and
PowerBI
Thomas Claudius Huber – Principal Consultant
@thomasclaudiush

2. #SDF16
Thomas Claudius Huber
Principal Consultant
– Microsoft Most Valuable Professional (MVP)
Focus on C#, TypeScript, Angular and Azure
– Part of Cloud-Unit @Trivadis
Book- and Pluralsight-Author
– See also www.thomasclaudiushuber.com

3. #SDF16
Agenda
The Big Picture
Azure Event Hubs
Stream Analytics and Power BI

4. #SDF16
The Big Picture

5. #SDF16
The big picture

6. #SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI

7. #SDF16
Azure Event Hubs

8. #SDF16
Azure Event Hubs
Scales up to millions of events
Great approach for telemetry data
Great approach for sensor-data

9. #SDF16

10. #SDF16
Azure Event Hubs: Partitions
Event stream is segmented by partitions for scale
– Scaling for event consumers
– Scaling for event submitters
You can have between 2 and 32 partitions
Azure Support can enable up to 1024 or even more «under special conditions»

11. #SDF16
Azure Event Hubs: Throughput Units
Throughput units (TUs) are important for billing. 1 Unit means
– Write maximum 1MB/sec and 1000 messages per second
– Read maximum 2MB/sec
Number of partitions has to be equal or greater than number of TUs
TUs are applied on namespace level
– You can have up to 10 event hubs in a namespace
– Namespace is used for the URI, for example
https://<yourNamespace>.servicebus.windows.net

12. #SDF16
Azure Event Hubs: Consumer Groups
Views on the Event Stream
– Similar to topic subscriptions that are also a «view» on an event stream
– Organizational anchor for checkpointing
$Default consumer group is always there
– Up to 20 named consumer groups are supported
Limitations
– 5 concurrent reading connections per consumer group
• It’s recommended to use a separate consumer group for each job

13. #SDF16
Azure Event Hub vs. Azure IoT Hub
Quote: «It is not uncommon to use both IoT Hub and Event Hubs in the same solution»
Official docmentation: https://azure.microsoft.com/en-us/documentation/articles/iot-hub-
compare-event-hubs/
Event Hub IoT Hub
Communication Device to cloud Device to cloud and cloud to device
Security
Event hub wide shared
access policy
Per device identity
Scale
5000 simultaneous
connections
Millions of simultaneous connections
SDKs
Supported on .NET and C,
also has AMQP and HTTP
(REST) interfaces
Device SDKs for several platforms.

14. #SDF16
But why not always using IoT Hub?
IoT Hub scales on devices
– a lot of concurrent connections
– but is expensive for a few thousand devices and a lot events
Event Hub scales on events
– but is limited to 5000 concurrent connections

15. #SDF16
So what about pricing?
Event Hub
CHF 0.0253 per million events
1 Throughput Unit = CHF 0.0135/hr (~CHF 10/month)
– 1 Throughput Unit = 1MB / 1000 Messages Ingress, 2MB read

16. #SDF16
Creating an Event Hub: Step 1
Go to the https://portal.azure.com
Select «New  Internet of Things  Event Hubs»
Enter Namespace-Name and Resource Group and click «Create»

17. #SDF16
Creating an Event Hub: Step 2
So far you’ve created a namespace
– the namespace can have up to 10 event hubs
When you navigate to the namespace (via «Resources»), you can add a new event
hub

18. #SDF16
Demo: Create an Event Hub

19. #SDF16
Connect to the Event Hub
To push events to an event hub, you have several options:
– From a .NET application you can use the NuGet-package
WindowsAzure.ServiceBus and the EventHubClient-class
• Samples: https://github.com/thomasclaudiushuber/EventHub.Clients
– You can also use a REST-API that is available automatically for each event hub
• https://github.com/thomasclaudiushuber/EventHub.RestClientGenerator
You need to setup a Shared Access policy for your event hub
To access the event hub, you need the URI and the shared access key
– both is part of a Connection String that you find in the Shared Access policy

20. #SDF16
Pushing Events to the Event Hub from a .NET App
Use the NuGet-package WindowsAzure.Service
– It has an EventHubClient-class
Events are JSON-data items

21. #SDF16
Consuming Events from the Event Hub in a .NET App
Same EventHubClient-class
Grab the consumerGroup and create the receivers for the partitionIds

22. #SDF16
Demo: Access the Event Hub

23. #SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI

24. #SDF16
The big picture

25. #SDF16
What is Stream Analytics
it’s a real-time event processing engine
Use a SQL-variant to query the event stream
High scalable

26. #SDF16
Stream Analytics: Inputs and Outputs

27. #SDF16
Stream Analytics: Windowed Aggregates
There are different windowed aggregates, one is the Tumbling Window
select
DeviceName,max(value),count(*)
into
[SDFPBOutput]
from
[SDFEventHub99]
Timestamp by Readtime
group by devicename,
TumblingWindow(
Duration(second, 5),
Offset(millisecond, -1))

28. #SDF16
Demo: Setting up Stream Analytics

29. #SDF16
Power BI as an Output
BI-tool to visualize your data
– Access it via https://www.powerbi.com
Stream Analytics can directly push its outupt to Power BI
In Power BI you can work with the Stream Analytics Data

30. #SDF16
Build a Live Dashboard
in Power BI

31. #SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI

32. #SDF16
The big picture

33. #SDF16
Questions
Thomas Claudius Huber
Principal Consultant
thomas.huber@trivadis.com
@thomasclaudiush