SlideShare ist ein Scribd-Unternehmen logo
1 von 51
Downloaden Sie, um offline zu lesen
Azure Stream Analytics  : Analyse Data in Motion
Deepthi Anantharam
Technology Evangelist
@deananth
Ruhani Arora
Technology Evangelist
@infinitydlimit
The need for evolution – Identified 2 years ago
… data warehousing has reached the most
significant tipping point since its inception. The
biggest, possibly most elaborate data
management system in IT is changing.
– Gartner, “The State of Data Warehousing in 2012”
Data sources
The “Traditional” Data Warehouse
4
Data sources
Increasing
data volumes
1
Real-time
data
4
Non-Relational Data
New data
sources & types
2
Cloud-born
data
3
Evolving Approaches to Analytics
ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Data Marts
Data Lake(s)
Dashboards
Apps
ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Data Marts
Data Lake(s)
Dashboards
Apps
Evolving Approaches to Analytics
ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Scale-out
Storage &
Compute
(HDFS, Blob Storage,
etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
Evolving Approaches to Analytics
ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Scale-out
Storage &
Compute
(HDFS, Blob Storage,
etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
Evolving Approaches to Analytics
Real Time data analytics
Agenda
• Azure Data Factory
• Azure Stream Analytics
Azure Data Factory Overview
• New Azure service for data developers & IT
• Compose data processing, storage and movement services to create & manage
analytics pipelines
• Initially focused on Azure & hybrid movement to/from on premises SQL Server.
Overtime will expand to more storage & processing systems throughout
• Rich, simple end-to-end pipeline monitoring and management
Operationalizing Information Production With
Data Factory
Example Scenario:
Customer Profiling (game usage analytics)
Customer Profiling – Game Usage Analytics
2277,2013-06-01 02:26:54.3943450,111,164.234.187.32,24.84.225.233,true,8,1,2058
2277,2013-06-01 03:26:23.2240000,111,164.234.187.32,24.84.225.233,true,8,1,2058-2123-2009-2068-2166
2277,2013-06-01 04:22:39.4940000,111,164.234.187.32,24.84.225.233,true,8,1,
2277,2013-06-01 05:43:54.1240000,111,164.234.187.32,24.84.225.233,true,8,1,2058-225545-2309-2068-2166
2277,2013-06-01 06:11:23.9274300,111,164.234.187.32,24.84.225.233,true,8,1,223-2123-2009-4229-9936623
2277,2013-06-01 07:37:01.3962500,111,164.234.187.32,24.84.225.233,true,8,1,
2277,2013-06-01 08:12:03.1109790,111,164.234.187.32,24.84.225.233,true,8,1,234322-2123-2234234-12432-344323
…
Log Files Snippet (10s of TBs per day in cloud storage)
User Table
UserID FirstName LastName State …
2277 Pratik Patel Oregon
664432 Dave Nettleton Washington
8853 Mike Flasko California
New User Activity Per Week By Region
profileid day state duration rank weaponsused interactedwith
1148 6/2/2013Oregon 216 33 1 5
1004 6/2/2013Missouri 22 40 6 2
292 6/1/2013Georgia 201 137 1 5
1059 6/2/2013Oregon 27 104 5 2
675 6/2/2013California 65 164 3 2
1348 6/3/2013Nebraska 21 95 5 2
Terminologies
• Linked Services
• Data Sets
• Pipeline
• Diagram View
• Create a Data factory
• Add Data Sources
• Define Tables and Pipelines
• Deploy & Start
• Monitor and Manage
Steps
Example: Game Logs, Customer Profiling
On Premises SQL Server Azure Blob Storage
1000’s Log FilesNew User View
Azure Data Factory
Example: Game Logs, Customer Profiling
On Premises SQL Server Azure Blob Storage
1000’s Log FilesNew User View
Azure Data FactoryViewOf
Game Usage
ViewOf
New Users
New User
Activity
Example: Game Logs, Customer Profiling
ViewOf
On Premises SQL Server Azure Blob Storage
1000’s Log FilesNew User View
Copy “NewUsers” to
Blob Storage
Cloud New
Users
Azure Data FactoryViewOf
Game Usage
ViewOf
New Users
New User
Activity
Pipeline
Example: Game Logs, Customer Profiling
On Premises SQL Server Azure Blob Storage
1000’s Log FilesNew User View
Copy NewUsers to
Blob Storage
Cloud New
Users
Azure Data FactoryViewOf
Game Usage
ViewOf
Mask & Geo-
Code
New Users
Geo Dictionary
Geo Coded
Game Usage
HDInsight
New User
Activity
Pipeline
Pipeline
Example: Game Logs, Customer Profiling
On Premises SQL Server Azure Blob Storage
1000’s Log FilesNew User View
Copy NewUsers to
Blob Storage
Cloud New
Users
Azure Data FactoryViewOf
Game Usage
ViewOf
RunsOn
Mask & Geo-
Code
New Users
Geo Dictionary
Geo Coded
Game Usage
Join &
Aggregate
HDInsight
New User
Activity
ViewOf
Pipeline
Pipeline
Pipeline
“GeoCoded Game Usage” Table:
Step 3: Define Tables & Pipelines
Pipeline Definition:
Step 3: Define Tables & Pipelines
Powershell
// Deploy Table
New-AzureDataFactoryTable
-DataFactory“GameTelemetry“
-File NewUserActivityPerRegion.json
// Deploy Pipeline
New-AzureDataFactoryPipeline
-DataFactory “GameTelemetry“
-File NewUserTelemetryPipeline.json
// Start Pipeline
Set-AzureDataFactoryPipelineActivePeriod
-Name “NewUserTelemetryPipeline“
-DataFactory “GameTelemetry“
-StartTime 10/29/2014 12:00:00
Incremental Data Production
Dataset2
Dataset3
Hourly
12-1
1-2
2-3
Daily
Monday
Tuesday
Wednesday
Daily
Monday
Tuesday
Wednesday
Hive
Activity
GameUsage
GeoCodeDictionary
Geo-Coded
GameUsage
Custom Actions
• Allows running any .NET code wrapped within an ADF activity
• Can be used to connect to new sources/destination
• Can be used to create custom transformation activities
• Example: Invoke Azure ML model
• SDK for custom activity creation:
Coordination:
• Rich scheduling
• Complex dependencies
• Incremental rerun
Authoring:
• JSON & Powershell/C#
Management:
• Lineage
• Data production policies (late data, rerun, latency, etc)
Hub: Azure Hub (HDInsight + Blob storage)
• Activities: Hive, Pig, C#
• Data Connectors: Blobs, Tables, Azure DB, On Prem SQL Server, MDS [internal]
Data Factory – Available Today
Analyze your data in motion
What is Streaming Data?
Data in MotionData at Rest
Azure Stream Analytics
Managed real-time analytics
Mission-critical reliability and scale
Rapid development
Point of
Service Devices
Self Checkout
Stations
Kiosks
Smart
Phones
Slates/
Tablets
PCs/
Laptops
Servers
Digital
Signs
Diagnostic
EquipmentRemote Medical
Monitors
Logic
Controllers
Specialized
DevicesThin
Clients
Handhelds
Security
POS
Terminals
Automation
Devices
Vending
Machines
Kinect
ATM
How do customers create a real-time
streaming solution?
Customers using ASA?
Using Azure Analytic Service
Data Source
Collect Process
Consume
Deliver
Event Inputs
- Event Hub
- Azure Blob
Transform
- Temporal joins
- Filter
- Aggregates
- Projections
- Windows
- Etc.
Enrich
Correlate
Outputs
- SQL Azure
- Azure Blobs
- Event Hub
- Table Storage
Azure
Storage
Azure Stream Analytics
Reference Data
- Azure Blob
Sample Scenario : Toll Station
TollId EntryTime
License
Plate
State Make Model Type Weight
1 2014-10-25T19:33:30.0000000Z JNB7001 NY Honda CRV 1 3010
1 2014-10-25T19:33:31.0000000Z YXZ1001 NY Toyota Camry 2 3020
3 2014-10-25T19:33:32.0000000Z ABC1004 CT Ford Taurus 2 3800
2 2014-10-25T19:33:33.0000000Z XYZ1003 CT Toyota Corolla 2 2900
1 2014-10-25T19:33:34.0000000Z BNJ1007 NY Honda CRV 1 3400
2 2014-10-25T19:33:35.0000000Z CDE1007 NJ Toyota 4x4 1 3800
… … … … … … … …
EntryStream - Data about vehicles entering toll stations
TollId ExitTime LicensePlate
1 2014-10-25T19:33:40.0000000Z JNB7001
1 2014-10-25T19:33:41.0000000Z YXZ1001
3 2014-10-25T19:33:42.0000000Z ABC1004
2 2014-10-25T19:33:43.0000000Z XYZ1003
… … …
ExitStream - Data about cars leaving toll stations
LicensePlate RegistartionId Expired
SVT6023 285429838 1
XLZ3463 362715656 0
QMZ1273 876133137 1
RIV8632 992711956 0
… … ….
ReferenceData - Commercial vehicle registration data
Query Language - Overview
DML Statements
• SELECT
• FROM
• WHERE
• GROUP BY
• HAVING
• CASE
• JOINS
• UNION
Scaling Functions
• WITH
• PARTITION BY
Date and Time Functions
• DATENAME
• DATEPART
• DAY
• MONTH
• YEAR
• DATETIMEFROMPARTS
• DATEDIFF
• DATADD
Windowing Extensions
• Tumbling Window
• Hopping Window
• Sliding Window
Aggregate Functions
• SUM
• COUNT
• AVG
• MIN
• MAX
String Functions
• LEN
CONCAT
• SUBSTRING
• CHARINDEX
• PATINDEX
Tumbling Windows
SELECT TollId, COUNT(*)
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, TumblingWindow(second, 10)
Count the total number of vehicles entering each toll booth every interval of 10 seconds.
1 5 4 26 8 6 5
0 5 2010 15
Time
(secs)
1 5 4 26
8 6
25
A 10-second Tumbling Window
30
3 6 1
5 3 6 1
Hopping Windows
SELECT COUNT(*), TollId
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, HoppingWindow (second, 10,5)
Count the number of vehicles
entering each toll booth every
interval of 10 seconds; update
results every 10 seconds
1 5 4 26 8 7
0 5 2010 15
Time
(secs)
25
A 10-second Hopping Window with a 5-second “Hop”
30
4 26
8 6
5 3 6 1
1 5 4 26
8 6 5 3
6 15 3
Sliding Windows
Give me the count of all the toll
booths which have served more than
10 vehicles in the last 10 seconds
1 5
0 5 2010 15 Time
(secs)
25
A 10-second Sliding Window
8
8
51
9
51 9
1
SELECT TollId, Count(*)
FROM EntryStream ES
GROUP BY TollId, SlidingWindow (second, 10)
HAVING Count(*) > 10
Intake millions of events per second
Process data from connected devices/apps
Integrated with highly-scalable publish-subscriber ingestor
Easy processing on continuous
streams of data
Transform, augment, correlate, temporal operations
Detect patterns and anomalies in streaming data
Correlate streaming with reference
data
Input and Output
Management
Transformations
Management
Programmatic Access with REST APIs
Jobs Management
Start Job
Stop Job
Create Job
Delete Job
List Jobs
Update Job
Create Input / Output
Delete Input / Output
List Input / Output
Update Input / Output
Create Transformation
Delete Transformation
Get Transformation
Update Transformation
The full functionality of
Azure Stream Analytics is
through REST APIs.
Enables programmatic
access
Useful for automation
through scripting
Embed in other
applications/tools
Demo: Scaling , Monitoring & Logging
Scaling Concepts – Partitions
Step Result1
Step Result2
Step Result3
PartitionId=1
PartitionId=3
PartitionId=2
PartitionId = 1
PartitionId = 2
PartitionId = 3
Event HubSELECT COUNT(*) AS Count, TollBoothId
FROM EntryStream Partition By PartitionId
GROUP BY TumblingWindow (minute, 3),
TollBoothId
41
• Preview services
• Offers ability to deal with new age problem in processing and
analyzing data
• Scale, Speed, Economy
ADF & ASA
Recommended/related sessions
Inside Azure Storage – Options, abstractions and Best Practices
Data, Sabha2, 11.00 AM – 11.55 AM tomorrow
1
Choosing Right platform for BigData
Data, Sabha2, 3.00 PM to 3.55 PM tomorrow
2
Practical Machine Learning
Data, Sabha2 , 4.15 to 5.10 Today
3
References
Related references for you to expand your knowledge on the subject
Azure Stream Analytics Documentation
http://azure.microsoft.com/en-
in/documentation/services/stream-analytics/
Stream Analytics Query Language Reference
https://msdn.microsoft.com/en-
us/library/azure/dn834998.aspx
Azure Portal
http://azure.microsoft.com
Azure Updates
http://azure.microsoft.com/blog/
Microsoft Virtual Academy
aka.ms/mva
Developer Network
msdn.microsoft.com/
Azure Support
Must know resources to get online help for Azure.
Azure Support Options
http://azure.microsoft.com/en-
us/support/options/
Azure Support Plans
http://azure.microsoft.com/en-
us/support/plans/
Ask questions, & get answers
Post questions
in the Azure
forums
Tag questions
with the keyword
Azure.
Azure Vidyapeeth
A platform for learning – Choose your topic, choose your time
• Register to attend Azure Vidyapeeth Live webinars @
www.aka.ms/azure-vidyapeeth
• Collect free $100 Azure gift pass by registering for our Azure Vidyapeeth series at the Expo zone!
• Point your mobile phone here to download the Azure Vidyapeeth Mobile App :
www.aka.ms/av-app
Tell us what you think
Help us shape future events by
sharing your valuable feedback.
Scan the QR code to evaluate
this session.
< QR Code will be given 2 days before
the Conference >
Thank you
Twitter: @deananth
@infinitydlimit
Follow us online
Pricing (Today)
You write declarative queries in SQL
No code compilation, easy to author and deploy
Unified programming model
Brings together event streams, reference data and
machine learning extensions
Temporal Semantics
All operators respect, and some use, the temporal
properties of events
Built-in operators and functions
These should (mostly) look familiar if you know
relational databases
Filters, projections, joins, windowed (temporal)
aggregates, text and date manipulation
50
Why Event Processing in the Cloud?
Event data is already
in the Cloud
Event data is
globally distributed
Reduced TCO Scale Managed service,
not infrastructure
Bring the processing to the data,
not the data to the processing!
Application Components
Components of an Azure Stream Analytics Application
AzureSQLDB
AzureEvent Hubs
AzureBlobStorageAzureBlob Storage
AzureEvent Hubs
ReferenceData
Queryrunscontinuouslyagainstincomingstreamofevents
Events
Have a defined schema and
are temporal (sequenced in
time)

Weitere ähnliche Inhalte

Was ist angesagt?

DBP-010_Using Azure Data Services for Modern Data Applications
DBP-010_Using Azure Data Services for Modern Data ApplicationsDBP-010_Using Azure Data Services for Modern Data Applications
DBP-010_Using Azure Data Services for Modern Data Applicationsdecode2016
 
Analyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeAnalyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeBizTalk360
 
Real time big data stream processing
Real time big data stream processing Real time big data stream processing
Real time big data stream processing Luay AL-Assadi
 
5 Comparing Microsoft Big Data Technologies for Analytics
5 Comparing Microsoft Big Data Technologies for Analytics5 Comparing Microsoft Big Data Technologies for Analytics
5 Comparing Microsoft Big Data Technologies for AnalyticsJen Stirrup
 
Part 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure SynapsePart 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure SynapseNilesh Gule
 
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLiftThe Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLiftRui Quintino
 
Accessing Google Cloud APIs
Accessing Google Cloud APIsAccessing Google Cloud APIs
Accessing Google Cloud APIswesley chun
 
Overview on Azure Machine Learning
Overview on Azure Machine LearningOverview on Azure Machine Learning
Overview on Azure Machine LearningJames Serra
 
Microsoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMicrosoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMark Kromer
 
Is there a way that we can build our Azure Synapse Pipelines all with paramet...
Is there a way that we can build our Azure Synapse Pipelines all with paramet...Is there a way that we can build our Azure Synapse Pipelines all with paramet...
Is there a way that we can build our Azure Synapse Pipelines all with paramet...Erwin de Kreuk
 
1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for releaseJen Stirrup
 
Real Time Power BI
Real Time Power BIReal Time Power BI
Real Time Power BIDavide Mauri
 
BTUG - Dec 2014 - Hybrid Connectivity Options
BTUG - Dec 2014 - Hybrid Connectivity OptionsBTUG - Dec 2014 - Hybrid Connectivity Options
BTUG - Dec 2014 - Hybrid Connectivity OptionsMichael Stephenson
 
Real-Time Event & Stream Processing on MS Azure
Real-Time Event & Stream Processing on MS AzureReal-Time Event & Stream Processing on MS Azure
Real-Time Event & Stream Processing on MS AzureKhalid Salama
 
Databricks: A Tool That Empowers You To Do More With Data
Databricks: A Tool That Empowers You To Do More With DataDatabricks: A Tool That Empowers You To Do More With Data
Databricks: A Tool That Empowers You To Do More With DataDatabricks
 
Big Data Adavnced Analytics on Microsoft Azure
Big Data Adavnced Analytics on Microsoft AzureBig Data Adavnced Analytics on Microsoft Azure
Big Data Adavnced Analytics on Microsoft AzureMark Tabladillo
 
Big Data on azure
Big Data on azureBig Data on azure
Big Data on azureDavid Giard
 

Was ist angesagt? (20)

DBP-010_Using Azure Data Services for Modern Data Applications
DBP-010_Using Azure Data Services for Modern Data ApplicationsDBP-010_Using Azure Data Services for Modern Data Applications
DBP-010_Using Azure Data Services for Modern Data Applications
 
Analyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeAnalyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data Lake
 
Real time big data stream processing
Real time big data stream processing Real time big data stream processing
Real time big data stream processing
 
Modern data warehouse
Modern data warehouseModern data warehouse
Modern data warehouse
 
5 Comparing Microsoft Big Data Technologies for Analytics
5 Comparing Microsoft Big Data Technologies for Analytics5 Comparing Microsoft Big Data Technologies for Analytics
5 Comparing Microsoft Big Data Technologies for Analytics
 
Super charged prototyping
Super charged prototypingSuper charged prototyping
Super charged prototyping
 
Part 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure SynapsePart 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure Synapse
 
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLiftThe Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift
The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift
 
Yahoo's Next Generation User Profile Platform
Yahoo's Next Generation User Profile PlatformYahoo's Next Generation User Profile Platform
Yahoo's Next Generation User Profile Platform
 
Accessing Google Cloud APIs
Accessing Google Cloud APIsAccessing Google Cloud APIs
Accessing Google Cloud APIs
 
Overview on Azure Machine Learning
Overview on Azure Machine LearningOverview on Azure Machine Learning
Overview on Azure Machine Learning
 
Microsoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMicrosoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the Cloud
 
Is there a way that we can build our Azure Synapse Pipelines all with paramet...
Is there a way that we can build our Azure Synapse Pipelines all with paramet...Is there a way that we can build our Azure Synapse Pipelines all with paramet...
Is there a way that we can build our Azure Synapse Pipelines all with paramet...
 
1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release
 
Real Time Power BI
Real Time Power BIReal Time Power BI
Real Time Power BI
 
BTUG - Dec 2014 - Hybrid Connectivity Options
BTUG - Dec 2014 - Hybrid Connectivity OptionsBTUG - Dec 2014 - Hybrid Connectivity Options
BTUG - Dec 2014 - Hybrid Connectivity Options
 
Real-Time Event & Stream Processing on MS Azure
Real-Time Event & Stream Processing on MS AzureReal-Time Event & Stream Processing on MS Azure
Real-Time Event & Stream Processing on MS Azure
 
Databricks: A Tool That Empowers You To Do More With Data
Databricks: A Tool That Empowers You To Do More With DataDatabricks: A Tool That Empowers You To Do More With Data
Databricks: A Tool That Empowers You To Do More With Data
 
Big Data Adavnced Analytics on Microsoft Azure
Big Data Adavnced Analytics on Microsoft AzureBig Data Adavnced Analytics on Microsoft Azure
Big Data Adavnced Analytics on Microsoft Azure
 
Big Data on azure
Big Data on azureBig Data on azure
Big Data on azure
 

Andere mochten auch

Big data streaming with Apache Spark on Azure
Big data streaming with Apache Spark on AzureBig data streaming with Apache Spark on Azure
Big data streaming with Apache Spark on AzureWillem Meints
 
Azure api app métricas com application insights
Azure api app métricas com application insightsAzure api app métricas com application insights
Azure api app métricas com application insightsNicolas Takashi
 
Microsoft NYC 14
Microsoft NYC 14Microsoft NYC 14
Microsoft NYC 14SwitchPitch
 
Enterprise Data Workflows with Cascading and Windows Azure HDInsight
Enterprise Data Workflows with Cascading and Windows Azure HDInsightEnterprise Data Workflows with Cascading and Windows Azure HDInsight
Enterprise Data Workflows with Cascading and Windows Azure HDInsightPaco Nathan
 
Fraud Detection using Hadoop
Fraud Detection using HadoopFraud Detection using Hadoop
Fraud Detection using Hadoophadooparchbook
 
Go Serverless with Azure Functions
Go Serverless with Azure FunctionsGo Serverless with Azure Functions
Go Serverless with Azure FunctionsJim O'Neil
 
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...Mike Martin
 
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)Sascha Dittmann
 
Going serverless
Going serverlessGoing serverless
Going serverlessTechExeter
 
2016-08-25 TechExeter - going serverless with Azure
2016-08-25 TechExeter - going serverless with Azure2016-08-25 TechExeter - going serverless with Azure
2016-08-25 TechExeter - going serverless with AzureSteve Lee
 
Azure functions
Azure functionsAzure functions
Azure functionsvivek p s
 
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloud
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloudAzure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloud
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloudToradex
 
Open up to a better learning ecosystem
Open up to a better learning ecosystemOpen up to a better learning ecosystem
Open up to a better learning ecosystemKatie Bradford
 
Spark on Azure HDInsight - spark meetup seattle
Spark on Azure HDInsight - spark meetup seattleSpark on Azure HDInsight - spark meetup seattle
Spark on Azure HDInsight - spark meetup seattleJudy Nash
 
Microsoft Azure For Solutions Architects
Microsoft Azure For Solutions ArchitectsMicrosoft Azure For Solutions Architects
Microsoft Azure For Solutions ArchitectsRoy Kim
 
Building big data solutions on azure
Building big data solutions on azureBuilding big data solutions on azure
Building big data solutions on azureEyal Ben Ivri
 

Andere mochten auch (20)

Big data streaming with Apache Spark on Azure
Big data streaming with Apache Spark on AzureBig data streaming with Apache Spark on Azure
Big data streaming with Apache Spark on Azure
 
Azure api app métricas com application insights
Azure api app métricas com application insightsAzure api app métricas com application insights
Azure api app métricas com application insights
 
Microsoft NYC 14
Microsoft NYC 14Microsoft NYC 14
Microsoft NYC 14
 
Enterprise Data Workflows with Cascading and Windows Azure HDInsight
Enterprise Data Workflows with Cascading and Windows Azure HDInsightEnterprise Data Workflows with Cascading and Windows Azure HDInsight
Enterprise Data Workflows with Cascading and Windows Azure HDInsight
 
Fraud Detection using Hadoop
Fraud Detection using HadoopFraud Detection using Hadoop
Fraud Detection using Hadoop
 
Go Serverless with Azure Functions
Go Serverless with Azure FunctionsGo Serverless with Azure Functions
Go Serverless with Azure Functions
 
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...
Belgian Windows Server 2012 Launch windows azure insights for the enterprise ...
 
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)
SQLSaturday #230 - Introduction to Microsoft Big Data (Part 1)
 
Azure IOT
Azure IOTAzure IOT
Azure IOT
 
Going serverless
Going serverlessGoing serverless
Going serverless
 
Azure HDInsight
Azure HDInsightAzure HDInsight
Azure HDInsight
 
Software scope
Software scopeSoftware scope
Software scope
 
2016-08-25 TechExeter - going serverless with Azure
2016-08-25 TechExeter - going serverless with Azure2016-08-25 TechExeter - going serverless with Azure
2016-08-25 TechExeter - going serverless with Azure
 
Azure functions
Azure functionsAzure functions
Azure functions
 
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloud
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloudAzure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloud
Azure IoT Hub on a Toradex Colibri VF61 – Part 1 - Sending data to the cloud
 
Open up to a better learning ecosystem
Open up to a better learning ecosystemOpen up to a better learning ecosystem
Open up to a better learning ecosystem
 
Spark on Azure HDInsight - spark meetup seattle
Spark on Azure HDInsight - spark meetup seattleSpark on Azure HDInsight - spark meetup seattle
Spark on Azure HDInsight - spark meetup seattle
 
Microsoft Azure For Solutions Architects
Microsoft Azure For Solutions ArchitectsMicrosoft Azure For Solutions Architects
Microsoft Azure For Solutions Architects
 
Building big data solutions on azure
Building big data solutions on azureBuilding big data solutions on azure
Building big data solutions on azure
 
Going serverless
Going serverlessGoing serverless
Going serverless
 

Ähnlich wie Azure Stream Analytics : Analyse Data in Motion

Big Data Analytics in the Cloud with Microsoft Azure
Big Data Analytics in the Cloud with Microsoft AzureBig Data Analytics in the Cloud with Microsoft Azure
Big Data Analytics in the Cloud with Microsoft AzureMark Kromer
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMark Kromer
 
Building IoT and Big Data Solutions on Azure
Building IoT and Big Data Solutions on AzureBuilding IoT and Big Data Solutions on Azure
Building IoT and Big Data Solutions on AzureIdo Flatow
 
Building Custom Big Data Integrations
Building Custom Big Data IntegrationsBuilding Custom Big Data Integrations
Building Custom Big Data IntegrationsPat Patterson
 
WSO2 Analytics Platform - The one stop shop for all your data needs
WSO2 Analytics Platform - The one stop shop for all your data needsWSO2 Analytics Platform - The one stop shop for all your data needs
WSO2 Analytics Platform - The one stop shop for all your data needsSriskandarajah Suhothayan
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming VisualizationGuido Schmutz
 
Serverless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsServerless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsAmazon Web Services
 
[WSO2Con USA 2018] Patterns for Building Streaming Apps
[WSO2Con USA 2018] Patterns for Building Streaming Apps[WSO2Con USA 2018] Patterns for Building Streaming Apps
[WSO2Con USA 2018] Patterns for Building Streaming AppsWSO2
 
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft Private Cloud
 
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...Riccardo Zamana
 
Les objets connectés : de nombreux cas d'usage
Les objets connectés : de nombreux cas d'usage Les objets connectés : de nombreux cas d'usage
Les objets connectés : de nombreux cas d'usage Jedha Bootcamp
 
Getting Started with Real-time Analytics
Getting Started with Real-time AnalyticsGetting Started with Real-time Analytics
Getting Started with Real-time AnalyticsAmazon Web Services
 
High-performance database technology for rock-solid IoT solutions
High-performance database technology for rock-solid IoT solutionsHigh-performance database technology for rock-solid IoT solutions
High-performance database technology for rock-solid IoT solutionsClusterpoint
 
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...Amazon Web Services
 
Introduction to Azure DocumentDB
Introduction to Azure DocumentDBIntroduction to Azure DocumentDB
Introduction to Azure DocumentDBDenny Lee
 

Ähnlich wie Azure Stream Analytics : Analyse Data in Motion (20)

Big Data Analytics in the Cloud with Microsoft Azure
Big Data Analytics in the Cloud with Microsoft AzureBig Data Analytics in the Cloud with Microsoft Azure
Big Data Analytics in the Cloud with Microsoft Azure
 
Microsoft Azure Big Data Analytics
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data Analytics
 
Building IoT and Big Data Solutions on Azure
Building IoT and Big Data Solutions on AzureBuilding IoT and Big Data Solutions on Azure
Building IoT and Big Data Solutions on Azure
 
Building Custom Big Data Integrations
Building Custom Big Data IntegrationsBuilding Custom Big Data Integrations
Building Custom Big Data Integrations
 
Implementing Real-Time IoT Stream Processing in Azure
Implementing Real-Time IoT Stream Processing in Azure Implementing Real-Time IoT Stream Processing in Azure
Implementing Real-Time IoT Stream Processing in Azure
 
WSO2 Analytics Platform - The one stop shop for all your data needs
WSO2 Analytics Platform - The one stop shop for all your data needsWSO2 Analytics Platform - The one stop shop for all your data needs
WSO2 Analytics Platform - The one stop shop for all your data needs
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
 
Serverless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsServerless Streaming Data Processing using Amazon Kinesis Analytics
Serverless Streaming Data Processing using Amazon Kinesis Analytics
 
WebAction-Sami Abkay
WebAction-Sami AbkayWebAction-Sami Abkay
WebAction-Sami Abkay
 
[WSO2Con USA 2018] Patterns for Building Streaming Apps
[WSO2Con USA 2018] Patterns for Building Streaming Apps[WSO2Con USA 2018] Patterns for Building Streaming Apps
[WSO2Con USA 2018] Patterns for Building Streaming Apps
 
Serverless SQL
Serverless SQLServerless SQL
Serverless SQL
 
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
 
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...
 
Les objets connectés : de nombreux cas d'usage
Les objets connectés : de nombreux cas d'usage Les objets connectés : de nombreux cas d'usage
Les objets connectés : de nombreux cas d'usage
 
Getting Started with Real-time Analytics
Getting Started with Real-time AnalyticsGetting Started with Real-time Analytics
Getting Started with Real-time Analytics
 
Patterns for Building Streaming Apps
Patterns for Building Streaming AppsPatterns for Building Streaming Apps
Patterns for Building Streaming Apps
 
High-performance database technology for rock-solid IoT solutions
High-performance database technology for rock-solid IoT solutionsHigh-performance database technology for rock-solid IoT solutions
High-performance database technology for rock-solid IoT solutions
 
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...
Emerging Prevalence of Data Streaming in Analytics and it's Business Signific...
 
Analytics&IoT
Analytics&IoTAnalytics&IoT
Analytics&IoT
 
Introduction to Azure DocumentDB
Introduction to Azure DocumentDBIntroduction to Azure DocumentDB
Introduction to Azure DocumentDB
 

Kürzlich hochgeladen

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 

Kürzlich hochgeladen (20)

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 

Azure Stream Analytics : Analyse Data in Motion

  • 2. Deepthi Anantharam Technology Evangelist @deananth Ruhani Arora Technology Evangelist @infinitydlimit
  • 3. The need for evolution – Identified 2 years ago … data warehousing has reached the most significant tipping point since its inception. The biggest, possibly most elaborate data management system in IT is changing. – Gartner, “The State of Data Warehousing in 2012” Data sources
  • 4. The “Traditional” Data Warehouse 4 Data sources Increasing data volumes 1 Real-time data 4 Non-Relational Data New data sources & types 2 Cloud-born data 3
  • 5. Evolving Approaches to Analytics ETL Tool (SSIS, etc) EDW (SQL Svr, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Data Marts Data Lake(s) Dashboards Apps
  • 6. ETL Tool (SSIS, etc) EDW (SQL Svr, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Ingest (EL) Original Data Data Marts Data Lake(s) Dashboards Apps Evolving Approaches to Analytics
  • 7. ETL Tool (SSIS, etc) EDW (SQL Svr, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Ingest (EL) Original Data Scale-out Storage & Compute (HDFS, Blob Storage, etc) Transform & Load Data Marts Data Lake(s) Dashboards Apps Streaming data Evolving Approaches to Analytics
  • 8. ETL Tool (SSIS, etc) EDW (SQL Svr, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Ingest (EL) Original Data Scale-out Storage & Compute (HDFS, Blob Storage, etc) Transform & Load Data Marts Data Lake(s) Dashboards Apps Streaming data Evolving Approaches to Analytics Real Time data analytics
  • 9. Agenda • Azure Data Factory • Azure Stream Analytics
  • 10. Azure Data Factory Overview • New Azure service for data developers & IT • Compose data processing, storage and movement services to create & manage analytics pipelines • Initially focused on Azure & hybrid movement to/from on premises SQL Server. Overtime will expand to more storage & processing systems throughout • Rich, simple end-to-end pipeline monitoring and management
  • 12. Example Scenario: Customer Profiling (game usage analytics)
  • 13. Customer Profiling – Game Usage Analytics 2277,2013-06-01 02:26:54.3943450,111,164.234.187.32,24.84.225.233,true,8,1,2058 2277,2013-06-01 03:26:23.2240000,111,164.234.187.32,24.84.225.233,true,8,1,2058-2123-2009-2068-2166 2277,2013-06-01 04:22:39.4940000,111,164.234.187.32,24.84.225.233,true,8,1, 2277,2013-06-01 05:43:54.1240000,111,164.234.187.32,24.84.225.233,true,8,1,2058-225545-2309-2068-2166 2277,2013-06-01 06:11:23.9274300,111,164.234.187.32,24.84.225.233,true,8,1,223-2123-2009-4229-9936623 2277,2013-06-01 07:37:01.3962500,111,164.234.187.32,24.84.225.233,true,8,1, 2277,2013-06-01 08:12:03.1109790,111,164.234.187.32,24.84.225.233,true,8,1,234322-2123-2234234-12432-344323 … Log Files Snippet (10s of TBs per day in cloud storage) User Table UserID FirstName LastName State … 2277 Pratik Patel Oregon 664432 Dave Nettleton Washington 8853 Mike Flasko California New User Activity Per Week By Region profileid day state duration rank weaponsused interactedwith 1148 6/2/2013Oregon 216 33 1 5 1004 6/2/2013Missouri 22 40 6 2 292 6/1/2013Georgia 201 137 1 5 1059 6/2/2013Oregon 27 104 5 2 675 6/2/2013California 65 164 3 2 1348 6/3/2013Nebraska 21 95 5 2
  • 14. Terminologies • Linked Services • Data Sets • Pipeline • Diagram View • Create a Data factory • Add Data Sources • Define Tables and Pipelines • Deploy & Start • Monitor and Manage Steps
  • 15. Example: Game Logs, Customer Profiling On Premises SQL Server Azure Blob Storage 1000’s Log FilesNew User View Azure Data Factory
  • 16. Example: Game Logs, Customer Profiling On Premises SQL Server Azure Blob Storage 1000’s Log FilesNew User View Azure Data FactoryViewOf Game Usage ViewOf New Users New User Activity
  • 17. Example: Game Logs, Customer Profiling ViewOf On Premises SQL Server Azure Blob Storage 1000’s Log FilesNew User View Copy “NewUsers” to Blob Storage Cloud New Users Azure Data FactoryViewOf Game Usage ViewOf New Users New User Activity Pipeline
  • 18. Example: Game Logs, Customer Profiling On Premises SQL Server Azure Blob Storage 1000’s Log FilesNew User View Copy NewUsers to Blob Storage Cloud New Users Azure Data FactoryViewOf Game Usage ViewOf Mask & Geo- Code New Users Geo Dictionary Geo Coded Game Usage HDInsight New User Activity Pipeline Pipeline
  • 19. Example: Game Logs, Customer Profiling On Premises SQL Server Azure Blob Storage 1000’s Log FilesNew User View Copy NewUsers to Blob Storage Cloud New Users Azure Data FactoryViewOf Game Usage ViewOf RunsOn Mask & Geo- Code New Users Geo Dictionary Geo Coded Game Usage Join & Aggregate HDInsight New User Activity ViewOf Pipeline Pipeline Pipeline
  • 20. “GeoCoded Game Usage” Table: Step 3: Define Tables & Pipelines
  • 21. Pipeline Definition: Step 3: Define Tables & Pipelines
  • 22. Powershell // Deploy Table New-AzureDataFactoryTable -DataFactory“GameTelemetry“ -File NewUserActivityPerRegion.json // Deploy Pipeline New-AzureDataFactoryPipeline -DataFactory “GameTelemetry“ -File NewUserTelemetryPipeline.json // Start Pipeline Set-AzureDataFactoryPipelineActivePeriod -Name “NewUserTelemetryPipeline“ -DataFactory “GameTelemetry“ -StartTime 10/29/2014 12:00:00
  • 24. Custom Actions • Allows running any .NET code wrapped within an ADF activity • Can be used to connect to new sources/destination • Can be used to create custom transformation activities • Example: Invoke Azure ML model • SDK for custom activity creation:
  • 25. Coordination: • Rich scheduling • Complex dependencies • Incremental rerun Authoring: • JSON & Powershell/C# Management: • Lineage • Data production policies (late data, rerun, latency, etc) Hub: Azure Hub (HDInsight + Blob storage) • Activities: Hive, Pig, C# • Data Connectors: Blobs, Tables, Azure DB, On Prem SQL Server, MDS [internal] Data Factory – Available Today
  • 26. Analyze your data in motion
  • 27. What is Streaming Data? Data in MotionData at Rest
  • 28. Azure Stream Analytics Managed real-time analytics Mission-critical reliability and scale Rapid development Point of Service Devices Self Checkout Stations Kiosks Smart Phones Slates/ Tablets PCs/ Laptops Servers Digital Signs Diagnostic EquipmentRemote Medical Monitors Logic Controllers Specialized DevicesThin Clients Handhelds Security POS Terminals Automation Devices Vending Machines Kinect ATM
  • 29. How do customers create a real-time streaming solution?
  • 31. Using Azure Analytic Service Data Source Collect Process Consume Deliver Event Inputs - Event Hub - Azure Blob Transform - Temporal joins - Filter - Aggregates - Projections - Windows - Etc. Enrich Correlate Outputs - SQL Azure - Azure Blobs - Event Hub - Table Storage Azure Storage Azure Stream Analytics Reference Data - Azure Blob
  • 32. Sample Scenario : Toll Station TollId EntryTime License Plate State Make Model Type Weight 1 2014-10-25T19:33:30.0000000Z JNB7001 NY Honda CRV 1 3010 1 2014-10-25T19:33:31.0000000Z YXZ1001 NY Toyota Camry 2 3020 3 2014-10-25T19:33:32.0000000Z ABC1004 CT Ford Taurus 2 3800 2 2014-10-25T19:33:33.0000000Z XYZ1003 CT Toyota Corolla 2 2900 1 2014-10-25T19:33:34.0000000Z BNJ1007 NY Honda CRV 1 3400 2 2014-10-25T19:33:35.0000000Z CDE1007 NJ Toyota 4x4 1 3800 … … … … … … … … EntryStream - Data about vehicles entering toll stations TollId ExitTime LicensePlate 1 2014-10-25T19:33:40.0000000Z JNB7001 1 2014-10-25T19:33:41.0000000Z YXZ1001 3 2014-10-25T19:33:42.0000000Z ABC1004 2 2014-10-25T19:33:43.0000000Z XYZ1003 … … … ExitStream - Data about cars leaving toll stations LicensePlate RegistartionId Expired SVT6023 285429838 1 XLZ3463 362715656 0 QMZ1273 876133137 1 RIV8632 992711956 0 … … …. ReferenceData - Commercial vehicle registration data
  • 33. Query Language - Overview DML Statements • SELECT • FROM • WHERE • GROUP BY • HAVING • CASE • JOINS • UNION Scaling Functions • WITH • PARTITION BY Date and Time Functions • DATENAME • DATEPART • DAY • MONTH • YEAR • DATETIMEFROMPARTS • DATEDIFF • DATADD Windowing Extensions • Tumbling Window • Hopping Window • Sliding Window Aggregate Functions • SUM • COUNT • AVG • MIN • MAX String Functions • LEN CONCAT • SUBSTRING • CHARINDEX • PATINDEX
  • 34. Tumbling Windows SELECT TollId, COUNT(*) FROM EntryStream TIMESTAMP BY EntryTime GROUP BY TollId, TumblingWindow(second, 10) Count the total number of vehicles entering each toll booth every interval of 10 seconds. 1 5 4 26 8 6 5 0 5 2010 15 Time (secs) 1 5 4 26 8 6 25 A 10-second Tumbling Window 30 3 6 1 5 3 6 1
  • 35. Hopping Windows SELECT COUNT(*), TollId FROM EntryStream TIMESTAMP BY EntryTime GROUP BY TollId, HoppingWindow (second, 10,5) Count the number of vehicles entering each toll booth every interval of 10 seconds; update results every 10 seconds 1 5 4 26 8 7 0 5 2010 15 Time (secs) 25 A 10-second Hopping Window with a 5-second “Hop” 30 4 26 8 6 5 3 6 1 1 5 4 26 8 6 5 3 6 15 3
  • 36. Sliding Windows Give me the count of all the toll booths which have served more than 10 vehicles in the last 10 seconds 1 5 0 5 2010 15 Time (secs) 25 A 10-second Sliding Window 8 8 51 9 51 9 1 SELECT TollId, Count(*) FROM EntryStream ES GROUP BY TollId, SlidingWindow (second, 10) HAVING Count(*) > 10
  • 37. Intake millions of events per second Process data from connected devices/apps Integrated with highly-scalable publish-subscriber ingestor Easy processing on continuous streams of data Transform, augment, correlate, temporal operations Detect patterns and anomalies in streaming data Correlate streaming with reference data
  • 38. Input and Output Management Transformations Management Programmatic Access with REST APIs Jobs Management Start Job Stop Job Create Job Delete Job List Jobs Update Job Create Input / Output Delete Input / Output List Input / Output Update Input / Output Create Transformation Delete Transformation Get Transformation Update Transformation The full functionality of Azure Stream Analytics is through REST APIs. Enables programmatic access Useful for automation through scripting Embed in other applications/tools
  • 39. Demo: Scaling , Monitoring & Logging
  • 40. Scaling Concepts – Partitions Step Result1 Step Result2 Step Result3 PartitionId=1 PartitionId=3 PartitionId=2 PartitionId = 1 PartitionId = 2 PartitionId = 3 Event HubSELECT COUNT(*) AS Count, TollBoothId FROM EntryStream Partition By PartitionId GROUP BY TumblingWindow (minute, 3), TollBoothId
  • 41. 41 • Preview services • Offers ability to deal with new age problem in processing and analyzing data • Scale, Speed, Economy ADF & ASA
  • 42. Recommended/related sessions Inside Azure Storage – Options, abstractions and Best Practices Data, Sabha2, 11.00 AM – 11.55 AM tomorrow 1 Choosing Right platform for BigData Data, Sabha2, 3.00 PM to 3.55 PM tomorrow 2 Practical Machine Learning Data, Sabha2 , 4.15 to 5.10 Today 3
  • 43. References Related references for you to expand your knowledge on the subject Azure Stream Analytics Documentation http://azure.microsoft.com/en- in/documentation/services/stream-analytics/ Stream Analytics Query Language Reference https://msdn.microsoft.com/en- us/library/azure/dn834998.aspx Azure Portal http://azure.microsoft.com Azure Updates http://azure.microsoft.com/blog/ Microsoft Virtual Academy aka.ms/mva Developer Network msdn.microsoft.com/
  • 44. Azure Support Must know resources to get online help for Azure. Azure Support Options http://azure.microsoft.com/en- us/support/options/ Azure Support Plans http://azure.microsoft.com/en- us/support/plans/ Ask questions, & get answers Post questions in the Azure forums Tag questions with the keyword Azure.
  • 45. Azure Vidyapeeth A platform for learning – Choose your topic, choose your time • Register to attend Azure Vidyapeeth Live webinars @ www.aka.ms/azure-vidyapeeth • Collect free $100 Azure gift pass by registering for our Azure Vidyapeeth series at the Expo zone! • Point your mobile phone here to download the Azure Vidyapeeth Mobile App : www.aka.ms/av-app
  • 46. Tell us what you think Help us shape future events by sharing your valuable feedback. Scan the QR code to evaluate this session. < QR Code will be given 2 days before the Conference >
  • 49. You write declarative queries in SQL No code compilation, easy to author and deploy Unified programming model Brings together event streams, reference data and machine learning extensions Temporal Semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions These should (mostly) look familiar if you know relational databases Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation
  • 50. 50 Why Event Processing in the Cloud? Event data is already in the Cloud Event data is globally distributed Reduced TCO Scale Managed service, not infrastructure Bring the processing to the data, not the data to the processing!
  • 51. Application Components Components of an Azure Stream Analytics Application AzureSQLDB AzureEvent Hubs AzureBlobStorageAzureBlob Storage AzureEvent Hubs ReferenceData Queryrunscontinuouslyagainstincomingstreamofevents Events Have a defined schema and are temporal (sequenced in time)

Hinweis der Redaktion

  1. Let us start with a statement that was made 2 years ago. Gartner stated that DW has reached a significant tipping point since its inception. The biggest, possibly the most elaborate data management system in IT is changing DW is not going anywhere, just that there are more tools for developers such as Hadoop and NoSQL
  2. Each pipeline has an activity that is depicted by the blue box
  3. .
  4. Economy – Yes, the services are spoken of in mills/sec etc Go to pricing portal
  5. Stream Analytics is priced on two variables: Volume of data processed Streaming units required to process the data stream
  6. Now let us dig deeper into what a typical ASA application looks like. An ASA application has three major components: Input – Inputs are the sources of events. Note that the ‘original’ source of streaming events are devices, machines, applications, sensors, applications etc. However, ASA is not intended to connect to them directly. Rather ASA lets Azure Event Hubs be the primary interface to the wide variety of event sources. ASA is optimized to get streaming data from Azure Event Hubs and Azure Blob Storage. Azure Blob Storage is the likely place where log data is stored. The list of input sources that ASA directly integrates with may increase in the future, but Azure Event Hubs and Azure Blob Storage will be the primary sources. Query – Queries are the main component of an ASA application. Queries implement the “analytics logic”. Queries are a set of transformations that are applied to the input stream to produce another set of output events. Queries are the only thing that an ASA application developers actually ‘develops’. Everything else is done through guided wizards in the Azure Portal. Note that ASA has a SQL-like query language but unlike traditional databases, ASA queries run continuously against the stream of incoming events. The queries stop being applied only when the job itself stops. Output – As queries execute they continuously produce results. The results can be stored in Blob Storage, Event Hubs or Azure SQL database. Note that if the output is stored in Event Hub or Blob Storage, it can become the input to another ASA job. So it is possible to ‘chain’ together multiple jobs to implement a series of transformations.