Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings. If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.

1. Sponsored & Brought to you by
Analyzing StackExchange data with Azure
Data Lake
Tom Kerkhove
http://www.twitter.com/TomKerkhove
https://be.linkedin.com/in/tomkerkhove

2. Analysing StackExchange data
with Azure Data Lake
Analysing StackExchange data with Azure Data Lake

3. Nice to meet you
Tom KERKHOVE
➔ Integration Professional
➔ IoT Competency Lead
➔ Windows Development &
Microsoft Azure MVP
tom.kerkhove@codit.eu
+32 473 701 074
@TomKerkhove
be.linkedin.com/in/tomkerkhove
github.com/tomkerkhove

4. Agenda
• Why should we care about Big Data?
• Big Data in Azure
• Azure Data Lake
• Demo
• Q & A
4

6. Integration of ThingsInternet of Things
6

7. Connect and scale
with efficiency
Analyze and act
on new data
Integrate and transform
business processes

8. Event producers & gateways Ingestion & transformation Report, Act, Predict

9. Microsoft Patterns & Practices – IoT Journey

10. 10

11. 11
Cluster Management

12. 12
Languages

13. Platform Services
Infrastructure Services
Web Apps
Mobile
Apps
API
Management
API Apps
Logic Apps
Notification
Hubs
Content
Delivery
Network (CDN)
Media
Services
BizTalk
Services
Hybrid
Connections
Service Bus
Storage
Queues
Hybrid
Operations
Backup
StorSimple
Azure Site
Recovery
Import/Export
SQL
Database
DocumentDB
Redis
Cache
Azure
Search
Storage
Tables
Data
Warehouse Azure AD
Health Monitoring
AD Privileged
Identity
Management
Operational
Analytics
Cloud
Services
Batch
RemoteApp
Service
Fabric
Visual Studio
App
Insights
Azure
SDK
VS Online
Domain Services
HDInsight Machine
Learning
Stream
Analytics
Data
Factory
Event
Hubs
Mobile
Engagement
Data
Lake
IoT Hub
Data
Catalog
Security &
Management
Azure Active
Directory
Multi-Factor
Authentication
Automation
Portal
Key Vault
Store/
Marketplace
VM Image Gallery
& VM Depot
Azure AD
B2C
Scheduler

14. Overview in Azure
14
DocumentDB
Data Factory Stream Analytics Data Lake HDInsight Data Lake
(Store & Analytics)
Virtual Machine
IoT Hub SQL Data
Warehouse
SQL DatabaseStorageEvent Hubs
Document Db
Data Ingestion Data Storage
Data Pipelines
Machine Learning
Data Analytics

15. Cortana Analytics Suite

16. 16

17. Analysing Big Data in Azure
Azure Data Lake Family
HDInsight Data Lake Store Data Lake Analytics
• Unlimited storage
• WebHDFS Store
• Managed cluster service
• Open-source technology
• Runs on Windows or Linux
• Managed job service
• U-SQL batch-processing

18. Azure Data Lake Store
➔ WebHDFS compatible
➔ Any size
➔ Any format as-is
➔ Write-once-read-many
➔ Enterprise-grade security
➔ Thé big data store in Azure
18

19. Characteristics
➔ Data Warehousing
➔ Structured data
➔ Defined set of schemas
➔ Requires Extract-Transform-
Load (ETL) before storing
➔ Known for some of us
➔ Exploratory analysis is hard
because of transforming the
data
19
Data Lake vs DataWarehousing
➔ Data Lake
➔ Raw data
(unstructured/semi-structured/structured)
➔ “Dump” all your data in the
lake
➔ Data scientists will
interpret data from the lake
➔ Without metadata, turns in
a data swamp pretty fast

20. 20Martin Fowler on Data Lake & Data Warehouses(link)

21. Azure Data Lake Analytics
➔ Run analytics jobs on managed clusters
➔ Don’t worry about scale
➔ Written in U-SQL
➔ SQL Syntax
➔ Extensibility in C#
➔ Easily scaled with Analytics Units
➔ Pay for processing time only
21

22. Writing U-SQL scripts
22
Extract from data source by
using built-in or custom
extractors.
Transform / Analyse the data
using SQL-syntax, in-line C# or
C# method calls
Output the result to a data
source by using built-in or
custom extractors

23. 23

24. Data Lake Analytics - Data Sources
U-SQL
Query Query
Azure
Storage Blobs
Azure
Data Lake Store
Azure
SQL Database
Azure
SQL Data Warehouse
Azure SQL
in VMs
Azure Data Lake Analytics

25. 25

26. Meet StackExchange
➔ Over 280 subwebsites
➔ 150+ GB of open-source data
➔ Different kinds of data
➔ Posts
➔ Users
➔ Votes
➔ ...
➔ A big data sample data set

27. What AreWe GoingTo Do?
• Downloading the
original data set
Acquiring The
Data
• Upload data set to
Azure
• Determine what
service to use
Moving The
Data • Merging data from
each site into one
file
• Conversion from
XML to CSV
Aggregating
The Data
• Run business logic
on it
• Attempt to gain
knowledge from it
Analyzing The
Data • Visualize what we’ve
learned
Visualizing The
Data
27

28. Azure Data Lake tools forVisual Studio
➔ Projects / Solutions / Source control
➔ Store Explorer
➔ Browse store
➔ Download complete / subset of file
➔ Preview
➔ JobVisualizer
➔ Determine bottlenecks by using heatmaps
➔ Playback jobs based on telemetry
➔ Query optimization
➔ Job Profiler
➔ Off-Line execution
28

29. Integration with Azure Services
➔ Integrate in your data pipelines in Azure Data Factory
➔ Move data from Azure Data Lake Store to other store
➔ Move data to Azure Data Lake Store
➔ Run U-SQL query within pipeline
➔ Integration with Azure Data Catalog
➔ Register your Azure Data Lake Store assets
29

30. Pricing
➔ Data Lake Store
➔ $0,08/GB stored per month
➔ $0,14 per 1M transactions
• 1 transaction is block of up to 128 kB
➔ Egress will be billed but not know yet
➔ Data Lake Analytics
➔ $0,05 per job
➔ $0,05 per minute per Analytics Unit for processing time
30

31. Azure Data Lake Store vs Blob Storage
31
No Limitations
Store whatever you
want in any format
Security
Built-in Azure Active
Directory support
Pricing
More expensive than
Storage RA-GRS
Redundancy
It’s there but no control
over it
Built for Scale
Optimized for high-
scale reads
Integration
With Data Factory, Data
Catalog & HDInsight

32. 32

33. Summary
➔ Big Data is not just a hype so get ready
➔ Azure Data Lake Store
➔ Analyse today & explore tomorrow
➔ Data Swamps
➔ Data Lake Analytics
➔ No cluster management
➔ Re-use existing skills
➔ Pay for what we use
➔ Big Data in Azure? Azure Data Lake family and it’s easy!

35. 35

36. 36

37. 37