Slide deck by Nik Shahriar for the presentation topic Azure Data Factory, Azure Logic Apps at the C# Corner Toronto chapter Feb 2019 meetup

1. Presented by : Nik – Shahriar Nikkhah
Date : 2019-02-25 Toronto, Canada
Azure Data Factory And
Azure Logic Apps Design

2. • Microsoft Canada (MS-Mississauga)
Everyone at Microsoft
• C-Sharp Corner (Toronto)
Nilesh Shah (Toronto Chapter Lead)
Nik – Shahriar Nikkhah (Executive Member of Toronto Chapter)
Vivek Patel (Toronto Power BI Chapter Lead)
• SQL Data Side Inc.
Thank you…
And thank you to you all.

3. • Nik – Shahriar Nikkhah
Snr Data Engineer, Snr Azure Data Integration Lead/Design, Snr BI Consultant , Snr Technical
Team Lead
(IBM, LGS, SNC-Lavalin)
Microsoft Certified, MCSE, MCSA, MCP, MCITP, MCTS
Main focus: Big Data, Azure Data Factory, Azure Databrick, Azure Logic App, IoT Hub, Azure Stream Analytics,
Azure SQL Data Warehouse, Azure SQL Database, Azure Storage Account (Blob), Azure Data Lake Storage,
Azure Analysis Services, Power BI, SSIS, SQL Tabular DM, Azure PowerShell and On-Prem SQL Server
• SQLDataSide@gmail.com Tel: (647) 621-2242
• http://sqldataside.blogspot.ca/ https://www.linkedin.com/in/nnikkhah/
• https://www.c-sharpcorner.com/members/shahriar-nikkhah (Executive Member of Toronto Chapter)
About me
Please ask questions

4. • Nik - Shahriar Nikkhah
• Sr. Technical Team Lead
• Sr. BI Consultant
• C# Corner MVP
• Nilesh Shah
• Tech Lead, .NET & O365 dev.
• TOGAF 9.1 EA
• MS MVP, C# Corner MVP
• Vivek Patel
• Team Lead, Data & Analytics
• Microsoft Data Platform MVP
Toronto chapter – founder members
C# Corner

5. C# Corner Toronto Chapter
https://www.c-sharpcorner.com/

6. Toronto Power BI
https://www.meetup.com/Toronto-Power-BI-Meetup/

7. Agenda
We will be focusing on the above agenda
• What is Azure Data Factory (ADF)?
• What is Azure Logic Apps (ALA)?
• Unzip files in Logic Apps
• Looking from a different Angle.
• Typical Data Flow
• Unzip Files using ADF and ALA
• Mix and Match ADF ALA & ADB
• Unzip Demo
• Motivation slide

8. • Azure Subscription
• Azure Data Factory V2+
• Azure Storage Account (Blob Storage)
• Visual studio 2017+
• Azure tools for Visual Studio
• Microsoft Azure Storage Explorer
• Azure PowerShell
Requirements
https://azure.microsoft.com/en-ca/free/

9. • Cloud Data Integration Service
• Compose data storage, movement, …., processing service
into automated data pipelines
• ETL or ELT Tool in the cloud
• Azure Databricks Cluster
• Parallel processing
What is ADF?
Can we compare ADF with SSIS ?

10. • Cloud Data Integration Service
• Automate and orchestrate tasks, business process, Work
flows, When you need to integrate apps, data, systems,
and services across organisations.
• ETL or ELT Tool in the cloud
• No documentation found on Clustering for ALA
• Parallel processing
What is ALA?
Can we somehow ALA, ADF and SSIS?

11. Unzip files in Logic Apps
Do you see any limitation? and if so is ALA the right tool to unzip?

12. • Think as if you are working for google.
• Think big, think differently (Left join in google reports)
• Learn from scratch (How to create a table in ASDW in the
cloud vs creating a table in on-prem SQL Server)
• Profiling the data with different Tools (ADK/PBI).
• <= 2500 Pipelines, 200 ADF, 150 - 1000 ADB, 100
table/data dependency
• Replacing old features with new releases (Tool mature).
Looking From a Different Angle
How fast can you replace a old component with a new component(s)?
Considering the time & work needed (Dev, Test, QA, UAT, Deployment…)

13. Read files from a path
Filter out unwanted
files
Loop Files one by one
Typical Data Flow
Framework and prediction approach is critical in your design.
• 1 - GM_GetListOfFiles
• 2 - FL_SelectRequiredFilesOnly
• 3 - FE_LoopFiles
• Copy action is conducting in the loop

14. Unzip files using ADF and ALA
Two services with the same feature, what’s the difference/limitation?

15. Pipeline 2
Mix and Match ADF ALA & ADB
All parameters must have the same names.
Do they have the same parameters in the above samples?
Read files from a path
Filter out unwanted
files
Loop Files one by one
Pipeline 3
Pipeline 4
Pipeline 5
Pipeline 1

16. • Understand the Flow (Profiling Profiling Profiling).
• Covers all technical and non technical requirements (SOW).
• R&D, POC and get second opinion (by MS if possible).
• Rough sketch your Pipeline Flow (Categorise your ADFs).
• Plan a deployment plan with the devops team once a week.
• Stress test, run jobs everyday overnight, Check the logs, use
Azure Advisor, Azure Security Center, failure is the key.
• Predict massive changes & how easy you can change them.
• Reusability and Documentation (Cursor story)
Building ADF
The goal is to hand over the work to someone junior.

17. Motivation Slide
Think big think data. Azure IoT DevKit

18. DEMO
Demo / Hands-on
Let’s Visualize.

19. References
http://sqldataside.blogspot.ca/

20. • IoT Hub DevKit from MS (temperature)
(Microsoft Canada HQ in Mississauga)
Webinar, March ??, 2019
- ?:00 PM to ?:00 PM
• Introduction to Azure IoT SuiteLocation:
• Microsoft Canada HQ in MississaugaLocation:
Future Events…
C# Corner ---

21. THANK YOU
Please feel free to survey
1. How did we/I do?
2. How can we improve?
3. What other public event are you interested?
Stay in touch:
SQLDataSide@gmail.com
http://sqldataside.blogspot.ca/
Sincerely
Nik
Thank You For Your Time