Understanding Azure Data Factory: The What, When, and Why (Presented at NIC on February 6th, 2020)

1. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)

2. Understanding
Azure Data Factory
The What, When, and Why
Cathrine Wilhelmsen
NIC · February 6th, 2019

3. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Understanding Azure Data Factory

4. @cathrinew
cathrinew.net

5. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

6. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

7. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

8. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare

9. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)

10. Azure
Data Factory

11. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?

12. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

13. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What is inside Azure Data Factory?
Pipelines
Activities Datasets
Linked
Services
Integration
Runtimes
Triggers
Templates

14. DEMO
Let's look inside
Azure Data Factory!

15. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

16. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What is the Copy Data Activity?
*
* Cathrine's opinion :)

17. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Process: Binary Files
Source Sink

18. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Process: Complex Files
Source Sink
Serialization
Deserialization
Compression
Decompression
Column
Mapping

19. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Process: Complex Files
Source Sink
Serialization
Deserialization
Compression
Decompression
Column
Mapping

20. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Process: Complex Files
Source Sink
Serialization
Deserialization
Compression
Decompression
Column
Mapping

21. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Process: Complex Files
Source Sink
Serialization
Deserialization
Compression
Decompression
Column
Mapping

22. DEMO
Let's copy
some data!

23. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
what if my
systems are
on-premises?

24. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Hybrid Azure Data Factory

25. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Integration Runtimes?

26. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Integration Runtime

27. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Self-Hosted Integration Runtime

28. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Copy Data Scenarios

29. DEMO
Let's connect to an
on-prem SQL Server!

30. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Ok, so we can copy data…
Copy Data Transform Data

31. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
…what about transforming data?
Copy Data Transform Data

32. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Mapping or Wrangling

33. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?

34. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
How do Mapping Data Flows work?

35. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Wrangling Data Flows?

36. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
How do Wrangling Data Flows work?

37. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Mapping Data Flows Wrangling Data Flows

38. DEMO
Let's transform
some data!

39. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
how do we
schedule data
pipelines?

40. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Trigger pipelines…

41. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Triggers: Schedule

42. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Triggers: Tumbling Window

43. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Triggers: Event Based

44. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Triggers: Now

45. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Monitoring Triggers

46. DEMO
Let's schedule
some pipelines!

47. Azure Data
Architectures

48. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Advanced Analytics on Big Data
https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/

49. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Real-time Analytics
https://azure.microsoft.com/en-us/solutions/architecture/real-time-analytics/

50. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Modern Data Warehouse
https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/

51. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Azure
Data Factory
Serve
Azure Synapse
Analytics
Visualize
Power BI
Store
Azure Data Lake Storage
Data Pipeline Orchestration and Monitoring
Azure Data Factory
Sources
Cloud
SaaS
Prepare
Wrangling
Data Flows
Transform
Mapping
Data Flows
On-Premises

52. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Serve
Azure Synapse Analytics
Visualize
Power BI
Sources
Cloud
SaaS
On-Premises

53. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!

54. © 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!