Pipelines and Packages: Introduction to Azure Data Factory (Presented at DATA:Scotland on September 13th, 2019)

1. Pipelines and Packages:
Introduction to Azure Data Factory
Cathrine Wilhelmsen
DATA:Scotland · September 13th, 2019

2. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines and Packages: Introduction to Azure Data Factory

4. @cathrinew
cathrinew.net

5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare

8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
10 years ago…

10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
SSIS
SQL Server Integration Services

11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Then…

13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ADF v1
Azure Data Factory Version 1

14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Today…

17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ADF v2
Azure Data Factory Version 2

18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why?
How?
What?

20. Azure
Data Factory

21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?

22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is inside Azure Data Factory?
Pipelines
Activities
Datasets
Linked
Services
Data Flows
Templates
Triggers

24. DEMO
Let's look inside
Azure Data Factory!

25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Wait…
I already have
thousands of SSIS
packages!

26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
And…
You told me to use
this Biml thing!

27. SSIS
Lift and Shift

28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What does Lift and Shift mean?

29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
*
Why should you Lift and Shift SSIS?

30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
1. Configure Azure-SSIS Integration Runtime
2. Deploy SSIS Packages to SSISDB in Azure
3. Orchestrate SSIS Packages in Azure Data Factory
How do you Lift and Shift SSIS?

31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
1. Configure Azure-SSIS Integration Runtime
2. Deploy SSIS Packages to SSISDB in Azure
3. Orchestrate SSIS Packages in Azure Data Factory
How do you Lift and Shift SSIS?

32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure-SSIS Integration Runtime
like all VMs

33. DEMO
Let’s lift and shift
some SSIS Packages!

34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
≈
≈
≈
≈
≈
≈
ADF vs SSIS

35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline
Linked Service
Source
Sink
Activity
Data Flow
≈
≈
≈
≈
≈
≈
Package
Connection Manager
Source
Destination
Control Flow Task
Data Flow
ADF vs SSIS

36. Mapping
Data Flows

37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?

38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How do Mapping Data Flows work?

39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Mapping Data Flows?

40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Schema Drift?

41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Schema Drift in SSIS

42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Schema Drift in ADF

43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Oh no!

44. DEMO
Let's transform
some data!

45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lessons Learned

46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!

47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!

48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)