Building Dynamic Pipelines in Azure Data Factory (Presented at Data Saturday Holland on October 5th, 2019)

1. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

2. Dynamic Data Pipelines
in Azure Data Factory
Cathrine Wilhelmsen
Data Saturday Holland · Oct 5, 2019

3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Building Dynamic Data Pipelines in Azure Data Factory

4. @cathrinew
cathrinew.net

5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
You are an
Azure Data Factory
developer

6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
…this is getting
pretty tedious…

8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
There has to be
a better way!

9. Dynamic Data
Pipelines

10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use dynamic solutions?

11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why not use dynamic solutions?

12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you make dynamic solutions?
1.
2.
3.
4.

15. Parameters

16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters

17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters

18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service

19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline

20. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
data flow

21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
activity

22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline
activity
data flow

23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

24. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

27. Expressions
and Functions

28. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions
@

29. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Functions

30. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions with Functions
@

31. Combining Strings

32. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
@concat
@{…}

33. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
String Concatenation
@concat(
' TRUNCATE TABLE dbo. ',
pipeline().parameters.TableName
)

34. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
String Interpolation
TRUNCATE TABLE dbo.
@{pipeline().parameters.TableName}

35. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
String Concatenation String Interpolation

36. DEMO
Let's parameterize
some pipelines!

37. ForEach Loops

38. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops

39. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops
@item()

40. Array Items

41. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items

42. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
colors
@item()

43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
@item()
parts
@item()
colors
@item()
sets

44. Object Items

45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items

46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item()

47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item().Name

48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
@item().Name
parts
@item().Name
colors
@item().Name
sets

49. DEMO
Let's create
some loops!

50. Lookups

51. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup

52. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output

53. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}

54. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
firstRow

55. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
firstRow
.Name

56. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value

57. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value[1]

58. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value[1]
Name

59. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
@{activity('Lookup').output.firstRow.ColumnName}
@{activity('Lookup').output.value}

60. DEMO
Let's add a lookup to
control our loop!

61. Controls and
Conditionals

62. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

63. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Branching and Chaining
Success Failure Completion Skipped

64. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
If Condition

65. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Variables

66. DEMO
Let's create some
if statements!

67. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

68. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Until

69. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata

70. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Wait

71. DEMO
Let's check for
file existence!

72. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!

73. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!