Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (Presented at DataMinutes #2 on January 21st, 2022)

1. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Lessons Learned:
Understanding Pipeline Pricing
in Azure Data Factory and Azure Synapse Analytics
Cathrine Wilhelmsen
DataMinutes #2 • January 21st, 2022

2. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Abstract
Azure Data Factory and Synapse Pipelines pricing is easy, right? No upfront
costs. Pay only for what you use. It's a wonderful world for data integration
developers. Just a few clicks and your solution is ready!
But what do you present to management when they ask for cost estimates? "I
guess we just have to wait for the next invoice" is rarely an acceptable answer.
It is definitely not an acceptable answer if that invoice turns out to be
unexpectedly high.
In this session, we will look at the bits and pieces that make up the Azure Data
Factory and Synapse Pipelines pricing models, how to monitor consumption
and manage costs, and cover a few important gotchas to be aware of…

3. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net

4. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines are
awesome, right?

5. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
No upfront costs!
Just start developing!
Pay only for usage!

6. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
“But how much will
it cost per month?”

7. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
“I guess we just have to
wait for the invoice!"

8. Pipeline
Pricing

9. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations
Orchestration and Execution
Data Flows

10. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations
(Pretty cheap, but everything has a cost)
Orchestration and Execution
Data Flows

11. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations: Azure Data Factory
Read / Write: $0.50 per 50 000 entities modified
Operations: Create, Read, Update, Delete
Monitoring: $0.25 per 50 000 run records retrieved
Operations: Get, List
* Prices in USD from January 2022

12. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations: Azure Synapse
First 1 Million Operations: Free!
Operations: Create, Read, Update, Delete, Get, List
Additional Operations: $0.25 per 50 000 operations
Operations: Create, Read, Update, Delete, Get, List
* Prices in USD from January 2022

13. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations
Orchestration and Execution
(Be aware of the biggest gotchas)
Data Flows

14. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Integration Runtimes
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

15. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Orchestration
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

16. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Orchestration
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per run:
Activity runs, debug runs, trigger executions
$0.001 $0.001 $0.0015
Data Movement per DIU-hour:
Copy Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

17. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Data Movement
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

18. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Data Movement
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

19. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Data Movement
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

20. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Data Movement
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
Auto = 4+

21. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Data Movement
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
Minimum = 2
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

22. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Pipeline Activity
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

23. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Pipeline Activity
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebook, Databricks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
200x 500x

24. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: External Activity
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

25. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: External Activity
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
4000x 10000x

26. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

27. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Rounding Up
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

28. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Rounding Up
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

29. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Rounding Up
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

30. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Rounding Up
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
0.0167 hours
= 1 minute
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

31. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline Pricing: Rounding Up
Azure IR Azure Managed VNet IR Self-Hosted IR
Orchestration per 1000 runs:
Activity runs, debug runs, trigger executions
$1 $1 $1.50
Data Movement per DIU-hour:
Copy Data Activity
$0.25 $0.25 $0.10
Pipeline Activity per hour:
Lookup, Get Metadata, Delete etc.
$0.005 $1 $0.002
External Activity per hour:
Stored Procedure, Notebooks etc.
$0.00025 $1 $0.0001
0.0333 hours
= 2 minutes
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up.

32. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Operations
Orchestration and Execution
Data Flows
(Can be expensive, but worth it for the right use cases)

33. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows
Basic: $0.286 per vCore-hour
General Purpose
Standard: $0.346 per vCore-hour
Memory Optimized
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

34. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Rounding Up
Basic: $0.286 per vCore-hour
General Purpose
Standard: $0.346 per vCore-hour
Memory Optimized
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

35. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Cluster Size
Basic: $0.286 per vCore-hour
General Purpose
Standard: $0.346 per vCore-hour
Memory Optimized
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

36. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Cluster Size
Basic: $0.286 per vCore-hour
General Purpose
Standard: $0.346 per vCore-hour
Memory Optimized
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

37. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Cluster Size
Basic: $0.286 per vCore-hour
General Purpose
Standard: $0.346 per vCore-hour
Memory Optimized
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

38. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Cluster Size
Basic: $0.286 per vCore-hour = $2.288 per hour
General Purpose: 4 (+4 Driver Cores)
Standard: $0.346 per vCore-hour = $2.786 per hour
Memory Optimized: 4 (+4 Driver Cores)
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

39. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Flows: Cluster Size
Basic: $0.286 per vCore-hour = $77.792 per hour
General Purpose: 256 (+16 Driver Cores)
Standard: $0.346 per vCore-hour = $94.112 per hour
Memory Optimized: 256 (+16 Driver Cores)
* Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.

40. Lessons
Learned

41. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Just because you can…
doesn’t mean you should…

42. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example

43. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
= $0.0000835 = $0.0167 = $0.000008325

44. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
$0.053

45. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
$0.053
$1.59

46. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
$0.053
$159

47. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
$0.053
$3816

48. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Example
$0.053
$45792

49. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Again
Just because you can…
doesn’t mean you should…

50. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Everything has a price
We need to understand the pricing
model to create solutions optimized
for performance and cost

51. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
“I guess we just have to
wait for the invoice!"

52. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Here’s our estimate!
azure.microsoft.com/pricing/details/data-factory
azure.microsoft.com/pricing/details/synapse-analytics

53. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net)
Thank you!
hi@cathrinew.net
@cathrinew
cathrinew.net