This document summarizes Mark Kromer's presentation on using Azure Data Factory and Azure Databricks for ETL. It discusses using ADF for nightly data loads, slowly changing dimensions, and loading star schemas into data warehouses. It also covers using ADF for data science scenarios with data lakes. The presentation describes ADF mapping data flows for code-free data transformations at scale in the cloud without needing expertise in Spark, Scala, Python or Java. It highlights how mapping data flows allow users to focus on business logic and data transformations through an expression language and provides debugging and monitoring of data flows.

1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2. Mark Kromer
Sr. Azure Data Program Manager
Microsoft
ETL Made Easy with Azure
Data Factory & Azure
Databricks
#UnifiedAnalytics #SparkAISummit

3. Azure Data Factory
Cloud ETL Patterns with ADF
3#UnifiedAnalytics #SparkAISummit

4. Nightly ETL Data Loads Code-free

5. Slowly Changing Dimension Scenario

6. Load Star Schema DW Scenario

7. Data Lake Data Science Scenario

8. Azure Data Factory
Workflow Data Pipelines/Control Flow
9#UnifiedAnalytics #SparkAISummit

12. Azure Data Factory
Mapping Data Flows
13#UnifiedAnalytics #SparkAISummit

13. What is ADF Mapping Data Flow?
• Transform Data, At Scale, in the
Cloud, Zero-Code
– Cloud-first, scale-out ELT
– Code-free dataflow pipelines
• Serverless scale-out transformation
execution engine
• Maximum Productivity for Data
Engineers
– Does NOT require understanding of
Spark / Scala / Python / Java
• Resilient Data Transformation
Flows
– Built for big data scenarios with
unstructured data requirements
– Operationalize with Data Factory
scheduling, control flow and
monitoring

14. Code-free Data Transformation At Scale
• Does not require understanding of Spark, Big Data Execution
Engines, Clusters, Scala, Python …
• Focus on building business logic and data transformation
– Data cleansing
– Aggregation
– Data conversions
– Data prep
– Data exploration

15. Transformation Function Expression Language

16. Debug Data Flows with Data Preview and Data
Sampling

17. Deep Monitoring Introspection of Data Transformations

18. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT