Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Global Azure Norway 2021)

•

2 gefällt mir•1,107 views

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Global Azure Norway on April 16th, 2021)

Daten & Analysen

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines & Data Flows:
Introduction to Data Integration
in Azure Synapse Analytics
Cathrine Wilhelmsen
Global Azure Norway · April 16th, 2021

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Abstract
Do you regularly need to get data for your projects?
Yep! 🙋‍♀️
Data is at the core of every Business Intelligence, Data Science, and Machine Learning project.
You need data to understand what has happened in the past, to predict what may happen in
the future, to discover patterns and anomalies, and to gain the insight necessary for making
faster and better decisions.
But before you can do any of those things, you need to ingest, store, transform, integrate, and
prepare your data. Guess what? You can do all of those things in Azure Synapse Analytics –
without having to write any code!
In this session, we will cover the fundamentals of data integration in Azure Synapse Analytics.
First, we will discuss when Azure Synapse Analytics is the right tool of choice. Then, we will go
through what Pipelines and Data Flows are, and when to use them. Finally, we will see how
easy it is to ingest and transform data both on-premises and in the cloud.

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Store
Transform
Integrate
Prepare

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse Analytics
What?
Pipelines & Data Flows
How?
Data Integration
When?
…the next 45 minutes…

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
All-in-one platform for analytical projects:
• Data Lake (All Data)
• Data Warehouse (Relational Data)
• Data Analytics (SQL, Spark)
• Data Integration (Ingest, Transform)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
Deeply integrated with other services:
• Azure Purview
• Azure Machine Learning
• Azure Cosmos DB
• Power BI

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Who can use Azure Synapse Analytics?
Built for collaboration between:
• Data Engineers
• Data Analysts
• Data Scientists
• Data Consumers

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Integration
Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Copy Data Transform Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Pipelines?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Activities?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Datasets?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Linked Services?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flows?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Triggers?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse or Azure Data Factory?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/adf

Weitere ähnliche Inhalte

Was ist angesagt?

Data Quality Patterns in the Cloud with Azure Data Factory

Mark Kromer

Azure data factory

David Giard

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Azure Synapse Analytics Overview (r1)

James Serra

Azure Synapse 101 Webinar Presentation

Matthew W. Bowers

Azure Data Factory

HARIHARAN R

DataMinds 2022 Azure Purview Erwin de Kreuk

Erwin de Kreuk

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

Cathrine Wilhelmsen

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Mark Kromer

Azure Data Factory v2

inovex GmbH

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Cathrine Wilhelmsen

Microsoft Data Integration Pipelines: Azure Data Factory and SSIS

Mark Kromer

Azure data factory

BizTalk360

Azure Data Factory Data Flow

Mark Kromer

TechEvent Databricks on Azure

Trivadis

In essence, a data lake is commodity distributed file system that acts as a repository to hold raw data file extracts of all the enterprise source systems, so that it can serve the data management and analytics needs of the business. A data lake system provides means to ingest data, perform scalable big data processing, and serve information, in addition to manage, monitor and secure the it environment. In these slide, we discuss building data lakes using Azure Data Factory and Data Lake Analytics. We delve into the architecture if the data lake and explore its various components. We also describe the various data ingestion scenarios and considerations. We introduce the Azure Data Lake Store, then we discuss how to build Azure Data Factory pipeline to ingest the data lake. After that, we move into big data processing using Data Lake Analytics, and we delve into U-SQL.

Building the Data Lake with Azure Data Factory and Data Lake Analytics

Khalid Salama

Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.

Building a modern data warehouse

James Serra

Azure Data Factory presentation with links

Chris Testa-O'Neill

Azure Data Factory Data Flows Training (Sept 2020 Update)

Mark Kromer

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

The new Microsoft Azure SQL Data Warehouse (SQL DW) is an elastic data warehouse-as-a-service and is a Massively Parallel Processing (MPP) solution for "big data" with true enterprise class features. The SQL DW service is built for data warehouse workloads from a few hundred gigabytes to petabytes of data with truly unique features like disaggregated compute and storage allowing for customers to be able to utilize the service to match their needs. In this presentation, we take an in-depth look at implementing a SQL DW, elastic scale (grow, shrink, and pause), and hybrid data clouds with Hadoop integration via Polybase allowing for a true SQL experience across structured and unstructured data.

Introducing Azure SQL Data Warehouse

James Serra

Was ist angesagt? (20)

Data Quality Patterns in the Cloud with Azure Data Factory

Azure data factory

Azure Synapse Analytics Overview (r1)

Azure Synapse 101 Webinar Presentation

Azure Data Factory

DataMinds 2022 Azure Purview Erwin de Kreuk

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Azure Data Factory v2

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Microsoft Data Integration Pipelines: Azure Data Factory and SSIS

Azure data factory

Azure Data Factory Data Flow

TechEvent Databricks on Azure

Building the Data Lake with Azure Data Factory and Data Lake Analytics

Building a modern data warehouse

Azure Data Factory presentation with links

Azure Data Factory Data Flows Training (Sept 2020 Update)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Introducing Azure SQL Data Warehouse

Ähnlich wie Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Global Azure Norway 2021)

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Cathrine Wilhelmsen

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)

Cathrine Wilhelmsen

Analytics in a Day Virtual Workshop

CCG

Azure synapse by usama whaba khan

Usama Wahab Khan Cloud, Data and AI

Analytics in a Day Ft. Synapse Virtual Workshop

CCG

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Cathrine Wilhelmsen

Analytics in a Day Ft. Synapse Virtual Workshop

CCG

New ways to apply infrastructure data for better business outcomes

accenture

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

Cathrine Wilhelmsen

Ingesting Click Data for Analytics

ClickMeter

Francesco Furiani - Marketing is a serious business, moreover tracking and monetizing the campaign that allows your marketing to flourish is very important: our tool allows anyone to monitor, compare and optimize all those campaigns (delivered via links) in one place and to deliver insights about who's using those links. Making this infrastructure, making it works, deliver results in real-time (when necessary) and keep everyone happy from the customer to the CFO will be the point of this talk, from the design to the final result with an eye on the costs/risks/benefits of having everything in the cloud.

Ingesting click events for analytics

Data Driven Innovation

1 Introduction to Microsoft data platform analytics for release

Jen Stirrup

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Cathrine Wilhelmsen

The Briefing Room with Barry Devlin and WhereScape Live Webcast on June 10, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=5230c31ab287778c73b56002bc2c51a The data warehouse is intended to support analysis by making the right data available to the right people in a timely fashion. But conditions change all the time, and when data doesn’t keep up with the business, analysts quickly turn to workarounds. This leads to ungoverned and largely un-managed side projects, which trade short-term wins for long-term trouble. One way to keep everyone happy is by creating an integrated environment that pulls data from all sources, and is capable of automating both the model development and delivery of analyst-ready data. Register for this episode of The Briefing Room to hear data warehousing pioneer and Analyst Barry Devlin as he explains the critical components of a successful data warehouse environment, and how traditional approaches must be augmented to keep up with the times. He’ll be briefed by WhereScape CEO Michael Whitehead, who will showcase his company’s data warehousing automation solutions. He’ll discuss how a fast, well-managed and automated infrastructure is the key to empowering faster, smarter, repeatable decision making. Visit InsideAnlaysis.com for more information.

Smarter Analytics: Supporting the Enterprise with Automation

Inside Analysis

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)

Cathrine Wilhelmsen

Deep architectural competency for deploying azure solutions

Synergetics Learning and Cloud Consulting

Cortana Intelligence Solutions

Darwin Schweitzer

Analytics in a Day Ft. Synapse Virtual Workshop

CCG

It is a fascinating, explosive time for enterprise analytics. It is from the position of analytics leadership that the mission will be executed and company leadership will emerge. The data professional is absolutely sitting on the performance of the company in this information economy and has an obligation to demonstrate the possibilities and originate the architecture, data, and projects that will deliver analytics. After all, no matter what business you’re in, you’re in the business of analytics. The coming years will be full of big changes in enterprise analytics and Data Architecture. William will kick off the fourth year of the Advanced Analytics series with a discussion of the trends winning organizations should build into their plans, expectations, vision, and awareness now.

2022 Trends in Enterprise Analytics

DATAVERSITY

Ähnlich wie Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Global Azure Norway 2021) (20)

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)

Analytics in a Day Virtual Workshop

Azure synapse by usama whaba khan

Analytics in a Day Ft. Synapse Virtual Workshop

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Analytics in a Day Ft. Synapse Virtual Workshop

New ways to apply infrastructure data for better business outcomes

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

Ingesting Click Data for Analytics

Ingesting click events for analytics

1 Introduction to Microsoft data platform analytics for release

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Smarter Analytics: Supporting the Enterprise with Automation

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)

Deep architectural competency for deploying azure solutions

Cortana Intelligence Solutions

Analytics in a Day Ft. Synapse Virtual Workshop

2022 Trends in Enterprise Analytics

Mehr von Cathrine Wilhelmsen

Data Factory in Microsoft Fabric (MsBIP #82)

Cathrine Wilhelmsen

Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...

Cathrine Wilhelmsen

Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)

Cathrine Wilhelmsen

Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...

Cathrine Wilhelmsen

Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)

Cathrine Wilhelmsen

The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)

Cathrine Wilhelmsen

Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...

Cathrine Wilhelmsen

Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...

Cathrine Wilhelmsen

Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)

Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...

Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023) We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout. The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits. I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...

Cathrine Wilhelmsen

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023) Do you sometimes think the world is moving so fast that you're struggling to keep up? Does it make you feel a little uncomfortable? Awesome! That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it. It just might not be easy. All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning. Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...

Cathrine Wilhelmsen

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022) What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights? We had no idea, but we gave it a go! Our ambition was to think big, start small, and act fast – to deliver business value early and often. Did we succeed? Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...

Cathrine Wilhelmsen

6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022) Do you feel nervous about getting on stage to deliver a presentation? That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed? There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice. You can do this!

6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)

Cathrine Wilhelmsen

Azure Data Factory for the SSIS Developer (SentryOne Webinar)

Cathrine Wilhelmsen

Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)

Cathrine Wilhelmsen

Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...

Cathrine Wilhelmsen

Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)

Cathrine Wilhelmsen

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

Cathrine Wilhelmsen

Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)

Cathrine Wilhelmsen

Mehr von Cathrine Wilhelmsen (20)