This session focuses on the all Platform as a Service (PaaS) solution of Azure SQL Database (DB)/Managed Instance (MI) + SQL Server Integration Services (SSIS) in Azure Data Factory (ADF) to lift & shift, modernize, and extend Extract, Transform & Load (ETL) workflows. We will first show you how to provision Azure-SSIS Integration Runtime (IR) – dedicated ADF servers for running SSIS – with SSIS catalog (SSISDB) hosted by Azure SQL DB/MI, configure it to access data on premises using Windows authentication and Virtual Network (VNet) injection/Self-Hosted IR as a proxy, and extend it with custom/Open Source/3rd party components. We will next show you how to use the familiar SQL Server Data Tools (SSDT)/SQL Server Management Studio (SSMS) to design/debug/deploy/configure/execute your SSIS packages in the cloud just like you do on premises. We will then show you how to modernize your ETL workflows by invoking/scheduling SSIS package executions as first-class activities in ADF pipelines and combining/chaining them with other activities, allowing you to trigger your pipeline runs by events, automatically (de)provision SSIS IR just in time, etc. We will also show you how to enable the following scenarios: assessing/testing SSIS packages in ADF from SSDT, running SSIS packages stored in file system/Azure Files/SQL Server database (MSDB) hosted by Azure SQL MI, accessing Azure SQL DB/MI with OLEDB connectors that are configured for Azure Active Directory (AAD) authentication with ADF managed identity, scheduling SSIS jobs in ADF from SQL Server Agent/3rd party orchestrators that invoke Azure-enabled dtexec.exe, etc. We will finally show you the complete SSIS Migration Playbook to perform package assessments and package/job migrations in batches using Data(base) Migration Assistant/Service, dtutil.exe, scripts, etc.
9. SSIS + ADF
Analogy
• Running traditional ETL workloads on premises is like being
on the ground floor
• Running modern ELT workloads in the cloud on a fully
managed Platform as a Service (PaaS) is like being on the 7th
floor, so jumping 7 stories straight up is virtually impossible
10. SSIS + ADF = SSIS in ADF
Analogy
• Running traditional ETL workloads on premises is like being
on the ground floor
• Running modern ELT workloads in the cloud on a fully
managed Platform as a Service (PaaS) is like being on the 7th
floor, so jumping 7 stories straight up is virtually impossible
• Running SSIS in ADF is like taking an escalator that
seamlessly carries you to the 7th floor step-by-step with the
option to get on/off anytime at any floor.
32. SSIS IR node
Container
ISV Setup1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key
3rd Party Extensibility – Implementation
37. Batch Assess Batch Migrate
Edit
TestAssess
DBA
BI Developer
SSDT
Database Migration
Assistant (DMA)/SSDT
Azure Files/file system
Database Migration
Service
(DMS)/Deployment
Wizard/dtutil
utility/manual copy
SSISDB hosted by Azure
SQL DB/MI, MSDB
hosted by Azure SQL MI
38. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
39. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
40. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
MSDB • Same as above • MSDB to MSDB via DMS (TBD)
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Export to Azure Files/file
system via SSMS/dtutil
• Recreate in ADF via SSMS
• Recreate in ADF
41. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
MSDB • Same as above • MSDB to MSDB via DMS (TBD)
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Export to Azure Files/file
system via SSMS/dtutil
• Recreate in ADF via SSMS
• Recreate in ADF
SSIS Package
Store
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Export to Azure Files/file
system via SSMS/dtutil
• Same as above
51. Why Should You Care? Better TCO & ROI!
Source: The Business Value of Microsoft Azure SQL Database Services, IDC, Mar 2015
The Total Economic Impact of Microsoft Azure SQL Database Managed Instance, Forrester Consulting, Sep 2018
52. Why Should You Care? BYOL to Get AHB!
Maximize the value of on-premises SQL Server license investments by migrating
SSIS in ADF/Azure-SSIS Integration Runtime (IR) Editions
SQL Server with Software Assurance
53. Why Should You Care? Huge Opportunity!
Source: Worldwide Database Server Forecast, IDC, Dec 2016
Worldwide Database Server Forecast
Modernizing
opportunity
with SQL
Server
37% of all units
run Microsoft,
>50% of them
run 2008/R2
or older versions
Non-Relational
Open Source
PaaS & IaaS
Microsoft
Oracle
IBM
SAP
New SQL
Other
MySQL
No
SQL
54. Why Should You Care? Do The Math!
Source: Feb 2019 snapshot of active SSIS instances extrapolated from SSIS 2016/2017 telemetry & CSS case volume for SQL 2008-2017
SSIS 2008-2017 on
premises/Azure VMs/AWS VMs
(100%)
SSIS 2008-2017 on premises
(88%)
SSIS 2008/R2 EOS w/o SSISDB
(13%)
SSIS 2012-2017 w/ SSISDB
(21%)
Total Addressable Market
Present Lift & Shift
Target
Immediate
Target
Past Target
56. How Should You Care? Fear is a Great Motivator!
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
57. Fear of losing investments
Too many SSIS packages to convert from using OLEDB connection via Win Auth to using ADO.NET connection via AAD Auth w/ ADF
managed identity
Too many SSIS packages to convert from Package Deployment Model w/o SSISDB to Project Deployment Model w/ SSISDB
Too many SSIS jobs scheduled by 3rd party orchestrators (e.g. Active Batch, Control-M, etc.) invoking dtexec
Too many packages using absolute paths w/ hard-coded drives (e.g. C:)
SQL MI Agent doesn’t support SSIS environment/32-bit runtime/property override
SQL MI Agent can’t invoke dtexec to execute SSIS packages stored in MSDB/file system/SSIS package stores
Network configuration/policy is too complex/strict to access on-premises data via VNet
Custom setups w/ script are too complex/error-prone
How Should You Care? Fear is a Great Motivator!
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
58. How Should You Care? Fear is a Great Motivator!
Fear of losing investments
Too many SSIS packages to convert from using OLEDB connection via Win Auth to using ADO.NET connection via AAD Auth w/ ADF
managed identity
– Support OLEDB connection via AAD Auth w/ ADF managed identity, switch at run-time via property overrides
Too many SSIS packages to convert from Package Deployment Model w/o SSISDB to Project Deployment Model w/ SSISDB
– Support file system/Azure Files/MSDB hosted by SQL MI to store packages, turn SSISDB from a blocker into an upsell
Too many SSIS jobs scheduled by 3rd party orchestrators (e.g. Active Batch, Control-M, etc.) invoking dtexec
– Azure-enable dtexec to invoke Execute SSIS Package activities in ADF pipelines
Too many packages using absolute paths w/ hard-coded drives (e.g. C:)
– Replace absolute paths w/ UNC paths at run-time via property overrides
SQL MI Agent doesn’t support SSIS environment/32-bit runtime/property override
– Achieve feature parity w/ SQL Server Agent
SQL MI Agent can’t invoke dtexec to execute SSIS packages stored in MSDB/file system/SSIS package stores
– ADF-enable SQL MI Agent to invoke executions of packages stored in file system/Azure Files/SSIS IR package stores
Network configuration/policy is too complex/strict to access on-premises data via VNet
– Support Self-Hosted IR as proxy for Azure-SSIS IR to access on-premises data
Custom setups w/ script are too complex/error-prone
– Support custom setups w/o script
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
59. How Should You Care? Fear is a Great Motivator!
Minimizing fear of losing investments
– No more lift & shift blockers
Rebalancing your risks/fears
Maximizing fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
ADO.NET/OLEDB/Storage connection via AAD Auth w/ ADF managed identity
AKV support on Execute SSIS Package activity
SSIS ADF connectors (80+)
PQ Source
DQS migration to cloud alternatives running on SSIS in ADF
More cloud-first/cloud-only features, etc.
62. https://www.slideshare.net/SandyWinarko/paasport-to-paradise-back-to-the-future-
with-ssis-in-azure-data-factory
Documentations
Provisioning Azure-SSIS IR Using UI: https://docs.microsoft.com/azure/data-factory/tutorial-deploy-ssis-packages-azure
Using PSH: https://docs.microsoft.com/azure/data-factory/tutorial-deploy-ssis-packages-azure-powershell
Using ARM template: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime#azure-
resource-manager-template
Provisioning Azure-SSIS IR using Azure SQL
DB/Managed Instance w/ VNet service
endpoint/private endpoint to host SSISDB
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Joining a VNet: https://docs.microsoft.com/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
Provisioning Azure-SSIS IR using AAD Auth w/
ADF managed identity
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Enabling AAD Auth: https://docs.microsoft.com/azure/data-factory/enable-aad-authentication-azure-ssis-ir
Provisioning Azure-SSIS IR to access data on
premises
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Joining a VNet: https://docs.microsoft.com/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
Using Self-Hosted IR as a proxy: https://docs.microsoft.com/azure/data-factory/self-hosted-integration-runtime-proxy-
ssis
Monitoring Azure-SSIS IR Using UI/PSH: https://docs.microsoft.com/azure/data-factory/monitor-integration-runtime#azure-ssis-integration-
runtime
Using Azure Monitor: https://docs.microsoft.com/azure/data-factory/monitor-using-azure-monitor#monitor-ssis-
operations-with-azure-monitor
63. Documentations
Configuring Azure-SSIS IR Using UI/PSH: https://docs.microsoft.com/azure/data-factory/manage-azure-ssis-integration-runtime
Configuring Azure-SSIS IR to use Enterprise Edition Using UI/PSH: https://docs.microsoft.com/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition
Configuring Azure-SSIS IR to use custom setups Using UI/PSH: https://docs.microsoft.com/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup
Configuring Azure-SSIS IR for high performance Using UI/PSH: https://docs.microsoft.com/azure/data-factory/configure-azure-ssis-integration-runtime-
performance
Configuring Azure-SSIS IR for disaster recovery Using UI/PSH: https://docs.microsoft.com/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime
Deploying projects/packages to run on Azure-SSIS IR Connecting SSMS to SSISDB: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-to-
catalog-database?view=sql-server-2017
Using UI/PSH: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-deploy-run-monitor-
tutorial?view=sql-server-2017
Using dtutil: https://docs.microsoft.com/azure/data-factory/azure-ssis-integration-runtime-package-
store#deploying-multiple-packages-with-dtutil
64. Documentations
Connecting to data stores using Win Auth on Azure-
SSIS IR
Using 4 methods: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-
auth?view=sql-server-2017
Connecting to data stores using AAD Auth w/ ADF
managed identity on Azure-SSIS IR
Using Azure Storage connections: https://docs.microsoft.com/sql/integration-services/connection-
manager/azure-storage-connection-manager#managed-identities-for-azure-resources-authentication
Using ADO.NET connections: https://docs.microsoft.com/sql/integration-services/connection-manager/ado-net-
connection-manager#managed-identities-for-azure-resources-authentication
Using OLEDB connections: https://docs.microsoft.com/sql/integration-services/connection-manager/ole-db-
connection-manager#managed-identities-for-azure-resources-authentication
Connecting to file shares and processing files on Azure-
SSIS IR
Using local file system/file shares: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-files-
file-shares?view=sql-server-2017
Using Win Auth: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-
auth?view=sql-server-2017
Running packages on Azure-SSIS IR From SSDT: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-ssdt
Using SSMS/stored procedures/scripts/code: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-
azure-run-packages?view=sql-server-2017
Using AzureDTExec: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-azure-enabled-
dtexec
As Execute SSIS Package activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-
invoke-ssis-package-ssis-activity
As Stored Procedure activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-invoke-
ssis-package-stored-procedure-activity
65. Documentations
Scheduling package executions on Azure-SSIS IR Using SSMS/ADF/other methods: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-
schedule-packages?view=sql-server-2017
Using SSMS: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-schedule-packages-
ssms?view=sql-server-2017
As Execute SSIS Package activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-
invoke-ssis-package-ssis-activity
Using Azure SQL MI Agent: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-
managed-instance-agent
Scheduling Azure-SSIS IR provisioning on demand/just
in time
Using Web activities: https://docs.microsoft.com/azure/data-factory/how-to-schedule-azure-ssis-integration-
runtime
Cleaning up package execution logs in SSISDB Using Elastic Jobs: https://docs.microsoft.com/azure/data-factory/how-to-clean-up-ssisdb-logs-with-elastic-jobs
Assessing SSIS packages for migration Using DMA: https://docs.microsoft.com/sql/dma/dma-assess-ssis?view=sql-server-ver15
Using SSDT: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-ssdt#assess-ssis-
projectpackages-for-executions-in-azure
Migrating SSIS packages SSIS Migration Playbook: https://docs.microsoft.com/azure/data-factory/scenario-ssis-migration-overview
Using DMS: https://docs.microsoft.com/azure/dms/how-to-migrate-ssis-packages-managed-instance
Practicing SSIS CI/CD Using Azure DevOps: https://docs.microsoft.com/sql/integration-services/devops/ssis-devops-overview?view=sql-
server-ver15
Developing premium/licensed components for Azure-
SSIS IR
Using Cluster ID: https://docs.microsoft.com/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-
components
Procuring 3rd party components for Azure-SSIS IR Partner list: https://techcommunity.microsoft.com/t5/SQL-Server-Integration-Services/Enterprise-Edition-Custom-
Setup-and-3rd-Party-Extensibility-for/ba-p/388360
Continuous and increased product investments for different customer cohorts.
Continuous and increased product investments for different customer cohorts.
Continuous and increased product investments for different customer cohorts.
Typical SQL Server estate including primary Value-Added Services (VAS), excluding secondary ones (MDS/DQS).
Full compatibility rehost for DB + ETL engines.
Full compatibility rehost vs. partial rehost/complete replacement.
SSIS in ADF as glue between disparate resources on premises and in the cloud.
Pop quiz: What is AWS ETL service called?
ADF complements/fills gaps in SSIS and vice versa.
We have cloud alternatives of DQS built as extensions running on SSIS in ADF.
2017 is our break-out year.
Preserved skillsets by ensuring that traditional on-premises tools can still be used in the cloud.
Transformed ADF into a first-class job scheduler for Azure SQL DB.
Lift & shift motivation/migration drive from the economic/financial angle for enterprises.
Lift & shift motivation/migration drive from the economic/financial angle for enterprises.
Pop quiz: If you have on-premises SQL Server license for 100 cores and you’ve traded in 50 cores for Azure SQL DB/MI, how many cores can you use for SQL on IaaS/SSIS in ADF?
Lift & shift motivation/migration drive from the business opportunity angle for System Integrators (SIs) and DB/BI consultancies.
Lift & shift motivation/migration drive from the business opportunity angle for System Integrators (SIs) and DB/BI consultancies.
Lift & shift motivation/migration drive from the timing angle for all.
Pop quiz: What kind of insect is in my photo?
Lift & shift motivation/migration drive from the risk/fear angle (pros & cons) for enterprises.
Pros = fear of missing out.
Lift & shift motivation/migration drive from the risk/fear angle (pros & cons) for enterprises.
In the past, pros = fear of missing out < cons = fear of losing investments.
We’ve addressed/are addressing each fear of losing investments.
Now, pros = fear of missing out > cons = fear of losing investments.
