Dave Wells of Eckerson Group discusses why cloud data warehousing has become popular, the many benefits, and the corresponding challenges. Migrating an existing data warehouse to the cloud is a complex process of moving schema, data, and ETL. The complexity increases when architectural modernization, restructuring of database schema, or rebuilding of data pipelines is needed.

1. © David L Wells
Data Warehousing in the Cloud
Practical Migration Strategies
Dave Wells
dwells@eckerson.com

2. © David L Wells 2
Dave Wells
Director, Data Management Practice
Eckerson Group
www.eckerson.com
• Advisory consultant
• Educator
• Industry analyst
• Business Intelligence
• Analytics
• Data Management

3. © David L Wells
Cloud Data Warehousing – What and Why?
3
Challenges of Conventional Data Warehousing
Growth Management
Workload Fluctuation
Data Center Management
Data Center & Operations Costs
Processing Bottlenecks & Delays
Projects Wait for Infrastructure
Business Critical with Risks
Security & Governance Challenged
Complex Database Management

4. © David L Wells
Benefits of Cloud Data Warehousing
4
Overcoming the Challenges
Growth Management
Workload Fluctuation
Data Center Management
Data Center & Operations Costs
Processing Bottlenecks & Delays
Projects Wait for Infrastructure
Business Critical with Risks
Security & Governance Challenged
Complex Database Management
Scalability: growth in data, processing & users
Elasticity: adapt to workload peaks and valleys
Managed Infrastructure: reduce data center overhead
Cost Savings: cut cost of hardware, staffing, etc.
Processing Speed: fast data pipelines, no bottlenecks
Deployment Speed: agility, instant infrastructure
Disaster Recovery: benefits of virtualization
Security & Governance: service provider features + VPC
RDBMS in the Cloud: gracefully accepts existing schema

5. © David L Wells
Technologies for Cloud Data Warehousing
5
Cloud Data Warehouse Platforms

6. © David L Wells
Technologies for Cloud Data Warehousing
6
Migration Tools – Integration Platform as a Service (iPaaS)

7. © David L Wells
Technologies for Cloud Data Warehousing
7
Migration Tools – Data Warehouse Automation

8. © David L Wells
Technologies for Cloud Data Warehousing
8
Migration Tools – Data Virtualization

9. © David L Wells
Step-by-Step Data Warehouse Migration
9
The Big Picture
Migration
Technology Selection
Migration Strategy
Architectural Assessment
Business Case
Planning
Testing and Operationalization
incrementalmigration
Scope, Timing, Resources, Schedule,
User Transparency, Testing Plan
Drivers, Costs, Benefits, Risk of Migrating,
Risk of Not Migrating
Reliability, Availability, Performance,
Scalability, Adaptability, Maintainability
Lift and Shift or Incremental by Workload,
Workload Breakdown and Priorities
Cloud Data Warehousing Platform,
Migration Tools
Schema, Data, Process, Metadata,
Users and Applications
Function Test, Performance Test, DQ Audit,
Scheduling, Monitoring, Support

10. © David L Wells
Step-by-Step Data Warehouse Migration
10
Business Case
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
What are the drivers to move to the cloud?
What are the business benefits? Who cares about them?
What are the technical benefits? Who cares about them?
What are the business disadvantages of not migrating? Who feels the pain?
What are the technical disadvantages of not migrating? Who feels the pain?

11. © David L Wells
Step-by-Step Data Warehouse Migration
11
Architectural Assessment
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case Architecture
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Business Architecture
Organization Architecture
Data Architecture
Integration Architecture
Technology Architecture
Reliability Availability Performance Scalability Adaptability Maintainability
Suited to purpose
Fits gracefully into the environment
Structurally sound
Compliant with codes and regulations
Sustainable through expected lifespan
Aesthetically pleasing

12. © David L Wells
Step-by-Step Data Warehouse Migration
12
Architectural Assessment
Change Warehouse Positioning
Change Data Flow
Change Data Models
Change Data Stores
Change Technology
Change Architectural Concept
Beside data lake, inside data lake …
Landing, staging, warehouse, data marts …
Normalized, denormalized, dimensional …
Divide, combine, partition, retire …
Automation, virtualization ...
Hub-and-spoke, bus, hybrid …
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Migrate as is
Review and refine
Redesign
Architecture

13. © David L Wells
Step-by-Step Data Warehouse Migration
13
Migration Strategy
COMPLEX MEGA-PROJECT
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Migrate as is
Review and refine
Redesign
Lift and shift
• By pain points
• By subject area
• By data source
• By user groups
Migration StrategyArchitecture

14. © David L Wells
Step-by-Step Data Warehouse Migration
14
Migration Strategy
1
2
3
Incremental migration of individual
workloads on a case-by-case basis.
• When to migrate?
• Migrate as is?
• Modify and migrate?
• Replace with new data mart?
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Migrate as is
Review and refine
Redesign
Lift and shift
• By pain points
• By subject area
• By data source
• By user groups
Migration StrategyArchitecture

15. © David L Wells
Step-by-Step Data Warehouse Migration
15
Technology Selection
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Migrate as is
Review and refine
Redesign
Lift and shift
• By pain points
• By subject area
• By data source
• By user groups
Migration Strategy
Cloud platform
Migration tools
Technology
Selection
Architecture

16. © David L Wells
Step-by-Step Data Warehouse Migration
16
Migration
Plan
Migrate
Test
Operationalize
Agility
Performance
Growth
Cost Savings
Labor Savings
Business Case
Current State
Assessment
of Data
Warehouse
Architecture
Good
Okay
Flawed
Migrate as is
Review and refine
Redesign
Lift and shift
• By pain points
• By subject area
• By data source
• By user groups
Migration Strategy
Cloud platform
Migration tools
Technology
Selection
Migration
pipelines
ETL meta
data
dataschema
users
&
apps
Architecture

17. © David L Wells
Step-by-Step Data Warehouse Migration
17
Schema Migration
pipelines
ETL meta
data
dataschema
users
&
apps
Do you need to
change …
• Structure
• Indexing
• Partitioning
• Optimization
• Pre-Joining
• Derivation
• Aggregation

18. © David L Wells
Step-by-Step Data Warehouse Migration
18
Data Migration
pipelines
ETL meta
data
dataschema
users
&
apps
How much data are you moving?
What is your network capacity and what else uses the network?
How long will it take to migrate and what can you do to accelerate?
Do you need to transform data due to schema adjustment?
Should you transform in stream or pre-process?

19. © David L Wells
Step-by-Step Data Warehouse Migration
19
ETL
pipelines
ETL meta
data
dataschema
users
&
apps
Change the code base to optimize for platform performance?
Change data transformations to sync with data restructuring?
Reorganize data flows?
Reduce data latency?
Migrate ETL processing to the cloud?

20. © David L Wells
Step-by-Step Data Warehouse Migration
20
Data Pipelines
pipelines
ETL meta
data
dataschema
users
&
apps
Rebuild pipelines instead of migrating existing ETL?
Package individual transform actions as executable objects?
Assemble objects as modules?
Configure modules for workflow and dataflow?
Gain performance, agility, or maintainability?

21. © David L Wells
Step-by-Step Data Warehouse Migration
21
Metadata
pipelines
ETL meta
data
dataschema
users
&
apps
Source-to-target mappings and tracing data lineage?
Can metadata be readily moved to cloud platform?
Can you export and import metadata?
Reverse engineer or rebuild from scratch?

22. © David L Wells
Step-by-Step Data Warehouse Migration
22
Users & Applications
pipelines
ETL meta
data
dataschema
users
&
apps
Uninterrupted business operations
Security and access authorizations
Communication and coordination
Connecting BI and analytics tools and applications

23. © David L Wells
Bringing it all Together
23
Cloud Data Warehouse (CDW) - Many good solutions are
available. Select the one that is the best fit with your business
model, your budget and your existing systems.
Integration Platform as a Service (iPaaS) – Use to connect and
migrate data from multiple, different endpoints. Make sure your
iPaaS is strong not just for application integration but also cloud
DW, big data & data lake integration.
Data Warehouse Automation (DWA) – Ideal for schema
replication from the legacy data store to the new CDW and for
schema management. Select a DWA that can best automate routine
developer tasks.
Data Virtualization (DV) – Use to provide easier access to data by
your “customers” using a modern interface. Ideal for incremental
data migration.

24. © David L Wells
Getting Started with Cloud Data Warehousing
24
What’s Next?
Should your data warehouse move to the cloud?
What benefits would you get from cloud data warehousing?
What challenges and risks will you face?
How would you approach migrating your data warehouse?
What people, tools, and resources will you need?

25. © David L Wells
Questions …

26. © David L Wells
Get the Whitepaper at SnapLogic.com/resources
Email me at dwells@eckerson.com
Follow my blogs at eckerson.com/blogs/data-management
Thank You!
Free Demo available on SnapLogic.com