Migrating data has become one of the most challenging initiatives for IT managers. Although these projects yield high business benefits (such as cost savings, increased productivity, and improved data manageability), they tend to involve a high level of risk due to the volume and criticality of the data being moved. So What Is The #1 Reason Why Many Data Migration Projects Fail? Failure to validate and test throughout the data migration process. You can save your project from this common pitfall! Attend the webinar to learn: - Why validation and testing is so important - How to integrate data testing into your project schedule - How to automate data validation and testing - How to show project stakeholders that your migration is free of bad data - What tool and process will help formalize your validation and testing

1. QuerySurge
Bill Hayduk
CEO
Christopher Thompson
Senior Domain Expert
Data Migrations:
How to avoid the #1 project pitfall

2. Today’s Agenda
• Data Migration
o Business Reasons
o Risks
o Costs
o Scary Metrics
o Common Defects
• #1 Project Pitfall
• QuerySurge Solution
• Demo
AGENDA
Data Migration
o Business Reasons
o Risks
o Costs
o Scary Metrics
o Common Defects
#1 Project Pitfall
QuerySurge Solution
Demo
built by

3. QuerySurge™
FACTS
Founded:
1996
Locations:
New York, NY
(Headquarters)
Customer profile:
• Fortune 1000
• 700+ customers
Software Offering
QuerySurge (2011)
Tech Partners:
IBM, Microsoft, HPE,
Oracle, Teradata,
Cloudera,
HortonWorks, MongoDB
RTTS is the parent company of QuerySurge
and is the premier pure-play QA & Testing organization
that specializes in test automation
a software division of

4. 80 percent of data migration projects fail to meet
expectations, running over time and budget.
- Gartner
83 percent of migrations experience problems
- IBM survey
80% of organizations… will underestimate the costs
related to the data acquisition tasks by an average
of 50 percent
- Gartner
Data Migrations
built by
QuerySurge™

5. Data Migration – Top Reasons
Top reasons for Data Migration
• Decommissioning legacy systems
• Adding new line of business
• New deployment (i.e. ERP or CRM system)
• Consolidation of systems
• Relocation
• Switching vendors
built by
QuerySurge™

6. • the average cost for a Data Migration = $875,000
• 34% of migrations have missing or lost data
• 38% have some form of data corruption
• 64% of migration projects have unexpected
outage/downtime
• typical cost of downtime:
o $6.5 million/hour - Brokerage industry
o $2.8 million/hour - Energy Industry
• 72% of organizations deferred because data migration is
too risky
Data Migration – Scary Metrics
built by
QuerySurge™

7. • Unexpected downtime
• Budget overruns
• Customer or Brand Impact
• Data corruption
• Application performance issues
• Data loss
Data Migration – Key Risks
Key Risks
built by
QuerySurge™

8. • Remediation activities
• Server outage
• Application outage
• Labor costs
• Software Tools
• System Maintenance
Data Migration – Cost Factors
Key Cost Factors
built by
QuerySurge™

9. Data Migration – Key Pitfalls
• Failing to engage the lines of business at the outset
• Absence of data governance policies and organizational
structure
• Poor data quality in a legacy system
• Neglecting to validate and redefine business rules
• Failure to validate and test the data-migration process
Key Pitfalls of Migration Projects
• Failure to validate and test the data-migration process
built by
QuerySurge™

10. "You really need to make sure that you're validating and testing
throughout the process”.
“Unfortunately companies often don't spend enough time
aligning the data testing, validation, and migration cycles to the
project timeline”. - InformationWeek
Question:
How are you going to test the data?
Failure to validate and test the process
built by
QuerySurge™

11. Data Migration – Popular Test Strategy
Post-Migration Testing
• Row Counts
• Comparing migrated records to
source using sampling
Question:
Is there a better way?
built by
QuerySurge™

12. 12
Our Solution:
Automated Testing of the
Migration Process
built by
QuerySurge™

13. a software division of

14. is the leading Data Testing solution
for automated validation & testing of Big Data
QuerySurge
QuerySurge™
Use Cases
a software division of

15. • Reduce your costs & risks
• Improve your data quality
• Accelerate your testing cycles
• Share information with your team
with QuerySurge™ you can:
built by
QuerySurge™
• Provides huge ROI (i.e. 1,300%)*
*based on client’s calculation of Return on Investment

16. QuerySurge™
Automate the entire testing cycle
 Automate the test execution, the comparison, auto-email results
Query Wizards - no coding needed
 Query Wizards ensure minimal time & effort to create tests
Test across different platforms
 Data Warehouse, Hadoop, NoSQL, DB, flat files, XML, mainframe
Analyze & Collaborate
 Data Health dashboard, shared tests & auto-emailed reports
Verify more data & do it quickly
 Verifies up to 100% of all data up to 1,000 x faster
Integrate for Continuous Delivery (DevOps)
 Integrates with most Build, Configuration, ETL & QA mgmt solutions
a software division of

17. Web-based…
Installs on...
Connects through…
…to any JDBC compliant data store
QuerySurge™
QuerySurge
Controller
QuerySurge
Server
QuerySurge
Agents
a software division of
Installs in...
the Cloud a VM

18. QuerySurge™
a software division of
QuerySurge supports the following JDBC drivers:
• Amazon Redshift
• Cassandra
• Couchbase
• Exasol
• Flat Files (delimited, fixed-width)
• JSON
• Hadoop/Hive (Apache, Cloudera, Hortonworks, Pivotal)
• HPE Vertica
• IBM (DB2, Netezza, Informix)
• Mainframe
• MAPR
• Microsoft (SQL Server, PDW, SSAS, Excel, Access, Sharepoint)
• MongoDB
• Oracle (Oracle, MySQL, Exadata)
• Pivotal GreenPlum
• PostgreSQL
• Salesforce
• SAP (HANA, IQ, ASE, SQL Anywhere)
• Spark
• Teradata, Aster
• WorkDay
• XML
And any other JDBC-compliant data store

19. 6/8/2017 19
Developer: Codes data movement based on Mapping Requirements
Target Data
Warehouse
ETL
Tester: Tests data movement based on Mapping Requirements
Data Mart
ETL
Source Data
Target Big Data
Mainframe
and other ETL tools…

20. QuerySurge™
Design
Library
SchedulingRun
Dashboard
Deep-Dive
Reporting
Data Health
Dashboard
Query
Wizards
a software division of

21. Fast and Easy.
No programming needed.
QuerySurge™
• Perform 80% of all data tests with no SQL coding
• Opens up testing to novices & non-technical members
• Speeds up testing for skilled coders
• provides a huge Return-On-Investment
a software division of

22. Design Library
• Create custom Query Pairs (source & target SQLs)
• Great for team members skilled with SQL
Scheduling
 Build groups of Query Pairs
 Schedule Test Runs for:
• immediately
• at a specific date/time
• automatically after build or
ETL process
built by
QuerySurge™

23. Design Library
• Create custom Query Pairs (source & target
SQLs for tests that have transformations)
Scheduling
 Build groups of Query Pairs
 Schedule Test Runs
• Run immediately
• Run at set date/time
• Have event kick it off
QuerySurge™
a software division of

24. Deep-Dive Reporting
 Examine and automatically
email test results
Run Dashboard
 View real-time execution
 Analyze real-time results
QuerySurge™
a software division of

25. QuerySurge™
• view data reliability & pass rate
• add, move, filter, zoom-in on any
data widget & underlying data
• verify build success or failure
a software division of

26. QuerySurge™
 Drive QuerySurge execution from your Test Management Solution
 See QuerySurge Pass/Fail results in your Test Management solution
 Click link to drill into detailed results in QuerySurge
Integration with leading
Test Management Solutions
• HPE ALM (Quality Center)
• Microsoft Team Foundation Server
• IBM Rational Quality Manager
a software division of

27. QuerySurge™
Automated
Testing
Automated
Reporting
Automated
Launch
Data Integration/ETL
solutions
QuerySurge™
and many others…
email
report
and many others…
QuerySurge™
Build/Configuration
solutions
email
report
Test Management
solutions
QuerySurge™
email
report
a software division of

28. QuerySurge™
a software division of
A 2-Step Process

29. Target Database
Data Migration: Data Flow
Source Data Migration Process
ETL &
EL
SQL
SQL
SQL
SQL
SQL
SQL
built by
QuerySurge™

30. • Automated Row Counts
• Automated Column Compares (no transformations)
• Automated Table Compares (no transformations)
#1: Automated testing with Query Wizards (no programming)
#2: Automated testing with custom QueryPairs (custom SQL)
• All tables with transformations are compared
• Upwards of 100% coverage of all data quickly
SQL
SQLSQL
SQL
SQL
SQL SQL
SQL
2 Steps:
built by
QuerySurge™

31. built by
QuerySurge™
3 Types of Data Comparison Wizards:
The also provide you with automated features for:
o filtering (‘Where’ clause) and
o sorting (‘Order By’ clause)
Column-Level Comparison:
This is great for Big Data stores and Data Warehouses where tables will have some columns
containing transformations and some columns with no transformations. Many tables and
columns can be compared simultaneously and quickly.
Table-Level Comparison:
This comparator is great for Data Migrations and Database Upgrades with no
transformations at all. Many tables can be compared simultaneously and quickly.
Row Count Comparison:
Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades.
Many tables and rows can be compared simultaneously and quickly.

32. 32
Fast and Easy. No programming needed.
• Perfect for data migrations & upgrades
• Great for verifying that all rows have
come over without data issues
• Superb at verifying table-to-table
transfers
• Ability to graphically create SQL stubs
for advanced scripting
Step #1:
built by
QuerySurge™

33. QuerySurge™
a software division of

34. Uses:
• Verify that the amount of rows match
the amount that were expected
• Great for smoke testing
Result:
Quickly compares rows of all tables
Skill set / Value added:
beginner: no coding needed
experienced user: saves time
built by
QuerySurge™

35. Uses:
Tests the columns that have no
transformations, which means it tests
approximately 80% of your data store without
you writing any SQL code
Tests:
Big data, data warehouses
Skill set / Value added:
novice or non-technical: no coding needed,
productive immediately
experienced user: saves time
built by
QuerySurge™

36. Uses:
- Verifies tables from source
and target match
Result:
Can perform 100s of tables in
a few minutes.
Skill set / Value added:
beginner: no coding needed
experienced user: saves time
built by
QuerySurge™

37. Step #2:
built by
QuerySurge™
Uses:
- Create custom tests to mimic your transformations
Result:
- Can test all of you transformed data in your columns
Skill set/Value added:
- Need SQL skills. Can test all of your data
Design Library
• Create custom Query Pairs (source & target SQLs for tests that have transformations)

38. • Improve your data governance & data quality
• Accelerate your data delivery cycles
• Reduce your costs & risks
• Realize a huge ROI
QuerySurge™
a software division of

39. Ensuring Data Warehouse Qualitybuilt by
QuerySurge™
Christopher Thompson
Senior Domain Expert
QuerySurge
To see the video of this demonstration please visit:
http://bit.ly/2sHWyO7

40. QuerySurge
built by
QuerySurge™ a software division of
Any questions
before the
demo?

41. 41
• SAP Data Migration: Turning Risk into Opportunity, Jerome Delune, EIM Global Practice
Director, SAP Consulting
• Data Migration Best Practices and Nondisruptive Migration Service Capability for
Enterprise Storage, Hitachi Data Systems
• Risks and Challenges in Data Migrations and Conversions, Gartner
• Data Migration – Key Considerations, Glen Bell, SearchDatamanagement.com
• Data Migration in the Global 2000, Bloor Research
• Zero-Downtime Upgrades and Migrations with Oracle GoldenGate, Oracle
• 10 Big Data Migration Mistakes, InformationWeek
• How to Implement an Effective Data Migration Testing Strategy, DataMigration Pro
magazine, David Katzoff, Managing Director of Product Development, Valiance Partners
• The Hidden Costs of Data Migration, IBM
References
built bybuilt by