More Related Content Similar to 6 Principles of Modern Change Data Capture: How to Build Fast, Agile, Reliable Data Pipelines (20) More from Enterprise Management Associates (20) 6 Principles of Modern Change Data Capture: How to Build Fast, Agile, Reliable Data Pipelines1. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
6 Principles of Modern, Multi-Modal Change Data Capture
- How to Build Fast, Agile, Reliable Data Pipelines
John Santaferraro
Research Director
EMA
Nir Livneh
Founder and CEO
Equalum
2. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Watch the On-Demand Webinar
Slide 2
6 Principles of Modern, Multi-Modal Change Data Capture On-
Demand webinar: https://info.enterprisemanagement.com/six-
principles-of-modern-change-data-capture-webinar-ws
Check out upcoming webinars from EMA here:
https://www.enterprisemanagement.com/freeResearch
3. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Featured Speakers
Slide 3 © 2020 Enterprise Management Associates, Inc.
John Santaferraro, Research Director, EMA
John is the research director for analytics, business intelligence, and data management
at EMA. He has 23 years of experience in data and analytics, from startups to executive
positions at Fortune 50 companies. His deep understanding of the industry comes from
years of leadership in implementation, product and marketing organizations, along with
multiple big data imagineering efforts for finance, communications, retail, manufacturing,
healthcare, events, oil and gas, and utilities.
Nir Livneh, Founder and CEO, Equalum
Nir is a thought leader with over 20 years of experience in big data architecture and
performance. He led product management at Quest Software (Acquired by Dell) for all
big data products. Nir also led big data architecture projects at the Israeli Military
Intelligence unit (Equivalent to NSA). He is currently a member of the Forbes
Technology Council.
4. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Agenda
4 © 2020 Enterprise Management Associates, Inc.
• Modernization Trends and the EMA Hybrid Data Ecosystem
• EMA Advisory: 6 Principles of Modern, Multi-Modal CDC
• EMA Guidance: Additional Checklist Items
• EMA Best Practices for Modern, Multi-Modal CDC Implementation
• Equalum Data Ingestion Platform
• Market Perspectives: The Advantages of Modern, Multi-Modal CDC
5. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Trends Driving the Need for
Modern, Multi-Modal CDC
5 © 2020 Enterprise Management Associates, Inc.
6. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
• Digital superiority is the new
mandate.
• Real time is the new
measurement.
• Intelligent automation is the
new means.
6 © 2020 Enterprise Management Associates, Inc.
The Post-Pandemic Digital Shift
7. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
EMA Hybrid Data Ecosystem Architecture
• No one single platform for
all use cases
• New platforms added
based on value creation
• Interoperability is a must,
must play will with others
Slide 7 © 2020 Enterprise Management Associates, Inc.
8. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
HDE Composition: Number of Platforms
Slide 8 © 2020 Enterprise Management Associates, Inc.
9. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Operating a Hybrid Data Ecosystem
• Orchestration defines the building of data pipelines to
acquire, prepare, provision, and adapt data for consistent
use across the entire data and application landscape.
• Operation is the delivery, administration, security,
availability, and automation of data movement among all
platforms and applications within the ecosystem.
• Optimization is the process of continually improving the
operation and orchestration of data pipelines.
• Obsolescence is the means by which legacy data
integrations and pipelines reach end of life and migrate to
new systems.
Slide 9 © 2020 Enterprise Management Associates, Inc.
10. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Consumer-Centric Design
The most important part
of the Hybrid Data
Ecosystem is outside
the technology circle:
the consumers.
Slide 10 © 2020 Enterprise Management Associates, Inc.
11. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
EMA Advisory:
Six Principles of
Modern, Multi-Modal CDC
Slide 11 © 2020 Enterprise Management Associates, Inc.
12. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC
• It is modern because it works in modern data ecosystems
• It is multi-modal because it supports…
• Real time and batch (fresh data, fast data)
• CDC and transformation
• Multi-structured data
• Multiple modes of replication
• Multi-directional data pipelines
• Hybrid ecosystems
Slide 12 © 2020 Enterprise Management Associates, Inc.
13. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Principle #1: Enable Comprehensive Data
Replication for CDC and Beyond
13 © 2020 Enterprise Management Associates, Inc.
Streaming-First
Track data from
the point of the
event.
Rapid Replication
Replicate
changes as they
occur in near real
time.
Broad Replication
Initial data
capture
Schema migration
Grouping
Batch
14. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Principle #2: Streamline Interconnectivity
Among Broad Set of Sources and Targets
14 © 2020 Enterprise Management Associates, Inc.
• Sources become targets and targets become sources.
Technology Ecosystem Support
• Applications: SaaS and On-
Premises
• Data: Cloud and On-Premises
• Platforms: SQL, NoSQL,
Blockchain, others
Interconnectivity
Data Preparation
Bi-Directional
15. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
• Structured Data
• Multi-schema
• Semi Structured Data
• All File Formats
• Complex File Formats
o JSON, XML
• Mixed File Formats
o JSON in XML, XML in JSON
Standardization
Single Set of Operators
Easy to Combine
Common Ingestion
15 © 2020 Enterprise Management Associates, Inc.
Principle #3: Standardize Support
for Multi-Structured Data
16. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
• Transformation
• Variables
• Aggregation
• Summarization
• Concatenation
• Correlations
• Others
Internet of Things:
Data collected by sensors often
includes irrelevant data and
requires enrichment before it is
analyzed. Data must be captured,
filtered, and enriched before it is
analyzed and elicits an intelligent
response.
16 © 2020 Enterprise Management Associates, Inc.
Principle #4: Combine CDC and
Transformation for All Data Streams
17. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
• Orchestration should automate
the resolution of conflicts using
priority based on performance,
business criticality, and
dependencies.
• Optimization should continually
improve the operation and
orchestration of data pipelines.
17 © 2020 Enterprise Management Associates, Inc.
Principle #5: Orchestrate and Optimize
Complex Stream-Based Operations
Abstraction Layer
Pipeline Complexity
Open Source Complexity
18. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING18 © 2020 Enterprise Management Associates, Inc.
Principle #6: Create Interoperability in
Hybrid Environments
• 53% of all data is now in the cloud
• Interoperability includes CDC, replication, migration, and multi-
directional data pipelines.
On-premises Cloud SaaS Distributed Internet
Providers Apps of Things
19. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
EMA Guidance:
Additional Checklist Items for
Modern, Multi-Modal CDC
Slide 19 © 2020 Enterprise Management Associates, Inc.
20. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
20 © 2020 Enterprise Management Associates, Inc.
Simple, easy to use
Outperform legacy platforms
Agile for changing data and business requirements
Reliable to the point of unbreakable.
Evolve with the latest open source projects
21. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
21 © 2020 Enterprise Management Associates, Inc.
Simple, easy to use
• Drag and drop
• No code
• Single pane of glass
22. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
22 © 2020 Enterprise Management Associates, Inc.
Outperform legacy platforms
• Performance speed
• Elasticity
• Time to Capture
• Time to Transform
• Time to Ingestion
• Time to Response
23. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
23 © 2020 Enterprise Management Associates, Inc.
Agile for changing data and business requirements
(Increase in data drift)
• Simple monitoring, versioning, etc.
• Auto-detection
• Auto-update
• Simple manual changes
• Exactly once in distributed systems
24. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
24 © 2020 Enterprise Management Associates, Inc.
Reliable to the point of unbreakable.
of participants are
spending between
30% and 50% of
their time fixing
breaks in data
pipelines.
61%
25. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC Checklist
25 © 2020 Enterprise Management Associates, Inc.
Evolve with the latest open source projects
26. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
EMA Best Practices:
Modern, Multi-Modal CDC
Implementation
Slide 26 © 2020 Enterprise Management Associates, Inc.
27. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Enrich and combine data streams for more
intelligence.
27 © 2020 Enterprise Management Associates, Inc.
• Enrich and combine data streams for more intelligence.
• Standardize operators on data streams for dev speed.
• Audit data streams for greater accuracy.
28. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTINGSlide 28 © 2020 Enterprise Management Associates, Inc.
Data Ingestion Platform with
Infinite Scalability & Speed
29. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING29
Equalum is an end-to-end streaming-first data ingestion platform Scalable
Fully Orchestrated
Easy to Deploy & Use
When choosing a
Data Ingestion Platform,
look for:
Performance & Reliability
Ease of Use
Broad Support for Sources &
Targets
Multi-Structured Data Support
Extensive ETL-Like Operations
OSS Foundation
Flexible Deployment
30. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING30 © 2020 Enterprise Management Associates, Inc.
© 2020 Equalum, Inc. All rights reserved.
Equalum Dashboard
31. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING31 © 2020 Enterprise Management Associates, Inc.
© 2020 Equalum, Inc. All rights reserved.
Zero-Coding, Drag-and-Drop Data Pipelines
32. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Market Perspectives:
The Advantages of
Modern, Multi-Modal CDC
Slide 32 © 2020 Enterprise Management Associates, Inc.
33. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC v. DIY
33 © 2020 Enterprise Management Associates, Inc.
Modern, Multi-Modal CDC Do-It-Yourself
Time to Insight Hours Months
CDC Built In Integration Required
Broad Replication Built In Integration Required
Transformation Built In Integration Required
Source/Target Support Built In Separate Connectors
Multi-Structed Data Built In Separate Convertors
Hybrid Environments Built In Integration Required
Enrichment Built In Difficult to Build
Reliability Built In Difficult to Build
Agility Built In Difficult to Build
Ease of Use Built In Extremely Difficult
Validation Built In Integration Required
Orchestration Built In Difficult to Build
Optimization Built In Difficult to Build
34. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC v. Legacy CDC
and Integration
34 © 2020 Enterprise Management Associates, Inc.
Modern, Multi-Modal CDC Legacy CDC, Integration
Time to Insight Hours Weeks
CDC Built In Legacy Only
Broad Replication Built In Legacy Only
Transformation Built In Built for Legacy
Source/Target Support Built In Mostly Legacy
Multi-Structed Data Built In Legacy Only
Hybrid Environments Built In Separate Products
Enrichment Built In Legacy Only
Reliability Built In Legacy Only
Agility Built In Legacy Only
Ease of Use Built In Legacy Only
Validation Built In Legacy Only
Orchestration Built In Legacy Only
Optimization Built In Legacy Only
35. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Modern, Multi-Modal CDC v. Modern CDC
35 © 2020 Enterprise Management Associates, Inc.
Modern, Multi-Modal CDC Modern CDC
Time to Insight Hours Limited
CDC Built In Little or No Legacy Support
Broad Replication Built In Little or No Legacy Support
Transformation Built In Little or No Legacy Support
Source/Target Support Built In Built In
Multi-Structed Data Built In Minimal Legacy Support
Hybrid Environments Built In Minimal Legacy Support
Enrichment Built In Minimal Legacy Support
Reliability Built In Minimal Legacy Support
Agility Built In Minimal Legacy Support
Ease of Use Built In Minimal Legacy Support
Validation Built In Minimal Legacy Support
Orchestration Built In Minimal Legacy Support
Optimization Built In Minimal Legacy Support
36. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING36 © 2020 Enterprise Management Associates, Inc.
Equalum’s
Vision
for the
Future
37. IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Question and Answer: Log Questions in the Q&A panel
© 2020 Enterprise Management Associates, Inc.
Interested in Learning More?
LEARN ABOUT:
• Main Benefits of Using CDC
• 3 Common Approaches to CDC
• Disadvantages Enterprises Experience
When Relying On Legacy Tools
DOWNLOAD
www.equalum.io/modernize-wp