Welcome to my post on ‘Architecting Modern Data Platforms’, here I will be discussing how to design cutting edge data analytics platforms which meet the ever-evolving data & analytics needs for the business.
https://www.ankitrathi.com
2. Content
• Data Architecture Principles
• Data Lake Basics
• High Level Architecture
• Data Characteristics
• Putting It All Together
• Product-Driven Data Architecture
• Reference Architecture
3. Data Architecture Principals
• Adhere to ADDA (Accessibility, Definition, Decoupling, Agility)
• Design for RSM (Reliability, Scalability, Maintainability)
• Use Right Tools
• Cloud Native/Agnostic
• Be Cost Conscious
4. Adhere to ADDA
Accessibility
Easily accessible data
for business
Definition
Data catalog for
simplified data
discovery
Decoupling
Decoupled layers for
flexibility
Agility
Agile enough to cater
evolving business
requirements
6. Use Right Tools
Data Structure
Structured, Semi-
structured, Unstructured
Latency
Low, Medium, High
Throughput
High, Medium, Low
Access Pattern
Key-value, Search,
Transactions
7. Cloud Native/Agnostic
Cloud Native Cloud Agnostic
Pros:
• Better performance
• Better efficiency
• Lower costs (generic services)
Pros:
• Flexibility
• Minimal vendor lock-in
• Standard performance
Cons:
• Vendor lock-in
• Higher costs (specific services)
Cons:
• Underutilization of vendor capabilities
• Solution can become complex
• Performance, logging and monitoring
can take a hit
8. Be Cost Conscious
• Efficient consumption of services
• Select cost-conscious options
• Enforce policies and controls
9. Data Lake
• Data Lake Definition
• An architectural approach
• Massive heterogenous data stored centrally
• Available to diverse group of users
• To be categorized, processed, analyzed & consumed
• Data Lake Characteristics
• Structured, semi-structured & unstructured data
• Scaled out as required
• Diverse set of storage, analytics and ML/AI tools
• Designed for low-cost storage and analytics
11. Ingest
Source Data Type Data
Web/Mobile Apps Records Transactions
Databases Records Transactions
Logging Search documents Files
Logging Log files Files
Messaging Messages Events
IoT Data Streams Events
12. Data Characteristics
Hot Warm Cold
Volume MB-GB GB-PB PB-EB
Item Size B-KB KB-MB KB-TB
Latency ms ms, sec min, hrs
Durability Low-high High Very high
Request Rate Very high High Low
Cost/GB $$-$ $-¢¢ ¢¢-¢
13. Data Characteristics
• Type of Data Structures
• Fixed Schema
• Schema Free
• Key-Value
• Type of Access Patterns
• Key-Value
• Simple relations (1:N, M:N)
• Multi-table joins, transactions
• Faceting, Search