Suche senden
Hochladen
YugaByte DB Internals - Storage Engine and Transactions
•
Als PPTX, PDF herunterladen
•
7 gefällt mir
•
2,687 views
Yugabyte
Folgen
Presented at NorCal DB Day https://sites.google.com/view/norcaldb18/home
Weniger lesen
Mehr lesen
Software
Melden
Teilen
Melden
Teilen
1 von 51
Jetzt herunterladen
Empfohlen
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data Analytics
Alluxio, Inc.
Distributed SQL Databases Deconstructed
Distributed SQL Databases Deconstructed
Yugabyte
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
StreamNative
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...
Databricks
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks
[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf
[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf
Chris Hoyean Song
Empfohlen
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data Analytics
Alluxio, Inc.
Distributed SQL Databases Deconstructed
Distributed SQL Databases Deconstructed
Yugabyte
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
StreamNative
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...
The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analy...
Databricks
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks
[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf
[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf
Chris Hoyean Song
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQL
Yugabyte
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
HostedbyConfluent
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
Deep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
Amazon Web Services
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
Apache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the Covers
ScyllaDB
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdf
Alkin Tezuysal
How Uber scaled its Real Time Infrastructure to Trillion events per day
How Uber scaled its Real Time Infrastructure to Trillion events per day
DataWorks Summit
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Amazon Web Services
Building an open data platform with apache iceberg
Building an open data platform with apache iceberg
Alluxio, Inc.
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
ScaleGrid.io
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Databricks
Introduction to Greenplum
Introduction to Greenplum
Dave Cramer
An Introduction to Druid
An Introduction to Druid
DataWorks Summit
Apache Iceberg Presentation for the St. Louis Big Data IDEA
Apache Iceberg Presentation for the St. Louis Big Data IDEA
Adam Doyle
A Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
confluent
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
HostedbyConfluent
Using ClickHouse for Experimentation
Using ClickHouse for Experimentation
Gleb Kanterov
YugaByte + PKS CloudFoundry Meetup 10/15/2018
YugaByte + PKS CloudFoundry Meetup 10/15/2018
AlanCaldera
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
Carlos Andrés García
Weitere ähnliche Inhalte
Was ist angesagt?
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQL
Yugabyte
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
HostedbyConfluent
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
Deep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
Amazon Web Services
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
Apache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the Covers
ScyllaDB
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdf
Alkin Tezuysal
How Uber scaled its Real Time Infrastructure to Trillion events per day
How Uber scaled its Real Time Infrastructure to Trillion events per day
DataWorks Summit
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Amazon Web Services
Building an open data platform with apache iceberg
Building an open data platform with apache iceberg
Alluxio, Inc.
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
ScaleGrid.io
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Databricks
Introduction to Greenplum
Introduction to Greenplum
Dave Cramer
An Introduction to Druid
An Introduction to Druid
DataWorks Summit
Apache Iceberg Presentation for the St. Louis Big Data IDEA
Apache Iceberg Presentation for the St. Louis Big Data IDEA
Adam Doyle
A Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
confluent
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
HostedbyConfluent
Using ClickHouse for Experimentation
Using ClickHouse for Experimentation
Gleb Kanterov
Was ist angesagt?
(20)
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQL
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Deep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
Apache Iceberg: An Architectural Look Under the Covers
Apache Iceberg: An Architectural Look Under the Covers
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdf
How Uber scaled its Real Time Infrastructure to Trillion events per day
How Uber scaled its Real Time Infrastructure to Trillion events per day
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Deep Dive on the Amazon Aurora PostgreSQL-compatible Edition - DAT402 - re:In...
Building an open data platform with apache iceberg
Building an open data platform with apache iceberg
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Introduction to Greenplum
Introduction to Greenplum
An Introduction to Druid
An Introduction to Druid
Apache Iceberg Presentation for the St. Louis Big Data IDEA
Apache Iceberg Presentation for the St. Louis Big Data IDEA
A Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Using ClickHouse for Experimentation
Using ClickHouse for Experimentation
Ähnlich wie YugaByte DB Internals - Storage Engine and Transactions
YugaByte + PKS CloudFoundry Meetup 10/15/2018
YugaByte + PKS CloudFoundry Meetup 10/15/2018
AlanCaldera
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
Carlos Andrés García
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
VMware Tanzu
Timesten Architecture
Timesten Architecture
SrirakshaSrinivasan2
Scale Transactional Apps Across Multiple Regions with Low Latency
Scale Transactional Apps Across Multiple Regions with Low Latency
Yugabyte
times ten in-memory database for extreme performance
times ten in-memory database for extreme performance
Oracle Korea
Kudu: Fast Analytics on Fast Data
Kudu: Fast Analytics on Fast Data
michaelguia
Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWS
Kimmo Kantojärvi
Running Stateful Apps on Kubernetes
Running Stateful Apps on Kubernetes
Yugabyte
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
Hakka Labs
YugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on Kubernetes
DoKC
Brian Bulkowski. Aerospike
Brian Bulkowski. Aerospike
Volha Banadyseva
Capital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream Processing
confluent
Leveraging Scala and Akka to build NSDb
Leveraging Scala and Akka to build NSDb
radicalbit
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
HostedbyConfluent
Container Attached Storage (CAS) with OpenEBS - SDC 2018
Container Attached Storage (CAS) with OpenEBS - SDC 2018
OpenEBS
Design Choices for Cloud Data Platforms
Design Choices for Cloud Data Platforms
Ashish Mrig
Apache Kudu: Technical Deep Dive
Apache Kudu: Technical Deep Dive
Cloudera, Inc.
Laboratorio práctico: Data warehouse en la nube
Laboratorio práctico: Data warehouse en la nube
Software Guru
Avoiding Common Pitfalls: Spark Structured Streaming with Kafka
Avoiding Common Pitfalls: Spark Structured Streaming with Kafka
HostedbyConfluent
Ähnlich wie YugaByte DB Internals - Storage Engine and Transactions
(20)
YugaByte + PKS CloudFoundry Meetup 10/15/2018
YugaByte + PKS CloudFoundry Meetup 10/15/2018
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
A Planet-Scale Database for Low Latency Transactional Apps by Yugabyte
Timesten Architecture
Timesten Architecture
Scale Transactional Apps Across Multiple Regions with Low Latency
Scale Transactional Apps Across Multiple Regions with Low Latency
times ten in-memory database for extreme performance
times ten in-memory database for extreme performance
Kudu: Fast Analytics on Fast Data
Kudu: Fast Analytics on Fast Data
Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWS
Running Stateful Apps on Kubernetes
Running Stateful Apps on Kubernetes
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
YugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on Kubernetes
Brian Bulkowski. Aerospike
Brian Bulkowski. Aerospike
Capital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream Processing
Leveraging Scala and Akka to build NSDb
Leveraging Scala and Akka to build NSDb
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
Container Attached Storage (CAS) with OpenEBS - SDC 2018
Container Attached Storage (CAS) with OpenEBS - SDC 2018
Design Choices for Cloud Data Platforms
Design Choices for Cloud Data Platforms
Apache Kudu: Technical Deep Dive
Apache Kudu: Technical Deep Dive
Laboratorio práctico: Data warehouse en la nube
Laboratorio práctico: Data warehouse en la nube
Avoiding Common Pitfalls: Spark Structured Streaming with Kafka
Avoiding Common Pitfalls: Spark Structured Streaming with Kafka
Kürzlich hochgeladen
MYjobs Presentation Django-based project
MYjobs Presentation Django-based project
AnoyGreter
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
OnePlan Solutions
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
Lionel Briand
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
Akihiro Suda
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
31events.com
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
Marharyta Nedzelska
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
BradBedford3
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
OnePlan Solutions
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Angel Borroy López
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
smiwainfosol
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
Christian Birchler
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Natan Silnitsky
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
Hanief Utama
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Cizo Technology Services
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdf
FerryKemperman
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
How To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTRO
motivationalword821
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
Łukasz Chruściel
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
Andreas Granig
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Ahmed Mohamed
Kürzlich hochgeladen
(20)
MYjobs Presentation Django-based project
MYjobs Presentation Django-based project
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdf
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
How To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTRO
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
YugaByte DB Internals - Storage Engine and Transactions
1.
1© 2018 All
rights reserved. Introducing YugaByte DB Kannan Muthukkaruppan, Co-Founder/CEO Mikhail Bautin, Co-Founder/Architect NorCal DB Day, May 2018
2.
2© 2018 All
rights reserved. About Us Kannan Muthukkaruppan, CEO Nutanix ♦ Facebook ♦ Oracle IIT-Madras, University of California-Berkeley Karthik Ranganathan, CTO Nutanix ♦ Facebook ♦ Microsoft IIT-Madras, University of Texas-Austin Mikhail Bautin, Software Architect ClearStory Data ♦ Facebook ♦ D.E.Shaw Nizhny Novgorod State University, Stony Brook Founded Feb 2016 Apache HBase committers and early engineers on Apache Cassandra Built Facebook’s NoSQL platform powered by Apache HBase Scaled the platform to serve many mission-critical use cases • Facebook Messages (Messenger) • Operational Data Store (Time series Data) Reassembled the same Facebook team at YugaByte along with engineers from Oracle, Google, Nutanix and LinkedIn Founders
3.
3© 2018 All
rights reserved. What is YugaByte DB? A transactional, high-performance database for building planet-scale cloud services.
4.
4© 2018 All
rights reserved. Why another database?
5.
5© 2018 All
rights reserved. Typical Stack Today Fragile infra with several moving parts Datacenter 1 SQL Master SQL Slave Application Tier (Stateless Microservices) Datacenter 2 SQL for OLTP data Manual sharding Cost: dev team Manual replication Manual failover Cost: ops team NoSQL for other data App aware of data silo Cost: dev team Cache for low latency App does caching Cost: dev team Data inconsistency/loss Fragile infra Hours of debugging Cost: dev + ops team
6.
6© 2018 All
rights reserved. Does AWS change this? Datacenter 1 SQL Master SQL Slave Datacenter 2 Elasticache Aurora DynamoDB Still Complex it’s the same architecture Application Tier (Stateless Microservices)
7.
7© 2018 All
rights reserved. TRANSACTIONAL PLANET-SCALEHIGH PERFORMANCE Distributed ACID Transactions Document-Based, Strongly Consistent Storage Low Latency, Tunable Reads High Throughput OPEN SOURCE Apache 2.0 Popular APIs Extended Apache Cassandra, Redis and PostgreSQL (BETA) Auto Sharding & Rebalancing Global Data Distribution YugaByte DB CLOUD-NATIVE Built For The Container Era Self-Healing, Fault-Tolerant
8.
8© 2018 All
rights reserved. Architecture tablet 1’ Portable across clouds tablet3-leader tablet2-leader tablet1-leader … … … tablet2-follower tablet2-follower tablet3-follower tablet3-follower tablet1-follower tablet1-follower SMACK Apps … Mature ecosystems tablet 1’ tablet 1’ tablet 1’ DocDB Storage Transactional key-document store, based on a heavily customized version of RocksDB Raft-Based Replication Highly resilient, used for both data replication & leader election node1 node2 node3 Flexible storage engine with single- row & multi-row ACID txns Transaction Manager Tracks ACID txns across multi-row ops, incl. clock skew mgmt. tablet1-leader tablet2-leader tablet3-leader tablet2-follower tablet3-follower tablet2-follower tablet3-follower tablet1-follower tablet1-follower … …… Automated Sharding & Load Balancing Popular APIs extended for app dev agility YCQL Cassandra-compatible YEDIS Redis-compatible BETA
9.
9© 2018 All
rights reserved. Architecture
10.
10© 2018 All
rights reserved. Design Goals ✓ Highly scalable & resilient ✓ Transactional - strong consistency ✓ All layers in C++ for high performance ✓ No dependencies on external systems ✓ Cloud-native – online re-configuration
11.
11© 2018 All
rights reserved. Consistency Goals – similar to Google Spanner CAP Consistency Partition Tolerant HA on failures – new leader elected in seconds PACELC No failure: Low latency On failure: Trade off latency for consistency
12.
12© 2018 All
rights reserved. API Goals – similar to Azure Cosmos DB ✓ Multi-model ✓ Start with well known APIs ✓ Extend to fill functionality gaps ✓ APIs supported Cassandra Query Language Redis PostgreSQL in works
13.
13© 2018 All
rights reserved. ACID Transactions Globally Consistent SQL API only Not Transactional Multi-Model High Performance Best of Cloud-Native Meets Open Source Not Globally Consistent Lower Performance
14.
14© 2018 All
rights reserved. DB Features SQL Strong consistency Secondary indexes ACID transactions Expressive query language NoSQL Tunable read latency Write optimized for large data sets Data expiry with TTL Scale out and fault tolerant
15.
15© 2018 All
rights reserved. Distributed ACID Transactions Multi-Row/Multi-Shard Operations At Any Scale YCQL
16.
16© 2018 All
rights reserved. Native JSON Data Type Modeling document & flexible schema use-cases YCQL
17.
17© 2018 All
rights reserved. Auto Data Expiry with TTL Database tracks and expires older data YCQL YEDIS Query the key right away Query the key after 10 seconds Write a key with a 10 second expiry
18.
18© 2018 All
rights reserved.
19.
19© 2018 All
rights reserved. Data Persistence in DocDB • DocDB is YugaByte DB’s LSM storage engine • Persistent key to data-structure store keys = ordered (composed of hash and range components) values = primitive (int32, double, etc.) or objects (maps, nested maps) • Extends and enhances RocksDB
20.
20© 2018 All
rights reserved. DocDB: A Key-to-Object/Document Store • Document key = CQL/SQL primary key or Redis key • Documents = CQL / SQL rows and Redis data structures
21.
21© 2018 All
rights reserved. DocDB: A Key-to-Object/Document Store Generated RocksDB keys have this format: DocKey, SubKey1, …, SubKeyN, Timestamp -> Value “Subkeys”: e.g. CQL/SQL column, Redis map key, etc. INSERT INTO products (prod_id, attrs, price) VALUES ('p1', {'h' : 7, 'w': 7}, 99) DocKey('p1'), HybridTime(1526000000) -> {} DocKey('p1'), ColumnId(attrs), HybridTime(1526000000) -> {} DocKey('p1'), ColumnId(attrs), 'h', HybridTime(1526000000) -> 7 DocKey('p1'), ColumnId(attrs), 'w', HybridTime(1526000000) -> 5 DocKey('p1'), ColumnId(price), HybridTime(1526000000) -> 99
22.
22© 2018 All
rights reserved. Some of the RocksDB enhancements • WAL and MVCC enhancements o Removed RocksDB WAL, re-uses Raft log o MVCC at a higher layer o Coordinate RocksDB memstore flushing and Raft log garbage collection • File format changes o Sharded (multi-level) indexes and Bloom filters • Splitting data blocks & metadata into separate files for tiering support • Separate queues for large and small compactions
23.
23© 2018 All
rights reserved. More Enhancements to RocksDB • Data model aware Bloom filters • Per-SSTable key range metadata to optimize range queries • Server-global block caches & memstore limits • Scan-resistant block cache (single-touch and multi-touch)
24.
24© 2018 All
rights reserved. Raft Related Enhancements • Leader Leases • Leader Balancing • Group Commits • Observer Nodes / Read Replicas (Tunable Read Consistency)
25.
25© 2018 All
rights reserved. Raft Extension: Leader Leases Tablet Peer (old leader) Tablet Peer (new leader) Tablet Peer (follower) x=10 x=10 x=10 Network partition Client writes x=20, and the new leader replicates it Client Without leader leases: the client can still reach the old leader, read x=10 1 2 4 3x=20 x=20
26.
26© 2018 All
rights reserved. Raft Extension: Leader Leases TimeTablet Server 1 is the leader of a tablet Leader lease Tablet server 2 becomes leader, cannot take load until the old leader’s lease expires Tablet Server 2 is a follower Tablet Server 1 Tablet Server 2
27.
27© 2018 All
rights reserved. Highly Scalable Source: https://blog.yugabyte.com/scaling-yugabyte-db-to-millions-of-reads-and-writes-fb86cea5ff15 • Stress tested up to 50 node cluster sizes • Scales linearly • Automatic sharding and load balancing
28.
28© 2018 All
rights reserved. High Performance and Data Density Source: https://blog.yugabyte.com/building-a-strongly-consistent-cassandra-with-better-performance-aa96b1ab51d6 • Better than the most performant NoSQL DBs • 2x-5x better performance vs Cassandra
29.
29© 2018 All
rights reserved. Transactions
30.
30© 2018 All
rights reserved. Single Shard Transactions Raft Consensus Protocol . . . INSERT INTO t SET x=10 IF NOT EXISTS Lock Manager (in memory, on leader only) Acquire a lock on x DocDB / RocksDB Read current value of x Submit a Raft operation for replication: set x=10 at hybrid_time 100 Raft log Tablet follower Tablet follower Replicate to majority of tablet peers Apply to RocksDB and release lock x=10 @ht=100 1 2 5 3 4
31.
31© 2018 All
rights reserved. MVCC based on Hybrid Time • HybridTime is an always increasing cluster-wide timestamp http://users.ece.utexas.edu/~garg/pdslab/david/hybrid-time-tech-report-01.pdf • Every RocksDB key includes a HybridTime at the end • Allows reads at a particular snapshot without locking • Compactions: o Overwritten/deleted entries are garbage-collected as soon as all read operations at old timestamps are done o TTL-expired entries turn into delete markers o On minor compactions, delete markers have to be kept!
32.
32© 2018 All
rights reserved. Single Shard Transactions • HybridTime values are strictly increasing in each tablet • Each tablet maintains a “safe time” that is used for reads o Highest timestamp such that the view as of that timestamp is fixed o In the common case it is just before the hybrid time of the next uncommitted record in the tablet
33.
33© 2018 All
rights reserved. Distributed Transactions • A fully decentralized architecture • Every tablet server can act as a Transaction Manager • A distributed transaction status table • Every transaction is assigned to a status tablet
34.
34© 2018 All
rights reserved. Distributed Transactions – Write Path
35.
35© 2018 All
rights reserved. Distributed Transactions – Write Path Step 1: Client request
36.
36© 2018 All
rights reserved. Distributed Transactions – Write Path Step 2: Create status record
37.
37© 2018 All
rights reserved. Distributed Transactions – Write Path Step 2: Create status record
38.
38© 2018 All
rights reserved. Distributed Transactions – Write Path Step 3: Write provisional records
39.
39© 2018 All
rights reserved. Distributed Transactions – Write Path Step 4: Atomic commit
40.
40© 2018 All
rights reserved. Distributed Transactions – Write Path Step 5: Respond to client
41.
41© 2018 All
rights reserved. Distributed Transactions – Write Path Step 6: Apply provisional records
42.
42© 2018 All
rights reserved. Isolation Levels • Currently Snapshot Isolation is supported o Write-write conflicts detected when writing provisional records • Serializable isolation (roadmap) o Reads in RW txns also need provisional records • Read-only transactions are always lock-free
43.
43© 2018 All
rights reserved. Clock Skew and Read Restarts • Need to ensure the read timestamp is high enough o Committed records the client might have seen must be visible • Optimistically use current Hybrid Time, re-read if necessary o Reads are restarted if a record with a higher timestamp that the client could have seen is encountered o Read restart happens at most once per tablet o Relying on bounded clock skew (NTP, AWS Time Sync) • Only affects multi-row reads of frequently updated records
44.
44© 2018 All
rights reserved. Distributed Transactions – Read Path
45.
45© 2018 All
rights reserved. Distributed Transactions – Read Path Step 1: Client request; pick ht_read
46.
46© 2018 All
rights reserved. Distributed Transactions – Read Path Step 2: Read from tablet servers
47.
47© 2018 All
rights reserved. Distributed Transactions – Read Path Step 3: Resolve txn status
48.
48© 2018 All
rights reserved. Distributed Transactions – Read Path Step 4: Respond to YQL Engine
49.
49© 2018 All
rights reserved. Distributed Transactions – Read Path Step 5: Respond to client
50.
50© 2018 All
rights reserved. Distributed Transactions – Conflicts & Retries • Every transaction is assigned a random priority • In a conflict, the higher-priority transaction wins o The restarted transaction gets a new random priority o Probability of success quickly increases with retries • Restarting a transaction is the same as starting a new one • A read-write transaction can be subject to read-restart
51.
51© 2018 All
rights reserved. Questions? Try it at docs.yugabyte.com/quick-start
Jetzt herunterladen