SlideShare ist ein Scribd-Unternehmen logo
1 von 68
Downloaden Sie, um offline zu lesen
1
The State of HBase Replication
Jean-Daniel Cryans
May 5th, 2014
©2014 Cloudera, Inc. All rights reserved.
About me
2
• Software Engineer at Cloudera, Storage team
• Apache HBase committer since 2008, PMC
member
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-tolerant;
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-tolerant;
• highly available; and
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-tolerant;
• highly available; and
• almost magic.
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-tolerant;
• highly available; and
• almost magic.
3
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
4
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
• It’s used to replicate data between thousands
of nodes across continents.
4
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
• It’s used to replicate data between thousands
of nodes across continents.
• It’s used for Disaster Recovery, geo-
distributed serving, and more.
4
©2014 Cloudera, Inc. All rights reserved.
5
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Design
• Clusters are distinct
• Pull VS push
• Sync VS Async
6
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
7
Master
20 RS
Slave
15 RS
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t replicated
7
Master
20 RS
Slave
15 RS
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t replicated
• Regions can be different
7
Master
20 RS
Slave
15 RS
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t replicated
• Regions can be different
• Security has to be configured for each cluster
7
Master
20 RS
Slave
15 RS
©2014 Cloudera, Inc. All rights reserved.
Push instead of Pull
8
MySQL
Master
MySQL
Slave
Get binlog
Apply locally
MySQL Replication uses Pull
Cluster A Cluster B
©2014 Cloudera, Inc. All rights reserved.
Push instead of Pull
9
RS RSreplicate entries
Apply to cluster
HBase Replication uses Push
Cluster A Cluster B
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Synchronous Replication
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Put
2
3
1
Synchronous Replication
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Put
2
3
1
Ack Ack
Put
5
6
4
78
Synchronous Replication
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
Cluster A
RS
HLog
MemStore
Put
Ack
2
3
1
4
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
Cluster A
RS
HLog
MemStore
Put
Ack
2
3
1
4
Cluster B
RS
HLog
MemStore
Ack
Put
3
4
2
5
HLog
Tailing
Thread
1
©2014 Cloudera, Inc. All rights reserved.
First Release - 0.90.0
• Simple master-slave (only one)
• Disabled by default
• Uses ZK as a metadata store
12
©2014 Cloudera, Inc. All rights reserved.
Original Implementation
13
replicateLogEntries()Replication
Source
ZooKeeper
Watcher
Region Server on
Master Cluster
Replication
Sink
HTable
Put
Delete
Region Server on
Slave Cluster
©2014 Cloudera, Inc. All rights reserved.
First Lesson Learned
• HDFS doesn’t support tailing files being
written to. It requires:
• open()
• seek()// go where we stopped last time
• while (not EOF || enoughData)
• read()
• close()
• repeat
14
©2014 Cloudera, Inc. All rights reserved.
Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have an atomic move operation
• Doubles # ops needed, race conditions
15
©2014 Cloudera, Inc. All rights reserved.
Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have an atomic move operation
• Doubles # ops needed, race conditions
15
/hbase
/replication
/RS1
/1
/hlog1
/hlog2
...
/hbase
/replication
/RS2
/1-RS1
/hlog1
1. create new hlog2
2. delete old hlog2
©2014 Cloudera, Inc. All rights reserved.
Second Release - 0.92.0
• Cyclic replication
• Multi-slave (scope LOCAL or GLOBAL)
• Enable / disable peer
• Special configurations
16
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row X
Row X is from 1
Don’t replicate!
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X Put Row X
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog
HLog
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog
HLog
HLog
Is the peer enabled?
©2014 Cloudera, Inc. All rights reserved.
Special Configurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replication when
deleting data.
20
©2014 Cloudera, Inc. All rights reserved.
Special Configurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replication when
deleting data.
• MIN_VERSION
• With TTL, makes it easy to configure a slave that
contains only the last few days of data.
20
©2014 Cloudera, Inc. All rights reserved.
Third Lesson Learned
• It’s easy to DDOS yourself.
• Replication was using the normal handlers...
• ... and using them to write back!
21
Handler1: Put
Handler2: Delete
Handler3: Replicate
Handler4: Get
Handler5: Put
Replicated Put goes in the queue
©2014 Cloudera, Inc. All rights reserved.
Fourth Lesson Learned
• Instinctively, what would something called
stop_replication do?
22
©2014 Cloudera, Inc. All rights reserved.
Fourth Lesson Learned
• Instinctively, what would something called
stop_replication do?
• Good intentions, bad outcomes, HBASE-8861
22
start/stop_replication
X
©2014 Cloudera, Inc. All rights reserved.
Third Release - 0.96.0 / 0.98.0
• Replication enabled by default!
• Completely refactored for readability/
extensibility (Chris Trezzo)
• ReplicationSyncUp tool (HBASE-9047)
• Throttling (HBASE-9501)
• Finer grained replication controls
(HBASE-8751)
23
©2014 Cloudera, Inc. All rights reserved.
ReplicationSyncUp Tool
• Works on an offline cluster
• Can finish replicating the queues in ZK
• Useful to finish draining a master cluster
24
HBase
HDFS
ZooKeeper
HBase
HDFS
ZooKeeper
ReplicationSyncUp
©2014 Cloudera, Inc. All rights reserved.
Finer Grained Replication Controls
> set_peer_tableCFs '2', "table1;
table2:cf1,cf2; table3:cfA,cfB"
• Meaning: enable replication to peer #2 for:
• All of table1
• cf1 and cf2 from table2
• cfA and cfB from table3
25
©2014 Cloudera, Inc. All rights reserved.
26
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Flurry
• Two data centers, coast to coast
• Three clusters, in master-master pairs
• 1200 nodes
• 800 nodes
• 30 nodes
• Replication traffic: 2Gbps
• Latency between DCs: 85ms
27
©2014 Cloudera, Inc. All rights reserved.
Opower
• Two clusters, same data center
• Master: tens of nodes
• Slave: tens of nodes
• Replication traffic: 1GB/day
• Bulk load replication traffic: 180GB/day
• Recent use case
28
©2014 Cloudera, Inc. All rights reserved.
Lily HBase Indexer
• Collaboration between NGData & Cloudera.
• NGData are the creators of the Lily data
management platform.
• Lily HBase Indexer
• Service which acts as a HBase replication listener.
• Custom sink writes to SolrCloud.
• Integrates Cloudera Morphlines library for ETL of
rows.
29
©2014 Cloudera, Inc. All rights reserved.
30
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Stop Relying on Permanent Znodes
• Current rule is to never rely on znodes to
survive cluster restarts, upgrades, etc.
• State data should be kept in an HBase table.
• Notification done through a new mechanism
• See: https://issues.apache.org/jira/browse/
HBASE-10295
31
©2014 Cloudera, Inc. All rights reserved.
Define a Replication Interface
• Replication is somewhat extendable but it
lacks stable interfaces.
• The HBase Indexer is such an extension and it
required surgery every time a committer
sneezed.
• See: https://issues.apache.org/jira/browse/
HBASE-10504
32
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the current value; and
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the current value; and
3.Put’ing the newly incremented value.
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the current value; and
3.Put’ing the newly incremented value.
• This breaks in Master-Master because the
Puts are overwriting each other.
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the current value; and
3.Put’ing the newly incremented value.
• This breaks in Master-Master because the
Puts are overwriting each other.
• See https://issues.apache.org/jira/browse/
HBASE-2804
33
©2014 Cloudera, Inc. All rights reserved.
More Tooling
• Replication management console, one shell to
rule all the clusters!
• Replication bootstrapping tool.
• Tool that can move queues between region
servers.
• Tool that can throttle replication on a live
cluster.
34
©2014 Cloudera, Inc. All rights reserved.
Questions?
• Or ping me async:
• @jdcryans
• jdcryans@cloudera.com
• jdcryans on #hbase irc.freenode.net
35

Weitere ähnliche Inhalte

Was ist angesagt?

Indexes, Filters, and Other Animals
Indexes, Filters, and Other AnimalsIndexes, Filters, and Other Animals
Indexes, Filters, and Other AnimalsScyllaDB
 
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation BuffersHBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation BuffersCloudera, Inc.
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance TuningLars Hofhansl
 
Container Storage Best Practices in 2017
Container Storage Best Practices in 2017Container Storage Best Practices in 2017
Container Storage Best Practices in 2017Keith Resar
 
MariaDB MaxScale monitor 매뉴얼
MariaDB MaxScale monitor 매뉴얼MariaDB MaxScale monitor 매뉴얼
MariaDB MaxScale monitor 매뉴얼NeoClova
 
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/Spring
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/SpringPacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/Spring
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/SpringTakatoshi Matsuo
 
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBayReal-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBayAltinity Ltd
 
Alfresco勉強会#24 コンテンツのライフサイクル
Alfresco勉強会#24 コンテンツのライフサイクルAlfresco勉強会#24 コンテンツのライフサイクル
Alfresco勉強会#24 コンテンツのライフサイクルJun Terashita
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent
 
하둡 고가용성(HA) 설정
하둡 고가용성(HA) 설정하둡 고가용성(HA) 설정
하둡 고가용성(HA) 설정NoahKIM36
 
Introduction to rook
Introduction to rookIntroduction to rook
Introduction to rookRohan Gupta
 
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon
 
TripleOの光と闇
TripleOの光と闇TripleOの光と闇
TripleOの光と闇Manabu Ori
 
Manage Add-On Services with Apache Ambari
Manage Add-On Services with Apache AmbariManage Add-On Services with Apache Ambari
Manage Add-On Services with Apache AmbariDataWorks Summit
 
Introduction to Apache Sqoop
Introduction to Apache SqoopIntroduction to Apache Sqoop
Introduction to Apache SqoopAvkash Chauhan
 
MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11NeoClova
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
 
HBase Operations and Best Practices
HBase Operations and Best PracticesHBase Operations and Best Practices
HBase Operations and Best PracticesVenu Anuganti
 
The Future of GlusterFS and Gluster.org
The Future of GlusterFS and Gluster.orgThe Future of GlusterFS and Gluster.org
The Future of GlusterFS and Gluster.orgJohn Mark Walker
 

Was ist angesagt? (20)

Indexes, Filters, and Other Animals
Indexes, Filters, and Other AnimalsIndexes, Filters, and Other Animals
Indexes, Filters, and Other Animals
 
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation BuffersHBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
 
Apache HBase Performance Tuning
Apache HBase Performance TuningApache HBase Performance Tuning
Apache HBase Performance Tuning
 
Container Storage Best Practices in 2017
Container Storage Best Practices in 2017Container Storage Best Practices in 2017
Container Storage Best Practices in 2017
 
MariaDB MaxScale monitor 매뉴얼
MariaDB MaxScale monitor 매뉴얼MariaDB MaxScale monitor 매뉴얼
MariaDB MaxScale monitor 매뉴얼
 
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/Spring
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/SpringPacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/Spring
Pacemaker+PostgreSQLレプリケーションで共有ディスクレス高信頼クラスタの構築@OSC 2013 Tokyo/Spring
 
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBayReal-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay
Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay
 
Alfresco勉強会#24 コンテンツのライフサイクル
Alfresco勉強会#24 コンテンツのライフサイクルAlfresco勉強会#24 コンテンツのライフサイクル
Alfresco勉強会#24 コンテンツのライフサイクル
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
 
하둡 고가용성(HA) 설정
하둡 고가용성(HA) 설정하둡 고가용성(HA) 설정
하둡 고가용성(HA) 설정
 
Introduction to rook
Introduction to rookIntroduction to rook
Introduction to rook
 
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
 
TripleOの光と闇
TripleOの光と闇TripleOの光と闇
TripleOの光と闇
 
Manage Add-On Services with Apache Ambari
Manage Add-On Services with Apache AmbariManage Add-On Services with Apache Ambari
Manage Add-On Services with Apache Ambari
 
Introduction to Apache Sqoop
Introduction to Apache SqoopIntroduction to Apache Sqoop
Introduction to Apache Sqoop
 
MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
HBase Operations and Best Practices
HBase Operations and Best PracticesHBase Operations and Best Practices
HBase Operations and Best Practices
 
The basics of fluentd
The basics of fluentdThe basics of fluentd
The basics of fluentd
 
The Future of GlusterFS and Gluster.org
The Future of GlusterFS and Gluster.orgThe Future of GlusterFS and Gluster.org
The Future of GlusterFS and Gluster.org
 

Ähnlich wie The State of HBase Replication

Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera FieldHBaseCon
 
Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Kathleen Ting
 
UKOUG2018 - I Know what you did Last Summer [in my Database].pptx
UKOUG2018 - I Know what you did Last Summer [in my Database].pptxUKOUG2018 - I Know what you did Last Summer [in my Database].pptx
UKOUG2018 - I Know what you did Last Summer [in my Database].pptxMarco Gralike
 
What's New and Upcoming in HDFS - the Hadoop Distributed File System
What's New and Upcoming in HDFS - the Hadoop Distributed File SystemWhat's New and Upcoming in HDFS - the Hadoop Distributed File System
What's New and Upcoming in HDFS - the Hadoop Distributed File SystemCloudera, Inc.
 
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2Cloudera, Inc.
 
Hive on spark berlin buzzwords
Hive on spark berlin buzzwordsHive on spark berlin buzzwords
Hive on spark berlin buzzwordsSzehon Ho
 
HBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and SparkHBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and SparkHBaseCon
 
Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Wei-Chiu Chuang
 
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast DataKudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast DataCloudera, Inc.
 
HBase replication
HBase replicationHBase replication
HBase replicationwchevreuil
 
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...Yahoo Developer Network
 
Multi-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase DeploymentsMulti-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase DeploymentsDataWorks Summit
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosHeiko Loewe
 
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in HadoopKudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoopjdcryans
 
Kafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupKafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupGwen (Chen) Shapira
 

Ähnlich wie The State of HBase Replication (20)

Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera Field
 
Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)Hadoop Operations for Production Systems (Strata NYC)
Hadoop Operations for Production Systems (Strata NYC)
 
UKOUG2018 - I Know what you did Last Summer [in my Database].pptx
UKOUG2018 - I Know what you did Last Summer [in my Database].pptxUKOUG2018 - I Know what you did Last Summer [in my Database].pptx
UKOUG2018 - I Know what you did Last Summer [in my Database].pptx
 
What's New and Upcoming in HDFS - the Hadoop Distributed File System
What's New and Upcoming in HDFS - the Hadoop Distributed File SystemWhat's New and Upcoming in HDFS - the Hadoop Distributed File System
What's New and Upcoming in HDFS - the Hadoop Distributed File System
 
Hadoop Operations
Hadoop OperationsHadoop Operations
Hadoop Operations
 
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2
Strata + Hadoop World 2012: High Availability for the HDFS NameNode Phase 2
 
Hive on spark berlin buzzwords
Hive on spark berlin buzzwordsHive on spark berlin buzzwords
Hive on spark berlin buzzwords
 
HBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and SparkHBaseCon 2015: HBase and Spark
HBaseCon 2015: HBase and Spark
 
Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)Hadoop 3 (2017 hadoop taiwan workshop)
Hadoop 3 (2017 hadoop taiwan workshop)
 
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast DataKudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
 
HBase replication
HBase replicationHBase replication
HBase replication
 
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
 
YARN
YARNYARN
YARN
 
Multi-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase DeploymentsMulti-tenant, Multi-cluster and Multi-container Apache HBase Deployments
Multi-tenant, Multi-cluster and Multi-container Apache HBase Deployments
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and Mesos
 
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in HadoopKudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
 
Kafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka MeetupKafka & Hadoop - for NYC Kafka Meetup
Kafka & Hadoop - for NYC Kafka Meetup
 
Empower Hive with Spark
Empower Hive with SparkEmpower Hive with Spark
Empower Hive with Spark
 
Kudu austin oct 2015.pptx
Kudu austin oct 2015.pptxKudu austin oct 2015.pptx
Kudu austin oct 2015.pptx
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 

Mehr von HBaseCon

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on KubernetesHBaseCon
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on BeamHBaseCon
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at HuaweiHBaseCon
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in PinterestHBaseCon
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程HBaseCon
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at NeteaseHBaseCon
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践HBaseCon
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台HBaseCon
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comHBaseCon
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architectureHBaseCon
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at HuaweiHBaseCon
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMiHBaseCon
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0HBaseCon
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon
 

Mehr von HBaseCon (20)

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
 

Kürzlich hochgeladen

%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...masabamasaba
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...SelfMade bd
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park masabamasaba
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsArshad QA
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesVictorSzoltysek
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfonteinmasabamasaba
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrainmasabamasaba
 
8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech studentsHimanshiGarg82
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...Health
 
Define the academic and professional writing..pdf
Define the academic and professional writing..pdfDefine the academic and professional writing..pdf
Define the academic and professional writing..pdfPearlKirahMaeRagusta1
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrandmasabamasaba
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastPapp Krisztián
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionOnePlan Solutions
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension AidPhilip Schwarz
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️Delhi Call girls
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...panagenda
 
%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Hararemasabamasaba
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...masabamasaba
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Bert Jan Schrijver
 

Kürzlich hochgeladen (20)

%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
Define the academic and professional writing..pdf
Define the academic and professional writing..pdfDefine the academic and professional writing..pdf
Define the academic and professional writing..pdf
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
 
%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 

The State of HBase Replication

  • 1. 1 The State of HBase Replication Jean-Daniel Cryans May 5th, 2014
  • 2. ©2014 Cloudera, Inc. All rights reserved. About me 2 • Software Engineer at Cloudera, Storage team • Apache HBase committer since 2008, PMC member
  • 3. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: 3
  • 4. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; 3
  • 5. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; 3
  • 6. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and 3
  • 7. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and • almost magic. 3
  • 8. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and • almost magic. 3
  • 9. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. 4
  • 10. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. • It’s used to replicate data between thousands of nodes across continents. 4
  • 11. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. • It’s used to replicate data between thousands of nodes across continents. • It’s used for Disaster Recovery, geo- distributed serving, and more. 4
  • 12. ©2014 Cloudera, Inc. All rights reserved. 5 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  • 13. ©2014 Cloudera, Inc. All rights reserved. Design • Clusters are distinct • Pull VS push • Sync VS Async 6
  • 14. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs 7 Master 20 RS Slave 15 RS
  • 15. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated 7 Master 20 RS Slave 15 RS
  • 16. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated • Regions can be different 7 Master 20 RS Slave 15 RS
  • 17. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated • Regions can be different • Security has to be configured for each cluster 7 Master 20 RS Slave 15 RS
  • 18. ©2014 Cloudera, Inc. All rights reserved. Push instead of Pull 8 MySQL Master MySQL Slave Get binlog Apply locally MySQL Replication uses Pull Cluster A Cluster B
  • 19. ©2014 Cloudera, Inc. All rights reserved. Push instead of Pull 9 RS RSreplicate entries Apply to cluster HBase Replication uses Push Cluster A Cluster B
  • 20. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Synchronous Replication
  • 21. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Put 2 3 1 Synchronous Replication
  • 22. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Put 2 3 1 Ack Ack Put 5 6 4 78 Synchronous Replication
  • 23. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication
  • 24. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication Cluster A RS HLog MemStore Put Ack 2 3 1 4
  • 25. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication Cluster A RS HLog MemStore Put Ack 2 3 1 4 Cluster B RS HLog MemStore Ack Put 3 4 2 5 HLog Tailing Thread 1
  • 26. ©2014 Cloudera, Inc. All rights reserved. First Release - 0.90.0 • Simple master-slave (only one) • Disabled by default • Uses ZK as a metadata store 12
  • 27. ©2014 Cloudera, Inc. All rights reserved. Original Implementation 13 replicateLogEntries()Replication Source ZooKeeper Watcher Region Server on Master Cluster Replication Sink HTable Put Delete Region Server on Slave Cluster
  • 28. ©2014 Cloudera, Inc. All rights reserved. First Lesson Learned • HDFS doesn’t support tailing files being written to. It requires: • open() • seek()// go where we stopped last time • while (not EOF || enoughData) • read() • close() • repeat 14
  • 29. ©2014 Cloudera, Inc. All rights reserved. Second Lesson Learned • Single threaded, non-batched ZK is slow • ZK didn’t have an atomic move operation • Doubles # ops needed, race conditions 15
  • 30. ©2014 Cloudera, Inc. All rights reserved. Second Lesson Learned • Single threaded, non-batched ZK is slow • ZK didn’t have an atomic move operation • Doubles # ops needed, race conditions 15 /hbase /replication /RS1 /1 /hlog1 /hlog2 ... /hbase /replication /RS2 /1-RS1 /hlog1 1. create new hlog2 2. delete old hlog2
  • 31. ©2014 Cloudera, Inc. All rights reserved. Second Release - 0.92.0 • Cyclic replication • Multi-slave (scope LOCAL or GLOBAL) • Enable / disable peer • Special configurations 16
  • 32. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X
  • 33. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X
  • 34. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X
  • 35. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X Row X is from 1 Don’t replicate!
  • 36. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X
  • 37. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X
  • 38. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X
  • 39. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread
  • 40. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread Is the peer enabled?
  • 41. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog Is the peer enabled?
  • 42. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog Is the peer enabled?
  • 43. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog Is the peer enabled?
  • 44. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog HLog Is the peer enabled?
  • 45. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog HLog HLog Is the peer enabled?
  • 46. ©2014 Cloudera, Inc. All rights reserved. Special Configurations • KEEP_DELETED_CELLS • Must be used on slaves with replication when deleting data. 20
  • 47. ©2014 Cloudera, Inc. All rights reserved. Special Configurations • KEEP_DELETED_CELLS • Must be used on slaves with replication when deleting data. • MIN_VERSION • With TTL, makes it easy to configure a slave that contains only the last few days of data. 20
  • 48. ©2014 Cloudera, Inc. All rights reserved. Third Lesson Learned • It’s easy to DDOS yourself. • Replication was using the normal handlers... • ... and using them to write back! 21 Handler1: Put Handler2: Delete Handler3: Replicate Handler4: Get Handler5: Put Replicated Put goes in the queue
  • 49. ©2014 Cloudera, Inc. All rights reserved. Fourth Lesson Learned • Instinctively, what would something called stop_replication do? 22
  • 50. ©2014 Cloudera, Inc. All rights reserved. Fourth Lesson Learned • Instinctively, what would something called stop_replication do? • Good intentions, bad outcomes, HBASE-8861 22 start/stop_replication X
  • 51. ©2014 Cloudera, Inc. All rights reserved. Third Release - 0.96.0 / 0.98.0 • Replication enabled by default! • Completely refactored for readability/ extensibility (Chris Trezzo) • ReplicationSyncUp tool (HBASE-9047) • Throttling (HBASE-9501) • Finer grained replication controls (HBASE-8751) 23
  • 52. ©2014 Cloudera, Inc. All rights reserved. ReplicationSyncUp Tool • Works on an offline cluster • Can finish replicating the queues in ZK • Useful to finish draining a master cluster 24 HBase HDFS ZooKeeper HBase HDFS ZooKeeper ReplicationSyncUp
  • 53. ©2014 Cloudera, Inc. All rights reserved. Finer Grained Replication Controls > set_peer_tableCFs '2', "table1; table2:cf1,cf2; table3:cfA,cfB" • Meaning: enable replication to peer #2 for: • All of table1 • cf1 and cf2 from table2 • cfA and cfB from table3 25
  • 54. ©2014 Cloudera, Inc. All rights reserved. 26 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  • 55. ©2014 Cloudera, Inc. All rights reserved. Flurry • Two data centers, coast to coast • Three clusters, in master-master pairs • 1200 nodes • 800 nodes • 30 nodes • Replication traffic: 2Gbps • Latency between DCs: 85ms 27
  • 56. ©2014 Cloudera, Inc. All rights reserved. Opower • Two clusters, same data center • Master: tens of nodes • Slave: tens of nodes • Replication traffic: 1GB/day • Bulk load replication traffic: 180GB/day • Recent use case 28
  • 57. ©2014 Cloudera, Inc. All rights reserved. Lily HBase Indexer • Collaboration between NGData & Cloudera. • NGData are the creators of the Lily data management platform. • Lily HBase Indexer • Service which acts as a HBase replication listener. • Custom sink writes to SolrCloud. • Integrates Cloudera Morphlines library for ETL of rows. 29
  • 58. ©2014 Cloudera, Inc. All rights reserved. 30 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  • 59. ©2014 Cloudera, Inc. All rights reserved. Stop Relying on Permanent Znodes • Current rule is to never rely on znodes to survive cluster restarts, upgrades, etc. • State data should be kept in an HBase table. • Notification done through a new mechanism • See: https://issues.apache.org/jira/browse/ HBASE-10295 31
  • 60. ©2014 Cloudera, Inc. All rights reserved. Define a Replication Interface • Replication is somewhat extendable but it lacks stable interfaces. • The HBase Indexer is such an extension and it required surgery every time a committer sneezed. • See: https://issues.apache.org/jira/browse/ HBASE-10504 32
  • 61. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 33
  • 62. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 33
  • 63. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 33
  • 64. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. 33
  • 65. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. • This breaks in Master-Master because the Puts are overwriting each other. 33
  • 66. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. • This breaks in Master-Master because the Puts are overwriting each other. • See https://issues.apache.org/jira/browse/ HBASE-2804 33
  • 67. ©2014 Cloudera, Inc. All rights reserved. More Tooling • Replication management console, one shell to rule all the clusters! • Replication bootstrapping tool. • Tool that can move queues between region servers. • Tool that can throttle replication on a live cluster. 34
  • 68. ©2014 Cloudera, Inc. All rights reserved. Questions? • Or ping me async: • @jdcryans • jdcryans@cloudera.com • jdcryans on #hbase irc.freenode.net 35