Lars George and Jon Hsieh presented archetypes for common Apache HBase application patterns. They defined archetypes as common architecture patterns extracted from multiple use cases to be repeatable. The presentation covered "good" archetypes that are well-suited to HBase's capabilities, such as storing simple entities, messaging data, and metrics. "Bad" archetypes that are not optimal fits for HBase included using it as a large blob store, naively porting a relational database schema, and as an analytic archive requiring frequent full scans. A discussion of access patterns and tradeoffs concluded the overview of HBase application archetypes.
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Â
A Survey of HBase Application Archetypes
1. Headline Goes Here
Speaker Name or Subhead Goes Here
DO NOT USE PUBLICLY
PRIOR TO 10/23/12
Apache HBase Application
Archetypes
Lars George | @larsgeorge | Cloudera EMEA Chief Architect | HBase PMC
Jonathan Hsieh | @jmhsieh | Cloudera HBase Tech lead | HBase PMC
HBaseCon 2014
May 5th , 2014
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
1
2. About Lars and Jon
Lars George
⢠EMEA Chief Architect
@Cloudera
⢠Apache HBase PMC
⢠OâReilly Author of HBase â The
Definitive Guide
⢠Contact
⢠lars@cloudera.com
⢠@larsgeorge
Jon Hsieh
⢠Tech Lead HBase Team
@Cloudera
⢠Apache HBase PMC
⢠Apache Flume founder
⢠Contact:
⢠jon@cloudera.com
⢠@jmhsieh
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
2
3. About Supporting HBase at Cloudera
⢠Supporting Customers using HBase since 2011
⢠HBase Training
⢠Professional Services
⢠Team has experience supporting and running HBase since 2009
⢠8 committers on staff
⢠2 HBase book authors
⢠As of Jan 2014, ~20,000 HBase nodes (in aggregate) under
management
⢠Information in this presentation is either aggregated customer data
or from public sources.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
3
4. An Apache HBase Timeline
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
20142008 2009 2010 2011 20132012
Aprâ11: CDH3 GA
with HBase 0.90.1
May â12:
HBaseCon 2012
Jun â13:
HBaseCon 2013
Summerâ11:
Messages
on HBaseSummer â09
StumbleUpon
goes production on
HBase ~0.20
Nov â11:
Cassini
on HBase
Jan â13
Phoenix
on HBase
Summerâ11:
Web Crawl
Cache
4
Septâ11:
HBase TDG
published
Novâ12:
HBase in
Action
published
2015
May â14:
HBaseCon 2014
Aug â13
Flurry 1k-1k node
cluster replication
Summer â14
HBase v1.0.0
released
Janâ14: Cloudera
has ~20k Hbase
nodes under
management
7. A vocabulary for HBase Archetypes
Definitions
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
7
8. Defining HBase Archetypes
⢠There are a lot of HBase applications
⢠Some successful, some less so
⢠They have common architecture patterns
⢠They have common tradeoffs
⢠Archetypes are common architecture patterns
⢠Common across multiple use-cases
⢠Extracted to be repeatable
⢠Our Goal: Define patterns Ă la âGang of Fourâ (Gamma, Helm,
Johnson, Vlissides)
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
8
9. So you want to use HBase?
⢠What data is being stored?
⢠Entity data
⢠Event data
⢠Why is the data being stored?
⢠Operational use cases
⢠Analytical use cases
⢠How does the data get in and out?
⢠Real time vs. Batch
⢠Random vs. Sequential
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
9
10. What is being stored?
There are primarly two kinds of big data workloads. They have
different storage requirements.
Entities Events
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
10
11. Entity Centric Data
⢠Entity data is information about current state
⢠Generally real time reads and writes
⢠Examples:
⢠Accounts
⢠Users
⢠Geolocation points
⢠Click Counts and Metrics
⢠Current Sensors Reading
⢠Scales up with # of Humans and # of Machines/Sensors
⢠Billions of distinct entities
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
11
12. Event Centric Data
⢠Event centric data are time-series data points recording successive points
spaced over time intervals.
⢠Generally real time write, some combination of real time read or batch read
⢠Examples:
⢠Sensor data over time
⢠Historical Stock Ticker data
⢠Historical Metrics
⢠Clicks time-series
⢠Scales up due to finer grained intervals, retention policies, and the passage
of time
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
12
13. Events about Entities
⢠Majority Big Data use cases are dealing with event-based data
⢠|Entities| * |Events| = Big data
⢠When you ask questions, do you hone in on entity first?
⢠When you ask questions, do you hone in on time ranges first?
⢠Your answer will help you determine where and how to store
your data.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
13
14. Why are you storing the data?
⢠So what kind of questions are you asking the data?
⢠Entity-centric questions
⢠Give me everything about entity e
⢠Give me the most recent event v about entity e
⢠Give me the n most recent events V about entity e
⢠Give me all events V about e between time [t1,t2]
⢠Event and Time-centric questions
⢠Give me an aggregates on each entity between time [t1,t2]
⢠Give me an aggregate on each time interval for entity e
⢠Find events V that match some other given criteria
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
14
15. How does data get in and out of HBase?
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
Bulk Import
HBase Client
15
HBase
Replication
HBase
Replication
16. How does data get in and out of HBase?
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
16
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
17. What system is most efficient?
⢠It is all physics
⢠You have a limited I/O budget
⢠Use all your I/O by parallelizing access
and read/write sequentially.
⢠Choose the system and features that
reduces I/O in general
⢠Pick the systems best for your workload
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
17
IOPs/s/disk
18. The physics of Hadoop Storage Systems
Workload HBase HDFS
Low latency ms, cached mins, MR
+ seconds, Impala
Random Read primary index - index?, small files problem
Short Scan sorted + partition
Full Scan 0 live table
+ (MR on snapshots)
MR, Hive, Impala
Random Write log structured - Not supported
Sequential Write hbase overhead
bulk load
minimal overhead
Updates log structured - Not supported
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
18
19. The physics of Hadoop Storage Systems
Workload HBase HDFS
Low latency ms, cached mins, MR
+ seconds, Impala
Random Read primary index - index?, small files problem
Short Scan sorted + partition
Full Scan 0 live table
+ (MR on snapshots)
MR, Hive, Impala
Random Write log structured - Not supported
Sequential Write hbase overhead
bulk load
minimal overhead
Updates log structured - Not supported
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
19
20. The physics of Hadoop Storage Systems
Workload HBase HDFS
Low latency ms, cached mins, MR
+ seconds, Impala
Random Read primary index - index?, small files problem
Short Scan sorted + partition
Full Scan 0 live table
+ (MR on snapshots)
MR, Hive, Impala
Random Write log structured - not supported
Sequential Write HBase overhead
bulk load
minimal overhead
Updates log structured - not supported
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
20
22. HBase application use cases
⢠The Good
⢠Simple Entities
⢠Messaging Store
⢠Graph Store
⢠Metrics Store
⢠The Bad
⢠Large Blobs
⢠Naïve RDBMS port
⢠Analytic Archive
⢠The Maybe
⢠Time series DB
⢠Combined workloads
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
22
24. Archetype: Simple Entities
⢠Purely entity data, no relation between entities
⢠Batch or real-time, random writes
⢠Real-time, random reads
⢠Could be a well-done denormalized RDBMS port.
⢠Often from many different sources, with poly-structured data
⢠Schema:
⢠Row per entity
⢠Row key => entity ID, or hash of entity ID
⢠Col qualifier => Property / field, possibly time stamp
⢠Geolocation data
⢠Search index building
⢠Use solr to make text data searchable.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
24
25. Simple Entities access pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
25
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
HBase
Replication
Solr
26. Archetype: Messaging Store
⢠Messaging Data:
⢠Realtime Random writes: Emails, SMS, MMS, IM
⢠Realtime random updates: Msg read, starred, moved, deleted
⢠Reading of top-N entries, sorted by time
⢠Records are of varying size
⢠Some time series, but mostly random read/write
⢠Schema:
⢠Row = users/feed/inbox
⢠Row key = UID or UID + time
⢠Column Qualifier = time or conversation id + time.
⢠Use CFâs for indexes.
⢠Examples:
⢠Facebook Messages, Xiaomi Messages
⢠Telco SMS/MMS services
⢠Feeds like tumblr, pinterest
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
26
27. Facebook Messages - Statistics
Source: HBaseCon 2012 - Anshuman Singh
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
27
28. Messages Access Pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
28
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
29. Archetype: Graph Data
⢠Graph Data: All entities and relations
⢠Batch or realtime, random writes
⢠Batch or realtime, random reads
⢠Its an entity with relation edges
⢠Schema:
⢠Row = Node.
⢠Row key => Node ID.
⢠Col qualifier => Edge ID, or properties:values
⢠Examples:
⢠Web Caches â Yahoo!, Trend Micro
⢠Titan Graph DB with HBase storage backend
⢠Sessionization (financial transactions, clicks streams, network traffic)
⢠Government (connect the bad guy)
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
29
30. Graph Data Access Pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
30
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
31. Archetype: Metrics
⢠Frequently updated Metrics
⢠Increments
⢠Roll ups generated by MR and bulk loaded to HBase
⢠Poor manâs datacubes
⢠Examples
⢠Campaign Impression/Click counts (Ad tech)
⢠Sensor data (Energy, Manufacturing, Auto)
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
31
32. Metrics Access Pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
32
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
34. Current HBase weak spots
⢠HBaseâs architecture can handle a lot
⢠We make engineering trade offs to optimize for them.
⢠HBase can still do things it is not optimal for.
⢠However, other systems are fundamentally more efficient for some
workloads.
⢠Weâve often seen some folks forcing apps into HBase.
⢠If one of these is your only workloads on this data, use another system
⢠If you are in a mixed workload case, some of these become âmaybesâ.
⢠Just because it is not good today, doesnât mean it cant be better
tomorrow.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
34
35. Bad Archetype: Large Blob Store
⢠Saving large objects >3MB per cell
⢠Schema:
⢠Normal entity pattern, but with some columns with large cells.
⢠Examples
⢠Raw photo or video storage in HBase
⢠Large frequently updated structs as a single cell
⢠Problems:
⢠Will get crushed due to write amplification when reoptimizing data for read.
(compactions on large unchanging data)
⢠Will crush write pipeline if there are large structs with frequently updated subfields.
Cells are atomic, and hbase must rewrite an entire cell.
⢠Some work adding LOB support
⢠This requires new architecture elements
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
35
36. Bad Archetype: NaĂŻve RDBMS port
⢠A naïve port the RDBMS onto HBase, directly copying the schema.
⢠Schema
⢠Many tables, just like an RDBMS schema.
⢠Row key: primary key or auto-incrementing key, like RDBMS schema
⢠Column qualifiers: field names
⢠Manually do joins, or secondary indexes (not consistent)
⢠Solution:
⢠HBase is not a SQL Database.
⢠No multi-region/multi-table in HBase transactions (yet).
⢠Must to denormalize your schema to use Hbase.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
36
37. Large blob store, NaĂŻve RDBMS port access patterns
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
37
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
38. Bad Archetype: Analytic archive
⢠Store purely chronological data, partitioned by time
⢠Real time writes, chronological time as primary index
⢠Column-centric aggregations over all rows.
⢠Bulk reads out, generally for generating periodic reports
⢠Schema
⢠Row key: date+xxx or salt+date+xxx
⢠Column qualifiers: properties with data or counters
⢠Example
⢠Machine logs organized by date.
⢠Full fidelity clickstream
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
38
39. Bad Archetype: Analytic archive Problems
⢠HBase non-optimal as primary use case.
⢠Will get crushed by frequent full table scans.
⢠Will get crushed by large compactions.
⢠Will get crushed by write-side region hot spotting.
⢠Instead
⢠Store in HDFS; Use Parquet columnar data storage + Impala/Hive
⢠Build rollups in HDFS+MR; store and serve rollups in HBase
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
39
40. Analytic Archive access patterns
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
40
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
41. And this is crazy | But hereâs my data, | serve it, maybe!
Archetypes: The Maybe
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
41
42. The Maybeâs
⢠For some applications, doing it right gets complicated.
⢠These more sophisticated or nuanced cases require considing
these questions:
⢠When do you choose HBase vs HDFS storage for time series data?
⢠Are there times where bad archetypes are ok?
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
42
43. Time Series: in HBase or HDFS?
⢠IO Patterns:
⢠Reads: Collocate related data
⢠Make reads cheap and fast.
⢠Writes: Spread writes out as much as possible
⢠Maximize write throughput
⢠HBase: Tension between these goals
⢠Spreading writes spreads data making reads inefficient
⢠Colocating on write causes hotspots, underutilizes resources by limiting write
throughput
⢠HDFS: The sweet spot.
⢠Sequential writes and and sequential read.
⢠Just write more files in date-dirs; physically spreads writes but logically groups data.
⢠Reads for time centric quieries just read files in date-dir
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
43
44. Time Series data flows
⢠Ingest
⢠Flume or similar direct tool via app
⢠HDFS
⢠Batch queries and generate rollups in Hive/MR
⢠Faster queries in Impala
⢠No user time serving
⢠HBase for recent, HDFS for historical
⢠HBase
⢠Serve individual events
⢠Serve pre-computed aggregates
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
44
45. Archetype: Entity Time Series
⢠A time series access pattern suitable for HBase
⢠Random write to event data, random read specific event or aggregate data
⢠Generate aggregates via counters, donât directly compute aggregate on
query
⢠HBase is system of record
⢠Schema:
⢠Rowkey: entity-timestamp or hash(entity)-timestamp, possibly with salt
added after entity.
⢠Col qualifiers: property
⢠Use custom aggretation to consolidate old data
⢠Use TTLâs to bound and age off old data
⢠Examples:
⢠OpenTSDB does this well for numeric values; Lazily aggregates cells for
better performance.
⢠Facebook Insights, ODS
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
45
46. Entity Time Series access pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
46
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
Flume
Custom App
47. Archetypes: Hybrid Entity Time Series
⢠Essentially a combo of the Metric Archetype and Entity Time
Series Archetype, with bulk loads of rollups via HDFS.
⢠Land data in HDFS and HBase
⢠Keep all data in HDFS for future use
⢠Aggregate in HDFS and write to HBase
⢠HBase can do some aggregates too (counters)
⢠Keep serve-able data in HBase.
⢠Use TTL to discard old values from Hbase.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
47
48. Hybrid time series access pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Hive or MR:
Bulk Import
HBase Client
48
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
HDFS
Flume
49. Meta Archetype: Combined workloads
⢠In these cases, the use of HBase depends on workload
⢠Cases where we have multiple workloads styles.
⢠Many cases we want to do multiple things with the same
data
⢠primary use case (real time, random access)
⢠secondary use case (analytical)
⢠Pick for your primary, hereâs some patterns on how to do
your secondary.
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
49
50. Real time workloads and Analytical access
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
50
poor latency!
full scans
interfere with
latency!
high throughput
MapReduce
HBase Scanner
HBase Client
Put, Incr,
Append
Bulk Import
HBase Client
HBase
Replication
51. Real time workloads and Analytical access
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
51
HBase
Replication
low latency
Isolated from full scans
high throughput
MapReduce
HBase Scanner
HBase Client
Put, Incr,
Append
Bulk Import
HBase Client
HBase
Replication
high throughput
52. MR over Table Snapshots (0.98, CDH5.0)
⢠Previously MapReduce jobs over
HBase required online full table
scan
⢠Take a snapshot and run MR job
over snapshot files
⢠Doesnât use HBase client
⢠Avoid affecting HBase caches
⢠3-5x perf boost.
⢠Still requires more IOPs than hdfs
raw files
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
map
map
map
map
map
map
map
map
reduce
reduce
reduce
map
map
map
map
map
map
map
map
reduce
reduce
reduce
snapshot
52
53. Analytic Archive access pattern
HBase Client
Put, Incr,
Append
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
HBase Client
Get, Scan
Bulk Import
HBase Client
53
HBase
Replication
HBase
Replication
low latency
high throughput
Gets
Short scan
Full Scan,
MapReduce
HBase Scanner
55. Multitenancy (in progress)
⢠We want to MR for analytics while
serving low-latency requests in one
cluster.
⢠Performance Isolation
⢠Limit performance impact load on
one table has on others. (HBASE-
6721)
⢠Request prioritization and scheduling
⢠Toda default is FIFO
⢠Need to schedule some requests
before others (HBASE-10994)
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
55
1 1 2 1 1 3 1
1 1 21 1 31
Delayed by long
scan requests
Rescheduled so
new request get
priority
Mixed workload
Isolated
workload
57. Big Data Workloads
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
57
Low
latency
Batch
Random Access Full ScanShort Scan
HDFS + MR
(Hive/pig)
HBase
HBase + Snapshots
-> HDFS + MR
HDFS
+ Impala
HBase + MR
58. Big Data Workloads
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
58
Low
latency
Batch
Random Access Full ScanShort Scan
HDFS + MR
(Hive/pig)
HBase
HBase + Snapshots
-> HDFS + MR
HDFS
+ Impala
HBase + MR
Current Metrics
Graph data
Simple Entities
Hybrid Entity Time series
+ Rollup serving
Messages
Analytic archive
Hybrid Entity Time series
+ Rollup generation
Index building
Entity Time series
59. HBase is evolving to be an Operational Database
⢠Excels at consistent single row centric operations
⢠Dev efforts aimed at using all machine resources efficiently,
reducing MTTR, and improving latency predictability.
⢠Projects built on HBase that enable secondary indexing and
multi-row transactions
⢠Apache Phoenix (incubating) or Impala provide a SQL skin for
simplified application development
⢠Analytic workloads?
⢠Can be done but will be beaten by direct HDFS +
MR/Spark/Impala
5/5/14 HBaseCon 2014; Lars George,
Jon Hsieh
59