2. Agenda •
1. Column Advantage
2. Storage and Process
3. Hadoop Related
3. History
2001 PAX
Mike Stonebraker, Daniel Abadi, Adam Batkin, Xuedong Chen, Mitch
Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, …
C-Store: A Column Oriented DBMS
D. J. Abadi, etc: Integrating Compression and Execution in Column-O
riented Database Systems. In SIGMOD, pages 671–682, 2006.
D. J. Abadi, etc: Materialization Strategies in a Column-Oriented DB
MS. In ICDE, pages 466–475, 2007.
6. Columnar Store vs Row Store
● IO-1 (basic column store): Every storage block contain
s data from only ONE column.
● IO-2: Aggressive compression.
● IO-3: No record-ids.
● CPU-4: A column executor
● CPU-5: Executor runs on compressed data.
● CPU-6: Executor can process columns that are key se
quence or entry sequence.
7. Columnar Store advantage
●
Compression
RLE, Bitmap ..
●
Ppd
reduce IO
●
Late Materialization
less memeory and CPU overhead
●
Block Iteration (Vectorization)
less CPU overhead
●
Invisible Join
– block as join key
8. Compression
● Run-length Encoding ● High Selectivity :
● ENCODING DELTAVAL Gender ,age
● Bit Vector Encoding ● Mid Selectivity :
● BLOCK_DICT City , Category
data skew ● Low Selectivity :
compound item_id , user_id
Price,quantity,
comment
12. late materialization
Construct Row
Apply Filter + Projection
Projections column only needed(also ppd)
Decoding Column First
Wait util process
Different Compression have difference behavior
15. Common Confusion IO
Choose more column ,more close to row store
IO <5%
record-ID
Row store free space at block tail
variable length field
IO Access Pattern means scalability
Hardware Trend
Compression rate
16. Common Confusion SerDe
Row or PAX SerDe
cpu cache miss
no columnar compression
Block Iteration (construct tuple or row)
Java vs C/C++
C/c++ direct memory mapping
Java Fastutil
17. Index and MV
Reduce IO Scalability
Avoid Sort Storange cost
Index join Complex desige
Lookup Hard maintain
Pre-computation : High latency
Join Slow down loading
Group by Lost Details
Query Rewrite
19. Hadoop Related
File Format
Trenvi vs IBM CIF
Schema Evolution
Portable File Format
Bigger Block Size
IO Pattern
SerDe network influence
20. Hadoop Related
Storage Cost
NameNode
Less block
Bigger block size
Cold data even bigger
No Intermediate Level
JobTracker
Each Job have Less Map and reduce number
DataNode
21. Hadoop Related
Real Data ingestion
Hbase + Flume
Balanced Data
Write avro file format first, then sort merge
SerDe memory reduce
Tuple Structure not row
Batch Update+Delete+Insert
22. Hadoop Related
MR Performance Boost
Block Shuffle (3 times faster)
Skew data have less overhead
Less map number and bigger spill
Reduce side combine
Light Compression Codec(snappy not LZO)
Combiner or in-memroy combiner deprecated
23. Hadoop Related
Easier Performance Tuning
mapred.min.split.size(deprecated)
mapred.child.java.opts
mapred.compress.map.output(deprecated)
io.sort.mb
io.sort.spill.percent(deprecated)
Io.sort.factor
mapred.reduce.parallel.copies(deprecated)
Map and reduce number easier estimate
Reduce algorithm will change
24. Hadoop Related
Easy Management
Less Partition or Dynamic Partition
Integrity constraints and Referential integrity
Statistic make simple query engine
Cold Data automatic merge
Trojan Layout vs Columnar Projections
Less Design complexity
Map join vs Fat Table
Group by + Index