Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
RocksDB
Dhruba Borthakur
Feb 3, 2016.
Open Discussion on Roadmap and Vision
How many of you have hands-on experience with RocksDB?
Use RocksDB?
Contribute to RocksDB?
RocksDB is an open source-project
You get to decide and contribute how to take it forward
What is RocksDB?
•Key-Value persistent store
•Embedded
•Optimized for fast storage
•Server workloads
RocksDB API
▪ Keys and values are arbitrary byte
arrays.
▪ Data are stored sorted by key.
▪ Update Operations: Put/Delete/...
RocksDB Architecture
Write Request
Read Request
Flush
Compaction
Active
MemTable
ReadOnly
MemTable
log
log
log
LS
M
sstsst...
Why does FB develop RocksDB?
RocksDB in the mainstream
•Reduces a 5 TB MongoDB instance to 285 GB on
MongoRocks
MySQL Innodb vs RocksDB
•LinkBench: open source benchmark for Facebook’s workload
•Reduces MySQL flash storage space by 50...
It is easy to use RocksDB in your software
RocksDB: Open & Pluggable
Pluggable
Memtable format
in RAM
Write Request from ApplicationGet or Scan Request from Applicat...
RocksDB is a tool, not a solution by itself.
Embed it into your software solution.
RocksDB Integrations
•LinkedIn Feed , Yahoo Sherpa, Airbnb, Pinterest
•Microsoft Bing Platform (development)
•Apache Samza...
Any SSD Vendors out here today?
Make RocksDB use your SSD efficiently!
Any FileSystem vendors here today?
Make RocksDB use your filesystem efficiently!
Want to extend RocksDB functionality?
Use StackableDB
e.g. TTL Support, Geo-index, Redis-style
Vision for the future
•Most performant database engine on ram, SSD and disks
• Optimize for next-generation storage hardwa...
Roadmap for 2016
What do you want from RocksDB?
Here are some probable enhancements for RocksDB in 2016
Raw Devices
RocksDB would run on a device lun
RocksDB could use directIO and bypass OS cache
MergeOperator & CompactionFilter
RocksDB could support a lua-based Filter
RocksDB could support a Javascript-based Filter
Reduce storage footprint
Dictionary based compression
More tradeoffs between cpu and compression (zstd, etc)
Production support
Database repair
Standalone utilities to inspect files, blocks, MANIFEST
Tiered Storage
Single db can have RAM, SSD, disk & next-gen-hardware
Intelligent movement of data between storage tiers
Open Discussion
Nächste SlideShare
Wird geladen in …5
×

The Hive Think Tank: Rocking the Database World with RocksDB

1.279 Aufrufe

Veröffentlicht am

Dhruba Borthakur, Facebook

Dhruba Borthakur is an engineer at Facebook. He has been one of the founding engineer of RocksDB, an open-source key-value store optimized for storing data in flash and main-memory storage. He has been one of the founding architects of the Apache Hadoop Distributed File System and has been instrumental in scaling Facebook's Hadoop cluster to multiples of petabytes. Dhruba has contributed code to the Apache HBase project. Earlier, he contributed to the development of the Andrew File System (AFS). He has an M.S. in Computer Science from the University of Wisconsin, Madison and a B.S. in Computer Science BITS, Pilani, India.

Veröffentlicht in: Technologie
  • Login to see the comments

The Hive Think Tank: Rocking the Database World with RocksDB

  1. 1. RocksDB Dhruba Borthakur Feb 3, 2016. Open Discussion on Roadmap and Vision
  2. 2. How many of you have hands-on experience with RocksDB? Use RocksDB? Contribute to RocksDB?
  3. 3. RocksDB is an open source-project You get to decide and contribute how to take it forward
  4. 4. What is RocksDB? •Key-Value persistent store •Embedded •Optimized for fast storage •Server workloads
  5. 5. RocksDB API ▪ Keys and values are arbitrary byte arrays. ▪ Data are stored sorted by key. ▪ Update Operations: Put/Delete/Merge ▪ Queries: Get/Iterator
  6. 6. RocksDB Architecture Write Request Read Request Flush Compaction Active MemTable ReadOnly MemTable log log log LS M sstsst sst sst sst sst d Switch Memory Persistent Storage Switch
  7. 7. Why does FB develop RocksDB?
  8. 8. RocksDB in the mainstream •Reduces a 5 TB MongoDB instance to 285 GB on MongoRocks
  9. 9. MySQL Innodb vs RocksDB •LinkBench: open source benchmark for Facebook’s workload •Reduces MySQL flash storage space by 50% for LinkBench
  10. 10. It is easy to use RocksDB in your software
  11. 11. RocksDB: Open & Pluggable Pluggable Memtable format in RAM Write Request from ApplicationGet or Scan Request from Application Transaction logPluggable sst data format on storage Pluggable Compaction Blooms Customizable WAL
  12. 12. RocksDB is a tool, not a solution by itself. Embed it into your software solution.
  13. 13. RocksDB Integrations •LinkedIn Feed , Yahoo Sherpa, Airbnb, Pinterest •Microsoft Bing Platform (development) •Apache Samza •Rippled server (bitcoin style) •RedHat CEPH •Open Channel SSD
  14. 14. Any SSD Vendors out here today? Make RocksDB use your SSD efficiently!
  15. 15. Any FileSystem vendors here today? Make RocksDB use your filesystem efficiently!
  16. 16. Want to extend RocksDB functionality? Use StackableDB e.g. TTL Support, Geo-index, Redis-style
  17. 17. Vision for the future •Most performant database engine on ram, SSD and disks • Optimize for next-generation storage hardware •Flexibility to be deployed on varied environments • RocksDB’s components are pluggable • Enables software vendors to customize their solution
  18. 18. Roadmap for 2016 What do you want from RocksDB? Here are some probable enhancements for RocksDB in 2016
  19. 19. Raw Devices RocksDB would run on a device lun RocksDB could use directIO and bypass OS cache
  20. 20. MergeOperator & CompactionFilter RocksDB could support a lua-based Filter RocksDB could support a Javascript-based Filter
  21. 21. Reduce storage footprint Dictionary based compression More tradeoffs between cpu and compression (zstd, etc)
  22. 22. Production support Database repair Standalone utilities to inspect files, blocks, MANIFEST
  23. 23. Tiered Storage Single db can have RAM, SSD, disk & next-gen-hardware Intelligent movement of data between storage tiers
  24. 24. Open Discussion

×