Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Rakuten LeoFs - distributed file system

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 33 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Rakuten LeoFs - distributed file system (20)

Anzeige

Weitere von Rakuten Group, Inc. (20)

Aktuellste (20)

Anzeige

Rakuten LeoFs - distributed file system

  1. S3 Compatible Storage “LeoFS” Rakuten. Inc, RIT Yosuke Hara 25/05/2012 1
  2. Table of Contents 1. Motivation 2. Overview 3. Inside of LeoFS 4. WEB Console 2
  3. 1. Motivation 3
  4. Why NFS? Which is suitable storage for storing the media files? ? Low ROI Possibility of SPOF Storage Expansion is difficult during increasing data 4
  5. 5
  6. Object Storage Farm for IaaS 6
  7. 2. Overview 7
  8. Overview Storage Gateway Manager 8
  9. System Layout Request from Web Application(s) Load Balancer LeoFS-Manager S3-API REST over HTTP LeoFS-Gateway RPC w/Cache Server SNMP RPC LeoFS-Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router WEB Console META Object Store META Object Store META Object Store 9
  10. System Layout Request from Web Application(s) Load Balancer Gateway Manager LeoFS-Manager HTTP Cluster Request/Response Handling S3-API Management REST over HTTP LeoFS-Gateway + RPC w/Object Cache w/Cache Server Ring Watcher (AWS S3-API) Node Watcher SNMP RPC LeoFS-Storage Storage GUI Console Object Storage, Meta data Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router + META Replicator/Recoverer Object Store META META Object Store Object Store 10
  11. 3. Inside of LeoFS 11
  12. Architecture HTTP Gateway (stateless) Erlang RPC Erlang RPC Storage Cluster (multi master) Erlang RPC Process Monitor Manager Cluster 12
  13. Architecture Architecture - Gateway/Storage LeoFS Gateway Cacher REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC LeoFS-Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage Metadata Storage 13
  14. Architecture Architecture - Manager Cluster Erlang Mnesia RING Member / Cluster State Auth / ACL Process Monitor Gateway / Storage Cluster 14
  15. Architecture 15
  16. Three “HIGH” High Cost Performance Monolithic Storage System Storage Engine For Unstructured Files Traffic Restrain Mechanism > File Cache System (Gateway Plugin) 16
  17. Three “HIGH” High Reliability NO SPOF Split Brain Measure “Erlang OTP” > Nine Nines (99.9999999%) 17
  18. Three “HIGH” High Scalability Elastic Storage System > Able to dynamic attach/detach nodes > Able to over 100-nodes > NOT Mesh-connected Mutual Servers 18
  19. Cache Mechanism 19
  20. Cache Mechanism Gateway Buffer Pool Slab Alloc Skiplist {$filename, $etag} request from Client response Gateway match: {ok, match} NOT match: {ok, $metadata, $body,} High I/O efficiency Low Latency 20
  21. Storage Engine 21
  22. Storage Engine Metadata + Object Storage LeoFS-Storage Storage Engine / Replicator / Recoverer Object’s Attribute Storage (metadata) Object Storage Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store. 22
  23. Storage Engine Retrieve an object from Storage Storage Engine < META DATA > ID Filename Offset Size Checksum Data Header Metadata File Footer Object Container 23
  24. Storage Engine Insert an object into Storage Storage Engine Add a Metadata Meta Data Server Data Append a File 24
  25. Storage Engine Reduce unnecessary files Compaction 25
  26. 4. WEB Console 26
  27. Web Console File Manager WEB Console Node Stats Log Search 27
  28. Web Console Node State Monitor Log Analyzer / Searcher 28
  29. Web Console Web Console System Layout Manager Storage Gateway or GUI Console Producers and Admins Logstatsh 29
  30. Wrap Up 30
  31. Wrap Up High Cost Performance High Reliability High Scalability 31
  32. 32
  33. Thank you for your time 33

×