SlideShare ist ein Scribd-Unternehmen logo
1 von 23
Downloaden Sie, um offline zu lesen
DISTRIBUTED STORAGE
SYSTEM
Mr. Dương Công Lợi
Company: VNG-Corp
Tel: +84989510016
Email:loiduongcong@gmail.com
CONTENTS
 1. What is distributed-computing system?
 2. Principle of distributed database/storage
system
 3. Distributed storage system paradigm
 4. UniversalDistributedStorage
1. WHAT IS DISTRIBUTED-COMPUTING
SYSTEM?
 Distributed-Computing is the process of solving a
computational problem using a distributed
system.
 A distributed system is a computing system in
which a number of components on multiple
computers cooperate by communicating over a
network to achieve a common goal.
DISTRIBUTED DATABASE/STORAGE
SYSTEM
 A distributed database system, the database is
stored on several computers .

 A distributed database is a collection of multiple
, Logic computer network .
DISTRIBUTED SYSTEM ADVANCE
 Advance
 Avoid bottleneck & single-point-of-failure
 More Scalability
 More Availability
 Routing model
 Client routing: client request to appropriate server to
read/write data
 Server routing: server forward request of client to
appropriate server and send result to this client
* can combine the two model above into a system
DISTRIBUTED STORAGE SYSTEM
 Store some data {1,2,3,4,6,7,8} into 1 server
 And store them into 3 distributed server
1,2,3,4,
6,7,8
1,2,3
4,6
7,8
2. PRINCIPLE OF DISTRIBUTED
DATABASE/STORAGE SYSTEM
 Shard data key and store it to appropriate server
use Distributed Hash Table (DHT)
 DHT must be consistent hashing:
 Uniform distribution of generation
 Consistent
 Jenkins, Murmur are the good choice; MD5, SHA
slower
CANONICAL PROBLEMS IN DISTRIBUTED
SYSTEMS
 Distributed data independence
 Distributed transactions: ACID (Atomicity,
Consistency, Isolation, Durability) requirement
 Fault tolerance
 Transparency
3. DISTRIBUTED STORAGE SYSTEM
PARADIGM
 Data Hashing/Addressing
 Determine server for data store in
 Data Replication
 Store data into multi server node for more available,
fault-tolerance
DISTRIBUTED STORAGE SYSTEM
ARCHITECT
 Data Hashing/Addressing
 Use DHT to addressing server (use server-name) to a
number, performing it on one circle called the keys
space
 Use DHT to addressing data and find server store it
by successor(k)=ceiling(addressing(k))
 successor(k): server store k
0
server3
server1
server2
DISTRIBUTED STORAGE SYSTEM
ARCHITECT
 Addressing – Virtual node
 Each server node is generated to more node-id for
evenly distributed, load balance
Server1: n1, n4, n6
Server2: n2, n7
Server3: n3, n5
0
server3
server1
server2
n7
n1
n5
n2
n4
n6
n3
n6
DISTRIBUTED STORAGE SYSTEM
ARCHITECT
 Data Replication
Data k1 store in server1 as master and store in
server2 as slave
0
server3
server1
server2
k1
UNIVERSALDISTRIBUTEDSTORAGE
a distributed storage system
4. UNIVERSALDISTRIBUTEDSTORAGE
 UniversalDistributedStorage is a distributed
storage system develop for:
 Distributed data independence
 Distributed transactions (ACID)
 Fault tolerance
 Leader election (decision for join or leave server node)
 Replicate with multiple master replication
 Transparency
UNIVERSALDISTRIBUTEDSTORAGE
ARCHITECTURE
 Overview
Bussiness
Layer
Distrib
uted
Layer
Storage
Layer
Bussiness
Layer
Distrib
uted
Layer
Storage
Layer
Bussiness
Layer
Distrib
uted
Layer
Storage
Layer
ARCHITECTURE OVERVIEW
UNIVERSALDISTRIBUTEDSTORAGE
FEATURE
 Data hashing/addressing
 Use Murmur hashing function
UNIVERSALDISTRIBUTEDSTORAGE
FEATURE
 Leader election
 Use Bully Leader Election algorithm
UNIVERSALDISTRIBUTEDSTORAGE
FEATURE
 Multi-master replication
 Problem of multi-master replication
UNIVERSALDISTRIBUTEDSTORAGE
FEATURE
 Multi-master replication
 Data store to main master (called sub-leader), then
this data post to queue to sync to other master.
UNIVERSALDISTRIBUTEDSTORAGE
STATISTIC
 System information:
 3 machine 8GB Ram, core i5 3,220GHz
 LAN/WAN network
 7 physical servers on 3 above mechine
 Concurrence write 16500000 items in 3680s, rate~
4480req/sec (at client computing)
 Concurrence read 16500000 items in 1458s, rate~
11320req/sec (at client computing)
* It doesn’t limit of this system, it limit at clients (this
test using 3 client thread)
Q & A
Contact:
Duong Cong Loi
loiduongcong@gmail.com
https://www.facebook.com/duongcong.loi

Weitere ähnliche Inhalte

Was ist angesagt?

Sensor Protocols for Information via Negotiation (SPIN)
Sensor Protocols for Information via Negotiation (SPIN)Sensor Protocols for Information via Negotiation (SPIN)
Sensor Protocols for Information via Negotiation (SPIN)
rajivagarwal23dei
 
Localization in WSN
Localization in WSNLocalization in WSN
Localization in WSN
Yara Ali
 
Introduction to Data-Link Layer
Introduction to Data-Link LayerIntroduction to Data-Link Layer
Introduction to Data-Link Layer
Abdullaziz Tagawy
 
security in wireless sensor networks
security in wireless sensor networkssecurity in wireless sensor networks
security in wireless sensor networks
Vishnu Kudumula
 

Was ist angesagt? (20)

Routing Protocols in WSN
Routing Protocols in WSNRouting Protocols in WSN
Routing Protocols in WSN
 
Security issues in manet
Security issues in manetSecurity issues in manet
Security issues in manet
 
Sensor Protocols for Information via Negotiation (SPIN)
Sensor Protocols for Information via Negotiation (SPIN)Sensor Protocols for Information via Negotiation (SPIN)
Sensor Protocols for Information via Negotiation (SPIN)
 
Routing protocols for ad hoc wireless networks
Routing protocols for ad hoc wireless networks Routing protocols for ad hoc wireless networks
Routing protocols for ad hoc wireless networks
 
Localization in WSN
Localization in WSNLocalization in WSN
Localization in WSN
 
WLAN
WLANWLAN
WLAN
 
Eucalyptus, Nimbus & OpenNebula
Eucalyptus, Nimbus & OpenNebulaEucalyptus, Nimbus & OpenNebula
Eucalyptus, Nimbus & OpenNebula
 
Storage Area Network (San)
Storage Area Network (San)Storage Area Network (San)
Storage Area Network (San)
 
Introduction to Data-Link Layer
Introduction to Data-Link LayerIntroduction to Data-Link Layer
Introduction to Data-Link Layer
 
RPL - Routing Protocol for Low Power and Lossy Networks
RPL - Routing Protocol for Low Power and Lossy NetworksRPL - Routing Protocol for Low Power and Lossy Networks
RPL - Routing Protocol for Low Power and Lossy Networks
 
security in wireless sensor networks
security in wireless sensor networkssecurity in wireless sensor networks
security in wireless sensor networks
 
Domain name system
Domain name systemDomain name system
Domain name system
 
WSN IN IOT
WSN IN IOTWSN IN IOT
WSN IN IOT
 
Client Centric Consistency Model
Client Centric Consistency ModelClient Centric Consistency Model
Client Centric Consistency Model
 
MEDIUM ACCESS CONTROL
MEDIUM ACCESS CONTROLMEDIUM ACCESS CONTROL
MEDIUM ACCESS CONTROL
 
netconf and yang
netconf and yangnetconf and yang
netconf and yang
 
Wireless Sensor Networks
Wireless Sensor NetworksWireless Sensor Networks
Wireless Sensor Networks
 
Cs6703 grid and cloud computing unit 3
Cs6703 grid and cloud computing unit 3Cs6703 grid and cloud computing unit 3
Cs6703 grid and cloud computing unit 3
 
SENSOR NETWORK PLATFORMS AND TOOLS
SENSOR NETWORK PLATFORMS AND TOOLSSENSOR NETWORK PLATFORMS AND TOOLS
SENSOR NETWORK PLATFORMS AND TOOLS
 
Mac protocols for ad hoc wireless networks
Mac protocols for ad hoc wireless networks Mac protocols for ad hoc wireless networks
Mac protocols for ad hoc wireless networks
 

Andere mochten auch

7 distributed storage_open_stack
7 distributed storage_open_stack7 distributed storage_open_stack
7 distributed storage_open_stack
openstackindia
 
Strategies for Distributed Data Storage
Strategies for Distributed Data StorageStrategies for Distributed Data Storage
Strategies for Distributed Data Storage
kakugawa
 

Andere mochten auch (20)

7 distributed storage_open_stack
7 distributed storage_open_stack7 distributed storage_open_stack
7 distributed storage_open_stack
 
DumpFS - A Distributed Storage Solution
DumpFS - A Distributed Storage SolutionDumpFS - A Distributed Storage Solution
DumpFS - A Distributed Storage Solution
 
Distribute Storage System May-2014
Distribute Storage System May-2014Distribute Storage System May-2014
Distribute Storage System May-2014
 
Survey of distributed storage system
Survey of distributed storage systemSurvey of distributed storage system
Survey of distributed storage system
 
Deploying pNFS over Distributed File Storage w/ Jiffin Tony Thottan and Niels...
Deploying pNFS over Distributed File Storage w/ Jiffin Tony Thottan and Niels...Deploying pNFS over Distributed File Storage w/ Jiffin Tony Thottan and Niels...
Deploying pNFS over Distributed File Storage w/ Jiffin Tony Thottan and Niels...
 
Strategies for Distributed Data Storage
Strategies for Distributed Data StorageStrategies for Distributed Data Storage
Strategies for Distributed Data Storage
 
Tachyon: An Open Source Memory-Centric Distributed Storage System
Tachyon: An Open Source Memory-Centric Distributed Storage SystemTachyon: An Open Source Memory-Centric Distributed Storage System
Tachyon: An Open Source Memory-Centric Distributed Storage System
 
A Design of Distributed Storage System over HTTP for Collecting Sensor Data
A Design of Distributed Storage System over HTTP for Collecting Sensor DataA Design of Distributed Storage System over HTTP for Collecting Sensor Data
A Design of Distributed Storage System over HTTP for Collecting Sensor Data
 
Distributed storage performance for OpenStack clouds using small-file IO work...
Distributed storage performance for OpenStack clouds using small-file IO work...Distributed storage performance for OpenStack clouds using small-file IO work...
Distributed storage performance for OpenStack clouds using small-file IO work...
 
Torus
TorusTorus
Torus
 
Ceph - A distributed storage system
Ceph - A distributed storage systemCeph - A distributed storage system
Ceph - A distributed storage system
 
Auditing Distributed Preservation Networks
Auditing Distributed Preservation Networks Auditing Distributed Preservation Networks
Auditing Distributed Preservation Networks
 
[HATCH! FAIR 2013] Decision Making for Startups - Mr. Nguyen Tat Dac
[HATCH! FAIR 2013] Decision Making for Startups - Mr. Nguyen Tat Dac[HATCH! FAIR 2013] Decision Making for Startups - Mr. Nguyen Tat Dac
[HATCH! FAIR 2013] Decision Making for Startups - Mr. Nguyen Tat Dac
 
ICDE2015 Research 3: Distributed Storage and Processing
ICDE2015 Research 3: Distributed Storage and ProcessingICDE2015 Research 3: Distributed Storage and Processing
ICDE2015 Research 3: Distributed Storage and Processing
 
Clustered and distributed
 storage with
 commodity hardware 
and open source ...
Clustered and distributed
 storage with
 commodity hardware 
and open source ...Clustered and distributed
 storage with
 commodity hardware 
and open source ...
Clustered and distributed
 storage with
 commodity hardware 
and open source ...
 
Tachyon workshop 2015-07-19
Tachyon workshop 2015-07-19Tachyon workshop 2015-07-19
Tachyon workshop 2015-07-19
 
Identity Based Secure Distributed Storage Scheme
Identity Based Secure Distributed Storage SchemeIdentity Based Secure Distributed Storage Scheme
Identity Based Secure Distributed Storage Scheme
 
Integrated Distributed Solar and Storage
Integrated Distributed Solar and StorageIntegrated Distributed Solar and Storage
Integrated Distributed Solar and Storage
 
Use Distributed Filesystem as a Storage Tier
Use Distributed Filesystem as a Storage TierUse Distributed Filesystem as a Storage Tier
Use Distributed Filesystem as a Storage Tier
 
Distributed Storage and Compute With Ceph's librados (Vault 2015)
Distributed Storage and Compute With Ceph's librados (Vault 2015)Distributed Storage and Compute With Ceph's librados (Vault 2015)
Distributed Storage and Compute With Ceph's librados (Vault 2015)
 

Ähnlich wie Distributed storage system

Fredrick Ishengoma - HDFS+- Erasure Coding Based Hadoop Distributed File System
Fredrick Ishengoma -  HDFS+- Erasure Coding Based Hadoop Distributed File SystemFredrick Ishengoma -  HDFS+- Erasure Coding Based Hadoop Distributed File System
Fredrick Ishengoma - HDFS+- Erasure Coding Based Hadoop Distributed File System
Fredrick Ishengoma
 
Distributed operating system
Distributed operating systemDistributed operating system
Distributed operating system
Moeez Ahmad
 

Ähnlich wie Distributed storage system (20)

Apache ignite v1.3
Apache ignite v1.3Apache ignite v1.3
Apache ignite v1.3
 
Fredrick Ishengoma - HDFS+- Erasure Coding Based Hadoop Distributed File System
Fredrick Ishengoma -  HDFS+- Erasure Coding Based Hadoop Distributed File SystemFredrick Ishengoma -  HDFS+- Erasure Coding Based Hadoop Distributed File System
Fredrick Ishengoma - HDFS+- Erasure Coding Based Hadoop Distributed File System
 
1 ddbms jan 2011_u
1 ddbms jan 2011_u1 ddbms jan 2011_u
1 ddbms jan 2011_u
 
cluster computing
cluster computingcluster computing
cluster computing
 
Data Grids with Oracle Coherence
Data Grids with Oracle CoherenceData Grids with Oracle Coherence
Data Grids with Oracle Coherence
 
final-unit-ii-cc-cloud computing-2022.pdf
final-unit-ii-cc-cloud computing-2022.pdffinal-unit-ii-cc-cloud computing-2022.pdf
final-unit-ii-cc-cloud computing-2022.pdf
 
Data Redundancy on Diskless Client using Linux Platform
Data Redundancy on Diskless Client using Linux PlatformData Redundancy on Diskless Client using Linux Platform
Data Redundancy on Diskless Client using Linux Platform
 
Basics of the Highly Available Distributed Databases - teowaki - javier ramir...
Basics of the Highly Available Distributed Databases - teowaki - javier ramir...Basics of the Highly Available Distributed Databases - teowaki - javier ramir...
Basics of the Highly Available Distributed Databases - teowaki - javier ramir...
 
Everything you always wanted to know about highly available distributed datab...
Everything you always wanted to know about highly available distributed datab...Everything you always wanted to know about highly available distributed datab...
Everything you always wanted to know about highly available distributed datab...
 
Distributed operating system
Distributed operating systemDistributed operating system
Distributed operating system
 
Highly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowakiHighly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowaki
 
getFamiliarWithHadoop
getFamiliarWithHadoopgetFamiliarWithHadoop
getFamiliarWithHadoop
 
Cassandra
CassandraCassandra
Cassandra
 
IRJET- Distributed Decentralized Data Storage using IPFS
IRJET- Distributed Decentralized Data Storage using IPFSIRJET- Distributed Decentralized Data Storage using IPFS
IRJET- Distributed Decentralized Data Storage using IPFS
 
Unit-1 Introduction to Big Data.pptx
Unit-1 Introduction to Big Data.pptxUnit-1 Introduction to Big Data.pptx
Unit-1 Introduction to Big Data.pptx
 
Hadoop data management
Hadoop data managementHadoop data management
Hadoop data management
 
Hadoop
HadoopHadoop
Hadoop
 
L018137479
L018137479L018137479
L018137479
 
BDA Mod2@AzDOCUMENTS.in.pdf
BDA Mod2@AzDOCUMENTS.in.pdfBDA Mod2@AzDOCUMENTS.in.pdf
BDA Mod2@AzDOCUMENTS.in.pdf
 
Apache cassandra
Apache cassandraApache cassandra
Apache cassandra
 

Kürzlich hochgeladen

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Kürzlich hochgeladen (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 

Distributed storage system