SlideShare ist ein Scribd-Unternehmen logo

分散Key-valueストアkumofsの思想と設計

•
•
Sadayuki Furuhashi
Sadayuki Furuhashi

丸レクセミナー2010 分散Key-valueストア「kumofs」について紹介いたします。近年では、スケーラビリティを重視した分散データストアが数多く登場してきました。その中でkumofsは、特に可用性の高さや動的な運用のしやすさを重視しており、また、高い単体性能を誇ります。このセッションではkumofsの特性についてご紹介すると共に、稼働中のサーバの追加やサーバ故障時の復旧方法などのデモを交えながら、kumofsの運用について具体的に紹介いたします。

Technologie
1 von 72
Downloaden Sie, um offline zu lesen
kumofs
mpio MessagePack
100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
Server Server Manager Gateway
Tokyo Cabinet Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
Server Server Server Server Server Server
Server Manager Gateway
Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
Server Server Server Manager Server Server Manager Gateway Application
Server Server Server Manager Server Server Manager Gateway Application
Server Server Server Manager Server Server Manager Server Gateway Application
Gateway Application
Server Server Server localhost:11211 Gateway RPC / Application
Server Server Server Manager
Manager Manager Server Server svr001 svr002
Manager Manager Server Server Server svr001 svr002 svr003
Manager Manager Server Server Server svr001 svr002 svr003 Server svr005
Manager Manager Server svr001 svr002 svr003 Server Server Server svr005 svr006 svr007
110,000 req/sec 114,000 req/sec 114,000 85,500 Linux 2.6.27.10 x86_64 AMD Athlon 64 X2 5000+ 57,000 Linux 2.6.27.19 x86_64 AMD Athlon 64 X2 4800+ 28,500 key 32 value 1KB 8 32 key 960,000 key 0 memcached kumofs
(requests/sec) 3,200,000 kumofs 2,400,000 Linux 2.6.27.21 i686 Intel Pentium 4 3.20GHz Linux 2.6.27.21 x86_64 Intel QuadCore Xeon X3350 1,600,000 50 10 60 key 8 value 32 32 800,000 256 key 51,200,000 key 0 10 20 30 40 50 60 ( )
Demo

Empfohlen

NoSQLを知る
NoSQLを知るNoSQLを知る
NoSQLを知るSadayuki Furuhashi
 
Ethereum blockchain
Ethereum blockchainEthereum blockchain
Ethereum blockchainMad Devs
 
Docker-OVS
Docker-OVSDocker-OVS
Docker-OVSsnrism
 
Misc
MiscMisc
MiscShamz Deez
 
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiIp sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiHuy Eav
 
愛のSuperCollider
愛のSuperCollider愛のSuperCollider
愛のSuperColliderMasayuki Akamatsu
 
An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)Naoto MATSUMOTO
 
Openv switchの使い方とか
Openv switchの使い方とかOpenv switchの使い方とか
Openv switchの使い方とかkotto_hihihi
 

Empfohlen

NoSQLを知る
NoSQLを知るNoSQLを知る
NoSQLを知るSadayuki Furuhashi
 
Ethereum blockchain
Ethereum blockchainEthereum blockchain
Ethereum blockchainMad Devs
 
Docker-OVS
Docker-OVSDocker-OVS
Docker-OVSsnrism
 
Misc
MiscMisc
MiscShamz Deez
 
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiIp sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiHuy Eav
 
愛のSuperCollider
愛のSuperCollider愛のSuperCollider
愛のSuperColliderMasayuki Akamatsu
 
An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)Naoto MATSUMOTO
 
Openv switchの使い方とか
Openv switchの使い方とかOpenv switchの使い方とか
Openv switchの使い方とかkotto_hihihi
 
Containers on Windows
Containers on WindowsContainers on Windows
Containers on WindowsPratik Khasnabis
 
How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineRitesh Rushiya
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]Naoto MATSUMOTO
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSRyo ONODERA
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingNaoto MATSUMOTO
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5Vasanth Reddy
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
 
Netmiko library
Netmiko libraryNetmiko library
Netmiko libraryManjunath annure
 
Docker Networking
Docker NetworkingDocker Networking
Docker NetworkingWeaveworks
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化Yoko Nakano
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloValerio Balbi
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpnAshwajit Maske
 
ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースSadayuki Furuhashi
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728jieun kim
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyLeif Hedstrom
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOIJohnson Liu
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Yongyoon Shin
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04Hiroshi SHIBATA
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSazuredayit
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014Hiroshi SHIBATA
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogicRakuten Group, Inc.
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)markmcclain
 

Weitere ähnliche Inhalte

Was ist angesagt?

How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineRitesh Rushiya
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]Naoto MATSUMOTO
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSRyo ONODERA
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingNaoto MATSUMOTO
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5Vasanth Reddy
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
 
Docker Networking
Docker NetworkingDocker Networking
Docker NetworkingWeaveworks
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化Yoko Nakano
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloValerio Balbi
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpnAshwajit Maske
 

Was ist angesagt? (12)

Containers on Windows
Containers on WindowsContainers on Windows
Containers on Windows
 
How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command line
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPS
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
 
Netmiko library
Netmiko libraryNetmiko library
Netmiko library
 
Docker Networking
Docker NetworkingDocker Networking
Docker Networking
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & Controllo
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpn
 

Ähnlich wie 分散Key-valueストアkumofsの思想と設計

ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースSadayuki Furuhashi
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728jieun kim
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyLeif Hedstrom
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOIJohnson Liu
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Yongyoon Shin
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04Hiroshi SHIBATA
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSazuredayit
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014Hiroshi SHIBATA
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogicRakuten Group, Inc.
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)markmcclain
 
Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio Soares
 
New idc architecture
New idc architectureNew idc architecture
New idc architectureMason Mei
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVMJohn Lee
 
Betting On Data Grids
Betting On Data GridsBetting On Data Grids
Betting On Data Gridsgojkoadzic
 
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewCloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewChip Childers
 
Run Run Trema Test
Run Run Trema TestRun Run Trema Test
Run Run Trema TestHiroshi Ota
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Joe Arnold
 
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamSharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamOfficience
 
Learn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceLearn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceOpenCity Community
 

Ähnlich wie 分散Key-valueストアkumofsの思想と設計 (20)

ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケース
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a Proxy
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOI
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)
 
Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11
 
New idc architecture
New idc architectureNew idc architecture
New idc architecture
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVM
 
Dfhghg
DfhghgDfhghg
Dfhghg
 
Betting On Data Grids
Betting On Data GridsBetting On Data Grids
Betting On Data Grids
 
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewCloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
 
Run Run Trema Test
Run Run Trema TestRun Run Trema Test
Run Run Trema Test
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012
 
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamSharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
 
Learn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceLearn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practice
 

Mehr von Sadayuki Furuhashi

Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Sadayuki Furuhashi
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesSadayuki Furuhashi
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Sadayuki Furuhashi
 
Fluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupFluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupSadayuki Furuhashi
 
DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？Sadayuki Furuhashi
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container EraSadayuki Furuhashi
 
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11Sadayuki Furuhashi
 
Fighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkFighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkSadayuki Furuhashi
 
Embulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダSadayuki Furuhashi
 
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsPlugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsSadayuki Furuhashi
 
Embulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderEmbulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderSadayuki Furuhashi
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Sadayuki Furuhashi
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreSadayuki Furuhashi
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi
 

Mehr von Sadayuki Furuhashi (20)

Scripting Embulk Plugins
Scripting Embulk PluginsScripting Embulk Plugins
Scripting Embulk Plugins
 
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
 
Making KVS 10x Scalable
Making KVS 10x ScalableMaking KVS 10x Scalable
Making KVS 10x Scalable
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics Pipelines
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理
 
Fluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupFluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes Meetup
 
DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
 
Fighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkFighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with Embulk
 
Embulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダ
 
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsPlugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGems
 
Embuk internals
Embuk internalsEmbuk internals
Embuk internals
 
Embulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderEmbulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loader
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
 
Prestogres internals
Prestogres internalsPrestogres internals
Prestogres internals
 
Presto+MySQLで分散SQL
Presto+MySQLで分散SQLPresto+MySQLで分散SQL
Presto+MySQLで分散SQL
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect More
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for Presto
 

Kürzlich hochgeladen

What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CVKhem
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 

Kürzlich hochgeladen (20)

What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 

分散Key-valueストアkumofsの思想と設計

  • 2. mpio MessagePack
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. 100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
  • 18. 100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37. Server Server Manager Gateway
  • 38. Tokyo Cabinet Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
  • 39. Server Server Server Server Server Server
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 56. Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
  • 57. Server Server Server Manager Server Server Manager Gateway Application
  • 58. Server Server Server Manager Server Server Manager Gateway Application
  • 59. Server Server Server Manager Server Server Manager Server Gateway Application
  • 61. Server Server Server localhost:11211 Gateway RPC / Application
  • 62.
  • 63. Server Server Server Manager
  • 64. Manager Manager Server Server svr001 svr002
  • 65. Manager Manager Server Server Server svr001 svr002 svr003
  • 66. Manager Manager Server Server Server svr001 svr002 svr003 Server svr005
  • 67. Manager Manager Server svr001 svr002 svr003 Server Server Server svr005 svr006 svr007
  • 68. 110,000 req/sec 114,000 req/sec 114,000 85,500 Linux 2.6.27.10 x86_64 AMD Athlon 64 X2 5000+ 57,000 Linux 2.6.27.19 x86_64 AMD Athlon 64 X2 4800+ 28,500 key 32 value 1KB 8 32 key 960,000 key 0 memcached kumofs
  • 69. (requests/sec) 3,200,000 kumofs 2,400,000 Linux 2.6.27.21 i686 Intel Pentium 4 3.20GHz Linux 2.6.27.21 x86_64 Intel QuadCore Xeon X3350 1,600,000 50 10 60 key 8 value 32 32 800,000 256 key 51,200,000 key 0 10 20 30 40 50 60 ( )
  • 70.
  • 71.
  • 72. Demo