Everything You Need to Know About Membase

San Diego

Thursday, January 6, 2011

What is Membase?


Membase is a distributed database

Application user

Web application server Web application server Web application server

Membase Servers

In the data center On the administrator console

3


Membase is Simple, Fast, Elastic

Five minutes or less to a working
cluster
• Downloads for Linux and Windows
• Start with a single node
• One button press joins nodes to a cluster
Easy to develop against
• Just SET and GET – no schema required
• Drop it in. 10,000+ existing applications
already “speak membase” (via memcached)
• Practically every language and application
framework is supported, out of the box
Easy to manage
• One-click failover and cluster rebalancing
• Graphical and programmatic interfaces
• Conﬁgurable alerting

4



Predictable
• “Never keep an application waiting”
• Quasi-deterministic latency and throughput
Low latency
• Built-in Memcached technology
High throughput
• Multi-threaded
• Low lock contention
• Asynchronous wherever possible
• Automatic write de-duplication

5



Zero-downtime elasticity
• Spread I/O and data across commodity
servers (or VMs)
• Consistent performance with linear cost
• Dynamic rebalancing of a live cluster
All nodes are created equal
• No special case nodes
• Any node can replace any other node, online
• Clone to grow
Extensible
• Filtered TAP interface provides hook points
for external systems (e.g. full-text search,
backup, warehouse)
• Data bucket – engine API for specialized
container types

6


Built-in Memcached Caching Layer

Memcached Membase Cache

Membase Database Membase Database

Memcached Mode Membase Mode

Fact: Membase development team has also contributed over
half of the code to the Memcached project. 7


Use Cases


Ad targeting

40 milliseconds to come
up with an answer.

profiles, real time campaign
3 statistics

2
1 profiles, campaigns
events

9


Search and Gaming Portal

Database

10


Membase Caching at The Knot
Jason Sirota


Membase Architecture


Clustering
• Underlying cluster
functionality based on
erlang OTP
• Have a custom, vector
clock based way of storing
and propagating...
– Cluster topology
– vBucket mapping
• Collect statistics from many
nodes of the cluster
– Identify hot keys, resource
utilization

13


TAP

• A generic, scalable method of streaming mutations
from a given server
– As data operations arrive, they can be sent to arbitrary TAP
receivers
• Leverages the existing memcached engine interface,
and the non-blocking IO interfaces to send data
• Three modes of operation
Data
Mutations Working set

Data
Mutations Working set Working set

15


Membase data ﬂow – under the hood

SET request arrives at SET acknowledgement
KEY’s master server 1 2 returned to application

3 Listener-Sender
3

RAM

membase storage engine
4

Disk Disk Disk

Disk Disk Disk

Replica Server 1 for KEY Master server for KEY Replica Server 2 for KEY
16


Clients, nodes and other nodes

Client moxi + Client

port 11210
port 11211 memcached operations REST/comet
memcached operations cluster topology
and vbucket map

memcached operations
moxi ns_server

membase ns_server
(memcached + membase engine)

TAP
memcached operations
with tap commands vbucketmigrator

17


Data buckets are secure membase “slices”

Application user

Web application server

Bucket 1
Bucket 2
Aggregate Cluster Memory and Disk Capacity

Membase data servers

In the data center On the administrator console
18


vBucket mapping

19


Disk > Memory

Dataset may have many memory quota

Bucket Conﬁguration
items infrequently accessed. mem_high_wat

However, memcached has mem_low_wat
different behavior (LRU) than
wanted with membase.

Still, traditional (most)
RDBMS implementations are
not 100% correct for us
either. The speed of a miss
is very, very important.

20


Membase Demo


Key-Value Patterns


Key-Value

23


Key-Value

Items have: Operations include:
Key Get/Set
Value Increment/Decrement
Expiration Append/Prepend
Flags
CAS (more on this later)
23


Key-Value

(with a replica )
Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/

23


Membase Datatypes

24


Membase Datatypes

• byte[]
– Does your data have
1s and 0s?

24


Membase Datatypes

• byte[]
1s and 0s?

“Any customer can have
a car painted any colour
that he wants so long as
it is black.”

24


Membase Datatypes

• byte[]
1s and 0s?
• Items do have flags
– Many clients use flags
– Data type options
• Google protobuf “Any customer can have
a car painted any colour
• Thrift that he wants so long as
• Avro it is black.”

24


Transactions

• Lock == slow me down
• CAS operations User 1 User 2
– Optimistic locking

Su
• Very useful with complex

cc

il!
es

Fa
datatypes

s
– Imagine two clients trying to
update a complex item
• You’re likely using CAS
already... if you use a CPU

25


Common Use: Sessions

• Web user sessions
– Highly read, less writes in many case
– Protocol advantage of memcached
• Options already for PHP, Ruby and Java
• Application state
– Not necessarily “entity” style things
– May be appropriate for a “cache” pool

26


Common Use (cache): Rate Limiting

• Want to provide API calls
into the system Your Users

– Twitter search h !
uc
– Google search services ¡O

• Use the atomic increment Your App

– Set an item with a unique ID
– Upon API request,
increment and check
• HTTP 420: go away and come
back later

27


Looking Ahead: NodeCode


NodeCode – Motivation

Beyond key-value
• Indexing/Range Queries
• Advanced Data Structures
• Sub-object direct manipulation

Validation and In-ﬂight transformation
• Block mutations failing validation
• Enrich or transform objects

Connectors (Integrate easily with other systems)
• Solr
• Hadoop
• MySQL

29


Q&A

31


Attributions

• http://commons.wikimedia.org/wiki/File:Flag_of_China.png
• http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg
• http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg

32


Everything You Need to Know About Membase

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (9)

Ähnlich wie Everything You Need to Know About Membase

Ähnlich wie Everything You Need to Know About Membase (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Everything You Need to Know About Membase