Typical Memcached deployments do not comprehensively address web site requirements for high availability. Depending on your web architecture, a single failure can disable your web caches. This presentation offers real world solutions to solving <a>high availability</a> challenges common to large, dynamic websites with Memcached, specifically: * Options and benefits for deploying high availability services within Memcached * How companies are approaching high availability * Considerations on building and deploying high availability o Recommendations for a typical Memcached environment o Open source tools available o High level costs for deployment

1. Creating High Availability Services for Your
Memcached Environment
Presented by:
Bill Takacs – Director, Product Management
25 August 2009

2. Agenda
• Overview of Memcached
• High Availability defined
• Why HA Memcached
• Options for HA Memcached
• Gear6 Solution
2 : Copyright 2009 Gear6 Inc.

3. Memcached
“A high performance, distributed memory object
caching system, generic in nature, but intended for
use in speeding up dynamic web applications by
alleviating database load”
Ref: http://www.danga.com/memcached/
• Big hash table
• Created by Danga Interactive for Live Journal
• Significantly reduced database load
• Perfect for web sites with high database load
• In use by Facebook, Twitter, MyYearBook, others
3 : Copyright 2009 Gear6 Inc.

4. More on Memcached
• Takes advantage of available DRAM
• Open source
• Distributed under BSD license
• Server - Current version is 1.4
» http://www.danga.com/memcached/download.bml
• Many Clients
» http://code.google.com/p/memcached/wiki/Clients
4 : Copyright 2009 Gear6 Inc.

5. How Does Memcached Work?
• Client - Server
• Hash Function
• Memcached
» Two stage
» Key / value pair
5 : Copyright 2009 Gear6 Inc.

6. Memcached Server
• Slab allocator
• Libevent based
• Simple Protocol (no xml)
• Server has internal hash table
• Servers are dumb – they don’t know about each other
6 : Copyright 2009 Gear6 Inc.

7. Memcached Server
• Limits
» Key size = (250 bytes)
» 1MB Limit
» 32bit/64bit (maximum size of process)
• LRU
» Least recently accessed items are cycled out
» One LRU exists per “slab class”
» LRU “evictions” need not be common
• Threads? Yes
7 : Copyright 2009 Gear6 Inc.

8. Memcached Clients
• Client hashes key to server list
• Serializes the object
• Many client libraries
• Authentication
8 : Copyright 2009 Gear6 Inc.

9. Commands
• Storage commands:
» Set / Add / Replace /
Append / Prepend / CAS
• Retrieval commands:
» Get / Gets
• Delete / increment /
decrement
• Stats
9 : Copyright 2009 Gear6 Inc.

10. High Availability
• High availability: System
design and implementation
that ensures a certain
absolute degree of
operational continuity
during a given measurement
period.
• Organizations
» OpenSAF Foundation
» SAFourm
• HA for Memcache
10 : Copyright 2009 Gear6 Inc.

11. Key Concepts
• Clustering
• Failover
• Replication
11 : Copyright 2009 Gear6 Inc.

12. The case for HA Memcache
• What Memcache is NOT:
» A persistent data store
» A database
» Application specific
» Clustered
» A large object cache
• The intent is if its not in cache hit the DB
• Great when your site’s traffic is small
• Let’s look at the evolution of a site
12 : Copyright 2009 Gear6 Inc.

13. The Beginning
Web Servers
MySQL
Database
13 : Copyright 2009 Gear6 Inc.

14. Web traffic grows
Memcached Web Servers
Memcached
MySQL Memcached
Database
Dynamic content:
Result: Site slows down or worse site unavailable
14 : Copyright 2009 Gear6 Inc.

15. Add Memcached as traffic increases
Web Servers
MySQL
Memcached
Database
Memcached
Increases performance but as the site grows so does Memcached
15 : Copyright 2009 Gear6 Inc.

16. Even more traffic
Web Servers
MySQL
Databases
Memcached
Memcached Memcache Server(s)
Memcached
Reliance on memcached for performance
16 : Copyright 2009 Gear6 Inc.

17. Options for HA Memcached
Not Many…
• DIY
• Or - Find a solution that leverages memcached that
has high availability features
17 : Copyright 2009 Gear6 Inc.

18. Consistent Hashing
• Why?
• How does it work?
• Where
18 : Copyright 2009 Gear6 Inc.

19. Consistent Hashing - Example
• PHP – Flexihash – code.google.com/p/flexihash
<?php
$hash = new Flexihash();
// bulk add
$hash->addTargets(array('cache-1', 'cache-2', 'cache-3'));
// simple lookup
$hash->lookup('object-a'); // "cache-1"
$hash->lookup('object-b'); // "cache-2"
// add and remove
$hash
->addTarget('cache-4')
->removeTarget('cache-1');
// lookup with next-best fallback (for redundant writes)
$hash->lookupList('object', 2); // ["cache-2", "cache-4"]
// remove cache-2, expect object to hash to cache-4
$hash->removeTarget('cache-2');
$hash->lookup('object'); // "cache-4"
19 : Copyright 2009 Gear6 Inc.

20. Replication
• How
» Repcache
» Libmemcached
• Why
» Two datacenters – master / slave dc’s
» Keys stored on multiple machines to make them ‘more”
available
20 : Copyright 2009 Gear6 Inc.

21. Replication - Repcache Example
• Downloaded the memcached-(version)-repcached-
(version).tar.gz source and simply compiled it.
./configure --enable-replication
make
sudo make install
Start two instances of memcached on ports 11211 & 11222
btak@csn07:~/php$ memcached -p 11211 -m 64 -x 127.0.0.1 -v
replication: connect (peer=127.0.0.1:11212)
replication: marugoto copying
replication: close
replication: listen
btak@csn07:~/php$ memcached -p 11222 -m 64 -x 127.0.0.1 -v
replication: connect (peer=127.0.0.1:11212)
replication: marugoto copying
Replication: start
21 : Copyright 2009 Gear6 Inc.

22. Replication – Repcache Example Cont
Now set and get a value on instance on Connect to port 11222 and try getting
port 11211 this value
btak@csn07:~$ telnet localhost 11211 btak@csn07:~$ telnet localhost 11222
Trying 127.0.0.1... • Trying 127.0.0.1...
Connected to localhost. • Connected to localhost.
Escape character is '^]'. • Escape character is '^]'.
set hello 0 0 5 • get hello
world • VALUE hello 0 5
STORED • world
get hello • END
VALUE hello 0 5
world
END
22 : Copyright 2009 Gear6 Inc.

23. Replication – Repcache Example Cont
Suppose server on 11211 goes down Let’s see if its replicated on 11211
On port 11211 btak@csn07:~$ telnet localhost 11211
Killed Trying 127.0.0.1...
btak@csn07:~/php$ memcached -11211 Connected to localhost.
-m 64 -x 127.0.0.1 -v Escape character is '^]'.
replication: connect get hello
(peer=127.0.0.1:11222) VALUE hello 0 5
replication: marugoto copying world
replication: start END
get key
On port 11222
VALUE key 0 5
<---snip
myval
replication: close
END
replication: listen
replication: accept
replication: marugoto start
replication: marugoto 2
replication: marugoto owari
<---snip-->
23 : Copyright 2009 Gear6 Inc.

24. Clustering
• www.linux-ha.org
• Provides high availability clustering solution
• Can enable memcached cluster
• Combine with consistent hashing
24 : Copyright 2009 Gear6 Inc.

25. About Gear6
• First and leading provider
of Memcached solutions
• Memcached solution including
• High density
• High Availability
• Advanced memory
management
• Enhanced reporting
capabilities
• Support for multi-tenancy
• Disruption free
software upgrades
• 100% client compatible
25 : Copyright 2009 Gear6 Inc.

26. References
• http://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-
deployment.html
• http://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-
hashtypes.html
• http://jayant7k.blogspot.com/2009/04/memcached-replication.html
• http://www.lexemetech.com/2007/11/consistent-hashing.html
• http://www8.org/w8-papers/2a-
webserver/caching/paper2.htmlhttp://www.last.fm/user/RJ/journal/2007/04/10/rz
_libketama_-_a_consistent_hashing_algo_for_memcache_clients
• http://bazaar.launchpad.net/~libmemcached-
developers/libmemcached/trunk/revision/539
26 : Copyright 2009 Gear6 Inc.

27. Questions?
Thank you for attending
“Creating High Availability Services for Your Memcached
Environment presented” by Gear6
Bill Takacs
Gear6
salesinfo@gear6.com
+1 650 587 7118
27 : Copyright 2009 Gear6 Inc.