Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Memcached
1. Memcached
http://download.tangent.org/talks/Memcached%20Study.pdf
Thursday, April 23, 2009
2. memcached is a high-performance, distributed memory object
caching system, generic in nature, but intended for use in
speeding up dynamic web applications by alleviating
database load.
Thursday, April 23, 2009
6. LiveJournal
• Origin of memcached
• 30G of cache. Terabytes of data
• Writes to DB based on reads from the DB,
not cache
Thursday, April 23, 2009
7. Mixi
• Memcached on dedicated servers
• 300 servers in production (reuse old
MySQL Servers)
• 4/8 gigs of Memory
• 15,000 qps / 400Mbps throughput
Thursday, April 23, 2009
13. Server
• Slab Allocator
• Libevent based
• Simple Protocol (no xml)
• Server has Internal Hash Table
• Servers know nothing about each other
Thursday, April 23, 2009
14. Clients
• Client hashes Key to Server List (distribution)
• Serializes the Object (server just likes byte arrays)
• Compresses data
Thursday, April 23, 2009
16. So what should I ask?
• How do I dump data?
• How is it redundant? *
• How does it handle failover?*
• How does it authenticate?
Thursday, April 23, 2009
17. Details on the Server?
• set/get/replace/add
• append/prepend
• increment/decrement
• cas (compare and swap atomic!)
• stats (detail)
Thursday, April 23, 2009
18. Platforms
• FreeBSD and Linux are top tier
• Windows exists
• Solaris (as of 1.2.5)
• OSX Good Support
Thursday, April 23, 2009
21. # Set up client object
require 'memcache'
servers = ['127.0.0.1:43042', '127.0.0.1:43043']
CACHE = MemCache.new(servers, :namespace => 'my_app')
Thursday, April 23, 2009
22. # Get; fall through to Rails' MySQL load if missing
key = "recent_posts"
# Try to get; returns nil on failure
posts = CACHE.get(key)
unless posts
# Load from DB
posts = Post.find(:all, :limit => 10, :order => 'id DESC')
# Cache the new value, which is automatically serialized
CACHE.set(key), posts, 60
end
Thursday, April 23, 2009
23. # Ghetto locking implementation for memcache-client
# from http://fauna.rubyforge.org/svn/interlock/trunk/lib/interlock/lock.rb
def lock(key, lock_expiry = 30, retries = 5)
retries.times do |count|
# Try to acquire the lock
response = CACHE.add("lock:#{key}", "Locked by #{Process.pid}", lock_expiry)
if response == "STOREDrn"
# We got it
begin
# Yield the current value to the closure
value = yield(CACHE.get(key))
# Set the new value returned from the closure CACHE.set(key, value)
# We're done ('ensure' block will still run)
return value
ensure
# Release the lock
CACHE.delete("lock:#{key}")
end
else
# Exponentially back off requests if the lock can't be acquired
sleep((2**count) / 2.0)
end
end
# We waited and waited but our turn never came
raise MemCacheError, "Couldn't acquire lock for #{key}"
end
Thursday, April 23, 2009
25. /**
* Initalize Memcache object and add local server 127.0.0.1 using port 11211
*/
$cache = new Memcache;
$cache->addServer("127.0.0.1",11211);
Thursday, April 23, 2009
26. /** Look in cache, if not found, set object (misses null) **/
if (!$user = $cache->get($user_key)) {
/**
* Set object with expire of 1 hour and no compression
*/
$cache->set($user_key,$value,NULL, $expire);
$user = $value;
}
Thursday, April 23, 2009
27. /* Get user counter value (does not handle add fail) */
if (!$counter = $cache->get($counter_key)) {
/* No counter, set to 1 */
$cache->add($counter_key,1);
} else {
/* Increment by 1 */
$cache->increment($counter_key);
}
Thursday, April 23, 2009
28. /* Print out stats from memcached server */
print_r($cache->getStats());
/* Print out extended stats from memcached server */
print_r($cache->getExtendedStats());
/* Flush memcached (bad idea in production)*/
var_dump($cache->flush());
Thursday, April 23, 2009
37. lighttpd/mod_memcache
• Cache files from disk
• Specify mime/types
• Create Expire Times
Thursday, April 23, 2009
38. Apache (mod_memcached)
• CAS operations exposed
• GET/PUT/DELETE operations
• Still Alpha
• (pandoraport.com!)
Thursday, April 23, 2009
39. NGINX
• Support variable Time outs
• Based on Perl
• http://wiki.codemongers.com/
NginxHttpMemcachedModule
• http://www.igvita.com/2008/02/11/nginx-
and-memcached-a-400-boost/
Thursday, April 23, 2009
43. Installation
• CREATE FUNCTION memc_servers_add RETURNS INT SONAME
"libmemcached_functions_mysql.so";
• CREATE FUNCTION memc_set RETURNS INT SONAME
"libmemcached_functions_mysql.so";
• CREATE FUNCTION memc_get RETURNS STRING SONAME
"libmemcached_functions_mysql.so";
Thursday, April 23, 2009
44. Functions Available
• memc_servers_set();
• memc_servers_behavior_set()
• memc_set()
• memc_get()
• memc_append()
• memc_prepend()
Thursday, April 23, 2009
46. Trigger
DROP TRIGGER IF EXISTS feed_insert;
CREATE TRIGGER feed_insert BEFORE INSERT ON feeds FOR EACH ROW
BEGIN SET @mm= memc_set(concat('feeds:',md5(NEW.url)),
NEW.url);END |
insert into feeds (url) values ('http://grazr.com/feedlist.xml');
select memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml')));
+------------------------------------------------------------------+|
memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml'))) |
+------------------------------------------------------------------+|
http://grazr.com/feedlist.xml |
+------------------------------------------------------------------+
Thursday, April 23, 2009
47. Memcached Replication
via MySQL
INSERT INTO table_a VALUES (“key”, “value”, “values”);
INSERT INTO blackhole_table VALUES (memc_delete(“key”));
Thursday, April 23, 2009
49. Limits
• Key Size (250 bytes) (1.2 <)
• Data Size (under 1 megabyte)
• 32bit/64bit (maximum size of the process)
• Maxbytes (limits item cache, not everything!)
Thursday, April 23, 2009
50. LRU
• Least recently accessed items are up for
eviction
• One LRU exists per “slab class”
• LRU evictions don't need to be common
• Can be common if you set expiration to 0
• Monitor evictions via 'stats items' command
Thursday, April 23, 2009
51. Threads
• No single-threading in versions past 1.2
• Great for large instances (16G+)
• Also great for large multiget requests
• Improvements in 1.3+
• Don't set too many threads
• One per CPU
• No more than 8 total for 1.2
Thursday, April 23, 2009
52. Slab Allocator
• Memory permanently allocated from OS
• Classes created by chunk size
• Cannot (presently) reassign slab pages
• Carefully use 'stats sizes' to test efficiency
Thursday, April 23, 2009
53. Binary Protocol
24 HEADER Required
bytes
EXTRA FIELD
(Command Specific)
Sizes are
defined in KEY As needed
the header
VALUE
* Values must be in network byte order.
Thursday, April 23, 2009
54. Binary Protocol
MAGIC Opcode Key Length
(1 byte) (1 byte) (2 bytes)
Extra Length Data Type Reserved
(1 byte) (1 byte) (2 bytes)
Total Body Length
(4 bytes)
Opaque
(4 bytes)
CAS (Compare and Swap)
(8 bytes)
Thursday, April 23, 2009
55. Response Header
MAGIC Opcode Key Length
(1 byte) (1 byte) (2 bytes)
Extra Length Data Type Status
(1 byte) (1 byte) (2 bytes)
Total Body Length
(4 bytes)
Opaque
(4 bytes)
CAS (Compare and Swap)
(8 bytes)
Thursday, April 23, 2009
57. Protocol
• Telnet’able (see doc/protocol.txt)
• Store commands are binary (no escaping)
• WireShark support for Binary Protocol
Thursday, April 23, 2009
68. Connections
• Don't fear setting -c (max conns) too high!
• Watch 'listen_disabled_num' (1.2.8+)
• Audit any firewalls on the system (can run
out of firewall states!)
• Watch for connection overload. Probably an
application bug
Thursday, April 23, 2009
69. Memory
• Don't set -m (max bytes) too high
• -m only controls memory for stored values
• Other memory is used for connection
handling, LRU tracking, the hash table, etc
• Especially if running under 32bit (-m 2048
bad!)
• Careful of -k. Could force server to swap
• Monitor server well! If you are swapping,
memcached will be slow
Thursday, April 23, 2009
70. Things to remember!
• Memcached is great for scaling, but abuse
will slow down your page rendering time
• Fetching 10,000 separate keys to build a
page will not be fast
• Use persistent connections if possible
• Don't fetch more than you need. Takes time
to serialize/deserialize 100k+ of data
Thursday, April 23, 2009
71. Shrink data!
• Especially if you use multiget, try to use
shorter keys
• “fooooooooooooooooooooooo-$MD5”
will take more data packets than “foo1”
• Try using smaller keys anyway, saves
memory if your data is small too
Thursday, April 23, 2009
72. Future Solutions!
• Evaluate the binary protocol in 1.3
• New tricks will be possible, such as
smashing many commands into fewer
packets
• 'noreply' commands. Don't wait for the
response if you don't have to
• ('noreply' is in 1.2, but not recommended
for use)
Thursday, April 23, 2009
73. 1.2.8
• Bugfixes
• New stats
• listen_disabled_num
• cmd_flush (are you flushing your cache
and not realizing it? :) )
• evicted_time under 'stats items'
• Only bugfixes and minor features planned
for 1.2 series
Thursday, April 23, 2009
74. 1.3.* Beta
• Scalability improvements
• New stats (per-slab get/miss/etc counters)
• Binary protocol
• Memory optimizations (disable CAS
support)
Thursday, April 23, 2009
75. ASCII Protocol vs Binary Protocol (memcached-1.3.1 Development Branch)
task completion time (secs)
ASCII Protocol
Binary Protocol
concurrent connections
Binary Protocol
Thursday, April 23, 2009
76. Future
• 1.4.0 stable
• Multiple Engine Support
• Durable
• Highly Threaded
• New Slab Features
Thursday, April 23, 2009
77. More Resources...
• http://danga.com/memcached/
• #memcached on Freenode
• http://tangent.org/552/libmemcached.html
• http://groups.google.com/groups/
memcached
• http://dev.mysql.com/doc/refman/5.1/en/ha-
memcached.html
Thursday, April 23, 2009
