Caching can bring speed to the slow systems in your enterprise. In this session, we'll explore how to avoid expensive computations and reduce database (or other systems) load with caching. Considerations include when to cache (cache policy), how to cache, and when/how to evict. This session will also explore and discuss problems and gotchas like cache sizing, eviction mistakes, and where your cache should live. Finally, we'll look at how caching is implemented in the "memory-first" Couchbase architecture.

1. C.R.E.A.M. get the memory (dollar dollar bill ya'll)
August 2022
Matthew D. Groves | Product Marketing Manager
aka “Passionate Bravo”
Cache Rules Everything
Around Me

2. 2
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315

3. 3
The Need for Speed
https://www.slideshare.net/devonauerswald/walmart-pagespeedslide

4. 4
Cache Rules Everything Around Me
Dollar dollar bill ya'll
Android apps iPhone apps
CPU cache
CDNs
Web browser
DNS

5. 01/
02/
03/
04/
05/
06/
What is caching?
Use Cases
Reading from Cache
Writing into Cache
Cache Invalidation
Gotchas
Agenda

6. 6
ho am I?
• Matthew D. Groves
• Microsoft MVP, author at Pluralsight, Manning, Apress
• Product Marketing Manager at Couchbase
• https://twitter.com/mgroves
• https://www.linkedin.com/in/mgroves/
• C# Advent: https://csadvent.christmas/

7. 7
1 hat is Caching?

8. 8
Caching
Collection of duplicated data to serve future requests faster.
• First formally described by Maurice Wilkes in 1965
• "Slave Memories and Dynamic Storage Allocation"
https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf
(https://bit.ly/WilkesPaper)
What is Caching?
Key Data
1 Proteck Ya Neck
2 C.R.E.A.M.
3 Brooklyn Zoo
4 Criminology
… …
Key Data
2 C.R.E.A.M.
4 Criminology
Records Cache

9. 9
Powerpoint as a Cache
What is Caching?
Key Value
What year was Wu-Tang
Clan Formed?
1992

10. 10
2 Caching Use Cases

11. 11
Caching Use Cases
What is Caching used for?

12. 12
Caching Use Case 1: Faster Database Access
• Databases can rely heavily on disk
• Reading the same data over again
could mean accessing the disk over
again
• Store frequently accessed data in a
cache, to reduce pressure on
database/disk

13. 14
Caching Use Case 1: Faster Database Access
DATABASE
APP
LAN
CACHE
CACHE
CACHE

14. 15
Caching Use Case 2: Faster Web Browsing
• Any given URL could result in
hundreds of files that are loaded
over HTTP
• A single JS file may be loaded by
every page on a site.
• Cache static assets (JS, images,
sound, icons, etc) on a user's laptop
to avoid loading them over the
network every time

15. 16
Caching Use Case 2: Faster Web Browsing
CACHE

16. 17
Caching Use Case 3: Efficient API Use
• API usage may involve an internet
call (with latency)
• API calls may cost per use, or may
be limited
• Storing results in cache may help
reduce latency AND reduce costs

17. 19
Caching Use Case 3: Efficient API Use
API APP
LAN/WAN
CACHE
CACHE
CACHE

18. 20
Caching Use Case 4: Maintaining Availability
• Retrieving data from other
processes
• Microservices
• Latency is secondary concern

19. 21
Caching Use Case 4: Maintaining Availability
CACHE

20. 22
Cache Rules Everything Around Me
Dollar dollar bill ya'll
Disconnected Availability
Slow
Repetition
Constrained
Costs

21. 23
3 Reading From Cache

22. 24
Using a Cache 101
Your Code
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
Cache
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
D Criminology
E Triumph
F Uzi
G Method Man
H Wu-Tang Clan Ain't . . .
. . . . . .
Database
1. Lookup "A"
2. Lookup "B"
3. Lookup "C"
HIT!
HIT!
MISS!
4. Lookup "C" again?
WALK OF SHAME

23. 25
4 Writing Into Cache

24. 26
What about writes?
• Data is duplicated, stored in cache and record
• The system is responsible for keeping the cache updated
• Commons methods:
• write-through
• write-around
• write-back
Stuff needs to get in the cache somehow

25. 27
write-through
1. System gets a new/updated piece of data
2. Write to cache and record "at the same time"
1. Write to record
2. If that didn't succeed, the write failed.
3. Write to cache
4. If that didn't succeed, the write failed.
5. If the system reaches this point, the write succeeded!
Why or why not use this?

26. 28
write-around
1. System gets a new/updated piece of data
2. Write the data to the record ONLY
3. Invalidate the record in the cache
Why or why not use this?

27. 29
write-back
1. System gets a new/updated piece of data
2. Write the data to the cache.
3. Update the record asynchronously (don't wait on it to finish)
4. If the cache write succeeded, then consider the write succeeded.
Why or why not use this?

28. 30
Cache Writing Methods
Method Write performance Write integrity Best for…?
write-through
➖ ➕ High % reads
write-around
➖ ➕ High % read later
write-back
➕ ❔ Mixed reads / writes

29. 31
Couchbase
A database with a built-in managed cache
https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html

30. 32
Couchbase and Writing to the Cache
await collection.InsertAsync("A", new { title = "Protect Ya Neck" });
await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options =>
{
options.Durability(DurabilityLevel.None);
// OR: options.Durability(DurabilityLevel.Majority);
// OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive);
// OR: options.Durability(DurabilityLevel.PersistToMajority);
});

31. 33
Durability (with 3 replicas)
None
Memory
Disk
Node 1 Node 2 Node 3
✔
⚠
⚠
⚠
⚠
⚠
Node 4
⚠
⚠

32. 34
Durability (with 3 replicas)
Majority
Memory
Disk
Node 1 Node 2 Node 3
✔
⚠
⚠
⚠
✔
⚠
Node 4
✔
⚠

33. 35
Durability (with 3 replicas)
MajorityAndPersistToActive
Memory
Disk
Node 1 Node 2 Node 3
✔
✔
⚠
⚠
✔
⚠
Node 4
✔
⚠

34. 36
Durability (with 3 replicas)
PersistToMajority
Memory
Disk
Node 1 Node 2 Node 3
✔
✔
⚠
⚠
✔
✔
Node 4
✔
✔

35. 37
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315

36. 38
5 Cache Invalidation

37. 39
Powerpoint as a Cache
Key Value
What year was Wu-Tang Clan
Formed?
1992
What was Wu-Tang's first album
called?
Enter the Wu-Tang
What year was Maurice Wilkes
born?
1913
What year did Maurice Wilkes
die?
2010
What is the most recent Wu-
Tang studio album?
Once Upon a Time in Shaolin

38. 40
Cache Invalidation is Hard
There are only two hard things in Computer Science: cache
invalidation and naming things.
-- Phil Karlton
How to decide what to invalidate (what data to "evict"):
cache policy https://www.nndb.com/people/400/000031307/

39. 41
Cache Policy 1: LRU (Least Recently Used)
Key How new?
A 0
B 1 3
C 2
D 4
Cache max size: 3
Order of reads: A, B, C, B, D
X
---------------------------------------------------------------------------------

40. 42
Cache Policy 2: NRU (Not Recently Used)
Lower scores evicted (at random)
Key Referenced? Modified? Score
A 0 0 0
B 0 0 0
C 0 1 1
D 1 0 2
E 1 1 3

41. 43
Cache Policy 3: None
Don't evict anything

42. 44
More Cache Policies
• RR – random replacement
• FIFO and LIFO – treat a cache like a queue
• MRU – MOST recently used
• LFU – least frequently used
• Belady's algorithm – not possible to implement
• Machine learning
• https://en.wikipedia.org/wiki/Cache_replacement_policies

43. 45
6 Gotchas

44. 46
Gotcha 1: Cache Size
• Problem: too small or too big cache
• Symptom: High turnover and churn (many evictions) or long lifetimes (few
evictions)
• Solutions:
• Monitoring
• Size the cache
• "Value" vs "Full" eviction

45. 47
Gotcha 2: "Close" Caching
• Problem: running the cache on the same machine as the application
• Symptom: redundancy, hot spots, inconsistency, availability
• Solutions:
• Use a distributed cache
• Running on its own machine(s) / cluster
• Couchbase, of course!

46. 48
Gotcha 3: Bad Eviction Policy
• Problem: using the wrong eviction policy
• Symptom: lots of overhead, cache checking
• Solutions:
• Don't Write an Eviction Policy (or better yet don't write a Cache)
• Benchmarking: "hit ratio" and "latency"
• "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio")
• "Latency" – how long it takes from request to response (e.g. 10μs latency)

47. THANK YOU

48. 50
References
• https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/
• https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback-
caching-with-python
• https://en.wikipedia.org/wiki/Maurice_Wilkes
• https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf
• https://stackoverflow.com/a/4087315/40015
• https://www.techtarget.com/searchstorage/definition/cache
• https://en.wikipedia.org/wiki/Cache_replacement_policies
• https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability
• https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure-
considerations.html#durable-writes