2. OUTLINE
• Introduction and Overview
• CouchDB Basics
• Special Topics in Relaxation: Scaling CouchDB
• Use Cases In the Wild
• Takeaways
Windy City DB 2 June 26, 2010
3. HI
• Alan Hoffman
• @_hoffman
• alan@cloudant.com
• Experimental particle physicist
• Background: machine learning, big
data analysis, distributed systems
• Co-founder of Cloudant (Hosted
Couch)
• Not a committer, but...
Windy City DB 3 June 26, 2010
4. COUCH: THE BIG PICTURE
• Apache project
• Schema-free document database management system
• Robust, concurrent, fault-tolerant
• RESTful JSON API
• Custom persistent views using MapReduce
• Bi-directional incremental replication
• Futon web admin console
Windy City DB 4 June 26, 2010
5. WHO CARES?
The internet happened, and we ignored it.
In retrospect, that was a mistake.
-Bill Warner (Avid, Wildfire, Techstars)
Summer, 2008
Disruptive technologies enable new business
Windy City DB 5 June 26, 2010
6. DOCUMENTS
Primary Key
MVCC
&
Insta-cache
Nested Structures
• Reserved fields are prefixed with an
underscore
• MVCC _rev deterministically generated
from doc content
Binary Attachments • Binary attachments
Windy City DB 6 June 26, 2010
7. RESTFUL API
• Create
PUT /mydb/mydocid
• Retrieve
GET /mydb/mydocid “Built of the Web
Completely embraces... HTTP”
• Update
PUT /mydb/mydocid
-Jacob Kaplan-Moss
• Delete October 2007
DELETE /mydb/mydocid
GET /mydb/_all_docs?include_docs=true
http://wiki.apache.org/couchdb/Reference
Windy City DB 7 June 26, 2010
8. VIEWS
value
ap du ce
m re
key
• Docs can be indexed by any attribute using views. Custom, persistent
representations of the data.
• Each view must have a map function and may also have a reduce function
• View indices are stored in B-trees for efficient lookup by map key
• Stored in special documents called _design documents
Windy City DB 8 June 26, 2010
9. INCREMENTAL
• Computing a view can be expensive, so CouchDB saves
the result in a B-tree and keeps it up-to-date
• Only new docs or changed docs get ‘re-indexed’
• Leaf nodes store map results, inner nodes store reductions
of children
http://horicky.blogspot.com/2008/10/couchdb-implementation.html
Windy City DB 9 June 26, 2010
10. ROBUST
• Never overwrite previously committed
data
• Append only b+trees, ‘copy-on-write’
• Server crash, power failure? just restart
CouchDB -- there is no “repair”
• Take snapshots with “cp”
J.C. Anderson
• ACID at the single document level
Windy City DB 10 June 26, 2010
11. REPLICATION
source target
progress
The beauty of MVCC one click
CouchDB => “Cloud ready”
Windy City DB 11 June 26, 2010
12. REPLICATION
• Peer-based, bi-directional replication using normal HTTP
• Mediated by a replicator process which can
live on the source, target, or somewhere else
entirely
• Replicate a subset of documents in a DB
meeting criteria defined in a custom filter
function
• Applications (_design documents) replicate
along with the data
• Ideal for offline applications: “ground
computing”
Windy City DB 12 June 26, 2010
13. FILTERED REPLICATION
Write the filter function
Embed it in a design
doc
Specify in the
replication request
Windy City DB 13 June 26, 2010
14. MULTI-COUCH SETUPS
Master-Slave Robust Multi-Master
Master-Master
Windy City DB 14 June 26, 2010
15. CONFLICTS
PUT /a/foo PUT /b/foo
replicate
Conflict
• Replication can introduce conflicts in a multi-master setup
• CouchDB deterministically chooses a winner but the loser is saved with
the document as a conflicting rev
• Conflicting revs are replicated; both source and target will agree on
winning and losing revs
• Compacting the DB removes all losing revs
Windy City DB 15 June 26, 2010
16. BUILDING A BIG COUCH
D oesn’t
Why CouchDB ^ Doesn’t Scale
Windy City DB 16 June 26, 2010
17. WHAT WE TALK ABOUT WHEN WE
TALK ABOUT SCALING
• Horizontal scaling: more servers creates more capacity
• Transparent to the application: adding more capacity should not affect
the business logic of the application.
• No single point of failure. Physics Joke!
Pseudo Scalars
http://adam.heroku.com/past/2009/7/6/sql_databases_dont_scale/
Windy City DB 17 June 26, 2010
18. COUCHDB LOUNGE
• Proxy-based partitioning and clustering PUT/GET
application
• Designed originally for use at Meebo Dumbproxy
(nginx)
• Uses consistent hashing to partition docs
across nodes
• Dumbproxy - nginx module that handles
simple GETs and PUTs
• Smartproxy - A twisted/python daemon
that handles view requests Smartproxy
• Want to know more? R. Leeds (tilgovi)
http://tilgovi.github.com/couchdb-lounge/
GET /_deisgn/...
Windy City DB 18 June 26, 2010
19. OPEN CLOUDANT
• Clustering in a ring (a la Dynamo)
PUT http://alan.cloudant.com/dbname/blah?w=2 • Any node can handle a request
• O(1) lookup
N=3
Load Balancer
• Quorum system (N, R, W)
W=2
R=2
• Views distributed like documents
24
Node 1
No
• Distributed erlang
de A B C D de
No B 2
Y
Z
A
C
D
• Masterless
X hash(blah) = E E
C N
od ✓ Horiziontally Scalable
e
D 3
E ✓ No SPOF
F
✓ Transparent to the
D
application
No
E
de
4
F
Coming soon to a
G
github near you!
Windy City DB 19 June 26, 2010
20. IN THE WILD
• 15+ million deployments
• Activecommercial support
• 3 books
• 1.0 imminent
• Vibrant, open community
Windy City DB 20 June 26, 2010
21. CASE #1: REALTIME ANALYTICS
• Analytics on high-rate advertising data
• ETL analysis workflow too slow for their customers (24 hr cycle)
• Needed a realtime solution
• Complicated SQL stored procedures for social graph analysis
required 40+ postgres tables
• Replaced it all with a single CouchDB document type and two
views:
• group level collation to bin data at multiple granularities => customers get
updated results in seconds, not hours
• single view (30 lines of JS) for graph analysis.
Windy City DB 21 June 26, 2010
22. MONEY QUOTE
Migrating to CouchDB really opened a lot of doors
for us product-wise. The time delay between data
arriving in our systems and becoming available to our
customers went from 24 hours to less than 30 min - on
similar hardware - even while we greatly increased the
level of granularity that our processing provided
Windy City DB 22 June 26, 2010
23. CASE #2: EASYBIB
• Online bibliography service, ~10 years old, initially built on MySQL
(and Coldfusion)
• Had suffered through many migrations
• Choice: massive sharding and replication of MySQL v. “another
option”
• Why Couch:
• Schema Free (replacing 40 - 50 tables with 3 DBs)
• Easily scalable
• Strong community support
“In your best Borat voice: ‘Great Success!’”
Windy City DB 23 June 26, 2010
24. CASE #3: MEEBO
• “All
your friends and networks, from wherever you are.”
• Why Couch?
• No Schema (and ergo, no schema migrations)
• Replication
• Could deal with queries that would break on a sharded RDBMS
• REST interface -- easy to re-use existing tools and libraries
• Easy to write a proxy layer that keeps sharding out of the app
logic
• Wishes? Speed, API stability, native clustering
Windy City DB 24 June 26, 2010
25. PARAPHRASING THE MASSES
• Why CouchDB?
• Simple, robust, concurrent, fun
• successful in production
• Why Not Couch?
• Missing Features
• ad hoc queries
• authz/authn
• doesn’t scale
• Too New -- api still changing, still alpha
• “Too Slow”
Windy City DB 25 June 26, 2010
26. PARAPHRASING THE MASSES
• Why CouchDB?
• Simple, robust, concurrent, fun, scalable, powerful
• successful in production, active community, industry adoption
• Why Not Couch?
• Missing Features
• ad hoc queries
• authz/authn
• doesn’t scale
• Too New -- api still changing, still alpha
• “Too Slow”
Windy City DB 25 June 26, 2010
27. PARAPHRASING THE MASSES
• Why CouchDB?
• Simple, robust, concurrent, fun, scalable, powerful
• successful in production, active community, industry adoption
• Why Not Couch?
• Missing Features
• ad hoc queries True, by design
• authz/authn Included in 0.11
• doesn’t scale Lounge, Pillow, Open Cloudant, etc
• Too New -- api still changing, still alpha
• “Too Slow” 0.11 Feature freeze and 1.0 imminent
Perhaps, but...
Windy City DB 25 June 26, 2010
28. DESERVING OF MORE TIME
• CouchApp: HTML+JS framework for building
lightweight, portable apps and serving them directly
from CouchDB
• http://github.com/couchapp/couchapp/
• External indexers like CouchDB-Lucene
• http://github.com/rnewson/couchdb-lucene
• The plethora of client libraries and tools...
Windy City DB 26 June 26, 2010
29. TRY IT OUT
Hosted Free:
Cloudant.com
Easy Offline:
CouchDBX
Windy City DB 27 June 26, 2010
30. THANK YOU
• Books
• CouchDB: The Definitive Guide. J. Chris Anderson, Jan
Lehnardt, Noah Slater
• Beginning CouchDB. Joe Lennon
• Web
• http://wiki.apache.org/couchdb/
• http://planet.couchdb.org/
• IRC
• Freenode #couchdb
• Freenode #cloudant
Windy City DB 28 June 26, 2010
32. AUTHZ/AUTHN
• Remember, Couch acts like a web service
• Authentication:
• 0.11+ ships with support for OAuth, cookie, and basic
• Handlers specified in a config file
• Users defined in authentication database (“_users” by default)
• Authorization
• 3 levels: DB reader, DB admin, Server Admin
• Per DB roles defined in security document
Windy City DB 30 June 26, 2010
33. EXAMPLES
User Document
Security Document
Caution!
Do not leave arrays blank
http://wiki.apache.org/couchdb/
Security_Features_Overview
Windy City DB 31 June 26, 2010
34. DRAWBACKS
• “Futon -- difficult to use for installations that have a lot
of DBs (1000+)”
• “Tools for managing design docs are deficient”
• “Client
libraries too focused on Couch as the ‘M’ in
MVC apps.”
• “Couch 1.0 is a moving target”
Windy City DB 32 June 26, 2010
