2. Who Am I?
✤ Robert Treat
✤ OmniTI
✤ Design, Development,
Database, Ops
Monday, October 25, 2010
3. Who Am I?
✤ Robert Treat
✤ OmniTI
✤ Design, Development,
DATABASE, Ops
Monday, October 25, 2010
4. Who Am I?
✤ Robert Treat
✤ OmniTI
✤ Design, Development,
DATABASE, Ops
✤ Etsy, Allisports,
National Geographic,
Gilt, etc...
Monday, October 25, 2010
5. Who Am I?
✤ Robert Treat
✤ Postgres
✤ Web, Advocacy,
phpPgAdmin
✤ Major Contributor
Monday, October 25, 2010
6. Who Am I?
✤ Postgres 6.5 -> 9.1alpha1
✤ Terabytes of data
✤ Millions of transactions per day
✤ OLTP, ODS, DW
✤ Perl, PHP, Java, Ruby, C#
Monday, October 25, 2010
7. Who Am I?
OBSERVATION == LEARNING
(hopefully)
Monday, October 25, 2010
8. Scalability
It is the ability of a computer
application or product (hardware
or software) to continue to
function well when it (or its
context) is changed in size or
volume in order to meet a user
need.
Monday, October 25, 2010
9. Scalability
Given ever increasing load
Monday, October 25, 2010
10. Scalability
Given ever increasing load
NEVER GO DOWN
ALWAYS PERFORM WELL
Monday, October 25, 2010
11. Scalability
Given ever increasing load
NEVER GO DOWN
ALWAYS PERFORM WELL
impossible goal, but we’ll try
Monday, October 25, 2010
12. Scalability
Given ever increasing load
NEVER GO DOWN
ALWAYS PERFORM WELL
NOTE! data loss is not a goal, but ideally we won’t lose it :-)
Monday, October 25, 2010
14. ✤ Get over schema purity
✤ add column default not null
Monday, October 25, 2010
15. ✤ Get over schema purity
✤ add column default not null
Good performance comes from good schema
design, HOWEVER, perfect relational
modeling is NOT THE GOAL
Monday, October 25, 2010
16. ✤ Devs must own schema and queries
✤ they design, you refine
Monday, October 25, 2010
17. ✤ Devs must own schema and queries
✤ they design, you refine
Performance and scalability cannot be
managed solely within the database; both
require application level knowledge. To
achieve this, application developers need to
have visibility of the resources they work on
Monday, October 25, 2010
25. Gain Visibility
ALL alerts, graphs, query reports, etc...
MUST be available to EVERYONE on
the team AT ALL TIMES
Monday, October 25, 2010
26. Hands on
You can’t succeed without first putting
the right culture in place.
Once you are on the right path, make
sure you have the right technology
Monday, October 25, 2010
27. Postgres Versions
✤ MINIMUM: 8.3
✤ removes xid for read only queries, significant reduction in vacuum
activity
Monday, October 25, 2010
28. Postgres Versions
✤ MINIMUM: 8.3
✤ removes xid for read only queries, significant reduction in vacuum
activity
seriously!
Monday, October 25, 2010
29. Postgres Versions
✤ MINIMUM: 8.3
✤ removes xid for read only queries, significant reduction in vacuum
activity
✤ BETTER: 8.4
✤ revised free space map management leads to more efficient
vacuuming
Monday, October 25, 2010
30. Postgres Versions
✤ MINIMUM: 8.3
✤ removes xid for read only queries, significant reduction in vacuum
activity
✤ BETTER: 8.4
✤ revised free space map management leads to more efficient
vacuuming
✤ WHY NOT? 9.0
✤ Hot standby / streaming replication couldn’t hurt
Monday, October 25, 2010
31. Speaking of replication
✤ Common practice for scaling websites
✤ Good for READ based loads
✤ We have used many:
✤ slony, rubyrep, bucardo, 9.0 built-in, mammoth, wrote-our-own
Monday, October 25, 2010
33. Speaking of replication
✤ No favorite system for this, evaluate based on:
✤ avoid solutions that duplicate writes at sql level (imho)
✤ how comfortable am I debugging the system?
✤ do you need automated schema changes?
✤ how much redundancy / complexity do you need?
✤ how does the system handle node failure for N nodes?
Monday, October 25, 2010
34. So what would you use? (tm)
✤ 2 Nodes, master + standby: Postgres 9.0
✤ Master + multiple slaves: Slony
✤ Master-Master: Bucardo
All choices subject to change!!
Monday, October 25, 2010
35. A word about “Sharding”
✤ Distributed computing is hard(er)
✤ we think of things in a singular global state
✤ the more we can work in that model, the better
✤ RDBM offer poor solutions for multiple masters
✤ you must manage that complexity on your own
Monday, October 25, 2010
36. A word about “Sharding”
✤ Splitting systems by service:
✤ separate db for login, forums, sales, etc...
✤ allows for growth
✤ provides simple interface
Monday, October 25, 2010
37. Pooling
✤ Postgres connections are expensive!
✤ fork new process per connection
✤ keep 1 process open per connection
✤ 1000+ processes you will notice trouble
Monday, October 25, 2010
38. Pooling
✤ Postgres connections are expensive!
✤ fork new process per connection
✤ keep 1 process open per connection
✤ 1000+ processes you will notice trouble
✤ POOLING
✤ JDBC, mod-perl
✤ pgbouncer ftw!
Monday, October 25, 2010
39. Summary
✤ Schema / Queries should be shared between dev, dba teams!
✤ Monitoring + Visibility!
✤ >= 8.3 Required!
✤ Replication, jump in it!
✤ Use connection pooling!
Monday, October 25, 2010
