1. Scaling with SkyTools
& More
Scaling-Out Postgres with Skype’s Open-Source Toolset
Gavin M. Roy
September 14th, 2011

2. About Me
• PostgreSQL ~ 6.5
• CTO @myYearbook.com
• Scaled initial infrastructure
• Not as involved day-to-day database
operational and development
• Twitter: @Crad

3. Scaling?

4. Concurrency
Requests per Second
6am 8am 10am 12pm 2pm 4pm 6pm 8pm 10pm 12am 2am 4am 6am
Hourly breakdown

5. Increasing Size-On-Disk
Size in GB
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

6. Scaling and
PostgreSQL Behavior

7. Size on Disk

8. Tuples, Indexes,
Overhead

9. Table Size
+
Size of all combined Indexes
Relations Indexes

10. Constraints
• Available Memory
• Disk Speed
• IO Bus Speed

11. Keep it in memory.

12. Get Fast Disks & I/O.

13. Process Forking
+
Locks

14. Client Connections

15. One Connection per
Concurrent Request

16. Apache+PHP
One connection per backend for each pg_connect

17. Python
One connection per connection*

18. ODBC
One connection to Postgres per ODBC connection

19. Master Process
Lock Stats
Collector
Contention? Autovacuum
Each backend for a connected Wall Writer
client has to check for locks
Wall Writer
Connection
Client Connection
Backend

20. Master Process
New Client Stats
Collector
Connection?
Autovacuum
Wall Writer
Access Share
Access Exclusive Wall Writer
Exclusive
Share Connection
Client Connection
Backend
Share Row Exclusive
Share Update Connection
Client Connection
Row Share
Backend
Row Exclusive

21. Master Process
Stats
Collector
Too many Autovacuum
connections? Wall Writer
Wall Writer
Slow performance Connection
Client Connection
Backend
Connection
Client Connection
Backend
...
Connection
Client Connection
Backend

22. 250 Apache Backends
x
1 Connection per Backend
x
250 Servers
=
62,500 Connections

23. Solvable Problems!

24. The Trailblazers

25. Solving Concurrency

26. pgBouncer

27. Session Pooling

28. Transactional Pooling

29. Statement Pooling

30. Connection Pooling
Clients Clients Clients
Hundreds Hundreds Hundreds
pgBouncer
Tens Tens Tens
Postgres Postgres Postgres
Server #1 Server #2 Server #3

31. Add Local Pooling
Clients Clients Clients
Hundreds Hundreds Hundreds
Local pgBouncer Local pgBouncer Local pgBouncer
Tens Tens Tens
pgBouncer
Tens Tens Tens
Postgres Postgres Postgres
Server #1 Server #2 Server #3

32. Easy to run
Usage: pgbouncer [OPTION]... config.ini
-d, --daemon Run in background (as a daemon)
-R, --restart Do a online restart
-q, --quiet Run quietly
-v, --verbose Increase verbosity
-u, --user=<username> Assume identity of <username>
-V, --version Show version
-h, --help Show this help screen and exit

33. userlist.txt
“username” “password”
“foo” “bar”

34. pgbouncer.ini

35. Specifying Connections
[databases]
; foodb over unix socket
foodb =
; redirect bardb to bazdb on localhost
bardb = host=localhost dbname=bazdb
; access to dest database will go with single user
forcedb = host=127.0.0.1 port=300 user=baz password=foo
client_encoding=UNICODE datestyle=ISO connect_query='SELECT
1'

36. Base Daemon Config
[pgbouncer]
logfile = pgbouncer.log
pidfile = pgbouncer.pid
; ip address or * which means all ip-s
listen_addr = 127.0.0.1
listen_port = 6432
; unix socket is also used for -R.
;unix_socket_dir = /tmp

37. Authentication
; any, trust, plain, crypt, md5
auth_type = trust
#auth_file = 8.0/main/global/pg_auth
auth_file = etc/userlist.txt
admin_users = user2, someadmin, otheradmin
stats_users = stats, root

38. Stats Users?
SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|VERSION
SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM
pgbouncer=# SHOW CLIENTS;
type | user | database | state | addr | port | local_addr | local_port | connect_time
------+-------+-----------+--------+-----------+-------+------------+------------+---------------------
C | stats | pgbouncer | active | 127.0.0.1 | 47229 | 127.0.0.1 | 6000 | 2011-09-13 17:55:46
* Truncated columns for display purposes

39. psql 9.0+ Problem?
psql -U stats -p 6432 pgbouncer
psql: ERROR: Unknown startup parameter
Add to pgbouncer.ini:
ignore_startup_parameters = application_name

40. Pooling Behavior
pool_mode = statement
server_check_query = select 1
server_check_delay = 10
max_client_conn = 1000
default_pool_size = 20
server_connect_timeout = 15
server_lifetime = 1200
server_idle_timeout = 60

41. Skytools

42. Scale-Out Reads
Clients Clients Clients Clients
pgBouncer
Load Balancer
Read Only Copy Read Only Copy Read Only Copy Read Only Copy
Canonical
Database

43. PGQ

44. The Ticker

45. ticker.ini
[pgqadm]
job_name = pgopen_ticker
db = dbname=pgopen
# how often to run maintenance [seconds]
maint_delay = 600
# how often to check for activity [seconds]
loop_delay = 0.1
logfile = ~/Source/pgopen_skytools/%(job_name)s.log
pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid

46. Getting PGQ Running
Setup our ticker:
pgqadm.py ticker.ini install
Run the ticker daemon:
pgqadm.py ticker.ini ticker -d

47. Londiste

48. replication.ini
[londiste]
job_name = pgopen_to_destination
provider_db = dbname=pgopen
subscriber_db = dbname=destination
# it will be used as sql ident so no dots/spaces
pgq_queue_name = pgopen
logfile = ~/Source/pgopen_skytools/%(job_name)s.log
pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid

49. Install Londiste
londiste.py replication.ini provider install
londiste.py replication.ini subscriber install

50. Start Replication Daemon
londiste.py replication.ini replay -d

51. DDL?

52. Add the Provider
Tables and Sequences
londiste.py replication.ini provider add public.auth_user

53. Add the Subscriber
Tables and Sequences
londiste.py replication.ini subscriber add public.auth_user

54. Great Success!

55. PL/Proxy

56. Scale-Out Reads & Writes
plProxy Server
A-F Server G-L Server M-R Server S-Z Server

57. How does it work?

58. Simple Remote
Connection
CREATE FUNCTION get_user_email(username text)
RETURNS SETOF text AS $$
CONNECT 'dbname=remotedb';
SELECT email FROM users WHERE username = $1;
$$ LANGUAGE plproxy;

59. Sharded Request
CREATE FUNCTION get_user_email(username text)
RETURNS SETOF text AS $$
CLUSTER “usercluster”;
RUN ON hashtext(username);
$$ LANGUAGE plproxy;

60. Sharding Setup
• Need 3 Functions:
• plproxy.get_cluster_partitions(cluster_name
text)
• plproxy.get_cluster_version(cluster_name text)
• plproxy.get_cluster_config(in cluster_name text,
out key text,
out val text)

61. get_cluster_partitions
CREATE OR REPLACE FUNCTION
plproxy.get_cluster_partitions(cluster_name text)
RETURNS SETOF text AS $$
BEGIN
IF cluster_name = 'usercluster' THEN
RETURN NEXT 'dbname=part00 host=127.0.0.1';
RETURN NEXT 'dbname=part01 host=127.0.0.1';
RETURN;
END IF;
RAISE EXCEPTION 'Unknown cluster';
END;
$$ LANGUAGE plpgsql;

62. get_cluster_version
CREATE OR REPLACE FUNCTION
plproxy.get_cluster_version(cluster_name text)
RETURNS int4 AS $$
BEGIN
IF cluster_name = 'usercluster' THEN
RETURN 1;
END IF;
RAISE EXCEPTION 'Unknown cluster';
END;
$$ LANGUAGE plpgsql;

63. get_cluster_config
CREATE OR REPLACE FUNCTION plproxy.get_cluster_config(
in cluster_name text,
out key text,
out val text)
RETURNS SETOF record AS $$
BEGIN
-- lets use same config for all clusters
key := 'connection_lifetime';
val := 30*60; -- 30m
RETURN NEXT;
RETURN;
END;
$$ LANGUAGE plpgsql;

64. get_cluster_config
values
• connection_lifetime
• query_timeout
• disable_binary
• keepalive_idle
• keepalive_interval
• keepalive_count

65. SQL/MED

66. SQL/Med Cluster
Definition
CREATE SERVER a_cluster FOREIGN DATA WRAPPER plproxy
OPTIONS (
connection_lifetime '1800',
disable_binary '1',
p0 'dbname=part00 hostname=127.0.0.1',
p1 'dbname=part01 hostname=127.0.0.1',
p2 'dbname=part02 hostname=127.0.0.1',
p3 'dbname=part03 hostname=127.0.0.1'
);

67. PLProxy + SQL/Med
Behavior
• PL/Proxy will prefer SQL/Med cluster
definitions over the plproxy.get_* functions
• PL/Proxy will fallback to plproxy.get_*
functions if there are no SQL/Med clusters

68. SQL/MED User Mapping
CREATE USER MAPPING FOR bob
SERVER a_cluster
OPTIONS (user 'bob', password 'secret');
CREATE USER MAPPING FOR public
SERVER a_cluster
OPTIONS (user 'plproxy', password 'foo');

69. plproxyrc
• plpgsql based api for table based
management of PL/Proxy
• Used to manage complicated PL/Proxy
infrastructure @myYearbook
• BSD Licensed
https://github.com/myYearbook/plproxyrc

70. “Server-to-Server”
Postgres Postgres Postgres
Server #1 Server #2 Server #3
pgBouncer

72. Complex PL/Proxy and pgBouncer
Environment
Clients Local pgBouncer Load Balancer
Clients Local pgBouncer
Clients Local pgBouncer pgBouncer pgBouncer
Postgres plProxy Server plProxy Server
Server #1
pgBouncer
Postgres
Load Balancer
Server #3
pgBouncer
Postgres
Server #3

73. Other Tools and
Methods?

74. Questions?