Slides from a presentation and workshop on using Amazon's EC2 with PoolParty. Given in Pasadena, CA in Feb 09.

1. Getting Started with
PoolParty & EC2

2. PoolParty

3. PoolParty
• PoolParty makes it easy and simple to
configure any cloud of computers

4. PoolParty
• PoolParty makes it easy and simple to
configure any cloud of computers
• Written in Ruby and Erlang

5. PoolParty

6. PoolParty
• Not Ruby or Rails specific

7. PoolParty
• Not Ruby or Rails specific
• Not even tied to Amazon EC2

8. sample config

9. Amazon

11. EC2

12. EC2 S3

13. EC2 S3
EBS

14. EC2 S3
EBS AMI

15. EC2 S3
Elastic IP
EBS AMI

16. EC2 S3
Elastic IP
Keypairs
EBS AMI

17. EC2 S3
Elastic IP
Keypairs
Security Groups
EBS AMI

18. EC2 S3
Elastic IP
Keypairs
CloudFront
Security Groups
EBS AMI

19. EC2 S3
??? AMI
Elastic IP
Keypairs
CloudFront
Security Groups
EBS

20. Amazon: EC2

21. Amazon: EC2
• Platform to launch machine instances

22. Amazon: EC2
• Platform to launch machine instances
• ec2-* (git-style) commands such as:

23. Amazon: EC2
• Platform to launch machine instances
• ec2-* (git-style) commands such as:
• ec2-run-instances

24. Amazon: EC2
• Platform to launch machine instances
• ec2-* (git-style) commands such as:
• ec2-run-instances
• ec2-terminate-instances

25. Amazon: EC2
• Platform to launch machine instances
• ec2-* (git-style) commands such as:
• ec2-run-instances
• ec2-terminate-instances
• ec2-allocate-address

26. Amazon: EC2
• Platform to launch machine instances
• ec2-* (git-style) commands such as:
• ec2-run-instances
• ec2-terminate-instances
• ec2-allocate-address
• etc.

27. Amazon: EC2 (AMIs)

28. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image

29. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image
• Conceptually similar to a Xen or VMware
image

30. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image
• Conceptually similar to a Xen or VMware
image
• Base to build on

31. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image
• Conceptually similar to a Xen or VMware
image
• Base to build on
• Hundreds pre-built

32. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image
• Conceptually similar to a Xen or VMware
image
• Base to build on
• Hundreds pre-built
• Snapshots

33. Amazon: EC2 (AMIs)
• AMI - Amazon Machine Image
• Conceptually similar to a Xen or VMware
image
• Base to build on
• Hundreds pre-built
• Snapshots
• PoolParty default is the widely used
- ec2ubuntu
ami-1cd73375

34. Amazon: EC2

35. Amazon: EC2
• Primarily uses ephemeral storage

36. Amazon: EC2
• Primarily uses ephemeral storage
• e.g. node crashes = data gone (sortof)

37. Amazon: EC2
• Primarily uses ephemeral storage
• e.g. node crashes = data gone (sortof)
• requires new way to think about servers

38. Amazon: EC2
• Primarily uses ephemeral storage
• e.g. node crashes = data gone (sortof)
• requires new way to think about servers
• forces redundancy and clustering issues to
the front

39. Amazon: S3

40. Amazon: S3
• Simple Storage Service (S3)

41. Amazon: S3
• Simple Storage Service (S3)
• Persistent

42. Amazon: S3
• Simple Storage Service (S3)
• Persistent
• Data stored in “buckets”

43. Amazon: S3
• Simple Storage Service (S3)
• Persistent
• Data stored in “buckets”
• EC2 does not use S3 *directly*

44. Amazon: EBS

45. Amazon: EBS
• Elastic Block Store

46. Amazon: EBS
• Elastic Block Store
• Persistent disks for EC2 instances

47. Amazon: EBS
• Elastic Block Store
• Persistent disks for EC2 instances
• Can store snapshots on S3

48. Amazon: EBS
• Elastic Block Store
• Persistent disks for EC2 instances
• Can store snapshots on S3
• Fast

49. Amazon: EBS
• Elastic Block Store
• Persistent disks for EC2 instances
• Can store snapshots on S3
• Fast
• Mounts to only one instance at a time (not a
SAN)

50. Amazon: Elastic IPs

51. Amazon: Elastic IPs
• Instance IP addresses are dynamic

52. Amazon: Elastic IPs
• Instance IP addresses are dynamic
• Elastic IPs are static

53. Amazon: Elastic IPs
• Instance IP addresses are dynamic
• Elastic IPs are static
• Free to use, but $0.01/hr if unassociated
with an instance

54. EC2 Security Groups

55. EC2 Security Groups
• Amazon has a strict firewall

56. EC2 Security Groups
• Amazon has a strict firewall
• ec2-authorize

57. EC2 Security Groups
• Amazon has a strict firewall
• ec2-authorize
• default security group nodes have
unlimited (network) access to each other

58. EC2 Security Groups
• Amazon has a strict firewall
• ec2-authorize
• default security group nodes have
unlimited (network) access to each other
• everything else is turned off

59. EC2 Security Groups
ec2-add-group web -d quot;Web server groupquot;
ec2-authorize -p 80 web
ec2-authorize -p 22 web

60. EC2 Security Groups
ec2-add-group web -d quot;Web server groupquot;
ec2-authorize -p 80 web
ec2-authorize -p 22 web
ec2-authorize -P icmp -t -1:-1 web

61. Amazon: CloudFront

62. Amazon: CloudFront
• Amazon’s “self service” CDN

63. Amazon: CloudFront
• Amazon’s “self service” CDN
• Pay-as-you-go

64. Amazon: CloudFront
• Amazon’s “self service” CDN
• Pay-as-you-go
• Upload to S3

65. Amazon: CloudFront
• Amazon’s “self service” CDN
• Pay-as-you-go
• Upload to S3
• Content is distributed & cached on
Amazon’s machines

66. PoolParty

67. PoolParty overview

68. PoolParty overview
• binary tools

69. PoolParty overview
• binary tools
• configuration files

70. PoolParty overview
• binary tools
• configuration files
• erlang-driven messenger

71. PoolParty: Binary Tools

72. PoolParty: Binary Tools
• pool-* binaries

73. PoolParty: Binary Tools
• pool-* binaries
• server-* binaries

74. PoolParty: Binary Tools
Shell Example

75. PoolParty Configuration

76. PoolParty Configuration

77. PoolParty Configuration

78. PoolParty Configuration

79. PoolParty Configuration
Shell Example
a look at clouds/plugins/poolparty-mysql-plugin/mysql.rb

80. how does this actually work?

81. Puppet

82. Puppet

83. Puppet
• Puppet is a system for automating system
administration tasks.

84. Puppet
• Puppet is a system for automating system
administration tasks.
• Being used at Google to manage all Mac
desktops and laptops, expanding into Linux
clients.

85. Puppet
• Puppet is a system for automating system
administration tasks.
• Being used at Google to manage all Mac
desktops and laptops, expanding into Linux
clients.
• Red Hat - In the process of moving legacy
systems onto Puppet.

86. Puppet

87. Puppet
• Resources: files, folders, permissions, cron
jobs, mounted directories, packages, ssh
keys, services, arbitrary commands, etc.

88. Puppet
• Resources: files, folders, permissions, cron
jobs, mounted directories, packages, ssh
keys, services, arbitrary commands, etc.
• Mini-language

89. Puppet
• Resources: files, folders, permissions, cron
jobs, mounted directories, packages, ssh
keys, services, arbitrary commands, etc.
• Mini-language
• Dependencies

90. Puppet
• Resources: files, folders, permissions, cron
jobs, mounted directories, packages, ssh
keys, services, arbitrary commands, etc.
• Mini-language
• Dependencies
• If/unless conditions

91. Puppet
• Resources: files, folders, permissions, cron
jobs, mounted directories, packages, ssh
keys, services, arbitrary commands, etc.
• Mini-language
• Dependencies
• If/unless conditions
• Variables

92. Puppet

93. Puppet
• Client/Server architecture

94. Puppet
• Client/Server architecture
• puppetmasterd

95. Puppet
• Client/Server architecture
• puppetmasterd
• puppetd

96. Puppet: Language Example

97. Puppet: Language Example

98. Puppet: Language Example

99. Puppet: Language Example

100. Puppet: Language Example

101. Puppet

102. Puppet
• Puppet base config:
/etc/puppet/puppet.conf

103. Puppet
• Puppet base config:
/etc/puppet/puppet.conf
• PoolParty’s Puppet config:
/etc/puppet/manifests/classes/poolparty.pp

104. Puppet Configuration
Shell Example
a look at /etc/puppet/manifests/classes/poolparty.pp

105. Puppet
• Cron jobs:
* 1 * * * . /etc/profile && puppetmasterd --verbose
*/15 * * * * /usr/bin/puppetrunner

106. Puppet

107. Puppet
• Builds dependency graph

108. Puppet
• Builds dependency graph
• Provisions the machines (e.g. runs the
commands)

109. Puppet Security

110. Puppet Security
• Uses SSL certificates as the required and only form
of authentication

111. Puppet Security
• Uses SSL certificates as the required and only form
of authentication
• Master authenticates the certificates

112. Puppet Security
• Uses SSL certificates as the required and only form
of authentication
• Master authenticates the certificates
• Test certificate from node:
/usr/sbin/puppetd --waitforcert 60
--server master --test --verbose

113. Puppet Security
• Uses SSL certificates as the required and only form
of authentication
• Master authenticates the certificates
• Test certificate from node:
/usr/sbin/puppetd --waitforcert 60
--server master --test --verbose
• If certificate doesn’t match, probably cached on
server. Try on master:
puppetca --clean node1.compute-1.internal

114. Puppet Cache

115. Puppet Cache
• Keeps a cache of the parsed configuration in:
/var/lib/puppet/localconfig.yaml

116. Troubleshooting Puppet

117. Troubleshooting Puppet
• tail -f /var/log/syslog

118. Troubleshooting Puppet
• tail -f /var/log/syslog
• Can the nodes contact master?

119. Troubleshooting Puppet
• tail -f /var/log/syslog
• Can the nodes contact master?
• Do the certificates match?

120. Troubleshooting Puppet
• tail -f /var/log/syslog
• Can the nodes contact master?
• Do the certificates match?
• Does /var/lib/puppet/localconfig.yaml
contain the right information?

121. Troubleshooting Puppet
• tail -f /var/log/syslog
• Can the nodes contact master?
• Do the certificates match?
• Does /var/lib/puppet/localconfig.yaml
contain the right information?
• Does /etc/puppet/manifests/classes/
contain the right information?
poolparty.pp

122. Troubleshooting Puppet
• tail -f /var/log/syslog
• Can the nodes contact master?
• Do the certificates match?
• Does /var/lib/puppet/localconfig.yaml
contain the right information?
• Does /etc/puppet/manifests/classes/
contain the right information?
poolparty.pp
• Does contain the
/var/poolparty/poolparty.pp
right information? (more on that later)

123. Troubleshooting Puppet

124. Troubleshooting Puppet
• server-provision -n app -i 1
--slave --verbose --debug

125. master

126. PoolParty DNS
on master
root@master:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 master
75.101.128.147 master puppet localhost
10.252.166.18 node1.app node1
10.252.167.47 master.app master

127. PoolParty DNS
on master
root@master:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 master
75.101.128.147 master puppet localhost
10.252.166.18 node1.app node1
10.252.167.47 master.app master
on node1
root@node1:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 node1
75.101.128.147 puppet master
10.252.167.47 master.app master
10.252.166.18 node1.app node1

128. PoolParty DNS
• /etc/hosts modified by PoolParty when you
cloud provision

129. HAProxy

130. HAProxy

131. HAProxy
• Extremely fast

132. HAProxy
• Extremely fast
• Reports of HAProxy saturating gigabit fiber
(http://is.gd/4myI)

133. HAProxy
• Extremely fast
• Reports of HAProxy saturating gigabit fiber
(http://is.gd/4myI)
• Reputation of being reliable and secure

134. HAProxy
• Extremely fast
• Reports of HAProxy saturating gigabit fiber
(http://is.gd/4myI)
• Reputation of being reliable and secure
• Installed and setup by PoolParty by default

135. HAProxy

136. HAProxy
• Config kept in:
/etc/haproxy.cfg

137. HAProxy
• Config kept in:
/etc/haproxy.cfg
• Reload config by:
/etc/init.d/haproxy reload

138. HAProxy
• View stats by:
http://ec2-your-ip/poolparty

139. HAProxy
• View stats by:
http://ec2-your-ip/poolparty

140. PoolParty Messenger

141. PoolParty Messenger

142. PoolParty Messenger
• Three parts:

143. PoolParty Messenger
• Three parts:
• Master

144. PoolParty Messenger
• Three parts:
• Master
• Node

145. PoolParty Messenger
• Three parts:
• Master
• Node
• Client

146. PoolParty Messenger
• Three parts:
• Master
• Node
• Client
• “Instance Glue”

147. PoolParty Messenger
• Three parts:
• Master
• Node
• Client
• “Instance Glue”
• Master provisions nodes (will change)

148. PoolParty Messenger

149. PoolParty Messenger
• Erlang - excellent node-to-node monitoring

150. PoolParty Messenger
• Erlang - excellent node-to-node monitoring
• Gathers load data

151. PoolParty Messenger
• Erlang - excellent node-to-node monitoring
• Gathers load data
• Decides when to launch new nodes

152. PoolParty Messenger

153. PoolParty Messenger
• Changes happening in architecture

154. PoolParty Messenger
• Changes happening in architecture
• Will become neighborhood-based instead of
master/node-based

155. PoolParty Messenger
• Changes happening in architecture
• Will become neighborhood-based instead of
master/node-based
• The brains of the operation when humans
aren’t around

156. Using PoolParty

157. Using PoolParty

158. Using PoolParty
• cloud start

159. Using PoolParty
• cloud start
• cloud configure --verbose --debug

160. Using PoolParty
• cloud start
• cloud configure --verbose --debug
• cloud provision --verbose --debug

161. Using PoolParty: What Happens

162. Using PoolParty: What Happens
• PoolParty reads clouds.pool

163. Using PoolParty: What Happens
• PoolParty reads clouds.pool
• Generates templates, puppet configs, etc in a local
storage directory (/tmp/poolparty)

164. Using PoolParty: What Happens
• PoolParty reads clouds.pool
• Generates templates, puppet configs, etc in a local
storage directory (/tmp/poolparty)
• Files are rsync’d to /var/poolparty on
master

165. Using PoolParty: What Happens
• PoolParty reads clouds.pool
• Generates templates, puppet configs, etc in a local
storage directory (/tmp/poolparty)
• Files are rsync’d to /var/poolparty on
master
• A gang of bootstrapping tasks are run (via
Capistrano)

166. Using PoolParty: What Happens
• PoolParty reads clouds.pool
• Generates templates, puppet configs, etc in a local
storage directory (/tmp/poolparty)
• Files are rsync’d to /var/poolparty on
master
• A gang of bootstrapping tasks are run (via
Capistrano)
• Puppet config is cp’d from /var/poolparty to /etc/
puppet/manifests/classes/poolparty.pp

167. Using PoolParty: What Happens

168. Using PoolParty: What Happens
• Puppet is started

169. Using PoolParty: What Happens
• Puppet is started
• Puppet tries to provision everything

170. Using PoolParty: What Happens
• Puppet is started
• Puppet tries to provision everything
• Puppet starts PP Messenger master

171. Using PoolParty: What Happens

172. Using PoolParty: What Happens
• PP Messenger master contacts Amazon and boots
more instances if needed.

173. Using PoolParty: What Happens
• PP Messenger master contacts Amazon and boots
more instances if needed.
• PP Messenger bootstraps the new node

174. Using PoolParty: What Happens
• PP Messenger master contacts Amazon and boots
more instances if needed.
• PP Messenger bootstraps the new node
• The bootstrap process starts PP Messenger Node

175. Using PoolParty: What Happens
• PP Messenger master contacts Amazon and boots
more instances if needed.
• PP Messenger bootstraps the new node
• The bootstrap process starts PP Messenger Node
• PP Messenger initiates Puppet provisioning on the
new node

176. Using PoolParty: What Happens
• PP Messenger master contacts Amazon and boots
more instances if needed.
• PP Messenger bootstraps the new node
• The bootstrap process starts PP Messenger Node
• PP Messenger initiates Puppet provisioning on the
new node
• time passes

177. Using PoolParty: What Happens

178. Using PoolParty: What Happens
• Puppet runs on master and master discovers a
new node has been started.

179. Using PoolParty: What Happens
• Puppet runs on master and master discovers a
new node has been started.
• In the case of HAProxy/Apache, Puppet re-
generates the HAProxy config to include the new
node.

180. Using PoolParty: What Happens
• Puppet runs on master and master discovers a
new node has been started.
• In the case of HAProxy/Apache, Puppet re-
generates the HAProxy config to include the new
node.
• As specified in poolparty.pp, when haproxy.cnf gets
updated, it asks HAProxy to reload.

181. Using PoolParty: What Happens
• Puppet runs on master and master discovers a
new node has been started.
• In the case of HAProxy/Apache, Puppet re-
generates the HAProxy config to include the new
node.
• As specified in poolparty.pp, when haproxy.cnf gets
updated, it asks HAProxy to reload.
• HAProxy now starts distributing the load to the
new node

182. Using PoolParty: What Happens

183. Using PoolParty: What Happens
• Similar config changes happen throughout the
system

184. Real World Misc.

185. Real World Misc.

186. Real World Misc.
• tail -f /var/log/syslog is your friend

187. Real World Misc.
• tail -f /var/log/syslog is your friend
• ssh into master:
cloud ssh

188. Real World Misc.
• tail -f /var/log/syslog is your friend
• ssh into master:
cloud ssh
• ssh into i’th node:
cloud ssh -i 1

189. Real World Misc.

190. Real World Misc.
• scp
scp -i /Users/me/.ec2/id_rsa-clouds_app
root@ec2-75-101-191-10.compute-1.amazonaws.com:/
etc/snmp/snmpd.conf .

191. Real World Misc.
• scp
scp -i /Users/me/.ec2/id_rsa-clouds_app
root@ec2-75-101-191-10.compute-1.amazonaws.com:/
etc/snmp/snmpd.conf .
• rsync
rsync -av -e quot;ssh -i /var/poolparty/id_rsa-clouds_appquot;
/var/www/cpohunterfan.com/documents node1:/var/www/cpohunterfan.com/

192. Workshop

193. Agenda:

194. Agenda:
• Launch, login, terminate an EC2 instance

195. Agenda:
• Launch, login, terminate an EC2 instance
• into our instances and look around
cloud ssh

196. Agenda:
• Launch, login, terminate an EC2 instance
• into our instances and look around
cloud ssh
• Take a closer look at clouds.pool

197. Agenda:
• Launch, login, terminate an EC2 instance
• into our instances and look around
cloud ssh
• Take a closer look at clouds.pool
• Examine a basic PoolParty plugin