This is a presentation to JClouds Meetup @ Twitter's HQ. Presented by Toni Batchelli. It discusses the methodologies that the GoGrid engineering team is employing to quickly and accurately produce timely builds.
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Agile Development at GoGrid with Pallet and JClouds
1. Complex Infrastructure Made Easy™
Agile Development at GoGrid with
Pallet and JClouds
Presented to JClouds Meetup @ Twitter’s HQ
Toni Batchelli, Consultant and Entrepreneur wannabe
tbatchelli@gmail.com, @tbatchelli, @disclojure
February 2, 2011
02/01/11
2. GoGrid’s Challenges
GoGrid is a cloud provider, building their own
software
Developing and testing their cloud
management software is complex. They need
to create and maintain realistic environments
in order to load them with their latest software
builds
~10 different servers types
~20 different services
3. The problem?
Current time to setup a development/QA
environment is 2 days
• Mostly manual work
Building the environments manually is very
error prone
A high level of interconnectivity between
services
4. Goal
To be able to create development and
testing environments dynamically, in a
customizable way, and on cloud
infrastructure
A Cloud in a Cloud
5. Additional Challenges
Environments need to be verified as fully functional before
deploying software on them
Must support multiple locations
Must be able to scale dynamically
Must support concurrent development
• Different feature branches
Must manage/version non-software components in a
lockstep with the software:
• Network configuration
• System-level configurations
• Application-level configurations
7. Abstraction layer on resource management
If you code your infrastructure management
code to the jclouds API, you can port your
code to all cloud providers
JVM-based
Provides access to GoGrid’s infrastructure by
abstracting the communication GoGrid’s
Cloud API
8. Uses JClouds to create and manage nodes
Provisions nodes with software and
configuration
Verifies node and service correct operation
Operates the services in the nodes
Nodes are grouped by groups (formerly
known as ‘tags’). All nodes in a group are
equivalent
9. Clojure
Pallet is built on Clojure
Clojure is a JVM-based lisp with superb
integration with Java and concurrency
Can use any existing Java library (Tomcat, jdbc,
etc...)
Provides a high level of abstraction, which is
what is needed in this case
Growing community: there are 7 books on Clojure
either published or in preparation
11. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
12. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service
13. GoGrid Cloud Infrastructure
gimme
Service Service
nodes!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service
14. GoGrid Cloud Infrastructure
gimme
Service Service
nodes!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service
15. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
16. GoGrid Cloud Infrastructure
Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
17. GoGrid Cloud Infrastructure
do stuff! Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
18. GoGrid Cloud Infrastructure
do stuff! Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
22. GoGrid Cloud Infrastructure
make me Service Service
a dev cloud!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service
23. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
24. GoGrid Cloud Infrastructure
install cloud
Service Service
infrastructure!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
25. GoGrid Cloud Infrastructure
install cloud
Service Service
infrastructure!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
26. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
27. GoGrid Cloud Infrastructure
is all working Service Service
fine?
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
28. GoGrid Cloud Infrastructure
is all working Service Service
fine?
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
OK!
Service Service
Service
29. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
30. GoGrid Cloud Infrastructure
deploy Service Service
new build!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
31. GoGrid Cloud Infrastructure
deploy Service Service
new build!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service
32. GoGrid Cloud Infrastructure
deploy GoGrid Cloud Infrastructure
Service Service
new build!
et
GoGrid Cloud API
pall
GoGrid Cloud API
Service Service Service
pallet Service Service
Service
Service Service
Service
33. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
34. GoGrid Cloud Infrastructure
awesome! GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
35. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
36. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service
37. GoGrid Cloud Infrastructure
let’s test:
GoGrid Cloud Infrastructure
Service Service
gimme nodes!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service
38. GoGrid Cloud Infrastructure
let’s test:
GoGrid Cloud Infrastructure
Service Service
gimme nodes!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service
39. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service
40. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service
41. GoGrid Cloud Infrastructure
I’m done, GoGrid Cloud Infrastructure
Service Service
thanks!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service
44. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud API
45. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud API
GoGrid Cloud API
46. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
47. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API
48. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API
environment
for QA please?
49. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
GoGrid Cloud Infrastructure QA
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API
GoGrid Cloud API
environment
for QA please?
51. GoGrid’s Development Process
Create a feature branch
Create an environment (cloud infrastructure) for
this branch
Verify new environment (end-to-end tests)
Automatically build and deploy the new software
every time new code is committed to the branch.
Destroy environment and free resources when
done
52. An Environment
Usually one environment for active
development branch
Each environment is specified by:
• The development branch it tracks
• Service types and node count for each service type
• Network configuration
• External services/DB provisioning
53. Development Environments
Each development environment tracks a
development feature branch
Each environment has a node with Hudson
CI, configured to track the environment’s
branch
When new code is pushed to the development
branch, the environment’s Hudson server
builds the software and updates the
environment with it.
55. Why Pallet?
There are well known solutions for node
provisioning in the cloud, why pallet?
• It integrates with infrastructure management
• Service interconnections are first class entities in
the framework
• Scalable programming model
• Integrates well with SCM and with other OPS tools
• Comprehensive support for all development cycles
56. ✓ No central server
✓ No agents on nodes
✓ Works with standard and custom images
✓ First-class support for complex multi-service
configurations
✓ All node configuration done remotely via
generated scripts over SSH
✓ DSL for generating scripts, resources
✓ Testable
57. crates
Pallet’s basic configuration unit
A crate is a grouping of functions around a
service
Functions usually:
• manipulate/create internal representations
• result in resources deployed or scripts being run in
the target nodes
58. proxied
Custom Crates
tomcat reverse
bootstrap tomcat
deploy proxy
Standard Crates
automated haproxy
admin user
java tomcat
59. A Node
(core/defnode
proxied
"Basic
web
app,
served
by
tomcat"
{:os-‐family
:ubuntu
:os-‐description-‐matches
"10.04"
:inbound-‐ports
[8080
22]}
:bootstrap
(resource/phase
(crates/bootstrap))
:configure
(resource/phase
(crates/tomcat))
:deploy
(resource/phase
(crates/tomcat-‐deploy
"webapp.war"))
:restart-‐tomcat
(resource/phase
(service/service
"tomcat6"
:action
:restart)))
60. jclouds section of a node
(core/defnode
proxied
"Basic
web
app,
served
by
tomcat"
{:os-‐family
:ubuntu
:os-‐description-‐matches
"10.04"
:inbound-‐ports
[8080
22]}
:bootstrap
(resource/phase
(crates/bootstrap))
:configure
(resource/phase
(crates/tomcat))
:deploy
(resource/phase
(crates/tomcat-‐deploy
"webapp.war"))
:restart-‐tomcat
(resource/phase
(service/service
"tomcat6"
:action
:restart)))
62. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB
63. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB
64. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB
65. Cloud Infrastructure
Crates proxy proxy
jclouds
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB
66. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB
67. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
definitions DB DB
68. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions
69. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions
70. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions
71. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions
72. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions
73. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions
74. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions
mysql mysql
75. How are systems interconnected?
Phase 1
• Retrieve and/or create needed nodes
• For each node in each group: run phases for crates
and
a) Collect configuration info
b) Schedule configuration actions to be performed
on node
Phase 2
• Generate and execute scripts and resources
78. Operation
user => (start branch-1-environment)
user => (verify branch-1-environment)
user => (destroy branch-1-environment)
79. Challenge with Pallet
It’s a newcomer in the field.
• Documentation needs work
• Untested support for some combinations of flavors
and versions of Linux
Steep learning curve, a new language
OMG! It’s A Lisp!
80.
81. Some pointers...
http://jclouds.org Adrian Cole et al.
http://palletops.com Hugo Duncan et al.
http://gogrid.com For more info on GoGrid
services please contact:
Paul Lappas
VP Engineering & Co-Founder of GoGrid
paul@gogrid.com
GoGrid is hiring devops and SW engineers
for their SF office (contact Paul too)
82. Questions?
Toni Batchelli
tbatchelli@gmail.com
@tbatchelli
@disclojure
02/01/11