Thoughts on how to utilize Amazons EC2 (Elastic Compute Cloud) web service for hosting applications, especially with Ruby on Rails in mind. A major issue is the non persistence of local storage in the EC2 environment. We outlined several approaches on how to possibly solve this issue, minimizing potential data loss. by Adam Groves and Martin Rehfeld

1. Amazon S3 + EC2
+ Rails = Dream Team?
Thoughts by
Adam Groves and Martin Rehfeld
@ BRUG 05-Apr-2007

2. Amazon S3
Simple Storage Service
• Pricing
• $0.15 per GB-Month NO CO
ST
(10 GB = $1.50 per month)
DRIVE
R
• $0.20 per GB of data transferred BEWA
RE
HIGH
TRAFF
IC
(100 GB = additional $20 per month) SITES

3. Amazon S3
Common Terms
• buckets:
S3
global name space
some bucket A
• objects:
accessed by key [~ path+name] my bucket #1
can have metadata object 1, key = „images/icons/smile.png“, encoding=...
object 2, key = „images/icons/cry.png“, encoding=...
• prefixes:
}
identical prefix
searching by prefix emulates
directory structure my bucket #2
• ACL: some bucket B
...
everyone, authenticated users,
owner / named users

4. S3Fox Extension

5. Amazon EC2
Elastic Compute Cloud
• Pricing
• $0.10 per instance-hour consumed IDEAL
STAGIN FOR
GA
SCALIN ND
(1 month 24x7 ~ $72) G
• $0.20 per GB of data transferred
(just internet traffic; TRAFF
WARN IC
no charge for EC2-S3 traffic)
IN
APPLIE G
S

6. Amazon EC2
Instance Specs
• ~ 1.7 GHz x86 CPU
ACTUA
LLY
A XEN
VIRTUA
INSTAN L
CE
• 1.75 GB RAM
• 160 GB local disk space
• 250 Mb/s network bandwidth

7. Amazon EC2
Common Terms
• images:
named OS images stored in S3: „AMI“
• instances:
virtual maschines running an AMI
• bundling:
saving customized images back to S3
• non-persistence: BUG O
FEATU R
local disk storage will not survive instance RE?
shutdown or failure

8. Amazon EC2
Network Security
access group quot;defaultquot;
access group quot;webquot;
Internet
access group quot;backendquot;

9. Amazon EC2
Rails Scenario
}
EC2 S3
Web Static
Server Content

Application
Code
Server
memcached,
backgrounDRb,
...

SQL HOW
TO
GET A
PERSIS
Database TENT
DB?

10. Amazon EC2
Database Persistence
Flavor A:
• on instance failure
Frequent backup to S3 you will lose all
changes since last
backup
S3 • backup using a lot
backup job
of resources
DB
• no automatic
failover

11. Amazon EC2
Database Persistence
Flavor B:
• on instance failure
Backup write ahead logs to S3 you will lose all
changes since last
log switch
•
S3
DB
backup is fast, but
backup job
recovery will take
WAL
longer
• still no automatic
failover

12. Amazon EC2
Database Persistence
Flavor C:
• you might still lose all
Shadow database changes since last log
switch
•
Master Shadow backup is fast, recovery is
usually not neccessary
(only if master and slave
continous
DB recovery
should fail at the same
time)
WAL transfer DB
WAL
• no load balancing
between master and slave
via
• automatic failover can be
achieved
S3

13. Amazon EC2
Database Persistence
•
Flavor D: data loss only on failure
of all instances
(Multi-Master-)Replication • reduced write
performance
Master #1 Master #2
• sophisticated
configuration
 can be tricky
two-way
• should be combined with
replication
DB flavor A or B for backup
DB
• load balancing can be
backup job
achieved
• automatic failover can be
achieved
S3

14. Amazon EC2
Database Persistence
Flavor E (highly Experimental): • unless caching is used
there should be no data
DB files on S3DFS loss
• (very?) reduced
performance
• reliability of S3DFS yet to
S3 be proven
• can be combined with
DB
S3DFS fuse filesystem
flavors A or B for backup
• can be combined with
flavors C or D for
failover and load
balancing (flavor D only)

15. Resources
• Amazon EC2 / S3
aws.amazon.com/ec2 / aws.amazon.com/s3
• Rails & EC2 & Capistrano tasks
• Amazon Web Services Ruby Gem (aws/s3)
http://niblets.wordpress.com/2007/02/12/capistrano-ec2-sitting-in-a-tree-k-i-s-s-i-n-g/
http://amazon.rubyforge.org/
• S3DFS filesystem driver
http://www.openfount.com/blog/s3dfs-for-ec2
• Discussion on persistence with EC2
http://blog.awswebshop.com/2007/01/07/thoughts-on-current-functionality-requests-for-future-functionality/
• Postgres WAL / continuous archiving
http://www.postgresql.org/docs/8.2/static/wal-intro.html / http://www.postgresql.org/docs/8.2/static/continuous-archiving.html
• MySQL multi-master replication
http://capttofu.livejournal.com/1752.html

16. Q &A
!
Adam Groves
adam.groves@gmail.com
Martin Rehfeld
martin.rehfeld@glnetworks.de