I describe a distributed load testing tool I built. It deploys some number of ec2 micros, configures them and then launches the load balancing tool in parallel against the website. github.com/j2labs/microarmy

1. Micro
Army:
Load
Tes1ng
By
James
Dennis
(@j2labs)
h?p://j2labs.net

2. What
is
Micro
Army?
• Website
Load
Tes1ng
with
EC2
micros
– Micros
are
cheap
– I spent $2 building the tool
– Brubeck
vs.
Tornado
• this
is
basically
Eventlet
vs.
Tornado
– Easily
scalable,
• at
least
high
enough
to
slaughter
a
server

3. Challenges
• Arbitrary
number
of
boxes
should
be
possible
– I
call
them
“cannons”
– They
should
be
configurable
via
some
script.
• With
a
liAle
more
work,
it
could
be
an
admin
tool
• Parallel
– Deploying
the
cannons
must
be
parallel
• I’m
impaEent
– The
cannons
must
fire
in
parallel

4. Python
Modules
• Boto:
AWS
for
Python
– Used
to
deploy
EC2
instances
• Paramiko:
SSH
– Required
a
liAle
work
– Fairly
old
• Not
sure
if
that’s
bad
though
• Eventlet:
paralleliza1on
(and
other
goodness)
– GreenPiles
are
easy
– Free
nonblocking
I/O
• Listen
up,
Tornado
users

5. Using
It
• Deploy
–
Hopefully
AWS
gives
us
our
boxes
– We’ll
need
a
list
of
hostnames
– SSH
to
each
host
and
setup
the
boxes
• Upload
and
run
script
• Must
be
parallel
or
large
numbers
hurts
– Prints
host
list
• The
deploy
script
will
eval()
it
for
you!

6. Using
It
• Firing
the
cannons
– SSH
to
all
boxes
and
run
`siege`
• siege c 200 t 10s `hostname`
• I usually test with c 500
• Siege can handle GET and POST args (!)
– Each
SSH
session
blocks
unEl
compleEon
– Eventlet
makes
this
easy
to
manage
– Quick
and
dirty
parsing
of
out
to
generate
a
CSV

7. Show
Me
Code

8. Micro
Army:
SeUngs
Easily
overrideable
with
local_seUngs.py
### Create n instances
aws_access_key = ‘...'
aws_secret_key = ‘...'
ami_key = 'ami-ccf405a5’ # official ubuntu image
ec2_ssh_username = 'ubuntu' # ami specific
security_groups = [’MicroArmyGroup']
key_pair_name = ’micro_army_test'
num_cannons = 5
placement = 'us-east-1a’
instance_type = 't1.micro’
ec2_ssh_key = '/some/path/ec2_testing.pem'

9. Boto:
AWS
Find
an
OS
image
(official
Ubuntu)
import boto
# First, get a connection
ec2_conn = boto.connect_ec2(aws_access_key, aws_secret_key)
# Get *all* images that match this unique AMI key
images = ec2_conn.get_all_images(ami_key)
image = images[0]

10. Boto:
AWS
Instan1ate
N
boxes
### Create n instances
r = image.run(min_count=num_cannons,
max_count=num_cannons,
placement=placement,
security_groups=security_groups,
key_name=key_pair_name,
instance_type=instance_type)

11. Boto:
AWS
Launching
is
a
li?le
verbose…
while True:
# Give AWS some time to work
time.sleep(5)
# Aggregate our instance status info
for i in r.instances:
i.update()
status = [i.state for i in r.instances]
# Finish only if Amazon says all instances are up
if status.count('running') == len(r.instances):
print 'Done!’
# Return list of host names
return [i.public_dns_name for i in r.instances]

12. Paramiko:
SSH
Everything
you
need:
to
use
build
an
SSH
abstracEon
jd@gibson : 14:42:44 : ~/Projects/microarmy/microarmy
$ wc -l communications.py
61 communications.py
jd@gibson : 14:42:45 : ~/Projects/microarmy/microarmy
$ grep "def" communications.py
def ssh_connect(host, port=22):
def exec_command(transport, command, return_stderr=False):
# And half an SFTP abstraction...
def sftp_connect(transport):
def put_file(transport, local_path, remote_path):
def put_files(transport, paths):

13. Micro
Army:
build_cannon.sh
apt-­‐get
can
handle
everything
### Let's get recent stuff
apt-get update
### Install
apt-get -y install
python-dev build-essential autoconf automake
libtool uuid-dev git-core mercurial python-pip
siege

14. Paramiko:
SSH
Configure
each
cannon
with
a
script
def _setup_a_cannon(hostname):
# Get a connection (paramiko calls it a transport)
ssh_conn = ssh_connect(hostname)
# copy script to cannon and make it executable
script_path = env_scripts_dir + '/' + CANNON_INIT_SCRIPT
put_file(ssh_conn, script_path, CANNON_INIT_SCRIPT)
response = exec_command(ssh_conn, 'chmod 755 ~/%s' % CANNON_INIT_SCRIPT)
if response: # response would be error output
return False
# execute the setup script (expect this call to take a while)
response = exec_command(ssh_conn, 'sudo ./%s' % CANNON_INIT_SCRIPT)
return (hostname, response)

15. Eventlet:
GreenPools
Setup
each
host
in
parallel
def setup_cannons(hostnames):
print 'Loading cannons... ',
# Use eventlet’s abstraction of a collection of tasks: a GreenPile
pile = eventlet.GreenPile(pool)
# Spawn a coroutine in our pile for each host
for hostname in hostnames:
pile.spawn(_setup_a_cannon, hostname)
# Iterating the pile causes eventlet to do the work
responses = list(pile)
return responses

16. Micro
Army:
Cannons
are
GO!
Time
to
blast
some
poor
web
server
(that
you
own)
def fire_cannon(cannon_host, target):
ssh_conn = ssh_connect(cannon_host)
# 200 simultaneous connections requesting data for 10 seconds
remote_command = 'siege -c200 -t10s %s' % (target)
# Siege writes stats to stderr
response = exec_command(ssh_conn,
remote_command,
return_stderr=True)
return response

17. Micro
Army:
Fire
Cannons!
Same
strategy
as
before,
with
a
GreenPile
def slam_host(cannon_hosts, target):
# Familiar pattern
pile = eventlet.GreenPile(pool)
for hostname in cannon_hosts:
pile.spawn(fire_cannon, hostname, target)
responses = list(pile)
# Parse output from each host for CSV generation
report = parse_responses(responses)
return report

18. Micro
Army:
Finishing
up.
Just
the
facts,
micro.
Num_Trans,Elapsed,Tran_Rate
3679,
9.54,
385.64
3635,
9.48,
383.29
3535,
9.33,
378.89

19. Micro
Army:
Recap
• LAUNCH
some
number
of
EC2
micros
• INSTALL
siege,
etc
on
instances
• SLAM
web
host
from
micros
in
parallel
• AGGREGATE
the
results
in
a
CSV

20. Hopefully
you
learned
how
to…
• deploy
instances
on
EC2
with
Python
• execute
a
script
on
many
boxes
in
parallel
• use
SSH
from
Python
– You
should
sEll
checkout
Fabric
(fabfile.org)
• use
the
excellent
module,
Eventlet:
– Free
nonblocking
I/O
– Simple,
but
efficient
concurrency
– Read
the
webpage,
it
actually
gets
beAer
• Green
threads!

21. Try
it!
h?ps://github.com/j2labs/microarmy
h?ps://console.aws.amazon.com/ec2/home
Read
more!
h?p://eventlet.net
h?p://www.lag.net/paramiko/
h?p://boto.cloudhackers.com/

22. Ques1ons
?
James
Dennis
(@j2labs)
jdennis@gmail.com