19. I
want
to
process
data,
for
example
encode
movies,
…
Job Queue
EC2
Instance
Data Processing
20. …
and
store
the
results
in
S3.
Job Queue
EC2
Instance
S3
Data Processing
21. And
I
want
to
be
no@fied
on
comple@on
Job Queue
EC2
Instance
Result
e-mail
S3
Data Processing
22. I
like
using
SNS,
because
…
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing
23. I
like
using
SNS,
because
…
I
can
integrate
other
systems
via
SQS,
or
HTTP(S)
web-‐hooks
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing SQS
HTTP
24. I
should
be
done
...
right?
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing
25. Not
quite
–
I’m
also
cost
conscious
…
Don’t
want
to
pay
anything
when
there’s
no
work
Autoscaling
Job Queue
Charge
empty
$0
Topic Result
e-mail
S3
Data Processing
26. Need
a
controller
…
Capacity Control (on-demand launch)
Autoscaling
Job Queue
Charge
$0
Topic Result
e-mail
S3
Data Processing
27. Measure
…
Detect
work
…
Capacity Control (on-demand launch)
CW Alarm
CW Metric
#msgs
Autoscaling
Job Queue
Charge
$0
Topic Result
e-mail
S3
Data Processing
28. When
there’s
a
message
queued
…
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
Charge
$0
Topic Result
e-mail
S3
Data Processing
29. When
there’s
a
message
queued
…
start
an
EC2
instance
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing
30. ...
or
5000
EC2
instances
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
keep adding instances
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
5000
EC2
Instances Topic Result
e-mail
S3
Data Processing
31. When
the
work
is
done
…
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing
32. When
the
work
is
done
…
Terminate
the
EC2
instance
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
TERM’D
Topic Result
e-mail
S3
Data Processing
33. For
full
produc@on
use,
add
IAM
Users,
Permissions,
Policies,
OperaFonal
Alarm,
etc.
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
EC2
Instance
Topic Result
e-mail
S3
IAM
User
IAM
User IAM
IAM
Permissions
IAM
Permissions
IAM
Permissions
IAM
Policy
Policy
Data Processing
34. On
Demand
SQS
Worker
Example
Capacity Control (on-demand launch)
CW Alarm
Autoscaling
Policy
add instance
CW Metric if #msg > 0
#msgs
Autoscaling
Job Queue
EC2
Instance
Topic Result
e-mail
S3
Data Processing
No
glue-‐scrip.ng
(or
UI
code)
required
…
We
let
you
focus
on
the
business
problem!
36. One
substack
per
e.g.
Video
format
to
encode
Parent
template/stack
Parameterized
by
e.g.
format/resolu1on
Job Queue
Encode for
low-res, … S3 S3
Job Queue
Encode for
Job: high-res, … S3 S3
Render X Topic
Job Queue
S3 S3
Job Queue
S3 S3
64. WIEN2K Parallel
Performance
H size 56,000 (25GB)
Runtime (16x8 processors)
Local (Infiniband) 3h:48
Cloud (10Gbps) 1h:30 ($40)
1200 atom unit cell; SCALAPACK+MPI
diagonalization, matrix size 50k-100k
Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
110. Netflix needed to transcode
17,000 titles (80TB of data) to
support the launch of Sony PS3.
They provisioned 1200 Amazon
EC2 instances and completed
the transcoding process in just
days.
Source: Adrian Cockroft (Netflix)
138. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.github.com
Inspiration and ideas from
Matt Wood & Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license