This document discusses using Amazon Web Services for scientific computing. It describes how AWS provides scalable, cost-effective, and reliable cloud infrastructure that researchers can use on-demand. Specific AWS services that can benefit scientific applications are mentioned, including Amazon EC2 for flexible computing power, S3 for data storage, and Elastic MapReduce for hosted Hadoop services. Examples are given of scientific projects that have successfully used AWS, such as genome analysis and astrophysics simulations. The document argues that AWS enables new approaches to doing science by providing access to vast, on-demand computing resources and platforms for distributed applications and data sharing.

1. Scien&fic
Compu&ng
with
Amazon
Web
Services
Deepak
Singh
ACAT
2010.
Jaipur,
India

3. Via Reavel under a CC-BY-NC-ND license

6. life science industry

7. Credit: Bosco Ho

9. By ~Prescott under a CC-BY-NC license

13. data

14. Image: Wikipedia

15. Image: Matt Wood

18. couldn’t find a good picture for
arrays of sensors in the ocean

19. Image
via
image
editor
under
a
CC-­‐BY
License

20. years

21. weeks

22. days

23. days

24. mi
nu
tes
days ?

25. gigabytes

26. terabytes

27. petabytes

28. petabytes

29. ex
ab
y tes
petabytes ?

30. Image: Chris Dagdigian

32. scale has implications

33. data management

34. data processing

35. data sharing

36. amazon web services

37. the cloud

38. has_many :definitions

39. infrastructure as a service

41. Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

42. Payments On-Demand
Parallel Processing Messaging
Content Delivery Amazon Flexible Workforce
Amazon Elastic Amazon Simple
Amazon CloudFront Payments Service Amazon Mechanical
MapReduce Queue Service (SQS)
(FPS) Turk
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

43. Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS Toolkit for .NET Cloud
Payments On-Demand
Parallel Processing Messaging
Content Delivery Amazon Flexible Workforce
Amazon Elastic Amazon Simple
Amazon CloudFront Payments Service Amazon Mechanical
MapReduce Queue Service (SQS)
(FPS) Turk
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

44. Your Custom Applications and Services
Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS Toolkit for .NET Cloud
Payments On-Demand
Parallel Processing Messaging
Content Delivery Amazon Flexible Workforce
Amazon Elastic Amazon Simple
Amazon CloudFront Payments Service Amazon Mechanical
MapReduce Queue Service (SQS)
(FPS) Turk
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

45. scalable

46. scalable
cost effective

47. go
o u
s y
scalable ay a
P
cost effective

48. scalable
cost effective
reliable

49. scalable
cost effective
reliable
secure

50. Amazon EC2

51. servers on demand

52. highly scalable

54. elastic

55. 3000 CPU’s for one firm’s risk management application
3444JJ'
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
344'JJ'
I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1='
>?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

56. highly available systems

58. “Everything fails, all the time”
-- Werner Vogels

59. 2.3% AFR in population of 13,250
3.3% AFR in population of 22,400
4.2% AFR in population of 246,000
Source: James Hamilton

60. assume sw/hw failure

61. design apps to be resilient

62. automate & bootstrap

63. nothing fails

64. elastic block store

65. elastic IP

66. SQS

67. US East Region
Availability Availability
Zone A Zone B
Availability Availability
Zone C Zone D

68. on-demand instances
reserved instances
spot instances

69. data storage

70. one size does not fit all

72. Amazon S3

73. distributed object store

74. durable

75. available

76. !"#$%&'()*+
T T
T

77. scalable

78. fast

79. simple

81. structured data anyone?

82. Amazon SimpleDB

83. zero administration

84. highly available

85. schema less

86. key-value store

87. Amazon Relational Data Service

88. single API call

89. MySQL database

90. automatic backup

91. scale up with API call

93. hosted hadoop service

94. hadoop easy and simple

95. Amazon Elastic
MapReduce
Amazon EC2 Instances
End
Deploy Application
Hadoop Hadoop Hadoop
Elastic Elastic
MapReduce MapReduce
Hadoop Hadoop Hadoop Notify
Web Console, Command
line tools Input output
dataset results
Input
S3
Output
S3
Get Results
Input Data
bucket bucket
Amazon S3

96. apache hive
http://hadoop.apache.org/hive/

97. apache pig
http://hadoop.apache.org/pig/

98. cascading
http://www.cascading.org/

100. computing platforms

103. http://cyclecomputing.com

104. sudo gem install cloud-crowd
http://cyclecomputing.com
http://wiki.github.com/documentcloud/cloud-crowd

105. http://www.rightscale.com

106. Amazon Elastic
MapReduce
Amazon EC2 Instances
End
Deploy Application
Hadoop Hadoop Hadoop
Elastic Elastic
MapReduce MapReduce
Hadoop Hadoop Hadoop Notify
Web Console, Command
line tools Input output
dataset results
Input
S3
Output
S3
Get Results
Input Data
bucket bucket
Amazon S3

107. application platforms

108. http://heroku.com

109. software distribution

110. http://www.cloudbiolinux.com/

111. http://bitbucket.org/galaxy/galaxy-central/wiki/Home

113. data distribution

115. http://aws.amazon.com/publicdatasets/

116. problem solving

118. 3.7 million classifications in just over three days
~15 million in less than a month
>2.6 million clicks in 100 hours

119. software & algorithms

121. Crossbow: Rapid whole genome SNP analysis
Preprocessed reads
Map: Bowtie
Sort: Bin and partition
Reduce: SoapSNP
Langmead B, Trapnell C, Pop M, Salzberg SL. Genome Biol 10 (3): R25.

122. Crossbow
condenses
over
1,000
hours
of
resequencing
computa:on
into
a
few
hours
without
requiring
the
user
to
own
or
operate
a
computer
cluster

123. doing science

124. http://bioteam.net

125. BLAT @ U. PENN
Map 100 million, 100 base paired end reads
Quad core with 5 GB of RAM would take 16 days
30 high-memory instances; 32 hours; $195

126. GALAXY MAPPING
Goal: Create an astrometric catalog of a billion
stars with micro arc second precision
Gaia satellite launched 2011; observations till
2017; catalog ready 2019
Problem: Single pass through the data for image
processing would take 30 years (on one CPU)
Solution: Use AWS

127. Capacity Capacity
Resources
Resources
Demand Demand
Time Time
Static data center Data center in the cloud
Unused resources

128. HEAVY-ION COLLISIONS
Problem: Quark matter physics conference
imminent but no compute resources handy
Solution: NIMBUS context broker allowed
researchers to provision 300 nodes and get the
simulations done

129. BELLE MONTE CARLO
Credit: Tom Fifield

130. AWS for the sciences

131. available resources

132. task-based resources

133. shared dataspaces

134. new software architectures

135. new computing platforms

136. http://aws.amazon.com/education

137. the cloud works

138. today

140. Thank
you!
deesingh@amazon.com
Twi<er:@mndoci
Presenta?on
ideas
from
James
Hamilton
and
@lessig