Slides of the Presentation (30 july 2010) of Grid on Demand

1. Welcome
Grid on Demand
Willem Toorop and Alain van Hoof
{wtoorop,ahoof}@os3.nl
June 30, 2010
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39

2. Introduction Research Question
Research Question
Can Grid computing be offered as a Cloud service and what
type of applications benefit from this?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39

3. Introduction The Grid and The Cloud
The Grid and The Cloud: Sharing Resources
Sharing Resources to minimizing idle resources
Resource Capacity Goods
Processor Flops Compute
Hard-disk memory Bytes Storage
Network Bytes/sec Communication Bandwidth
Grid Cloud (IaaS)
Distribution By Collaboration By Cloud Provider
Resource Type Heterogeneous Homogeneous
Abstraction Middleware (open standards) Virtualization
Sharing Guarantee Security infrastructure Financial infra
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39

4. Introduction The Grid and The Cloud
The Grid and The Cloud: Dividing the Resources
VO
Grid User User
Cloud
Grid Grid
User User
User
User User
VO
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39

5. A Grid in the Cloud Urgent Computing
A Grid in The Cloud: Urgent Computing
GRID
on
Demand
Using the elasticity and scalability of the Cloud and providing a Grid
interface, current e-Science (computationally intensive science)
applications can use a Grid on Demand when there is a high demand for
resources for a short period of time.
Example
UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictions
when actual flooding happens, Grid on Demand can provide this.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39

6. A Grid in the Cloud Grid on Demand
Grid on Demand
Job Submission Client
OS: Ubuntu 10.04
Grid: Globus 4.2.1
PBS: Torque 2.6.8
Compute Nodes
Head Node
Grid Interface: Globus Toolkit 4.2 (open standard, widely used)
Queue Manager (Distribution of jobs to nodes): Torque 2.6.2
Amazon EC2 Cloud Provider (de-facto standard with open API)
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39

7. Implementation What makes the Grid, the Grid
What makes the Grid, the Grid
Typical Grid usage example spanning multiple organizations
Organization B
ta Compute
st 1 o
f da acc
1 2 te 12X e
dat ss
use mpu a
o
Organization C to c Organization A
use my data
to compute X Grid store data
Scheduler Storage
User
use 2 nd
1 ata
to co 2 of da ss d
mpu 1 ta
te X acce
2 Compute
Organization D
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39

8. Implementation What makes the Grid, the Grid
Proxy certificates for delegation
The Grid is a collaboration between multiple organizations.
Grid users need to be authorized to use remote resources
Those resources should be authorized to perform actions
on behalf of the Grid user
Proxy certificates enable delegation of authority
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39

9. Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
compute X
create cert.
request
sign this
Compute
Okay
compute X
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

10. Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
compute X
create cert.
request
sign this
Compute
Okay
compute X
Virtual Organizations are based on
delegation by proxy certificates.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

11. Implementation What makes the Grid, the Grid
Grid user
Proxy certificates allow for
dynamic resource allocation,
create proxy Scheduler
But the hosts still need a
CA signed certificate.
compute X
This is a problem when hostname’s
create cert. are assigned dynamically!
request
sign this
Compute
Okay
compute X
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

12. Implementation What makes the Grid, the Grid
Grid user
Proxy certificates allow for
dynamic resource allocation,
create proxy Scheduler
But the hosts still need a
CA signed certificate.
compute X
This is a problem when hostname’s
create cert. are assigned dynamically!
request
sign this
Compute
Fixed CA in image?
Okay
compute X
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

13. Implementation What makes the Grid, the Grid
Grid user
Proxy certificates allow for
dynamic resource allocation,
create proxy Scheduler
But the hosts still need a
CA signed certificate.
compute X
This is a problem when hostname’s
create cert. are assigned dynamically!
request
sign this
Compute
Fixed CA in image?
Okay
compute X Dynamic CA on every Launch?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39

14. Implementation Amazon Elastic Block Store
Easy Image management with Amazon Elastic Block Store
configure
launch
instance IA Running
(A, S) Pending
VS ← S (IA , VS , H) term
inate
p
sto
Shutting-down
Stopping Pending
Terminated
rt
sta
IA
inate
term
create AMI A Stopped
(A , S )
S ← VS (IA , VS )
Life cycle of Amazon Machine Images using Elastic Block Storage
A = Amazon Machine Image, S = Snapshot,
I = Instance, V = Volume, H = Hostname
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39

15. Implementation EBS for configuration stages
EBS for configuration stages
configure configure
launch provide Amazon Access Credentials
instance IA Running (configure cluster) Running
(A, S)
VS ← S (IA , VS , H) (IA , VS , H)
sh shar
sh wn
for
ng
ut
for utdo
shutdown
shutdown
for reuse
for reuse
do ing
ari
start
start
wn
sh
launch IA launch IA
VS ← S VS ← S
Stopped Stopped Stopped
(IA , VS ) (IA , VS ) (IA , VS )
create AMI A create AMI A create AMI A
S ← VS S ← VS S ← VS
(A , S ) (A , S ) (A , S )
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39

16. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39

17. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39

18. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39

19. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39

20. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39

21. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39

22. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39

23. Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39

24. Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39

25. Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39

26. Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39

27. Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39

28. Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39

29. Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39

30. Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39

31. Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39

32. Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39

33. Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39

34. Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39

35. Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39

36. Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39

37. “Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a Biomedical
Application and a workflow manager are used
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

38. “Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a Biomedical
Application and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

39. “Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a Biomedical
Application and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39

40. “Real use” and Performance Connecting the Grid on Demand to a Workflow Manager
Connecting the Grid on Demand to a Workflow Manager
WAVE
Biomedical
Compute OS: Ubuntu 10.04
Application
Grid: Globus 4.2.1
PBS: Torque 2.6.8
Compute Nodes
Head Node
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 34 / 39

41. “Real use” and Performance Preliminary performance results: total execution time
Preliminary performance results: total execution time
Grid on demand compared to a 32 node physical UvA cluster (DAS3) with
Globus Grid interface
Grid Jobs Total execution time
DAS3 28 00:42:58
Grid on Demand 28 00:46:22
DAS3 98 02:46:36
Grid on Demand 98 00:52:26
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 35 / 39

42. “Real use” and Performance Preliminary performance results: 30 Jobs
Preliminary performance results of job pending and
running time: 30 jobs
Performance on DAS3 - 30 Jobs Performance on Grid on Demand 30 Jobs
01:20:00 01:20:00
Running time Running time
Pending time Pending time
01:10:00 01:10:00
01:00:00 01:00:00
00:50:00 00:50:00
Time (HH:MM:SS)
Time (HH:MM:SS)
00:40:00 00:40:00
00:30:00 00:30:00
00:20:00 00:20:00
00:10:00 00:10:00
00:00:00 00:00:00
0 5 10 15 20 25 30 0 5 10 15 20 25 30
Job Number Job Number
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 36 / 39

43. “Real use” and Performance Preliminary performance results: 100 Jobs
Preliminary performance results of job pending and
running time: 100 jobs
Performance on DAS3 - 100 Jobs Performance on Grid on Demand - 100 Jobs
Running time Running time
02:30:00 Pending time 02:30:00 Pending time
02:15:00 02:15:00
02:00:00 02:00:00
01:45:00 01:45:00
Time (HH:MM:SS)
Time (HH:MM:SS)
01:30:00 01:30:00
01:15:00 01:15:00
01:00:00 01:00:00
00:45:00 00:45:00
00:30:00 00:30:00
00:15:00 00:15:00
00:00:00 00:00:00
0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100
Job Number Job Number
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 37 / 39

44. Conclusion
Conclusion
Grid on Demand can offer Grid as a Cloud service
Parallel applications run/scale very well on Grid on Demand
Grid on Demand can be a solution for Urgent Computing demands
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 38 / 39

45. The End
Questions
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 39 / 39