2. How to use the cloud?
1. Get an account on the supported cloud
2. Start a master instance via the cloud web
console or CloudLaunch
3. Use CloudMan s web interface on the master
instance to manage the platform
3. • Launch an instance
• Demonstrate the following CloudMan
features and prepare for the data
analysis part:
Manual & Auto-scaling
Using an S3 bucket as a data source
Accessing an instance over ssh
Customizing an instance
Submitting an SGE job
Sharing-an-instance
•
•
•
•
•
•
Interaction flow
Agenda details
5. Launch an instance
1. Load biocloudcentral.org
2. Enter the access key and secret key
provided at http://bit.ly/bioworkshop
3. Type any name as the cluster name
4. Set any password (and remember it)
5. Keep Large instance type
6. Start your instance
Wait for the instance to start (~2-3 minutes)
For more details, see
wiki. galaxyproject.org/CloudMan
7. • Launch an instance ✓
• Demonstrate the following CloudMan
features and prepare for the data
analysis part:
Manual & Auto-scaling
Using an S3 bucket as a data source
Accessing an instance over ssh
Customizing an instance
Submitting an SGE job
Sharing-an-instance
•
•
•
•
•
•
Interaction flow
Workshop plan
8. Manual scaling
• Explicitly add 1 worker node to your cluster
• Node type
corresponds to
node processing
capacity
• Research use of
Spot instances
13. Accessing an instance over ssh
Use the terminal (or install Secure Shell for Chrome)
SSH using user ubuntu and the password you chose when
launching an instance:
[local machine]$ ssh ubuntu@<instance IP address>
14. Once logged in
• You have full system access to your instance,
including sudo; use it as any other system
• A suite of bioinformatics tools readily installed
• Can submit jobs to a job manager via the
standard qsub command
15. Distributed Job Manager
(DRM)
• Controls job execution on a (set of) resource(s)
• A job: an invocation of a program
• Manages job load
• Provides ability to monitor system and job status
• Popular DRMs: Sun Grid Engine (SGE), Portable
Batch Scheduler (PBS), TORQUE, Condor, Load
Sharing Facility (LSF)
16. Customize your instance install a new tool
$ cd /mnt/galaxy/export
$ wget http://heanet.dl.sourceforge.net/project/
dnaclust/parallel_release_3/dnaclust_linux_release3.zip
$ unzip dnaclust_linux_release3.zip
$ cd dnaclust_linux_release3
$ chmod +x *
Get a copy of a sample dataset
$ cp /mnt/workshop-data/mtDNA.fasta .
Don’t forget
17. Use the new tool in the cluster mode
1. Create a new sample shell file to run the tool; call it job_script.sh
with the following content:
#$ -cwd
./dnaclust -l -s 0.9 /mnt/workshop-data/mtDNA.fasta
2. Submit single job to SGE queue
qsub job_script.sh
3. Check the queue: qstat -f
4. Job output will be in the local directory in file job_script.sh.o#
5. Start a number of instances of the script:
qsub job_script.sh (*10)
watch qstat ‒f
1. See all jobs lined up
6. See auto-scaling in (using /cloud) [1.5-2 mins]
7. Go back to command prmopt, see jobs being distributed
18. Sharing-an-Instance
• Share the entire CloudMan platform
• Includes all of user data and even the
customizations
• Publish a self-contained
analysis
• Make a note of the
share-string and send
it to your neighbor
