2. CI: Helping Push Research Boundaries
• Developing CI: A one step process?
– “If we build it, will they come?” “Will it be usable?”
• Interplay of (sustainable, longterm, and broadly
usable) CI and Research more complex
– Research & Applications requirements inform the
development of CI
– In response, developed CI “roughly” sets the
boundaries of applications and their usage mode
– Novel applications and usage modes that can
exploit CI will push the boundaries of research...
2
3. Outline
• Scientific Grid Applications
• Computing Free Energies in Biological Systems
– STIMD (200304), SPICE (200506)
• Challenges of Distributed Environments
• HARC: A tool for coallocating resources
– GENIUS: GridEnabled Neurosurgical Imaging
Using Simulations (200708)
• Simple API for Grid Applications (SAGA)
• Regional CI Example Louisiana
• Software: Cactus, HARC, Petashare, SAGA...
• People: LONI Institute and NSF Cybertools
• Novel eScience Applications
3
4.
Source: NSF report on Cyberinfrastructure for Biology
5. Computing Free Energies: Motivation
Cellullar messangers
e.g. Growth Factors,
Thermodynamic quantity of maximum
cytokines significance.
Transmembrane
receptor Characterizes the binding accurately:
Recognition of
Inhibit specific protein domains
activated receptor by Cell signalling events
Intelligent drug design ...
SH2 domain
Rapid & accurate determination is
critical:
Gene switched where FE difference maybe just
one part of the overall “system”
library of ligands to explore
6. Scientific Grid Computing: An Exemplar
• Computing FE is computationally very expensive. Balance
between accuracy and rapid determination. Some
experimental timescales are 23 days.
• Algorithmic advances (e.g., O(N logN)) have helped; but
more than just algorithmic advances are required.
• Computational ‘Grid’ Science: Which approaches can be
adapted to exploit gridbased computation? Interplay of
physical algorithm(s) and grid architecture(s).
7. Computing a Free Energy Difference Using
Thermodynamic Integration (TI)
Src SH2 domain
λ=0 λ=1
∆G1
∆GB
∆GA
ligand
Free energy of binding of the ligand to the
∆G2
larger protein characterises the strength of the
binding.
TI provides a formalism to compute the difference of free energy of binding (GAB) between
two somewhat similar, yet different peptides. The key concept in TI is that of a
thermodynamic cycle varying the value of lambda from 0 (peptide A) to 1 (peptide B).
∆ ∆GAB = ∆GB ∆GA
= ∆G1 ∆G2
8. TI Calculation: Modified Workflow
H
λ Launch initial job, use real time analysis to
determine when to spawn next simulation
Starting Check for convergence t at a new λ value. Spawning simulation
conformation
continues until sufficient data collected.
Need to control several jobs.
λ=0.10 time
H
λ=0.25 λ
λ=0.33
In general, use realtime analysis to dynamically
determine best next value of lambda. λ
…
λ=0.9 Combine and calculate data from all runs to
compute the integral to get ∆∆GAB.
9. Infrastructure (Software) Developed for Application
Steering Library: Correct functional abstraction from app.
perspective.
steering_control(), register_params(), emit_data()
Library determines destination, transport mechanism, fopen,
fwrite Architecture of a steered application.
Details of infrastructure & middleware (layers) hidden from app
10. Extensions of the Distributed TI Concept
Computational Techniques
Replica Exchange Methods
Need for “intelligent” infrastructure to be coupled with analysis method
Ensemble MD
– Simulate each system many times from same starting position
– Allows conformational sampling. Can’t say how much a priori.
Start Conformation Series of Runs End Conformations
C1
Cx C2
C3
Equilibration Protocols
eq1 eq2 eq3 eq4 eq5 eq6 eq7 eq8 C4
11. RNA Translocation Through Protein Pores
Molecular Biology: Critical and
ubiquitous process. A model for:
Gene expression in eukaryotic cells
Viral infections rely on import of viral
genome via nuclear pore
Translocation across bacterial
membrane during phage infection
Technical Applications: Artificial pores
(similar to natural pore) for high
throughput DNA screening
Theoretical Physics: Long standing
problem of a semiflexible polymer
motion in a confined geometry
12. Simulated Pore Interactive Computing Environment
Size, complexity & timescale: Computations expensive.
Millions of CPU hours using “vanilla” MD. Not good enough.
Free Energy Profile: Extremely challenging but yields
maximal insight and understanding of translocation process.
Novel Algorithm: Steered Molecular Dynamics to “pull DNA
through the pore”. Jarzynksi's Equation to compute
equilibrium free energy profile from such nonequilibrium
pulling.
e ß∆F = ‹ e ßW ›
13. Grid Computing Using Novel Algorithms
SMD+JE: Need to determine
“optimal” parameters before
simulations at the optimal values.
Requires: Interactive simulations
and distributing many large,
parallel simulations
Interactive “Live coupling”: use
visualization to steer simulation
Reduces computational cost by a factor of ca. 100.
Solve a computationally “intractable” problem using novel
algorithm. Our solution not only exploits grid infrastructure, but
requires it.
15. Grid Computing, Interactivity and Analysis
• Interactive simulations used to determine:
Optimal value of forceconstant & pulling
velocity, choice of subtrajectory length and
location for optimal value simulations
• Use visualization to provide input to the
running simulation. Require 256px (or more)
of HPC for interactivity. Steadystate data
stream (up & down)
Interactive simulations perform better when using optical
lightpaths between simulation and visualization
Due to network characteristics. Typical “legacy” app (NAMD) not written
for network I/O. “Unreliable” transfer can stall simulations.
16. “Global” Grid Infrastructure UK NGS
NGS
HPCx
Leeds
US TeraGrid
Starlight (Chicago) Manchester
Netherlight
(Amsterdam)
SDSC Oxford
NCSA
PSC RAL
UKLight
App Sci DEISA
All sites connected by
production network
Computation Network PoP Visualization
17. Recap: FE Exemplars
Both FE Algorithms are good candidates for distributed
resource utilization
– i.e., “pleasantly” distributable
Similar Infrastructure
– Software (ReG Steering Services), middleware..
– Federated Grids
SPICE more complex than STIMD:
– Complexity of tasks different
– Needs coscheduling of heterogenous resources
– number of components/degreeoffreedom different
18. VORTONICS: Vortex Dynamics
on Transatlantic Federated Grids
USUK TGNGS Joint Projects Supported by NSF,
EPSRC, and TeraGrid
Computational challenge: Enormous problem sizes, memory
requirements, and long run times: Largest runs require
geographically distributed domain decomposition (GD3)
19. Run Sizes to Date / Performance
• Using an early version of MPICHG2, 3D lattice
sizes up to 6453 across six sites on TG/NGS
• NCSA, SDSC, ANL, TACC, PSC, CSAR (UK)
• Amount of data injected into network.
Strongly bandwidth limited.
• Effective SUPS/processor
• Reduced by factor approximately equal to 1 2 3 4 sites
number of sites
• Therefore SUPS approximately constant as
problem grows in size sites kSUPS/Proc
– If too large to fit onto one machine, GD3 over 1 600
N resources simultaneously is no worse than
2 300
N sequential runs
4 149
6 75
20. Outline
• Scientific Grid Applications
• Computing Free Energies in Biological Systems
– STIMD (200304), SPICE (200506)
• Challenges of Distributed Environments
• HARC: A tool for coallocating resources
– GENIUS: GridEnabled Neurosurgical Imaging
Using Simulations (200708)
• Simple API for Grid Applications (SAGA)
• Regional CI Example
• Software: Cactus, HARC, Petashare, SAGA...
• People: LONI Institute and NSF Cybertools
• Novel eScience Applications
20
21. Challenges of Distributed Environments
Lessons learnt from Pilot Projects
• Hiding the Heterogeneity; providing uniformity
We interface application code to grid middleware through well
defined userlevel APIs. No code refactoring required.
Hides heterogeneity of software stack and sitespecific details:
Vortonics: MPICHG2 hides lowlevel details (communication,
networktopology, resourceallocation and management)
SPICE RealityGrid steering library
• Need for usable, stable and extensible infrastructure
Infrastructure relatively “easy” for demo(s); difficult for routine use!
Science requires stable & persistent infrastructure
Motivation for SAGA Efforts at OGF
23. Challenges of Distributed Environments
Federated Grids
• Current barrier to utilise federated grids still high:
Many degreesoffreedom need coordination
Collective InterGrid Debugging required
• Federated Grids must be interoperable in practice:
Stress test using real applications Requires additional “user level
middleware” (MPICHG2, ReG steering infrastructure), to work
across grids
• Paper on the theory, implementation and experiences of the three
joint projects: (CLADE 2006, Sp. Issue Cluster Comp)
http://www.realitygrid.org/publications/triprojects_clade_final.pdf
•Application level Interoperability; Influenced the creation of GIN
26. How does HARC Work?
• Client makes request, from
command line, or other tool via
Client API
• Request goes to the HARC
Acceptors, which manage the co
allocation
• The Acceptors talk to individual
Resource Managers which make
the individual reservations by
talking to the local schedulers
27. HARC is Extensible
(Community Model)
• Modular Design throughout
– Not just compute resources. New resource types can be
added, then coallocated with all other types of resource
– No modification to Acceptors is needed. Just provide
Resource Manager code to schedule the resource
– And extend the Client API with new classes (again, no mods to
existing code)
– Even works from the command line
• Example: Network Resource Manager
$ harc-reserve -n EnLIGHTened/RA1-BTH
-c bluedawg.loni.org/8 -s 12:00 -d 1:00
– Coallocates a lightpath between LSU & MCNC, with 8
processors on bluedawg...
– Was used to schedule lightpaths in EnLIGHTened testbed for
Thomas Sterling’s HPC Class, broadcast in Highdef video
28. GENIUS: Overview
PI: Coveney, UCL
Goals:
Provide a better understanding of
cerebral fluid flow,
Inform clinicians of best surgical approaches.
Approach:
Model large scale patient specific cerebral blood flow within clinically
relevant time frames
Provides:
Reliable & effective patientspecific imagebased models
Efficient LB blood flow simulation
Real time blood flow volume rendering – Visualisation
29. HemeLB fluid flow solver
A fast fluid flow simulation of a very large system
requires the use of an efficient parallel fluid solver
several processors
• LatticeBoltzmann method; Parallel MPI code,
• Efficient algorithms for sparse geometries,
• Topologyaware graph growing partitioning technique,
• Optimized inter and intramachine communication
patterns,
• Full checkpoint capabilities...
32. NGS UK NGS
US/UK Grid Infrastructure HPCx
Leeds
Manchester
TeraGrid Oxford
RAL
LONI
LSU HSCTech
La
ULM
NSU
The GENIUS project makes use of Alex
SLU
SU LSU
infrastructure provided by LONI,
TeraGrid and NGS, connected by ULL UNO
McNeese LSU HSC
dedicated switched optical light paths Tulane
33. Using HARC...
• Our aim is to get HARC available to users as part of
the basic Grid infrastructure
• Current deployments
– LONI (Louisiana Optical Network Initiative)
• production mode
– UK NGS, Manchester, Leeds and Oxford NGS2
– TeraGrid Coscheduling testbed machines
(SDSC/NCSA IA64)
– NWGrid (Lancaster, Manchester)
• Everything is open source too
• See:
– http://www.cct.lsu.edu/~maclaren/HARC/
34. Rough Taxonomy of Applications
• Some applications are Gridunaware and want to remain so
– Use tools/environments (e.g, NanoHub, GridChem)
– May run on Gridaware/Gridenabled environments (e.g.
Condor) or programming environment (e.g, MPICHG2)
• Some applications are explicitly Gridaware
– Control, Interact & Exploit distributed systems at the
application level
34
35. SAGA: In a Nutshell
• A lack of:
• Programming interface that provides common grid
functionality with the correct level of abstractions?
• Ability to hide underlying complexities, varying semantics,
heterogenities and changes from application program(er)
• Simple, integrated, stable, uniform, highlevel interface
• Simplicity: Restricted in scope, 80/20
• Measure(s) of success:
• Does SAGA enable quick development of “new”
distributed applications?
• Does it enable greater functionality using less code?
37. SAGA Example: Copy a File
Highlevel, uniform
#include <string>
#include <saga/saga.hpp>
void copy_file(std::string source_url, std::string target_url)
{
try {
saga::file f(source_url);
f.copy(target_url);
}
catch (saga::exception const &e) {
std::cerr << e.what() << std::endl;
}
}
• Provides the high level abstraction, that application programmers
need; will work across different systems
• Shields gory details of lowerlevel m/w system
• Like MapReduce – leave details of distribution etc. out
10/22/2006 LCSD'06 37
38. SAGA: Scope
• Is:
– Simple API for GridAware Applications
• Deal with distributed infrastructure explicitly
– Highlevel (= applicationlevel) abstraction
– An uniform interface to different middleware(s)
– Clientside software
• Is NOT:
– Middleware
– A service management interface!
– Does not hide the resources remote files, job (but
the details)
39. SAGA API: Towards a Standard
• The need for a standard programming interface
– “Go it alone” versus “Community” model
– Reinventing the wheel again, yet again, and again
– MPI as a useful analogy of community standard
– OGF the natural choice; establish SAGARG
• “Tedium” of the standardisation process?
– Not all technology needs to be standardised upfront
– Standardisation not a guarantee to success
• Requirements Document
– Quick skim through the Requirements document re
– Design and requirements derived from 23 Use Cases
– Different projects, applications and functionality
45. GridSAT
First Principles Grid Application
• Grid implementation of the
satisfiability problem: To
determine if the variables of
given Boolean formula can
be assigned such as to
make it TRUE.
• Adaptive: computation to
communication ratio
need/can be adjustable (!)
• Allows new domain science
– beats zChaff (time taken
and problem) Adapted from slides by Wolski & Chakrab
46. GridSAT Characteristics
• Parallel, distributed SAT solver
– Both CPU and Memory Intensive
– Splitting leads to better performance
– Allows sharing: clause learned in solver shared
• Grid Aware Application:
– Heterogenous (single, clusters & supercomputers)
– Dynamical Resource Usage
• Unpredictable runtime behaviour
– How much time? How many resources? When
to split? Which process splits first?
– Problems vary: easy to hard, short to long
– Need to be adaptive, “add resources as you go”
47. GridSAT: Programming Requirements
• RPC, Dynamic resource & Job
Error Handling, scheduling and
management checkpointing
SAGA provides the required programming functionality, at the
correct level of abstraction and thus makes it easier to
manage, deploy and extend (for new functionality) GridSAT
49. Replica Exchange Algorithm
• Create replicas of initial R1
configuration
• Spawn 'N' replicas over R2
different machine
R3
• Run for time t
RN
• Attempt configuration
swap of Ri <> Rj
• Run for further time t hot
• ... T
• Repeat till termination
t Exchange attempts 300K
t
50. RE: Programming Requirements
RE can be implemented using following “primitives”
• Read job description
– # of processors, replicas, determine resources
• Submit jobs
– Move files, job launch
• Access simulation data & analysis
• Checkpoint and relaunch simulations
– Exchange, RPC (to swap or not)
Implement above using “grid primitives” provided by SAGA
Separated “distributed” logic from “simulation” logic
Independent of underlying code/engine
Science kernel is independent of details of distributed
resource management
Desktop akin to Highend supercomputer!!
51. Programming Distributed Applications
Parallel Programming Analogy
Status of distributed programming today, (somewhat) similar
to parallel programming preMPI days
MPI was a “success” in that it helped many new applications
– MPI was simple
– MPI was a standard (stable and portable code!)
• SAGA is to the grid application developer, what MPI is to the
parallel program developer (“gridprimitives”)
• SAGA conception & trajectory similar to MPI
– SAGA is simple to use
– OGF specification; on path to becoming a standard
•Therefore, SAGA's Measure(s) of success:
• Does SAGA enable “new” grid applications?
52. Outline
• Scientific Grid Applications
• Computing Free Energies in Biological Systems
– STIMD (200304), SPICE (200506)
• Challenges of Distributed Environments
• HARC: A tool for coallocating resources
– GENIUS: GridEnabled Neurosurgical Imaging Using
Simulations (200708)
• Simple API for Grid Applications (SAGA)
• Regional CI Example: LONI (Now part of TeraGrid!)
• Hardware: Compute + Network
• Software: Cactus, HARC, Petashare, SAGA...
• People: LONI Institute and NSF Cybertools
• Novel eScience Applications 52
53. 3 Axes:
LONI
CyberTools
LONI Institute
~ 100TF IBM, Dell
LA Tech Supercomputers
National Lambda Rail
LSU
SUBR
UNO
ULL Tulane
58. • Goal: Enable underlying infrastructure to manage the
lowlevel data handling issues.
• Novel approach: treat data storage resources and the
tasks related to data access as first class entities just like
computational resources and compute tasks.
• Key technologies being developed: dataaware storage
systems, dataaware schedulers (i.e. Stork), and cross
domain metadata scheme.
• PetaShare exploits 40 Gb/sec LONI connections
between 5 LA Universities : LSU, LaTech, Tulane, ULL &
UNO
Cybertools: Not just compute!
PI: Tevfik Kosar (CCT/LSU)
59. Participating institutions in the PetaShare project, connected
through LONI. Sample research of the participating researchers
pictured (i.e. biomechanics by Kodiyalam & Wischusen, tangible
interaction by Ullmer, coastal studies by Walker, and molecular
biology by Bishop).
High Energy Physics
Biomedical Data Mining
LaTech
Coastal Modeling
Petroleum Engineering
LSU Computational Fluid Dynamics
Synchrotron X-ray Microtomography
UNO
Biophysics
ULL Tulane
Geology Molecular Biology
Petroleum Engineering Computational Cardiac Electrophysiology
60. LONI Institute
• Build on LONI infrastructure, create bold new inter
university superstructure
– New faculty, staff, students; train others. Focus on CS, Bio,
Materials, but all disciplines impacted
– Promote collaborative research at interfaces for innovation
– Much stronger recruiting opportunities for all institutions
• Two new faculty at each institution (12 total)
– Six each in CS, Comp. Bio/Materials with half PKSFI matching;
fully covered after five years
• Six Computational Scientists
– Support 7090 projects over five years; lead to external funding
• Graduate students
– 36 new students funded, trained; two years each
62. Resource Performance Monitoring Application
• NWS, BQP – only 1 resource at a time!!
• jfdja;
• How to choose M resources out of N ?
e.g. MPICHG2 Application, which M
• Cactus + SAGA + LONI (Lightpaths)
62
CENTER FOR COMPUTATION & TECHNOLOGY AT LOUISIANA STATE UNIVERSITY