The document summarizes the experience of building Thailand's national grid infrastructure, known as ThaiGrid. It describes ThaiGrid's goals of enabling better research through high-performance computing and grid technologies. The infrastructure connects computational resources from 21 clusters across 16 sites, totaling over 1,000 CPU cores. Notable projects supported include virtual screening of medicinal herbs for drug discovery. Challenges in building the grid included issues of scale, manageability, and software/hardware compatibility. Monitoring tools and a meta-scheduler help manage the distributed resources as a unified "super cluster".

1. Building a National Grid
Infrastructure: A ThaiGrid
Experience
PutchongUthayopas,
Department of Computer Engineering,
Faculty of Engineering,Kasetsart University, Thailand
pu@ku.ac.th

2. The Need for National E-
Infrastructure
• Efficient Resources
Management for Government
• Information dissemination for
people
• Emergency Response
– Natural Disaster: Flooding,
earth quake, Tsunami
• Better Quality of Live
– Environmental monitoring,
protection

3. National E-Infrastructure
• Fast computer
– massive computing power
• Large and fast storage for data
intensive applications
• Fast network
– Information sharing
– Collaboration
• Middleware
– Security, connectivity, access
• Portal
– Hiding the complexity of access,
data movement, and execution from
users
August 29,2008 TGCC 2008, Khon Khan University , Thailand

4. Introduction to ThaiGrid
• A National Project under Software
Industry Promotion Agency (Public
Organization) , Ministry of Information
and Communication Technology
• Started in 2005 from 14 member
organizations
• Expanded to 22 organizations in 2008
Objective
• Driving better research and
education using grid and HPC as
an enabling technology
• Making Thailand more
competitive by applying HPC and
Grid Technology
August 29,2008 TGCC 2008, Khon Khan University , Thailand

5. Thai Grid Infrastructure
19 sites
1 Gbps
About 1000 CPU core.
155 Mbps
2.5 Gbps
155
Mbps
August 29,2008 TGCC 2008, Khon Khan University , Thailand

6. ThaiGrid Usage
• ThaiGrid provides about 290
Digital
years of computing time for Animation
members Nano Life
science Science
– 9 years on the grid
– 280 years on tera
• 41 projects from 8 areas are ThaiGrid Geo
being support on Teraflop Physics
Science
machine Application
• More small projects on each
machines Computer
Chemistry
Science
CFD
August 29,2008 TGCC 2008, Khon Khan University , Thailand

7. Medicinal Herb Research
• Partner
– Cheminormetics Center, Kasetsart
Univesity (Chak Sangma and team)
• Objective
– Using 3D-molecular databse and virtual
screening to verify the traditional
medicinal herb
• Benefit
– Scientific proof of the ancient
traditional drug
– Benefit poor people that still rely on
the drug from medicinal herb
– Potential benefit for local
pharmaceutical industry Virtual
Screening
Infrastructure
Lab Test
August 29,2008 TGCC 2008, Khon Khan University , Thailand

8. NanoGrid
Computing Resources
Computing Resources
2 MS-Gateway
3
1
MS-Gateway
ThaiGrid
• Objective
– Platform that support computational Nano science
research
• Technology used
– AccelRys Materials Studio
– Cluster Scheduler: Sun Grid Engine and Torque
August 29,2008 TGCC 2008, Khon Khan University , Thailand

9. Building ThaiGrid

10. Challenges
• Size and Scale
• Manageability
– Deployment
– Configuration
– Operation
• Software and
Hardware
Compatibility

11. Principle Design Goal
• Openness
• Usability
• Functionality
• Performance

12. Some statistics about Thai Grid
Infrastructure
• 21 clusters from 16 sites around the country
• More than than ~1,000 cores
• A real super computer 4.6 Teraflops machines
– 200 nodes of 800 cores of Intel Xeon 3.0GHz
• ThaiGrid provides more than 111 years of computing
time for member
– 7 years on the grid and more than 104 years on tera
server
• More than 30-40 projects from 8 areas are being
support

13. Grid System Architecture
• Clusters
– Satellite Sets
• 16 clusters delivered from
ThaiGrid for initial members
• Composed of 5 nodes of IBM
eServerxSeries 336
– Intel Xeon 2.8Ghz (Dual
Processor)
– x86_64 architecture
– Memory: 4 GB (DDR2 SDRAM)
– Other sets
• Various type of servers and
number of nodes
• Provided by member institutes
of ThaiGrid

15. Grid Software Architecture
Grid Applications
Programming:
Management/
Grid Scheduler: Ninf-G
Monitoring:
GridWay MPICH-G2
SCMS
Globus WSRF
(Provide auto-configuration)
Grid Middleware:
Globus Toolkit 4.0
ThaiGrid Roll
Management/
Grid Scheduler: Programming:
Monitoring:
SGE, PBS, MPICH
SCMS,
Condor, SCMS OpenMPI
Ganglia
Cluster OS:
Linux / Rocks 4.0 – 4.3 on x86/x86_64 machines
UNINET (10Gbps)

16. Production CA Infrastructure
CA Operator RA Operator User
secure face-to-face
VTC or Skype
secure
Secure Web Online
CA RA
Interface Repository

17. Experiences
• Building a quick deployment technology is a key!
– NPACI Rocks is a key technology being used
– Collaboration SDSC/ TNGC under PRAGMA
– ThaiGrid Roll is built to facilitate the fast deployment
time
• Building a good infrastructure monitoring and
management system
– Know what your grid are doing.
– Develop system monitoring, accounting, testing tools

18. Grid Monitoring and Scheduling
• Using SCMSWeb to monitor and probe for Grid status
– Grid services probe is mainly used to categorize services
level in ThaiGrid
– Used in resource certification system
• Grid Meta Scheduler
– Meta scheduler is installed in central site
– GridWay 5.2 is installed and experiment in some projects
• Drug Design Grid
• Information is pulled from SCMSWeb

19. Grid as a Super Cluster
Middleware
Grid Scheduler • Gridway
• Globus 4.x
• GridWay 5.2.3
• SGE, PBSpro, Condor
GCC • SCMS
REN H
H
H
C C C C H C C C C
C C C C
C C C C
August 29,2008 TGCC 2008, Khon Khan University , Thailand

21. Experiences
• Strong standard compliance is
critical to make the systems
interoperable
• Site certification program
– Level 1: Network firewall
compliance
– Level 2: Middleware
compliance (MPI/G2, Ninf-g)
– Level 3: Job scheduling
compliance
– Level 4: Data Grid compliance

22. TERA Cluster
2.5Gbps to Uninet
Storage 48 TB
KU Fiber Backbone
• 1 Frontend (HP
ProLiant DL360 G5 (1Gbps Fiber)
Server) and 192 1 Gbps Ethernet/Fiber
computer nodes
– Intel Xeon 3.2
GHz (Dual core, Edge Switch 1Gbps Ethernet
Dual processor)
– Memory 4 GB
(8GB for
Frontend FE FE WinHPC TERA Anatta SPARE1 SPARE2
&infiniband Sunyata Araya (FE) (FE) (FE) (FE) (FE)
nodes)
– 70x4 GB SCSI
HDD (RAID1)
• 4 Storage Servers 96 nodes
– Lustre file 64 + 15
4 nodes 4 nodes
system for TERA nodes 16 spare nodes
cluster's storage nodes
– Attached with
Smart Array
P400i Controller
for 5TB space 200 Ports Gigabit Ethernet switch
Storage Tier 5TB Lustre FS
FS FS FS FS
1 2 3 4
August 29,2008 TGCC 2008, Khon Khan University , Thailand

23. Experience
• For a large user group, users will cluster them self
based on their application interest. Create a
divers set of requirement in term of Operating
system, Scheduler and End user application
– Single large cluster may not be used effectively
• Partition large cluster into several set of partition
that can be modified seems to work well.
– We have two set of infiniband for parallel job
– Unix and windows for single batch/interactive

24. Future
• The world are moving toward Cloud
– More cost effective
– Better manage through centralized control
– More elastic and on demand
• Thailand cover with cloud
– E-government cloud
– National Crisis Management Cloud
– Research Cloud
– Education Cloud
• Support mobile learning
• Nationwide broadband infrastructure is needed
– 4G/4G , Wifi, ADSL , Fiber to Home

25. Acknowledgement
• I would like to Thank you
– Dr. VaraVaravithya for being the co-founder of
the project and spent numerous hours helping
on laying out many technical detail.
– Dr. ArnonRungsawang and ThaiGrid staffs for
the hard work during the project to prove that
we can do it!
– All the university contact persons for helping
make our dream come true for a while
– Kasetsart University that allow me to do this for
our country.
– MICT, SIPA, NECTEC, NANOTECH, NSTDA for
their support
• Any honor and award should go to them for
their contributions . I can not do this alone!

26. Thank you