High Performance Computing (HPC)*Supercomputing*Beowulf Clusters*Scyld Cluster Mangement Software*IT Infrastrucutre*Linux*Open Source -- Penguin Computing Sr. Software Engineer Dr. Joseph Dunn presented at the Penguin Computing booth theater at SC18 how best to manage a cluster in order to meet the most demanding computing requirements that can help maximize HPC potential.

1. Pro Tips:
Best Practices in Cluster Management
Dr. Joseph Dunn, Sr. Software Engineer, Penguin Computing
Penguin Computing Theater SC18

2. © 2018 Penguin Computing - Confidential
Measure Twice, Cut Once
2
Plan the Cluster You Need
▪ Don't boil the ocean with a myriad of
node classes in a small cluster
▪ Find the "sweet spot" configuration
that makes most users happy
▪ Put specialized needs in the cloud

3. © 2018 Penguin Computing - Confidential
Simplify Management
3
Make it easy to reproduce, backup, and expand management services.
▪ Automate the building of the VMs
(ansible, puppet, etc.)
Automation frameworks provide
documentation in the form of
deployment template.
▪ Employ VMs for servers that do not
require bare-metal hardware
Change management for license
servers, head nodes, scheduling
servers, and login nodes is easier on
a VM.

4. © 2018 Penguin Computing
Organize Your Applications
4
Organizing applications and crucial libraries into stand-alone
folders or containers on a shared file system simplifies change
management.
▪ Upgrade without "stepping on" older
installations.
▪ Conduct OS upgrades without fear
that Perl, Python, or system library
changes will affect applications
▪ Treat critical libraries like
applications (e.g. ATLAS, FFTW)
▪ Treat compilers like applications for
optimal results (GCC, Perl, Python)
▪ Simplifies application backups

5. © 2018 Penguin Computing
Organizing Through Unique --prefix
5
▪ Organize into --prefix defined folders. This enables installing
upgrades without "stepping on" a previous installation.
/apps/OpenFoam/{2.3.1,4.1,5.0}/intel_{2017,2018}
▪ Leverage Modules or LMOD to enable users to switch between
versions.
module load openfoam/2.3.1/intel_2018

6. © 2018 Penguin Computing
Do I Containerize?
6
Containerization requires steps to
design and QA. The effort results in
highly portable, reproducible
workloads.
Ideal if the application is business
critical, or you plan to migrate between
various platforms or clouds.

7. © 2018 Penguin Computing
High Availability
7
Things break, so focus HA spending where it matters.
▪ Focus the HA budget on business
impacting services: storage and core
switching.
▪ Buy spare parts to easily repair
failures.
▪ Obscure parts (e.g. switch PSUs) can
be costly to repair years down the line.
Invest at purchase time.
▪ Plan for failures.
▪ Larger clusters are more resilient
to partial failures and require less
HA at edge networks.
▪ Use an intelligent scheduler that
can run health checks and
schedule jobs around failures.

8. © 2018 Penguin Computing
Continuously Monitor
8
▪ Invest in monitoring and alerting
infrastructure.
▪ Keep track of cluster performance
to catch partially degraded
systems.
▪ Benchmark at purchase, then
benchmark using excess capacity
throughout the life of the cluster.
Monitor health, not just uptime.

9. © 2018 Penguin Computing
Use Commercially Supported Software
9
▪ Hand off 3rd-party driver integration
to a commercial vendor.
▪ Design the environment to rapidly
segment parts of your cluster for
upgrades and testing.
▪ Commercial OS distros provide
faster security updates.
▪ Invest in commercial support for
technologies where there are
knowledge gaps.
Supported software allows admins to focus on the interesting problems.

10. © 2018 Penguin Computing 10
To learn more, go to:
www.penguincomputing.com/products/software/scyld-
clusterware/
And get a 45-day FREE trial of Scyld ClusterWare11!
Want More Info?
PENGUIN COMPUTING
@PenguinHPC
PENGUIN COMPUTING
PENGUIN COMPUTING

11. www.penguincomputing.com
1-888-PENGUIN