1. HP Insight CMU
Cluster Management Utility Tour
Sébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant
7th HPC Day conference, Kiev, Ukraine
October 2012
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

2. Agenda
 HP Insight CMU Introduction & Review
 Introduction, History / Customers
 Product mindset
 Insight CMU v7.0 tour
 Provisioning (Cloning / Autoinstall / Diskless)
 Monitoring ( TimeView / Collectl / GPGPUs…)
 Scalable/Frictionless administration ( cmudiff…)
 Custom GUI & partners integration
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

3. Insight CMU
Introduction & History
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

4. Insight CMU introduction
CMU = Cluster Management Utility
‚CMU optimizes the TCO of compute farms‛
 CMU scaling specification: 4k nodes
 CMU has lots of industrial clusters in production with 2k/3k+ nodes
 CMU has a strong presence in the TOP500 (www.top500.org)
 CMU at customer site since 2000
4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

5. Insight CMU major milestones
2000: initial implementation for Tru64 Unix (Alphaserver)‫‏‬
2001: port to Alpha Linux, 1600 servers commercial cluster
2002: port to x86 & IA64 Linux / HPUX Itanium
2004: port to x86_64 Linux. (only port maintained*)
2007: Swedish gov, 6th @ TOP500
2010: Tsubame 2, HP first public 1+ PFlop cluster, 5th @ TOP500
*ARM port in progress...
5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

6. Worldwide CMU Deployments
HP ships >2 CMU clusters per week WW
UNIVERSITIES ENGINEERING
GOVERNMENT and RESEARCH LABS ENERGY
6
6
April 2009
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

7. Insight CMU project mindset
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

8. Insight CMU project mindset
CMU provides the core functionalities for a farm/cloud
 runs any HP* server (even mix) / any Linux distribution (even mix)‫‏‬
 independent of many architectural aspects of the system:
interconnects / GPGPUs / IO accelerators...
network topology (open cluster, guarded cluster, WAN…)
batch/job schedulers, MPI stacks, math libraries, compilers....
CMU is not a ‘predefined’ SW supercomputer appliance
 >90% systems delivered as ‚turn-key solutions‛
 CMU can also be purchased standalone with support and manuals
8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

9. Insight CMU v7.0 tour
 CMU functionalities / typical CMU implementation
 CMU Provisioning
 CMU Monitoring
 CMU Scalable / Frictionless administration
 CMU Custom GUI & partners integration
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

10. CMU starts here: typical { farm / HPC cluster } implementation
{ high speed Interconnect }
{ Highly Avail.}Head node
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

11. Insight Cluster Management Utility Basics
 CMU is a single package running on the head node (upgrade is trivial)
CMU mgt node can be an HA cluster (HP service guard,Redhat Cluster, SLES HA)
 Provides a full fledged interactive CLI
 Provides cmu_* commands as an API (for scripting)
For integration with other software or command-line activity (see partner’s integration)
 Provides GUI client for single dashboard control
launch from a web page served from head node (JAVA© webstart technology)
run on a local laptop/desktop
user mode for monitoring
admin mode for remote administrationcontained herein is subject to change without notice.
11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information

12. The three pillars of HP Insight CMU
Provision Monitor Control
• Simplified discovery, • ‘At a glance’ view of • GUI & CLI & API
firmware audits entire system / • Easy GUI, friction-
• Fast & scalable partition less control of
cloning • Customizable remote servers
• Legacy support of • Lightweight • Scalable pdsh with
Kickstart/Autoyast/D • Instant 2D view cmudiff analyser
ebian Preseed • TimeView, 3D live
• Diskless support history
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

13. Insight CMU v7.0 Tour
 CMU Provisioning
 Scalable Cloning
 Legacy/Compatible Autoinstall (Kickstart/Autoyast/Preseed)
 Diskless
 Firmware audit
 Bare metal netboot low level tools (hpacucli / hponcf / ipmitool)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

14. CMU provisioning engine
 backup/cloning (up to 4k nodes)‫‏‬
 RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only compute nodes)‫‏‬
 performance only depends on the image size & harddrive speed
 no architectural dependence on trunked/IB/10gig networks, 1Gig ethernet is sufficient.
 22 minutes to clone 1000 nodes with SAS drives and 10 GB image
 continuous cloning for reprovisionning from batch schedulers
 autoinstall ‫ ‏‬CMU bridge to legacy/standard tools
:
 Redhat Kickstart / SLES Autoyast / Debian Preseed
 do not use above 100 nodes
 diskless (advised if improving the density of the solution and/or data security)
 statefull diskless engine (hybrid NFS ro + rw personalities)
14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

15. CMU firmware / bare metal netboot tools
 firmware hooks:
 firmware version checks (HP conrep based currently / HP rcu soon)
 firmware settings audit (HP conrep + cmudiff)
 firmware flashing engine (to feed with SCEXE HP files)
 bare metal netboot tools (available from ‚pre_reconf ‚/ ‚reconf‛ )
 hpacucli : configure HP smartarray controllers
 locfg.pl/ hponcfg: configure HP ILO from the CMU netboot environment
 ipmitools: configure IPMI capable BMC from the CMU netboot environment
19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

16. Insight CMU v7.0 Tour
 CMU Monitoring
 scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)
 2D Instant View / 3D Time View (Live History)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

17. CMU monitoring
 Backend: ‚HPC aware‛ monitoring since years
 Scalable monitoring ( proven on 4k nodes system )
 Non intrusive (leverage collectl + ‚HPC synchro‛ mode)
 Programmable (monitor anything you can script )
 Nvidia & AMD GPGPUs monitoring tool
 Extended Monitoring to inject arbitrary monitoring data
 Alerting system & CMU Reactions
 Frontend GUI (JAVA client/server) / CLI
 GUI: Instant view 2D / TimeView 3D (Live History)
 cmu_dynamic_user groups (see later in presentation)
21 CLI/API: cmu_monstat and flat human readable files
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

18. ‚Instant View*‛ CMU Display
* renamed « Instant View » since CMU v7.0
22 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

19. 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

20. TimeView (Live History)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

21. Existing ‘well known’ CMU Display since 2004
25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

22. 3D Display of Sensor Histories
readability, efficiency, precision
26
2
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

23. 3D Display of Sensor Histories
global job overview
27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

24. 28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

25. 29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

26. 30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

27. 31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

28. GPGPUs monitoring
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

29. CMU GPGPU Support
CMU provides a tool for extracting GPU metric data
from GPU driver
‚cmu_get_nvidia_gpu‛ monitors:
load, mem_util, mem_alloc, power_state, and ECC_double_bit
alerts by default
Power_usage, various clock speeds, fan speeds, and
temperature also configured but commented out by default
33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

30. 37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
F
3ooter goes here

31. Extended monitoring
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

32. CMU Extended Monitoring
Inject monitoring data from another source into CMU
Extended metrics will be used for:
Server hardware metrics (ILO4 out-of-band & agentless monitoring)
• Temperatures, fan speeds, power usage
• Gathered out-of-band
• OS-neutral
Cluster peripherals
• MCS temperatures, switch status
Workload schedulers
39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

33. CMU alerts & reactions
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

34. CMU alerts & CMU reactions
CMU monitoring engine can trigger alerts
CMU alerts can trigger scripts as reactions to alerts
reaction examples:
• SNMP traps (send all alerts to an SNMP sink such as HPSIM)
• Send an email
• Remove a compute node from a batch scheduler…
44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

35. Insight CMU v7.0 Tour
 Scalable / Frictionless monitoring
 Interactive command broadcast (ssh, BMC interfaces)
 cmudiff non interactive scalable command output analyzer
 GUI accelerators (power off / UID leds/ three clicks ‘en masse’ cloning….)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

36. Insight CMU GUI basics
Cluster mgmt
panel
displays all
nodes in
selected
groupings: by
switch
location; by
image; or by
custom
grouping
node states
display
current state
of each node CMU Main Alerts displayed
47
Display Panel
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
along the bottom

37. CMU GUI Basics
– Right-click to select sensors to
display
– CMU pre-configured with standard
sensors: CPU and memory usage,
and disk and network I/O
– Simple to add any sensor or alert
– CMU provides simple support for
monitoring GPU temp and ECC
errors
– Three clicks to clone compute nodes
!
48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

38. Friction-less remote control of target nodes
Selected Power
nodes Broadcast
commands
commands
Provisioning
commands
User-defined
commands
49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

39. CMU remote management commands
• Multi-window broadcast command (access OS or console)
51
type here…
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
...and see it there

40. cmudiff
scaling the command line.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

41. Compare node outputs with Scalable Text Analyser (cmudiff)
Single-window pdsh with
cmu_diff example
One command
executed across a
set of selected
nodes…
…finds one node
running with an old
BIOS version!
53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

42. dshbak vs cmudiff: round #1…. ‘date’ on five hosts
cmudiff
dshbak
57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

43. dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts
cmudiff
dshbak
58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
58 HP Confidential

44. Partners software integration
& Custom menu GUI
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

45. 62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

46. 63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

47. CMU Custom Menu Support
/opt/cmu/etc/cmu_custom_menu
64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

48. cmu_dynamic_user_groups
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

49. Insight CMU as a (job) power monitor
70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

50. Insight CMU Partner Integrations
 Moab – Dynamic Provisioning
 PBS Pro – Green Scheduling & OS Provisioning
 LSF – Platform HPC
 ScaleMP – create large virtual SMP nodes
 StackIQ – CMU part of HP ‚roll‛
 HP Matrix CMU CloudMap
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

51. Thank you for your interest in
HP Insight CMU
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.