PEARC17: Building bridges - The System Administration Tools and Techniques Used to Deploy Bridges

Building Bridges
The System Administration Tools and
Techniques Used to Deploy Bridges
Richard Underwood
HPC System Administrator
richardu@psc.edu

2
© 2010 Pittsburgh Supercomputing Center
What is Bridges?
• NSF Funded
– NSF Award #1445606
– Omni Path Connected
– 908 Nodes
• 752 Compute
• 48 GPU
• 42 3TB Large Memory
• 4 12TB Large Memory
• 42 Administration and Application Servers
• 20 Storage Servers

3
What is Bridges? (cont)
• Focuses on new user communities
– Groups and disciplines new to supercomputing
• Wide variety of computing programs and paradigms
– MPI, Hadoop, Virtual Machines, science gateways, etc
• Bridges Virtual Tour
– https://www.psc.edu/bridges-virtual-tour

4
Scheduling - Slurm
• Simple Linux Utility for Resource Management
• Replaces PBS as the standard for many large installations
these days
– XSEDE especially
• Topology Aware
– Bridges has islands of high connectivity with weak overall
connectivity
• similar command line tools to PBS (qstat vs sinfo)
• Gives QoS tools, that allow us to push things like anti-
stuffing, and debugging queues

5
Booting - Openstack Ironic
• Openstack Ironic is Openstack’s utility for booting physical
hardware using openstack’s own images
• Booting all Bridges nodes(except 3 bootstrap nodes, and
the 12TB nodes) is done through Ironic.
• Benefits
– Fast deployment
– Heterogeneity
– Fast redeployment
• Shortcomings
– We were one of the first large deployments at this scale
– PXE boot only (just recently got local disk boot to work)

6
Monitoring and Reporting Workflow

7
Configuration Management

8
Puppet – Configuration Management
• Flexible configuration management tool
• Bridges uses puppet to configure and customize most
things on Bridges
– packages, Omni Path, file system mounts, swap, utilities (NTP,
Duo), Naemon
• Handles a plethora of machine types
– Administration, compute (3 flavors), GPU, Openstack hypervisors,
storage nodes, data transfer among others
– Done through node type inheritance, and overrides
• Allows for quick rebuilds and quick repurposing
• Puppet is extended by a number of tools: puppetdb,
puppetboard, mcollective

9
PuppetDB
• Puts a database backend behind puppet, to keep reports on
state of the machines under its purview.
• Fully searchable by RESTful interface
– Everything from reports to facts(environmental variables) about
each system
• Allows for external resources (a way to pool resources to
one host)
– ssh keys from each host to a specific master host
– monitoring instructions (Nagios/Naemon)

10
Puppetboard
• Web frontend for puppetdb
– Enterprise puppet has this but is not free
• Takes puppetdb and makes it easily viewable
• Full searching for facts, reports
• Easy to see how machines are reacting to puppet
• Only status cannot change from the website

11
Puppetboard – Cont.

12
mCollective
• Marionette Collective
• Controls puppet agents from a mco command line tool
• Uses middleware layer – activemq
• Allows you to update multiple hosts at once, to push out an
important update
• mco extensible - nrpe, rpc, services, facts

13
Monitoring

14
Monitoring - Historical
• Monitoring at PSC
– Done with Mon in the Past
• Easy to configure
• Works with pagers
• Doesn’t break down
• Not good for large systems with lots of parts
• Nagios
– Standard monitoring tool
– highly extensible
– harder to configure (than mon)
– scales decently (can slow down with really large instances)
– Community version no longer being developed
– Can be automatically configured with Puppet / Puppetdb

15
Monitoring - Naemon
• Fork of Nagios
– Developers and users of Nagios
– forked after development of Nagios XI stopped development on
community edition
• Similar to mysql -> mariadb
– Many different forks (ichinga, centreon, shinken)
• Benefits we liked
– input engine parallelized
– enhanced grouping
– drop in replacement for nagios
– LiveStatus database-like interface to current and historical data
• Allows for complicated node up introspect scripts

16
Puppet – Naemon Integration
• Puppet defines naemon rules for each host
• When puppet runs on a host, these rules are loaded into
puppetdb (External Resources)
• When the naemon server runs puppet, these rules are
combined and if there’s a change in the rules, naemon
restarts
• Allows for automatic deployment of naemon rules through
puppet.

17
Naemon – Checking In through Passive Checks
• Lets the host check in with the naemon master instead of
naemon directing the check
• Useful in cases where the host under test may be load
• Allows hosts to check in at their leisure
• In the past this was done through NCSA
– Nagios Service Check Acceptor
– Not good for parallelization
– Not maintained
• New method NRDP - Nagios Remote Data Protocol
– Uses secure apache to send the code RESTfully
– multiple formats - including JSON and XML
– easier to maintain

18
Monitoring – InfluxDB
• Tool for long term state data retention and visualization
– Lets you make pretty graphs
• Most long term state data tools use RRDs
– Round Robin Databases
– great for looking at trends
– Do not save all data over time
• it pares down to averages over time
• Originally we used OpenTSDB
– did not scale well to a large cluster or even a small cluster for that
matter

19
InfluxDB - Cont
• Hosts locally run a client that collects the data and push it
up to the InfluxDB server
• Very similar to Naemon Passive checks
• We would like to use RabbitMQ to handle all messages
from hosts
– Push logs to ELK stack
– Push host status to Nagios
– Push state data to InfluxDB

20
Grafana
• Visualization Front end for InfluxDB
• Allows for multiple graphs to be tracked over time

21
Grafana

22
Grafana Cont

23
Logging
• rsyslog
– Standard on most linux systems
• Great for retention
• Bad for Searching
• Bad for Analyzing Patterns
• Splunk
– Uses Elastic Search
• Great for searching
• Great for Analyzing
• Expensive
• Data potentially stored in the cloud

24
Logging – ELK Stack
• Replacement for Splunk
– Harder to set up than Splunk
– Free
• Apache 2.0 License
– Web GUI front end
• ELK Stands for
– Elastic Search
• Search Engine for Logging
• Hadoop Extensions
• Relevance Algorithm
– Logstash
– Kibana

25
ELK Stack - Cont
• Elastic Search
– Search Engine for Logging
– Hadoop Extensions – Parallelizable
– Relevance Algorithm
• Logstash
– Replacement for rsyslog – Stores all events as objects
– Hadoop Extensions – Parallelizable
• Kibana
– Web Front End for ELK stack
– Powerful search
– Easy to visualize patterns, and display graphs of data

26
Kibana - Cont

27
Questions?
• I’d love to collaborate with my colleagues at other sites, and
discuss what works for them as well.

PEARC17: Building bridges - The System Administration Tools and Techniques Used to Deploy Bridges

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to PEARC17: Building bridges - The System Administration Tools and Techniques Used to Deploy Bridges

Similar to PEARC17: Building bridges - The System Administration Tools and Techniques Used to Deploy Bridges (20)

Recently uploaded

Recently uploaded (20)

PEARC17: Building bridges - The System Administration Tools and Techniques Used to Deploy Bridges