1. Embedding Existing Heterogeneous Monitoring Techniques into
a Lightweight, Distributed Integration Platform
ADVCOMP 2009 (October 12th 2009, Sliema)
Jan Niemann, GNS Systems
Doreen Seider, German Aerospace Center (DLR)
Folie 1
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into … > 12.10.2009

2. DLR Research Institution
German Aerospace Center Space Agency
Project Management Agency
GNS Systems UNIX/Windows System Management
High-Performance-Computing
Engineering Data Management
Software Development
Folie 2
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

3. Motivation
Monitoring in HPC/CAE Environments
In CAE environments, a lot of software components do monitoring!
Folie 3
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

4. Motivation
Different Purposes for Monitoring
Resource Management
Monitoring Resources like Load, Licenses, …
Examples: Grid Engine, PBS, Platform LSF
Health Management
Monitoring Resources like Load,
Heat, SMART-Status
Examples: Ganglia, Nagios
Resource Planning
Esp. License Monitoring
Examples: We have created License Monitoring and Accounting
Solutions for some of our customers.
Folie 4
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

5. Motivation
A typical setup has 3 Software Components that are monitoring
All with different types of Sensors
All with different administrative and user interfaces
Some Resources are monitored more than once (Load, Licenses)
Let‘s do it once!
But allow reusability of existing solutions!
Folie 5
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

6. Monitoring Techniques
Ganglia (gmetric)
Basically, a Sensor-Script for Ganglia just reports a simple value.
May be written in sh, perl, python, …
Whatever you/your Administrators like!
#!/bin/sh
echo 60.3
# gmetric --name temperature --value
> $(cputemp) --type float --units Celsius
There are a lot of these scripts available on the internet for free.
Measure Infiniband performance
Temperatures via IPMI
…
Folie 6
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

7. Monitoring Techniques
Grid Engine (Load Sensors)
We use a lot of scripts for the Grid Engine to make the scheduler aware
of license usage.
A bit more complicated, but again: may be written in any language.
#!/bin/sh
while read x ; do
echo begin
echo “global:lic_abq_free:5“
echo “global:lic_abq_used:8“
echo “global:lic_abq_total:13“
echo end
done
Platform LSF and PBS use similar mechanics.
Administrators know how to write these scripts.
We think that acceptability is a key factor for success!
Folie 7
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

8. Realizing the Monitoring Environment
Requirements
Collect heterogeneous monitoring data from distributed hosts
Provide them centralized via one unified interface
Reuse existing sensor scripts
Solution
Unify accumulated data by wrapping the scripts with adapters
Embed the scripts (i.e. the adaptors) in an integration platform
Make use of the platform‘s infrastructure to collect the data
Realize an embedded component providing the data
Folie 8
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

9. Realizing the Monitoring Environment
Unify the Heterogeneous Monitoring Data
Wrapping each sensor script with an adapter which
Executes the script
Transforms the data
Provides them via one unified interface
Folie 9
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

10. Realizing the Monitoring Environment
Integration Platform
Unified environment with base software components
Data management, distribution, privilege management, ...
Integrating diverse and specific applications
Benefit
Integrated applications simply use already realized software components
Enables basically independent applications to
Interact
Operate on same data
Base software components are realized only once
Folie 10
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

11. Realizing the Monitoring Environment
Remote Component Environment (RCE)
RCE is such an integration platform
Characteristics
Component-based
Distributed
OSGi/Eclipse RCP
Developed by
German Aerospace Center (DLR) – http://www.dlr.de
Fraunhofer SCAI – http://scai.fraunhofer.de
http://sourceforge.net/projects/rcenvironment
Folie 11
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

12. Realizing the Monitoring Environment
Remote Component Environment (RCE)
Distributed deployment scenario of RCE
Folie 12
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

13. Realizing the Monitoring Environment
Adapter concept
Distributed integration platform RCE
Folie 13
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

14. Monitoring Environment
On Top of RCE
Folie 14
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

15. Outlook
Implement clients using the monitoring environment
GUI
Alert system
Resource management system
Implement more adapters as needed
Folie 15
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

16. Questions?
Folie 16
Jan.Niemann@gns-systems.de, Doreen.Seider@dlr.de
ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009