Troy Lea's presentation on Leveraging and Understanding Performance Data and Graphs. The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. Leveraging and Understanding
Performance Data and Graphs
Troy Lea
troy@box293.com
Twitter: @Box293
http://exchange.nagios.org/directory/Owner/Box293/1

2. 2
About Me
IT Consultant
Nagios Developer
Love tinkering with Nagios
Why Nagios XI?
It’s a virtual appliance - ready to go

3. 3
About This Presentation
Understanding how performance data is stored
in the back end and how Nagios accesses it
Goal is to give you key pieces of information
A good reference for understanding concepts
This presentation is centered around Nagios XI
Valid for other Nagios implementations

4. 4
Basic Concepts - Part 1

5. 5
Basic Concepts - Part 2
./check_nt -H SERVER -s "" -p 12489 -v USEDDISKSPACE -l C -w 80 -c 95
C: - total: 39.99 Gb - used: 25.28 Gb (63%) - free 14.71 Gb (37%) | 'C: Used
Space'=25.28Gb;32.00;38.00;0.00;39.99

6. 6
Basic Concepts - Part 3
Service check command is executed by the monitoring engine
Monitoring engine receives the result of the check
Data received has performance data
Performance data is anything after the | (pipe)
The performance data is inserted into an RRD file
When viewing the performance graph, PNP4Nagios retrieves the
performance data from the RRD file and generates a pretty graph
Every time the service check receives performance data, it inserts
this performance data into the RRD file which allows you to look at
trends over time

7. 7
Plugins
The power of Nagios is in the plugins!
Monitor what you want, how you want!
Resources available that clearly define the
guidelines around creating plugins
Nagios Plug-in Developer Guidelines
http://nagiosplug.sourceforge.net/developer-
guidelines.html
PNP Documentation
http://docs.pnp4nagios.org/pnp-0.4/doc_complete

8. 8
Plugin Output Explained - Part 1
Plugins produce data divided into two parts
The pipe symbol “|” is used as a delimiter
Example check_icmp
OK - 127.0.0.1: rta 2.687ms, lost 0% |
rta=2.687ms;3000.000;5000.000;0; pl=0%;80;100;;
Data to the left of the pipe symbol is processed
by the monitoring engine
Data to the right of the pipe symbol is used for
inserting into RRD and XML files

9. 9
Plugin Output Explained - Part 2
The exit code Nagios receives from the plugin
determines the state of the service
0 = OK
1 = WARNING
2 = CRITICAL
3 = UNKNOWN
The exit code is not “visible” when running a
check from the command line or looking at the
output returned from the plugin

10. 10
Plugin Output Explained - Part 3
No performance data = no pretty graphs
You can create a plugin using whatever
language and tools are available
All that matters is the end result which is
returned back to Nagios when the plugin has
finished running

11. 11
Plugin Output Explained - Part 4
Examples:
Shell script
Something you might want to check on the Nagios
host itself
perl script
Remotely checking a device using SNMP OR using
third party APIs like the VMware vSphere SDK to
remotely access virtual environments
Visual Basic script
Using NSClient on a Windows host to perform a
check (like RDP usage)

12. 12
Performance Data Specifics - Part 1
Asterix (*) fields are required fields, everything
else is optional
In this instance, rta is the FIRST DS, or DS 1

13. 13
Performance Data Specifics - Part 2
Multiple DS
Each DS is separated by a space
rta=2.687ms;3000.000;5000.000;0; pl=0%;80;100;;
The label can have spaces however the label
MUST be enclosed by single quotes
'Round Trip Average'=2.687ms;3000.000;5000.000;0;
'Packet Loss'=0%;80;100;;
13

14. 14
Basic Plugin - Part 1
Example shell script demonstrating how a plugin
outputs performance data
NUMBER1=$[ ( $RANDOM % 100 ) + 1 ]
NUMBER2=$[ ( $RANDOM % 1000 ) + 1 ]
echo ""OK - Number 1: $NUMBER1 Number 2:
$NUMBER2" | 'Number 1'=$NUMBER1;;;; 'Number
2'=$NUMBER2;;;;“
exit "0"

15. 15
Basic Plugin - Part 2
Here is the output each time it is run:
OK - Number 1: 4 Number 2: 74 | 'Number 1'=4;;;; 'Number 2'=74;;;;
OK - Number 1: 52 Number 2: 758 | 'Number 1'=52;;;; 'Number 2'=758;;;;
OK - Number 1: 73 Number 2: 60 | 'Number 1'=73;;;; 'Number 2'=60;;;;
OK - Number 1: 29 Number 2: 338 | 'Number 1'=29;;;; 'Number 2'=338;;;;
OK - Number 1: 87 Number 2: 612 | 'Number 1'=87;;;; 'Number 2'=612;;;;

16. 16
Basic Plugin - Part 3
Performance data
displayed as a
pretty graph
Demonstration of
how you can
generate
performance data
in a plugin

17. 17
Basic Plugin - Part 4
Now lets add warning and critical thresholds to
the performance data string
Number1
WARNING @ 50
CRITICAL @ 75
Number2
WARNING @ 500
CRITICAL @ 750
echo ""OK - Number 1: $NUMBER1 Number 2:
$NUMBER2" | 'Number 1'=$NUMBER1;50;75;;
'Number 2'=$NUMBER2;500;750;;"

18. 18
Basic Plugin - Part 5
Here is the output each time it is run:
OK - Number 1: 4 Number 2: 74 | 'Number 1'=4;50;75;; 'Number 2'=74;500;750;;
OK - Number 1: 52 Number 2: 758 | 'Number 1'=52;50;75;; 'Number 2'=758;500;750;;
OK - Number 1: 73 Number 2: 60 | 'Number 1'=73;50;75;; 'Number 2'=60;500;750;;
OK - Number 1: 29 Number 2: 338 | 'Number 1'=29;50;75;; 'Number 2'=338;500;750;;
OK - Number 1: 87 Number 2: 612 | 'Number 1'=87;50;75;; 'Number 2'=612;500;750;;

19. 19
Basic Plugin - Part 6
This demonstrates
how the
performance data
does not have any
effect on the state
of the service
Warning and
Critical thresholds
are inside the .xml
file
19

20. 20
.rrd and .xml files
Used for recording the results from Nagios checks
Useful for observing daily trends of your environment
Invaluable for helping resolve performance issues
RRD = Round Robin Database
XML = Information about the Nagios check
PNP4Nagios uses the RRD and XML files to
generate pretty graphs

21. 21
Location of .rrd and .xml files
When a service check returns performance data,
Nagios dumps this into:
/usr/local/nagios/var/spool/perfdata
A background process detects the spooled data
and creates / updates the relevant .rrd and .xml
The Performance Data files live in:
/usr/local/nagios/share/perfdata/<host>

22. 22
Extract .rrd data
You can extract data from an .rrd file
Example (from the CLI):
rrdtool fetch
/usr/local/nagios/share/perfdata/localhost/_HOST_.rrd MAX
-r 900 -s -1h

23. 23
.rrd and .xml Gotchya - Part 1
The .xml file can contain sensitive data
<NAGIOS_SERVICECHECKCOMMAND>check_emc_clariion!$HOSTADDRESS$!-u
readonly!-p Str0ngPassw0rd!-t sp_cbt_busy!--sp A!--warn 70!--crit 90!
</NAGIOS_SERVICECHECKCOMMAND>

24. 24
.rrd and .xml Gotchya - Part 2
Perhaps use a central credential file
<NAGIOS_SERVICECHECKCOMMAND>check_vmware_host!
check_vmware_config_vcenter01!cpu!90!95!!!!
</NAGIOS_SERVICECHECKCOMMAND>

25. 25
.rrd and .xml Gotchya - Part 3
RRD Data is averaged out over time
Looking at performance graphs for past day / week /
month / year will show results with less spikey data
This generally only occurs with data that has lots of
peaks and troughs
Constant data like disk space used will generally not
average out that much
It all depends on your environment!
When reviewing RRD data you need to take into
consideration these factors, it’s all relative!

26. 26
Graphs - How Templates Are Used - Part 1
http://docs.pnp4nagios.org/pnp-0.4/tpl

27. 27
Graphs - How Templates Are Used - Part 2
PNP4Nagios queries the XML file for the
<TEMPLATE> tag
Each datasource has it’s own <TEMPLATE> tag
<TEMPLATE>check-host-alive</TEMPLATE>
Also can be a trailing string in the performance
data (good for distributed monitoring)
OK - 127.0.0.1: rta 2.687ms, lost 0% |
rta=2.687ms;3000.000;5000.000;0; pl=0%;80;100;;
[check_icmp]

28. 28
Graphs - How Templates Are Used - Part 3
From the example graphs:
<TEMPLATE>check-host-alive</TEMPLATE>
<TEMPLATE>check_local_load_alt</TEMPLATE>
PNP4Nagios looks for a php file with this name
in the following folders:
/usr/local/nagios/share/pnp/templates.dist
/usr/local/nagios/share/pnp/templates

29. 29
Graphs - How Templates Are Used - Part 4
check-host-alive
/usr/local/nagios/share/pnp/templates.dist/check-host-
alive.php
This PHP file generates the performance graph
check_local_load_alt
check_local_load_alt.php does NOT exist
Default template is used:
/usr/local/nagios/share/pnp/templates.dist/default.php
29

30. 30
Graphs - Creating Your Own Template - Part 1
The check_command name is what Nagios uses
to insert into the <TEMPLATE> tag in the XML
file (how PNP determines which template to use)
So for this example I have created a copy of an
existing command
check_xi_service_nsclient_alt

31. 31
Graphs - Creating Your Own Template - Part 2
The service definition using the new command

32. 32
Graphs - Creating Your Own Template - Part 3
The graph currently being generated
Default Template being used
Check Command being used
.rrd and .xml files currently contain valid data

33. 33
Graphs - Creating Your Own Template - Part 4
Copy the file:
/usr/local/nagios/share/pnp/templates.dist/default.php
To the following location with the name:
/
usr/local/nagios/share/pnp/templates/check_xi_servic
e_nsclient_alt.php
Edit check_xi_service_nsclient_alt.php

34. 34
Graphs - Creating Your Own Template - Part 5
In the graph we are removing the bottom two lines
Default Template
Check Command command name
Which are lines 62 and 63
$def[$i] .= 'COMMENT:"Default Templater" ';
$def[$i] .= 'COMMENT:"Check Command ' .
$TEMPLATE[$i] . 'r" ';
Save check_xi_service_nsclient_alt.php
34

35. 35
Graphs - Creating Your Own Template - Part 6
How easy was that!
Updated graph
Template Name and Check Command removed

36. 36
PNP Templates In Detail - Part 1
Lets get into specifics
Template we just
modified
It’s not that
complicated! (LOL)
36

37. 37
PNP Templates In Detail - Part 2
.rrd files can have multiple datasources (DS)
Round Trip Time and Packet Loss for example

38. 38
PNP Templates In Detail - Part 3
Example of .rrd file with five DS
Two graphs generated using these DS

39. 39
PNP Templates In Detail - Part 4
Default Template creates one graph per DS
This is a simple PHP foreach loop
The code within the loop references the relevant
DS by the $i variable

40. 40
PNP Templates In Detail - Part 5
This section of the template uses three DS
One graph will be generated using three DS
$opt[1] and $def[1] is a reference for the first graph
being generated

41. 41
PNP Templates In Detail - Part 6
Number formatting
Our modified template and the relative code
The relevant information:
%3.4lf

42. 42
PNP Templates In Detail - Part 7
The three DS template and the relative code
The relevant information:
%4.0lf

43. 43
PNP Templates In Detail - Part 8
Numbers are displayed with four decimal points
%3.4lf
Numbers are displayed as whole numbers
%4.0lf

44. 44
PNP Templates In Detail - Part 9
PNP documentation defines the number
formatting using the printf standard defined here
http://en.wikipedia.org/wiki/Printf
The number (1) and the letter "L" look alike
%3.4lg contains a lower case "L"
The syntax is
%[parameter][flags][width][.precision][length]type

45. 45
PNP Templates In Detail - Part 10
width
When the number is generated on the graph, it will
allocate a minimum specific width, this helps you
align numbers in a column style
precision
Determines if the number displayed is a whole
number, or a number with a specific number of digits
following the decimal place

46. 46
PNP Templates In Detail - Part 11
%3.4lf
width = 3
precision = .4
hence the displayed number is 25.3800
%4.0lf
width = 4
precision = .0
hence the displayed number is 14
Because the precision is 0, NO decimal place is used

47. 47
MRTG - Part 1
MRTG = Multi Router Traffic Grapher
Nagios Addon that is useful for monitoring
network switch and router bandwidth using SNMP
Can be complicated to understand configuration

48. 48
MRTG - Part 2
Nagios XI Wizard called “Network Switch /
Router” automates the configuration of MRTG
MRTG configuration file
/etc/mrtg/mrtg.cfg
MRTG runs as a cron job every five minutes
cron comes from the Greek word for time, χρόνος
[chronos]
Hence cron is a software utility on linux which is a
time-based job scheduler
In the windows world it's the Task Scheduler

49. 49
MRTG - Part 3
When MRTG runs, it gathers data from the
devices defined in the mrtg.cfg file
It dumps this data into the folder
/var/lib/mrtg
For every port monitored, an .rrd file is created
(no .xml file created at this point)
Another background process will then take the
data in /var/lib/mrtg and put it into the correct
location
/usr/local/nagios/share/perfdata/<host>

50. 50
MRTG Gotchya - Part 1
When the Wizard populates the mrtg.cfg file it will
add ALL ports on the switch to the config file
Even if you only selected to monitor 10 ports on
the switch
The Nagios XI Service Configuration will only have 10
ports defined as service definitions
Every time the MRTG cron job runs, it will collect
data from all ports on the switch (as defined in the
mrtg.cfg file)
Extra CPU cycles, extra disk space
50

51. 51
MRTG Gotchya - Part 2
On a 48 port switch this might not concern you
But in a stack of two 48 port switches this
becomes 96 ports + also other internal ports like
link aggregation ports (another 32 ports perhaps)
So these additional 128 ports have now added
8700+ configuration lines to the mrtg.cfg file
128 ports consume about 24 MB of .rrd disk
space
In my past environment, the mrtg.cfg file was
59,000 lines long!
51

52. 52
MRTG Gotchya - Part 3
Suggestion
Clean up the mrtg.cfg file
Remove the ports you do not wish to gather data on
Can this cause Problems?
Yes!
Problem 1
Monitoring additional ports later using the wizard will
not work
The wizard will NOT re-add the ports to the mrtg.cfg file
Wizard detects switch / router is already in the mrtg.cfg file

53. 53
MRTG Gotchya - Part 4
Problem 2 - Adding a switch (or module) to an
existing switch
Monitoring additional ports later using the wizard will
not work
The wizard will NOT add newly detected ports to the
mrtg.cfg file
Wizard detects switch / router is already in the mrtg.cfg file
Very similar behaviour to Problem 1
Only relevant when the new switch / module is managed
through the existing IP Address / FQDN
Common with stacked switches, adding another switch to
the stack

54. 54
MRTG Gotchya - Part 5
Solutions to Problems 1 & 2
cfgmaker
This is how the Wizard configures mrtg.cfg
The wizard updates the existing mrtg.cfg using a php
function (not available from the CLI)
Run cfgmaker @ CLI to generate a config file
Add the contents of the config file to the existing mrtg.cfg
cfgmaker --noreversedns “public@192.168.1.1" --output=output.txt

55. 55
MRTG Gotchya - Part 6
Problem 3 - With a frequently changing
environment, keep mrtg.cfg clean
Monitoring WAN links for remote routers?
WAN link no longer exists?
Disable / Delete service definition(s) in Core Configuration
Manager (CCM)
You will NEED to remove device from mrtg.cfg
Why?
MRTG will still try and collect data from WAN links no longer
accessible
Causes delays and can make MRTG run past the default 5
minute schedule ... can cause graph anomalies

56. 56
MRTG Gotchya - Part 7
Problem 4 - Firmware Upgrade causes port
numbering to change
Major firmware revision applied to switch / router
New data collected for ports is no longer the same pattern
Internal port numbering has changed
mrtg.cfg queries specific port numbers, does not use port
names or descriptions
Example
Old Firmware: WAN = Port 1 LAN = Port 2
New Firmware: WAN = Port 0 LAN = Port 1
Have seen this behaviour on SonicWALL Firewalls

57. 57
Questions
Questions ?

58. 58
Discount Offer
But wait, there's more ...
When visiting the Nagios XI use my affiliate link
http://www.nagios.com/#ref=3oHG00