Ibm tivoli monitoring version 5.1.1 creating resource models and providers sg246900
1. Front cover
IBM Tivoli Monitoring
Version 5.1.1
Creating Resource Models and Providers
Practical step-by-step development
examples
Creating cross-platform Java
resource models
Provider engineering and
implementation
Tony Bhe
Kiyonobu Inayama
Craig Lister
Massimiliano Parlione
Michael Vesich
ibm.com/redbooks
2.
3. International Technical Support Organization
IBM Tivoli Monitoring Version 5.1.1
Creating Resource Models and Providers
August 2003
SG24-6900-00
20. Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
AIX® Notes® Tivoli®
™ OS/400® Tivoli Enterprise™
^™ Perform™ Tivoli Enterprise Console®
eServer™ Redbooks™ TME®
IBM® Redbooks (logo) ™ WebSphere®
ibm.com® RMF™
NetView® S/390®
The following terms are trademarks of other companies:
Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other
countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the
United States, other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun
Microsystems, Inc. in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure
Electronic Transaction LLC.
Other company, product, and service names may be trademarks or service marks of others.
xviii IBM Tivoli Monitoring: Creating Resource Models and Providers
22. in the teams that developed and designed Tivoli Java™ Management Extensions
(TMX4J), Tivoli Web Component Manager (TWCM), and ported ITM Version
5.1.1 on OS400 and OS2. Currently, he is working on the ITM 5.2 OEM Java
engine. Before joining IBM, Mr. Parlione worked as an independent consultant
for the Italian Research Council (CNR) and as an employee of the “Regione
Marche” regional government. He received his degree (“laurea”) in Computer
Science from the University of L'Aquila, Italy, in July 1995 and a doctorate in
Computer Science from the University "La Sapienza" of Rome in April 2000. He
is co-author of the Introducing IBM Tivoli Monitoring for Web Infrastructure,
SG24-6618 redbook.
Michael Vesich is a member of the Tivoli Services organization in North
America. He has over fourteen years experience in the information systems field
with nine years in software development. Prior to joining IBM in 1998, Michael
worked for Kvaerner Engineering, initially as an Automation Engineer developing
real-time industrial process control systems and later becoming the I.T. Director
for the Merrillville facility. Since joining IBM, he has focused on the Performance
and Availability product set, with a more recent role as an ITM Subject Matter
Expert. He has contributed to the Tivoli Certification program, Business Partner
enablement, and was the initial developer of the Tivoli Knowledge Toolkit.
Thanks to the following people for their contributions to this project:
Joanne Luedtke, Edson Manoel, Stephen Hochstetler, Lupe Brown, Wade
Wallace, and Chris Blatchley
International Technical Support Organization, Austin Center
Wade Allen
Software Developer, IBM Software Group
Kevin P. Ferguson
Software Developer, IBM Software Group
Steve Gutierrez
Software Developer, IBM Software Group
Jason Hooper
Senior Software Engineer, IBM Software Group
Bill Horne
Software Developer, IBM Software Group
Richard F. Reed
Systems Integrator Consultant, IBM Software Group
xx IBM Tivoli Monitoring: Creating Resource Models and Providers
23. Theo Winkelmann
Technical Evangelist, IBM Software Group
Become a published author
Join us for a two- to six-week residency program! Help write an IBM Redbook
dealing with specific products or solutions, while getting hands-on experience
with leading-edge technologies. You'll team with IBM technical professionals,
Business Partners and/or customers.
Your efforts will help increase product acceptance and customer satisfaction. As
a bonus, you'll develop a network of contacts in IBM development labs, and
increase your productivity and marketability.
Find out more about the residency program, browse the residency index, and
apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our Redbooks™ to be as helpful as possible. Send us your comments
about this or other Redbooks in one of the following ways:
Use the online Contact us review redbook form found at:
ibm.com/redbooks
Send your comments in an Internet note to:
redbook@us.ibm.com
Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. JN9B Building 003 Internal Zip 2834
11400 Burnet Road
Austin, Texas 78758-3493
Preface xxi
24. xxii IBM Tivoli Monitoring: Creating Resource Models and Providers
28. 1.1 Overview
IBM Tivoli Monitoring Version 5.1.1 (ITM) is a Resource Model based monitoring
infrastructure that provides Resource Models for best practice monitoring of
common resources. ITM also provides System Administrators a means to extend
its monitoring capabilities by providing the ability to create and import custom
Resource Models and dynamically pluggable instrumentation. The focus of this
redbook is on the methods used to extend the monitoring capabilities of the
product through the use of IBM Tivoli Monitoring Workbench 5.1.1 (ITMWB),
which is supplied with the product, and through the development of custom Java
instrumentation.
After the development of a custom Resource Model is complete, the package
created by the ITMWB can be installed into your Tivoli Enterprise environment
from the command line. Once installed into the Tivoli Enterprise environment, the
new Resource Model can then be used in the same manner as the ITM provided
Resource Models.
1.2 High level architecture
Figure 1-1 on page 5 gives a high level overview of the full data flow of IBM Tivoli
Monitoring Version 5.1.1. You will note that the flow commences with either the
distribution of a supplied Resource Model or the development of a custom
Resource Model from ITMWB, and completes with the measured metrics being
supplied to one or all of the following:
IBM Tivoli Data Warehouse
IBM Tivoli Web Health Console
IBM Tivoli Event Console
The cycle of instrumenting and reporting will repeat, with the above outputs, until
the ITM Engine is stopped.
4 IBM Tivoli Monitoring: Creating Resource Models and Providers
29. Trend Analy sis
CuCu Trend Ana lysis
s t s to
om m
ize ize/
De /D Dis Data Warehouse
Data Warehouse
Profile
Profile D
fa e
is t r
tri ib
ulfau bu utRollup
ts lt
s
te e Rollup
y Web Health
la Web Health
TMR
TMR
TMR
TMR D isp
Display Console
Console
Ge
tD
ITM
ITM ata
ll
Tivoli Event Console
In s ta HeartBeat
HeartBeat Get Data
Endpoint
Di Unix/Linux
tall
Resource
In s st r
Model ib
Resource Design Distributeut Endpoint Endpoint
Model
e Endpoint
Windows
ITM Engine
Uni x/Linux
Create Windows
Design , Create, Deb ug NT/2000
NT/2000
Debug
ITM Engine ITM Engine
ITM Engine
Workbench
Workbench
Figure 1-1 High level architecture view
1.3 Engine features and abilities
ITM eliminates the need to transfer large amounts of diagnostic data back to a
central location for evaluation. On each localized endpoint, it has the ability to:
Perform root-cause analysis
Isolate individual faults
Initiate programmed corrective actions
The ITM Engine is able to initiate the following functions on the endpoint:
Event analysis
Chapter 1. IBM Tivoli Monitoring architecture 5
30. Event correlation (Windows® only)
Event aggregation
Event management
Data logging
The engine’s ability to perform these functions locally is what frees the system
administrators from the analysis of multiple data sources, including large log
files, event logs, and performance metrics, to detect resource failures and
bottlenecks.
A further extension of the local engine’s capabilities is obtained when the data
logging feature is used in conjunction with IBM Tivoli Data Warehouse. This
enables system administrators to gain a trend analysis tool for any resource
profile from which data logging is enabled.
All instrumentation sources for ITM are provided to the engine through a
Common Information Model (CIM) interface. In a Windows environment, this is
provided by Microsoft®’s implementation of CIM, Windows Management
Instrumentation (WMI). The UNIX® component of CIM is provided via IBM
Tivoli’s own CIM implementation, which is called Touchpoint.
1.4 Endpoint integration
There are several aspects that make up the ITM integration with the Tivoli
Endpoint. It helps to be able to identify these aspects in order to aid in problem
diagnosis. First, understanding the flow of information through the endpoint to
the ITM components will assist in determining which components may be at fault;
is the problem high level (TMR or specifically MDist2), mid-level (the assigned
Gateway), or low-level (the Endpoint or ITM Engine itself)? Second, the directory
structures of the engine will define which files or paths should be investigated
when a problem is believed to be on the endpoint directly.
1.4.1 ITM data flow
It is helpful to understand the flow of configuration information as it relates to the
ITM components. Figure 1-2 on page 7 provides a high level view of the ITM
Engine upon which later chapters will provide additional granularity.
6 IBM Tivoli Monitoring: Creating Resource Models and Providers
31. Data Flow
TME
1
LCF
ITM Engine
2
3 Event
Analyzer
Handling
ITM Engine
4
TMWService
(Java Class)
5
Platform Specific CIM Implementations
Reference s ITM Logical Diagrams:
Touchpoint (Java)
-or-
WMI (Windows)
Provider Layer
Resources (OS & Applications)
Figure 1-2 ITM data flow diagram: all platforms
Chapter 1. IBM Tivoli Monitoring architecture 7
32. Figure 1-2 on page 7 contains five boxes numbered one through five. Each of
these boxes represents a data flow and is defined in the subsequent numbered
lists. Each list number represents the number in the box in Figure 1-2 on page 7.
1. Management Framework & Endpoint
Management [M9] information (that is, Reference Model and Parameters) and
Manageability [M12] information (that is, Dynamic Model) reach the endpoint
through a conduit, currently the Tivoli Endpoint agent, LCFD.
2. ITM Engine
The ITM Engine works as a façade between the LCFD and the ITM
components. It receives Tmw2k profile distributions and configuration
changes from the LCFD and configures the management function, the
analyzer component, with the management information, Resource Model and
parameters, and the Touchpoint with the manageability information, and
Dynamic Model.
3. Dynamic Model
The Dynamic Model is loaded in the Touchpoint. The manageability
information presented in M12 MOF format will be loaded in the underlying
CIMOM.
4. Analyzer
The Analyzer component is actually a script engine, which means the
configuration that it receives from the façade is presented in script form. In
this script, the TMWService object is instantiated and initialized with
information about the resource that it needs to collect metrics on. At run time,
the code inside the script invokes the collect method of TMWService, which in
turn accesses the Touchpoint to collect the resource metrics.
5. TMWService
a. Java platform
Driven by the script, TMWService Java class accesses the Touchpoint
using the Touchpoint service layer (TSL) interface on Java to collect the
resource metrics.
b. Windows platform
Driven by the script, the TMWService COM object accesses the WMI
using the WMI APIs to collect the resource metrics. The metrics may be
direct WMI Providers or ILT compliant Providers utilizing the Tivoli
Touchpoint Java engine.
The metric data collected is reported back to the Resource Model script
where it is processed according to the best practices coded within the script.
An event handling component is then used to allow the analyzer to
communicate the result of the best practices appliance to other components.
8 IBM Tivoli Monitoring: Creating Resource Models and Providers
33. CIM components and the Provider layer
The Common Information Model (CIM) allows the ITM Engine to handle system
resources in an industry standard object oriented fashion, while the Provider
layer allows for dynamically pluggable instrumentation within the ITM Engine.
The concepts and technologies behind the CIM implementation and the Provider
layer are discussed at length in Chapter 4, “Providers” on page 109.
Resources
Resources refers to the physical resources that the Provider components provide
instrumentation for. Instrumentation is handled through either operating system
libraries or Application Programming Interfaces (APIs), such as Java or vendor
supplied APIs. Such resources include, but are not limited to, the following:
System hardware
Operating system
Applications
1.4.2 Directory structures
When troubleshooting, it is important to know where to find the files you need.
This section will give some insight into where log files and other ITM components
are located.
Windows platform
Figure 1-3 on page 10 corresponds to the Framework (LCFD) components of the
endpoint. The primary log file, lcfd.log, can be found in the lcfdat1 directory. This
log file can prove to be very useful while diagnosing many problems.
Chapter 1. IBM Tivoli Monitoring architecture 9
34. Figure 1-3 Windows LCFD directory structure
It is also important to note that the lcfbinw32-ix86JREDMAE directory is where
Java is installed by default when you distribute a profile from the command line
using the wdmdistrib command and the -J switch.
10 IBM Tivoli Monitoring: Creating Resource Models and Providers
35. Another directory of interest is lcfbinw32-ix86tools. This is where all the tool
type programs (perl, bash, ntprocinfo, and so on) reside.
Figure 1-4 corresponds to the ITM components of the endpoint. The main ITM
log file resides in the lcfdat1LCFNEWTmw2k directory and is named
Tmw2k.log. It is in here that you will find most of your ITM debugging information
when you run into problems.
Figure 1-4 Windows ITM directory structure
Another useful file is located in lcfdat1LCFNEWAMGlogs and is named
logging.properties. This file is a Java config file that gives you the ability to initiate
logging for the various Java modules. This is easily done by looking for lines
such as islogging=False and changing them to islogging=true, and then
restarting the ITM Engine.
You will notice another directory under the preceding directory called log files.
This directory relates to the log file adapter Resource Model discussed later in
this book. It is in this file that the offsets are stored when reading log files. If the
engine is stopped and Java is also stopped, you can remove this file. When the
Chapter 1. IBM Tivoli Monitoring architecture 11
36. engine restarts, the ITM log file adapter will start reading from the beginning of
your log file.
In the directory lcfdat1LCFNEWAMWlogs, you will find three other log files
named ILTManagerForJava1.log, trace_ILTManagerForJava.log, and
msg_ILTManagerForJava.log. These can be useful when debugging Java ILT
issues.
If you have enabled logging in your profile, then the directory
lcfdat1LCFNEWTmw2kdb is where the log information is written on a
Windows endpoint and stored in a Microsoft Access database.
The lcfdat1LCFNEWTmw2kDec directory is particularly interesting. This
directory is where your code, be it VBA or Java, is stored on the local endpoint. If
you stop your ITM Engine by use of the wdmcmd -stop -e <endpointname>
command, you are able to make changes to this script on the local endpoint. You
will then be able to restart the ITM Engine and validate your changes. This can
be a great time saver, as you do not have to go all the way out to the ITMWB and
back through the distribution process.
In the lcfdat1LCFNEWTmw2kMof directory, you will find your MOF files that
you compiled from the ITMWB. There is also a directory within this tree named
compiled. It houses the locally compiled MOF files. So if your MOF file is in the
top directory but not in the compiled directory, then you know that you have a
problem.
Finally, the lcfdat1LCFNEWTmw2kUnixClassesWizGenRM directory is the
storage area for any dependant jar files that you may have included in your
Resource Model.
Java platform on UNIX
Figure 1-5 on page 13 corresponds to the Framework (LCFD) components of the
endpoint. The main LCFD log file, lcfd.log, can be found in the lcfdat1 directory.
This log file can prove to be very useful while diagnosing many problems.
It is also important to note that lcfbinaix4-r1JREDMAE is where Java is
installed by default when you distribute a profile from the command line using the
wdmdistrib command and the -J switch.
12 IBM Tivoli Monitoring: Creating Resource Models and Providers
37. Figure 1-5 UNIX directory structure
It is in the lcfdat1LCFNEWAMWlogs directory, in Figure 1-6 on page 14,
where you will find the trace_dmxengine.log, msg_dmxengine.log,
trace_dmxeu.log and trace_dmxntv.log log files. The file that will be most useful
Chapter 1. IBM Tivoli Monitoring architecture 13
38. is trace_dmxengine.log. This file is the equivalent to the Windows endpoint
Tmw2k.log. The best way to view the file, as it is XML formatted, is to open the
file in a text editor and set word wrap on.
Figure 1-6 Java ITM on UNIX directory structure
14 IBM Tivoli Monitoring: Creating Resource Models and Providers
39. Another useful file is located in lcfdat1LCFNEWAMGlogs and is named
logging.properties. This file is a Java config file, which give you the ability to
instigate logging for the various Java modules. This is easily done by looking for
lines like islogging=False and changing it to islogging=true, and then restarting
the ITM Engine.
If you have enabled logging in your profile, then the
lcfdat1LCFNEWTmw2kUnixdataITMLoggerdblogger directory is where the
log information is written on a UNIX endpoint. The logging information is stored in
an Open Source Database by Quadcap Software.
The lcfdat1LCFNEWTmw2kUnixDec directory is particularly interesting. It is
here that your code, be it VBA or Java, is stored on the local endpoint. If you stop
your ITM Engine by use of the wdmcmd -stop -e <endpointname> command, you
are able to make changes to this script on the local endpoint. After restarting the
ITM Engine, you can then validate your changes. This can be a great time saver,
as you do not have to go all the way out to the ITMWB and back through the
distribution process.
Finally, the lcfdat1LCFNEWTmw2kUnixClassesWizGenRM directory is the
storage area for any dependant jar files that you may have included in your
Resource Model.
1.5 ITM Engine architecture
Figure 1-7 on page 16 highlights the logical components that comprise the
Windows platform ITM Engine.
Chapter 1. IBM Tivoli Monitoring architecture 15
40. Windows, ITM Engine Logical Components
Analyzer
Resource Resource Resource Resource Resource Resource Resource
Model Model Model Model Model Model Model
TMWService
WMI APIs
WMI (CIMOM)
DM Classic Monitor Probes
COM Objects
ILT Manager for Java (WMI Provider DLL)
Custom Scripts
WMI Provider DLLs
Java Virtual Machine (JVM)
Launch (Class Loader)
ILT
ILT JMX ILT
Provider
MBean
Providers
JNI Server
Binary
Resource Resource Resource Resource Resource MBean1
Library
Resources (OS & Applications)
Figure 1-7 ITM Engine components for Windows
Figure 1-8 on page 17 highlights the logical components that comprise the Java
platform ITM Engine.
16 IBM Tivoli Monitoring: Creating Resource Models and Providers
41. Java, ITM Engine Logical Components
Analyzer
Resource Resource Resource Resource Resource
Model Model Model Model Model
DM Classic Monitor Probes Service Object
Launch (Class Loader)
Custom Scripts
Java Class Loader
ILT
ILT JMX ILT
Provider
MBean
Providers
JNI Server
Native
Resource Resource Resource MBean(s)
Binaries
Resources (OS & Applications)
Figure 1-8 ITM Engine components for Java
1.5.1 Engine input
To get the engine to monitor your endpoint, you need to create a profile that
contains at least one Resource Model and distribute it to the endpoint.
A Resource Model is a way of monitoring certain properties of the endpoint to
which the profile, which contains the Resource Model, is distributed. The data is
derived through the Service Object API using WMI or CIM from the endpoint’s
operating system.
When the Resource Model is distributed to the endpoint, the following types of
files are placed in the directories described in 1.4.2, “Directory structures” on
page 9 and “Java platform on UNIX” on page 12:
Managed Object Format (MOF)
Visual Basic or JavaScript
Chapter 1. IBM Tivoli Monitoring architecture 17
42. Dependent jar files
A Resource Model may have the following elements defined depending on the
desired functionality:
Visual Basic or JavaScript, as contained in the VisitTree function
Events
Thresholds
Parameters
Logging
Dependencies
The script component
Every Resource Model requires some programatic instructions on how to deal
with the metric data supplied. It needs to compare these values against your
thresholds and decide when it is appropriate to generate the alerting events.
The public function VisitTree contains your monitoring algorithm, which will be
called cyclically, as specified in your Resource Model.
Events and thresholds
Each Resource Model can include one or more events which will be checked
against its corresponding threshold.
Each period cycle, the monitored resource is checked against the specified
numeric threshold. If the threshold is exceeded, then an event will be escalated.
Events can have many causative indications or only one. If there are many
indications, they must be aggregated before they trigger an event. If the
indication does not exceed its particular threshold, it is called a hole. Holes are
used to reset indications. In other words, if an indication exceeds its threshold for
three cycles in a row without any holes, then an event would be generated. If, on
the second cycle, the threshold was not exceeded, then the engine would restart
the threshold breech count. This examples assumes that you have configured
the indications and holes for the Resource Model to be 3 and 0, respectively.
Parameters
While thresholds must be numeric, parameters can be multiple numeric or alpha
characters. You are able to customize your Resource Models by coding the
parameters in ITM Workbench as defaults or they can be entered at profile
definition time.
18 IBM Tivoli Monitoring: Creating Resource Models and Providers