Pre-Con Ed: Best Practices for Setting Thresholds in CA SYSVIEW®

World®
’16
The Process of Setting Thresholds in
CA SYSVIEW Performance Management
Tom Quinn, Sr Engineering Services Architect
CA Mainframe Solution Center
MFX02E
MAINFRAME AND WORKLOAD AUTOMATION

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
The “best practice” for setting mainframe monitoring thresholds is a
process, not a set of variables and numbers. Only you can decide what
the most important variables are and what thresholds should be set for
your shop.
Join this informative session to learn how you can set meaningful
threshold values in CA SYSVIEW® Performance Management that are
specific to your environment and business goals for proactive
performance monitoring.
Tom Quinn
CA Technologies
Sr Engineering Services
Architect, Mainframe
Solution Center

Agenda
PROCESS OVERVIEW
HOW CA SYSVIEW THRESHOLDS AND STATES WORK
STEPS IN THE PROCESS
INTEGRATING WITH MAINFRAME TEAM CENTER: OPERATIONAL INTELLIGENCE
1
2
3
4

Setting Thresholds in CA SYSVIEW®
Performance Management
Process Overview

The Process of Setting Thresholds in CA SYSVIEW
Evaluate
Measure
Analyze
Update
Validate

§ Setting thresholds is an on-going, cyclical process that should
be part of your monitoring strategy.
§ The information you need is available on your system through
CA SYSVIEW historical data logstreams.
§ Business requirements play an important role in this process
as well.
Best Practice

§ You must first understand what is “normal” and the variability
of your workload.
§ What is “normal”?
– Time of Day
– Time of Month
– Day of the Week
– Workload type
Best Practice

Why do we go through this process?
§ Performance Management is the process of monitoring and
managing end-user experience and key performance
indicators (KPIs).
– Availability
– Response Time
– Throughput
– Utilization
Performance Management

How CA SYSVIEW States and Thresholds
Work

§ CA SYSVIEW provides the ability to monitor and alert upon
both thresholds and states.
– Thresholds definitions assign numeric warning and problem limit
values to a metric or resource. Thresholds are always numeric.
– State definitions assign a status to a specific condition of a metric or
resource. States always have character values: ACTIVE, INACTIVE,
ONLINE, OFFLINE.
§ All available variables in CA SYSVIEW are listed in the VARS
command display.
How CA SYSVIEW Threshold and State Definitions Work

§ Threshold rule types define the algorithm that is used to
determine when a collected metric has exceeded the
allowable resource usage.
§ Multiple rule types or checks can be defined for a single
metric:
– UPPER - if the metric value exceeds the warning or problem limit values
– LOWER - If the metric value falls below the warning or problem limit values
– CHANGE - defines the amount that a metric can change, up or down, in terms
of standard deviations from the historical average.

§ STATEs do not have multiple rule types.
– STATE exception processing assigns a status to the current state of a
metric or resource.
§ Exception processing, or threshold processing, is part of the
CA SYSVIEW data collection process and controlled by the
Event Scheduler.

The Process

Evaluate
Measure
Analyze
Update
Validate

Evaluate is the process of reviewing your monitoring strategy.
– Are you defining monitoring at the system or application level?
– Do you have business requirements for monitoring the system and the
application?
– Do you have different tiers of monitoring in place?
§ Monitoring tiers allow for the definition of monitoring levels based on the
importance of the system or application.
§ More important, or higher tiered apps, may receive more monitoring than a
less important, lower tiered app.
Evaluate

Determining ImportanceMonitoring Tiers
– Disaster recovery plan
– Business requirements
§ Each time you are in the evaluation phase of this process you
will review all systems, applications or business requirements
to look for new or changed requirements.
Evaluate

Evaluate
Measure
Analyze
Update
Validate

Measure is the process of monitoring your systems and
gathering the data required to analyze both your system data
and the effectiveness of your thresholds.
§ By having CA SYSVIEW installed, configured and continually
running and collecting data you have the ability to measure
your system’s performance and the effectiveness of your
thresholds.
Measure

§ In CA SYSVIEW, it is recommended to collect exception data by
utilizing the Exception Log capabilities (XLOG), and historical
data through the various CA SYSVIEW logstreams and CA
SYSVIEW SMF 255 records.
§ Besides the XLOG, the other logstreams include IMS region
and summary records; MQ history records; CICS detail,
summary and system records; and historical PLOT records.
Measure

§ Audit log and CICS Exec Interface Trace are available log data types that
are not necessarily used in this process.
§ It is recommended you gather the historical record types that you are able
to in order to best provide you with the necessary data to make
meaningful decisions regarding threshold levels. This data will be
reviewed both on-line and through batch reporting.
§ https://docops.ca.com/display/CSPM10/Usage+of+SMF+Records
Measure

Evaluate
Measure
Analyze
Update
Validate

Analyze is the process of reviewing your measured data to determine what
metrics need thresholds, what those thresholds should be, and how effective
your thresholds are.
The very first time you go through this process, this will be the most time-
consuming. Analyze may be the most important step in the process cycle.
Analyze

Where do I begin?
§ CA SYSVIEW contains many out of the box thresholds as
examples
§ A good place to start is with the CA SYSVIEW VARS command
– VARS lists all of the metrics that can be used for thresholding
Analyze

PLOTLOG
§ As an example, PLOTLOG displays hourly interval records for
z/OS and subsystem data collection metrics.
§ A log can potentially contain a large number of records,
therefore it is recommended to use the FILTER command
within the PLOTLOG command display (CA SYSVIEW r14.1 and
greater) to limit the data you wish to search and return.
Analyze

PLOTLOG Example – CPU usage
Analyze

Analyze
CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP
12:0
0:00
AM
1:00:
00
AM
2:00:
00
AM
3:00:
00
AM
4:00:
00
AM
5:00:
00
AM
6:00:
00
AM
7:00:
00
AM
8:00:
00
AM
9:00:
00
AM
10:0
0:00
AM
11:0
0:00
AM
12:0
0:00
PM
1:00:
00
PM
2:00:
00
PM
3:00:
00
PM
4:00:
00
PM
5:00:
00
PM
6:00:
00
PM
7:00:
00
PM
8:00:
00
PM
9:00:
00
PM
10:0
0:00
PM
11:0
0:00
PM
CA11
Average of Average 0.3540.3220.3210.3940.4420.4010.4220.3840.4150.4580.556 0.58 0.5760.5890.5550.5130.4710.4590.4840.4850.4770.4370.5380.397
Average of Minimum 0.173 0.17 0.1640.2170.2180.2230.2150.2140.2260.2780.3630.3930.404 0.42 0.3830.3480.3240.2940.3340.3460.3310.2730.2920.209
Average of Maximum 0.6910.6070.6290.6990.7210.6990.7240.6330.7390.7620.771 0.79 0.7320.7790.7760.7460.6770.6480.6870.6620.6640.6940.8510.687
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
%
AVG GP% by Hour
01/17 - 02/04/2016

This data should start to give insight to your system.
– Are there certain times of days we see spikes?
– Are there periods of time that are consistently higher or at 100%?
Based on these trends, think again about your workload.
– Time of day – batch window?
– Time of month, year
Once you start to understand your data, system and business
requirements you can start to set meaningful thresholds.
Analyze

CICS
§ MENU CHISTORY is a great place to start to look at your CICS
historical data.
– Remember: you must be collecting CICS historical data in the CA
SYSVIEW logstreams to use these commands.
– CREVIEW – by region, the number of transactions that ran in the past
24 hours and an hourly breakdown of those transactions.
– CSYSDATA – by region, 15-minute interval summary.
Analyze

CICS
§ History continued
– CTRANLOG – every CICS transaction that ran on your system.
§ Useful for setting transaction specific thresholds or exclusion rules.
– CTRANSUM – summary of CTRANLOG in 15-minute intervals.
Analyze

Report Writer
§ Batch reporting against this same historical data that is being
accessed by the online commands.
§ As an example – all CICS transactions that executed each day,
the average, minimum, maximum lifetime and CPU.
Analyze

Copyright (C) 2015 CA Technologies, Inc. EXPLORE Report Writer Release: 14.1 07/06/15 20
FROM 01/27/16 20.27.00 MONTHLY CICS TRANSACTION DETAIL REPORT 1 PAGE 1
TO 01/30/16 20.27.25 RESPONSE TIME AND CPU ANALYSIS REPORT DATE 02/10/16 11.44
SHIFT NONE
EACH MONTH
PERIOD NONE
TRAN TRANS AVG TRANS MIN TRANS MAX TRANS AVG MIN MAX
DATE ID COUNT LIFETIME LIFETIME LIFETIME CPU CPU CPU
-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------
01/01/16 CSOL 137 1887.437 1887.424 1887.552 0.000154 0.000125 0.001341
CWBG 72 0.002459 0.000404 0.137902 0.000397 0.000349 0.000653
CRSQ 9 0.000449 0.000371 0.000677 0.000360 0.000317 0.000601
CRMF 6 0.004195 0.000452 0.022358 0.000598 0.000401 0.001314
TKOA 1 3.967686 3.967686 3.967686 0.371662 0.371662 0.371662
CSKL 1 1.734323 1.734323 1.734323 0.028685 0.028685 0.028685
CSFU 1 0.539743 0.539743 0.539743 0.008386 0.008386 0.008386
CATR 1 0.010191 0.010191 0.010191 0.001518 0.001518 0.001518
-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------
SUMMARY 228 1134.146 0.000371 1887.552 0.002047 0.000125 0.371662
-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------
Analyze

§ Once you have this information you can start to build your
threshold definitions.
§ Ask yourself these additional questions:
– Are there company standards for warning or problem thresholds?
§ Ex. A warning threshold is 10% less than a problem threshold
– Are there business requirements for CPUor elapsed (response) time?
– Are there transactions that are critical to our business?
Analyze

Evaluate
Measure
Analyze
Update
Validate

Update is the process of adding new or updating existing
thresholds as needed based on your analysis of the data.
§ In CA SYSVIEW this is done either through the command
displays or through updating the CA SYSVIEW PARMLIB
members.
– Which one depends on how CA SYSVIEW and its data collectors are
started – WARM vs COLD.
Update

SubSystem
CA SYSVIEW Command PARMLIB Member
Thresholds States Threshold States
z/OS THRESH STATES MVSTHRSH MVSSTATE
CICS CTHRESH CSTATES CTHRESH CSTATES
IMS IMSTHRSH IMSSTATE IMSTHRSH IMSSTATE
MQ MQTHRESH MQSTATES MQSTHRSH MQSSTATE
TCPIP TCPTHRSH TCPSTATE TCPTHRSH TCPSTATE
Update

Evaluate
Measure
Analyze
Update
Validate

Validate is the process of verifying your thresholds are accurately being
triggered and exceptions are raised as expected.
ALERTS
– To validate your threshold levels look at the ALERTS and ALERTSUM commands
within SYSVIEW. If you are writing exceptions to the XLOG you can view all
triggered exceptions through the XLOG command and also run batch reports
using the SYSVIEW Report Writer against the XLOG datasets.
Validate

Example report against the CA SYSVIEW Exception Log (XLOG)
START VARIABLE RESOURCE EXC THRSHOLD MAX MIN WARNING PROBLEM
DATE TIME NAME 1 STATUS VALUE VALUE VALUE VALUE VALUE
-------- ------------ -------- -------- -------- -------- -------- -------- -------- --------
03/17/16 01.17.00.010 IMPLMAX IMSD WARNING 103K 105K 105K 98304 131K
01.17.00.011 IMPLSIZE IMSD PROBLEM 101K 104K 104K 60000 80000
01.18.00.014 IMPLMAX IMSD WARNING 104K 106K 106K 98304 131K
01.19.00.012 IMSLFULL IMSD PROBLEM 1 1 1 1 1
01.21.00.024 IMPLSIZE IMSD NORMAL 57344 57344 57344 60000 80000
01.23.00.010 IMPLSIZE IMSD WARNING 73728 73728 73728 60000 80000
01.24.00.017 IMPLSIZE IMSD NORMAL 32768 32768 32768 60000 80000
01.31.00.014 IMSLARCH IMSD NORMAL 1 1
01.31.00.016 IMSLFULL IMSD NORMAL 1 1
01.34.00.006 IMSLARCH IMSD PROBLEM 1 1 1 1 1
Validate

Evaluate
Measure
Analyze
Update
Validate

Mainframe Team Center: Operational
Intelligence

§ SYSVIEW r14.2 introduced Mainframe Team Center:
Operational Intelligence (MTC-OI)
Mainframe Team Center: Operational Intelligence

§ The core benefits of MTC-OI
– MTC-OI is about predictive analysis, creating dynamic thresholds based
on historical performance
§ There are other benefits of MTC-OI as well
– You can visualize and explore historical CA SYSVIEW data that has been
analyzed to determine a normal baseline
– MTC-OI models this historical data provided by CA SYSVIEW and
determines what’s normal for your environment

§ MTC-OI enables you to do the following:
– Easily access historical data from your system.
– View a historical graph of a metric that is overlaid with actual data and
colored bands that illustrate likeliness of metric performance. A "green
highway" is produced from analysis using Western Electric rules.
– Perform analysis to answer questions about a specific metric, such as:
§ What is normal for this metric in my shop?
§ Is the current value of this metric outside of the expected value?
§ Based on past observations, how tightly could I set a threshold without
producing too many false-positive alarms

The Process of Setting Thresholds in SYSVIEW PM
So how does this process change with Mainframe Team Center-
Operational Intelligence?
§ With MTC-OI the Analyze phase will improve for some metrics
by modeling the historical data for you and graphically
displaying the expected behavior.
– Can eliminate the need to run batch reports against your historical data
to determine these boundaries

What data is modeled in MTC-OI?
§ SYSTEM Interval Data
– PLOTLOG 255.2 records
§ CICS Transaction Summary Data
– CTRANSUM 255.25 records
§ CICS System Interval Data
– CSYSDATA 255.28 records

§ MTC-OI can assist you in the process of setting thresholds in
SYSVIEW
§ MTC-OI generates its own predictive alerts after analyzing and
modeling your data and will continue to adjust dynamic
thresholds as more data is provided into the modeling engine
§ The process of setting thresholds requires you to combine
business requirements and system information to use the
most meaningful values.

Recommended Sessions
SESSION # TITLE DATE/TIME
MFX154E Ideation: Intelligent Mainframe Management – Be Part of
Shaping the Future! 11/14/2016 at 4:45 pm
MFX05E Predicting the Fire with Operational Intelligence
11/15/2016 at 9:00 am
MFX06E How to Get Operational Intelligence Up and Running in a
Day 11/15/2016 at 10:00 am
MFX07E So You’re a New Systems Operator – Who Ya Gonna Call?
11/15/2016 at 11:00 am
MFX144E Improving Your Response to Events – Performance First
Responders 11/15/2016 at 1:00 pm
MFX145E Operational Intelligence – Monitoring the Past to Better
Understand Today 11/15/2016 at 2:45 pm
MFX146E Mainframe Operational Intelligence – Predicting the
Future 11/15/2016 at 4:00 pm

Must See Demos in the Mainframe Theater:
Intelligent
Mainframe:
Advanced
Analytics
CA Mainframe Operations
Intelligence:
Performance & Network Mgmt
Application Monitoring
Events & Automation
Intelligent
Mainframe:
Automation
CA Mainframe Operations
Intelligence
CA OPS/MVS® Event Management
& Automation
Intelligent
Mainframe:
End-to-End
Performance
CA Cross-Enterprise APM
CA Mainframe Application Tuner
CA UIM for z Systems
Intelligent
Mainframe:
Storage
Management
CA Vantage SRM
CA 1® Tape Management
System

Questions?

Thank you.
Stay connected at communities.ca.com

Pre-Con Ed: Best Practices for Setting Thresholds in CA SYSVIEW®

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Pre-Con Ed: Best Practices for Setting Thresholds in CA SYSVIEW®

Ähnlich wie Pre-Con Ed: Best Practices for Setting Thresholds in CA SYSVIEW® (20)

Mehr von CA Technologies

Mehr von CA Technologies (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pre-Con Ed: Best Practices for Setting Thresholds in CA SYSVIEW®