Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management

Best Practices for Optimizing Mainframe
Application Performance with CA Cross-Enterprise
Application Performance Management
Tom Quinn
Mainframe
CA Technologies
Sr Engineering Services Architect, Mainframe Solution Center
MFX16E
@tomquinn03

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
For Informational Purposes Only
Terms of this Presentation
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The presentation provided at CA
World 2015 is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer
references relate to customer's specific use and experience of CA products and solutions so actual results may vary.

Abstract
In this session you will see a technical overview
of the CA Cross-Enterprise Application
Performance Management (CA APM) solution
and how it extends CA APM into the mainframe.
You will also hear about common install or
startup problems, details about tracing end to
end transactions into the mainframe, and how
to utilize the CA Mainframe Performance Suite
integrations.
Tom Quinn
CA Technologies
Sr Engineering Services
Architect, Mainframe
Solution Center

Agenda
CA CROSS-ENTERPRISE APM TECHNICAL OVERVIEW
SOLUTION INSTALLATION
CA PRODUCT INTEGRATIONS
INSTALLATION VALIDATION
COMMON ERRORS AND RESOLUTIONS
UNDERSTANDING TRANSACTION TRACING
2
3
4
5
6
CA CROSS-ENTERPRISE APM ROADMAP1
7

How do I deliver a flawless experience every
time an application touches the mainframe?
In the application economy it’s all about your customers.
You need to think about your mainframe reframed.
Connect mobile-to-
mainframe applications
Create mainframe
infrastructure flexibility
for the future
Unleash the power of
data on the mainframe

Agile Operations for End-to-End APM
CA Cross-Enterprise APM
Transaction tracing across entire application architecture...
...provides reduced MTTR, greater availability and optimal customer experience
3rd Party Apps
Databases
Web Services/SAAS
MIDDLEWARE
Application
Server
NetworkUser Experience
CICS IMS
Datacom
DB2
TRANSACTIONSDATA
Systems of Operations Systems of RecordSystems of Engagement
Quickly pinpoint root
cause of issues for
faster MTTR

End to End Application Monitoring

VALUE DRIVERS
Strategic Investment Themes
Customer Value Drives Roadmap Themes
Audit and Compliance Reporting Maximize Existing ResourcesApplication Experience Continuity
Collaborative Application Development Real Time InsightsHigh-speed Accessibility with Integrity
Data Security is ParamountCreate and Extend Mainframe Services Speed, Ease of Use and Flexibility
Connect to applications and data on the
mainframe
Unleash the power of data on the
mainframe
Create infrastructure flexibility for the
future
Applications Data Infrastructure
ROADMAP THEMES
Secure Data and Service Enablement
Total Cost of Ownership
Business Continuity

Distributed/Mainframe
Operations Analysts
CA Performance Management Portfolio Vision
Unified Performance Management for z Systems
MF Systems/Network
Analysts
Distributed
Application Analysts
Network
Analysts
CA NetMaster®
Databases
Network
CICS & MQ
z/VM, KVM &
Linux on z
CA UIMCA UIM for z Systems
CA Cross Enterprise
APM
Integrated Mainframe
Management Console
CA APM
Planned
AvailableSystem Analysts
CA SYSVIEW®
Storage
Capacity Planner
CA MICS®
Storage
Administrator
CA Vantage™

CA Cross-Enterprise
Application Performance
Management

Technology
Goals
Product
Architecture
Goals
Technology and Product Architecture Goals
CA CE-APM
 Rapid deployment
 Simplified Install and deployment
 Open APIs
 Maintain MF currency support
 Transaction trace across entire MF
 Standardize Interface with APM

 Enhanced productivity a Re-
Engineered Application Triage Map
allowing users to naturally and
easily triage application problems
 Improved root cause analysis with
Differential Analysis by never
missing a period during which an
application suffered a significant
change in performance – without
having to configure specifically for
that application.
 Improved root cause analysis -
Automatically detect and monitor
performance of applications at
process boundaries
Roadmap: CA CE-APM
Timeline as of Nov 1, 2015
Marquee
Features/
Business
Value
Planned Under Consideration
Delivered Planned Under Consideration
Product/
Releases
CA CE-APM 10.0
Currently GA
CA CE-APM 10.2 CA CE-APM
 Improved Root cause analysis
through deeper visualization of the
MF application components
through tighter and broader
transaction tracing from mobile
through the mainframe
 Enhanced productivity with simplified
install and upgrade process.
through additional application
protocols
through tighter transaction
tracing from mobile through the
mainframe
 Enhanced productivity through
interface standardization for
install and configuration
 Enhanced productivity with
simplified install and upgrade
process.
 Secure Data and Service Enablement
 Business Continuity
 Total Cost of Ownership

Delivered
Feature Description Benefit
APM Team Center Overlay real world attributes on to complex
application relationship data to minimize the
spaghetti seen in typical Application Maps
Team Center allows users to quickly look at the
components that comprise their applications and roll
back time to the point when they first started
experiencing symptoms to find the "patient zero”
causing the current issue
Differential Analysis Spikes are mapped & instability visualized
across many apps and long time periods.
By using simple statistical techniques, CA APM
Performance Spike Detection discovers spikes in
application performance, proactively triggers
evidence (trace) collection, and displays a rich
performance story across applications, all with
simple configuration that works out-of-the-box
Entry Point Detection Automatically detect and monitor application
processes without requiring any deep knowledge of
either the application or application framework , all
without requiring any manual instrumentation
Broaden application monitoring coverage
applications/frameworks by automatically detecting
and monitoring performance of application at
process boundaries

Planned CE APM 10.2
Feature Description Benefit
APM Team Center
Support
Overlay real world attributes on to complex
application relationship data to minimize the
spaghetti seen in typical Application Maps
Team Center allows users to quickly look at the
components that comprise their applications and roll
back time to the point when they first started
experiencing symptoms to find the "patient zero”
causing the current issue
Cross system trace Provide deeper transaction visibility of the trace
within the MF environment across lpars
Enables operation to see deeper into the transaction
flow of the mainframe
Land Lightly Rationalization of system parameters. Reduce time to value of investment return for user
base. Make the product easier to install and easier to
configure

APM Team Center
Re-Engineered Application Triage Map
PAIN
Benefit
SOLUTION
[Planned Feature]
• Replaces the Application Triage Map
• Roll back time to the point when applications first started
experiencing symptoms to find the "patient zero” causing the current
issue
• Easily share this information with the owner of that resource to
resolve the issue quickly
• Attribute-based filtering allows for custom arrangement of data
• First step in an all-new user interface designed from the ground up
• Allows users to naturally and easily triage application problems, even with little-to-no
knowledge of the application (or the APM Team Center itself)
• Team Center can be used to overlay real-world attributes onto complex application
relationship data to minimize the "spaghetti" seen in typical application topology maps.
EASY.
• Inability to easily triage application problems without broad
knowledge of the application
• Application relationship data depicted in dense complex inter-
relationships in typical application topology maps

Cross system trace enhancements
Provide deeper transaction visibility of the trace within the MF environment
across LPARs
BENEFIT
[Planned Feature]
• User cannot see detail beyond first transaction link within the
MF
• Get a summary of the transaction on the MF not the detail
• Need ability for deeper triage
• Provide vision of cross system trace to include multiple level
of transaction
• Provide vision of the cross system trace within the same
LPAR
• Improved root cause analysis by allowing the user deeper visibility into the MF
• Improved root cause analysis enabling faster triage of MF issues
PAIN
SOLUTIONInsert screenshot here

Land Lightly
Reduce time to value of investment return for CE APM user base
BENEFIT
[Planned Feature]
• initial install for new customer is extremely hot in CPU
requirements
• install is complex and includes many steps
• Elimination of dynamic install requirement
• Elimination of the user SVC during installation
• Streamline security implementation and integration with
Enterprise Security solutions
• Rationalization of system parameters for lower overhead
during data collection
• Integrated Management through easier management of security across the Enterprise
• Simplify Software Usage through the rationalization of system parameters
• Accelerate Software Deployment eliminating unnecessary steps in deploying the newest updates for SYSVIEW
PAIN
SOLUTION

Under Consideration
Improved Root cause analysis through additional application protocols – IMS
Connect, …
Improved Root cause analysis through tighter transaction tracing from mobile
through the mainframe – can we expand to support IBM Mobile Workload pricing for
the mainframe
Enhanced productivity through interface standardization for install and configuration
Enhanced productivity with simplified install and upgrade process.

Technical Overview

CA APM / CA CE APM Terminology
 CA Application Performance Management (APM)
– Formerly “Wily” (name is no longer used)
– Introscope – the dashboards
– Investigator – the data view
– Management Modules – java files describing the thresholds, alerts,
metric groupings, dashboard definitions
– CA APM Team Center – the new browser based interface with
application mapping
 CA Cross-Enterprise APM
– Mainframe data extension for CA APM

CA APM / CA CE APM Architecture

Licensing
 The CE APM license allows you to install the following
components only for the use of CA CE APM
– CA Cross-Enterprise APM agent
– CA SYSVIEW PM CE APM Option
– CA SYSVIEW for DB2 PM zOS (formerly Insight for DB2)
 If you are already licensed for and using these products, those
installations will be modified and utilized

 CA Cross-Enterprise APM documentation is available as a
bookshelf
 Includes all of the documentation for CA CE APM
– Install Guide
– User Guide – contains a listing and definition of all metrics available
 Includes links to the other product bookshelves
– Install Guides
Documentation

 Installed Components
 Agent Configuration
 CA SYSVIEW® Configuration
 CA SYSVIEW® for DB2 Configuration
 CA NetMaster® for TCP/IP Configuration
– CA NetMaster is not part of the CA CE APM Product, but does connect to CA
APM
Installation

 The CE APM Agent must be installed
– The release of the agent must be equal to or less than the release of the CA APM
Enterprise Manager
 CA SYSVIEW is used for zOS, CICS, IMS, MQ, Datacom and TCP/IP metrics
 CA SYSVIEW for DB2 zOS is used for DB2 metrics
– Formerly CA Insight for DB2 zOS
– CA SYSVIEW for DB2 connects directly to the agent via XNET, it does not need to
connect to CA SYSVIEW zOS for CA CE APM
Installed Components

 Agent runs as a started task on zOS
– The proc uses the included STDENV member to locate the zfs file system
 Run-time files are located in zfs and are java based
– File system must be permanently mounted
– During SYSVIEW installation a zfs file system is created and must be
permanently mounted if using SYSVIEW for CE APM
 Edit the STDENV member of the SAMPJCL library. This member
determines the paths for java, the CE APM agent runtime and SYSVIEW
runtime files in zfs
Agent Configuration

 In the zfs directory there are two files you can edit
– Introscope_Cross-Enterprise_APM.profile
 Allows for generic agent property information
 ppz.smf.socket.port=15029
– default for tracing socket, must match what is in SYSVIEW GROUPS
for CICSWILY and IMSWILY portlist
Agent Configuration

 Introscope_Cross-Enterprise_APM.profile
 APM EM hostname and port
– introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=hostname
– introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=5001
 # Logging Configuration
– Dictates whether log activity is sent to zOS joblog (console) or zfs log file in the
CE APM agent directory
 Various debugging features available
– #log4j.logger.IntroscopeAgent=VERBOSE#com.wily.util.feedback.Log4JSeverityLevel,logfile
Agent Configuration

– Cross-Enterprise_APM_Dynamic.properties
 This is the primary configuration file for CE APM
 The EULA must be accepted or the agent will not start!
– CA.Cross-Enterprise.APM.I.Read.And.Accept.End.User.License.Agreement=Yes
 SYSVIEW.update.interval=120
– How often CA SYSVIEW data is retrieved. Default is 120 seconds. 15-300 seconds
is possible.
– This has an impact on mainframe resource utilization
Agent Configuration

– Cross-Enterprise_APM_Dynamic.properties
 Basic CA SYSVIEW data collection is controlled here
– SYSVIEW.Metrics.collect=yes
 Detailed SYSVIEW data collection is controlled here. You can determine
whether to collect all, none, or specific portions of data.
– SYSVIEW.CICS.Regions.regex=*
– This has a direct impact on resource utilization
Agent Configuration

 Cross-Enterprise_APM_Dynamic.properties
 Basic SYSVIEW for DB2 data collection is controlled here
– Insight.metrics.collect=yes
 XNET and Passticketuser information
– Insight.passticket.support=yes
Agent Configuration

 Only the CEAPM option needs defined in the INSTALL job
 CICS, IMS, MQ, etc. are otherwise configured if data is to be
collected for those subsystems
CA SYSVIEW Configuration – CA CE APM Only

 CA SYSVIEW PM r14.0 and higher requires CA Common Services CA Master (FMID
CEI0E10) to be present
 Internal product licensing for CA CE APM allows certain CA SYSVIEW data collection and
commands to be utilized
 The CA CE APM Installation Guide includes a section on “Useful CA SYSVIEW
Commands”
– This is a list of commands that are available to CA CE APM only customers that can be used to validate
installation, data collection and explain how to use GROUPS in CE APM
– MENU CEAPM is available in CA SYSVIEW r14.1 showing useful commands for CA CE APM only users

CA SYSVIEW MENU CEAPM

MENU CEAPM as Primary Menu

 The CEAPM Installation Guide includes what CA SYSVIEW steps must be run
and which can be skipped for a CA CE APM only installation
– LOGGER files do not need to be created, except AUDIT if your installation requires it.
– The CICS and IMS LOGR Tasks are used for transaction tracing only, the logger files are
not required.
 CICS data collection is configured as normal
 HELP WILY0000 provides information on supported tracing methods and CICS
data collection GLUE information

Dashboards

 Cross-Process Tracing
– Details of the transaction
across all of the
monitored components of
the transaction path,
distributed and
mainframe.

 Supported Cross-Process Tracing Methods
– MQ to CICS or IMS
– Web Services to CICS
– CTG to CICS
 Distributed APM agent passes a GUID and trace request to the
CEAPM Agent.
– The CEAPM agent tells SYSVIEW to send the SMF details to the APM EM
for that transaction record.
– The APM EM correlates the transaction information using the GUID.

 CA SYSVIEW for DB2 is installed as normal
 Only specific DSQ Requests are needed for CE APM
– This will limit the data retrieved to only what was needed for
CE APM
 THRESHDS; DCL$A$D; DCL$E$P; DCL$Q$Z
 DSQPARMS; DSQAPMSS; DSQAPMGS – if datasharing
 History can be turned off as it is not used for CE APM
CA SYSVIEW for DB2 zOS – CA CE APM Only

 CA SYSVIEW for DB2 connects to CE APM using XNET
 XNET configuration is set in Cross-Enterprise_APM_Dynamic.properties
 Hostname
 Port
 Uses PASS Tickets for authentication
– Configuration set in Cross-Enterprise_APM_Dynamic.properties
– Can also hardcode a user id and password

 Does not require CA SYSVIEW to be installed
 Does not require integration with CA SYSVIEW be configured
 Does not provide trace data to CA APM
– DB2 data is included in the end to end application trace details when
CICS calls DB2.

 CA NetMaster does not use the CE APM Agent
 Integration to CA APM was developed independently using an
APM EPAgent
 Only TCPIP data is sent to CA APM
 The EPAgent must run from a Windows/Unix machine
– This is the “agent” that receives the data from CA NetMaster
– Default listening port is 8000
CA NetMaster TCPIP – CA CE APM Only Installation

 Connectivity parameters are set from within CA NetMaster
itself, not a .profile like other agents
– /PARMS
– F APMEPAGENT
 Set data collection on/off
 Set IP Address of EPAgent
 Set port for EPAgent (default 8000)

Installation Validation
 It is recommended to start CA SYSVIEW and/or CA SYSVIEW
for DB2 prior to starting the CA CE APM agent
 Start the CA CE APM agent task
– View the task log for messages during start-up
 Once the CA CE APM starts, it will attempt a connection to
the CA APM EM, CA SYSVIEW and CA SYSVIEW for DB2 (via
XNET) to begin requesting data

Installation Validation
 Using the CA APM Investigator you can verify if the CA CE APM agent
connected
– The CA CE APM agent will be visible in the Investigator tree of available agents
– You will see one agent per LPAR
– If there are only agent metrics available then the agent is connected to the EM but
not gathering metrics from CA SYSVIEW or CA SYSVIEW for DB2
 Using the CA APM Investigator, look for zOS, CICS, IMS, MQ, TCPIP and CA
Datacom metrics, as applicable, under the CA Cross-Enterprise APM agent
entry in the tree

Installation Validation – CA SYSVIEW
 Determine if the CA CE APM agent is connected to CA SYSVIEW
– In CA SYSVIEW, type the USERS command. Look for the CA CE APM
user name (usually WILYZOS)
– If present, type LISTL for the LISTLOG of this user to see that it is issuing
commands
– If not present
 Look in XSUSERS
 Look at the WILYZOS job log for any error messages
– Check ASADMIN
 Ensure XSXI and XSXS are running

Installation Validation – CA SYSVIEW for DB2 (Insight)
 Using the CA APM Investigator, look for DB2 metrics under the
CA Cross-Enterprise APM agent entry in the tree
 Determine if the CA CE APM agent is connected to
CA SYSVIEW for DB2 (Insight)
– There are no CA SYSVIEW for DB2 commands that validate
connectivity
– Review the CA CE APM agent task log for error messages
– Review the XNET task log for error messages

 CA CE APM Agent will not start
 CA SYSVIEW zfs directory structure no longer exists
 Agent connects to CA APM EM but there’s no data
 Agent not connected to EM
 No trace data present in Investigator
Common Errors and Resolutions

 This is extremely common. Besides basic zOS proc or security settings, always
check the variable paths in STDENV. Manually verify the variable paths match up
to were they are used in STDENV and the files in zfs are actually there
– JAVA_HOME
– SA_INSTALL
– SYSVIEW_PATH
 This is the most common mistake made. Use split screen and “f”ind to validate
the paths
 Often results in an “XAPI not found” type error
CA CE APM Agent Will Not Start

 Scenario: The CE APM agent is present in CA APM
Investigator but there is no zOS or subsystem data showing
 CA SYSVIEW
– Agent will start and connect to the CA APM EM even if it cannot
connect to SYSVIEW (and vice versa)
– Check CA CE APM agent log for errors
– Check ASADMIN for XSXS and XSXI
– Check USERS or XSUSERS for CA CE APM user name
 Default is WILYZOS
Agent Connects to CA CE APM but There’s No Data

 If CA SYSVIEW was restarted and CA CE APM agent was not,
this can sometimes throw them out of synch and agent will
not attempt a reconnection to CA SYSVIEW. Shutdown all
tasks, restart CA SYSVIEW before the agent
Agent Connects to CA CE APM but There’s No Data (continued)

 Scenario: The CA CE APM agent is present in CA APM
Investigator but there is no DB2 data showing
 CA SYSVIEW for DB2 (Insight)
– Check CE APM agent log for errors
– Check XNET log for errors
– Check .properties for PASSTICKET information
 Often try a real user’s id and password who have access to validate
connectivity. Often pass tickets are not configured correctly
Agent Connects to CA CE APM but There’s No Data

 In most cases the agents connect to an CA APM EM Manager of
Managers (MOM) which routes them to the most available server
 Validate the CA APM EM hostname and port is correct for the MOM
in the
Introscope_Cross-Enterprise_APM.profile
 Default APM EM listening port is 5001
 PING the hostname andor IP address of the CA APM
EM MOM from the LPAR in question
Agent Won’t Connect to EM or Disconnects

 Ensure firewall is open on the span of ports for the
MOM and individual EM’s. This is generally a range.
– You only specify the MOM in the .profile, but must have
access to all ports of all EM’s
Agent Won’t Connect to EM or Disconnects

 Transaction trace data only comes from CA SYSVIEW
 While DB2 information is available in the trace, it comes from the CICS
data, not from CA SYSVIEW for DB2.
 This can be a more difficult issue to debug as it spans platforms
 Validate application path has CA APM agents installed in all places
 Gaps prevent the GUID and trace indicator from passing along
– Validate PORTLISTs are setup in the CA SYSVIEW GROUPS command
– Validate the LOGGER (CICSLOGR and IMSLOGR) have matching
PORTLIST entries
No Trace Data Present

– For CICS
 validate a CORID is present in the CTRANLOG record
 Validate in CCONFIG WILY-INTROSCOPE is YES to indicate a trace
request
 Look at CICSLOGR for TCPSEND bytes count to determine if records
are being sent to the EM
No Trace Data Present

Understanding Transaction Tracing

CA APM CA CE APM Transaction Tracing
 CA Cross-Enterprise APM has the ability to provide application
details to CA APM in a transaction or cross-process trace
 Current supported methods are:
– Web Services to CICS
– CICS Transaction Gateway (CTG) to CICS
– MQ to CICS or IMS
– We need your input! IMS Connect? Sockets?
Supported Tracing Methods

 Transaction tracing is invoked through the CA APM interface
and must start with a non-mainframe agent
 We utilize the message header or application payload to
complete the trace request
– If you are using a custom middleware component we will need to
validate that it can pass this trace request
 The trace will follow the CICS transaction path on the same
LPAR or the first IMS transaction called
Supported Tracing Methods

 The CA APM transaction trace will present the CICS
transaction path on the same LPAR, showing the first program
executed and relevant timing information.
Trace Details

CA Performance Management Portfolio Vision
Unified Performance Management for z Systems Planned
Available
Distributed/Mainframe
Operations Analysts
MF Systems/Network
Analysts
Distributed
Application Analysts
Network Analysts
CA NetMaster®
Databases
Network
CICS & MQ
z/VM, KVM &
Linux on z
CA UIMCA UIM for z Systems
CA Cross Enterprise
APM
Integrated Mainframe
Management Console
CA APM
System Analysts
CA SYSVIEW®
Storage
Capacity Planner
CA MICS®
Storage Administrator
CA Vantage™

CA CE APM Product Integrations
 Alerts created from CA SYSVIEW are viewable in CA APM on a
typeview page
– zOS Alerts
– CICS Alerts
– IMS Alerts
– MQ Alerts
Using CA SYSVIEW with CA Cross-Enterprise APM

CA CE APM Product Integrations
CA SYSVIEW Alerts

Part of the bigger picture
CA SYSVIEW Event Capture
CA Mainframe Application
Tuner
CA OPS/MVS

Q & A

Recommended Sessions
SESSION # TITLE DATE/TIME
MFT56S
Infrastructure Performance Management: Flexibility
Combining Breadth, Depth and Speed
WED 11/18 at 3:00 pm
MFT15T
New Mainframe IT Analytics: Actionable Insight into
Root Cause Analysis of Performance Issues
WED 11/18 at 3:45 pm
MFT08S
Analyst Expert View: Mainframe in the Application
Economy - Gary Barnett Chief Analyst Ovum
THUR 11/19 at 10:30 am

Follow Conversations At…
DEMO STATION TECH TALKS
Demo Station: MNFSE009
Mobile to Mainframe App
Performance
Tech Talk MFT14T: WED 11/18 at 2:20 pm
Go Deep & Wide: No It’s Not a Football, It’s End-
to-End Performance Management
Tech Talk MFT15T: WED 11/18 at 3:45 pm
New Mainframe IT Analytics:
Actionable Insight into Root Cause Analysis of
Performance Issues

Because those who build the
apps will own the future ...
And those apps continue
to rely on the MAINFRAME

For More Information
To learn more, please visit:
http://cainc.to/Nv2VOe
CA World ’15

Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (7)

Ähnlich wie Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management

Ähnlich wie Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management (20)

Mehr von CA Technologies

Mehr von CA Technologies (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Best Practices for Optimizing Mainframe Application Performance with CA Cross-Enterprise Application Performance Management