Best practices for WebSphere Application Server monitoring

1. WebSphere Application
Server - Operational Best
Practices
Techniques in Performance Tuning &
Tools for Problem determination

2. Agenda
› Top 12 tips for performance tuning
› Tools for Problem Determination
– XCT
– PMI & JMX aggregation tools
– WAIT
– Health Center
– Memory Analyzer
– Edge monitoring

3. Top 12 Tuning recommendations
› Choose and Tune the Caches Correctly
› Understand and Follow Multi-core & Virtualization Guidelines
› Properly Tune the Operating System
› Keep Application Logging to a Minimum
› Understand & Tune Infrastructure
› Minimize HTTP Session Content
› Correctly Tune Thread Pools
› Tune JDBC Data Sources
› Create Cells To Group Like Applications
› Ensure Uniform Configuration Across Like Servers
› Correctly Tune The JVM
› Perform Proper Load Testing

4. Cross Component Tracing (XCT) – WAS 8.5
› Log/trace correlation technology.
› Determine which log/trace entries are part of each request.
› Recommended mode - Request ID and correlation log record mode
› Works best in combination with High Performance Extensible Logging
– HPEL stores XCT Request IDs
– HPEL can filter log/trace records by XCT Request ID
› Some overlap with Request Metrics
– PMI RM is used for performance tracking.
– XCT instrumentation is sparse mostly to track where requests change threads &
processes
› Allows you to see full chain of requests & how each tier responded
based on the XCT entries within the logs
› XCT Correlation Log Viewer tool in ISA

5. External Tools to consume PMI & JMX data
› ITCAM
– Collecting the historical data from the monitored servers
– Modeling, Identifying baselines and thresholds from the historical data
– Creating situations with appropriate severity
– Configuring actions and policies for automation when situations are raised
› Performance Tuning Toolkit (ISA download)
– Performance tuning & monitoring – servlet, ejb, jdbc, transaction, pools thread,
connection, sessions, JVM
– Health Check & predefined alerts and actions
– Operations & reporting engine
› WASMON (available for free from ISA)
– CSV based data collection utility
– Collects Version Details, Key performance settings & Performance data for
defined PMI counters

6. IBM Whole-system Analysis of Idle Time Tool
› Data collector scripts to collect JVM and system performance
› Information on multiple processes captured at the same with sleep time between
iterations
› At the end of the collection interval, generated data is compressed file and uploaded to
the data submission page.
› Advantages:
– Identify bottlenecks
– Zero install time and convenient reporting
– Non-disruptive
– Low-overhead
– Simple to use
– Centralized knowledge base
› https://wait.ibm.com/wait/public/showcase/1/
› https://wait.researchlabs.ibm.com/
› Start the WAIT data collector:
https://wait.researchlabs.ibm.com/submit/dataCollector.html

7. IBM Health Center
› Low overhead sampling profiler that runs in the IBM JVM
› Provides deep-dive information on method profiling, garbage
collection, I/O, lock analysis, threads, native memory …
› Health Center Agent & Client
– Agent ships with IBM JVM - is a JVMTI native library enabled using
generic JVM argument -Xhealthcenter
– Client ships with ISA - is an Eclipse-based GUI perspective used to
analyze agent data
› Best way to run the agent is in the headless mode
– Agent writes data to local file system that can later be loaded offline
– Ensure that the latest binaries of the agent are installed
– -Xhealthcenter:level=headless

8. Memory Analyzer Tool & IBM Extensions
› Memory leak detection &
footprint analysis
– Shallow heaps, retained
sets, finding objects, top
consumers, object query
language, multiple types
of leaks, thread stacks &
frame locals, java
extensions
› System /WAS dumps are
now default starting WAS
8002
› Use 64 bit version
shipped with ISA
› IEMA provides WAS and
Java specific knowledge:
– WAS overview, HTTP
Session Analysis, Thread
Pool Analysis, Web
Application Analysis,
Application classloader
leaks

9. IHS Monitoring Best Practices
› Always use %D - time taken to serve the request, in microseconds
– Others - %{JSESSIONID}C %{JSESSIONID}o %{Host}i SSL=%{HTTPS}e %{HTTPS_CIPHER}e
%{HTTPS_KEYSIZE}e %{HTTPS_SECRETKEYSIZE}e
› Mpmstats enabled by default - snapshot of what is happening within IHS.
– Similar to mod_status which can be requested through HTTP
– Watch bsy and wr to see if WAS is becoming backed up
– Set a ReportInterval of 30 seconds in case of problems
› Absolute maximum simultaneous requests is limited by the MaxClients settings
– MaxClients(default 600) = ThreadsPerChild * ServerLimit (1 single process model)
› Use ab (Apache Bench) from IHS bin dir to spot check backend server
performance & compare two different environments
– ab -n 1000 -c 10 http://localhost:9080/
› Monitor connections between WAS and Plugin using tools and scripts
– https://github.com/covener/plugin-tools/blob/master/README
– http://ibm.co/NtFZuH