NZS-4532 - Bringing Historical Data to Life with IBMs SMF Data Engine As presented at IBM InterConnect 2016

1. Bringing Historical Data to Life with
IBM’s SMF Data Engine
Alan Place
Senior Development Manager
Barry Klutz
zManage Offering Manager

2. Please Note:
1
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole
discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in
making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any
material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual
throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

3. SMF Introduction
2
• SMF data has been around for years - the bread and butter of the traditional Systems
Programmer
IBM System Management Facility
The standard way of writing z/OS records of activity to a file
Allows reporting or analysis of I/O, CPU utilisation, network activity, software usage, etc
• In today’s world there are pressures that are forcing us to rethink how we deal with this data
– Loss of skills
– Exponential increases in data volumes
– New technology bringing ever-more complicated metrics into the picture
– Economic pressures
This is the story of SMF at IBM and how we are using this
valuable data source to help customers run a more
efficient, cost effective business

4. SMF - Traditional Data Usage
3
Tivoli Decision Support for z/OS (TDSz)
• TDSz collects, organizes, and converts raw standard systems management data
into business relevant information that can help improve operational planning,
cost management, responsiveness, and decision-making process in organization
to effectively manage economic performance of the IT investment.
• Addresses performance reporting, capacity management, resource availability
management and accounting needs in an enterprise. Generates customized
reports for communicating and exchanging valuable information between
different departments in an enterprise.
• Removes customer pain points of high cost competitive tools and competitors’
proprietary database.
Accounting & Chargeback
Performance
Service Level Reporting

5. Extract, Categorize, Store
• Measure SLA compliance
• Quantify increased IT resource consumption or
abnormal spikes
• Compare trends to pinpoint where consumption
increased
• Converts raw data into business-relevant information
• Basis for mainframe accounting
Capacity Management Analytics (CMA)
Released in 2012 for a complete Capacity Management Solution
• Collection Engine (TDSz)
• Predictive Engine (SPSS)
• Reporting Engine (Cognos)
SMF – Beyond Traditional Reporting
4
Reporting
• Granularity / Statistical
• Forecasting / Prediction
• Application performance model
• Correlation of data / relationships
• Use beyond Capacity Management
Correlate & Forecast
• Granularity / Statistical
• Forecasting / Prediction
• Application performance model
• Correlation of data / relationships
• Use beyond Capacity Management

6. SMF – Using the Past to look into the Future
CMA uses predictive analytics to help
organizations use their data to make better
decisions by drawing reliable, data-driven
conclusions based on past and current events.
Future capacity requirements can be
forecasted to help ensure that sufficient
capacity is available when the business
needs it.
Capacity
Planning and
Forecasting
Question: Would I have enough capacity to handle my business growth in
the next three months?
5

7. SMF – Data as a Analytics tool
CMA Near real-time Anomaly Detection
Provides anomaly detection analysis on CICS transaction data. Helps customer find out which CICS
transaction is anomaly. And customer can use our result to tuning or fix problem of their production
environment.
6
Based on transaction CPU utilization and elapsed time
6

8. SMF – The TDSz Processing Engine
7
Input
Data
Update
Definition
TDSz Core Processing
Collects, manipulates, aggregates and stores log data
(SUM, AVG, MIN, MAX, etc)
Multiple options:
• Log archive
• Logstream
• SMF Buffer
• IMS Log
• Any fixed record
Record
Definition
Lookup
Tables
DB2
Log
Procedure
Record
Procedure
Combination allows logic to
be added (eg threshold
processing)
Assigns alternative values
to data or ranges (eg
peak/offpeak, MIPS rating,
banking applications)
Additional processing logic to
handle composite records (eg
IMS), split records, etc.
Special handling for complex or
different SMF formats
Defines what data is written
and the format
Bespoke language for defining
data types and SMF record
formatting

9. SMF – Using the data to create Insight
8
Tivoli Decision Support for z/OS 1.8.2
• Traditionally TDSz relied on customer experience and knowledge
• Using SMEs from z/OS, DB2, CICS and IMS the Key Performance metrics were created
• Cut down metrics to quickly check the state of the system
• Performance improved dramatically
• Simple reports to drill down into the problem areas

10. A Big Data view of SMF
9
IBM® DB2® Analytics Accelerator for z/OS (IDAA)
A high performance appliance
• Integrates IBM zSystems® infrastructure and IBM PureData™
• Powered by IBM Netezza® technology
Now we can have SMF Data in a repository for lightning fast queries
Storage is cheaper and allows data analysts to store months of data
Allows a deep dive of an event that may have happened weeks ago

11. SMF data as a SaaS Offering
10
IBM z Operational Insights
A New Way to Optimize your z Systems
Cut costs. Save time. No installation necessary.
Rapid, cloud-based analytics enable you to:
see quantified savings upfront
identify what actions to take next
compare your performance to others
Try the open beta today, for free! ibm.biz/try-zoi
Actionable insights for CICS, MQ , IMS & WAS in minutes, not hours - just add operational data.

12. SMF data as a SaaS Offering Benefits
11

13. Near Real Time SMF data
12
• SMF data can be processed more efficiently from a pure operational view
– A large North American bank cuts SMF logs every half an hour and collects/loads data using TDSz. Data is
available in DB2 within 50 minutes
Becomes an early warning system
No need to wait for the following day
• The logstream is another way of bringing data in sooner…………..

14. The SMF Data Engine
13
• Uses the standard SMF processing embedded in TDSz
• No DB2 pre-req
• Reads from the logstream
• Easily installed (hours instead of days)
• Highly targeted data customised for the problem being investigated
Available to any product with a need to process SMF data
TDSz reinvented as the standard way to read and process SMF data in
near real-time

15. Bringing SMF Data closer to Real Time
14
IBM Operations Analytics for z/OS Log Analyzer (IOAz)
Centralized,
Distributed, Cloud,
Resilient Architectures
Increase Data Volume
Logs,
Traces,..
Events
Metrics
Transactions
ConfigCore files
Contains the first production version of the SDE to provide insight into why messages are
happening and potential solutions to reduce MTR
The Challenge - Find the right needle in
one of many haystacks – QUICKLY
IOAz was created as a debug aid as part of the Predict, Search and Optimize strategy that
underpins the whole of the zManage portfolio

16. SMF Data - Near Real Time in Action
15
IBM Operations Analytics for z/OS Log Analyzer

17. The Power of Targeted SMF
16
Use Case: System Programmer wants detailed performance data
for a specific period of time to diagnose a performance problem.
Alert Request
Specific, tailored
definitions for type of
problem type
SDE
Batch
Job
Submit
DB2
Load
CSV
Output
destination and
format built into
requestTDSzReports
Locate
Start/end time
Record types and
fields
SMF
Data
Engine

18. The Future of SMF data
17
SDE Common Approach
Logstream
Hot
Data
Tier 1
Long term
Trending
Data
Short term
Analytical
Data
Near Real Time Tier 2
Real Time
BatchLog
Archive
Tier 3
SMF
Direct
• Single mechanism for handling all SMF data
• Read once – write many
• Designed to function in both online and batch mode.
Has the flexibility to target data and populate all three data tiers while keeping CPU consumption under control.
Multiple insights
from multiple
products

19. Other InterConnect Sessions
18
Tuesday 23-02-16 8:30am - 9:30am
4555 IT Analytics Keynote: IT Analytics for the Enterprise Barry Klutz
Tuesday 23-02-16 4:00pm - 5:00pm
2051 A New Era of MainframeOptimization: IBM Operational Insights for z Systems
Steven Horsman
Wednesday 24-02-16 8:30am - 9:30am
4561IBM OperationsAnalytics for z SystemsClient Experience Barry Klutz
Thursday 25-02-16 10:30am - 11:15am
5063 IT OperationsAnalytics = Bridging Business and IT Ann Dowling

20. Notices and Disclaimers
19
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or
their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

21. Notices and Disclaimers Con’t.
20
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

22. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.