The document summarizes a case study on tuning software costs for a large international bank. It describes how the consultant analyzed the client's workload profile, identified areas of high resource consumption like a CICS region with high RACF overhead, and made recommendations such as reducing security checks and spreading batch jobs to off-peak hours to lower software bills. Modeling showed how rescheduling backups could save an additional 20% of batch costs.
Boost PC performance: How more available memory can improve productivity
CMG 2012 - Tuning where it matters - Gerry Tuddenham
1. CMG Brazil Conference 2012
Tuning where it matters – a case study
Prepared by CPT Global Limited
Presenter Gerry Tuddenham
Telephone: +61 418 372 999
Email: gtuddenham@cptglobal.com
Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
2. PRESENTATION CONTENT
• Set the scene – the Client, the issue
• How we targeted the reduction efforts
• Approach taken – top down
• Examples of things we looked for / found
• Examples of changes recommended
• Where we are going next
• Results
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
3. SETTING THE SCENE
• Client is a large International bank
• Completely different culture
• Very risk-averse, and change averse
• Very untrusting of non-French speaking “experts”
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
4. THE ISSUE
• Growing monthly SCRT based software cost
• Approaching the point of requiring a large upgrade
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
5. TECHNICAL ENVIRONMENT
• Client had recently centralised a number of region
data centres onto 2 CECs in 2 data centres in a
central location
• Consolidation had just been physical, so LPARs
were still regional, though mainly ran the same
applications with regional flavours
• Throughout the project the client has been
migrating workloads and moving to a non-regional
configuration
• Applications mainly CICS / COBOL / DB2, though
some CICS / VSAM
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
6. TUNING FOCUS
Typical Challenge Approach
• Capacity Issue • Move workloads to quieter times
Run own • Tune what is left
systems • Reduce software bill • Need to focus on the peak 4HRA and
smoothing the workload
• Billed based on peak hour • Move workloads to quieter times
• Make outsourcer move housekeeping
Outsourced • Tune what is left
• Billed based on total • Tune
consumption
• Customer running out of • Move workloads to quieter times
capacity • Tune what is left
Outsourcer
• Customer pays for application • Only tune what is not billed, don’t tune
address spaces only system-wide parameters
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
7. TOP DOWN, FOCUSED APPROACH
• Understand the goals of tuning
• Understand the workload profile (RMF70, RMF72)
• Decide on area of focus
• Identify the big burners (SMF30)
• Use a sampling tool to understand what they are
doing
• Build and implement recommendations
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
8. FOCUS WHERE IT MATTERS
Data for CEC « B », Batch
the other CEC has a Peak monthly 4HRA driven TSO
similar profile Online
by online morning peak STC
6000 System
Uncaptured
5500 4HRA
5000
4500
4000
3500
3000
MIPS
2500
2000
1500
1000
500
0
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Fri 0h
Thu 0h
Mon 0h
Tue 0h
Wed 0h
Hourly intervals for week starting 2 December 2008
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
9. FOCUS WHERE IT MATTERS
Peak monthly 4HRA driven Area of focus
by online morning peak 4HRA
6000
5500
5000
4500
4000
3500
MIPS
3000
2500
2000
1500
1000
500
0
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Thu 0h
Mon 0h
Fri 0h
Tue 0h
Wed 0h
Hourly intervals for week starting 2 December 2008
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
10. FOCUS WHERE IT MATTERS
Batch
7000 Batch Large online peak Large midday TSO
Online
6500 throughout day batch suite STC
System
6000 Uncaptured
5500 4HRA
5000
4500
4000
3500
MIPS
3000
2500
2000
1500
1000
500
0
08h00
08h30
09h00
09h30
10h00
10h30
11h00
11h30
12h00
12h30
13h00
13h30
14h00
15 minute intervals for Tuesday 3 December 2008
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
11. INITIAL SAMPLE – LARGEST CICS REGION
High RACF overhead
MODULE SECTION PROCEDURE/FUNCTION CPU TIME PERCENT
NAME NAME NAME TOTAL
.SUPERVS IGC0013{ IRRRFC23 SUPERVISOR SERVICES 17.78
.NUCLEUS IEAVESL SUSPEND LOCK SERVICE 7.36
.DB2 DSNK2DM DSNKNXT2 FETCH NEXT TYPE 2 INDEX 4.64
.DB2 DSNK2DM DSNKFTCH FETCH TYPE 2 IDX ENTRY 2.61
.CICS DFHSIP DFHTRPX TR DOM-TRC PUT-FAST PATH 2.51
.DB2 DSNBBM DSNB1GET RETRIEVE REQUESTED PAGE 1.38
.LELIB CEEPLPKA CEEV#GTS GET HEAP STORAGE 2.00
.DB2 DSNIDM DSNIRNXT FETCH NEXT ROW TO PROG 1.37
.DB2 DSNIDM DSNIONX2 NEXT ON CUB ON IXED FAN 1.02
CICS Trace active Lots of DB2 activity LE Storage Parameters
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
12. REDUCING THE RACF OVERHEAD
• SIT security parameters
• Check the Xxxx parameters in SIT
• XPPT and XFCT are very expensive
• If they are set, ask why
• MRO session security
• With ATTACHSEC = IDENTIFY on a CICS MRO link,
CICS will perform 2 RACF checks at transaction attach:
one for the incoming userid, another for the session userid
• Letting the SESSION userid default to the region default userid
reduces this to 1
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
13. FOCUS WHERE IT MATTERS - 2
6000 Peak now overnight Area of focus
batch 4HRA in December 2008
5500 4HRA
5000
4500
4000
3500
3000
MIPS
2500
2000
1500
1000
500
0
Thu 0h
4h
8h
4h
8h
4h
8h
4h
8h
4h
8h
12h
16h
20h
12h
16h
20h
12h
16h
20h
12h
16h
20h
12h
16h
20h
Tue 0h
Wed 0h
Fri 0h
Mon 0h
Hourly intervals for week starting 2 March 2008
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
14. DEALING WITH BATCH
• Spread what you can into white space
• Tune or squeeze the rest
• The difficult bit is convincing the operations area
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
15. MODELING CHANGES TO BACKUPS
BATMD load shown
8500 Other Batch Medium over both CECs
8000 Consumption of ADRDSSU backups
7500
7000
6500
Batch all kicks off at 19h00
6000 Backups
5500
around 20% of
5000
4500
batch
4000
MIPS
3500
3000
2500
2000
1500
1000
500
0
Thu 0h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Tue 0h
Fri 0h
Wed 0h
Mon 0h
Hourly intervals for week starting 30 March 2009
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
16. MODELING CHANGES TO BACKUPS
8500 Original position
8000 Other Batch Medium (modeled)
7500
Modeled consumption of rescheduled backups
80% of backups now
7000 delayed until 02h00
6500
6000
Saving in hour
5500
5000
4500
4000
MIPS
3500
3000
2500
2000
1500
1000
500
0
Thu 0h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Tue 0h
Fri 0h
Wed 0h
Mon 0h
Hourly intervals for week starting 30 March 2009
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
17. MODELING CHANGES TO BACKUPS
8500 Original position
8000
Other Batch Medium (modeled)
7500
Modeled rescheduled & uncompressed backups
Assume 50%
7000 compression removed
6500
6000
5500
5000
4500
4000
MIPS
3500
3000
2500
2000
1500
1000
500
0
Thu 0h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Tue 0h
Fri 0h
Wed 0h
Mon 0h
Hourly intervals for week starting 30 March 2009
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
18. MODELING CHANGES TO BACKUPS
8500 Original position
8000 Unsqueezed Batch Medium (modeled)
7500 Squeezed Batch Medium (modeled) BATMD squeezed to
Modeled rescheduled & uncompressed backups
7000 5000 MIPS
6500
6000
5500
5000
4500
4000
MIPS
3500
3000
2500
2000
1500
1000
500
0
Thu 0h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
4h
8h
12h
16h
20h
Tue 0h
Fri 0h
Wed 0h
Mon 0h
Hourly intervals for week starting 30 March 2009
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
19. SYSTEM SETUP - MONITORS & TRACES
• The way to reduce CICS Trace overhead is to turn
off the all options you can
• Default for standard trace in CICS is level 1 for all
components (STNTRxx = 1 in SIT)
• Trace points from most of these domains are of no
use in debugging application problems
• For most sites application problems can be
diagnosed with trace running for application (AP),
EXEC Interface (EI), and File Control (FC) domains
• TMONCICS turns on some trace points (EI, DS, MN,
and ISC).
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
20. SYSTEM SETUP - MONITORS & TRACES
• The only thing you can go with monitors is try to
switch off as many options as you can get away with
TXSTGCOL means that
TMON645 < 4720 .02 .
TMON912 > 968 .17 . TMON GLOBAL TRACE
TMON924 > 5128 .21 . is monitoring transaction
TM2N950 > 11288 .19 .
TM2N952 > 24032 .02 .
storage utilisation. Most
TM2N958 > 8032 .10 . sites won’t be interested
TM2N960 TM2N960 > 13808 3.00 .*** in this, and the default is
TM2N969 > 2336 .08 .
TM2N973 > 4032 .36 . ON
TX2EIIN TXEIIN > 7930 2.28 .**
TX2EIOUT TXEIOUT 68038 1.20 .*
TX2EIOUT TXSTGCOL 856 2.17 .**
TX2EIOUT TXUCFAT 2288 .34 .
-----
TX2EIOUT TOTALS > 71182 3.71
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
21. INVESTIGATING HIGH DB2 ACTIVITY IN CICS
• Use your sampling tool to identify high burning
DBRMs
** SQL CPU USAGE SUMMARY **
SQL SQL STMT EXECUTION TIME/COUNT % CPU TIME
TYPE NAME CNT AVG-TIME TOTAL
……
DBRM NK1K600T 30,306 .0000 09:06:59 21/01/2005 .11 .
DBRM NK1NT0AT 194 .0035 16:31:23 13/04/2005 .12 .
DBRM NK1NT0IT 45 .0043 08:36:15 06/08/2007 .05 .
DBRM NK1NT0NT 18,860 .0027 07:16:23 21/07/2008 8.94 .*************************++++
DBRM NK1NT0OT 17,312 .0027 13:57:17 15/04/2005 11.00 .****************************++++++
DBRM NK1NT00T 0 .0000 08:36:59 06/08/2007 .02 .
DBRM NK1NT03T 131 .0163 15:50:43 09/12/2004 .58 .*
DBRM NK1NT14T 2 .4020 15:07:11 02/06/2006 .22 .
DBRM NK1U30ET 7,147 .0001 15:39:28 27/02/2008 .06 .
…
• Use DB2 Explain to review, and if possible improve
the access path
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
22. FOCUS ON APPLICATION CODE
• Cover simple code changes
• Look for tight loops, definition of index variables
MODULE SECTION PROCEDURE/FUNCTION STARTING PROCEDURE CPU TIME
NAME NAME NAME LOCATION LENGTH TOTAL
.LELIB CEEBINIT CEEVGTSI GET A STACK INCREMENT 13.36
.COBLIB IGZCPAC IGZCFPC COBOL LIBRARY SUBROUTINE 1.97
.COBLIB IGZCPAC IGZCNMV NUMVAL/NUMVAL-C FUNCTION 1.93
.COBLIB IGZCPAC IGZCFCC DYNAMIC CALL MANAGER 1.88
.COBLIB IGZCPAC IGZCFPCX COBOL LIBRARY SUBROUTINE 1.74
RK3DJAMV RK3DJAMV 0292A8 2 1.10
RKJB20 RKJB20 0806EC 2 1.10
RKJB20 RKJB20 0807BC 2 1.06
RK3DJAMV RK3DJAMV 0292D0 2 1.01
RKJB20 RKJB20 019BEC 2 .92
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
23. FOCUS ON APPLICATION CODE
• Offset points to area of code doing basic index
function
IF WSOL-NUCTE-SR > 900 AND < 961
MOVE WPRO TO WPRO5
END-IF.
IF ( WSOL-NUCTE-SR > 200 AND < 225 ) OR WSOL-NUCTE-SR = 298
MOVE WSCL TO WSCL5
END-IF.
• Variables are declared as characters
20 WSOL-NUCTE PIC 9(10) VALUE 0.
20 FILLER REDEFINES WSOL-NUCTE.
25 WSOL-NUCTE-SR PIC 9(3).
25 WSOL-NUCTE-RG PIC 9(7).
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
24. LINGERING RACF OVERHEAD
• RACF overhead is still high
• Noticed odd user sign-on behaviour in CICS logs
• Investigated further with Type 80 RACF records
• Spotted sign-on trend
• Piece together user behavior
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
25. ACCESSING CICS FROM BRANCH DESKTOP
Vtam / GOR Many AORs
TELNET
Application
on desktop
TOR
GOR
CTGs WOR
Websphere
on AIX
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
26. LOGON VIA GOR
Vtam / GOR Many AORs
Connects to TELNET
mainframe Autoinstall Signon
IPADDR : Port NETNAME TERMID ACEE
TXN1
No TERMID
Ship Signon
Single
TERMID
business
transaction TERMID ACEE
TXN1
TXNn
TXNn
CESF
Breaks Delete Signoff
connection
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
27. LOGON VIA GOR
Vtam / GOR Many AORs
TELNET
TERMID ACEE
IPADDR : New = NEW = NEW = NEW
Port number NETNAME TERMID ACEE
OVERHEAD
TXN1
No TERMID
Next Ship
Business TERMID
Transaction = NEW = NEW
TXN1
TERMID ACEE
TXNn
TXNn
CESF
Breaks Delete Signoff
connection
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
28. IDEAL LOGON VIA GOR
Vtam / GOR Many AORs
TELNET
First NETNAME TERMID ACEE
business
transaction TXN1
Ship TERMID
TERMID ACEE TXN1
TXNn
TXNn
Single
TXNn
Business
Day TXNn
Subsequent TXNn
business TXNn
transactions
TXNn
TXNn
TXNn
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
29. MEASURING RESULTS ON A REGION BASIS
• A key success point for early opportunities was building
confidence within the technical and operational areas
• Opportunies were measured at a very granular level to prove
the effectiveness
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
30. MEASURING SYSTEM WIDE RESULTS
Transactions per hour
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
31. SUMMARY OF OPPORTUNITIES RAISED
Number of Savings to 4HRA Savings to 4HRA
Opportunities at midday in batch peak
Move
6 200 MIPS 890 MIPS
workloads
System
43 1,940 MIPS 210 MIPS
settings
Application
29 244 MIPS 330 MIPS
changes
Access path
34 1,010 MIPS 110 MIPS
improvements
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
32. RESULTS OVER THE PROJECT LIFETIME
CICS CPU hours consumed in the month
Millions of transactions in the month
Start of tuning
implementations
+ 35%
- 13%
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
33. CONTINUE TO FOCUS WHERE IT MATTERS
Less batch Peak load around 800 Large midday Batch
7000 TSO
throughout MIPS lower running 38% batch suite Online
6500
the day more transactions now at 13h00 STC
System
6000 Uncaptured
5500 4HRA
5000
4500
4000
3500
MIPS
3000
2500
2000
1500
1000
500
0
08h00
08h30
09h00
09h30
10h00
10h30
11h00
11h30
12h00
12h30
13h00
13h30
14h00
15 minute intervals for Tuesday 2 March 2010
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
35. SUMMARY OF OPPORTUNITIES RAISED
Number of Savings to 4HRA Savings to 4HRA
Opportunities at midday in batch peak
Move
6 200 MIPS 890 MIPS
workloads
System
43 1,940 MIPS 210 MIPS
settings
Application
29 244 MIPS 330 MIPS
changes
Access path
34 1,010 MIPS 110 MIPS
improvements
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
36. A KEY TO SUCCESS - AREAS REVIEWED
We covered everything that runs on Other vendors concentrate on application
the system code base only
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
37. SUMMARY
• You need to establish the business reason for
tuning
• Focus where it will achieve the impact you want
• Start with big initiatives to gain momentum
• Take an iterative approach
• Track & publish results
• Keep reminding yourself what the goal is
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.
38. WHAT BENEFIT WILL CPT ENGAGEMENT
DELIVER?
• Your organization will make more profit
• Your organization will be able to grow without acquiring
more hardware
• The service that you provide to your customers will be in
more scalable, reliable and cost effective way
•Proibida cópia ou divulgação sem
permissão escrita do CMG Brasil.