Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Nächste SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Wird geladen in …3
×
1 von 55

Monitoring and problem determination of your mq distributed systems

1

Teilen

Herunterladen, um offline zu lesen

MQ monitoring presentation, covering events, system topics, administrative commands and more.

Ähnliche Bücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Monitoring and problem determination of your mq distributed systems

  1. 1. Integration Technical Conference 2019 ©2019IBMCorporation M07: Monitoringand problem determination of your MQ distributed systems MatthewWhitehead mwhitehead@uk.ibm.com
  2. 2. 1Integration Technical Conference 2019 ©2019IBMCorporation Agenda – the MQ Toolbox • What’s connected? • Re-routing in a cluster • Are messages flowing? • Monitoring activity over time • Where are messages going? • What are the apps doing? • How can I look back in time? NB: Capabilities differ in form between distributed and z/OS. This presentation is distributed focused Presentation contains MQSC example commands and output. Your own admin tools will vary by task, situation and preference. Tasks can be performed in PowerShell, MQ Explorer etc. as desired.
  3. 3. 2Integration Technical Conference 2019 ©2019IBMCorporation What’s connected?
  4. 4. Integration Technical Conference 2019 3©2019IBMCorporation DISPLAY CONN DISPLAY CONN(*) TYPE(HANDLE) ALL AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(HANDLE) OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE) ASTATE(NONE) HSTATE(INACTIVE) OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING) READA(NO) OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE) ASTATE(ACTIVE) HSTATE(ACTIVE) OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING) READA(NO) AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(CONN) PID(9740) TID(185) APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF) APPLTYPE(SYSTEM) ASTATE(NONE) CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1) CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad) UOWLOG( ) UOWSTDA(2014-04-08) UOWSTTI(13.24.00) UOWLOGDA( ) UOWLOGTI( ) URTYPE(XA) EXTURID(XA_FORMATID[DSAW] XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809] XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000 000000000001]) QMURID(0.7940075) UOWSTATE(ACTIVE) Use CONN to match TYPE(CONN) and TYPE(HANDLE) records TYPE(HANDLE) records let you find applications by the objects they access. See all open handles for an app in one place, unlike DIS QSTATUS records Long running UOW information. XID can be tied up with app server txn timeout Channel name + IP help identify client apps. MQ V7.5 and later JMS clients can supply an application name in the CF. MQ V9.1.2 and later, all bindings and client applications can provide a custom application name DISPLAY CONN(*) ALL All platforms
  5. 5. Integration Technical Conference 2019 4©2019IBMCorporation DISPLAY CHSTATUS DISPLAY CHSTATUS(*) ALL AMQ8417: Display Channel Status details. CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN) BUFSRCVD(17) BUFSSENT(13) BYTSRCVD(2296) BYTSSENT(2456) CHSTADA(2014-04-08) CHSTATI(15.26.59) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1) CURRENT EXITTIME(0,0) HBINT(5) JOBNAME(0000260C000000B9) LOCLADDR( ) LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59) MCASTAT(RUNNING) MCAUSER(pbroad) MONCHL(OFF) MSGS(6) RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example) SSLKEYDA( ) SSLKEYTI( ) SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) CURSHCNV(1) MAXSHCNV(1) RVERSION(00000000) RPRODUCT(MQJM) See SSLPEER information not in DIS CONN JOBNAME contains PID (except z/OS): 0x260C = PID(9740) On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185) Note that multiple CONN might share one SVRCONN channel instance Check suitable heartbeats are negotiated All platforms
  6. 6. 5Integration Technical Conference 2019 ©2019IBMCorporation Rerouting messages if nobody is connected
  7. 7. Integration Technical Conference 2019 6©2019IBMCorporation Cluster WLM • Cluster WLM algorithm doesn’t check whether consuming applications are connected to cluster queues • i.e. whether something is getting messages from them • You need to ensure that applications are consuming from all instances of a clustered queue to prevent messages building up • However, there is an alternative: AMQSCLM Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Cluster WLM App server 2 IPPROCS(0) IPPROCS(>=1) Round-robin message distribution IPPROCS(0) Sending appsSending appsSending apps All platforms
  8. 8. Integration Technical Conference 2019 7©2019IBMCorporation AMQSCLM: 1) Normal operation Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM Round-robin message distribution App server 1 App server 2 App server N . . . IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM AMQSCLM AMQSCLM All platforms
  9. 9. Integration Technical Conference 2019 8©2019IBMCorporation AMQSCLM: 2) Application failure Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) Messages start building up AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution Sending appsSending appsSending apps All platforms
  10. 10. Integration Technical Conference 2019 9©2019IBMCorporation AMQSCLM: 3) Failure Detection Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM checks the queue every 5 minutes and detects IPPROCS(0) AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution Sending appsSending appsSending apps All platforms
  11. 11. Integration Technical Conference 2019 10©2019IBMCorporation AMQSCLM: 4) Redirect Messages Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM reduces queue priority in cluster, to stop new messages arriving AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution Sending appsSending appsSending apps All platforms
  12. 12. Integration Technical Conference 2019 11©2019IBMCorporation AMQSCLM: 5) Un-maroon Msgs (optional) Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM tells qmgr to redistribute ‘stuck’ messages via the cluster AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution Sending appsSending appsSending apps All platforms
  13. 13. 12Integration Technical Conference 2019 ©2019IBMCorporation AMQSCLM summary • The cluster queue monitoring sample program (AMQSCLM) • Shipped with the product as a sample • Precompiled • Source code • Not shipped on z/OS, but distributed source code can be compiled and used on z/OS • More information in the MQ Knowledge Center
  14. 14. 13Integration Technical Conference 2019 ©2019IBMCorporation Are messages flowing?
  15. 15. Integration Technical Conference 2019 14©2019IBMCorporation Real-time/online monitoring – queues • Set detail level for queue manager. Override for individual queues • LOW, MEDIUM and HIGH set frequency of sampling, not type of stats generated • Gives live view of application responsiveness ALTER QMGR MONQ(MEDIUM) ALTER QLOCAL(QUEUE1) MONQ(HIGH) ALTER QLOCAL(QUEUE2) MONQ(OFF) AMQ8450: Display queue status details. QUEUE(QUEUE1) TYPE(QUEUE) CURDEPTH(16) IPPROCS(3) LGETDATE(2014-04-08) LGETTIME(17.05.59) LPUTDATE(2014-04-08) LPUTTIME(17.12.16) MEDIALOG( ) MONQ(HIGH) MSGAGE(112) OPPROCS(5) QTIME(10101414, 10101414) UNCOM(NO) DIS QSTATUS(QUEUE1) ALL Age in seconds of the oldest message on the queue Estimations of the time in microseconds that messages are waiting on the queue for processing. First value: Calculated from recent activity Second value: Calculated from longer term activity Timestamps of last PUT/GET to check for recent activity Without MONQ you only get the depth and how many handles are open All platforms
  16. 16. Integration Technical Conference 2019 15©2019IBMCorporation Real-time/online monitoring – channels Gives live view of channel throughput ALTER QMGR MONCHL(MEDIUM) MONACLS(MEDIUM) ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH) AMQ8417: Display Channel Status details. CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR) BATCHES(52) BATCHSZ(50) BUFSRCVD(55) BUFSSENT(1616) BYTSRCVD(1748) BYTSSENT(1192330) CHSTADA(2014-04-08) CHSTATI(17.49.03) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34) CURMSGS(50) CURRENT CURSEQNO(11823) EXITTIME(0,0) HBINT(5) INDOUBT(YES) JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557)) LONGRTS(999999999) LSTLUWID(0107445310001B33) LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51) LSTSEQNO(11773) MCASTAT(RUNNING) MONCHL(MEDIUM) MSGS(1580) NETTIME(137538,29555) NPMSPEED(FAST) RQMNAME(GATEWAY2) SHORTRTS(180) SSLCERTI( ) SSLKEYDA( ) SSLKEYTI( ) SSLPEER( ) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE) XQMSGSA(112) XQTIME(545784,3929968) RVERSION(07050002) RPRODUCT(MQMM) DIS CHSTATUS(MQHUB.GATEWAY2) ALL Short/long term calculations of how full your batches are getting, to help you tune BATCHSZ/BATCHINT Last time a message was sent over the channel Depth of messages on XMITQ for this channel (capped at 999) Short/long term calculations of how long messages are waiting on the XMITQ for transmission All platforms
  17. 17. 16Integration Technical Conference 2019 ©2019IBMCorporation Monitoring activity over a period of time
  18. 18. Integration Technical Conference 2019 17©2019IBMCorporation Accounting, statistics, and system topics MQ provides various mechanisms for understanding how a queue manager is being used Some are interactive commands as seen in the previous slides… • DISPLAY QMSTATUS() • DISPLAY QSTATUS() • DISPLAY CHSTATUS() • DISPLAY TPSTATUS() • DISPLAY CONN() Others are streams of data to consume… • Accounting & statistics messages • System topics Better suited to monitoring over a period of time
  19. 19. Integration Technical Conference 2019 18©2019IBMCorporation Accounting, statistics, and system topics Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE SYSTEM.BASE.TOPIC SYSTEM.ADMIN.TOPIC ‘$SYS/MQ’ user space MQ’s space Accounting & Statistics messages System Topics • Enabled by turning on certain types of event message on the queue manager • Turn on and off for specific queues and channels • Event messages build up on system queues for you to consume • No need to turn on – just subscribe to the correct topic • Better suited to multiple consumers who all want the same events • Provides more than just MQ data (system CPU, disk IO stats) • More granular authorisation Use SMF on z/OS Not on z/OS
  20. 20. Integration Technical Conference 2019 19©2019IBMCorporation Accounting, statistics, and system topics Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE SYSTEM.BASE.TOPIC SYSTEM.ADMIN.TOPIC ‘$SYS/MQ’ user space MQ’s space Accounting & Statistics messages System Topics • Enabled by turning on certain types of event message on the queue manager • Turn on and off for specific queues and channels • Event messages build up on system queues for you to consume • No need to turn on – just subscribe to the correct topic • Better suited to multiple consumers who all want the same events • Provides more than just MQ data (system CPU, disk IO stats) • More granular authorisation They provide a lot of overlapping information, but also have some differences Use SMF on z/OS Not on z/OS
  21. 21. Integration Technical Conference 2019 20©2019IBMCorporation Accounting, statistics, and system topics Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE SYSTEM.BASE.TOPIC SYSTEM.ADMIN.TOPIC ‘$SYS/MQ’ user space MQ’s space Accounting & Statistics messages System Topics • Enabled by turning on certain types of event message on the queue manager • Turn on and off for specific queues and channels • Event messages build up on system queues for you to consume • No need to turn on – just subscribe to the correct topic • Better suited to multiple consumers who all want the same events • Provides more than just MQ data (system CPU, disk IO stats) • More granular authorisation Use SMF on z/OS Not on z/OS
  22. 22. Integration Technical Conference 2019 21©2019IBMCorporation Accounting and statistics overview • Monitoring data sent as a PCF message at a configured interval • Statistics – scoped to a queue / channel / QMGR • Accounting – scoped to an individual CONN and queue / QMGR PCF accounting collection app Queues Channels CONNs PCF statistics collection app Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE Dist platforms Use SMF on z/OS New options in V9 Related attributes • ACCTQ • ACCTMQI • ACCTINT Related attributes • STATQ • STATMQI • STATCHL • STATACLS • STATINT
  23. 23. Integration Technical Conference 2019 22©2019IBMCorporation PCF accounting collection app PCF statistics collection app Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE MS0P Explorer Plugin (SupportPac) amqsmon Specifically for accounting and statistics amqsevt Generic PCF event message formatter, TEXT or JSON output Other • Tivoli • Custom app • E.g. Java app using com.ibm.mq.headers e.g. Accounting and statistics overview Dist platforms Use SMF on z/OS New options in V9
  24. 24. Integration Technical Conference 2019 23©2019IBMCorporation Taking a look at statistics using amqsmon • Overall QMGR busyness • Simple data format • Multiple values are [Persistent, NonPersistent] • One message every X seconds • Use amqsmon directly (perl/cron) • Low/high water marks for subscriptions • Grouped by subscription type • amqsmon is a sample so you can use it as a base for your own tools ALTER QMGR STATMQI(ON) Wait a bit, but not the default 30 minutes between stats records RESET QMGR TYPE(STATISTICS) amqsmon -m GATEWAY1 -t statistics -a -w 0 MonitoringType: MQIStatistics QueueManager: 'GATEWAY1' IntervalStartDate: '2014-04-09' IntervalStartTime: '00.00.35' IntervalEndDate: '2014-04-09' IntervalEndTime: '00.01.13' CommandLevel: 700 ConnCount: 35 PutCount: [271, 0] PutFailCount: 0 Put1Count: [2, 0] Put1FailCount: 0 PutBytes: [273976, 0] GetCount: [270, 0] GetBytes: [269468, 0] GetFailCount: 19 DurableSubscriptionHighWater: [0, 0, 0, 0] DurableSubscriptionLowWater: [0, 0, 0, 0] NonDurableSubscriptionHighWater: [0, 0, 0, 0] NonDurableSubscriptionLowWater: [0, 0, 0, 0] PutTopicCount: [0, 0] PutTopicFailCount: 0 Put1TopicCount: [0, 0] Put1TopicFailCount: 0 PutTopicBytes: [0, 0] PublishMsgCount: [0, 0] PublishMsgBytes: [0, 0] Use SMF on z/OS New options in V9
  25. 25. 24Integration Technical Conference 2019 ©2019IBMCorporation Taking a look with SupportPac MS0P Time period MQ Statistics at QMGR level Detailed queue statistics Use SMF on z/OS
  26. 26. Integration Technical Conference 2019 25©2019IBMCorporation Taking a look at accounting events with amqsevt Choose JSON or TEXT format Which type of event is this: ACCTMQI or ACCTQ?Which application does this event relate to? Use SMF on z/OS
  27. 27. Integration Technical Conference 2019 26©2019IBMCorporation Taking a look at statistics events with amqsevt Choose JSON or TEXT format Which type of event is this: STATMQI, STATQ, STATCHL, or STATACLS? Period of time the statistics relate to Use SMF on z/OS
  28. 28. Integration Technical Conference 2019 27©2019IBMCorporation Accounting, statistics, and system topics Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE SYSTEM.BASE.TOPIC SYSTEM.ADMIN.TOPIC ‘$SYS/MQ’ user space MQ’s space Accounting & Statistics messages System Topics • Enabled by turning on certain types of event message on the queue manager • Turn on and off for specific queues and channels • Event messages build up on system queues for you to consume • No need to turn on – just subscribe to the correct topic • Better suited to multiple consumers who all want the same events • Provides more than just MQ data (system CPU, disk IO stats) • More granular authorisation Dist platforms
  29. 29. Integration Technical Conference 2019 28©2019IBMCorporation System Topics • Distributed queue manager information is published to a range of system topic strings • $SYS/MQ/INFO/QMGR/…. • Authorised subscriptions receive their own stream of publications based on the topic string • Administrative subscriptions • E.g. For information to be continually sent to defined queues • Application subscriptions • E.g. To dynamically listen to information as required • Unlocks system level information for MQ administrators and DevOps teams • Administrators can grant access to subsets of the data, pertinent to different application teams SYSTEM.BASE.TOPIC ‘’ SYSTEM.ADMIN.TOPIC ‘$SYS/MQ’ user space MQ’s space Appliance/MQ V9 Not on z/OS
  30. 30. Integration Technical Conference 2019 29©2019IBMCorporation System Topics • Familiar statistics available through subscriptions • Queue manager wide statistics (connects, disconnects, opens, closes, puts, gets, …) • Queue usage statistics (opens, closes, puts, gets, …) • Application usage statistics (instances started, instances ended, …) • NB: statistics available from system topics are not a 1-1 mapping to those available from system queues • E.g no channel statistics, some missing information, some new information, some merged information • No support for accounting data • Extended to include CPU and disk usage. For example… • Queue manager CPU time, memory usage • Disk reads/writes, disk latency, etc. • Subscribe to meta-topic to learn which classes of statistics are available • $SYS/MQ/INFO/QMGR/QMGR1/Monitor/METADATA/CLASSES • Then subscribe to specific topics • $SYS/MQ/INFO/QMGR/QMGR-NAME/Monitor/class[/instance]/type] • See amqsrua sample program Appliance/MQ V9 Not on z/OS
  31. 31. Integration Technical Conference 2019 30©2019IBMCorporation • By default messages are published every 10 seconds • Configurable via a tuning parameter in qm.ini, e.g. TuningParameters: MonitorPublishHeartBeat=60 • Recent updates to amqsrua sample program let you subscribe to other parts of the $SYS topic tree • amqsrua –p “$SYS/Application/runmqsfb” • Applications can publish their own meta-data and events to $SYS/Application • Single place to consume MQ and application events from MQ V9.0.2 System Topics Appliance/MQ V9 Not on z/OS
  32. 32. Integration Technical Conference 2019 31©2019IBMCorporation $ amqsrua -m V9000_A CPU : Platform central processing units DISK : Platform persistent data stores STATMQI : API usage statistics STATQ : API per-queue usage statistics STATAPP : Per-application usage statistics Enter Class selection CPU SystemSummary : CPU performance - platform wide QMgrSummary : CPU performance - running queue manager Enter Type selection SystemSummary Publication received PutDate:20160411 PutTime:10465573 User CPU time percentage 0.01% System CPU time percentage 1.30% CPU load - one minute average 8.00 CPU load - five minute average 7.50 CPU load - fifteen minute average 7.30 RAM free percentage 2.02% RAM total bytes 8192MB Publication received PutDate:20160411 PutTime:10466573 User CPU time percentage 0.01% System CPU time percentage 1.30% … $ ==> ==> System Topics Appliance/MQ V9 Not on z/OS
  33. 33. Integration Technical Conference 2019 32©2019IBMCorporation • Meta-topics enable self-describing values • Gives classes of statistics as mentioned • Gives mapping of individual values to (NLS enabled) labels • Easy to extend - can add a new value just by including a new label to the meta-topics • Usable to produce graphs and charts… • Pub-sub model means consumption for chart doesn’t affect other consumers • Aids centralised system monitoring… See following charts Centralised System Monitoring
  34. 34. Integration Technical Conference 2019 33©2019IBMCorporation • Open source MQ sample demonstrates how you can subscribe to metrics and statistics and push them to a centralized time- series database such as Prometheus • Uses additional PCF commands to add information about channels to the other metrics • Makes it easier to monitor many queue managers from a single places Centralised System Monitoring • Available to download from Github
  35. 35. Integration Technical Conference 2019 34©2019IBMCorporation • Grafana can by used as the dashboard, connecting to a back-end time series database such as Prometheus Centralised System Monitoring
  36. 36. Integration Technical Conference 2019 35©2019IBMCorporation Accounting, statistics, and system topics recap Accounting Messages Designed to show you per-application data Statistics Messages Designed to show you per-object data ACCTMQI Includes Number of all API calls, bytes put/got etc. for each application/connection to the QM, across all objects the application uses. Includes everything in the “Not included” list for ACCTQ. Where relevant, includes other object types (e.g. number of opens/closes for objects such as namelists, auth info objects, process objects etc.) Example uses • Discover applications that frequently backout transactions • Discover which applications use pub/sub messaging (not included in ACCTQ events) • Charge-back to departments based on how much they use the queue manager STATMQI Includes Number of all API calls, bytes put/got etc. for the queue manager as a whole. Includes everything in the “Not included” list for STATQ. Where relevant, includes all object types (e.g. number of opens/closes for objects such as namelists, auth info objects, process objects etc.) Example uses Discover QMs that are very heavily or very lightly utilised Discover how often applications connect to a QM and how many are connected at any one time Discover how many messages are expired from the QM ACCTQ Includes Number of queue-related API calls, bytes put/got, queue events generated etc. for each application/connection using that queue. The max and min size of the messages used by each application. Not included Calls not related to messaging on a specific queue, e.g. • Topic/subscription related API calls • Inquire/set calls • Callback API calls • Commit/back API calls • Connect/disconnect activity Example uses • Discover which applications use QUEUE1. • See which of the applications using QUEUE1 create the most traffic. • Charge-back to departments based on how much they use a given queue STATQ Includes Summary of API activity on a specific queue, regardless of which application(s) are using it. Statistics such as • The min and max depth of the queue during the period • The average time messages spent on the queue during the period • The number of messages that expired during the period. Not included Calls not related to messaging on a specific queue, e.g. • Topic/subscription related API calls • Inquire/set calls • Callback API calls • Commit/back API calls Information about the size of messages put/got from the queue. Example uses Discover any queues that are never used Discover any queues that are being used as a database Dist platforms
  37. 37. Integration Technical Conference 2019 36©2019IBMCorporation Accounting, statistics, and system topics recap Accounting Messages Designed to show you per-application data Statistics Messages Designed to show you per-object data STATCHL/ STATACLS Includes Per-channel information about the flow of data over each channel: • Number of messages sent across it • Number of bytes in total • The maximum, minimum and average round-trip latency • Whether batches are being filled before sending Example uses Identify slow network connections Understand traffic flow between QMs Identify channels that may need tuning System Topics (e.g. $SYS/MQ/INFO/QMGR/QM1/Monitor/STATMQI/GET) STATMQI Includes Very similar data to the MQ events generated on the SYSTEM.ADMIN.STATISTICS.QUEUE (see previous slides) The main differences between $SYS STATMQI data and SYSTEM.ADMIN.STATISTICS.QUEUE data are: • Each MQI call (such as GET) or pairs of related calls (such as INQ & SET) are published on their own topic. Subscribe to each MQI call you are interested in • You first subscribe to a metadata topic that describes the contents of the actual event messages • Some fields show calculated messaging rates e.g. number of PUT bytes/sec STATQ Includes Very similar data to the MQ events generated on the SYSTEM.ADMIN.STATISTICS.QUEUE (see previous slides) The main differences between $SYS STATQ data and SYSTEM.ADMIN.STATISTICS.QEUE data are: • Each MQI call (such as GET) or pairs of related calls (such as INQ & SET) are published on their own topic. Subscribe to each MQI call you are interested in • You first subscribe to a metadata topic that describes the contents of the actual event messages • Some additional fields, e.g. amount of lock contention on the queue Dist platforms
  38. 38. Integration Technical Conference 2019 37©2019IBMCorporation Accounting, statistics, and system topics recap Note 1: The terms “STATMQI” and “STATQ” are intentionally used in relation to both statistics messages and system topic categories. This is because they offer similar information, even if they are not identical. Note 2: There is no equivalent of ACCTQ, ACCTMQI, or STATCHL on the $SYS system topics. These must still be generated by enabling them on the queue manager (e.g. ALTER QMGR ACCTQ(ON) ACCTMQI(ON) STATCHL(HIGH|MEDIUM|LOW) and consuming the event messages from the SYSTEM.ADMIN.ACCOUNTING.QUEUE or the SYSTEM.ADMIN.STATISTICS.QUEUE. System Topics (e.g. $SYS/MQ/INFO/QMGR/QM1/Monitor/DISK/QMgrSummary) STATAPP Includes • Summary of application instances of a specific application name. Statistics such as • The number of instances started and ended during the period • The number of instances currently connected • The number of instances that are marked as reconnectable. Example uses • Discover how many application instances are active CPU Includes • System CPU load over 1, 5 and 15 minute periods • System CPU load caused by QM and MQ applications • Free RAM (% and absolute) • RAM used by QM Example uses • Remotely monitor system load • Understand how much load MQ is putting on a system DISK Includes Amount of disk consumed by errors, trace & FDCs (% and absolute) Amount of disk consumed by QM (% and absolute) Log and log FS size, number of bytes written to log Log write latency Primary log currently in use Example uses • Remote monitor disk load • Understand how much disk MQ is consuming • Understand if MQ logs are correctly tuned Dist platforms
  39. 39. 38Integration Technical Conference 2019 ©2019IBMCorporation Where are messages going?
  40. 40. Integration Technical Conference 2019 39©2019IBMCorporation MQ networks can be complex • At each of the dots stuck / mis-sent messages are possible • MQOPENs of the wrong queue / queue manager by apps • Full queues • Stopped channels • Stopped apps • Incorrectly configured QREMOTE/QALIAS routing objects • Cluster membership problems • The standard problem diagnosis approach • Methodically checking channels/queues/DLQs at each point • Is there anything to speed up this process? App1 App2A App2B Requests Replies Replies SDR/RCVR SDR/RCVR Cluster All platforms
  41. 41. Integration Technical Conference 2019 40©2019IBMCorporation Trace route • MQ has the ability to inject trace route messages • (Can be) hidden from applications • Generate activity reports as they pass through, potentially accumulated in the message • Tools are available to trace routes using these reports • dspmqrte – command line tool supplied with the product • MS0P – Cat 2 SupportPac extension to MQ Explorer • Lets you see the path messages could have taken • Test connectivity through the MQ network • Test cluster workload balancing • Can quickly jump you close to the problem • The point your trace message veers off in the wrong direction • The point the trail goes cold • NB: there is also a related technology: activity recording which generates activity reports from real messages dspmqrte SDR/RCVR SDR/RCVR dspmqrte dspmqrteCluster All platforms
  42. 42. 41Integration Technical Conference 2019 ©2019IBMCorporation What are the apps doing?
  43. 43. Integration Technical Conference 2019 42©2019IBMCorporation Application activity trace • Information on all the MQI operations performed by an application in the order that they were done • Similar infrastructure to accounting & statistics • PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE • Configurable via mqat.ini • Can be changed without queue manager restart • Configurable detail level can include partial/full message payload • Frequency options for tuning • Can be enabled on a per-application basis • Via MQCONNX flags • Via application name Health warning: Performance impact http://ow.ly/vA8wB Not on z/OS New options in V9
  44. 44. Integration Technical Conference 2019 43©2019IBMCorporation • Enables scenarios such as • Application audit trail • Message duplication • Resource usage • Which queues or topics are actually being used • Problem determination • Which queue / queue manager is the application actually opening • Application coding standards • Does everyone use the MQI in the recommended way • And more … Health warning: Performance impact http://ow.ly/vA8wB Application activity trace Not on z/OS New options in V9
  45. 45. Integration Technical Conference 2019 44©2019IBMCorporation Looking at the data with the amqsact sample ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini amqsact -m GATEWAY1 -v MQI Operation: 6 Operation Id: MQXF_PUT ApplicationTid: 12451 OperationDate: '2014-04-09' OperationTime: '01:39:48' High Res Time: 1397003988665548 Completion Code: MQCC_OK Reason Code: 0 Hobj: 18225032 Put Options: 139330 Msg length: 460 Recs_present: 0 Known_dest_count: 1 Unknown_dest_count: 0 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'SENDINGAPP.REPLY' Object_Q_mgr_name: 'GATEWAY1' Resolved_Q_Name: 'SENDINGAPP.REPLY' Resolved_Q_mgr: 'GATEWAY1' Resolved_local_Q_name: 'SENDINGAPP.REPLY' Resolved_local_Q_mgr: 'GATEWAY1' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQHRF2' Priority: 4 Persistence: 0 Msg_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2603 '..DS .&. ' Correl_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2203 '..DS .". ' Reply_to_Q : ' ^D' Reply_to_Q_Mgr: ' ^C' Coded_char_set_id: 1208 Encoding: 273 Put_date: '20140409' Put_time: '00394866' Check the options used for coding standards Check queue name resolution, to find out why messages are going to the wrong place Track individual messages and request/reply scenarios with Msg_id and Correl_id As of V9 this is known as display mode Make sure you ask amqsact to show everything the QM has generated (-v) Not on z/OS New options in V9
  46. 46. Integration Technical Conference 2019 46©2019IBMCorporation Application activity trace – system topics • Application activity trace enabled through subscriptions rather than queue manager configuration • Subscribe to topic • E.g. $SYS/MQ/INFO/QMGR/QMGR1/ActivityTrace/ApplName/amqsput • Filter by application name, channel or connection id • When a subscription is created, PCF messages start to flow to the subscriber’s queue. When subscription is deleted, messages stop • Much easier to get just the data you want! Appliance/MQ V9 Not on z/OS
  47. 47. Integration Technical Conference 2019 47©2019IBMCorporation Application activity trace – dynamic sample • Sample provided to demonstrate usage and format output • Example below specifies application name (-a) so uses dynamic mode • Dynamic mode subscribes to system topic rather than uses system queue • Channel name and connection id also supported $ amqsact -m QMGR1 -a amqsput -w 60 Subscribing to the activity trace topic: '$SYS/MQ/INFO/QMGR/QMGR1/ActivityTrace/ApplName/amqsput' MonitoringType: MQI Activity Trace … QueueManager: ‘QMGR1' ApplicationName: 'amqsput' Application Type: MQAT_UNIX … ============================================================================= Tid Date Time Operation CompCode MQRC HObj (ObjName) 001 2016-04-14 09:56:53 MQXF_CONNX MQCC_OK 0000 - 001 2016-04-14 09:56:53 MQXF_OPEN MQCC_OK 0000 2 (QUEUE1) 001 2016-04-14 09:56:53 MQXF_PUT MQCC_OK 0000 2 (QUEUE1) 001 2016-04-14 09:56:53 MQXF_PUT MQCC_OK 0000 2 (QUEUE1) 001 2016-04-14 09:56:53 MQXF_CLOSE MQCC_OK 0000 2 (QUEUE1) 001 2016-04-14 09:56:53 MQXF_DISC MQCC_OK 0000 - ============================================================================= $ amqsput QUEUE1 QMGR1 Sample AMQSPUT0 start target queue is QUEUE1 Hello World Sample AMQSPUT0 end $ $ Appliance/MQ V9 Not on z/OS
  48. 48. 49Integration Technical Conference 2019 ©2019IBMCorporation How can I look back in time?
  49. 49. Integration Technical Conference 2019 50©2019IBMCorporation What happened to my messages at 2am this morning? • Enterprise monitoring solution • DLQ alerts, queue depth alerts, channel status alerts • Unresolved running units of work • Historical MQ monitoring, accounting and stats data • App logs from the time of the problem • Exceptions, MQ error codes, timeouts • MQ error logs for all qmgrs that could have been involved • Channel errors • Authentication issues • ??? – what else is there
  50. 50. Integration Technical Conference 2019 51©2019IBMCorporation What about the MQ recovery log? • For persistent messages inside transactions • MQ logs each operation performed • Outside of transactions persistent messages might be logged • Why can't we use this to • Look back in time to 2am and see what happened? • Recover the original payload if the app lost the message? • See what happened inside long-running units of work? • Provide a list of operations within the failed business transaction? • MQ documents how you can… if • You use the text formatting tool provided with MQ (dmpmqlog) • The logging is linear so the historical data is available in the tool • You follow the right steps to extract data from running qmgrs • You do the work to follow through the logs Use CSQ1LOGP on z/OS
  51. 51. Integration Technical Conference 2019 52©2019IBMCorporation dmpmqlog output is readable, but analysis is tedious • Wouldn't it be easier to let the computer do the tedious bit? LOG RECORD - LSN <0:0:954:44817> ********** HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH LogRecdType . . : AQM Put Message (257) Eyecatcher . . : ALRH Version . . . . : 1 LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM) XTranid . . . . : TranType: XA XID: formatID 1463898948, gtrid_length 36, bqual_length 54 gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2] bqual [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000 00000000000000001] QueueName . . . : Not known Qid . . . . . . : {Hash 2147211283, Counter: 5} ThisLSN . . . . : <0:0:0:0> PrevLSN . . . . : <0:0:954:43944> Version . . . . : 4 MapIndex . . . : 199 PrevLink.Locn . : 102408 PrevLink.Length : 8 PrevDataLink . : {High 0, Low 103424} Data.Locn . . . : 103424 Data.Length . . : 613 Data . . . . . : 00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ 00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç....... 00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥....... 00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM 00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000 00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............ 00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................ 00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........ 00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ 00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............ 00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR 00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........ 00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0 00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM 00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR 00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d 00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò.... 00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........ 00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS 00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client 00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219 00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 .... 00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤............... 00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤.... 00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR .... 00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms 00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd> 00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms 00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q 00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387 00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms>< 00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci 00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></ 00608: 6A 6D 73 3E 61 jms>a Ordered unique IDs for each record (LSN) A set of documented record types Transaction information with XIDs, or re-used MQ transaction IDs MQMD header data at discoverable offsets in the hex of a message Put The message payload itself Use CSQ1LOGP on z/OS
  52. 52. Integration Technical Conference 2019 53©2019IBMCorporation Check out dmpmqlog scraper tool • Takes the tedium out of analysing the output from dmpmqlog • Created by Peter Broadhurst • Download from http://www.ibm.com/support/docview.wss?uid=swg21660642 • Generates file per message PUT in the supplied data • Summary file java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o .
  53. 53. Integration Technical Conference 2019 54©2019IBMCorporation dmpmqlog scraper tool output • Generates file per message PUT in the supplied data. • Summary file
  54. 54. Integration Technical Conference 2019 55©2019IBMCorporation Summary • Lots of tools in your MQ toolbox! • On-line status commands • DISPLAY CONN • DISPLAY QSTATUS • DISPLAY CHSTATUS • Cluster monitoring – AMQSCLM • Off-line statistics and accounting • amqsmon and MS0P to view • Tracking • Trace-route • Application activity trace • MQ recovery logs • dmpmqlog scraper
  55. 55. Thank You Related session: • M08 – Monitoring and problem determination of an MQ z/OS system • Day 3 (Thursday) 9.50am • White Pine III

×