Cerner has hundreds of applications running on thousands of JVMs. AppDynamics helped Cerner get complete visibility into its application infrastructure using limited resources. Gain insight into planning and managing large AppDynamics infrastructures. Hear simple tricks that can be life savers and pitfalls to avoid. And learn about scripting for WebSphere in cells with multiple clients verses one client cells. Key takeaways: o Why you need to automate your processes o Insight to how to structure and plan your upgrades o How to make sure your users have skin in the game For more information go to: www.appdynamics.com

1. Thousands of JVMs, hundreds of applications,
and two people:
How Cerner learned to manage it
William Feist, IT Architect | Cerner Health
David Parrish | Cerner Health

2. APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 1
Notice
The information and materials included in this presentation (collectively,
the “Materials”) are the confidential and proprietary information of
AppDynamics, Inc. (the “Company”). No part of the Materials may be
reproduced, distributed, communicated or displayed in any form or by any
means, or used to make any derivative work, without prior written
permission from the Company. © 2016 AppDynamics, Inc. All rights
reserved.
All third party trademarks, including names, logos and brands, referenced
by the Company in this presentation are property of their respective
owners. All references to third party trademarks are for identification
purposes only and shall be considered nominative fair use under
trademark law.

3. AppDynamics
Bill Feist, Dave Parrish
November 16, 2016

4. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Agenda
Configuring and Installing AppDynamics agents
Sizing and tuning Controllers
Finding clients
License Control
Application Challenges
Organizing
Deeper Dive
Configuring

6. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Our Story

7. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Challenge
• How do we get it installed to all customers?
• How do we deliver the agent software?
• We have Multi client WebSphere cells and client specific
cells. How do we handle that?
• How many Controllers will we need?
• How will support be able to find a customer in a multi
customer environment?
• How will we handle licensing?
•

8. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
How do we get it installed to all customers?
• Bundle the Software
Agent software
Machine agent software
Community software
• Use Share Drives and Symbolic Links
• When possible use variables for JVM args.
${APPDYN_CONTROLLER_PORT} is
-Dappdynamics.controller.port=8090
• Scripts = Consistency
Static Variables vs Dynamic
Example – Tier and node dynamic
Multi-Tenant cell vs Standalone

9. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
We had no idea how many controllers we would need
Controller Hardware:
24 cpu
196 RAM
2 Terabytes db mnt
How many controllers do we need?

10. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
How many controllers do we need?
• Tune your Controller
Glassfish, Controller, Mysql, OS
• I/O is your enemy
• Check your CPU’s are all being used
• Use the AppDynamics Controller feature
http://server:8099/controller/?enableAccounts=true
• Tune MySQL
innodb_buffer_pool_size being undersized can hurt
• Watch for Data Explosion
• Set rules for Metric Retention, Events, Business,
Transactions

11. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Challenge
Started to experience the same clients in all controllers because their
applications are shared by other clients or sheer confusion as to last
install

12. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
I found 3 ways to do this:
How will support be able to find a client?

13. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
So what can you do with python/elastisearch/kibana?????????
Controller URLApplication Name
Nodes/JVM Names

14. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
So Python/Elastisearch/Kibana?????????
# of
Licenses
# of Applications
Usage by Applications

15. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Application Challenges
 What are we seeing ?
 Organizing the Information
 Going Deeper into the Application
 Avoiding “Alert Overload”
 Special Touches
 AppDynamics Configuration Exporter Utility

16. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
What are we seeing?
Soarian Financials
Browser-Based application to provide financial application
functions to hospitals
• Checkin, Checkout, Revise Encounter, Billing, Guarantor Processing,
interop to other applications such as Scheduling and Clinicals
Much more complex than we ever imagined
Support 75 PROD environments
• Largest processes 7 million requests per day
• Many TEST, Performance Lab environments
• Keeping everyone in sync is a major challenge

17. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Soarian Financials Application

18. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Organizing the Information
Define Nodes and Tiers
Nodes = JVMs
• Names must be unique across Tiers
Tiers
• Used to group similar Nodes together
• Important to get this right, as other AD components depend on this
Customize Business Transactions
Analyze the incoming traffic to a Tier
How granular is up to you
Can be hard to get the “right number”
• We have about 500 BT’s
• ~20 Custom Match Rules defining BT’s
• Working to cut that back, so we can add more
• Be careful not to overload your Controller

19. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Going Deeper into the Application
Data Collectors
Extracting HTTP Parameters
Cookie information (on backlog)
JMX Metrics
Some come out-of-the box
Can create other JMX Metrics from available Mbeans
Websphere PMI Metrics
• Messaging Engine metrics ex AvailableMessageCount
• Thread Pool usage
• DB Connection Pool Usage
Tomcat
• Thread Pool usage
• DB Connection Pool Usage

20. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Going Deeper…
HTTP Parameters

21. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Avoiding Alert Overload
Easy to bury support with Alerts
Use email until happy with Alert frequency and validity
Are Warnings Useful ? Not really for us. We only use Critical
Health Rules
How can we prevent outages ?
Using Baselines vs Hard-coded values
• Each has there pros and cons
Monitoring
• Heap Utilization, GC Time Spent
• Tier Avg Resp Time, Node Load (under normal utilization)
• DB Avg Resp Time, Errors Per Minute
• Stalls

22. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Avoiding Alert Overload…
Policies
Minimized Policies by pointing multiple Health Rules to a single
Policy
Different Action requirement determine our need for more Policies
*** Since Actions are fired on Health Rule Violation, you might
need a separate policy for “Warning” and “Critical” violations if
Actions are different

23. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches
Dashboards
Custom Actions
Futures
Custom Metrics
Service Endpoints

24. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Dashboards
Monitor Node Health
Heap Usage
CPU
Time Spent in Garbage Collection

25. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Dashboards (more)
Tier Performance

26. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Custom Actions
Remediation Script
Use AppDynamics to trigger an Action which runs a script
Variables available to script from AppDynamics to identify Node and other
information about the Health Rule that was violated
Successfully used to
• Trigger Connection Pool trace
• Trigger Heap Dumps
Pass Health Rule Violations to Ticketing System
Use the SNMP Trap Alerting Extension
Watch setting up your policy. The Health Rule Violation Events can be
important. This can cause a flood of traps assuming you select all health
Rule violation changes.

27. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Trigger Heap Dump

28. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Generate Ticket

29. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Special Touches: Futures
Custom Metrics
Currently 2 projects under construction
• Look for specific messages in JVM logs that indicate client customization has
failed
• Atomikos Connection Pool Size
 No metric published via Mbean
 Not available for 6 months from Development
 A Method Call is available to get the current value, but AppDynamics cannot use that directly
 Writing script to extract and publish to Controller Agent
Service Endpoints
Next on the list to investigate
Ex. Background transaction creates a message to process each request in
a database queue – currently shows as 1 BT snapshot with many
snapshots segments (seen over 5000 segments)

30. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
AppDynamics Configuration Exporter Utility
AppDynamics Tool to deploy components from one Application to
another
Policies, Health Rules, Actions
Business Transaction, Data Collectors
Call Graph Settings, AppAgent Properties
Several other components..
JMX Rules *
• Some may require further modification if Path contains Client-specific definition
• Ex.
WebSpherePMI:path0=StatGroup.SIBService,path1=StatGroup.SIBMessagingE
ngines,path2=SF_1030_PROD_TNT33_MESSAGING.000-
1030_PROD_SoarianSIB,path3=Destinations,path4=Topicspaces,*

31. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Config Exporter…

32. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Config Exporter…

33. Thank you

34. © 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.
Appendix
AppDynamics provides tuning recommendations for large clients
https://docs.appdynamics.com/display/PRO42/Tuning+for+Large+Scale+Deployments
Kibana
https://www.elastic.co/downloads/kibana
Python/Elastisearch
https://pypi.python.org/pypi/elasticsearch
AppDynamics SDK and Alerting for the Controller
https://www.appdynamics.com/community/exchange
AppDynamics Monitor
http://server:8099/controller/?enableAccounts=true
set it to system use the root id and password to get in.