The document discusses monitoring dynamic cloud applications in a constantly changing environment. It emphasizes that change is the new normal for applications and monitoring is needed to answer important business and operational questions. A software measurement framework is presented that covers metrics related to business outcomes, customer experience, application performance, engineering velocity, and service quality. Case studies are provided of how companies like Gannett/USA Today use monitoring to measure success during large events and speed up their release cycles.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1
Lee Atchison, Principal Cloud Architect at New Relic.
@leeatchison leeatchison
Cloud Monitoring: Change
Is The New Normal

2. Safe Harbor
This document and the information herein (including any information that may be incorporated by reference) is provided for
informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc.
(“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is
proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a
historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future
events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the
statement will include words such as “believes,” “anticipates,”, “expects” or words of similar import.
Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof,
and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting
business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of
the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and
are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results,
performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further
information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to
time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at http://ir.newrelic.com or the
SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not Intend to update these forward-looking statements, except as required by law. New
Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.

3. Who am I?
29 years in industry
4+ in New Relic
(Architecture Lead, Cloud, Service Migration)
7 in Amazon Retail & AWS
(Built SW/VG AppStore, AWS Elastic Beanstalk)
Who Specialize in:
Cloud computing
Services & Microservices
Scalability, Availability
@leeatchison leeatchison
Principal Cloud Architect

4. CustomerCare AppOwners
Executives
Customer Care
Operations Developers
App Owners
Marketing
Picture
Your
Apps
HERE
Behind every app is a team
Running a Digital Business is a Team Sport

5. What Data Do You Need To Run The Business?
How is my
business doing?
Business
Success
Am I open
for business?
Application
Performance
How are customers
engaging?
Customer
Experience

6. Asking Important Questions..
… About Constant Change
First
Deploy
in AWS
India AZ
Java
Security
Vulnerability?
Traffic
Spike from
Celebrity
Tweet!
New Mobile
App!
Flaky Ad
Network?
New Location
Feature
Experiment!
Cloud
Reboot!
Devs
Deploy
Docker
Acquire
New Team!
(They use
Node.js)
Apple
Changes
Terms of
Service
Key
Engineer
on Open
Source Tool
Leaves
6 million new
sensors now
lve!
Bug
Fix!
DNS
Outage?
Try New
CDN in
Europe
How is my
business doing?
Business
Success
Am I open
for business?
Application
Performance
How are customers
engaging?
Customer
Experience
New
Aviation
Service

7. Static vs. Dynamic Applications
Better
Data Center
Dynamic
Environment
How do we use the cloud to accomplish this?

8. Dynamic Cloud Application
• Services/Microservices
• Uses Mobile/Browser Apps
• Dynamic Provisioning
Server OS
Server (Virtual)
Hardware
Application &
Application
Microservices
Provisioning
Application &
Application
Microservices
Application &
Application
Microservices
BrowserMobile

9. Server OS
Server (Virtual)
Hardware
Application &
Application
Microservices
Provisioning
Application &
Application
Microservices
Application &
Application
Microservices
BrowserMobile
Monitoring Dynamic Cloud Application
Amazon
CloudWatch
AWS
CONSOLE
New Relic
Application
Monitoring
New Relic
Infrastructure
Monitoring
DASHBOARDS

10. Change Is The New Normal
Previous - STATIC World
Ops
Dev
Now - DYNAMIC World
Ops

11. Why Measurement Matters

12. Success for Modern Applications
Digital Intelligence
Application
Performance
Customer
Experience
Business
Outcome

13. It’s All About Data

14. 300ms
1.5s

15. .9s

17. The Data from Monitoring Your App
Dwarfs the data inside the App

18. User Experience
Business Outcome
Servers
Apps
Big Data Problem

19. What Game Are You Playing?
What Data Matters to You?
What is your Digital Intelligence?

20. In A Changing World…What Is Your Priority?
Am I Open for Business
and Making Money?
Are My Applications
Working?
Two different views into the same problem

21. Software Measurement Framework
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?

22. Software Measurement Framework
• Daily Revenue
• Cart Value / Volume
• What do you sell?
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance

23. Software Measurement Framework
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale

24. Software Measurement Framework
Service
Quality
• Uptime
• App Errors
• Successful
Deploys
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale

25. Software Measurement Framework
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale
Engineering
Velocity
• Lead Time
• Deploys
• MTTR
Service
Quality
• Uptime
• App Errors
• Successful
Deploys

26. Framework for a Journey
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale
Engineering
Velocity
• Lead Time
• Deploys
• MTTR
Service
Quality
• Uptime
• App Errors
• Successful
Deploys

27. A Digital Intelligence Journey
Am I Open for Business
and Making Money?
Are My Applications
Working?

28. A Digital Intelligence Journey
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale
Engineering
Velocity
• Lead Time
• Deploys
• MTTR
Service
Quality
• Uptime
• App Errors
• Successful
Deploys

29. How Do Other Companies Make This Journey?
Software
Business
• Daily Revenue
• Cart Value / Volume
• What do you sell?
Customer
Experience
• Conversion Rates
• Engagement / Interactions
• End-User Performance
Application
Performance &
Infrastructure
• App performance
• Slow Queries
• Burst Scale
Engineering
Velocity
• Lead Time
• Deploys
• MTTR
Service
Quality
• Uptime
• App Errors
• Successful
Deploys

30. Brian Lieberman
Systems Engineer, Platform as a Service Delivery
Gannett/USA Today

31. Monitoring a brand new Infrastructure on Election night

32. Measuring Success for Elections and Large events

33. Speeding up Release Cycle and Feedback Loop to real time

34. Thank You
leeatchison@leeatchison
Gannett/USA Today Office Hours
4pm Today
New Relic Booth #610