Are you spending hours trying to understand how customers are impacted by performance issues and faults in your service-oriented applications? In this session, we show you how customers are using AWS X-Ray to reduce mean time to resolution, get to the root cause faster, and determine customer impact. In addition, one of our X-Ray customers, ConnectWise, presents a case study and best practices on how it is leveraging X-Ray in its production environment. We also show you how to use X-Ray with applications built using AWS services, such as Amazon Elastic Container Service for Kubernetes (Amazon EKS), AWS Fargate, Amazon Elastic Container Service (Amazon ECS), and AWS Lambda to achieve the above.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep Dive into AWS X-Ray:
Monitor Modern Applications
Abhishek K Singh
General Manager
AWS X-Ray
D E V 3 2 4
Kiet Nguyen
Director of Cloud Services
ConnectWise, Inc.
Mena Joseph
Director of Application Services
ConnectWise, Inc.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What to expect from the session
What are modern applications
Challenges of monitoring modern applications
AWS X-Ray overview and benefits
Customer story
Demo
Q&A

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Modern applications
Each component is
autonomous and independent
Built on containers
and serverless
Microservices architecture
and distributed
See impact of
isolated errors

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traditional debugging does not scale
VPCOn-Premise Cloud
Database
Users
Local test
Add log
statements
Add
breakpoints
Local test
Add log
statements
Add
breakpoints
Local test
Add log
statements
Add
breakpoints

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tracing connects the dots in modern application
Get insights into
individual
operations
Discover
multiple
services
See issues
isolated within
a service
Perform root
cause analysis
for specific issue

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tracing for your modern application
VPCOn-Premise
Cloud
Users Amazon
API Gateway
Amazon
API Gateway
Amazon EC2 Amazon EKS
AWS
Lambda
Amazon
DynamoDB

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does AWS X-Ray help?
X-Ray helps answer three critical questions:
1. How is my application doing?
2. Why is my application performing the way it is?
3. Who is impacted by the issues?

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How the X-Ray service works

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Built for modern applications
End-to-end view of
individual services
Analyze and debug
issues quickly
Identify customer
impact
Cloud agnostic

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Supports multiple languages
Ruby (beta)
.NET
(with .NET Core 2.0)
Python
Java
Node.js
Go (beta)

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Value proposition for serverless
Get information on
AWS Lambda Cold
Start
Latency information on
Amazon API Gateway custom
authorizers
Identify Lambda
Timeouts

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable X-Ray for API Gateway

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable X-Ray for Lambda

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Value proposition for containers
Find issues in a specific
container
Pinpoint to a specific
availability zone
Supports Amazon
Elastic Compute Cloud
(Amazon ECS), Amazon
Elastic Container
Service for Kubernetes
(Amazon EKS), and
AWS Fargate

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable X-Ray for Amazon ECS/AWS Fargate
{
"environment": [],
"name": "xray-sidecar",
"mountPoints": [],
"image": "<repository_uri_xray_daemon>",
"cpu": 0,
"portMappings": [
{
"protocol": "udp",
"containerPort": 2000,
"hostPort": 2000
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs",
"awslogs-group": "/ecs/service-a"
}
},
"essential": true,
"volumesFrom": []
}
https://github.com/aws-samples/aws-microservices-deploy-options

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable X-Ray for Amazon EKS/K8s
https://github.com/aws-samples/aws-xray-kubernetes
https://aws.amazon.com/blogs/compute/application-tracing-on-kubernetes-with-aws-x-ray/

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ConnectWise transforms how technology teams
build, manage, and grow their businesses. Our
solutions increase productivity, efficiency, and
profitability.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Professional Quote and Proposal Automation
Hyper-Efficient Business Management
Robust Remote Monitoring and Management
Lightning-Fast Secure Remote Control
Unified Cloud Solutions Management
Marketing Campaign Management Tool
Immediate Response Support
Solution
The ConnectWise products

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The ConnectWise
Platform

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
24,300+ customers
103,000+ daily active users
ConnectWise Manage alone has 2,583 web
requests per second during peak time
900+ tables per customer

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prior to AWS
On-premise datacenter
High operational and maintenance cost
Hardware to deal with
Time to provision
AWS
All in with AWS in 2012
Able to design a true distributed architecture
High availability
No hardware to think about

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Distributed architecture
• Adopting microservices
• Integrating with many
AWS services

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
X-Ray adoption
• Lack visibility into critical application
• Visibility of application performance
• Efficiency of diagnosing
Integrated X-Ray with main architecture components
• Incoming requests
• Outgoing calls
• AWS services and client integration
• Data access layer

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Provides architects with a live architecture diagram
• Pinpoints the bottlenecks
• Helps optimize performance
• Helps optimize SQL queries
• Enables data-driven architecture decisions

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon
EC2
Amazon
S3
Lambda Amazon
EFS
DynamoDB
ElastiCache
ALBKinesis
SWF
Step
Functions
Amazon
SNS
Amazon
SES
Amazon
SQS
AWS KMS
API
Gateway

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Visibility into downstream AWS services integration

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
First line of defense

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Created subsegments for data
access layer to capture queries
performance
Troubleshooting SQL queries performance

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL performance logs
Latency (ms) Parameters SQL Command
7.0 @RecID Int 32639064 IF EXISTS (SELECT 1 FROM dbo.Schedule_Detail sd LEFT JOIN dbo.Schedule s on s.Schedule_RecID =
sd.Schedule_RecID . . .
34.8 IF EXISTS (SELECT sr.SR_Location_RecID FROM DBO.SR_LOCATION as sr WHERE sr.SR_Location_RecID =24)
SELECT . . .
1.0 @recid Int 65 SELECT Owner_Level_RecID FROM dbo.User_Defined_Field_Owner_Level WHERE
User_Defined_Field_RecID = @recid
18.6 @serviceRecId Int
11111899
SELECT Company_RecID FROM dbo.SR_Service WHERE SR_Service_RecID = @serviceRecId
123.7 @problemFlag Int 1 . . . exec dbo.usp_getSRDetailTable @problemFlag, @resolutionFlag, @internalAnalysisFlag, @serviceRecID,
@includeChild
1201.8 @problemFlag Int 1 . . . exec dbo.usp_getSRDetailTable @problemFlag, @resolutionFlag, @internalAnalysisFlag, @serviceRecID,
@includeChild
batch-get-traces --trace-ids <trace-ids>
<value> <value>
Pass trace-ids for SQL calls

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Troubleshooting service integrations
Entity Sync
Our web servers are pushing modules and entity update events to
Amazon SQS to be processed by auditing services and global search
indexing in Amazon CloudSearch
Amazon
EC2
Amazon
SQS
Amazon
ECS
Amazon
SNS
CloudSearch

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sending messages
synchronously
….
Still waiting for the
response

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Down to 0.0 ms
Syncing is now taking place on an asynchronous worker threads, no impact on main application

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Data intensive
• High transactional rate
• Highly customizable
• Different implementation per
customer
• Different configuration per user
X-Ray annotations & metadata
• Down to transactions per customer
• Down to single entity per user
• Down to single transaction per entity
Customer impact

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
By customer

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
By user

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
By record

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- Reduce cost by 75% (HUGE SAVINGS)
Down from $17k to $4k per quarter
- Average time to diagnose an issue from couple of hours to
minutes
Identifying bottlenecks and causes in real time
- X-Ray for analytics
Instrument annotations to represent different business
components
Business impact

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Usage
Business impact—Usage analytics

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Latency Usage Error rate
capture more than just analytics
Business impact—Usage analytics

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Taking full advantage of X-Ray
WHY?
WHO?
WHAT?

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics dashboards

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Context missing strategy—Wrapping X-Ray recorder
Wrapping Add
Annotation
Wrapping Add
Subsegment

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Context missing strategy – safe recording

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tracing & logging

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tracing & logging
X-Amzn-Trace-Id

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Helps in answering WHY unintended behaviors are taking place
Merging logging and tracing to empower your monitoring infrastructure
Connecting the Dots

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Overview
Perform root
cause
analysis

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap: What’s new with X-Ray
Sampling made easier Added API gateway integration Deep-dive into your application
Adjust sampling rate during
an operational event and
control cost
Update sampling on the fly
without need to redeploy

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
X-Ray Pricing
Free tier
 The first 100,000 traces recorded are free
 The first 1,000,000 traces retrieved or scanned are free
Additional charges
 Beyond the free tier, traces recorded cost $5.00 per million traces
 Beyond the free tier, traces retrieved or scanned cost $0.50 per million traces

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Takeaways
why who
annotations
metadata
groups
automated trace analysis latency errors
faults

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
X-Ray SDKs and Daemon
 Go: https://github.com/aws/aws-xray-sdk-go
 Java: https://github.com/aws/aws-xray-sdk-java
 Ruby: https://github.com/aws/aws-xray-sdk-ruby
 Node.js: https://github.com/aws/aws-xray-sdk-node
 Python: https://github.com/aws/aws-xray-sdk-python
 .NET & .NET Core: https://github.com/aws/aws-xray-sdk-dotnet
 X-Ray Daemon: https://github.com/aws/aws-xray-daemon

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample apps
 .NET: https://github.com/awslabs/aws-xray-dotnet-webapp
 Java: https://github.com/awslabs/eb-java-scorekeep/tree/xray
 Node.js: https://github.com/awslabs/eb-node-express-sample/tree/xray
 Python: https://github.com/awslabs/eb-py-flask-signup/tree/xray
 Alarms & Alerts: https://github.com/aws-samples/aws-xray-cloudwatch-event
 Lambda: https://github.com/awslabs/aws-xray-rekognition-lambda-sample
 Apache Kafka: https://github.com/aws-samples/aws-xray-apache-kafka-sample
 X-Ray + EKS: https://github.com/aws-samples/aws-xray-kubernetes

65. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Abhishek Singh (@abhik5ingh)
Kiet H Nguyen
Mean Joseph

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.