Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
04.03.19
O S L O
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analyzing your web and application logs
S3, Clou...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Hosting up a very simple web
Hosting stat...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What produces data?
• Metering
Records
• Mobile ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Logs, logs and more logs
Logs are important:
• D...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational analytics
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The explosion of machine-generated data
Transiti...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operational analytics requirements/challenges
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Not just for Geocities web pages
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to enable hosting in S3
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why hosting in S3 is cool!
• Simple static websi...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
Amazon Global Network
• Redundant 100GbE network
• Redundant private capacity
between all Regions except China
166 Global ...
Regional Edge Caches
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Going global
• Simple to enable
• Speeds up your...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticsearch Service is a cost-effective...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easy to Use
Deploy a production-ready Elasticsea...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application Monitoring & Root-cause Analysis
CAS...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Leading enterprises trust Amazon Elasticsearch S...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What does it do?
Application DataServer, applica...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
{ "title": "Star Wars",
"plot": "Luke Skywalker ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch creates an index for each field
Do...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Query your data
{
"query": {
"match": {
"title":...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
You can search for values by field, with Boolean...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
You can analyze field values to get statistics
a...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElasticSearch Service with Kibana
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easy to use and scalable
AWS SDK
AWS CLI Elastic...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security
• Public endpoints – IAM
• Private endp...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use IAM for public endpoints
{ "Version": "2012-...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add security groups for private endpoints
Specif...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture: Web Serving
Public
Subnet
Internet...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Encrypt your data
• Encrypted data at rest on
Am...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data is stored in indexes, distributed across sh...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How many instances?
• Index size is approximatel...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Instance type recommendations
Instance Workload
...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
It is elastic
• Run out of storage space?
• add ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Full text search
CASE STUDY: MirrorWeb
Make the ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deployment of indices to a cluster
• Index 1
– S...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cluster with dedicated masters
Amazon ES cluster...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Master node recommendations
Number of data nodes...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cluster with zone awareness
Amazon ES cluster
1
...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling Amazon Elasticsearch Service
• Scaling A...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
To scale correctly, iterate!
• Set config
• Inst...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use ES to analyze ES performance
https://aws.ama...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data pattern for time series data (or logs)
Amaz...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kinesis Firehose
CloudWatch Logs
Logstash
Amazon...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Small-ish hosted use cases
Public Subnet
Interne...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
XL hosted use cases
• Ingest supported through h...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Large or serverless use cases
• Data flows from ...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
What serverless is not about
Credit: Blue Waters main data center room in June 2010. Still empty except for power distribu...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The function
https://gitlab.com/ric_harvey/cf-es...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Permissions
Cluster access
Account ID
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Permissions
IAM Roles
Permissions to allow:
Acce...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affilia...
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you
Javier Ramirez
@supercoco9
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf
Nächste SlideShare
Wird geladen in …5
×

Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf

212 Aufrufe

Veröffentlicht am

Hosting scalable applications on Amazon S3 and making them globally availiable via Amazon Cloudfront has never been easier, in this presentation and demo we'll dig into getting more insights from your static hosted website by logging CloudFront to S3 and then using the power and scale of Lambda to push those logs into Amazon Elasticsearch Service for deep analysis.

  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. 04.03.19 O S L O
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Analyzing your web and application logs S3, Cloudfront, Lambda and the Elasticsearch Service Javier Ramirez AWS Tech Evangelist @supercoco9 B A R 2 O S L O 04.03.19
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Hosting up a very simple web Hosting static sites on AWS Scaling out any website with a Content Delivery Network The Elasticsearch Service Kibana Dashboards Using Lambda for serverless log ingestion Demo time
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why logs?
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What produces data? • Metering Records • Mobile Apps • IoT Sensors Web Clickstream • Enterprise Documents • Application Logs [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htd ocs/test
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Logs, logs and more logs Logs are important: • Debugging • Working out user flow • Monitoring Centralising those logs is even more important
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Operational analytics
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The explosion of machine-generated data Transition from IT to DevOps Increase in IoT and Mobile Devices Cloud-based architectures Machine-generated data is growing 10x faster than business data Source: insideBigData - The Exponential Growth of Data, February 16, 2017
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Operational analytics requirements/challenges
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s host a static website on AWS so we can generate some traffic logs
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Not just for Geocities web pages
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How to enable hosting in S3
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why hosting in S3 is cool! • Simple static website hosting with a simple workflow • 2 clicks to enable • Scales • Fast • Supports your own FQDN • You can extend it with lambda functions for dynamic content Developer Users Resolve DNS Fetch site direct from S3 Push Code
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling out with CloudFront so we can handle high volume traffic more efficiently
  15. 15. Amazon Global Network • Redundant 100GbE network • Redundant private capacity between all Regions except China 166 Global CloudFront PoPs a e o q i h Paris Sweden AWS GovCloud East First 5 years: 4 regions 2016–2020: 13 regions Next 5 years: 7 regions A W S REGIONS 2 0 R e g i o n s 6 1 A Z s d m c g b n s k v i i i i i i i i Milan i Cape Town
  16. 16. Regional Edge Caches
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Going global • Simple to enable • Speeds up your site for international users • Choice of regions • Allows you to enter multiple CNAME’s • Integrated with Route 53 alias records • Supports SSL certs from certificate manager
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Amazon Elasticsearch Service
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Elasticsearch Service is a cost-effective managed service that makes it easy to deploy, manage, and scale open-source Elasticsearch and Kibana for log analytics, full-text search, and more.
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Easy to Use Deploy a production-ready Elasticsearch cluster in minutes Simplifies time-consuming management tasks such as software patching, failure recovery, backups, and monitoring Open Get direct access to the Elasticsearch open- source API Fully compatible with the open-source Elasticsearch API, for all code and applications Secure Secure Elasticsearch clusters with AWS Identity and Access Management (IAM) policies with fine-grained access control access for users and endpoints Automatically applies security patches without disruption, keeping Elasticsearch environments secure Available Provides high availability using Zone Awareness, which replicates data between two Availability Zones Monitors the health of clusters and automatically replaces failed nodes, without service disruption AWS Integrated Integrates with Amazon Kinesis Firehose, AWS IoT, and Amazon CloudWatch Logs for seamless data ingestion AWS CloudTrail for auditing, AWS Identity and Access Management (IAM) for security, and AWS CloudFormation for cloud orchestration Scalable Scale clusters from a single node up to 20 nodes Configure clusters to meet performance requirements by selecting from a range of instance types and storage options, including SSD-powered EBS volumes Amazon Elasticsearch Service Benefits
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Application Monitoring & Root-cause Analysis CASE STUDY: EXPEDIA Logs, lots and lots of logs. How to cost effectively monitor logs? Require centralized logging infrastructure Did not have the man power to manage infrastructure P R O B L E M Quick insights: Able to identify and troubleshoot issues in real-time Secure: Integrated w/ AWS IAM Scalable: Cluster sizes are able to grow to accommodate additional log sources B E N E F I T S Streaming AWS CloudTrail logs, application logs, and Docker startup logs to Elasticsearch Created centralized logging service for all team members Using Kibana for visualizations and for Elasticsearch queries S O L U T I O N
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Leading enterprises trust Amazon Elasticsearch Service for their search and analytics applications Media & Entertainment Online Services Technology Other
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What does it do? Application DataServer, application, network, AWS, and other logs Amazon Elasticsearch Service Domain with index(es) 1. Send data as JSON via REST APIs 2. Data is indexed - all fields searchable, including nested JSON 3. Queries, via REST APIs, allow fielded matching, Boolean expressions, include sorting and analysis 1 2 3 Application users, analysts, DevOps, security
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. { "title": "Star Wars", "plot": "Luke Skywalker joins forces with a Jedi Knight, a cocky pilot, a wookiee and two droids to save the universe from the Empire's world-destroying battle-station, while also attempting to rescue Princess Leia from the evil Darth Vader.", "year": 1977, "actors": [ "Mark Hamill", "Harrison Ford", "Carrie Fisher” ], "directors": [ "George Lucas” ], "rating": 8.7, "genres" : [ "Action", "Adventure", "Fantasy", "Sci-Fi” ] } Elasticsearch works with structured JSON containing fields and values
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elasticsearch creates an index for each field Doc Name Value Name Value Name Value Name Value Name Value Name Value Fields Analysis Field indices Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 1, 4, 8, 12, 30, 42, 58, 100. .. Posting lists
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Query your data { "query": { "match": { "title": "iron man" } } } Title Score Iron Man 10.56436 Iron Man 2 8.631084 Iron Man 3 8.631084 Iron Sky 6.387543 The Man with the Iron Fists 6.185517 3 The Man in the Iron Mask 6.185517 3 The Iron Giant 5.218624 The Iron Lady 5.218624 77 hits
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. You can search for values by field, with Boolean expressions to get scored, sorted results 1, 4, 8, 12, 30, 42, 58, 100. ... 1, 4, 8, 12, 30, 42, 58, 100. ... Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 Analysis 1, 4, 8, 12, 30, 42, 58, 100 ... Posting lists Field1:value Field2:value Field3:value Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 Term 1 Term 2 Term 3 Term 4 Term 5 Term 6 Term 7 1, 12, 58, 100 58, 12, 1, 100 Result Key Idea
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. You can analyze field values to get statistics and build visualizations 58, 12, 1, 100, 115 123, 214, 947 Result GET GET POST GET PUT GET GET POST Field Data Buckets GET POST PUT 5 2 1 Counts Key Idea
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ElasticSearch Service with Kibana
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Get the right set up
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Easy to use and scalable AWS SDK AWS CLI Elasticsearch data nodes Elasticsearch master nodes Amazon Elasticsearch Service domain Developer Amazon Cognito Amazon CloudWatch AWS CloudTrail
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security • Public endpoints – IAM • Private endpoints – IAM and security groups • Encryption
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use IAM for public endpoints { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::XXXX:root” }, "Action": "es:*", "Resource": "arn:aws:es: us-west-2:XXXX:domain/YYYY/*” } ] } • To grant access for Kibana, use a CIDR Condition • To grant read-only access to an account or role, specify an es:HttpGet Action • To limit a user to a specific index, or API add it to the Resource • For Admin access, specify administrative es:* Actions
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add security groups for private endpoints Specify a subnet and security group to apply CIDR restrictions on inbound/outbound traffic security group security group Amazon Elasticsearch Service Data Master Data Master IAM IAM
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Architecture: Web Serving Public Subnet Internet Gateway ALB Private Subnet Httpd Amazon ES ENI Amazon ES Domain
  36. 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Encrypt your data • Encrypted data at rest on Amazon ES instances • Both EBS and ephemeral store • Encrypted automatic snapshots
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data is stored in indexes, distributed across shards Amazon Elasticsearch Service domain ID Field: value Field: value Field: value Field: value Index Shards Instances • Shards are primary or replica • Primary shard count can’t be changed • Elasticsearch distributes shards to instances elastically • Primary and replica are distributed to different instances
  38. 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How many instances? • Index size is approximately source size • Double this if you are deploying an index replica • Instance count based on storage requirements • Either local storage or up to 1.5 TB of Amazon Elastic Block Store (EBS) per instance Example: a 2 TB corpus will need 4 instances Assuming a replica and using EBS Given 1.5 TB of storage per instance, this gives 6TB of storage
  39. 39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instance type recommendations Instance Workload T2 Entry point. Dev and test. OK for dedicated masters. M3, M4 Equal read and write volumes. R3, R4 Read-heavy or workloads with high memory demands (e.g., aggregations). C4 High concurrency/indexing workloads I2,I3 Up to 1.6 TB of SSD instance storage.
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. It is elastic • Run out of storage space? • add data nodes or • increase EBS volume size. • Need more compute or RAM? • increase the instance type, or • add more data nodes. Amazon Elasticsearch Service is designed to apply those configuration changes without incurring downtime.
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Full text search CASE STUDY: MirrorWeb Make the UK Government and UK Parliament’s web archives searchable Large scale ingestion scenario: 120 TB of data (1.2 MM 100MB files), duplicates and bad data, Warc format P R O B L E M Scalability: Started on a 9-node, R4.4Xlarge cluster for fast ingest, reduced to 6 R4.Xlarge instances for search. Able to reconfigure the cluster with no down time Cost effective: Indexed 1.4 billion documents for $337 Fast: 146 MM docs per hour indexed. 14x faster than the previous best for this data set (using Hadoop) B E N E F I T S S O L U T I O N Amazon S3 (Storage) Amazon EC2 (Filtering) Amazon EC2 (Extraction) Amazon ES (Search) For more on this case, see http://tinyurl.com/ybqwbolq
  43. 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deployment of indices to a cluster • Index 1 – Shard 1 – Shard 2 – Shard 3 • Index 2 – Shard 1 – Shard 2 – Shard 3 Amazon ES cluster 1 2 3 1 2 3 1 2 3 1 2 3 Primary Replica 1 3 3 1 Instance 1, Master 2 1 1 2 Instance 2 3 2 2 3 Instance 3
  44. 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  45. 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster with dedicated masters Amazon ES cluster 1 3 3 1 Instance 1 2 1 1 2 Instance 2 3 2 2 3 Instance 3Dedicated master nodes Data nodes: queries and updates
  46. 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Master node recommendations Number of data nodes Master node instance type < 10 m3.medium+ < 20 m4.large+ <= 50 c4.xlarge+ 50-100 c4.2xlarge+ Always use an odd number of masters, >= 3
  47. 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  48. 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster with zone awareness Amazon ES cluster 1 3 Instance 1 2 1 2 Instance 2 3 2 1 Instance 3 Availability Zone 1 Availability Zone 2 2 1 Instance 4 3 3
  49. 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling Amazon Elasticsearch Service • Scaling Amazon Elasticsearch Service means scaling Elasticsearch • Amazon Elasticsearch Service delivers seamless deployment changes Storage Type, amount per- instance, and count Parallelism Add and remove instances Throughput and Capacity Change instance types
  50. 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. To scale correctly, iterate! • Set config • Instance type, storage • Monitor via CloudWatch and ES APIs • CloudWatch for CPU, JVM, and Disk • ES for indexing, merges, latencies • Slow Logs for bottlenecks • Adjust for workload Configure Cluster Run your workload Gather metrics
  51. 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use ES to analyze ES performance https://aws.amazon.com/blogs/database/analyzing-amazon-elasticsearch-service-slow-logs-using-amazon- cloudwatch-logs-streaming-and-kibana/
  52. 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data pattern for time series data (or logs) Amazon ES cluster logs_01.21.2017 logs_01.22.2017 logs_01.23.2017 logs_01.24.2017 logs_01.25.2017 logs_01.26.2017 logs_01.27.2017 Shard 1 Shard 2 Shard 3 host ident auth timestamp etc. Each index has multiple shards Each shard contains a set of documents Each document contains a set of fields and values One index per day
  53. 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kinesis Firehose CloudWatch Logs Logstash Amazon Lambda Transform Kinesis Firehose/ Streams CloudWatch Logs Amazon Elasticache/Redis Kafka Rabbit MQLogstash Amazon S3 Buffer Kinesis Firehose/ Streams Logstash Worker nodes Deliver Kinesis Agent CloudWatch Logs Agent Beats Fluentd Application Logstash Collect CloudWatch Logs Amazon Lambda
  54. 54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Small-ish hosted use cases Public Subnet Internet Gateway ALB Private Subnet Filebeat Redis ENI Kibana Amazon ES ENI Amazon ES Domain Redis Cluster
  55. 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. XL hosted use cases • Ingest supported through high- volume technologies like Spark or Kinesis • Up to 60 TB of data today • R3.8xlarge + 640GB data nodes • 3x m3.xlarge master nodes
  56. 56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Large or serverless use cases • Data flows from instances and applications via Lambda • SigV4 signing via Lambda/roles • Up to 5 TB of data • r3.2xlarge + 512 GB EBS data nodes • 3x m3.medium master nodes
  57. 57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda for serverless log ingestion
  58. 58. What serverless is not about Credit: Blue Waters main data center room in June 2010. Still empty except for power distribution cabinets, by Daniel Schwen https://commons.wikimedia.org/wiki/File:Blue_Waters_main_data_center_room.jpg
  59. 59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The function https://gitlab.com/ric_harvey/cf-es-log-ingester
  60. 60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Permissions Cluster access Account ID
  61. 61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Permissions IAM Roles Permissions to allow: Access to ES – (get granular and lock it down to a single cluster) Access S3 - (read only permissions to one bucket) CloudWatch Logs – (push logs from lambda app)
  62. 62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo
  63. 63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you Javier Ramirez @supercoco9

×