Learning Objectives: - Understand how Amazon ES security works - Learn how to use Amazon VPC endpoints with Amazon ES - Take away best practices for security and access control for your Amazon ES domains

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service
Security Deep Dive
November 9, 2017
Jon Handler
AWS Principal Solutions Architect
handler@amazon.com or @_searchgeek

2. Get started at https://aws.amazon.com/elasticsearch-service/
Introduction
• Amazon Elasticsearch Service supported public
endpoints. Access control used IAM and/or IP-based
policies
• October 17, we added VPC support
• You can still use IAM for fine-grained access, but VPC
support enhances security and simplifies communication
with Amazon ES

3. Get started at https://aws.amazon.com/elasticsearch-service/
Data
Analysis
InsightsActions
Evolution
Your
Busines
s
What is Elasticsearch all about? Let’s start with
the data
• Derive insights from
high-volume,
unstructured or semi-
structured data
• at scale
• securely
• and cost-effectively
• without diverting focus
from your mission
Your data drives your
business

4. Get started at https://aws.amazon.com/elasticsearch-service/
Some uses for your data
IT operations
monitoring
Security information and
event management
Application monitoring
and intelligence
IoT/TSDB monitoring
and data analysis
Search for your
application
Ad targeting

5. Get started at https://aws.amazon.com/elasticsearch-service/
How Amazon ES helps
Log Analytics
Analyze un-structured and semi-structured logs generated
by websites, mobile devices, servers, sensors, and more for
a wide variety of applications such as digital marketing,
operational intelligence, fraud detection, ad tech, gaming,
and IoT.
Full Text Search
Provide a highly performant, rich search and navigation
experience over a diverse set of documents with support for
features including text matching, faceting, filtering, fuzzy
search, auto complete, and highlighting.
Distributed Search
Engine
Power search for your application with
an easy to use, highly performant
JSON document-oriented platform that
can store and retrieve billions of
documents, with integrated replication
across Availability Zones.
Real-Time
Application Monitoring
Capture activity logs across your
customer-facing applications and
websites by indexing data for analysis
in near real-time (less than one
second), visualize it, and perform
statistical aggregations to identify root
cause and fix issues.
Click-Stream
Analytics
Deliver real-time metrics on digital
content and enable authors and
marketers to connect with their
customers. Stream billions of small
messages into Elasticsearch where you
can aggregate, filter, and process the
data to provide content performance
dashboards.

6. Get started at https://aws.amazon.com/elasticsearch-service/
Scalability AWS IntegrationSecurity &
Monitoring
Availability &
Data Protection
Ease of use &
TCO
Dev Platform &
Tools
• Fully-managed –
automated failure
management,
software patching
and maintenance
• Fast deployments
• Support for
search templates
• Zone awareness
to automatically
replicate data
across two
Availability Zones
• Automatic failure
recovery without
service disruption
• Automated
backups
• Authentication via
Amazon IAM
• Index level
access control
• Auditing via
Amazon
CloudTrail
• Monitoring and
Alerting via
Amazon
CloudWatch
• Seamless
scalability without
availability issues
• Wide variety of
instance types
and storage
options
• Up to 100 node
cluster support
• Programmatic
scale-up/ scale-
down support
based on
CloudWatch
metrics
• Data ingestion -
Amazon Kinesis
Firehose, Amazon
IOT, Amazon
CloudWatch Logs
• Amazon
CloudFormation
support
• Auditing and
Monitoring -
Amazon CloudTrail,
Amazon
CloudWatch
• Amazon IAM based
authentication
• Elasticsearch
open-source
APIs, supports
versions through
ES 5.5
• Logstash support
data for ingestion
and
transformation
• Kibana support
for visualization
• Support for 11
plugins covering
areas such as
extended Unicode
support, Phonetic
analysis etc.
Amazon ES benefits

7. Get started at https://aws.amazon.com/elasticsearch-service/
FT – Financial Times
P R O B L E M
• Which stories do our readers
care about? What’s hot?
• Required a custom
clickstream analytics
solution.
• Need a solution that delivers
analytics in real-time.
• Did not have a team to
manage analytics
infrastructure.
S O L U T I O N
• Streaming user data to
Elasticsearch Service for analysis.
Created their own custom
dashboards for editors and
journalists – Lantern.
• Lantern - ”shines a light” on reader
activity for the editors and
journalists at the FT.
• Critical tool for making editorial
decisions. Daily editorial meetings
B E N E F I T S
• Reliability - Lantern is used
throughout the day by
journalists and editors.
Relying on Amazon to
manage their systems for
maximum uptime.
• Cost savings - Able to easily
tune their cluster to meet their
specifics needs without much
management overhead

8. Security Model
for Amazon ES

9. Get started at https://aws.amazon.com/elasticsearch-service/
Service architecture
AWS SDK
AWS CLI
AWS CloudFormation
Elasticsearch
data nodes
Elasticsearch
master nodes
Elastic Load
Balancing
AWS IAM
CloudWatchCloudTrail
Amazon Elasticsearch Service domain
Internet or
your VPC
AWS Console

10. Get started at https://aws.amazon.com/elasticsearch-service/
AWS Security Responsibilities
• Creation of a service VPC that allows limited access to
the instances in your domain
• Application of security patches on the instances
• DDOS protection for the DNS name associated with the
domain via Route53
• Hides ports 9200 and 9300
• Built on top of AWS secure networking

11. Get started at https://aws.amazon.com/elasticsearch-service/
• Decide on your authentication strategy
• IAM
• VPC
• Identify key roles and build IAM policies or create
networking infrastructure to support their interaction with
Amazon ES
• Apply and maintain IAM policies and security groups to
AWS resources
Customer Security Responsibilities

12. Using IAM for Authentication

13. Get started at https://aws.amazon.com/elasticsearch-service/
Access via the internet

14. Get started at https://aws.amazon.com/elasticsearch-service/
Use a template or write your own
We strongly recommend against using an “open access” policy

15. Get started at https://aws.amazon.com/elasticsearch-service/
IAM policy application and resolution
role
Elastic IP
address
role
policy
policy
policy
Users have roles
Or policies
Instances have roles
with policies and IPs
Amazon ES domains
have policies
• IAM authenticates based on all applicable identification and all policies
are in play
User-based policies Resource-based policy

16. Access Policy Application & Resolution
• Deny ALWAYS wins over competing policy types
• If you do not explicitly state a policy, deny is default
Allowed in a
resource-based
policy
Denied in a
resource-based
policy
Neither allowed
nor denied in a
resource-based
policy
Allowed in an
identity-based policy
Allow Deny Allow
Denied in an
identity-based policy
Deny Deny Deny
Neither allowed nor
denied in an
identity-based policy
Allow Deny Deny

17. Get started at https://aws.amazon.com/elasticsearch-service/
Policy skeleton
{
"Version": "2012-10-17",
"Statement": [ {
"Effect":...
"Principal": ...
"Action": [...],
"Resource": ...,
"Condition": ...
} ]
}
• Effect: Allow or Deny
• Principal: AWS account ID
• Action
• HTTP verbs
• Service actions
• Resource: Amazon ES
domain/index
• Condition: IP Address

18. Get started at https://aws.amazon.com/elasticsearch-service/
Baseline IP-based access
Amazon ESIAM
Elastic IP
address
• Accessing from a known IP address
• The domain’s policy controls the Actions that are allowed/denied
• Access is anonymous
Policy

19. Get started at https://aws.amazon.com/elasticsearch-service/
• IP-based control, resource-based policy. All users, all
Actions, all indexes
{
"Sid": "",
"Effect": "Allow",
"Principal": { "AWS": "*” },
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*”
"Condition": {
"IpAddress": {
"aws:SourceIp": [”1.2.3.4"]
} }
}
Access policy for IP-based access

20. Get started at https://aws.amazon.com/elasticsearch-service/
Identity-based access
Amazon ESIAM
• Accessing with an identity, authenticated with
access/secret key
• Policy resolution as described
• Access is via signed request
Policy
AWS SigV4
signing
Lambda
AWS SigV4
signing

21. Get started at https://aws.amazon.com/elasticsearch-service/
• Requests must be signed. User-name-1 can run all
actions against all indices
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::12345678910:user/user-name-1”
]
},
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*”
}
Access policy

22. Get started at https://aws.amazon.com/elasticsearch-service/
Application access
Application
Amazon ES - search
Amazon ES - monitor
User
Amazon RDSUpdater
Kibana
Proxy
Administrator
DevOps/IT

23. Get started at https://aws.amazon.com/elasticsearch-service/
Create roles for access to Amazon ES
Administrator IT/DevOps Application Updater Proxy
Type User-based User-based Resource-based Resource-based Resource-based
Actions • es:CreateElasticsearchDomain
• es:Describe*
• es:DeleteElasticsearchDomain
• es:ListDomainNames
• es:AddTags
• es:ListTags
• es:RemoveTags
• es:Update*
• es:ESHttpGet
• es:ESHttpPut
• es:ESHttpDelete
• es:ESHttpPost
• es:Describe*
• es:ListDomainNames
• es:AddTags
• es:ListTags
• es:RemoveTags
• es:Update*
es:ESHttpGet es:ESHttpPost es:ESHttpGet
Resources Amazon ES search
Amazon ES monitor
Amazon ES search
Amazon ES monitor
Amazon ES search Amazon ES search Amazon ES Monitor
IPs No No EIP for the application
instance
EIP for the updater
instance
EIP for the proxy

24. Amazon Elasticsearch Service
in your VPC

25. Get started at https://aws.amazon.com/elasticsearch-service/
Amazon ES architecture with VPC
Availability Zone A
Availability Zone B
VPC subnet
security group
VPC subnet
security group
Amazon Elasticsearch Service
Data Master
Data
Master
IAM
IAM

26. Get started at https://aws.amazon.com/elasticsearch-service/
Set up for VPC access
• Select your VPC
• Select a subnet with
sufficient IP space and
ENIs for 3x your data
instances
• Select a security group
to apply to the Amazon
ES ENIs

27. Get started at https://aws.amazon.com/elasticsearch-service/
Simple VPC access
Availability Zone A
Subnet A
security group
Amazon Elasticsearch Service
Data
Master
Application
Instance(s)
Amazon RDS
Internet
gateway
Amazon Route 53

28. Get started at https://aws.amazon.com/elasticsearch-service/
Simple VPC access
• Internet gateway provides access for application users,
search, and monitoring traffic within the subnet
• Security group has normal inbound/outbound rules
• Because the IPs are within the security group, SigV4
signing is not required

29. Get started at https://aws.amazon.com/elasticsearch-service/
Application search within VPC
Availability Zone A
Availability Zone B
VPC subnet
security group
VPC subnet
security group
Amazon Elasticsearch Service
Data Master
Data
Master
IAMApplication
Application IAM
Internet
gateway

30. Get started at https://aws.amazon.com/elasticsearch-service/
Application search within the VPC
• With Zone Awareness enabled, the domain is in 2
subnets
• IAM provides additional security for IP-based or signed
requests

31. Get started at https://aws.amazon.com/elasticsearch-service/
Logging infrastructure in your VPC
security group
ELB
Logstash Indexers
Amazon Elasticsearch
Service Domain
Internet
gateway

32. Get started at https://aws.amazon.com/elasticsearch-service/
Logging infrastructure in your VPC
• Logstash colocated with the infrastructure you are
monitoring
• Use an ELB across an autoscaled group of indexers to
batch and forward to Amazon Elasticsearch Service
• Use a reverse proxy in the VPC to forward Kibana traffic
to Amazon ES

33. Get started at https://aws.amazon.com/elasticsearch-service/
Conclusions
• Amazon Elasticsearch Service now supports access
control using VPC security groups
• With IAM, you can further narrow access for particular
users
• All traffic remains within your VPC

34. Find out more:
https://aws.amazon.com/elasticsearch-service/
AWS Centralized Logging:
https://aws.amazon.com/answers/logging/centralized-logging/
Elasticsearch at the AWS Database Blog:
https://aws.amazon.com/blogs/database/category/elasticsearch/
Or ask your Solutions Architect!
Amazon
Elasticsearch
Service