"Shipping logs to Splunk from a container in AWS howto. Advantages of running containers in AWS Fargate" by Oleksii Makieiev, Senior systems engineer EPAM Ukraine

1. SHIPPING LOGS TO SPLUNK FROM A
CONTAINER IN AWS HOWTO.
ADVANTAGES OF RUNNING
CONTAINERS IN AWS FARGATE.

2. Необходимо предоставить решение по доставке
логов приложения в Спланк клауд из контейнера в
AWS, с использованием HEC
Миграция java сервисов с AWS EB на AWS Fargate

3. LOGGING
Неоходимо отказаться от использования on-premises kafka
логгирования и перейти на облачный сервис Спланк используя
HTTP Event Collector;
Отсылаем все что приложение пишет в stdout stderr;
Сделать это нужно не используя CloudWatch;
Мы запускаем контейнеры в AWS EB – 90% и AWS FG – 10%
The HTTP Event Collector (HEC) is a fast and efficient
way to send data to Splunk Enterprise and Splunk Cloud.
Notably, HEC enables you to send data over HTTP
(or HTTPS) directly to Splunk Enterprise or Splunk Cloud
from your application.

4. Fluentbit – Cloud native log forwarder
Fluentd – Unified logging Layer
Filebeat+Logstash
Filebeat и Fluentd не подошли по причине своей тяжеловесности.

5. Ради экперимента сделали контейнер c приложением + fluentbit
RUN mkdir -p /home/fluent-bit && cd /home/fluent-bit &&
wget https://fluentbit.io/releases/1.0/fluent-bit-1.0.6.tar.gz &&
tar xzf fluent-bit-1.0.6.tar.gz &&
rm fluent-bit-1.0.6.tar.gz &&
cd fluent-bit-1.0.6 &&
cmake . && make && make install
Возникла необходимость скриптом запускать оба процесса и контролировать
чтобы один из процессов не завершил работу.

6. [SERVICE]
Flush 5
Daemon on
Log_Level Info
Coro_Stack_Size 8092
[INPUT]
Name tail
Path /var/log/someservice.log
Refresh_Interval 7
Tag app
[FILTER]
Name nest
Match *
Operation nest
Wildcard *
Nest_under event
[FILTER]
Name modify
Match *
Add index main_dev
Add source http:
Add sourcetype someservice
[OUTPUT]
Name splunk
Host *.splunkcloud.com
Port 443
tls On
tls.Verify Off
Splunk_Token SPLUNK_TOKEN_PLACEHOLDER
Splunk_Send_Raw On
Match *

7. Что рекомендуют на сайте Спланка?
AWS Cloudwatch + AWS lambda

8. Docker-compose?!!
Application container + Fluentbit container !
Что предлагает AWS EB для мультиконтейнерных приложений?
“AWSEBDockerrunVersion”: 2
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html#create_deploy_docker_v2config_dockerrun_format

9. {
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "app",
"image": “some-service:@build_number@",
"essential": true,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 8080
}
],
"links": [
"fluentd"
],
"logConfiguration": {
"logDriver": "fluentd",
"options": {
"fluent-address": "fluentd:24224"
}
}
},
{
"name": "fluentd",
"image": “fluentd-logger:403323",
"essential": true
}
]
}
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "my-bucket",
"Key": "mydockercfg"
},
"Image": {
"Name": "quay.io/johndoe/private-image",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "1234"
}
],
"Volumes": [
{
"HostDirectory": "/var/app/mydb",
"ContainerDirectory": "/etc/mysql"
}
],
"Logging": "/var/log/nginx"
}

10. docker run --help

11. ERROR: Encountered error starting new ECS task: {cancel
the command.
"failures": [
{
"reason": "ATTRIBUTE",
"arn": "arn:aws:ecs:ap-northeast-
1:000000000000:container-instance/00000000-0000-
0000-0000-000000000000"
}
],
"tasks": []
}
ERROR: Failed to start ECS task after retrying 2 times.
ERROR: [Instance: i-00000000] Command failed on
instance. Return code: 1 Output:
beanstalk/hooks/appdeploy/enact/03start-task.sh failed.
For more detail, check /var/log/eb-activity.log using
console or EB CLI

12. files:
"/home/ec2-user/setup-available-log-dirvers.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
set -e
if ! grep splunk /etc/ecs/ecs.config &> /dev/null
then
echo 'ECS_AVAILABLE_LOGGING_DRIVERS=["json-
file","syslog","fluentd", "splunk"]' >> /etc/ecs/ecs.config
fi
container_commands:
01-configure-splunk:
command: /home/ec2-user/setup-available-log-dirvers.sh
02-stop-ecs:
command: stop ecs
03-stop-ecs:
command: start ecs
001_splunk.config

13. AWS ELASTICBEANSTALK
Почему получилось так что его использовали для
запуска контейнеров

14. AWS
ELASTICBEANSTALK
 01/2011
 04/23/2014 We are excited to announce that you
can now create and manage Docker containers in
AWS Elastic Beanstalk.
 Elastic Beanstalk is built on top of the proven AWS infrastructure.
It takes full advantage of Amazon EC2, Elastic Load Balancing,
Amazon CloudWatch, Auto Scaling, and other AWS services. You
get all of the economy and scalability of AWS in a form that’s
easier and quicker to deploy than ever before.
 With Elastic Beanstalk you can choose to gradually assert control
over a number of aspects of your application. You can start by
tuning a number of parameters (see my post on the Elastic
Beanstalk Console for more information about this). You can
choose the EC2 instance type that provides the optimal amount of
RAM and CPU power for your application. You can log in to the
EC2 instances to troubleshoot application issues, and you can
even take the default Elastic Beanstalk AMI (Amazon Machine
Image), customize it, and then configure Amazon Beanstalk to use
it for your application. This gradual assertion of control extends all
the way to “eleven” — you can choose to move your application
off of Elastic Beanstalk and manage the raw components yourself
if you so choose.
 Elastic Beanstalk was designed to support multiple languages and
application environments. We are already working with solution
providers to make this happen.
 Each of your Elastic Beanstalk applications will be run on one or
more EC2 instances that are provisioned just for your application.
Applications running on Elastic Beanstalk have the same degree of
security as those running on an EC2 instance that you launch
yourself.

15. AWS FARGATE
 Announced 11/29/2017
 AWS Fargate is a compute engine for deploying and
managing containers without having to manage any of the
underlying infrastructure. Fargate makes it easy to scale your
applications. You no longer have to worry about provisioning
enough compute resources for your container applications.
You can launch tens or tens of thousands of containers in
seconds.
 Previously, you needed to manage a cluster of Amazon EC2
instances, pick the instance types, manage the scheduling of
the containers, and optimize cluster utilization. With Fargate,
all of this goes away. Fargate seamlessly integrates with
Amazon ECS. You just define your application as you do
today for Amazon ECS. You package your application into
task definitions, specify the CPU and memory needed, define
the networking and IAM policies each container needs. Once
everything is setup, Fargate launches and manages your
containers for you.
 With Fargate, billing is at a per second granularity and you
only pay for what you use. You pay for the amount of vCPU
and memory resources your containerized application
requests. vCPU and memory resources are calculated from
the time your container images are pulled until the Amazon
ECS Task terminates, rounded up to the nearest second.

16. AWS EB pricing
 There is no additional charge for AWS
Elastic Beanstalk. You pay for AWS
resources (e.g. EC2 instances or S3
buckets) you create to store and run
your application.
Fargate pricing
per vCPU per hour $0.04048
per GB per hour $0.004445
Daily charge using Docker in AWS EB:
t2.small linux
24 * $0.0208 = 0.4992 + 0.0266(EBS price) =
0.5258
Daily charge using Fargate:
1vCPU + 2Gb
24*(0.25*0.04048+2*0.00445) = 0.45648
Effective Jan 07, 2019, we are reducing the price for AWS Fargate by 20% for vCPU and 65%
for memory across all regions where Fargate is currently available.

17. A few Fargate advantages:
• No cluster to manage
• Seamless scaling
• Much easier to deploy
• Supports shipping logs to SPLUNK out of the box