Sensu has become a critical component to keeping the modern visual effects studio of Industrial, Light & Magic in the business of creating the beautiful movies of our world and realizing the dreams we all enjoy on the big screen. In this talk from Sensu Summit 2018, Christopher J. Caillouet, Senior Dev|Ops Production Engineer at Industrial Light & Magic, looks behind the curtain and sees how the intelligence and uptime they gain by leveraging Sensu in the ILM monitoring infrastructure enables reliability and stable delivery within a large scale and geographically distributed set of datacenters.

1. MEANINGFUL
MONITORING &
MESSAGING
PROJECT
3M

2. PRESENTER
Christopher Caillouet
ILM Dev|Ops Engineering

3. What do businesses
want from
monitoring?

5. BUSINESS
INTELLIGENCE
THROUGH
MONITORING
Meaningful Monitoring
& Messaging

6. ●100% Open Source!! (Spend dollars for significant value)
●Ansible deployed infrastructure
●Ansible deployed services
●Dev, Staging and Production environments
●Sensors and Templates authored via code (YAML & Ruby)
●Monitoring occurs within datacenter local to monitored object
●Data visualization can occur across global footprint
●Scalable approach as we grow in sensor/service/studio count
●Flexible tool set for diverse consumption of platform
PROJECT 3M
DIRECTIVES

7. DEV
STAGING
PRODUCTION
OUR PIPELINE
APPROACH

8. Meaningful Monitoring
& Messaging
PLATFORM
BREAKDOWN

12. Sens
u

14. Ansible
Leveraging our
extensive internal set of
playbooks during
handled events is
building our path
toward automated
remediation of issues

15. Jenkins
We are always in need
of intelligence around
build times and job
status metrics to ensure
we are able to deliver
code to meet the needs
of the studio

16. SSL
Keeping us in touch
with the validation and
expiration of the various
certificates we have
throughout our services
and systems

17. CPU
Systemd
SNMP
Zendesk
Docker
Vsphere
Gitlab
HTTP
Varnish
PostgreSQL
Mongodb
Consul
Google Sheets
Memory
Supervisor
Graylog
LDAP
Kubernetes
Openstack
Tomcat
Memcached
Elasticsearch
RabbitMQ
Redis
Vault
Hipchat
BUT
WAIT…
...THERE’
S
MORE!

19. CONSUL

20. Consu
l

23. Elasti
c
SERVER
DATA
SERVER
DATA
SERVER
DATA
SERVER
DATA
TIMESERIES&LOGDATA

25. GRAFANA

26. Grafan
a
SENSU
DATA
CONSUL
DATA
ELASTIC
DATA
CUSTOM
DATA
INTERSECTIONSOFDATA

28. Ansible Filename: your_service.yml
---
- name: Configure 3m monitoring for your_service
hosts: your_service.lucasfilm.com
become: true
vars:
mmm_service_name: "your_service"
mmm_service_port: 80
mmm_service_process_name: "nginx"
roles:
- 3m_service
- custom_role
Sampl
e

29. Ansible Filename: your_service.yml
Would give you…
1.Service Discovery in Hashicorp Consul
a.Consul infrastructure monitored within Sensu
2.Centralized logging via Sensu handlers into ELK
3.Sensu monitoring of PING, CPU, RAM, HDD for base role
a.additional Ansible roles added for plugin based
check(s)
b.additional Ansible roles added for custom authored
check(s)
c.additional or custom Sensu handlers
i. elasticsearch, chat, email, SMS… and more!
4.Notifications for alerts where you want to get them
5.Customizable Grafana dashboard for your service
Sampl
e

36. QUESTIONS,
YOU
HAVE?

37. THANK YOU!

38. THANK YOU!

40. Filename: /etc/consul.d/3m_your_service.json
{
"services": [
{
"name": "your_service",
"tags": ["3m"],
"port": 80,
"check":{
"args": ["curl", "localhost:80"],
"interval": "20s"
}
}
]
}
consul.
d

41. Filename: /etc/sensu/conf.d/client.json
{
"client": {
"name": "your_service.lucasfilm.com",
"address": "1.2.3.4",
"environment": "prod",
"subscriptions": ["3m"]
}
}
sensu.d

42. THANK YOU!