These slides are from my talk in the BrownBag session in OpenStack Summit Paris. In this talk, I share our experience with building a highly scalable multi-tenant cloud service for logging, metrics and monitoring. Talking to other groups that are building similar services, I found that what we are building is unique in some aspects and the scale we are already using it is pretty cool! The video was also recorded and you can check that out here: http://youtu.be/wers9XNO26Q

1. Logging, Metrics and Monitoring
as a Service
Lakshminarayanan (LN) Renganarayana
Architect, Cloud Platform Engineering
LMM, Cloud Platform Engineering Nov 4, 2014 1

2. LMM, Cloud Platform Engineering Nov 4, 2014
2
The Symantec Team
• Cloud Platform Engineering
– We are building a consolidated cloud platform that provides infrastructure
and platform services for next generation Symantec products and services
– starting small, but scaling to tens of thousands of nodes across multiple data
centers
• Me
– Architect / Lead for LMM in Cloud Platform Engineering
– Love to build and break systems
– Previous life: Research Staff Member, IBM Research
– Automatic parallelization, Compilers, Deployment Automation (Heat)

3. Context for LMM: Symantec Cloud Platform
Common
Services
LMM, Cloud Platform Engineering
3
Symantec Services & Apps
Platform Services
BP, SP, KV, OBS
IaaS (OpenStack)
Bare Metal
Nov 4, 2014
LMM
Identity
Manager
CI / CD

4. LMM : Goals we started with …
• Design for both Developers and Ops
• Multi Tenant, Elastic Scalability
• One system for logs, metrics, monitoring, & other events
• Designed for under and over the cloud
• Build using open source tools and for open sourcing
LMM, Cloud Platform Engineering Nov 4, 2014
4

5. Guiding principle
Make it extremely simple to generate logs & metrics
Provide powerful query and visualizations for free
LMM, Cloud Platform Engineering Nov 4, 2014
5
Gen with 1 line: logger.warn(“WARN KVaaS too many failed writes”)
Query with all the power of ElasticSearch and visualize with Kibana
Gen with 1 line: statsd_client.incr(‘myapp_num_requests’)
Query with all the power of InfluxDB and visualize with Grafana

6. LMM Client Interface
expose
metrics
JMX
Push
Metrics
LMM, Cloud Platform Engineering Nov 4, 2014
6
App
Platform
Services
Writes app
metrics directly
Infrastructure
SNMP Vars
Pull
Metrics
LMM
Shipper
VM
logs
LMM
metrics log events
Alerts &
Config UI
Push: StatsD, metrics extension for openstack
Pull: CollectD. Shipper: logstash, lumberjack

7. Security and Multi-tenancy
• User authentication with Keystone for Query API & UI
• Tenant id and API key used for events sent to LMM
– Tenant ids from Keystone and API keys generated by LMM
• Every event is tagged with a tenant id
– Log events: tenant id as a field
– Metric events: tenant id prefixed to the metric name
• Keystone group membership used for sophisticated cross-tenant
event visibility / correlation
LMM, Cloud Platform Engineering Nov 4, 2014
7

8. Client App / Service
log & metrics shipper
log, metric, alert events
log, metric & alert events
Authentication, Validation, Alerts Processing
s1 s2 s3 s4 ... sn
LMM, Cloud Platform Engineering Nov 4, 2014
8
Kafka cluster
knode1
Keystone
knode2 knode3 ... knodeN
Storm cluster
Front End Cluster: Multi-tenancy and Kibana, Graphana Proxies
Elastic
Search
Elastic
ReSdeiasrch
Alerts email &
callbacks
Load Balancer
InfluxDB
InfluxDB
InfluxDB
Metrics Store
Elastic
Search
Elastic
Search
Elastic
Search
Log Store
LMM Under the Hood
Quota
&
Policy

9. Current deployment
• Multiple deployments : on bare KVM nodes, on OpenStack VMs
– On KVM nodes: 40+ VMs, 80+ TB storage, many large memory nodes
– All components are deployed in clustered mode for HA
– Some with active/active replication, some with active/passive
• Use by Platform and Infrastructure Services
– Hundreds of thousands of events per second
– Few TBs of data collected and indexed per day
– Queries are currently coming from Kibana and Grafana, in future from APIs
LMM, Cloud Platform Engineering Nov 4, 2014
9

10. In the Next Iteration …
• Support Health checks
– Exploring Sensu and related technologies
• Multi data center deployment
• Refactor to a data pipeline architecture
• Deployment automation using Murano
• Open Source LMM
LMM, Cloud Platform Engineering Nov 4, 2014
10

11. Thank you!
Questions, Comments, Suggestions?
We are interested in open sourcing LMM
and are looking for collaborators …
Interested? Feel free to contact me
LN_Renganarayana@Symantec.com
LMM, Cloud Platform Engineering Nov 4, 2014 11