This slide was used architecture show and tell at OpenStack Ops Meetup in Okinawa Open Days 2015.

1. Copyright © NTT Communications Corporation. All rights reserved.
Mahito Ogura <m.ogura@ntt.com>
NTT Communications
Collect, summarize and notify
of OpenStack’s log

2. Copyright © NTT Communications Corporation. All rights reserved.
Mahito Ogura <m.ogura@ntt.com>
Technology Development
DevOps Engineer
R&D for IaaS, NoSQL, DB as a Service,
Hadoop as a service
Contributing to Devstack
About me
2

3. Copyright © NTT Communications Corporation. All rights reserved.
Our team’s mission
● To evaluate new components and functions for production
● To accumulate knowledge of OpenStack’s operation for production
Challenge
● Continuously deploy OpenStack
while constantly keeping service functionality and performance
○ This challenging story was spoken in Tokyo summit.
“Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack”
Background
3

4. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark is helpful, but ...
4
Our challenge is successful !!
● Auto deployment and benchmarking can be constantly keeping service
functionality and performance
Output to many OpenStack’s logs
● Normal: 40 ~ 50 lines/min x 3 cluster（16 hosts）
● Benchmark: 220 ~ 260 lines/min x 3 cluster（16 hosts）
Debug: False
Only Nova

5. Copyright © NTT Communications Corporation. All rights reserved.
5
It is hard to manually find issues in logs...
barockschloss - https://www.flickr.com/photos/8663137@N04/4943743444/

6. Copyright © NTT Communications Corporation. All rights reserved.
Overview of System Architecture
OpenStack logs
Host resource data
Storage
visualizestore
notify
pull
forward
store
Stream processing server
Time series database
Team collaboration tool
Metrics dashbord
& graph editor
Data collector
6

7. Copyright © NTT Communications Corporation. All rights reserved.
Today’s talk target
OpenStack logs
Host resource data
notify alert
forward log data
7
pull summrized data
Stream processing server
Data collector

8. Copyright © NTT Communications Corporation. All rights reserved.
Fluentd
Fluentd is an open source data collector
Before Fluentd After Fluentd
Image from https://github.com/fluent/fluentd-docs 8

9. Copyright © NTT Communications Corporation. All rights reserved.
Collect OpenStack’s logs
OpenStack logs (fluent-plugin-tain)
9
nova-api.log
2015-10-19 17:29:37.832 25350 INFO nova.osapi_compute.wsgi.server [-]
10.1.16.22,127.0.0.1 "GET / HTTP/1.1" status: 200 len: 280 time: 0.0009720
{ "asctime":"2015-10-19 17:29:37.832",
”code":”200",
"loglevel":"INFO",
"objname":"nova.osapi_compute.wsgi.server”,
…
}
tag: openstack.nova.api.log
forward
parse

10. Copyright © NTT Communications Corporation. All rights reserved.
Forward and Output
OpenStack logs
Host resource data
Storage
tag:influxdb.**
tag:slack.**
tag:norikra.**
tag:**
tag:storage.**
10
<match openstack.**>
type copy
<store>
type map
tag (“norikra.” + tag)
</store> <store>
type map
tag (“influxdb.” + tag)
</store>
<store>
type map
tag (“storage.” + tag)
</store>
</match>
<match dstat.**>
…
Fluentd server configuration

11. Copyright © NTT Communications Corporation. All rights reserved.
Norikra
Norikra is Open source server software
● Enable to select/count/grouping/etc.. from receiving data by using SQL
● Enable to use time window (.win:time_batch(60 s))
11
Query Example:
SELECT hostname, loglevel, NULLABLE(message),
COUNT(*) AS count
FROM norikra_nova_compute_log.win:time_batch(1 min)
WHERE loglevel = 'WARNING’
GROUP BY hostname, loglevel, message
http://norikra.github.io/

12. Copyright © NTT Communications Corporation. All rights reserved.
Norikra vs other stream proecessing
Pros
● Easy to install
● Don’t write code! Enable to use SQL
Cons
● Disable to distribute data
● Disable to complex processing
● Doesn’t have availavility
12

13. Copyright © NTT Communications Corporation. All rights reserved.
Summarize ‘WARNING’ message
OpenStack logs
Host resource data
forward log data
13
pull summrized data
[
1445837968,
{"message":"No network configured!",
"loglevel":"WARNING",
"count":38,
"hostname":"compute-c3"}
]...
SELECT hostname, loglevel, NULLABLE(message),
COUNT(*) AS count
FROM norikra_nova_compute_log.win:time_batch(1 min)
WHERE loglevel = 'WARNING’
GROUP BY hostname, loglevel, message
Output summrized data per 1 min

14. Copyright © NTT Communications Corporation. All rights reserved.
Case: Notify a warning alert
OpenStack logs
Host resource data
tag: norikra.**
tag: slack.**
14
tag: slack.nova.compute.
warning.log
SELECT hostname, loglevel, NULLABLE
(message), COUNT(*) AS count
FROM norikra_nova_api_log.win:time_batch(1 min)
WHERE loglevel = 'WARNING’
GROUP BY hostname, loglevel, message
[
1445837968,
{"message":"No network configured!",
"loglevel":"WARNING",
"count":38,
"hostname":"compute-c3"}
]...

15. Copyright © NTT Communications Corporation. All rights reserved.
OpenStack’s logs are large and rapid stream
● All issues are hard to find in log stream
Fluentd & Norikra are powerfully software
● Easy to collect and forward for log data
● Summarizable to large log data by using SQL
● Enable to notify WARNING/ERROR alerts
Conclusion
15

16. Copyright © NTT Communications Corporation. All rights reserved.
16