openATTIC is the webUI for managing ceph storage in SUSE Linux and it is also accepted upstream as part of the default managing UI. Within openATTIC, it contains lots of different projects to provide all the functions including salt, Grafana and Prometheus.
This will not be a ceph or openATTIC focus talk, even we use it as the example of showing how everything work together. Instead we will look into openATTIC to see how each project work with each other. Mainly focus on Grafana and Prometheus, they are not only very useful for ceph and openATTIC but also equally powerful for monitoring your clusters, vm or cloud / container status.
Both Grafana and Prometheus are very easy to extend, which allow user or administrator to build the dashboard which fit their own need. Even the presentation example will be base on ceph / storage. Participants can use the same idea to monitor any system status they wanted after understand how they work.
3. Things related to openATTIC
● Ceph
● Salt and DeepSea
● Grafana + Prometheus
4. Ceph Overview
❖ https://github.com/ceph
❖ Ceph is a distributed storage system
❖ Object, Block and File storage all in one!
❖ Self Recovery and Self healing!
❖ openATTIC become part of the management
interface
➢ https://www.openattic.org/posts/ceph-manager-dashboard-v2/
6. Salt Overview
❖ https://github.com/saltstack/salt
❖ Salt is configuration management like ansible
➢ possible more :)
❖ Agent based distributed remote execution system,
➢ possible agentless with ssh
❖ Python base DevOps Automation tools using ZeroMQ
❖ Communication back channel for multiple product in
SUSE product line, including SES: DeepSea, SUSE
Manager and Cloud
10. OpenATTIC Overview
❖ https://bitbucket.org/openattic/openattic/
❖ Ceph Management and Monitoring GUI
❖ A easy to use administrative tools that actually work
❖ Highly customizable and scale out with ceph
❖ Expert / Local configuration aware without creating
inconsistent ceph cluster
❖ In 3.X above Grafana and Prometheus included
12. Grafana Overview
❖ https://github.com/grafana
❖ A beautiful Visualization and Monitoring platform
❖ Time series Analytics tool support multiple data store
❖ Highly customizable Dashboard design
❖ Query Editors, Alert Notification Service
❖ Community marketplace for Dashboard
14. Prometheus Overview
❖ https://github.com/prometheus
❖ Multi-dimensional data collects and store with time
series by metric / key value pairs
❖ Pull distributed model over http, scalable and
effective
➢ Push gateway is also supported
❖ Querying using PromQL is flexible and easy to use
❖ Grafana and other graphing dashboarding supported
16. Node Exporter Overview
❖ https://github.com/prometheus/node_exporter
❖ default running on port 9100
❖ Node_exporter is a metrics collectors written in
golang
❖ It act like agent allow prometheus collect data
18. Create your own Node Exporter
module
❖ After forking the node_exporter
➢ git clone https://github.com/AvengerMoJo/node_exporter.git
❖ Use other collect as example to start
➢ https://github.com/AvengerMoJo/node_exporter/blob/lio_exporter/collector/lio.go
❖ create iscsi module using NewLioCollector
70 func init() {
71 registerCollector("iscsi", defaultEnabled, NewLioCollector)
72 }
19. ❖ newLioMetric is internal function to create data structure
for node_exporter being pull from Prom
❖ lioCollector is internal structure store all the statistic data
❖ NewLioCollector return lioCollector with newLioMetric
statistics in the following code sample
75 func NewLioCollector() (Collector, error) {
… ( sysfs is node_exporter format need follow upstream )
80
81 metrics, _ := newLioMetric()
82
83 return &lioCollector{
84 fs: fs,
85 metrics: metrics}, nil
86 }
23. Build and Run Node Exporter
module
❖ You need to copy your update into the your home go dir
➢ go build github.com/prometheus/node_exporter
❖ Default collector doesn’t start / enable
❖ Enable module -collectors or --collectors after 0.15
version
➢ node_exporter -collectors.enabled iscsi
❖ Then you can check your metrics and data in port 9100
➢ http://node:9100/metrics
24. Node Exporter
http://node:9100/metrics
# HELP iscsi_lio_rbd_iops iSCSI RBD backstore transport operations.
# TYPE iscsi_lio_rbd_iops counter
iscsi_lio_rbd_iops{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 474
# HELP iscsi_lio_rbd_read iSCSI RBD backstore Read in byte.
# TYPE iscsi_lio_rbd_read counter
iscsi_lio_rbd_read{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 4.194304e+06
# HELP iscsi_lio_rbd_write iSCSI RBD backstore Write in byte.
# TYPE iscsi_lio_rbd_write counter
iscsi_lio_rbd_write{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 1.048576e+07