“System DataBase” (SysDB, https://sysdb.io/) is a multi-backend system management and inventory collection service. It may be used to (continuously) collect information about your systems from various backends (inventory services, monitoring services, etc. like Nagios, Puppet, collectd) and provides a unique interface to access the information independent of the active backends. This is done by storing and mapping the backend objects to generic objects and correlating the attributes to create a single hierarchical view of your infrastructure. This way, all important information about your systems is accessible from a central location allowing for use-cases such as central dashboards, cross-link monitoring or inventory information, identify missing pieces in your system configuration, and much more. This talk provides an overview over SysDB and its features as well as sample use-cases. The project is still in an early development stage but already usable. The talk also covers future directions and further integration with existing services.

1. SysDB – System DataBase
a system management and inventory collection service
Sebastian ‘tokkee’ Harl
<sh@tokkee.org>
SysDB developer
Open Source Data Center Conference 2014
April, 10 2014
Berlin

2. Disclaimer
WARNING:
SysDB is still under heavy development.
Flaming, bashing or other forms of constructive
feeback are very appreciated :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 2

3. Background / Motivation
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 3

4. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

5. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
Who also uses Puppet/Chef/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

6. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
Who also uses Puppet/Chef/etc.?
Who also uses collectd/Munin/etc.?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

7. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
Who also uses Puppet/Chef/etc.?
Who also uses collectd/Munin/etc.?
Who also uses some inventory service?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

8. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
Who also uses Puppet/Chef/etc.?
Who also uses collectd/Munin/etc.?
Who also uses some inventory service?
Who also maintains a CMDB?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

9. Motivation
Why SysDB?
Who uses Nagios/Icinga/Naemon/OpenNMS/etc.?
Who also uses Puppet/Chef/etc.?
Who also uses collectd/Munin/etc.?
Who also uses some inventory service?
Who also maintains a CMDB?
Who likes their setup? ;-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 4

10. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

11. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

12. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
Second thought: oh, then we can also add monitoring
information.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

13. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
Second thought: oh, then we can also add monitoring
information.
Then: this should be generic and extensible.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

14. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
Second thought: oh, then we can also add monitoring
information.
Then: this should be generic and extensible.
Then I realized that I don’t like web development.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

15. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
Second thought: oh, then we can also add monitoring
information.
Then: this should be generic and extensible.
Then I realized that I don’t like web development.
Also, a system like that would need a good back-end.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

16. History
Initially, I wanted a new front-end for collectd suitable for large,
heterogeneous environments.
First idea: group graphs using Puppet facts.
Second thought: oh, then we can also add monitoring
information.
Then: this should be generic and extensible.
Then I realized that I don’t like web development.
Also, a system like that would need a good back-end.
! the idea for SysDB was born
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 5

17. The System DataBase
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 6

18. SysDB – Overview
SysDB collects information about arbitrary hardware and software
systems.
The central idea is to get a central view of your infrastructure.
Simple examples:
Hosts and their attributes (“facts”)
Services and their attributes
Monitoring information (e.g. current state)
SysDB collects these information and correlates objects from
various backends.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 7

19. SysDB – core features
https://github.com/tokkee/sysdb
CI: https://travis-ci.org/tokkee/sysdb
60% code (function) unit-test coverage in the core
BSD license
Written in C
Easy to extend (simple plugin API)
Simple network protocol
Most of the code implemented as a library (reusable)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 8

20. SysDB – Plugins
Currently available:
collectd::unixsock – query collectd’s UNIX socket interface
mk-livestatus – query Monitoring systems (Nagios, Naemon,
Icing, Shinken) using Check MK Livestatus
puppet::store-configs – query Puppet
cname::dns – canonicalize host-names using DNS
syslog – syslog logging
Planned: Passive data collection (e.g. using Gearman), Foreman,
PuppetDB, $your favorite system (send patches!) :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 9

21. SysDB – Architecture
infra
backends
sysdbd
store FE
sysdb
network
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 10

22. The SysDB Store
The actual database implementation (in memory)
Stores generic objects:
Host – any kind of physical resource
Service – any kind of service
Attribute – attributes of hosts and services
Canonicalization of hosts
Each object stores the timestamp of the last update and the
(automatically calculated) update interval
Interface to query data
JSON is the external data representation
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 11

23. The SysDB Store – Examples
kvm0
kvm1
kvm2
ovirt0
vhost0
vhost0
vhost0
vhost0
vhost0
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 12

24. Internal Data Structures
message Host {
required string name;
required DateTime last_update;
required Interval update_interval;
repeated Attribute attributes;
repeated Object children; // Host or Service
optional Object parent;
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 13

25. Internal Data Structures
message Service {
required string name;
required DateTime last_update;
required Interval update_interval;
repeated Attribute attributes;
repeated Object children; // Host or Service
repeated Object parent; // multiple!
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 14

26. Internal Data Structures
message Attribute {
required string name;
required Type value;
required DateTime last_update;
required Interval update_interval;
optional Object parent;
}
Type is either an integer, floating point number, DateTime or binary
data.
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 15

27. The SysDB Client
Interactive client program for SysDB
Connects to a SysDB daemon
Interactive command shell
Receives and displays asynchronous log messages
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 16

28. The SysDB Query Language
Remotely similar to SQL
Meant to be easily usable in RPCs
Still under heavy development ;-)
Currently supported commands:
LIST – returns all hosts
FETCH hostname – returns details for a host
LOOKUP hosts WHERE expression – query details for
multiple hosts based on their attributes
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 17

29. The SysDB Query Language – Example(I)
sysdb= LIST;
{hosts:[{
name: nagios.lxc.tokkee.net,
last_update: 2014-04-03 10:26:41 +0200,
update_interval: 5m4s
},{
name: puppet.lxc.tokkee.net,
last_update: 2014-04-05 11:04:08 +0200,
update_interval: 5m2s
},{
name: whisky.mobile.tokkee.net,
last_update: 2014-04-05 11:09:15 +0200,
update_interval: 10s
}]}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 18

30. The SysDB Query Language – Example(II)
sysdb= LOOKUP hosts WHERE attribute.architecture = ’amd64’
AND service.name =˜ ’postgres’;
{ name: whisky.mobile.tokkee.net,
last_update: 2014-04-05 11:09:15 +0200,
update_interval: 10s,
attributes: [{
name: architecture, value: amd64,
last_update: 2014-04-03 10:26:41 +0200,
update_interval: 5m3s },{ ... }],
services: [{
name: cpu-0/cpu-idle,
last_update: 2014-03-31 23:34:06 +0200,
update_interval: 9.716754301s },{ ... }]
}
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 19

31. Use Cases
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 20

32. Use Cases
Extend (replace?) your CMDB
! put the CMDB behind and query it through SysDB
Compare the back-ends (monitoring)
! Which hosts / services are missing in a back-end?
! What is the global status of all Windows systems in some
data-center?
More flexible web-frontend combining multiple back-ends
! central dashboard
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 21

33. Future Directions
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 22

34. Future Directions
More documentation!
RDBMS store back-end (?)
Interface to query live data (monitoring status, metrics) from
backends
Distributed architecture (HA and load-balancing)
Web-Interface
Extend the type system and filters
Add support for related informationen (?):
persons / groups (ACLs, . . . )
Events / calendar (Change-Requests, . . . )
. . .
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 23

35. SysDB – System DataBase
Thank you for your attention!
Questions, comments, rants?
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 24

36. SysDB – System DataBase
Contakt:
Sebastian “tokkee” Harl
sh@tokkee.org
https://github.com/tokkee/sysdb
Please send patches! :-)
c 2014 Sebastian ‘tokkee’ Harl The System DataBase – slide 25