James Sweeney presents on "PuppetDB: A Single Source for Storing Your Puppet Data" at Puppet User Group NYC. Video: http://www.youtube.com/watch?v=HTr4b02aU7A Puppet NYC: http://www.meetup.com/puppetnyc-meetings/

1. IT Automation Software for System Administrators

2. James
Sweeny
james.sweeny@puppetlabs.com
Puppet
Labs,
Inc.
Professional
Services
A User’s Introduction to PuppetDB
2

3. What is PuppetDB
*The*
datastore
for
informaBon
generated
by
Puppet
3

4. PuppetDB Components
• HTTP
Server
• Middleware
Queue
• Worker
Threads
• Database
4

5. PuppetDB Components
• HTTP
Server
• Middleware
Queue
• Worker
Threads
• Database
5

6. Yes,it runs in a JVM
• WriMen
in
Clojure
• And
yes,
it’s
fast.
6

7. Lots of Data
7

8. Lots of Data
Small
environment:
~100
nodes
~800KiB
catalog
30
minute
interval
>3GiB/day
8

9. Lots of Data
Large
environment:
Start
adding
zeros.
Some
users
generate
over
750GiB/day
9

10. Puppet Data
• Facts
• Catalogs
• Reports
• Exported
Resources
10

11. Facts
• Collected
with
*every*
puppet
run
• Contain
lots
of
data
about
your
systems
• Available
from
the
inventory
service
11
architecture
=>
i386
augeasversion
=>
0.9.0
boardmanufacturer
=>
Intel
CorporaBon
boardproductname
=>
440BX
Desktop
Reference
Plaeorm
boardserialnumber
=>
None
domain
=>
localdomain
facterversion
=>
1.6.7
fqdn
=>
pe-­‐centos6.localdomain
hardwareisa
=>
i686
hardwaremodel
=>
i686
hostname
=>
pe-­‐centos6
id
=>
root
interfaces
=>
eth1,lo
ipaddress
=>
172.16.199.144
ipaddress_eth1
=>
172.16.199.144
ipaddress_lo
=>
127.0.0.1
is_virtual
=>
true
kernel
=>
Linux
kernelmajversion
=>
2.6
kernelrelease
=>
2.6.32-­‐220.13.1.el6.i686
kernelversion
=>
2.6.32
macaddress
=>
00:0C:
29:19:A2:33
macaddress_eth1
=>
00:0C:
29:19:A2:33
manufacturer
=>
VMware,
Inc.
memoryfree
=>
382.88
MB
memorysize
=>
499.15
MB
memorytotal
=>
499.15
MB
netmask
=>
255.255.255.0
netmask_eth1
=>
255.255.255.0
netmask_lo
=>
255.0.0.0
network_eth1
=>
172.16.199.0
network_lo
=>
127.0.0.0
operaBngsystem
=>
CentOS
operaBngsystemrelease
=>
6.2
osfamily
=>
RedHat
physicalprocessorcount
=>
1
processor0
=>
Intel(R)
Core(TM)
i7-­‐2620M
CPU
@
2.70GHz
processorcount
=>
1

12. Catalogs
• StaBc
configuraBon
of
every
Puppet
run
• What
the
node
*should*
be
12

13. Exported Resources
# Export the host file entry
@@host { $::fqdn:
host_aliases => $::hostname,
ip => $::ipaddress,
}
# Collect and apply all the host file entries
Host <<| |>>
13

14. Exported Resources
14

15. Exported Resources
@@nagios_host { $::fqdn:
ensure => present,
alias => $::hostname,
address => $::ipaddress,
use => ‘generic-host’,
}
@@nagios_service { “check_ping_${::hostname}”:
check_command => “check_ping!100.0,20%!500.0,60%”,
use => ‘generic-service’,
notification_period => ‘24x7’,
service_description => “${hostname}_check_ping”,
}
15

16. Exported Resources
Nagios_host <<||>> ~> Service[‘nagios’]
Nagios_service <<||>> ~> Service[‘nagios’]
16

17. Exported Resources - Removing
# puppet node deactivate <node name> [<another node>, ...]
17

18. Exporting Resources
18

19. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc

20. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc

21. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
F

22. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
CF C

23. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
F C
C

24. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

25. R
Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

26. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C R

27. Resource Collection
27

28. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
R

29. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
R

30. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
F
R
F

31. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
?F
R

32. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
?
R
F

33. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
? R
F

34. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
R
F

35. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
RF

36. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
RF

37. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
CC

38. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
C
C

39. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc
C

40. Failure and the DLO
40

41. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts Catalo
g
Resrc

42. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc

43. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
F

44. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
CF C

45. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
F C
C

46. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

47. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

48. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

49. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

50. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

51. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

52. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

53. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

54. Agent Master
PuppetDB Server
HTTP MQ
DB Workers
DLO
Facts
F
Catalo
g
Resrc
C
C

55. Dead Letter Office
Contains
complete
content
of
failed
transacBons
55

56. Installation
1.Install
Puppet
2.Enable
PuppetLabs
repos
3.
puppet module install puppetlabs/puppetdb
4.Classify
your
servers
56

57. Installation
node puppetmaster {
class { 'puppetdb': } ->
class { 'puppetdb::master::config': }
}
57

58. Scaling
1.Move
HSQLDB
to
Postgres
(at
>
100
nodes)
2.Separate
Postgres
3.MulBple
PuppetDB
servers
4.Clustered
Postgres
5.HTTP(s)
proxy
58

59. Querying
59

60. REST Interface
• By
default
listens
on
localhost:8080
• Can
easily
set
public
listener
class { ‘puppetdb’:
ssl_listen_address => $::ipaddress_eth0,
ssl_listen_port => ‘8081’,
}
60

61. REST Interface
• Queries
are
processed
immediately
• Commands
are
queued
61

62. REST Interface
• Responses
are
JSON
• Differ
by
query
type
62

63. Querying Facts
• /v2/fact_names
[ "architecture", "augeasversion",
"boardmanufacturer",
"boardproductname",...]
63

64. Querying Facts
• /v2/facts
• /v2/facts/<fact_name>
• /v2/facts/<fact_name>/<fact_value>
• Fact
Name
• Fact
Value
• Node
Name
64

65. Querying Facts
[ {
"certname" : "my_host.example.com",
"name" : "architecture",
"value" : "x86_64"
}, {
"certname" : "my_host.example.com",
"name" : "augeasversion",
"value" : "0.10.0"
}, ...
]
65

66. Querying Resources
• Query
all
resources
in
your
infrastructure
• /v2/resources/
• /v2/resources/<type>
• /v2/resources/<type>/<9tle>
66

67. Querying Resources
[ {
"parameters" : {
"ensure" : "running",
"enable" : true
},
"sourceline" : 139,
"sourcefile" : "/etc/puppetlabs/puppet/modules/puppetdb/
manifests/server.pp",
...
"title" : "pe-puppetdb",
"type" : "Service",
"certname" : "puppetdb"
}, ... ]
67

68. Querying Nodes
• Query
all
or
specific
nodes
for
facts
and
resources
• /v2/nodes
• /v2/nodes/<node
name>
• /v2/nodes/<node
name>/resources
• /v2/nodes/<node
name>/facts
68

69. Query Format
• HTTP
GET
• Endpoint
is
the
URL
• Query
is
the
data
69

70. Query Format
http://localhost:8080
/v2
/endpoint
query=[“=”, ”certname”, “my_host.com”]
70

71. Query Format
[
"and",
["=", "type", "Service"],
["=", "title", "httpd"]
]
71

72. Query Operators
• Binary
Operators
•
=
(equality)
•
>
(greater
than)
•
<
(less
than)
•
>=
(less
than
or
equal
to)
•
<=
(greater
than
or
equal
to)
•
~
(regexp
match)
• Boolean
Operators
•
and
•
or
•
not
72

73. Query Example
curl -G
-H ‘Accept: application/json’
http://localhost:8080/v2/resources
--data-urlencode
‘query=[“=”,“sourcefile”,“/etc/puppet/manifests/site.pp”]’
73

74. Helpful Curl Switches
-G
-H ‘Accept: application/json’
--data-urlencode <data>
--cacert /etc/puppet/ssl/certs/ca.pem
--cert /etc/puppet/ssl/certs/<node>.pem
--key /etc/puppet/ssl/private_keys/<node>.pem
74

75. Putting your Queries in Files
--data-urlencode query@<filename>
75

76. Helpful Filters
/v2/resources/User
["~","sourcefile","/opt/puppet/share/puppet/modules"]
/v2/resources
[
"and",
["=", "exported", true],
["=", "certname", "exporter.example.com"]
]
76

77. dalen/puppetdbquery
puppet query nodes '(Package["mysql-server"])'
puppet query nodes '(architecture=amd64)'
77

78. Upcoming Features
• Report
and
Event
endpoints
– available
now
under
/experimental/
• MCollecBve
discovery
plugin
78

79. Brief Tour
79

80. Other Resources
• hMp://docs.puppetlabs.com/puppetdb
• hMp://github.com/puppetlabs/puppetdb
• hMp://projects.puppetlabs.com/projects/
puppetdb
• #puppet
on
freenode
80

81. james.sweeny@puppetlabs.com
supercow
on
irc.freenode.net
@jsween_y
on
TwiMer
81