The document summarizes an agenda for a Belgian Puppet Users Group meeting about MCollective. The agenda includes an overview of orchestration and MCollective, hands-on setup of MCollective, the MCollective command line tool, MCollective agents, and future plans. It also provides details about installing and configuring MCollective clients, servers, and the ActiveMQ middleware.

1. Belgian Puppet Users Group
Something About MCollective
24th of June 2014
Hosted by Telenet/Hostbasket
Lochristi - Belgium

2. Agenda

3. Agenda
Orchestration & MCollective
Hands-on - Setting up MCollective
The 'mco' command
MCollective Agents
And what about Tomorrow ?

4. Orchestration

5. Orchestration on Wikipedia
Orchestration describes the automated
arrangement, coordination, and management of
complex computer systems, middleware, and
services.

6. Orchestration in human Language ?
Parallel job execution System

7. MCollective
Marionette Collective:
Framework
Uses Publish Subscribe Middleware
Very scalable (form small to huge clusters)
Broadcast paradigm (Network is the only source of truth)
no central database - no complex naming convention
Simple command line tools
Extremely pluggable
Community extentions available

8. MCollective
build upon existing middleware
uses existing authentication/authorisation models
uses existing clustering techniques
uses existing routing/network isolation methods

9. Marionette Collective
Pluggable core
middleware (STOMP compliant)
Authorisation
serialisation
Data Sources (Chef & Puppet supported + Facter[Community])
Mcollective as transport (eg, central service inventory system)

10. MCollective - Components

11. Overview - Components

12. The MCollective Server - mcollectived

13. The MCollective Client

14. MCollective Middleware Overview

15. Inside The MCollective Middleware

16. Middleware Choices
ActiveMQ - preferred
Best tested
Performance is great
Powerfull and flexible security features
Scaleable by clustering
Pain in the #$@% to configure
detailed docs on docs.puppetlabs.com
Connector is shipped with MCollective
RabbitMQ
Not that good tested as ActiveMQ
Not documented @ docs.puppetlabs
Connector is shipped with MCollective
Generic Stomp Connector (Deprecated)
Custom Connector Plugins

17. Getting dirty hands

18. Vagrant boxes

19. What we need
Centos vagrant boxes images
puppetlabs vagrant boxes
Centos 6.5 64bit nocm
Centos 6.5 32bit nocm
Minimal centos6.5 vagrant box
centos minimal 64 bit version
centos minimal 32 bit
My vagrantfile with bridged networking (with puppetlabs centos 6.5i nocm box)
Vagrantfile (showoff download link)

20. Vagrant setup
Based on the Vagrantfile from previous slide.
Only one ActiveMQ server (running on my laptop)
Only the ':johan' image is needed.
mkdir -p bpug_vagrant/puppet ; cd bpug_vagrant (puppet = shared folder)
download Vagrantfile
Used domainname = koewacht.net
change johan to 'yourname' (should be unique)
adjust box_url ( eg file://'downloaded box file' )
adjust memory settings (currently 1GB)
starting the vagrant box :
vagrant up 'yourname'
Having trouble -- shout !!
logging into your box
vagrant ssh 'yourname'
sudo -i

21. Info we need

22. The setup
One central ActiveMQ server (already up and running)
Many MCollective nodes
Your Virtual Boxes ...
Server role
Client role
Bridged mode, so we can see each others node
Installation done by hand

23. Info we need before hand
The ipaddress ActiveMQ server (dhcp based)
The passwords for configuration files :
client: 29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU=
server: 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw=
Pre-shared key: Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o=
( keys are generated with : $ openssl rand -base64 32 )

24. Requirements
We are using packages from Puppetlabs repos
Mcollective clients/servers
Working NTP
Ruby
1.8.7/1.9.3
2.0.0 not supported yet
1.9.0/1.9.2 will fail
Ruby stomp +1.2.2
Mcollective + 2.5.0
5MB disk
256 MB ram

25. Requirements Continued
Middleware Broker
500 MB ram
Messaging middleware :
ActiveMQ 5.8 with stomp connector
RabbitMQ 2.8 with stomp connector
Disk Space for Middleware server : 15MB
Some CPU & Network capacity (+2 connections per server)
platforms puppetlabs repo
RHEL 5|6|7
Fedora 19 - 20
Debian Lucid|Precise|Saucy|Sid|Squeeze|Trusty|Wheezy

26. Installing the packages

27. Installing Puppetlabs repos
osfamily == RedHat
$ sudo yum install http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
$ sudo yum install http://yum.puppetlabs.com/puppetlabs-release-fedora-20.noarch.rpm
osfamily == Debian
$ wget http://apt.puppetlabs.com/puppetlabs-release-sid.deb
$ sudo dpkg -i puppetlabs-release-sid.deb
$ sudo apt-get update
(replace sid with your version)

28. Installing ActiveMQ - How I Did It

29. Installing the package
On osfamily == RedHat
$ sudo yum install activemq
$ sudo chkconfig activemq on
On osfamily == Debian
$ sudo apt-get install activemq
$ sudo sysv-rc-conf activemq on

30. ActiveMQ Configuration
The /etc/activemq/activemq.xml
Line number correspond to download-able activemq.xml file
Enable Purging the Broker
35 <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" useJmx="true"
schedulePeriodForDestinationPurge="60000">
Disable producerFlowControl & memory cleanup
50 <destinationPolicy>
51 <policyMap>
52 <policyEntries>
53 <!-- MCollective generally expects producer flow control to be turned off. -->
54 <policyEntry topic=">" producerFlowControl="false" memoryLimit="1mb"/>
55 <!-- MCollective will generate many single-use reply queues,
56 which should be garbage-collected after five minutes to conserve memory. -->
57 <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/>

31. ActiveMQ Configuration - continued
The /etc/activemq/activemq.xml
define logins for clients and servers in simpleAuthenticationPlugins
104 <simpleAuthenticationPlugin>
105 <users>
106 <authenticationUser username="client" password="29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU="
groups="servers,clients,everyone"/>
107 <authenticationUser username="server" password="04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw="
groups="servers,everyone"/>
108 </users>
109 </simpleAuthenticationPlugin>

32. ActiveMQ Configuration - continued
The /etc/activemq/activemq.xml
Define permissions for clients and servers in authorizationPlugins
110 <authorizationPlugin>
111 <map>
112 <authorizationMap>
113 <authorizationEntries>
114 <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
115 <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
116 <authorizationEntry queue="mcollective.>" write="clients" read="clients" admin="clients" />
117 <authorizationEntry topic="mcollective.>" write="clients" read="clients" admin="clients" />
118 <authorizationEntry queue="mcollective.nodes" read="servers" admin="servers" />
119 <authorizationEntry queue="mcollective.reply.>" write="servers" admin="servers" />
120 <authorizationEntry topic="mcollective.*.agent" read="servers" admin="servers" />
121 <authorizationEntry topic="mcollective.registration.agent" write="servers" read="servers"
admin="servers" />
122 <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
123 </authorizationEntries>
124 </authorizationMap>
125 </map>
126 </authorizationPlugin>

33. ActiveMQ Configuration - continued
The /etc/activemq/activemq.xml
Transports - Only one transport should be enabled
156 <transportConnectors>
157 <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
158 </transportConnectors>
Disable web console (commented out)
170 <!-- disabled for security reasons
171 <import resource="jetty.xml"/>
172 -->

34. Fire it up - and check
$ service activemq start
$ netstat -an | grep 61613
$ tail -200f /var/log/activemq/activemq.log | less
In the real world
Adjust firewall (port 61613)
Selinux and equivalents

35. Installing MCollective Servers

36. Installing the package
osfamily == RedHat
$ sudo yum install mcollective
$ sudo chkconfig mcollective on
osfamily == Debian
$ sudo apt-get install ruby-stomp mcollective
$ sudo sysv-rc-conf mcollective on

37. MCollective Server Configuration
/etc/mcollective/server.cfg
(based on the downloadable server.cfg)
user and password are also defined in activemq.xml on messaging server
6 plugin.activemq.pool.size = 1
7 plugin.activemq.pool.1.host = activemq.koewacht.net
8 plugin.activemq.pool.1.port = 61613
9 plugin.activemq.pool.1.user = server
10 plugin.activemq.pool.1.password = 04BpZofasX1dDexFsqZcgfM1tkC4VCGI6hoziWMu7zw=
pre-shared-key form earlier slides
17 # Security provider
18 securityprovider = psk
19 plugin.psk = Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o=
Check the libdir directory
22 libdir = /usr/libexec/mcollective

38. Fire it up - and verify
$ service mcollective start
$ netstat -an | grep 61613
tcp 0 0 192.168.10.223:50737 192.168.10.231:61613 ESTABLISHED

39. The MCollective Client

40. Installing the MCollective Client Package
osfamily == RedHat
$ sudo yum install mcollective-client
oSfamily == Debian
$ sudo apt-get install mcollective-client

41. Configuring the MCollective Client
(based on the downloadable client.cfg)
user and password are also defined in activemq.xml on messaging server
3 connector = activemq
4 plugin.activemq.pool.size = 1
5 plugin.activemq.pool.1.host = activemq.koewacht.net
6 plugin.activemq.pool.1.port = 61613
7 plugin.activemq.pool.1.user = client
8 plugin.activemq.pool.1.password = 29l6wD2mIzbLpbp4GMnUzchHp2XWpKk8N8dcxXCnDRU=
9 plugin.activemq.heartbeat_interval = 30
pre-shared-key form earlier slides
17 # Security provider
18 securityprovider = psk
19 plugin.psk = Gw8nclOGn1YiIMvEAxgeZ7jrL1ErCdZZXm2e7JX2S4o=
Check the libdir directory
22 libdir = /usr/libexec/mcollective

42. Testing the Setup so Far

43. Testing the Basic Setup
The MCollective Ping Test
low level query
[vagrant@johan ~]$ mco ping
activeMQ.koewacht.net time=176.15 ms
johan.koewacht.net time=185.95 ms
Troubleshooting
Are the passwords & user/groups correct
middleware server : activemq.xml
mcollective server.cfg
mcollective client.cfg
Networking
check for port 61613

44. MCollective Command Line Client

45. Introduction mco command-line client
Connector
Clients uses 2 plugins
connector plugin (connection to middleware)
ActiveMQ
security plugin (sign & optionally encript data)
PSK (pre-shared key)
same connectors on all MCollective components
(clients/servers/middleware)

46. Introduction mco command-line client
Inventory
builtin plugin
gathers info about MCollective server
server configuration
server stats
available plugins
Configuration Classes
Facts (aka facter)

47. Introduction mco command-line client
Inventory - example run
$ mco inventory heliotrope
Inventory for heliotrope:
Inventory for heliotrope:
Server Statistics:
Version: 2.5.0
Start Time: Mon Apr 14 03:11:12 -0700 2014
Config File: /etc/mcollective/server.cfg
Collectives: mcollective
Main Collective: mcollective
Process ID: 1334
Total Messages: 16
Messages Passed Filters: 13
Messages Filtered: 3
Expired Messages: 0
Replies Sent: 12
Total Processor Time: 38.56 seconds
System Time: 128.22 seconds
Agents:
discovery rpcutil
Data Plugins:
agent fstat
Configuration Management Classes:
No classes applied
Facts:
No facts known

48. Inventory continued
custom output format
ruby script
use it as script argument
inventory do
format "%20s %8s %10s %-20s"
fields {[ identity, facts["architecture"],facts["operatingsystem"], facts["operatingsystemrelease"]]}
end
$ mco inventory --script inventory.mc
geode x86_64 CentOS 6.4
sunstone amd64 Ubuntu 13.10
heliotrope x86_64 CentOS 6.5

49. Discovery
mc plugin
built in
defined in client.cfg (mc plugin)
13 # Use auto-discovery
14 default_discovery_method = mc
sends broadcast queries
mco plugin doc mc
flatfile plugin
list of hostnames from file
mco plugin doc flatfile

50. Discovery
flatfile plugin
$ cat /path/to/hostlist
fireagate
heliotrope
$ mco rpc rpcutil ping --disc-method flatfile --disc-option /path/to/hostlist
Discovering hosts using the flatfile method .... 2
* [ ============================================================>] 2 / 2
heliotrope
Timestamp: 1385012042
fireagate
Timestamp: 1385012044
Finished processing 2 / 2 hosts in 146.13 ms
mco rpc rpcutil is how to invoke a direct call to the API without using the client application.

51. MCollective's filters
Can be used on all MCollective commands
$ mco help <command>
Host Filters
-W, --with FILTER Combined classes and facts filter
-S, --select FILTER Compound filter combining facts and classes
-F, --wf, --with-fact fact=val Match hosts with a certain fact
-C, --wc, --with-class CLASS Match hosts with a certain config management class
-A, --wa, --with-agent AGENT Match hosts with a certain agent
-I, --wi, --with-identity IDENT Match hosts with a certain configured identity

52. MCollective filters - examples
$ mco find -with-identity /i/
$ mco find -with-identity /^webd/
$ mco find -with-class webserver
$ mco find -with-fact operatingsystem=CentOS
$ mco find -with-agent package
Filters requires the mc Discovery Plugin.
Flatfile discovery only supports identity filter

53. MCollective combined filters
Types of combined filters
Puppet Classes & Facter facts
$ mco ping --with "/^webd/ operatingsystem=CentOS"
Select filter
combination of
Factes and Classes
Boolean Logic ( AND - OR - NOT|! )
$ mco ping --select "operatingsystem=CentOS and /nameserver/"
$ mco ping --select "operatingsystem=CentOS and !environment=dev"
$ mco ping --select "( /httpd/ or /nginx/ ) and is_virtual=true"
CentOS hosts named web followed by a number.
Ping only CentOS hosts which have the nameserver class applied to them.
Ping every CentOS host which isn’t in the dev environment.
match virtualized hosts with either the httpd or nginx Puppet class applied to them.

54. Add limitations to MCollective command
Limit option
Control how many servers get the request
--one
--limit
--limit matching server
$ mco ping --limit 15
$ mco ping --one --with-fact operatingsystem=CentOS
$ mco ping --limit 5 --with-class webserver
$ mco ping --limit 33% --with-class webserver
Fifteen servers of any type
Only one CentOS server
Five servers which have the webserver Puppet class applied to them
One third of the servers which have the webserver Puppet class applied to them

55. Add limitations to MCollective command
batch option
Controls how many servers receive the request in batch
Controls time between batches
$ mco ping --batch 5 --batch-sleep 30 --with-fact country=de
$ mco package upgrade sudo --batch 10 --batch-sleep 20
Ping batches of five German servers every 30 seconds
Fast upgrade sudo in batches of ten servers spaced twenty seconds apart

56. Controlling mco command output
--json
output in json format
--no-progress
supress status bar
--verbose
timing discovery
full RPC statistics

57. Facts
key/value pairs inventory server
Facter generate facts
Installing facter
osfamily == RedHat
$ sudo yum install facter
osfamily == Debian
$ sudo apt-get install facter

58. Facter facts & MCollective
configure /etc/mcollecive/server.cfg
30 # facter
31 factsource=yaml
32 plugin.yaml=/etc/mcollective/facts.yaml
Generate a facts.yaml file
$ facter -y > /etc/mcollective/facts.yaml
optionally add a crontab
$ cat /etc/cron.d/facts.sh
*/30 * * * * facter -y >/etc/mcollective/facts.yaml
restart mcollective
$ mco inventory nodename

59. MCollective & Puppet Classes
Only works with puppet
Puppet agents :
writes classes.txt
$statedir (/var/lib/puppet/state)
agent node runs MCollective server
puppet agent --configprint classfile
must match classesfile /etc/mcollective/server.cfg
We can simulate puppet classes by faking a classes.txt in /etc/mcollective/classes.txt

60. MCollective Agents
Extending MCollective

61. MCollective Agents
Connector
Agents uses 2 plugins
connector plugin (connection to middleware)
ActiveMQ
security plugin (sign & optionally encript data)
PSK (pre-shared key)
same connectors on all MCollective components
(clients/servers/middleware)
Agent Parts
Agent part (servers)
DDL (servers & clients)
Client part (clients)
Common part (servers & clients)

62. MCollective Agent - Installing from Packages
From the PuppetLabs Repositories
Install on every MCollective Server
Many community MCollective Agents (eg. github)
osfamily == RedHat
$ sudo yum install mcollective-filemgr-agent
$ sudo yum install mcollective-nettest-agent
$ sudo yum install mcollective-package-agent
$ sudo yum install mcollective-service-agent
osfamily == Debian
$ sudo apt-get install mcollective-filemgr-agent
$ sudo apt-get install mcollective-nettest-agent
$ sudo apt-get install mcollective-package-agent
$ sudo apt-get install mcollective-service-agent

63. MCollective Agent - Inside the Package
[vagrant@johan ~]$ rpm -ql mcollective-package-common-4.3.0-1.el6.noarch
/usr/libexec/mcollective/mcollective/agent/package.ddl
/usr/libexec/mcollective/mcollective/util/package
/usr/libexec/mcollective/mcollective/util/package/base.rb
/usr/libexec/mcollective/mcollective/util/package/packagehelpers.rb
/usr/libexec/mcollective/mcollective/util/package/puppetpackage.rb
[vagrant@johan ~]$ rpm -ql mcollective-package-agent-4.3.0-1.el6.noarch
/usr/libexec/mcollective/mcollective/agent/package.rb
[vagrant@johan ~]$ rpm -ql mcollective-package-client-4.3.0-1.el6.noarch
/usr/libexec/mcollective/mcollective/application/package.rb

64. MCollective Agent - The Components
The DLL file
DDL = Data Description Language
Definition remote methods
Description input format
Description generated output
metadata
author
version
license
...
Used for Validating Input
If you stick to code convention

65. MColelctive Agent - The Components
MCollective Agent DLL Example
[vagrant@johan ~]$ cat /usr/libexec/mcollective/mcollective/agent/package.ddl
metadata :name => "package",
:description => "Install and uninstall software packages",
:author => "R.I.Pienaar",
:license => "ASL 2.0",
:version => "4.3.0",
:url => "https://github.com/puppetlabs/mcollective-package-agent",
:timeout => 180
requires :mcollective => "2.2.1"
["install", "update", "uninstall", "purge"].each do |act|
action act, :description => "#{act.capitalize} a package" do
input :package,
:prompt => "Package Name",
:description => "Package to #{act}",
:type => :string,
:validation => :shellsafe,
:optional => false,
:maxlength => 90
output :output,
:description => "Output from the package manager",
:display_as => "Output"
output :epoch,
:description => "Package epoch number",
:display_as => "Epoch"
.................
...........

66. MCollective Agent - The Components
The Agent Plugin
Installed on all MCollective servers
Uses DLL for Meta Data & initialization
Defines Agent Actions
Action : individual tasks the agent can do

67. MCollective Agent - The Components
The Client
Installed only on MCollective clients
Provides access to agents and actions
Uses also DDL
eg. input validation
....
Clients - Agents - DLL are strongly coupled

68. MCollective Client Help
[vagrant@johan ~]$ mco help plugin package
MCollective Plugin Application
Usage: mco plugin package [options] <directory>
mco plugin info <directory>
mco plugin doc <plugin>
mco plugin doc <type/plugin>
mco plugin generate agent <pluginname> [actions=val,val]
mco plugin generate data <pluginname> [outputs=val,val]
info : Display plugin information including package details.
package : Create all available plugin packages.
doc : Display documentation for a specific plugin.
Application Options
-n, --name NAME Plugin name
--postinstall POSTINSTALL Post install script
--preinstall PREINSTALL Pre install script
--revision REVISION Revision number
....
-h, --help Display this screen
The Marionette Collective 2.5.2
[vagrant@johan ~]$

69. And What About Tomorrow

70. This is not the end,
Just the beginning
Delve much deeper into MCollective
Read, Read and Read even more
Experiment as much as you can
Secure your MCollective Infrastructure
Authentication connector
Tuning your ActiveMQ
Puppetlabs Docs on ActiveMQ & MCollective
Manage yout MCollective infrastructure with puppet
Puppetlabs MCollective Module on the Forge
Learning MCollective pupept module
Great for getting more insight in managing MCollective with
puppet

71. references
wikipedia - Orchastration(computing
PuppetLabs MCollective online docs
Introduction to orchestration using MCollective - Pieter Loubser
Inroduction to Mcollective - R.I. Pienaar
MCollective Installed. And now ? - Thomas Gelf
Learning MCollective - Jo Rhet (O'Reilly)
This Presentation on Github

72. ??? Questions ???

73. Thank You For Attending
- - -
Thanks go to our Host for Tonight
Telenet/Hostbasket
Do not forget the coolest T-Shirt