Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures

© 2014 Adobe Systems Incorporated.
All Rights Reserved.
Master Chef class
https://github.com/francoisledroff/connectcon-chef-repo
http://www.slideshare.net/francoisledroff/master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures
Francois Le Droff – Nicolas Peltier
Let’s cook delightful AEM infrastructures
/master-chef-class-learn-how-to-quickly-cook-delightful-cqaem-infrastructures

@francoisledroﬀ

@npeltier

Agenda
•!What
•!How
•!Why

What ?

“ An automation platform
that transforms
infrastructure into code ”

Infrastructure ?
noun ˈin-frə-ˌstrək-chər
[1] A Collection of
–!Resources:
•! Network nodes
•! File systems, files, folders, symbolic links, disk mounts
•! Users, groups
•! Packages, software
•! Configurations
–!Acting in concert
–!To oﬀer a service
–! [1] Introduction to Chef

Infrastructure as Code?
•! Does not only replace your shell scripts
•! Does allow:
–! to build your infra from a set of bare metal servers (called
Nodes) Even on heterogeneous OS and Architecture
–! to abstract the convergence of your infrastructure (your
Nodes) through code
–! to make this (Nodes) convergence idempotent
State A --> State B (ex: install)
Chef Client finished, 147/153 resources updated
State B --> State B (ex: check)
State C --> State B (ex: back to normal)

•! Created in 2009, Edited by Opscode/Chef
•! Apache License
•! On-top-of/in Ruby
•! Very active community
•! http://community.opscode.com
•! #learnChef
•! https://learnchef.opscode.com/
Chef ?
p://community.opscode.com

What does Chef code look like ?
•! Recipes
•! Attributes
•! Resources
•! Cookbooks

Recipes
Fundamental configuration element
•! Authored in Ruby
•! Queries , defines attributes
•! Manipulates Resources
•! Manipulates LW Resources
•! Leverages Libraries & Templates
•! Stored in Cookbooks
–! Default Attributes
–! Recipes
–! LW Resource Providers
–! Libraries
–! Templates
recipes
cookbooks

•! aem-cookbook/recipes/default.rb
case node['platform_family']
when 'rhel'
log 'this platform family is supported' do
level :info
end
else
log 'this platform family is not supported' do
level :warn
end
end
•! aem-cookbook/attributes/node.rb
default['aem']['mode'] = 'publish'
default['aem']['port'] = '4503'
default['aem']['url'] = 'http://localhost:'+ node['aem']
['port']
default['aem']['mode'] = 'publish'
default['aem']['port'] = '4503'
default['aem']['url'] = 'http://localhost:'+ node['aemaem']
['port']
aem-cookbook/recipes/default.rb
case node['platform_family']
when 'rhel'
log 'this platform family is supported' do
level :info
end
else
log 'this platform family is not supported' do
level :warn
end
end
Attributes
attributes

Recipes : Resources & LW Resources
•! aem-cookbook/recipes/default.rb
remote_file node['aem']['quickstart_jar'] do
source node['aem']['repo_url']
owner node['aem']['user']
group node['aem']['group']
checksum '3043859473'
action :create_if_missing
notifies :restart, 'service[aem]', :delayed
end
or
artifact_file node['aem']['quickstart_jar'] do
location 'com.day.cq:cq5:jar:5.5.0.20120220'
nexus_configuration nexus_configuration_object
end
artifact_file node['aem']['quickstart_jar'] do
location 'com.day.cq:cq5:jar:5.5.0.20120220'
nexus_configuration nexus_configuration_object
end
remote_file node['aem']['quickstart_jar'] do
source node['aem']['repo_url']
checksum '3043859473'
action :create_if_missing
end
recipes

Recipes : AEM LW Resources
•! tacit-aem-cookbook/recipes/author.rb
node[:aem][:author][:deploy_pkgs].each do |pkg|
aem_package pkg[:name] do
version pkg[:version]
aem_instance ‘author’
package_url pkg[:url]
update pkg[:update]
user node[:aem][:author][:admin_user]
password node[:aem][:author][:admin_password]
port node[:aem][:author][:port]
group_id pkg[:group_id]
recursive pkg[:recursive]
properties_file pkg[:properties_file]
version_pattern pkg[:version_pattern]
action pkg[:action]
end
end
node[:aem][:author][:deploy_pkgs].each do |pkg|
aem_package pkg[:name] do
version pkg[:version]
aem_instance ‘author’
package_url pkg[:url]
update pkg[:update]
user node[:aem][:author][:admin_user]
password node[:aem][:author][:admin_password]
port node[:aem][:author][:port]
group_id pkg[:group_id]
recursive pkg[:recursive]
properties_file pkg[:properties_file]
version_pattern pkg[:version_pattern]
action pkg[:action]
end
end
recipes

–! aem-cookbook/recipes/start.rb
ruby_block 'block_until_cq_operational' do
block do
Chef::Log.info 'Waiting until CQ is listening on port
'+node['aem']['port']
until CQHelper.service_listening?(node['aem']['port'])
sleep 1
Chef::Log.info('.')
end
Chef::Log.info 'Waiting until the CQ default page is
responding'
test_url = URI.parse(node['aem']['url'])
until CQHelper.endpoint_responding?(test_url)
sleep 1
Chef::Log.info('.')
end
end
action :nothing
end
Recipes: Libraries
recipes
© 2014 Adobe Systems Incorporated.© 2014 Adobe Systems Incorporated.
ruby_block 'block_until_cq_operational' do
block do
Chef::Log.info 'Waiting until CQ is listening on port
'+node['aem']['port']
until CQHelper.service_listening?(node['aem']['port'])
sleep 1
Chef::Log.info('.')
end
Chef::Log.info 'Waiting until the CQ default page is
responding'
test_url = URI.parse(node['aem']['url'])
until CQHelper.endpoint_responding?(test_url)
sleep 1
Chef::Log.info('.')
end
end
action :nothing
end

Cookbook patterns [1]
•! Library cookbooks:
–! https://github.com/RiotGames/artifact-cookbook
–! https://github.com/francoisledroﬀ/chef-vault-util
•! Application cookbooks
–! https://github.com/tacitknowledge/aem-cookbook
–! https://github.com/francoisledroﬀ/aem-cookbook
–! https://github.com/onehealth-cookbooks/apache2
–! https://github.com/socrata-cookbooks/java
–! https://github.com/hw-cookbooks/haproxy
–! https://github.com/stevendanna/logrotate
–! https://github.com/opscode-cookbooks/chef-splunk
•! Organization specific
–! Wrapper cookbooks
–! Base cookbooks
–! Environment cookbooks
[1] http://blog.vialstudios.com/the-environment-cookbook-pattern/
Library cookbooks:
ps://github.com/RiotGames/artifact-cookbook
ps://github.com/francoisledro
cookbooks

I got it

Wait a quick overview

Chef-server nodes

workstation
Git
Chef-server nodes
ssh
env.
roles attributes
recipes
cookbooks

workstation
Git
Chef-server nodes
RSA key Auth
ssh
knife
env.
roles attributes
recipes
Chef-DK
data bags
cookbooks RSA Keys

workstation
Git
Chef-server nodes
RSA key Auth
ssh
knife
env.
roles attributes
recipes
Chef-DK
data bags
cookbooks RSA Keys
Search API
cookbooksorg
data bags
attributes
env.
recipes
node objectrun-list
rolesWeb UI
versions

workstation
Git
Chef-server nodes
RSA key Auth
ssh
RSA key Auth
knife
env.
roles attributes
recipes
Chef-DK
data bags
cookbooks RSA Keys
Search API
cookbooksorg
data bags
attributes
env.
recipes
node objectrun-list
rolesWeb UI
versions
chef-clients
RSA Key

Search APIknife
workstation
env.
roles attributes
cookbooks
recipes
Git
org
Chef-DK
github
Nexus
opscode rubygem
data bags
attributes
env.
data bags
cookbooks recipes
Chef-server nodes
RSA key Auth
ssh
https
yymaven redhat
RSA Keys
node objectrun-list
rolesWeb UI
versions
RSA key Auth
chef-clients
RSA Key

How ?

Use case 0 : one AEM Author
author

Use case 0 : Chef Automation
–! Install the jdk
–! Download the jar
–! Install it as a service
–! A few aem cookbooks on github
•! https://github.com/francoisledroﬀ/aem-cookbook
•! https://github.com/tacitknowledge/aem-cookbook
•! https://github.com/manosriglis/chef-aem
•! https://github.com/QVCItalia/chef-aem

OSS from opscode and elsewhere
Chef, Ruby, rvm, bundler
knife
workstation
Chef-DK
opscode
$ curl –L
https://www.opscode.com/chef/install.sh |
sudo bash
$ curl -sSL https://get.rvm.io | bash -s stable –-
ruby=1.9.3
$ gem install bundler
http://www.getchef.com/downloads/chef-dk

workstation
Git
ssh
A Chef repo in Git
Every Chef automated infra needs a Chef Repository
github
$ git clone https://github.com/opscode/chef-repo.git
Made available for you at
https://github.com/francoisledroﬀ/connectcon-chef-repo

node
Get Few Machines/Nodes
to deploy your code/infra to
workstation
ssh
~/workspace/github/connectcon-chef-repo on ! master!
$ vagrant plugin install vagrant-berkshelf --plugin-version 2.0.1
$ cat Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "CentOS-6.4-x86_64"
config.vm.hostname = "connectcon-francois"
config.vm.box_url = "http://developer.nrel.gov/downloads/vagrant-boxes/
CentOS-6.4-x86_64-v20130427.box"
config.berkshelf.enabled = true
End
$ vagrant up
$ vagrant ssh
Welcome to your Vagrant-built virtual machine.
[vagrant@connectcon-francois ~]$

A Chef Server
comes in 3 flavors
•! On premise OS Chef Server
•! On premise Enterprise Chef
•! Hosted Enterprise Chef server
•! Local Alternatives:
•! Chef-zero
•! Chef-solo
Chef-server

A Chef Org
top-level entity for role-based access
org
Chef-server
https (ldap) Auth
Web UI
ps (ldap) Authps (ldap) Auth
https://chef.corp.adobe.com/organizations

workstation
Git
ssh
A few private keys
to associate your new chef repo with your chef server user and org
$ ll .chef
total 24
connectcon-validator.pem
knife.rb
ledroff.pem
org
Chef-server Web UI
https(ldap)Auth
RSA Keys

Bootstrap your nodes
knife
workstation Chef-server
RSAkeyAuth
node
org
RSA Keys
chef-clients
RSA Key
RSAkeyAuth
$ knife bootstrap <your-node-fqdn> --sudo -x <your-sudoer>

Chef-server nodes
RSA key Auth
Search API attributes node objectrun-list chef-clients
Node Objects

Start Coding:
Manage your Ruby dependencies
•! Use Bundler for RubyGem
$ cat Gemfile
source 'https://rubygems.org’
gem 'chef', '~> 11.10.0’
gem 'berkshelf', '~> 3.1.3‘
$ bundle install

Start Coding:
Manage your Chef dependencies
•! A few aem cookbooks on github
•! https://github.com/francoisledroﬀ/aem-cookbook
•! https://github.com/tacitknowledge/aem-cookbook
•! Use Berkshelf
$ cat Berksfile
source "https://api.berkshelf.com”
cookbook 'ntp', '~> 1.5.4'
cookbook 'chef-client', '~> 3.2.0’
cookbook 'artifact', git: 'https://github.com/francoisledroff/artifact-
cookbook.git', tag: '1.11.4’
cookbook 'aem', git: 'https://github.com/francoisledroff/aem-cookbook

Upload your cookbooks
chef-client and ntp declared in your chef server org
knife
workstation org
Chef-DK
Chef-server
httpsRSAkeyAuth
RSA Keyscookbooks
$ berks install
$ berks upload
cookbooks

Run-list
run_list
"recipe[chef-client::default]",
"recipe[chef-client::delete_validation]",
"recipe[ntp]",
"recipe[openssl::upgrade]"

Role
Base
name ”base”
Description ”connecton base server role”
run_list "recipe[chef-client::default]",
"recipe[ntp]",

Typical Roles
connectcon-chef-repo/roles/
name ”publish”
Description ”connecton aem publish server role”
run_list ”role[base]”,”recipe[aem::publish]”
Mongo Author
Publish
Dispatcher
HA/LB
name "lb_dispatcher"
description "connectcon roles for lb"
run_list ”role[base]”,”recipe[haproxy::app_lb]”
override_attributes('haproxy' =>
{'app_server_role' => ’dispatcher’})
Base
name ”base”
Description ”connecton base server role”
run_list "recipe[chef-client::default]",
"recipe[ntp]",

Apply Run-List to Nodesattributes
https://chef.corp.adobe.com/organizations
$ knife node list
$ knife node edit one.node.fqdn

Chef-client run
[root@ot1slu010 ~]# sudo chef-client
Starting Chef Client, version 11.12.8
resolving cookbooks for run list: ["chef-client::delete_validation", "ntp",
“aem::author", “aem::start”]
* remote_file[/apps/publish/aem-author-4502.jar] action create_if_missing (up to
date)
* file[/apps/publish/license.properties] action create (up to date)
* cookbook_file[/apps/author/serverctl] action create (up to date)
* execute[java] action run (skipped due to not_if)
* service[aem] action enable (up to date)
Recipe: aaem::start
* ruby_block[block_until_cq_operational] action nothing (skipped due to
action :nothing)
* log[ensure_cq_is_running] action write
Recipe: aaem::default
* service[aem] action start
- start service service[aem]
Recipe: aaem::start
* ruby_block[block_until_cq_operational] action create
- execute the ruby block block_until_cq_operational
Running handlers:
Running handlers complete
Chef Client finished, 2/27 resources updated in 9.770664 seconds

So we have an Author : Use Case 0
AuthorAuthor
{
"name": ”<author-connectcon-fqdn>",
"chef_environment": "dev”,
"run_list": [
"role[author]"
]
}
$ knife node edit author-connectcon-fqdn

Let’s add a publish : Use Case 1
{
"name": ”<publish-connectcon-fqdn>",
"run_list": [
"role[publish]"
]
}
$ knife node edit publish-connectcon-fqdn
Publish
Author

And there is magic: Chef search API
•! tacit-aem-cookbook/providers/replicator.rb
https://github.com/tacitknowledge/aem-cookbook
hosts = []
search(:node, %Q(role:"#{role}"
AND aem_cluster_name:"#{cluster_name}")) do |n|
log "Found host: #{n[:fqdn]}"
hosts << {
:ipaddress => n[:ipaddress],
…
:name => n[:fqdn]
}
end
…
hosts.each do |h|
…
end

Replication configuration happens
#{role}:”publish”
#{cluster_name}:”dev”
Author
Publish

Aem-Replicator API
aem_replicator "replicate_to_publish_servers" do
local_user node[:aem][:author][:admin_user]
local_password node[:aem][:author][:admin_password]
local_port node[:aem][:author][:port]
remote_hosts node[:aem][:author][:replication_hosts]
dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically]
cluster_name node[:aem][:cluster_name]
cluster_role node[:aem][:publish][:cluster_role]
type :publish
action :add
end

What about Secret Management ?
aem_replicator "replicate_to_publish_servers" do
local_user node[:aem][:author][:admin_user]
local_password node[:aem][:author][:admin_password]
local_port node[:aem][:author][:port]
remote_hosts node[:aem][:author][:replication_hosts]
dynamic_cluster node[:aem][:author][:find_replication_hosts_dynamically]
cluster_name node[:aem][:cluster_name]
cluster_role node[:aem][:publish][:cluster_role]
type :publish
action :add
end

Git
UX/Dev/QA/Ops
dev
dev-stable prod
ps
Chef-server
https RSA private
key Auth
chef-client
chef-client
chef-client
chef-client
https RSA private key Auth •! Chef encrypted
data bags
•! Encrypted for
•! admin users
•! whitelisted nodes
•! Managed by chef-vault ruby gem
•! Git Back up
•! Encrypted obviously
Encrypted Data Bags with Chef-vault

Chef-Vault in Action:
include_recipe 'chef-vault-util::default'
item = chef_vault_item(node['aem']['vault_accounts'], node['aem']
['vault_accounts_cq_admin_item'])
cq_admin_username = item[node['aem']['vault_accounts_username_property']]
cq_admin_password = item[node['aem']['vault_accounts_password_property']]
log 'the secret accounts credential was fetched from a chef-vault enabled
encrypted data_bag for username '+ cq_admin_username do
level :info
end
•! francois-aem-cookbook/recipes/secure.rb
https://github.com/francoisledroﬀ/aem-cookbook
https://github.com/francoisledroﬀ/chef-vault-util
$ knife vault update accounts prod_cq_admin -J accounts/prod_cq_admin.json
-A "ledroff," --mode client -S "role:aem_server environment:prod"

Let’s add a Dispatcher : Use Case 2
Dispatcher
Author
Publish
DispatcherDispatcher
{
"name": ”<dispatcher-connectcon-fqdn>",
"run_list": [
"role[dispatcher]"
]
}
$ knife node edit dispatcher-connectcon-fqdn
Another Search
Publish
Another Search

Let’s Add a Load Balancer : Use Case 3
HA/LB
Dispatcher
{
"name": ”<lb-connectcon-fqdn>",
"run_list": [
"role[lb_dispatcher]"
]
}
$ knife node edit lb-connectcon-fqdn
Author
Publish
Author
Publish Publish Publish
Dispatcher Dispatcher Dispatcher
Another Search
}
DispatcherDispatcher DispatcherDispatcher
Another Search

•! myapp-cookbook/recipes/log.rb
logrotate_app 'aem' do
cookbook 'logrotate'
path node['myapp']['log_dir']
frequency node['myapp']['logrotate']['frequency']
rotate node['myapp']['logrotate']['rotate']
end
include_recipe 'it-splunkforwarder::default’
•! myapp-cookbook/attributes/log.rb
default.splunkforwarder.inputs = [
{"input_path" =>
"#{node['aem']['log_dir']}/error.log",
… ]
default.splunkforwarder.inputs = [
{"input_path" =>
"#{node['aem']['log_dir']}/error.log",
… ]
Let’s add log monitoring: Use Case 4
myapp /recipes/log.rb
logrotate_app 'aem' do
cookbook 'logrotate'
path node['myapp']['log_dir']
frequency node['myapp']['logrotate']['frequency']
rotate node['myapp']['logrotate']['rotate']
end
include_recipe 'it-splunkforwarder::default’
Dispatcher
Author
Publish
Author
Publish Publish Publish
Dispatcher Dispatcher DispatcherDispatcherDispatcher DispatcherDispatcher DispatcherDispatcherDispatcherDispatcher
Publish
DispatcherDispatcherDispatcherDispatcherDispatcherDispatcher
Publish Publish
DispatcherDispatcher DispatcherDispatcher
Publish
DispatcherDispatcherDispatcherDispatcher
Add the above in
the base role
Add the above in
the base role

Let’s cluster things!

AEM Production Infrastructure
LB/HA
Dispatcher
Publish
Author
MongoDB servers
Dispatcher
!" 3rd parties (your legacy,
your cloud)

you got it ?

#LearnChef
Learn Stuﬀ
Automate IT infrastructure and application delivery
Learn Chef
Slide inspired by
Spice up your recipe
By Seth Vargo

What Stuff ?
•! Git Stuff
•! Ruby stuff
•! VM / Container stuff
•! Cloud stuff
•! Network stuff
•! *nix tools and stuff
•! OS Stuff
–! package management
–! services
•! Chef Community Stuff
•! Build and packaging stuff
•! Continuous Delivery stuff
•! Monitoring stuff
•! Analytics stuff
•! Messaging stuff
•! Security Stuff

Why ?

Devops?
“ You built it
You run it! ”

Chef, Devops ? No silver bullet.
https://twitter.com/DEVOPS_BORAT/status/52857016670105600

Chef, Devops ? No silver bullet.
https://twitter.com/mindweather/status/458653460234502144

Why ?
“Accelerate, Simplify, Scale”
•! Breaking down the wall of confusion
•! As infra is code, it becomes:
–! Testable
–! Versionable
–! Disposable
–! Reproducible
•! “If it’s not in the source control system,
it does not exist” @bdelacretaz

https://www.flickr.com/photos/francoisledroﬀ/6107220850/in/set-72157626126325552

https://www.flickr.com/photos/blmoregon/7883684692

One more publish?
LB/HA
Dispatcher
Publish
Author
!" MongoDB servers
Dispatcher
!" 3rd parties (cloud stuﬀ)

This needs to be configured
LB/HA
Dispatcher
Publish
Author
MongoDB servers
Dispatcher
!" 3rd parties (cloud stuﬀ)

A more complex production

Next scale of complexity…

Let’s Share the love

Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures

Ähnlich wie Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures (20)

Mehr von François Le Droff

Mehr von François Le Droff (13)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Master Chef class: learn how to quickly cook delightful CQ/AEM infrastructures