Killer R10K Workflow - Phil Zimmerman, Time Warner Cable

1. Killer R10K
Workflow
Automating the Killer
Robots, all 10K of Them

2. Who Am I
@phil_zimmerman

3. Early Days
R10K Workflow Awesomeness
Demo Time

5. The Journey of A
Thousand Miles Begins
With A Single, Monolithic
Repo

6. A Single Repo?
What the F*@K?!!

7. Why Put Everything in a
Single Repo?
Simplify Development
Easy Jenkins Flow
Puppet Code and Hiera Data Together

8. Just
Starting
Out

10. Single CI Job
• rspec-puppet
• syntax check
• lint
Single Release Job
• create/push tag
Single Deploy Job
• Capistrano tasks
• poor man’s dynamic environments
• kludgy git logic w/conditionals in Capfile
For All Modules

11. Forge Modules
Capistrano -> Puppet Module Tool
Worked Well… Until It Didn’t

12. Did I mention the
Capistrano tasks?
KLUDGY

13. task "create_puppet_env", :roles => :puppet_master do
if exists?(:branchname)
run "if [ -d #{environment_basedir}/#{branchname} ]; then cd
#{environment_basedir}/#{branchname} && git pull origin
#{branchname} ; else
git clone #{module_repository} #{environment_basedir}/
#{branchname} --branch #{branchname} ; fi"
else
puts "Please provide a valid git branch name as an argument"
end
end

14. task "create_puppet_env", :roles => :puppet_master do
if exists?(:branchname)
run "if [ -d #{environment_basedir}/#{branchname} ]; then cd
#{environment_basedir}/#{branchname} && git pull origin
#{branchname} ; else
git clone #{module_repository} #{environment_basedir}/
#{branchname} --branch #{branchname} ; fi"
else
puts "Please provide a valid git branch name as an argument"
end
end

15. Upgrade Forge Modules
Version Management == Face Palm
Upgrading Affects all Environments!

16. Let Me Count the Other
Ways…
This Started to Fail
Simple Comes at a Price

17. You mean I have to wait for tests
to run for ALL modules
before I know if my changes are
good?”
“I only changed one module….

19. I have to deploy
everything in order to
get my changes on the masters?
Sigh”
“I only changed one module….

20. everything just to get
my hiera data on the masters?”
“I need to change some hiera
data…. I have to deploy

21. I’m
Losing
My
Patience!

22. Oh - and that little problem where
upgrading forge modules can break
production…
Stop the Madness !!

24. Recap of Early Days:
Monolithic Repo
Long CI Cycles
All-Or-Nothing Deploys
Upgrading Forge Modules

26. Tools Can Be
Awesome
(we just needed a couple more)

27. Make Them Work For You

28. R10K

29. R10K

30. R10K!
https://github.com/
adrienthebo/r10k

31. Deploys Puppet Code
Handles Git/SvnFu
Is Awesome

32. R10K and Puppetfile
Match Made in Heaven (or Portland)
Manage Module Versions
Inventory of Puppet Environment

33. Puppetfile
Format

34. Inventory of
Modules
and Their
Versions
Puppetfile
lives in its
own
repository

35. Puppetfile
lives in its own
Repository

36. Inventory of
Modules and
Their Versions

37. Each module in its own
repository AND r10k

38. R10K
Deploy

39. r10k deploy environment test -p
deploys all modules in Puppetfile
for the test branch
r10k deploy module tomcat
deploys a single module!

40. CI Job Per Module
• rspec-puppet
• syntax check
• lint
Release Job Per Module
• create/push tag
• select module from dropdown list
Deploy Job For Each Module And Hiera
• simpler Capistrano tasks
• wrap r10k calls to each master/node

41. desc "for specified branch in puppetfile repo, use r10k to
deploy all modules for the specified environment."
task "update_environment", :roles => :puppet_master do
if exists?(:branchname)
run "r10k -v debug deploy environment #{branchname}
-p"
else
puts "Please provide a valid git branch name as an
argument"
end
end

42. Puppetfile
Manipulation and
Branch Creation

43. mod 'tomcat',
:git => 'git@github.webapps.rr.com:fylgia/tomcat.git',
:ref => 'RELEASE_1.0.13'
Tomcat Example

44. go to tomcat dir (production branch)
!
‘git checkout -b dev_change_foo’
‘git push origin dev_change_foo’

45. Create the
‘dev_change_foo’ branch
in Puppetfile repo

46. Change the :ref for
tomcat to
‘dev_change_foo’

47. Call r10k to deploy
‘dev_change_foo’
Notify Me When It’s
Finished

49. Explain the Magics
Hint:

50. Create Branch
r10k deploy environment

51. Modify Branch
r10k deploy module

52. Delete Branch
auto-delete Puppetfile branch

53. Testing Multiple
Modules in the
Same Environment

54. go to profile dir (production branch)
!
‘git checkout -b dev_change_foo’
‘git push origin dev_change_foo’

55. mod 'profile',
:git => 'git@github.webapps.rr.com:fylgia/profile.git',
:ref => 'RELEASE_0.1.124'
mod 'tomcat',
:git => 'git@github.webapps.rr.com:fylgia/tomcat.git',
:ref => 'RELEASE_1.0.13'
production
mod 'profile',
:git => 'git@github.webapps.rr.com:fylgia/profile.git',
:ref => ‘dev_change_foo’
mod 'tomcat',
:git => 'git@github.webapps.rr.com:fylgia/tomcat.git',
:ref => 'dev_change_foo'
dev_change_foo

56. Truly Dynamic
Environments!

57. The Post-Receive
Hook
reaktor

58. Default Setup
GitHub or GitHub Enterprise
Hipchat
Just provide some config!

59. REAKTOR_PUPPET_MASTERS_FILE
REAKTOR_HIPCHAT_TOKEN
REAKTOR_HIPCHAT_ROOM
REAKTOR_HIPCHAT_FROM
PUPPETFILE_GIT_URL

60. Other Chat Providers
Campfire, Slack, etc
If it has an API, it’s
pluggable!

61. Other Git Providers
Gitlab, Bitbucket, etc
Need to Determine Best
Approach

62. Create Release
Deploy - Ship It!

63. Create Release
Modulefile
RELEASE_1.0.13 -> RELEASE_1.0.14
versionfile (hieradata)

64. Deploy
Parameterized Job
environment
version to deploy

65. Puppetfile Manipulation
(again)
change :ref to selected version
r10k deploy selected environment
commit/push updated Puppetfile

66. One-Click
Production
Deploy

68. Workflow Recap:
R10K and Puppetfile
Each Module in Own Repo
Post-Receive Hook Goodness
Production Deployments

72. Conclusion:
Early Days
R10K Workflow Awesomeness
Demos

73. Helpful Links and
References
https://github.com/adrienthebo/r10k
http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/
http://garylarizza.com/blog/2014/03/07/puppet-workflow-part-3b/
http://philzim.com/2014/05/02/an-automated-r10k-workflow-that-works/
https://github.com/pzim/reaktor