Joseph Damato, Computology, LLC

1. Package Managers
and Puppet
Joe Damato
packagecloud.io

2. slides available at:
blog.packagecloud.io

3. hi, I’m joe!
• i think these things are cool:
• computer programs
• reproducible builds / infrastructure
• automation
• configuration management
• tahdig*
* an rice food

4. packagecloud.io
• I work on packagecloud.io
• packagecloud makes it easy to upload,
download, store, and delete software packages
• you should use it, it’s cool.

5. marc falardeau, https://flic.kr/p/8gKeGS

6. Wade M, https://flic.kr/p/5aghr9

7. Why?
• Central to maintaining, building, and testing
infrastructure.
• Packages are a primitive in Puppet.
• Understanding where packages come from, and how
to store them properly is a requirement for
infrastructure of any size.
• Packages and packaging are much trickier than they
seem!

8. Overview
• what is a package?
• what is a package manager?
• ./configure && make && make install pattern
• open source tools for package repositories
• HOWTO manage repos in your infra with puppet

9. What is a package?
Beck Gusler, https://flic.kr/p/4A15jm

10. What is a package?
• A package generally consists of:
• metadata (version, architecture, deps, etc)
• files to be written to the filesystem (/usr/sbin/
nginx, etc)

11. Common package types

12. Common package types
• RPM packages
• Used on CentOS, RHEL, Scientific Linux, Fedora, …
• files typically have the “.rpm” file extension
• can be inspected, installed, and removed with rpm
• are actually a:
• header structure (binary data)
• CPIO archive

13. man 8 rpm

14. Common package types

15. Common package types
• Deb packages:
• Used on Ubuntu, Debian, Knoppix, …
• files typically have the “.deb” file extension
• can be inspected, installed, and removed with
dpkg

16. Common package types
• Deb packages:
• are actually an AR archive with:
• version file: the debian format version
• data.tar.gz: the actual files to write to the filesystem
• control.tar.gz: the package metadata
• Can be GPG signed, but signatures are never checked!

17. man 1 dpkg

18. Common package types
• There are lots more! (ruby gems, npm, java,
python, …)
• Some packaging systems also have source
packages.

19. What is a source package?
• A source package consists of:
• metadata (version, architecture(s), build deps,
etc).
• source files (C source, C++ source, py scripts,
etc).
• Allows you to rebuild a binary package easily.

20. Install packages with puppet
Use the resource type ‘package’ to install packages:
package { 'pygpgme':
ensure => latest,
}

21. Install packages with puppet
package { 'pygpgme':
ensure => ‘0.3-11’,
}
Specify the version you want by setting ensure:

22. Summary
• Packages are a collection of files with metadata.
• The metadata usually has info like:
• architecture
• version
• dependency info
• and more.
• Installation is easy if you don’t have dependencies.

23. Dependencies
Nick Sieger, https://flic.kr/p/qQu1e

24. Dependencies
• Installing 1 package is as easy as:
• dpkg -i filename.deb
• rpm -ivh filename.rpm
• Of course, you should use puppet instead :D
• But what if your program needs other programs?
• For example: nginx depends on libssl, zlib, …

25. r-hol, https://flic.kr/p/6UZb98

26. So, what’s a package
manager?

27. Package manager
• A package manager is a collection of software
that allows you to:
• install, upgrade, remove packages
• query package info from local system or repos
• Some tools include more advanced features like
mirroring or more advanced caching features.

28. Common package managers
http://en.wikipedia.org/wiki/
Yellowdog_Updater,_Modified#mediaviewer/File:Yum.png

29. • yum (Yellowdog Updater, Modified)
• Common on RHEL, CentOS, Fedora, …
• Used for installing, removing, configuring, and
querying RPM packages and dependencies.
Common package managers

30. Common package managers
APT

31. Common package managers
• APT (Advanced Package Tool)
• Common on Debian, Ubuntu, KNOPPIX, …
• Used for installing, removing, configuring, and
querying Debian packages and dependencies.

32. Install packages with puppet
• When you install packages with puppet, puppet
will automatically detect which package
manager to use.
• You won’t need to worry about which command
to run, or what options to pass; puppet will take
care of that for you!

33. Summary
• package managers help you install software and
associated dependencies
• easily remove, upgrade, and query packages
• Puppet will automatically detect the system’s
package manager when you install a package.

34. Kellie Parker, https://flic.kr/p/mtNMb

35. A problem
• You run Ubuntu 10.04 LTS
• You want to install redis
• Ubuntu 10.04 comes with redis-server 1.2.0-1
• That’s too old! You need 2.8.19!
• So, now what?

36. Common (not great) solution
• A common solution to this sort of problem is
building redis (or ruby, or …) from source in your
puppet manifest
• Like this….

37. exec { 'install-redis':
command => "make && make install PREFIX=${redis_bin_dir}",
cwd => $redis_src_dir,
path => '/bin:/usr/bin',
unless => "test $(${redis_bin_dir}/bin/redis-server --version | cut -d ' ' -f
1) = ‘Redis'",
require => [ Exec['unpack-redis'], Class['gcc'] ],
}
Common (not great) solution

38. Why?
• It’s easy!
• ./configure && make && make install
• It works!
• I’m using puppet so it’s reproducible!

39. But…
• What happens if you need to:
• completely remove Redis?
• install a security update?
• install a new version?
• install the same exact Redis on 200 machines?

40. The not-so great side
• Not all Makefiles have uninstall targets, so you
have to remove files manually
• Leaving artifacts on the filesystem can cause
really, really hard to debug problems later
• If the build process changes version to version,
it can be painful to rollback

41. The not-so great side
• Rebuilding the same source does not necessarily
get you the same byte-for-byte binary
• If the binaries aren’t identical, you can end up
with bugs in some of the compiled binaries but
not others
• Painful to recreate source builds inside of puppet
• Makes writing tests for manifests painful

42. Make a package
• Install the same binary on every machine
• When the package is removed, all installed files are
removed
• Versioning of build process built in (with most tools)
• Keep your puppet manifests about config
management
• Your build steps are “factored out” into the package

43. Your new puppet manifest
package { 'redis':
ensure => latest,
}

44. Your package
• Your build steps get encapsulated in the package
itself
• Makes iterating on the build more straight forward
• Don’t need to apply (potentially) a bunch of
manifests to a machine every time you do a build

45. Duncan Hull, https://flic.kr/p/iVLZt

46. “How do I make a package?”

47. OZinOH, https://flic.kr/p/bRHn2v

48. Use tools!
• debbuild
• rpmbuild
• fpm
• mock and pbuilder (more advanced)

49. Tradeoffs
• Takes time to learn new tools
• Takes time to understand packaging
• No one ever has enough time

50. BUT…

51. Tradeoffs
• Once you learn how to make packages you can
build reproducible infrastructure much more
easily
• You can use your prod environment in dev and
test
• You can more easily build tests for your
infrastructure with beaker/kitchen.ci

52. Duncan Hull, https://flic.kr/p/iVLZt

53. “How do I store and
organize my packages?”

54. Package repositories
• Major linux distributions keep repositories of
packages for users:
• EPEL
• Ubuntu / Debian official repositories
• You can store a package and its dependencies to
make it easy to install them all on your infrastructure

55. OZinOH, https://flic.kr/p/bRHn2v

56. Package repositories
• createrepo: creates yum repositories
• reprepro: creates apt repositories
• Many other free tools available!
• Read the documentation carefully. Lots of tricky
options.
• I’ll show some examples to get you started!

57. createrepo
http://en.wikipedia.org/wiki/
Yellowdog_Updater,_Modified#mediaviewer/File:Yum.png

58. createrepo
• mkdir /var/www/myrepo
• cp /path/to/rpms/*.rpm /var/www/myrepo
• createrepo /var/www/myrepo
• gpg --detach-sign --armor /var/www/my/repo/repomd.xml

59. GPG is important
• Using GPG to sign the generated repository
guarantees that you generated the repository.
• This is important.
• This means that no one else modified, removed, or
inserted a package other than you.
• GPG signing the repository is not a very well known
security measure, but it is incredibly important!
• This is NOT the same as using rpmsign/rpm --sign.

60. Secure YUM repos
• Sign repository metadata with GPG
• Sign packages with GPG (use rpmsign)
• Serve repositories over SSL
• Enable all the right options for SSL verification,
repository GPG checking, AND package GPG
checking.

61. Wouldn’t it be cool to do all
that with Puppet instead?
Good news: you can!

62. createrepo via puppet
Puppet can create YUM repositories for you!
$ puppet module install palli-createrepo

63. createrepo { 'yumrepo':
repository_dir => '/var/yumrepos/yumrepo',
repo_cache_dir => '/var/cache/yumrepos/yumrepo'
}
createrepo via puppet

64. You still need to GPG sign
the repository yourself :(
exec { “gpg_sign_yumrepo”:
command => “gpg --detach-sign --armor
/var/yumrepos/yumrepo/repodata/repomd.xml“,
}

65. Once the repository is created, it must
be added to the client machines.

66. Add YUM repos with puppet
yumrepo { 'my_repo':
baseurl => "http://myurl.com/repo",
gpgcheck => 1,
repo_gpgcheck => 1,
gpgkey => “http://myurl.com/gpg.pub.key”,
sslverify => 1,
sslcacert => “/etc/pki/tls/certs/ca-bundle.crt”,
enabled => 1,
}
most people never turn on repo_gpgcheck or
sslverify, or set the ssl certificate path, but you
should!!

67. But that’s not all!
• You MUST have the ‘pygpgme’ package
installed on the system that will verify the
signatures.
• Without pygpgme, yum will not be able to verify
signatures!
• Some versions of CentOS / RHEL do not
automatically install pygpgme with yum!!

68. Make sure to install pygpgme
package { 'pygpgme':
ensure => latest,
}

69. reprepro
APT

70. reprepro
• mkdir /var/www/myrepo
• mkdir /var/www/myrepo/conf
• Create a file named “distributions” in the conf
directory

71. reprepro
Codename: precise
Components: main
Architectures: i386 amd64
SignWith: 7ABDB001
/var/www/myrepo/conf/distributions:

72. reprepro
• You can add more sections if you need more code
names (lucid, trusty, etc).
• SignWith specifies which GPG key to use for signing
repository metadata
• You can get your gpg key ID by looking at the output
of gpg —list-keys
• This is not the same as using debsigs/debsign !!!

73. reprepro
import your Ubuntu 12.04 packages:
reprepro -b /var/www/myrepo/ includedeb precise filename.deb

74. Wouldn’t it be cool to do all
that with Puppet instead?
Good news: you can!

75. reprepro via puppet
Puppet can create APT repositories for you!
$ puppet module install jtopjian-reprepro

76. # Base Directory shortcut
$basedir = '/var/lib/apt/repo'
!
# Main reprepro class
class { 'reprepro':
basedir => $basedir,
}
!
# Set up a repository
reprepro::repository { 'localpkgs':
ensure => present,
basedir => $basedir,
options => ['basedir .'],
}
reprepro via puppet

77. # Create a distribution within that repository
reprepro::distribution { 'precise':
basedir => $basedir,
repository => 'localpkgs',
origin => 'Foobar',
label => 'Foobar',
suite => 'precise',
architectures => 'amd64 i386',
components => 'main contrib non-free',
description => ‘My repo',
sign_with => 'F4D5DAA8',
not_automatic => 'No',
}
reprepro via puppet

78. Once the repository is created, it must
be added to the client machines.

79. Add APT repos with puppet
apt::source { 'myrepo':
location => ‘http://myurl.com/repo',
release => 'precise',
repos => 'main',
key => '7ABDB001',
key_source => ‘http://myurl.com/gpg.pub.key',
include_src => true,
}
$ puppet module install puppetlabs-apt

80. But that’s not all!
• You MUST have the ‘apt-transport-https’ package
installed on the system if your repository is served
over HTTPS!
• Without apt-transport-https, you can’t install
packages over HTTPS.
• You definitely want this.

81. Make sure to install apt-transport-https
package { ‘apt-transport-https‘:
ensure => latest,
}

82. Alosh Bennett, https://flic.kr/p/WJ7rE

83. Success
• You can now use beaker/kitchen.ci/etc to test your
infrastructure.
• Determine if the packages you need are actually
installed after your manifests are applied.
• Determine if the repositories you added are
actually added after your manifests are applied.
• Don’t need to wait forever for Ruby, redis, et al to
build during a test run.

84. BEST OF ALL !!!!
• You can now run Puppet on your development
VM using the same manifests you use in
production
• The manifests are applied and you are running
the same exact binaries you run in production
• Won’t catch ALL production bugs, but getting
closer to production during development is
super useful

85. Summary
• Creating package repositories can be tricky. Make
sure to GPG sign repository metadata.
• 99% of package repositories get this wrong.
• Carefully read the documentation of createrepo and
reprepro.
• Make sure to install necessary libraries for verifying
signatures and accessing repositories via HTTPS.
• Always serve up your repositories over HTTPS.

86. Use puppet to automate this.

87. ?@packagecloudio
https://packagecloud.io
joe@packagecloud.io