Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Packaging is the Worst Way 
to Distribute Software, 
except for everything else 
Ryan McKern | Puppet Labs 
mckern@puppetl...
Who is this?
What do I do now? 
Release Engineering at
Maybe you've used 
our products?
What have I done? 
System Administration 
… for 13 years.
I’ve probably been 
your customer
Caveat Audiens
"Prejudice is a great time saver. You 
can form opinions without having 
to get the facts." 
Attributed to E.B. White, 
So...
Let's talk about software!
Distributing software sucks 
Shipping new platforms is so hard 
Cross-platform packaging is so hard 
Unpredictable user-sp...
Everything is so hard
Who among us knows this pain? 
sad@roberto Downloads $ wget -­‐-­‐quiet http:// 
ftpmirror.gnu.org/gcc/gcc-­‐4.9.1/gcc-­‐4...
This was a problem because 
the customer's time has value
Behold! 
ryan@animatronio ~ $ sudo rpm -­‐Uvh http://my.mirror.co/pub/ 
el/7/x86_64/nano-­‐2.3.1-­‐10.el7.x86_64.rpm 
Retr...
What's so great 
about packages?
Dependency management 
calculon ~ # apt-­‐get install cmake 
Reading package lists... Done 
Building dependency tree 
Read...
Verification 
bender ~ # mv /usr/share/man/man8/applydeltarpm.8.gz ~/ 
bender ~ # rpm -­‐V deltarpm 
missing d /usr/share/...
Distribution 
ryan@tinnytim ~ $ gem push erniebert-­‐0.1.0.gem 
Pushing gem to BetterThanRubyGems.org... 
Successfully reg...
What could be better 
about packages?
Sometimes shipping bits 
really is hard
Security is often both the joke 
and the punchline 
ouch@killbot ~ $ dpkg-­‐sig -­‐-­‐verify  
puppet_3.7.1-­‐1puppetlabs1...
So, so, so many similar-but-different 
formats
Let's talk about some popular 
packaging formats
.rpm 
• Managed by the recursively named 
"RPM Package Manager" & yum 
• cpio compressed binaries & text files 
• Post-ins...
.deb 
• Managed by dpkg & apt, the 
"Advanced Package Tool" 
• ar compressed package with two 
gzipped tarballs & a small ...
Mac .pkg 
• Used by Mac OS X, and often delivered 
in a .dmg (disk image) or a .zip file 
• xar compressed archive, contai...
About all those post-install 
shell scripts 
Maybe they're not that safe, but the 
surface area of this problem is big. 
T...
Ruby .gem, Python .egg, 
and Node .npm 
• These are library managers with 
delusions of grandeur 
• Reuses the "download, ...
What about... ? 
#realtalk 
We only have 45 minutes, and I hope 
you're going to have some questions for 
me to evade
What are some alternatives?
Source tarballs
curl | bash
Full Disclosure 
• Puppet Labs does use the curl|bash 
technique as an option for our PE 
agent installation 
• If you don...
You just wanted Ruby 
but you got Cthulhu 
~ $ curl -­‐sSL https://get.rvm.io | bash -­‐s -­‐-­‐ -­‐-­‐fhtagn 
G̺̞ 
̯͔̮̫̥ ...
curl | bash often assumes 
• There is no air-gap 
• Every request is a safe & sane request 
• That HTTPS is good enough
curl | bash often forgets 
• >100% Broadband coverage 
• Mirrors exist 
• HTTPS secures transport, not content
curl | bash totally ignores 
• The benefits of reusability 
• The fragility of shell scripts 
• The fragility of shells
Security is hard 
• RVM recently introduced hand-rolled 
GPG signing* 
• Thread had 48 comments within a 
week, almost uni...
Omnibus
Isn't that from Chef? 
• Sure, but so is Test Kitchen 
• Builds packages while still controlling 
the entire dependency st...
Omnibus is one way to skin 
the entire cat 
• Abstracts (instead of removes) 
dependency management 
• Only builds package...
FPM
Effing Package Managers 
•General purpose swiss-army knife of 
package building 
•Works around a lot of the shortcomings 
...
"Common packaging patterns, a 
distaste for existing packaging 
practices, and some hate-driven 
development yielded FPM! ...
Effing FPM 
• Swiss army knives are rarely the best 
tool for a given job 
• General purpose in this case means a 
lot (~1...
Why so many alternatives? 
What went wrong?
RPM Packaging can 
be tough 
• RPM Spec files are weird 
• Kind-of M4, kind of Shell, all obtuse 
• Oh, and kind-of Make; ...
Deb Packaging can feels 
like penance 
• "debian/" directories are outright 
hostile to man & beast alike 
• Debian "Helpe...
Conflation of purpose 
• Some library managers try to install 
executables, e.g. gem, pip, npm 
• Remember when I said "de...
But really, I just have a 
hypothesis! 
• Developers love solving new problems 
• Sometimes they confuse their 
problems f...
Where do we go from here?
Sometimes the only choices you have 
are bad ones; but you still have 
to choose.
TL;DR: this problem is 
(mostly) solved 
Stop writing new installers 
from scratch 
Give your customers the best 
packages...
Thank you 
You're wonderful. Thank you for letting 
me rant at you for as long as you did. 
mckern@puppetlabs.com 
@the_mc...
Questions?
Packaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything Else
Nächste SlideShare
Wird geladen in …5
×

Packaging is the Worst Way to Distribute Software, Except for Everything Else

1.764 Aufrufe

Veröffentlicht am

As part of the 2014 USENIX Release Engineering Summit West, I presented a talk about packaging software and what's wrong with current trends.

Here's the abstract:
Reliably distributing software is a notoriously difficult problem, and almost every operating system and programming language vendor has tried to solve it. This has led to a herd of packaging systems, almost none of which are cross-compatible; some manage system-level software, while others focus on extending their own language (often by trampling on system-level software). And like all competing standards, every packaging system comes with its own sharp corners, dull edges, and hidden idiosyncrasies to deal with along the path to packaging happiness. In an attempt to answer the question "How do I install this software and ensure that its dependencies are fulfilled?", some novel solutions have begun to see popular adoption. But a lot of these newer tools and techniques tread the same ground as their predecessors while overlooking the lessons that were learned along the way.

I'll talk about the state of native packaging systems on some popular platforms (Debian/Ubuntu, RHEL/CentOS/Fedora, and Mac OS X), packaging systems for popular languages (Ruby, Python, Perl, and Node) and the ways that developers are attempting to work around the limitations of these systems. I'll review the reasons that tools like curlbash, FPM, and omnibus packages have become popular by sharing lessons I've learned while working through these systems. While this will be an amusing presentation, I'll show how native packages can address the concerns that have pushed Release Engineers and Developers away. I will also talk about what native packaging systems can learn from the next generation of packaging tools.

The original abstract is available here:
https://www.usenix.org/conference/ures14west/summit-program/presentation/mckern

Veröffentlicht in: Software, Technologie
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Packaging is the Worst Way to Distribute Software, Except for Everything Else

  1. 1. Packaging is the Worst Way to Distribute Software, except for everything else Ryan McKern | Puppet Labs mckern@puppetlabs.com
  2. 2. Who is this?
  3. 3. What do I do now? Release Engineering at
  4. 4. Maybe you've used our products?
  5. 5. What have I done? System Administration … for 13 years.
  6. 6. I’ve probably been your customer
  7. 7. Caveat Audiens
  8. 8. "Prejudice is a great time saver. You can form opinions without having to get the facts." Attributed to E.B. White, Source unknown
  9. 9. Let's talk about software!
  10. 10. Distributing software sucks Shipping new platforms is so hard Cross-platform packaging is so hard Unpredictable user-space is so hard Moving the packaged bits is so hard
  11. 11. Everything is so hard
  12. 12. Who among us knows this pain? sad@roberto Downloads $ wget -­‐-­‐quiet http:// ftpmirror.gnu.org/gcc/gcc-­‐4.9.1/gcc-­‐4.9.1.tar.bz2 sad@roberto Downloads $ tar xjf gcc-­‐4.9.1.tar.bz2 sad@roberto Downloads $ cd gcc-­‐4.9.1/ sad@roberto Downloads $ ./configure ./configure: line 532: sed: command not found ./configure: line 1371: sed: command not found ./configure: line 1920: sed: command not found ./configure: line 2291: sed: command not found configure: error: cannot run /bin/sh ./config.sub ./configure: line 361: sed: command not found ./configure: line 310: sort: command not found
  13. 13. This was a problem because the customer's time has value
  14. 14. Behold! ryan@animatronio ~ $ sudo rpm -­‐Uvh http://my.mirror.co/pub/ el/7/x86_64/nano-­‐2.3.1-­‐10.el7.x86_64.rpm Retrieving http://my.mirror.co/pub/el/7/x86_64/ nano-­‐2.3.1-­‐10.el7.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:nano-­‐2.3.1-­‐10.el7 ################################# [100%] ryan@animatronio ~ $
  15. 15. What's so great about packages?
  16. 16. Dependency management calculon ~ # apt-­‐get install cmake Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: cmake-­‐data emacsen-­‐common libarchive12 libnettle4 libxmlrpc-­‐core-­‐c3 The following NEW packages will be installed: cmake cmake-­‐data emacsen-­‐common libarchive12 libnettle4 libxmlrpc-­‐core-­‐c3 0 upgraded, 6 newly installed, 0 to remove and 51 not upgraded.
  17. 17. Verification bender ~ # mv /usr/share/man/man8/applydeltarpm.8.gz ~/ bender ~ # rpm -­‐V deltarpm missing d /usr/share/man/man8/applydeltarpm.8.gz bender ~ #
  18. 18. Distribution ryan@tinnytim ~ $ gem push erniebert-­‐0.1.0.gem Pushing gem to BetterThanRubyGems.org... Successfully registered gem: erniebert (0.1.0) ryan@tinnytim ~ $ gem install erniebert Fetching: ffi-­‐1.9.6.gem (100%) Building native extensions. This could take a while... Successfully installed ffi-­‐1.9.6 Fetching: erniebert-­‐0.1.0.gem (100%) Successfully installed erniebert-­‐0.7.1 1 gem installed ryan@tinnytim ~ $
  19. 19. What could be better about packages?
  20. 20. Sometimes shipping bits really is hard
  21. 21. Security is often both the joke and the punchline ouch@killbot ~ $ dpkg-­‐sig -­‐-­‐verify puppet_3.7.1-­‐1puppetlabs1_all.deb Processing ./puppet_3.7.1-­‐1puppetlabs1_all.deb... GOODSIG _gpgbuilder C093A3A56A6E0BEEA2821DD7133957EA11028DF3 1413702159 ouch@killbot ~ $ dpkg-­‐sig -­‐-­‐verify ./puppet_2.7.23-­‐1~deb7u3_all.deb Processing ./puppet_2.7.23-­‐1~deb7u3_all.deb... ouch@killbot ~ $
  22. 22. So, so, so many similar-but-different formats
  23. 23. Let's talk about some popular packaging formats
  24. 24. .rpm • Managed by the recursively named "RPM Package Manager" & yum • cpio compressed binaries & text files • Post-installation tasks are shell scripts
  25. 25. .deb • Managed by dpkg & apt, the "Advanced Package Tool" • ar compressed package with two gzipped tarballs & a small text file • Post-installation tasks are shell scripts
  26. 26. Mac .pkg • Used by Mac OS X, and often delivered in a .dmg (disk image) or a .zip file • xar compressed archive, containing a binary file, two archives, and an XML document • post-installation tasks are still shell scripts
  27. 27. About all those post-install shell scripts Maybe they're not that safe, but the surface area of this problem is big. That doesn't mean we needed "dash"
  28. 28. Ruby .gem, Python .egg, and Node .npm • These are library managers with delusions of grandeur • Reuses the "download, decompress, configure, build, install" patterns, which hasn't got much spam in it • Constant compilation is a bummer
  29. 29. What about... ? #realtalk We only have 45 minutes, and I hope you're going to have some questions for me to evade
  30. 30. What are some alternatives?
  31. 31. Source tarballs
  32. 32. curl | bash
  33. 33. Full Disclosure • Puppet Labs does use the curl|bash technique as an option for our PE agent installation • If you don't trust your own Puppet Master, who do you trust? • (ALL THE COOL KIDS WERE DOING IT)
  34. 34. You just wanted Ruby but you got Cthulhu ~ $ curl -­‐sSL https://get.rvm.io | bash -­‐s -­‐-­‐ -­‐-­‐fhtagn G̺̞ ̯͔̮̫̥ ͊̌͂ a͍͕͓̦͈̯̟̋ r̘̰̟e̓̓ ̦ C̋͋ͬt̂̅̓ t͇̻̩̲̬ ͇̪̹͔̾ ̟ḧ́΅ͭ ̩̿ͭ ͖̙̤ ͭl̅ͦ̓ ̝̙̭ ̗ ṷ ů̥͖ ͍͎͍ ̦̟ n̠̣̭ ̞̻̱̳ ̬ ̣̗ ̑ ̖͎ͩ hͯ ͐ ̝̤̊ ̞̭̳͚̞ ̘ s ̓ ͔̣ ̺̝͇ l̃ͪ͐̎̍ a̅͋̏̀ ̜̯͉ ͈͇̲ ̓̑ ̭̻l̂ͬ̽ ͮ ̙͇̼͍ r̭̂̋ͦ ̻̺̭ ̗͙̃ ̻̤̳ ̰̤ i̅̿̌ͫͣͪ ̺̙̽_̻͚ ̤ s ̮͇ e͍̞̚ ̿̌ͮ̍ ̝͕̳́̽ ̩̺ͅ ͉ r f͈̱͓͓ ̦̰̬̗ ̗̝̼ m̞̗͎͍̾̈͊ o͈̩ͪ̈ ̟̤̻͉ ̃ͨͬ́̉ͩ̓ ̰̝e͍͎h '̼̬̤̋̉̽lͪ ͣ ̗̼ͥ̉R ̱͚̏ ̫͙͓̰͔ ̃̂̂ ͕͓̲ y .̚ ̣̫ ̞͓͈ ̼̪̠s͔̹̞̟t͈̘͕ H͎̯ ̙̱a͍̟͍ͅ ̘̼ u͙͓̙̟r̥̹̫͇͎ ͚ ̻ͅ ̣ ̙̹ ͍̮ t ̻̳̮ h ̩̜̣_ e̠_̱̣͔ ̼ ̺͉U s͈̰̣̥ p̻ a͕̗̣̺ k ͇e̤͍̯ ̻̹͓̬ ̹̤̳a̠͍̪ b̩_ ̪_̥͖͎͍ e̪̻̣̣ ̙̼ ͈̬s͇̮̞ ḻ͇ ̥͖̠ ̹̩̖ ̙̲ a̺͈̹̤͈͉ͅ ̣̮͕̙ ̗ h ̠̟ ̰̜̜l̬̹̭ l̺̞ ̩̳̮̩̰͕ͅ ̻r̮̥̦͍͍ ͈̫ e̳̠̙̘ ̱ u̠͇r̮̣͓ ̘̬̰ t͔͚̳̹̰ ̰͖ n̗͍ ̥͕ ̥͉f̜͚r̯͍ͅ o͈̯̦ ̖ ̳͓̦͔ ¯ͭ̔ ̻͙̫̪̪̖͈ ͔̬ ̣̌ ̠̟̱̒ ͍m̻̟ ̭ ̼̠ ͍̣t͖h͔͉̞ e̬̫̦ ̋͂ ̖͇̼ ̊ͤ̓̋̄̐͌̾ ̩̝̮ ͓ d̰̼̞̤͕ ̤̘̣̭͍̖ ̻͈ ̟̭ ͡l̴l h͞ow l͢ f̕o ́̃̍̆̂̇̒ͫ k̲͖̻̻̆͋ͬ̑ ̩͍̭̙ ̥ k̙̣͕͔ ̘̮̤̻̜̳ r̯̰̱̬̭ͅ ḁ͙ ͔͔̺ ̠ ̗ s̻̱͎ ̙̦̝̗͍͎ ̞̪t̫͉̟̻ ͖ ạ̫ r͔̺ ͍.͔̖͚̺̹ ̰̫ Ǹ`ya͠rĺath̢ote͡p̢ ͟s̀`h ̨ a ̷reve̢r̀` ͑̎ ň̽̌ ḯ΅ ̣ ̃̄̇ͪ̂͑ ͉͔̙̤̪̜ h ̉ͦ ͕t͂̔ ͨ͋̅̿ ͔̆ͫ̓ͫ ̫͖̻ e ͫ͌͛ ͦ͆ͭ̽ ̊ͩͩ̇ͣ ̗ͅd̂ ā̇ͤ͋ͭͨ ̗̰ ͙̗̝͕̩̥ ̟͍ ř nͮͯ̑̿͒ e̍͒̅̄ͣ̀ ͅͅ ̪̠̗͕̥ ͋̋ ͙̹͎̺̠ s̊̈̽̊̌ s̈̌ͪ ̱̳ .̄̑̎ ͔̙̣ ̤̰̟̦̥ ͉̉ ͙̬
  35. 35. curl | bash often assumes • There is no air-gap • Every request is a safe & sane request • That HTTPS is good enough
  36. 36. curl | bash often forgets • >100% Broadband coverage • Mirrors exist • HTTPS secures transport, not content
  37. 37. curl | bash totally ignores • The benefits of reusability • The fragility of shell scripts • The fragility of shells
  38. 38. Security is hard • RVM recently introduced hand-rolled GPG signing* • Thread had 48 comments within a week, almost universally about the implementation • Broke semver, automation, and hearts * https://github.com/wayneeseguin/rvm/issues/3105
  39. 39. Omnibus
  40. 40. Isn't that from Chef? • Sure, but so is Test Kitchen • Builds packages while still controlling the entire dependency stack • Lots of love from users with complicated dependency stacks
  41. 41. Omnibus is one way to skin the entire cat • Abstracts (instead of removes) dependency management • Only builds packages for the platform it's installed on • You're going to want to know Ruby
  42. 42. FPM
  43. 43. Effing Package Managers •General purpose swiss-army knife of package building •Works around a lot of the shortcomings of existing package managers •Jordan Sissel is a SAINT (Shout out to #hugops!)
  44. 44. "Common packaging patterns, a distaste for existing packaging practices, and some hate-driven development yielded FPM! Add some amazing contributions in code, bugs, features, and support from the community and boom we have modern FPM." Jordan Sissel My inbox, Oct 10 2014
  45. 45. Effing FPM • Swiss army knives are rarely the best tool for a given job • General purpose in this case means a lot (~150ish) of command line flags • Still infinitely better than curl | bash
  46. 46. Why so many alternatives? What went wrong?
  47. 47. RPM Packaging can be tough • RPM Spec files are weird • Kind-of M4, kind of Shell, all obtuse • Oh, and kind-of Make; only kind-of • Sort-of competing RPM standards
  48. 48. Deb Packaging can feels like penance • "debian/" directories are outright hostile to man & beast alike • Debian "Helpers" usually don't • dpatch can use unified diffs (sane) or shell scripts (what?!)
  49. 49. Conflation of purpose • Some library managers try to install executables, e.g. gem, pip, npm • Remember when I said "delusions of grandeur"? (Google Image Search was kind of useless here)
  50. 50. But really, I just have a hypothesis! • Developers love solving new problems • Sometimes they confuse their problems for the customer's problems • Maybe packaging isn't a solved problem yet, but it's close
  51. 51. Where do we go from here?
  52. 52. Sometimes the only choices you have are bad ones; but you still have to choose.
  53. 53. TL;DR: this problem is (mostly) solved Stop writing new installers from scratch Give your customers the best packages possible Don't forget Pareto (any number of 80/20 rules)
  54. 54. Thank you You're wonderful. Thank you for letting me rant at you for as long as you did. mckern@puppetlabs.com @the_mckern
  55. 55. Questions?

×