Cooking Perl with Chef: Real World Tutorial with Jitterbug

Cooking Perl with Chef
Real World Tutorial with Jitterbug
Copyright © 2012 by David A. Golden
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License
Tutorial Version 1.0
http://perlchef.com/

Abstract

This tutorial provides a command-by-command walk-through for deploying the Jitterbug continuous
integration server using the Chef configuration management tool.

Prerequisites

This tutorial was written with Unix operating systems in mind and was tested using Ubuntu Linux
12.04 LTS. You will need to be familiar with Perl, CPAN, git and common Unix tools like ssh and
rsync. You will need an Internet connection to download files during this tutorial.

The tutorial is written using a Linode virtual machine for deployment. You will need a Linode account
or else must have a real/virtual machine available to which you can deploy a fresh operating system.

You will also need to install the Pantry tool, which is available on CPAN. Install it using your preferred
CPAN client.

Note that you won't need Chef installed on your own machine. You only need it on the virtual machine
you are deploying to and this bootstrap will be covered in Step 1 of the tutorial.

I assume that you have already seen the "Cooking Perl with Chef" overview presentation and "Hello
World" tutorial before you attempt this one, so I will not explain some Chef fundamentals again here.
If you have not seen these, please follow the links on http://perlchef.com/ before continuing.

Overview
Our goal is to deploy the Jitterbug continuous integration server (http://lumberjaph.net/jitterbug/) to a
small Linode virtual machine using Chef, Pantry and the Perl Chef cookbooks. This Jitterbug server
will be ready for testing Perl/CPAN modules using Github's webhook integration.

This tutorial is broken up into five distinct steps:

• Provision Linode, deploy SSH keys and install Chef
• Set up Pantry and third-party cookbooks
• Adapt Jitterbug for Chef and Carton
• Specify the configuration for the server
• Deploy and test

Cooking Perl with Chef: Real World Tutorial with Jitterbug Page 2

We will deploy these components as our "stack":

• Nginx web server
• Jitterbug web application (behind the nginx proxy)
• Jitterbug builder (the task worker process)
• Postfix mail server (to send failure reports)
• Perl 5.14.2

We will also configure a firewall to restrict everything except ports 22 and 80.

You can contact the author with praise, critique or questions at xdg@xdg.me or as @xdg on various
social networks.

Step 1. Provision a Linode, deploy SSH keys and install Chef

This tutorial uses Linode (http://linode.com/). If you don't already have a Linode account you should
create one now. If you don't want to use Linode, you should be able to adapt this provisioning step for
another virtual machine provider you prefer.

Add a linode 512 on a month-to-month (MTM) plan. (Linode will pro-rate you a refund when delete
the linode, so you'll only pay for a bit of usage if you only plan to use it for this demo.) Use only
10,000 MB for the main drive and deploy Ubuntu 12.04LTS. Remember the root password you assign;
you'll need that to deploy SSH keys (and then you can stop using the password or disable it). When
that is complete, boot the linode.

If you manage your own domain somewhere, setup a DNS A record mapping a server name to the
public IP address of the linode. Set linode's reverse DNS to map back to that hostname. If you don't
set up your own DNS, you'll need to use the default linode-provided hostname instead.

NOTE: When you see "jitterbug.example.com" in the tutorial, use your own hostname instead!

On your own computer (not the linode), you need to create a directory to hold your configuration
information. You should keep your configuration under version control, so this tutorial shows how to
do that with git.

$ mkdir /tmp/jitterbug-config
$ cd /tmp/jitterbug-config
$ git init

Next, you'll want to create SSH keys you'll use to connect to the jitterbug server. You will be
prompted for a passphrase for the private key, and it's a good idea to use one.

$ ssh-keygen -f jitterbug-key


Then, check the keys into git.

$ git add jitterbug-key*
$ git commit -m "added jitterbug SSH keys"

Now that the keys have been created, deploy them to the server (using the root password).

$ ssh-copy-id -i jitterbug-key root@jitterbug.example.com

Add the key to your SSH agent and try it out.

$ ssh-add jitterbug-key
$ ssh root@jitterbug.example.com

Once you're logged into remote machine, you'll need to install the Chef client on it. These steps are
adapted from the Opscode Chef wiki:

$ echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" >
/etc/apt/sources.list.d/opscode.list
$ mkdir -p /etc/apt/trusted.gpg.d
$ gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
$ gpg --export packages@opscode.com >
/etc/apt/trusted.gpg.d/opscode-keyring.gpg
$ apt-get update
$ apt-get upgrade -y
$ apt-get install -y opscode-keyring
$ DEBIAN_FRONTEND=noninteractive apt-get install -y chef
$ /etc/init.d/chef-client stop
$ update-rc.d -f chef-client remove
$ chef-solo --version

The version you see should be at least 10.4. Note that the regular Chef client is disabled because we'll
be using chef-solo instead.

Once Chef is installed, log out of the server.

$ exit

At this point, you should duplicate the disk drive and keep it as a base image with Chef already
bootstrapped in case you need to start over for any reason (or for future projects. On the Linode
dashboard, "edit" the drive and click "duplicate" to make a copy.

Step 2. Set up Pantry and third-party cookbooks

We use the Pantry tool to manage configuration and deploy with chef-solo. Start off by initializing the
current directory for Pantry.

$ pantry init


Next, you need to download several cookbooks with Chef deployment recipes. Some of these will
come from the Opscode community site, some will come from Opscode Github repositories, and some
will come from the Perl Chef Github repository. Some will be used directly as we configure the
Jitterbug server and others are dependencies.

Here is the list of cookbooks with an explanation of their purpose

• apt – ensures apt-get update is run before further configurations
• carton – used to deploy the Jitterbug application
• firewall – common firewall framework code
• hostname – set the hostname
• nginx – deploy and configure Nginx
• ntp – deploy ntpd
• ohai – common framework for Ohai plugins (used by Chef)
• perlbrew – deploy Perl intepreters
• postfix – deploy Postfix
• runit – used by carton for persistent services
• ufw – firewall configuration

I find it helpful to stage third-party cookbooks in a separate directory first before copying them to the
Pantry cookbooks directory. This lets me keep the Pantry cookbooks directory under version control
that is specific to my own configuration, while letting me browse third-party cookbook code
repositories independently. (git submodules could be used for this, but that gets complex.)

Start by creating another directory for staging cookbooks.

$ mkdir /tmp/jitterbug-src
$ cd /tmp/jitterbug-src

One handy trick is to get all the cookbooks in a similar directory structure, organized by source and
under a cookbooks directory. Some repositories are like this already, others need to be cloned to
specific locations. Afterwords, we can write a simple script to rsync them all to the right place in the
Pantry directory.

First, we'll get all the Opscode cookbooks, including a custom version from my own repository with
some bug fixes that haven't been merged upstream yet.

$ mkdir -p opscode/cookbooks
$ cd opscode/cookbooks
$ git clone git://github.com/opscode-cookbooks/apt.git
$ git clone git://github.com/opscode-cookbooks/firewall.git
$ git clone git://github.com/opscode-cookbooks/nginx.git
$ git clone git://github.com/opscode-cookbooks/ntp.git
$ git clone git://github.com/opscode-cookbooks/ohai.git
$ git clone git://github.com/opscode-cookbooks/postfix.git
$ git clone git://github.com/dagolden/runit.git -b CHEF-154
$ git clone git://github.com/opscode-cookbooks/ufw.git
$ cd ../..


The next cookbooks don't need extra subdirectories, because they already have a "cookbooks" directory
in the repository. We'll actually download a Jitterbug cookbook now, as well, even though in Step 3
we'll walk through it as if we were creating it from scratch.

$ git clone git://github.com/dagolden/perl-chef.git
$ git clone git://github.com/dagolden/jitterbug.git -b carton-chef

The hostname cookbook does not appear to have a repository, so we'll download a tarball instead.
(Apologies for the smaller font, but I wanted to keep "curl ..." all on one line.)
$ mkdir -p community/cookbooks
$ cd community/cookbooks
$ curl -L http://community.opscode.com/cookbooks/hostname/versions/0_0_2/downloads | tar xz
$ rm hostname.tgz
$ cd ../..

Now that we have all the cookbooks we need, we need to copy them over to the Pantry cookbook
directory. Create this little shell script to make it easy:

$ cat > copy-cookbooks.sh
#!/bin/bash
for d in *; do
if [[ -d $d ]]; then
rsync -av --exclude=.git $d/cookbooks/ /tmp/jitterbug-config/cookbooks
fi
done

Run that script, then go to the Pantry directory and check everything in.

$ chmod +x copy-cookbooks.sh
$ ./copy-cookbooks.sh
$ git add cookbooks
$ git commit -m "imported cookbooks"

Step 3. Adapt Jitterbug for Chef and Carton

In Step 2, we downloaded a cookbook for Jitterbug, but let's pretend it didn't exist and we had to create
it. In this section, I'll describe how I did that. If you're not interested in learning how to make a
cookbook, you can skip ahead to Step 4. (If you're really hard-core, you can delete the jitterbug
cookbook you downloaded, and recreate it using these instructions.)

To create the Jitterbug cookbook, I started by forking the project on Github
(git://github.com/franckcuny/jitterbug.git) and creating a new branch in my own repository:

$ cd ~/git
$ git clone git://github.com/dagolden/jitterbug.git
$ git checkout -b carton-chef


To adapt Jitterbug for Chef deployment with Carton, I need a carton.lock file to hold dependency
information and a Chef cookbook. The cookbook needs these files:

• attributes/default.rb – configuration attributes
• recipes/default.rb – a deployment recipe
• files/default/jitterbug.db – an empty SQLite database with the Jitterbug schema
• templates/default/config.yml.erb – Jitterbug's configuration file
• templates/default/jitterbug.conf.erb – app-specific Nginx configuration file

If I were creating a cookbook to upload to Opscode's community site, I'd also need to write a
metadata.rb file and a README.rdoc file, but I didn't do that for this tutorial.

Pantry has a command for creating a blank cookbook under the cookbooks directory. I could have run
that in the Pantry directory, but in this case, I ran it in the Jitterbug branch so I could share it later.

$ pantry create cookbook jitterbug

That creates several directories under cookbooks/jitterbug and touches some empty files to fill in. For
the files under the attributes and recipe directories, I then copied and adapted my Hello World tutorial
recipe.1 For the Nginx configuration file, I created one based on how Fletcher Nichol wrote one for
Jenkins.2 I still find cookbook creation to be a bit of a black-art, so this is a pretty common pattern for
me. I find things similar to what I want to do, use that as a base, and tweak it to my needs. The other
cookbook files, I had to create from scratch.

Creating the carton.lock file was straightforward. I used Perlbrew to install Perl 5.14.2 (which is what
I plan to deploy with), activated it, and installed the Carton module from CPAN. Then creating the
carton.lock file was just:

$ carton install

Next, I modified the attributes/default.rb file to contain the configuration attributes I need for the
recipe and template files. Here's what it looks like:
# perlbrew to execute with (should be a legal perlbrew target)
default['jitterbug']['perl_version'] = 'perl-5.14.2'

# Install directory, repo and tag
default['jitterbug']['deploy_dir'] = '/opt/jitterbug'
default['jitterbug']['deploy_repo'] = 'git://github.com/dagolden/jitterbug.git'
default['jitterbug']['deploy_tag'] = 'carton-chef'

# Service user/group/port
default['jitterbug']['user'] = "jitterbug"
default['jitterbug']['group'] = "jitterbug"
default['jitterbug']['port'] = 3000

# Jitterbug config
default['jitterbug']['db_dir'] = "/var/lib/jitterbug"
default['jitterbug']['conf_dir'] = "/etc/jitterbug"
default['jitterbug']['on_failure_subject_prefix'] = "[jitterbug] FAIL "
default['jitterbug']['on_failure_to_email'] = ""

1 https://github.com/dagolden/zzz-hello-world
2 https://github.com/fnichol/chef-jenkins/blob/master/recipes/proxy_nginx.rb


default['jitterbug']['on_failure_cc_email'] = "alice@example.com"
default['jitterbug']['on_failure_from_email'] = "donotreply@example.com"
default['jitterbug']['on_pass_subject_prefix'] = "[jitterbug] PASS "
default['jitterbug']['on_pass_to_email'] = ""
default['jitterbug']['on_pass_cc_email'] = "alice@example.com"
default['jitterbug']['on_pass_from_email'] = "donotreply@example.com"

I decided to deploy as the "jitterbug" user and group, so I'll need to remember to configure that user
later in the deployment recipe.

After defining attributes, it was time to create the templates for configuration files. The Jitterbug
repository already contains some sample configuration files, config.yml and example.yml, so I copied
the example.yml file to cookbooks/jitterbug/templates/default/config.yml.erb, tweaked it, and replaced
some of the sample configuration values with entries from the attributes/default.rb file. I only did a
partial replacement to demonstrate it for the tutorial. In theory, every one of the config values could be
parameterized. Here is the resulting file:
layout: "main"
logger: "console"
appname: "jitterbug"

builds_per_feed: 5
template: "xslate"
engines:
xslate:
path:
- "<%= node['jitterbug']['deploy_dir'] %>"
type: text
cache: 0

jitterbug:
reports:
dir: /tmp/jitterbug
build:
dir: /tmp/build
build_process:
builder: ./scripts/perlchef-capsule.sh
builder_variables:
on_failure: jitterbug::Emailer
on_failure_to_email: "<%= node['jitterbug']['on_failure_to_email'] %>"
on_failure_cc_email: "<%= node['jitterbug']['on_failure_cc_email'] %>"
on_failure_from_email: "<%= node['jitterbug']['on_failure_from_email'] %>"
on_failure_subject_prefix: "<%= node['jitterbug']['on_failure_subject_prefix'] %>"
on_failure_header:
on_failure_footer:
on_pass: jitterbug::Emailer
on_pass_to_email: "<%= node['jitterbug']['on_pass_to_email'] %>"
on_pass_cc_email: "<%= node['jitterbug']['on_pass_cc_email'] %>"
on_pass_subject_prefix: "<%= node['jitterbug']['on_pass_subject_prefix'] %>"
on_pass_from_email: "<%= node['jitterbug']['on_pass_from_email'] %>"
on_pass_header:
on_pass_footer:
reuse_repo: 1
options:
email_on_pass: 0

plugins:
DBIC:
schema:
skip_automake: 1
pckg: "jitterbug::Schema"
connect_info:
- "dbi:SQLite:dbname=<%= node['jitterbug']['db_dir'] %>/jitterbug.db"


The Nginx configuration file, templates/default/jitterbug.conf.erb is also straightforward:
server {
listen 80;
server_name <%= node[:fqdn] %>;

location / {
proxy_pass http://127.0.0.1:<%= node['jitterbug']['port'] %>;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
}

error_log <%= node[:nginx][:log_dir] %>/jitterbug-error.log;
access_log <%= node[:nginx][:log_dir] %>/jitterbug-access.log;
}

To create the empty SQLite database, I first ran Jitterbug's schema deployment tool, then moved the
resulting database to cookbooks/jitterbug/files/default/jitterbug.db:

$ carton exec -I lib -- scripts/jitterbug_db --config config.yml --deploy
$ mv jitterbug.db cookbooks/jitterbug/files/default/jitterbug.db

During deployment, this file will get deployed as the starting database, but only if it doesn't already
exist. This is a naively simple way to deploy an SQLite database. A more sophisticated recipe might
get the deployment database from a backup location to help with future disaster recovery deployment
and we'd seed the backup location with the empty database for first deployment.

Next, I need the deployment recipe in recipes/default.rb to tie all these components together. Because
it's long, I'll explain it it pieces, but you can see the whole thing in the location cloned during Step 2.

The first part of the recipe includes dependency cookboks, ensures that some required OS packages are
installed, and creates a jitterbug user.
include_recipe 'carton'
include_recipe 'perlbrew'
include_recipe 'nginx'

package 'git-core'
package 'libxml2-dev'
package 'libexpat-dev'
package 'zlib1g-dev'

user node['jitterbug']['user'] do
home '/home/jitterbug'
end

The next part of the recipe uses git to check out the jitterbug application from the repository. The
destination, repostory and source tag are all attributes. We also tell Chef to notify the applications
(defined later) to restart if anything has changed:
git node['jitterbug']['deploy_dir'] do
repository node['jitterbug']['deploy_repo']
reference node['jitterbug']['deploy_tag']
notifies :restart, "carton_app[jitterbug]"
notifies :restart, "carton_app[jitterbug-builder]";
end


Next, we deploy the Nginx template:
template "#{node[:nginx][:dir]}/sites-available/jitterbug.conf" do
source "jitterbug.conf.erb"
owner 'root'
group 'root'
mode '0644'

if File.exists?("#{node[:nginx][:dir]}/sites-enabled/jitterbug.conf")
notifies :restart, 'service[nginx]'
end
end

Then we deploy the Jitterbug configuration file into a specified directory:
directory node['jitterbug']['conf_dir'] do
owner node['jitterbug']['user']
group node['jitterbug']['group']
end

template "#{node['jitterbug']['conf_dir']}/config.yml" do
source "config.yml.erb"
mode '0644'
notifies :restart, "carton_app[jitterbug]";
notifies :restart, "carton_app[jitterbug-builder]";
end

Next, the database is deployed, also into a specific directory. Note the "action :create_if_missing" line
– that ensures we don't overwrite an existing database if we re-run the configuration. The extra "file"
resource stanza ensures the database has the right user/permissions, even if it does exist.
directory node['jitterbug']['db_dir'] do
end

cookbook_file "#{node['jitterbug']['db_dir']}/jitterbug.db" do
source "jitterbug.db"
mode "0644"
action :create_if_missing
end

file "#{node['jitterbug']['db_dir']}/jitterbug.db" do
mode "0644"
action :touch
end

With the configuration files and database deployed, the final step is to deploy two application services.
The first is the Jitterbug web application and the second is the Jitterbug task worker, which actually
does the testing.


carton_app "jitterbug" do
perlbrew node['jitterbug']['perl_version']
command "#{node['jitterbug']['deploy_dir']}/jitterbug.pl -p #{node['jitterbug']['port']}"
cwd node['jitterbug']['deploy_dir']
user node['jitterbug']['user']
environment ({ 'DANCER_CONFDIR' => node['jitterbug']['conf_dir'] })
action [:enable, :start]
end

carton_app "jitterbug-builder" do
perlbrew node['jitterbug']['perl_version']
command "perl #{node['jitterbug']['deploy_dir']}/scripts/builder.pl -c #{node['jitterbug']
['conf_dir']}/config.yml"
cwd node['jitterbug']['deploy_dir']
user node['jitterbug']['user']
action [:enable, :start]
end

Finally, the Jitterbug Nginx configuration is enabled.
nginx_site "jitterbug.conf" do
enable true
notifies :reload, 'service[nginx]'
end

Then I made sure all this work was checked into git and pushed up to Github, so it was ready for the
tutorial.

Step 4. Specify the configuration for the server

Now that you have all the cookbooks you'll need, it's time to create some roles and then apply the roles
and recipes to the server node. (We could do everything without roles, but I want to show how you
might use them.)

The first role is a "base" role that we would want to apply to any node. It does some basic
housekeeping, enables a firewall, and turns on NTP. (Make sure you're back in the Pantry directory
before you continue.)

$ pantry create role base
$ pantry apply role base -r apt -r ohai -r hostname -r ufw -r ntp

Note that we don't apply any firewall rules in the role, we merely ensure that the firewall is enabled (by
default only port 22 is allowed). We'll override that later in a "web" role to open up port 80.

The base roles can have attributes we want everywhere. For example, we can make sure that Perlbrew
always builds in parallel and without tests.

$ pantry apply role base -d perlbrew.install_options="-j 5 -n"

This doesn't cause perlbrew to run on a node, it just sets some default attributes for any node that does
actually configure perlbrew.


Next, we'll create a "web" role that deploys Nginx and opens up port 80 in the firewall.

$ pantry create role web
$ pantry apply role web -r nginx

We can also set an attribute that disables the default Nginx web page:

$ pantry apply role web -d nginx.default_site_enabled=false

Unfortunately, the data structure the ufw recipe expects for firewall data can't be specified on the Pantry
command line. You'll need to tell Pantry you want to manually edit the role JSON file and add the
necessary data structure directly.

Start by editing the file:

$ pantry edit role web

Then, edit the "default_attributes" data to add a section for the firewall configuration. The final result
should look like this:
{
"json_class" : "Chef::Role",
"run_list" : [
"recipe[nginx]"
],
"chef_type" : "role",
"override_attributes" : {},
"default_attributes" : {
"firewall" : {
"rules" : [
{
"http" : {
"port" : 80
}
}
]
},
"nginx" : {
"default_site_enabled" : false
}
},
"name" : "web"
}

Since Jitterbug wants to send out email reports when test fail, we need to configure a mail client.
Again, we'll create an "mx" (mail exchange) role, add postfix to that role, and configure postfix to be a
master (i.e. sends mail directly).

$ pantry create role mx
$ pantry apply role mx -r postfix -d postfix.mail_type=master

We don't create a firewall rule for the "mx" role, because we're only sending mail and not receiving it.
(If we needed to receive mail, we'd have to open up port 25.)


Now that the "base", "web" and "mx" roles have been created, the next step is to create a node and
configure it for those roles and the Jitterbug recipe we created.

$ pantry create node jitterbug.example.com
$ pantry apply node jitterbug -R base -R web -R mx -r jitterbug

Note that once the node is created with the fully qualified name, you only need to specify a unique
substring when Pantry operates on a node name. (You could even just say "j" instead of "jitterbug",
since that's the only node.)

The "hostname" recipe requires us to set an attribute for the desired hostname, so we add that:

$ pantry apply node jitterbug -d set_fqdn=jitterbug.example.com

We also need to configure Jitterbug itself. For this tutorial, we'll just configure the addresses used for
email. We'll use some shell loops to avoid repetitive typing. Replace "jdoe@example.com" with your
own email address:

$ for i in pass failure; do
for j in cc from; do
pantry apply node jitterbug
-d jitterbug.on_${i}_${j}_email=jdoe@example.com;
done;
done

You can look at the resulting node file to be sure everything was set correctly:

$ pantry show node jitterbug
{
"set_fqdn" : "jitterbug.example.com",
"jitterbug" : {
"on_pass_from_email" : "jdoe@example.com",
"on_failure_cc_email" : "jdoe@example.com",
"on_failure_from_email" : "jdoe@example.com",
"on_pass_cc_email" : "jdoe@example.com"
},
"run_list" : [
"role[base]",
"role[web]",
"role[mx]",
"recipe[jitterbug]"
],
"name" : "jitterbug.example.com"
}

Once all the configuration is done, we want to check everything into git.

$ git add .
$ git commit -m "jitterbug node configured"


Step 5. Deploy and test

With all the configuration work done, deployment is easy:

$ pantry sync node jitterbug

Now comes the hard part... waiting for it to finish.

Unfortunately, we have to let it work for a while as each component is installed and configured.
Generally, this is the time to go do something else while you wait.3

Once it's done, switch to a browser and enter the hostname you configured. You should see an empty
dashboard like this:

Congratulations! Your Jitterbug server has been deployed. Now it's time to test it with a Perl module
repository from Github.

Browse to Github and go to one of your repositories with a Perl module in it. For example, I used the
repo for my own IO::Prompt::Tiny at https://github.com/dagolden/io-prompt-tiny. This simple module
is actually a good torture test for Jitterbug because it's built with Dist::Zilla, so Jitterbug has to install
the full Dist::Zilla dependency tree in order to be able to test commits to the repository.

Under the "Admin" tab, the Service Hooks menu option lets you entire a WebHook URL
"http://jitterbug.example.com/hook/" like this:

3 If deployment crashes out while building Perl, just try it again. I've seen some rare transient errors I've yet to diagnose,
but the nice thing about idempotent deployment is that you can just try again and see what happens.


After you "Update Settings", go back to the WebHook and click "Test Hook". Now switch back to
your Jitterbug dashboard and refresh the page.

You should see your new repository being tested:

Refresh your dashboard every so often until the pending build task is gone. The first time might take a
long time as prerequisites get installed.4 You should then be able to click into the repository link and
see a "PASS" or "FAIL" notice for that commit:

4 If it seems like it's taking too long, you can ssh into the box and tail the file deep in the /tmp/jitterbug directory to see
what's going on.


Now, every time you push a commit to that repository, Github will push a task to your Jitterbug server
and tests will be run.

We're done — we just deployed a Jitterbug continuous integration server using Chef and Pantry.

Happy cooking!

Cooking Perl with Chef: Real World Tutorial with Jitterbug

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie Cooking Perl with Chef: Real World Tutorial with Jitterbug

Ähnlich wie Cooking Perl with Chef: Real World Tutorial with Jitterbug (20)

Mehr von David Golden

Mehr von David Golden (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Cooking Perl with Chef: Real World Tutorial with Jitterbug