Building a Perl5 smoketest environment in Docker using CPAN::Reporter::Smoker. Includes an overview of "smoke testing", shell commands to contstruct a hybrid environment with underlying O/S image and data volumes for /opt, /var/lib/CPAN. This allows maintaining the Perly smoke environemnt without having to rebuild it.

1. Smoke & Mirrors:
Containerizing Perl Testing
Steven Lembark
Workhorse Computing
lembark@wrkhors.com

2. What is “Smoking”?
● High-level sanity check, not detailed analysis.
Does it catch fire when we power it on?
● CPAN runs tests as part of installation.
We are unusual that way.
● CPAN::Reporter reports results of tests.
● CPAN::Reporter::Smoker builds with reporting.

3. Why do you smoke?
● Verify personal modules.
● Validate project platform.
● Addict (CPAN smoker).

4. What do you smoke?
● Whatever is in your sandbox.
● A project and all of its supporting code.
● All of CPAN, or the RECENTS file.

5. How do you smoke?
● “make -wk -C myproject all test” in your sandbox.
● Nightly build, git hooks, Travis CI.
● CPAN or CPANM smoke testers.

6. Risks of Smoking
● Pull in arbitrary code.
Maybe broken.
Possibly malicious.
More likely just stupid
system( rm -rf / *.foobar );

7. Second-hand Smoke
● Permanent side effects.
● Left over test files.
● Wasted disk space.
● Bandwidth.
● Still possible with personal tests:
Dependent modules.

8. Protect Yourself
● Install into a dedicated sandbox.
● Disposable environment.

9. One approach: Virtual Machines
● Heavy footprint.
● Fully configure a running machine to run a test.
● Inefficient use of hardware.

10. Lightweigit: “chroot”
● Filesystem isolation only.
● Tests can interfere with network, processes.
● Files pollute permenant storage.
● No resource limits.

11. Another approach: Process isolation.
● Keep processes from interfering with one another.
● Lightweight: Share logger, daemons, filesytems...
● More efficient use of CPU.
● Less work to set up.
● Not always easy to do right.
Jails & Zones

12. Better way: lxc
● Approach via IBM.
Mainframe LPAR ported to linux.
● Uses process groups.
● Excellent control.
● One problem: People used them like VM's.
● Two problem: lxc has hundreds of switches.

13. lxc for the Masses
● Docker: 80/20 of lxc.
Medium isolation.
● Generate an “image”.
Read-only stack of AUFS mounts.
● Run a “container”.
Mountpoint + resources + isolation.
● Decent manglement tools.

14. AUFS with lxc
● Images are layered.
Read-only.
● "Container layer"
updated.
discarded.
Good and bad:
No permenant updates.

15. Not so nice
● Images look good for smoking:
Read-only underneath: minimal harm.
Re-cycle O/S layer with Perly build.
Allows mix'n match testing.
● Catch:
No fixes or upgrades to smokebox.
Changes require complete rebuild.

16. One fix: Volumes
● Described as “data volumes”.
● Can store anything.
● Like Perl, build directory, CPAN mirror.
● Mountpoints and symlinks.
Dir's or files from O/S into container.
● Mount RO or RW.
Maintain Perl & Smoker.

17. Hybrid approach
● O/S is an image.
● Perl, CPAN on a volume.
O/S tools build and maintain perl.
perl maintains /var/lib/CPAN

18. perly volumes
● Tempting: perlbrew dir's as volumes.
Problems with storage in /home.
Hard to share across users.
Storage depends on user account.
Multiple accounts duplicate space.
● Alternate: Let Docker manage the volumes
Simpler sharing.
/var/lib/docker [hopefully] has more space.
Manage via docker.

19. Perl Testers Wiki
● Good instructions for CPAN::Reporter::Smoker.
● Start with an O/S image and get a working smoker.
● CPANM has its own version.
"minismokebox"

20. Grab a distro
● Docker Hub has hundreds of distro images.
● I'll use Gentoo here: it includes build environment.
No need to install gcc, make...
“stage3” is the starting point.

21. Installing Stage3
●
docker import $tarball into gentoo-stage3:YYYYMMDD.
● Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.
root's .exrc .bashrc
● Used to bulid perl, run smoke.

22. Building perl, smoker
● Volumes for /opt, /var/tmp & script files.
Standard paths simplify build scripts.
● “docker run” with command of build script.
build perl.
install & run minicpan.
configure reporting & smoker.
CPAN update to test reporting.

23. Aside: --tmpfs
● docker run –tmpfs has a problem:
The volume has execution turned off.
Fine for data.
Useless for build.
● Avoid the issue with a tmpfs volume.
Volume must be private to build.

24. Creating a tmpfs
● Private tmp
volume via PID.
● tmpfs usually
cleaned up
automatically on
the way out.
● Use volume rm
to be sure.
tmp=”var-tmp-$BASHPID”;
$docker volume create
--driver=local
--opt type=tmpfs
--opt device=tmpfs
--opt o=size=320M
$tmp
&&
docker run
...
-v $tmp:/var/tmp
...
;
# may fail due to non-existant volume.
docker volume rm $tmp;

25. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz:/var/tmp/perl.tar.gz‑
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

26. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

27. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

28. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

29. Start the ball rolling
● Assemble a container.
● Dispatch the build script.
● “-t” avoids issues with
core test that uses
column/row counts.
● “-a” gets output from
container into log.
$docker volume create
--driver=local
--opt type=tmpfs
--opt device=tmpfs
--opt o=size=320M
$tmp &&
$docker volume create
--driver=local
$perl &&
$docker run --rm
-t
--name=”build-$perl”
-a stdout
-a stderr
"${vols[@]}"
$image
/var/tmp/build-perl
2>&1 | tee build.out;
docker volume rm $tmp;

30. Building Perl
● The fun starts in /var/tmp/build-perl
● Mostly here-scripts.
● Install & configure smoketest engine.
● Includes common non-smoke modules.

31. Building Perl
HOME='/var/lib/CPAN';
(
cd /var/tmp;
gzip -dc < perl.tar.gz | tar xf -;
cd perl*;
jobs=$(grep '^processor' /proc/cpuinfo | wc -l);
export TEST_JOBS=$jobs;
./Configure
-de
-Dprefix=/opt/perl5
-Dman1dir='none'
-Dman3dir='none'
-Doptimize='-O2 -pipe -march=native'
&&
/usr/bin/make -j$jobs all test_harness install;
) ||
exit -1;
● Generic
“perl.tar.gz” via
docker volume
mapping.
● /opt/perl5 is in
external
volume.
● Don't knead
man pages.

32. Building Perl
HOME='/var/lib/CPAN';
● /home is in the
O/S image!
Changes in
container layer
are discarded!!
● Persistent files
go in
/var/lib/CPAN or
/opt.

33. Building Perl
TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness
install;
Files=2410, Tests=851044, 223 wallclock secs
(172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)

34. Aside: Testing perl builds
● Use tmpfs for /opt instead of disk volume.
Fast, disposable storage for validating your build scripts.
● git clone <whatever> /scratch;
cd /scratch/whatever;
● docker run … -v /scratch/whatever:/var/tmp/whatever … ;
● build script does cd /var/tmp/whatever; make all test install;

35. Testing bleeding edge perl
● Replace tar xvf with volume:
-v /gitclone/perl-5.25.X:/var/tmp/perl
● Rest of it stays the same.
Configure and make run quickly.
Minimal overhead if core modules don't change.
Lots “up to date” messages from CPAN.

36. Minor setups
● “h2ph”
validates
$PATH.
● Pick your
remote
server.
export PATH=/opt/perl5/bin:$PATH;
h2ph -r -l /usr/include || exit -1;
# at this point install seems usable
lib=$(ls -d /opt/perl5/lib/5* | tail -1);
site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);
cpan='/var/lib/CPAN';
build='/var/tmp/CPAN';
remote='http://mirror.uic.edu/CPAN/';
local="$cpan/sources/";

37. CPAN uses /var/lib, /var/tmp
● Push CPAN config onto
stable storage.
● CPAN config is specific
to perl version.
perl -MCPAN -e 'shell'<<CPAN;
yes
o conf cpan_home $cpan
o conf histfile $cpan/histfile
o conf prefs_dir $cpan/prefs
o conf build_dir $build
o conf keep_source_where $local
o conf auto_commit yes
o conf build_dir_reuse yes
o conf check_sigs yes
o conf inhibit_startup_message yes
o conf halt_on_failure no
o conf build_cache 1
o conf index_expire 1
o conf commit
CPAN

38. TLC
● Update CPAN.
● Deal with modules
that require user
input, special
choices.
(
CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;
perl -MCPAN -e 'install CPAN';
perl -MCPAN -e shell <<CPAN;
install Module::Signature
2
CPAN
AUTOMATED_TESTING=1; # avoid interactive prompt
perl -MCPAN -e 'shell <<CPAN;
install Term::Readline::Perl
CPAN
)

39. ● Common.
● Smoke modules.
● Iterate them with
${modules[@]}.
modules=(
YAML::XS
JSON::XS
CPAN::SQLite
IO::Socket::SSL
Net::SSL
Net::SSLeay
HTTP::Date
Test::Most
Metabase::Resource
Module::Version
Module::Version Bundle::CPAN
Log::Log4perl
Bundle::CPAN
CPAN::Mini
CPAN::Reporter
CPAN::Testers
Test::RequiresInternet
WWW::RobotRules
LWP::UserAgent
WWW::Mechanize
CPAN::Reporter::Smoker
Bundle::CPAN::Reporter::Smoker::Tests
Bundle::CPANReporter2
App::SmokeBox::Mini
App::cpanminus::reporter
);
Other modules

40. Generate ID file for Reporter
● ~ is /var/lib/CPAN
Not /home!
(
mkdir ~/.cpantesters;
cd ~/.cpantesters;
metabase-profile <<END;
Steven Lembark
lembark@wrkhors.com
password/secret
END
chmod 0400 metabase_id.json;
)

41. Configure CPAN::Reporter
# Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes
# Would you like me configure CPAN::Reporter now? [yes] <enter>
# email_from? [] example@example.com
# edit_report? [default:ask/no pass/na:no] <enter>
# send_report? [default:ask/yes pass/na:yes] <enter>
# transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] <enter>
# Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] <enter>
perl -MCPAN -e shell <<CPAN;
o conf init test_report
yes
yes
lembark@wrkhors.com
no
yes
CPAN

42. Voyeurism
● Fixing the build uses “docker exec”.
● Attach to a running container.
● Does not require ssh.
● Allows fixing, re-running the build.

43. Voyeurism
● Fixing the build uses “docker exec”.
● Attach to a running container.
● Does not require ssh.
● Allows fixing, re-running the build.
● Or just watching it:
docker exec -it build-perl-5.24.1 'top';

44. Access a running container
● Say you get something like:
t/07_plugins.t ........... ok
t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the
YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
/var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
at t/08_since_epoch.t line 12.
BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
# Looks like your test exited with 2 before it could output anything.
t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
Failed 11/11 subtests

45. Access a running container
● Say you get something like:
● Q: How do you fix it?
t/07_plugins.t ........... ok
t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the
YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
/var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
at t/08_since_epoch.t line 12.
BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
# Looks like your test exited with 2 before it could output anything.
t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
Failed 11/11 subtests

46. Access a running container
● Exec into the container.
● Use CPAN to install the module.
● Which leaves you with:
docker exec -it build-perl-5.24.1 bash –-login;

47. Aside: Container ID
●
docker ps shows what is running.
Includes the container id & name:
$ docker ps;
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..."
28 minutes ago Up 28 minutes build-perl-5.24.1
# same result:
$ docker exec -it 493d9f285faf 'bash login';‑‑
root@493d9f285faf ~ #
$ docker exec -it build-perl-5.4.21 'bash login';‑‑
root@493d9f285faf ~ #

48. Through the looking glass
● Inside the container you are root.
Kernel's proc struct has unchanged EUID.
● Limited view of the system.
Only container proc's in “ps” or “top”.
Nice on a crowded server.
Runlevel restrictions are ineffective!
● Run “perl -MCPAN” or “make install”.
Helpful if a non-essential test fails.

49. Through the Looking Glass
● Outside docker you have a specific PID & EUID.
● Inside docker the initial process has EUID 0 and PID 1.
● EUID 0 is manageable.
● PID 1 requires that the smoketest proc reap children.
Otherwise you end up with defunct proc's.
Will eventually eat all proc slots with long-running container.
One fix is Docker::Reaper.

50. Remember:
● Changes to the image go to a separate AUFS layer.
● They are discarded.
● Fixing files in /etc/ will rescue this build, not the next one.
● Changes to /opt & /var/lib/CPAN are persistent.

51. TMTOWTDI: minicpan
● Outside of docker.
Maintain CPAN cache in /var/lib.
● Inside docker.
Use /var/lib.

52. Running repeated jobs
● Docker can automatically restart running jobs.
● docker run -d restart=always ...‑‑
Downside: Lots of proc's started up at once.
Issues with tmpfs on system restarts.
● --restart=unless-stopped
Allows stopping volumes on the way down.
Start singly on way back up.

53. Smoketest lifecyle
● Run minicpan once on /var/lib/CPAN.
Global cache of modules.
● Run smoker with O/S images & perly vol's.
● This will only test *one* kernel!
● Use VM's running docker for multi-kernel smoke tests.

54. Zombie apocolypse
● Container startup command is process #1.
● When it dies the container exits.
● It has to reap all children.
Not a big issue for small, single-purpose containers.
Big problem for smoketests.

55. Fork & Reap
● Parent propagates signals.
● Exits with $? from child process.
● Loop reaps zombies.
Without this process slots will be
exhausted.
If that happens: log on as SU
and stop, rm the container.
for(;;)
{
my $pid = wait;
$pid != -1
or die "Lost child pid ($child)";
$pid != $child
and next;
return $?
}

56. Smoking
● Volumes for
perl
CPAN
/var/tmp
● Run smoker.
Never exits.
vols=(
"-v $perl:/opt:ro"
"-v $cpan:$cpan"
"-v $tmp:/var/tmp"
);
(
$docker volume create
...
$tmp
&&
exec $run
-t
--name="$cname"
${vols[@]}
$image
"/var/lib/CPAN/.cpantesters/docker-smoker"
) 2>&1 |
tee $log;

57. Propsed change
● Smoker processes recents and exit.
Simplifies iterating multiple perl verisons.
● Goal:
Product of images, perl vol's.
Automate bleed test.

58. Summary
● Smoking can be hazardous.
● LXC provides lightweight solution.
● Mix O/S image, perl volume.
O/S is fixed.
Volume updated to maintain smoker.
● git@github.com:lembark/docker-smoker.git
Mostly shell kwikhaks.
Working to make it cleaner, perly.