Learn why you should put your blackbox (or system/integration) tests into Docker Containers. Brief (remedial) overview of Docker for software testers who don't know docker, and only need to know the basics to wrap their regression tests inside of a container.

1. Containerize
your Blackbox
tests

2. Slides are Available on SlideShare
2

3. About Me
Software testing since 2005
● Desktop Applications
● Mobile Applications
● (Mobile) Device Drivers
● 802.11 B/G/WMM/WMM-PS
● Networking Devices (Routers/Switches)
● Load Balancing Appliances
● Software Defined Radios
● Web Applications
● Security Testing
3

4. The
Fine Print
Everything in this slide deck is either my own opinion coming from my own
personal experiences, or maybe Stack Overflow. In no way should anything from
this presentation be taken as how things are actually done at my current, former, or
future employers. All information is provided on an “AS IS” basis, without
warranties, and you the viewer accept all risks associated with the material herein.
4

5. Containers will not solve all of
your problems
› But they will solve some some of them
5

6. Testing Triangle
6

7. Basic Regression Topology
7
Regression
Node
Device/System
Under Test

8. We can’t update package X
because we don’t know if
package Y will still work.
Do our tests work with the
updated package X?
8

9. So, we updated package X, and
it doesn’t work anymore.
Also, we don’t actually know
what version package X was at
before...
9

10. Manually Created Test Executors
› Time consuming to create
› Often out of date.
› Apply security patches? It’s in the
lab, who cares?
› We should back it up but we don’t...
› We backed it up when we created it,
but haven’t updated the backup in years.
10

11. Solutions
› Robust backup strategy
› Move from physical systems to VM’s
› Robust snapshot strategy.
› Use Ansible/Puppet/Chef
› Containers
11

12. A new team member is starting,
how do we quickly get their
environment setup?
12

13. A Developer just did a large
refactor, the unit tests pass, but
they want to run the regression
suite before they merge their
branch...
13

14. Solutions
› Build a new System
› Clone a VM
› Use Ansible/Puppet/Chef
› Containers
14

15. Developer
Writes production code
Cares (hopefully) about
memory usage
Uses the best toolset for the
problem at hand, or the
toolset the rest of the team
is using
Developers and Testers are Different
Test Automator
Writes code to test other
code/systems
Not generally concerned about
efficiency/memory usage
(unless it becomes a problem)
Uses the best toolset to test
the system/device under test,
or the toolset the rest of the
team is using
15

16. Automated system tests
don’t often find problems.
Writing automated system
tests can find problems.
Typical Workflow
Running automation
afterwards usually makes
sure things don’t break after
being fixed.
Problems found are often
due to interactions between
components, and are
relatively straightforward.
16

17. Atypical Flow
1.
Developer writes new
code, commits it, plays a
game of foosball while CI
“works”
2.
Regression test suite runs,
and fails. DevTest
investigates to make sure
it isn’t an issue with the
automation.
3.
DevTest reviews logs,
determines its a legitimate
failure, tells Developer
about the problem.
4.
Developer can’t reproduce.
5.
Devtest can’t either
outside of the regression
suite, its some interaction
between a previous test
and the failing test.
6.
Developer is forced to
push changes and let CI
run to see if issues are
fixed. No one likes to see
the ❌next to their build.
17

18. Why Can’t Developers Just Run the Tests locally?
› Different environments that may not
play well together.
› Time consuming to spin up another
resource
› Expensive to have a dedicated test
environment for developers
› One more system to have to keep in
sync with the others.
› Tribal knowledge needed to run tests
18

19. Docker
Containers
Package your blackbox tests so
that they will run anywhere, and
by anyone.
No special tools, no secret
commands.
19

20. Why Containers?
› Images are Immutable.
› Images contain all dependencies
› Images are easy to share
› Dockerhub or Private Registry
› Easier then Ansible/Puppet/Chef
› Easy to integrate into CI/CD Pipelines
› Agents only need Docker installed
20

21. VM
● Clone an existing
devtest VM
● Resync from last clone
● Running VM’s take
resources
● Developers have to
know how to run the
tests
Developers running Regression Test Suites
Container
● Docker pull to get up to
date image
● Docker run to run tests
● Resources only being used
when tests are running
● Developers may still need
to know arguments for
how to run the tests.
21

22. Remedial Containers 0001
22

23. Remedial Containers 0001
Dockerfile
Text file that defines
how to create a
docker image.
Starts with a base
image
Add your secret
sauce
Entrypoint
Command (or script)
that gets run by
default when the
container is ran.
For containerized
blackbox tests, this
should be the
command needed to
run the tests.
Image
Contains the read-
only layers of the
container. When you
run a container, a
writeable layer is
added, which makes
the image itself
immutable.
23

24. >cat regression_test.py
#!/usr/bin/env python3
print(“Running Tests...”)
Containerize your blackbox tests
24

25. >ls
regression_test.py
Dockerfile
Containerize your blackbox tests
25

26. >cat Dockerfile
FROM python:3.6.8-slim-stretch
WORKDIR /tests
COPY . .
ENTRYPOINT python3 regression_test.py
Containerize your blackbox tests
26

27. >docker build -t regression -f Dockerfile .
Sending build context to Docker daemon 81.92kB
Step 1/3 : FROM python:3.6.9-slim-stretch
---> d4a811fcaf6a
Step 2/3 : COPY . .
---> 7573353bcfb1
Step 3/3 : ENTRYPOINT python3 regression_test.py
---> Running in 8c7563d2c6e6
Removing intermediate container 8c7563d2c6e6
---> 082942a6359c
Successfully built 082942a6359c
Successfully tagged regression:latest
Containerize your blackbox tests
27

28. >docker run regression
Running tests…
>docker run --entrypoint=pwd
/tests
>docker run --entrypoint=ls regression
Dockerfile
regression_test.py
Containerize your blackbox tests
28

29. #Start a container with an interactive bash shell
>docker run -it --entrypoint=/bin/bash regression
Debugging your Containerized
blackbox tests
29

30. Tag and Push docker
images to a registry to
allow others to be
able to use the image
without rebuilding
Docker Registry
Host your Own
Dockerhub
AWS
GCP
Azure
30

31. >docker build -t kjbeeman/regression:latest -f Dockerfile .
OR
>docker tag regression:latest kjbeeman/regression:latest
>docker push kjbeeman/regression:latest
Containerize your blackbox tests
31

32. What does this really give us?
32

33. Developers can run the tests by just
typing:
docker run kjbeeman/regression args
33

34. Regression environments only need
docker and Pipeline scripts just need...
docker run kjbeeman/regression args
34

35. >cat Dockerfile
FROM python:3.6.8-slim-stretch
COPY . .
ENTRYPOINT python3 regression_test.py
Upgrading Dependencies
35
>cat Dockerfile
FROM python:3.6.9-slim-stretch
COPY . .
ENTRYPOINT python3 regression_test.py

36. >cat Dockerfile
FROM python:3.6.8-slim-stretch
RUN apt-get update
&& apt-get install -y gparted=0.16.1-1
COPY . .
RUN pipenv install --system
ENTRYPOINT python3 hello_world.py
Upgrading Dependencies
36

37. FROM python:3.6.8-slim-stretch
#Install Nmap
RUN apt-get update
&& apt-get install -y libpcap-dev autoconf git wget build-essential checkinstall libpcre3-dev libssl-dev
&& git clone https://github.com/nmap/nmap.git
&& cd nmap
&& ./configure
&& make
&& make install
WORKDIR /test
COPY . .
ENTRYPOINT python3 hello_world.py
Upgrading Dependencies
37

38. Dependencies Live with the Code
The regression tests prior to
Product Version 1.0.4 require
OpenSSL 1.0.X, but OpenSSL
1.1.x is required starting with
version 1.1.0
If you need to go back and
run regressions, you can pull
the specific git commit and
rebuild the image.
Much easier than
maintaining separate
regression environments for
the two product versions.
38

39. Containers make it easier to
what you need to do without
worrying about managing the
infrastructure that is running it.
39

40. Review
Less Frustration
Anyone can run the tests
without having to worry
about dependencies
Immutable
A working image is a
working image, you can’t
break it without creating a
new image.
Change is Safe
Dockerfile should be
version controlled. Easy to
make changes and revert if
needed.
Resources
Other than disk space,
containers use no
resources when they aren’t
being used
Scale
Easier to expand
regression environments.
Buzzwords
Container is the hip word
right now. Even Financial
Analyst on CNBC are
talking about it.
40

41. Next
Stop?
Serverless...
Maybe not today...
41

42. THANKS!
Any questions?
You can find me at:
kevinbeeman@gmail.com
Linked In: https://www.linkedin.com/in/kevinbeeman
GitHub: https://github.com/kjbeeman/regression
42

43. Credits
Special thanks to all the people who made
and released these awesome resources for
free:
› Presentation template by SlidesCarnival
› Photographs by Startupstockphotos
43