3 years ago, Meetic chose to rebuild it's backend architecture using microservices and an event driven strategy. As we where moving along our old legacy application, testing features became gradually a pain, especially when those features rely on multiple changes across multiple components. Whatever the number of application you manage, unit testing is easy, as well as functional testing on a microservice. A good gherkin framework and a set of docker container can do the job. The real challenge is set in end-to-end testing even more when a feature can involve up to 60 different components. To solve that issue, Meetic is building a Kubernetes strategy around testing. To do such a thing we need to : - Be able to generate a docker container for each pull-request on any component of the stack - Be able to create a full testing environment in the simplest way - Be able to launch automated test on this newly created environment - Have a clean-up process to destroy testing environment after tests To separate the various testing environment, we chose to use Kubernetes Namespaces each containing a variant of the Meetic stack. But when it comes to Kubernetes, managing multiple namespaces can be hard. Yaml configuration files need to be shared in a way that each people / automated job can access to them and modify them without impacting others. This is typically why Meetic chose to develop it's own tool to manage namespace through a cli tool, or a REST API on which we can plug a friendly UI. In this talk we will tell you the story of our CI/CD evolution to satisfy the need to create a docker container for each new pull request. And we will show you how to make end-to-end testing easier using Blackbeard, the tool we developed to handle the need to manage namespaces inspired by Helm.

1. Kubernetes @ Meetic
Building an end-to-end testing strategy on top
of Kubernetes in a world of microservices

2. Sébastien Lavallée
Senior software engineer @ Meetic

3. Sébastien Le Gall
@seb_legall
Tech Lead back-end @ Meetic

4. Started in 2001
•
150 people in IT teams
•
Dating leader in Europe
•
Active in 15 countries
•
Millions of Monthly Active Users

5. Event bus
Meetic Now
Microservices
BackOffices
Consumers
Exposition Layer

6. Testing
There are only two hard things in Computer Science: cache
invalidation and naming things.
[Phil Karlton]
There are only three hard things in Computer Science: cache
invalidation and naming things and testing a microservice
architecture
[Me]

7. Testing strategy

8. Testing strategy
• Unit tests
• Behavioral tests
• End-to-end tests

9. Testing – Registration
Exposition Layer
Event bus
Consumers
Micro-services
ProfileEmailing
Behavioral Testing
Behavioral Testing
Behavioral Testing
Behavioral Testing
End-to-end Testing
Unit TestingUnit Testing
Unit Testing

10. Testing strategy
On push :
Unit Tests
Behavioural Tests
Fast Feedback End-to-end tests
Deployment

11. Continuous Integration

12. Continuous Integration

13. In PHP, but all languages we use follow this concepts
Let's focus on the span of our microservice profile used in the
registration process.
Continuous Integration
Profile

14. Build Strategy / example

15. Container strategy
Exposition Layer
Event bus
Consumers
Unit testing
Behavioral
testing
End-to-end testing

16. Base Container
Add a Dockerfile to our Profile repository so we
can run this tools to test the code
FROM artifact/meetic/centos:6.9
RUN yum install
rh-php56
rh-php56-php-fpm
...
COPY config/cli/php.ini /etc/opt/rh/rh-php56/php.ini
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod a+x /entrypoint.sh && chown root:root /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/bin/bash"]
CentOS 6.9
Meetic/CentOS 6.9
Meetic/PHP 5.6

17. Base Container
Test Unit
Test Behavioral
Profile
Behavioral Testing
Unit Testing

18. docker-compose
services:
oracle:
image: artefact/meetic/oracle-main-data
ports:
- "1521"
broker:
build: docker-images/broker
ports:
- "10234"
depends_on:
- oracle
php:
build: docker-images/php
environment:
COMPOSER_PATH: bin/composer
depends_on:
- memcached
- wiremock
volumes:
- "./:/srv/app"
Profile
Add a docker-compose file to our repository
microservice Profile

19. docker-compose
Test Unit
Test Behavioral
Profile
Behavioral Testing
Unit Testing

20. Scaling the model

21. Target architecture
1 base container 1 container /
application
1 docker-compose /
application 1 interface
Makefile

22. Base container
Extract the base Dockerfile from repository
Change the entrypoint.sh to make it extendable
CentOS 6.9
Meetic/CentOS 6.9
Meetic/PHP 5.6
Meetic/PHP-CI 5.6
Microservice container
#!/bin/bash
...
chown -R $USERNAME:$GROUPNAME /docker-entrypoint-init.d
for f in /docker-entrypoint-init.d/*; do
case "$f" in
*.sh) echo "$0: running $f"; gosu $USERNAME "$f" ;;
*) echo "$0: ignoring $f" ;;
esac
echo
Done
...

23. Application Container
Put New and smaller embedded Dockerfile
dockerize.sh (an extension of our entrypoint.sh)
#!/bin/bash
# wait for dependencies to be ready
dockerize -timeout 10s -wait tcp://oracle:1521
dockerize -timeout 10s -wait tcp://broker:10234
CentOS 6.9
Meetic/CentOS 6.9
Meetic/PHP 5.6
Meetic/PHP-CI 5.6
Microservice container
FROM artifact/meetic/php56-ci
COPY config/tnsnames.ora /etc/
COPY docker-entrypoint-init.d/dockerize.sh /docker-entrypoint-init.d/dockerize.sh
RUN chmod a+x /docker-entrypoint-init.d/dockerize.sh

24. Application docker-compose
services:
wiremock:
build: docker-images/wiremock
memcached:
build: docker-
images/memcached
ports:
- "11211"
php:
build: docker-images/php
environment:
COMPOSER_PATH:
bin/composer
depends_on:
- memcached
- wiremock
volumes:
- "./:/srv/app"
services:
oracle:
image: artefact/meetic/oracle-main-
data
ports:
- "1521"
broker:
build: docker-images/broker
ports:
- "10234"
depends_on:
- oracle
php:
build: docker-images/php
environment:
COMPOSER_PATH: bin/composer
PLATFORM_ENV: test
depends_on:
- broker
volumes:
- "./:/srv/app"
docker-compose Emailing:docker-compose Profile :

25. Makefile
• Create a unified set of commands
across our microservices
• Stored in each repositories
• Used by generic builder

26. Makefile
Makefile emailing :
Makefile Profile :
...
cache-warmup:
bin/console cache:warmup
debug-router:
bin/console debug:router
test-behavioural: cache-warmup
test-behavioural: test-behat-ci
...
$ make test-behavioural
...
cache-warmup:
app/console cache:warmup
test-behavioural: test-behat-ci
...
Continuous Integration

27. A CI environment per microservice
One docker-compose per microservice
Some images are build on the go in order to be able to
override some configuration value during tests
One per microservice. Provide a unified set of commands

28. Scaling testing architecture
Test Unit
Test Behavioral
Test end-to-end
Exposition Layer
Event bus
Consumers
Micro-services
ProfileEmailing
Behavioral TestingBehavioral Testing
Behavioral Testing
End-to-end Testing

29. End-to-end testing strategy

30. Testing – the problem
• Multiple people need to test different thing at the same time
• Small technical increments are easy to test but not revelant for product
owners
• User stories may need multiple microservice to be tested
• When coming to end-to-end testing, docker-compose reach its limitation

31. Testing at scale
End-to-end testing at Meetics means :
• Running ~30 microservices
• Running ~30 consumers
• Running ~6 different database/search engine technology
• Running kafka

32. Using VMs and docker-compose?
First thing we though of was using VMs and running services inside
docker containers linked together using docker-compose.

33. Using VMs and docker-compose?
But…
• Running multiple docker-compose on the same VM cause conflict
on port usage
• There are 150 people in the IT teams. Does it mean that we should
be prepared to run 150 VM? (One per user)
• Automated testing on VMs run by the CI are hard to handle
• Docker-compose does not handle well domain name. (Think about
kubernetes’ ingress like feature)

34. Introducing Kubernetes
• Kubernetes handle port /name conflict with namespaces
• Ingress is great to handle domain name / url associated to services
• Creating and deleting a namespace is easy
Let’s go with Kubernetes !

35. Introducing Kubernetes
Test use case :
• Manual tests done by Product Owners
 Need a user friendly interface
• Manuel tests done by developers
 Could be done using a CLI tool but a user friendly interface
could be nice as well
• Automated tests done by a scripts / the CI
 Must be done using a CLI tool.

36. Introducing Kubernetes
User / CI workflow :
• Create a new namespace
• Apply default configuration (Means deploying all the containers
using the latest version of each, the one used in production)
• Change the container tag for services concerned by the update
(related to the feature)
• Launch test
• Trash the namespace

37. Building containers for run

38. Docker Images (w/ code)
Kubernetes need Docker images
Objectives :
• Create docker images
• Push them in a place kubernetes can retrieve them

39. Continuous integration for containers

40. Kubernetes Dockerfile
Dockerfile specifies an image with microservice code
Dockerfile

41. Kubernetes Dockerfile
FROM artifact/meetic/php56
ARG version=master
ARG stage=release
ARG apiName=
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod a+x /entrypoint.sh && chown root:root /entrypoint.sh
COPY config/fpm/php-fpm.d/* /etc/opt/rh/rh-php56/php-fpm.d/
COPY config/tnsnames.ora /etc/
RUN wget "https://artifact/php-$stage/$apiName/$version.tar.gz" &&
tar zxvf "$version.tar.gz"
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/opt/rh/rh-php56/root/usr/sbin/php-fpm", "-F"]
CentOS 6.9
Meetic/CentOS 6.9
Meetic/PHP 5.6
Microservice container
PHP Dockerfile not versioned in repositories

42. The Data
A lot of data
We can't just clone the databases in our testing environment
Our data is stored across multiple technology
Two constraints :
• We need to create a sub-set of users that is coherent
• We need to anonymize the data

43. Database Images
Need Data coherence across all our storage instances
Extract a coherent subset of data.
- same ratio H/M
- same ratio of search
- same ratio of active/inactive
…
With the ID's, extract from all database the data for those members

44. Database Extract
Extracted
ID LIST
Anonymization
Anonymization
Anonymization
Anonymization
User List

45. Docker images ready

46. Managing testing environments

47. Helm
Helm is a package manager for Kubernetes.
• Creating packages (called charts)
• Sharing packages on a platform
• Deploying shared packages easily on kubernetes

48. Helm - Charts
“A chart is a collection of files that describe a related set of Kubernetes
resources.”
“Charts are created as files laid out in a particular directory tree, then they
can be packaged into versioned archives to be deployed.”
+
Templates Values Package

49. Helm - Charts

50. Helm – the problem
Helm is great but the use case does not really match our needs :
• Helm is a CLI tools (only)
• Helm comes with the features of a package manager that we don’t
care about
• Helm does not manage namespaces. It only handle package
deployment.

51. Testing env – our goal
api-profile version 0.1.2 api-search
version 1.3.2
Latest Version Latest Version
Template and default
values

52. Introducing Blackbeard
Blackbeard is a user friendly tool that handle package
configuration and deployment across multiple
kubernetes namespace.
+
Templates
Inventory
Inventory
Playbook

53. Blackbeard features
• Create namespace
• Apply configuration (deployments, services, etc.)
• Get useful informations about namespace state (Pods status,
exposed service via NodePort or Ingress)
• Delete namespace

54. Playbooks
Templates
Default values
Inventory for slegall
Config for slegall

55. Playbook - templates

56. Playbook - Inventory

57. Blackbeard usage
+

58. Working with Blackbeard
$ blackbeard create –n test # Create a namespace and default configuration
$ vi inventories/test_inventory.json # Update the values to match needs
$ blackbeard apply –n test # Apply the change to the namespace
$ blackbeard get services # Show exposed services
$ blackbeard delete –n test # Cleanup files and namespace

59. Blackbeard under the hood
• Written in the Go programming language
• Interact with the Kubernetes API using the go client
• Expose a collection of CLI commands using the Cobra framework
• Expose a REST API trough the Gin framework
• Rely on the .kube/config configuration file

60. Blackbeard under the hood

61. Conclusion
Inheritance in Docker is a game changer
when working with microservices and
different testing strategy
Give your users a base.
Let them override the configuration.
Don’t treat all the use cases with the same
patterns.

62. Conclusion
Re-create an entire stack in kubernetes
remains hard. You cannot just apply your
prod configuration in a container.
Some useful tricks stay undocumented.
The kubernetes source code is very hard to
understand for outsiders.
There are plenty of tools and stories around
kubernetes . But the reality is far from a
fairy tail.