Talk with @clementd. The great war is ended, systemd won over initd (the old), upstart and the other, and now it's the mainstream choice. But Why? What is the interest? How difficult is it to do a configuration on the systemd world? How does it work? Can I replace CRON on it?

1. What is systemd?
Why use it?
How does it work?
By @waxzce & @clementd
At Devoxx France 2017

2. Quentin ADAM - @waxzce

3. Clément Delafargue
@divarvel

7. Process PID 1 on linux kernel

8. Manage process lifecycle
Start/Stop
Reap Orphan processes
Launch Order
Monitoring
….

9. Scheduling tasks

10. For years we’ve used initd

11. Initd configuration is bash
😞😞

13. Example: gitlab initd bash script (~500 lines)
#! /bin/sh
# GITLAB
# Maintainer: @randx
# Authors: rovanion.luckey@gmail.com, @randx
### BEGIN INIT INFO
# Provides: gitlab
# Required-Start: $local_fs $remote_fs $network $syslog redis-server
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: GitLab git repository management
# Description: GitLab git repository management
# chkconfig: - 85 14
### END INIT INFO
###
# DO NOT EDIT THIS FILE!
# This file will be overwritten on update.
# Instead add/change your variables in /etc/default/gitlab
# An example defaults file can be found in lib/support/init.d/gitlab.default.example
###
### Environment variables
RAILS_ENV="production"
# Script variable names should be lower-case not to conflict with
# internal /bin/sh variables such as PATH, EDITOR or SHELL.
app_user="git"
app_root="/home/$app_user/gitlab"
pid_path="$app_root/tmp/pids"
socket_path="$app_root/tmp/sockets"
rails_socket="$socket_path/gitlab.socket"
web_server_pid_path="$pid_path/unicorn.pid"

14. Example: gitlab initd bash script (~500 lines)
sidekiq_pid_path="$pid_path/sidekiq.pid"
mail_room_enabled=false
mail_room_pid_path="$pid_path/mail_room.pid"
gitlab_workhorse_dir=$(cd $app_root/../gitlab-workhorse 2> /dev/null && pwd)
gitlab_workhorse_pid_path="$pid_path/gitlab-workhorse.pid"
gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr $socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket $rails_socket -
documentRoot $app_root/public"
gitlab_workhorse_log="$app_root/log/gitlab-workhorse.log"
gitlab_pages_enabled=false
gitlab_pages_dir=$(cd $app_root/../gitlab-pages 2> /dev/null && pwd)
gitlab_pages_pid_path="$pid_path/gitlab-pages.pid"
gitlab_pages_options="-pages-domain example.com -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090"
gitlab_pages_log="$app_root/log/gitlab-pages.log"
shell_path="/bin/bash"
gitaly_enabled=false
gitaly_dir=$(cd $app_root/../gitaly 2> /dev/null && pwd)
gitaly_pid_path="$pid_path/gitaly.pid"
gitaly_log="$app_root/log/gitaly.log"
# Read configuration variable file if it is present
test -f /etc/default/gitlab && . /etc/default/gitlab
# Switch to the app_user if it is not he/she who is running the script.
if [ `whoami` != "$app_user" ]; then
eval su - "$app_user" -c $(echo ")$shell_path -l -c '$0 "$@"'$(echo "); exit;
fi
# Switch to the gitlab path, exit on failure.
if ! cd "$app_root" ; then
echo "Failed to cd into $app_root, exiting!"; exit 1
fi
### Init Script functions
## Gets the pids from the files
check_pids(){

15. Example: gitlab initd bash script (~500 lines)
if ! mkdir -p "$pid_path"; then
echo "Could not create the path $pid_path needed to store the pids."
exit 1
fi
# If there exists a file which should hold the value of the Unicorn pid: read it.
if [ -f "$web_server_pid_path" ]; then
wpid=$(cat "$web_server_pid_path")
else
wpid=0
fi
if [ -f "$sidekiq_pid_path" ]; then
spid=$(cat "$sidekiq_pid_path")
else
spid=0
fi
if [ -f "$gitlab_workhorse_pid_path" ]; then
hpid=$(cat "$gitlab_workhorse_pid_path")
else
hpid=0
fi
if [ "$mail_room_enabled" = true ]; then
if [ -f "$mail_room_pid_path" ]; then
mpid=$(cat "$mail_room_pid_path")
else
mpid=0
fi
fi
if [ "$gitlab_pages_enabled" = true ]; then
if [ -f "$gitlab_pages_pid_path" ]; then
gppid=$(cat "$gitlab_pages_pid_path")
else
gppid=0
fi
fi
if [ "$gitaly_enabled" = true ]; then
if [ -f "$gitaly_pid_path" ]; then
gapid=$(cat "$gitaly_pid_path")

16. Example: gitlab initd bash script (~500 lines)
else
gapid=0
fi
fi
}
## Called when we have started the two processes and are waiting for their pid files.
wait_for_pids(){
# We are sleeping a bit here mostly because sidekiq is slow at writing its pid
i=0;
while [ ! -f $web_server_pid_path ] || [ ! -f $sidekiq_pid_path ] || [ ! -f $gitlab_workhorse_pid_path ] || { [ "$mail_room_enabled" = true ] && [ ! -f $mail_room_pid_path ]; } ||
{ [ "$gitlab_pages_enabled" = true ] && [ ! -f $gitlab_pages_pid_path ]; } || { [ "$gitaly_enabled" = true ] && [ ! -f $gitaly_pid_path ]; }; do
sleep 0.1;
i=$((i+1))
if [ $((i%10)) = 0 ]; then
echo -n "."
elif [ $((i)) = 301 ]; then
echo "Waited 30s for the processes to write their pids, something probably went wrong."
exit 1;
fi
done
echo
}
# We use the pids in so many parts of the script it makes sense to always check them.
# Only after start() is run should the pids change. Sidekiq sets its own pid.
check_pids
## Checks whether the different parts of the service are already running or not.
check_status(){
check_pids
# If the web server is running kill -0 $wpid returns true, or rather 0.
# Checks of *_status should only check for == 0 or != 0, never anything else.
if [ $wpid -ne 0 ]; then
kill -0 "$wpid" 2>/dev/null
web_status="$?"

17. Example: gitlab initd bash script (~500 lines)
else
web_status="-1"
fi
if [ $spid -ne 0 ]; then
kill -0 "$spid" 2>/dev/null
sidekiq_status="$?"
else
sidekiq_status="-1"
fi
if [ $hpid -ne 0 ]; then
kill -0 "$hpid" 2>/dev/null
gitlab_workhorse_status="$?"
else
gitlab_workhorse_status="-1"
fi
if [ "$mail_room_enabled" = true ]; then
if [ $mpid -ne 0 ]; then
kill -0 "$mpid" 2>/dev/null
mail_room_status="$?"
else
mail_room_status="-1"
fi
fi
if [ "$gitlab_pages_enabled" = true ]; then
if [ $gppid -ne 0 ]; then
kill -0 "$gppid" 2>/dev/null
gitlab_pages_status="$?"
else
gitlab_pages_status="-1"
fi
fi
if [ "$gitaly_enabled" = true ]; then
if [ $gapid -ne 0 ]; then
kill -0 "$gapid" 2>/dev/null
gitaly_status="$?"
else
gitaly_status="-1"

18. Example: gitlab initd bash script (~500 lines)
fi
fi
if [ $web_status = 0 ] && [ $sidekiq_status = 0 ] && [ $gitlab_workhorse_status = 0 ] && { [ "$mail_room_enabled" != true ] || [ $mail_room_status = 0 ]; } && { [
"$gitlab_pages_enabled" != true ] || [ $gitlab_pages_status = 0 ]; } && { [ "$gitaly_enabled" != true ] || [ $gitaly_status = 0 ]; }; then
gitlab_status=0
else
# http://refspecs.linuxbase.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
# code 3 means 'program is not running'
gitlab_status=3
fi
}
## Check for stale pids and remove them if necessary.
check_stale_pids(){
check_status
# If there is a pid it is something else than 0, the service is running if
# *_status is == 0.
if [ "$wpid" != "0" ] && [ "$web_status" != "0" ]; then
echo "Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran."
if ! rm "$web_server_pid_path"; then
echo "Unable to remove stale pid, exiting."
exit 1
fi
fi
if [ "$spid" != "0" ] && [ "$sidekiq_status" != "0" ]; then
echo "Removing stale Sidekiq job dispatcher pid. This is most likely caused by Sidekiq crashing the last time it ran."
if ! rm "$sidekiq_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi
fi
if [ "$hpid" != "0" ] && [ "$gitlab_workhorse_status" != "0" ]; then
echo "Removing stale GitLab Workhorse pid. This is most likely caused by GitLab Workhorse crashing the last time it ran."
if ! rm "$gitlab_workhorse_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi

19. Example: gitlab initd bash script (~500 lines)
fi
if [ "$mail_room_enabled" = true ] && [ "$mpid" != "0" ] && [ "$mail_room_status" != "0" ]; then
echo "Removing stale MailRoom job dispatcher pid. This is most likely caused by MailRoom crashing the last time it ran."
if ! rm "$mail_room_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi
fi
if [ "$gitlab_pages_enabled" = true ] && [ "$gppid" != "0" ] && [ "$gitlab_pages_status" != "0" ]; then
echo "Removing stale GitLab Pages job dispatcher pid. This is most likely caused by GitLab Pages crashing the last time it ran."
if ! rm "$gitlab_pages_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi
fi
if [ "$gitaly_enabled" = true ] && [ "$gapid" != "0" ] && [ "$gitaly_status" != "0" ]; then
echo "Removing stale Gitaly pid. This is most likely caused by Gitaly crashing the last time it ran."
if ! rm "$gitaly_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi
fi
}
## If no parts of the service is running, bail out.
exit_if_not_running(){
check_stale_pids
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; } &&
{ [ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
echo "GitLab is not running."
exit
fi
}
## Starts Unicorn and Sidekiq if they're not running.
start_gitlab() {
check_stale_pids

20. Example: gitlab initd bash script (~500 lines)
if [ "$web_status" != "0" ]; then
echo "Starting GitLab Unicorn"
fi
if [ "$sidekiq_status" != "0" ]; then
echo "Starting GitLab Sidekiq"
fi
if [ "$gitlab_workhorse_status" != "0" ]; then
echo "Starting GitLab Workhorse"
fi
if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" != "0" ]; then
echo "Starting GitLab MailRoom"
fi
if [ "$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" != "0" ]; then
echo "Starting GitLab Pages"
fi
if [ "$gitaly_enabled" = true ] && [ "$gitaly_status" != "0" ]; then
echo "Starting Gitaly"
fi
# Then check if the service is running. If it is: don't start again.
if [ "$web_status" = "0" ]; then
echo "The Unicorn web server already running with pid $wpid, not restarting."
else
# Remove old socket if it exists
rm -f "$rails_socket" 2>/dev/null
# Start the web server
RAILS_ENV=$RAILS_ENV bin/web start
fi
# If sidekiq is already running, don't start it again.
if [ "$sidekiq_status" = "0" ]; then
echo "The Sidekiq job dispatcher is already running with pid $spid, not restarting"
else
RAILS_ENV=$RAILS_ENV bin/background_jobs start &
fi
if [ "$gitlab_workhorse_status" = "0" ]; then

21. Example: gitlab initd bash script (~500 lines)
echo "The GitLab Workhorse is already running with pid $spid, not restarting"
else
# No need to remove a socket, gitlab-workhorse does this itself.
# Because gitlab-workhorse has multiple executables we need to fix
# the PATH.
$app_root/bin/daemon_with_pidfile $gitlab_workhorse_pid_path
/usr/bin/env PATH=$gitlab_workhorse_dir:$PATH
gitlab-workhorse $gitlab_workhorse_options
>> $gitlab_workhorse_log 2>&1 &
fi
if [ "$mail_room_enabled" = true ]; then
# If MailRoom is already running, don't start it again.
if [ "$mail_room_status" = "0" ]; then
echo "The MailRoom email processor is already running with pid $mpid, not restarting"
else
RAILS_ENV=$RAILS_ENV bin/mail_room start &
fi
fi
if [ "$gitlab_pages_enabled" = true ]; then
if [ "$gitlab_pages_status" = "0" ]; then
echo "The GitLab Pages is already running with pid $spid, not restarting"
else
$app_root/bin/daemon_with_pidfile $gitlab_pages_pid_path
$gitlab_pages_dir/gitlab-pages $gitlab_pages_options
>> $gitlab_pages_log 2>&1 &
fi
fi
if [ "$gitaly_enabled" = true ]; then
if [ "$gitaly_status" = "0" ]; then
echo "Gitaly is already running with pid $gapid, not restarting"
else
$app_root/bin/daemon_with_pidfile $gitaly_pid_path
$app_root/bin/with_env $gitaly_dir/env
$gitaly_dir/gitaly >> $gitaly_log 2>&1 &

22. Example: gitlab initd bash script (~500 lines)
fi
fi
# Wait for the pids to be planted
wait_for_pids
# Finally check the status to tell wether or not GitLab is running
print_status
}
## Asks Unicorn, Sidekiq and MailRoom if they would be so kind as to stop, if not kills them.
stop_gitlab() {
exit_if_not_running
if [ "$web_status" = "0" ]; then
echo "Shutting down GitLab Unicorn"
RAILS_ENV=$RAILS_ENV bin/web stop
fi
if [ "$sidekiq_status" = "0" ]; then
echo "Shutting down GitLab Sidekiq"
RAILS_ENV=$RAILS_ENV bin/background_jobs stop
fi
if [ "$gitlab_workhorse_status" = "0" ]; then
echo "Shutting down GitLab Workhorse"
kill -- $(cat $gitlab_workhorse_pid_path)
fi
if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; then
echo "Shutting down GitLab MailRoom"
RAILS_ENV=$RAILS_ENV bin/mail_room stop
fi
if [ "$gitlab_pages_status" = "0" ]; then
echo "Shutting down gitlab-pages"
kill -- $(cat $gitlab_pages_pid_path)
fi
if [ "$gitaly_status" = "0" ]; then
echo "Shutting down Gitaly"
kill -- $(cat $gitaly_pid_path)
fi

23. Example: gitlab initd bash script (~500 lines)
# If something needs to be stopped, lets wait for it to stop. Never use SIGKILL in a script.
while [ "$web_status" = "0" ] || [ "$sidekiq_status" = "0" ] || [ "$gitlab_workhorse_status" = "0" ] || { [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; } || {
[ "$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" = "0" ]; } || { [ "$gitaly_enabled" = true ] && [ "$gitaly_status" = "0" ]; }; do
sleep 1
check_status
printf "."
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; }
&& { [ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
printf "n"
break
fi
done
sleep 1
# Cleaning up unused pids
rm "$web_server_pid_path" 2>/dev/null
# rm "$sidekiq_pid_path" 2>/dev/null # Sidekiq seems to be cleaning up its own pid.
rm -f "$gitlab_workhorse_pid_path"
if [ "$mail_room_enabled" = true ]; then
rm "$mail_room_pid_path" 2>/dev/null
fi
rm -f "$gitlab_pages_pid_path"
rm -f "$gitaly_pid_path"
print_status
}
## Prints the status of GitLab and its components.
print_status() {
check_status
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; } &&
{ [ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
echo "GitLab is not running."
return
fi
if [ "$web_status" = "0" ]; then
echo "The GitLab Unicorn web server with pid $wpid is running."

24. Example: gitlab initd bash script (~500 lines)
else
printf "The GitLab Unicorn web server is 033[31mnot running033[0m.n"
fi
if [ "$sidekiq_status" = "0" ]; then
echo "The GitLab Sidekiq job dispatcher with pid $spid is running."
else
printf "The GitLab Sidekiq job dispatcher is 033[31mnot running033[0m.n"
fi
if [ "$gitlab_workhorse_status" = "0" ]; then
echo "The GitLab Workhorse with pid $hpid is running."
else
printf "The GitLab Workhorse is 033[31mnot running033[0m.n"
fi
if [ "$mail_room_enabled" = true ]; then
if [ "$mail_room_status" = "0" ]; then
echo "The GitLab MailRoom email processor with pid $mpid is running."
else
printf "The GitLab MailRoom email processor is 033[31mnot running033[0m.n"
fi
fi
if [ "$gitlab_pages_enabled" = true ]; then
if [ "$gitlab_pages_status" = "0" ]; then
echo "The GitLab Pages with pid $mpid is running."
else
printf "The GitLab Pages is 033[31mnot running033[0m.n"
fi
fi
if [ "$gitaly_enabled" = true ]; then
if [ "$gitaly_status" = "0" ]; then
echo "Gitaly with pid $gapid is running."
else
printf "Gitaly is 033[31mnot running033[0m.n"
fi
fi
if [ "$web_status" = "0" ] && [ "$sidekiq_status" = "0" ] && [ "$gitlab_workhorse_status" = "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" = "0" ]; } && { [
"$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" = "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" = "0" ]; }; then
printf "GitLab and all its components are 033[32mup and running033[0m.n"

25. Example: gitlab initd bash script (~500 lines)
fi
}
## Tells unicorn to reload its config and Sidekiq to restart
reload_gitlab(){
exit_if_not_running
if [ "$wpid" = "0" ];then
echo "The GitLab Unicorn Web server is not running thus its configuration can't be reloaded."
exit 1
fi
printf "Reloading GitLab Unicorn configuration... "
RAILS_ENV=$RAILS_ENV bin/web reload
echo "Done."
echo "Restarting GitLab Sidekiq since it isn't capable of reloading its config..."
RAILS_ENV=$RAILS_ENV bin/background_jobs restart
if [ "$mail_room_enabled" != true ]; then
echo "Restarting GitLab MailRoom since it isn't capable of reloading its config..."
RAILS_ENV=$RAILS_ENV bin/mail_room restart
fi
wait_for_pids
print_status
}
## Restarts Sidekiq and Unicorn.
restart_gitlab(){
check_status
if [ "$web_status" = "0" ] || [ "$sidekiq_status" = "0" ] || [ "$gitlab_workhorse" = "0" ] || { [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; } || { [
"$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" = "0" ]; } || { [ "$gitaly_enabled" = true ] && [ "$gitaly_status" = "0" ]; }; then
stop_gitlab
fi
start_gitlab
}

26. Example: gitlab initd bash script (~500 lines)
### Finally the input handling.
case "$1" in
start)
start_gitlab
;;
stop)
stop_gitlab
;;
restart)
restart_gitlab
;;
reload|force-reload)
reload_gitlab
;;
status)
print_status
exit $gitlab_status
;;
*)
echo "Usage: service gitlab {start|stop|restart|reload|status}"
exit 1
;;
esac
exit

28. Gitlab needs?
4 processes:
Unicorn (ruby webserver process)
Sidekiq (ruby async worker process)
Mailroom (go mail management process)
Workhorse (go, custom reverse proxy and git http management)
Depends on:
Web server (apache, nginx…)
Redis

29. Gitlab needs?
File system ready
Network ready
User management
4 processes:
Unicorn (ruby webserver process)
Sidekiq (ruby async worker process)
Mailroom (go mail management process)
Workhorse (go, custom reverse proxy and git http management)
Depends on:
Web server (apache, nginx…)
Redis

30. Has the process A been started?
So, do we start B?

31. If A crashes,
do we need to restart it?
What about B?

32. What about logs?

33. What about the users running the process?

34. About environment variables?

35. Double-fork

36. Using initd, process management quality
varies from bash script to bash script, as
well as bash script quality
And changing configuration requires to
write bash code

37. Enter
systemd

38. Configuration over Scripting

39. The features we need, out of the box

40. gitlab-unicorn.service
[Unit]
Description=GitLab Unicorn Server
Requires=redis.service
Wants=mysqld.service postgresql.service
After=redis.service mysqld.service postgresql.service
[Service]
User=git
WorkingDirectory=/home/git/gitlab
Environment=RAILS_ENV=production
SyslogIdentifier=gitlab-unicorn
PIDFile=/home/git/gitlab/tmp/pids/unicorn.pid
ExecStart=/usr/bin/bundle exec "unicorn_rails -D -c /home/git/gitlab/config/unicorn.rb -E production"
[Install]
WantedBy=multi-user.target

41. gitlab-sidekiq.service
[Unit]
Description=GitLab Sidekiq Worker
Requires=redis.service
Wants=mysqld.service postgresql.service
After=redis.service mysqld.service postgresql.service
[Service]
Type=forking
User=git
WorkingDirectory=/home/git/gitlab
Environment=RAILS_ENV=production
SyslogIdentifier=gitlab-sidekiq
PIDFile=/home/git/gitlab/tmp/pids/sidekiq.pid
ExecStart=/usr/bin/bundle exec "sidekiq -C config/sidekiq_queues.yml -c 5 -e production -P tmp/pids/sidekiq.pid -d -L log/sidekiq.log >>
log/sidekiq.log 2>&1"
ExecStop=/usr/bin/bundle exec "sidekiqctl stop /home/git/gitlab/tmp/pids/sidekiq.pid >> /home/git/gitlab/log/sidekiq.log 2>&1"
[Install]
WantedBy=multi-user.target

42. gitlab-mailroom.service
[Unit]
Description=Gitlab mailroom Worker
Requires=gitlab-unicorn.service
Wants=gitlab-unicorn.service
After=gitlab-unicorn.service
[Service]
User=git
Environment=RAILS_ENV=production
WorkingDirectory=/home/git/gitlab
SyslogIdentifier=gitlab-mailroom
PIDFile=/home/git/gitlab/tmp/pids/gitlab-mailroom.pid
Type=oneshot
RemainAfterExit=yes
ExecStart=/home/git/gitlab/bin/mail_room start
ExecStop=/home/git/gitlab/bin/mail_room stop
[Install]
WantedBy=multi-user.target

43. gitlab-workhorse.service
[Unit]
Description=Gitlab Workhorse handles slow HTTP requests for Gitlab.
Requires=gitlab-unicorn.service
Wants=gitlab-unicorn.service
After=gitlab-unicorn.service
[Service]
Type=forking
User=git
WorkingDirectory=/home/git/gitlab-workhorse
SyslogIdentifier=gitlab-workhorse
PIDFile=/home/git/gitlab/tmp/pids/gitlab-workhorse.pid
ExecStart=/home/git/gitlab/bin/daemon_with_pidfile /home/git/gitlab/tmp/pids/gitlab-workhorse.pid
/home/git/gitlab-workhorse/gitlab-workhorse -listenUmask 0 -listenNetwork unix -listenAddr
/home/git/gitlab/tmp/sockets/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket
/home/git/gitlab/tmp/sockets/gitlab.socket -secretPath /home/git/gitlab/.gitlab_workhorse_secret -
documentRoot /home/git/gitlab/public >> /home/git/gitlab/log/gitlab-workhorse.log 2>&1
[Install]
WantedBy=multi-user.target

44. Working using systemctl
start/stop/reload
status
daemon-reload
list...

45. unit

46. Service
application
Start / stop
Start conditions
Dependencies

47. Target
Logical groups of units (`network.target`, `multi-user.target`, …)

48. Socket, path
Starts a service when a specific socket is used (socket-based activation)
Same for path-based activation

49. Timer
Start a service periodically

50. Example, gitlab backup - .timer file
[Unit]
Description=Run gitlab request backup every day
After=network.target
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

51. Example, gitlab backup - .service file
[Unit]
Description=Gitlab request backup
After=network.target
[Service]
Type=oneshot
User=git
ExecStart=/usr/bin/gitlab-backup

52. Device, mount, automount, swap
File system mounting
Derived from `/etc/fstab`

53. Snapshot
System state snapshot

54. Slice
Control Group descriptor for resource restrictions

55. systemd-nspawn
Namespaced container
Virtualized file system, process tree
No resource isolation by default
Used by rkt to start containers

56. Work in an ephemeral snapshot
systemd-nspawn -D / -xb
Starts a container with the current state of the system, changes are removed
when quitting the container.

57. Run an image in a container
systemd-nspawn -M Fedora-Cloud-Base-25-1.3.x86_64.raw

58. systemd-run
Start transient services, on the fly resource limitations
systemd-run -p BlockIOWeight=10 updatedb

59. Coupon on clever-
cloud.com :
systemdevoxx
On twitter: @clementd & @waxzce