Presented by Markus Hansmair during the OpenShift Meetup (2019-11-25)

1. GitLab on OpenShift
Markus Hansmair
2019-11-25

2. GitLab on OpenShift Slide 2
Agenda
• Installing GitLab on OpenShift
• Preparational steps
• Installation
• Issues & fixes
• Upgrading
• Backup & Restore
• Desaster recovery
• Data migration

3. GitLab on OpenShift Slide 3
Installing GitLab on OpenShift
https://labs.consol.de/
Thank you!
Any questions?

4. GitLab on OpenShift Slide 4
Installing GitLab on OpenShift
Just kidding :-)

5. GitLab on OpenShift Slide 5
Installing GitLab on OpenShift
Preparational steps
 Install required software on local host
 Create an OpenShift project for GitLab
 Tweak permissions
 Provide TLS certificates
 Provide LDAP credentials
 Compose gitlab-values.yml

6. GitLab on OpenShift Slide 6
Installing GitLab on OpenShift
Install required software on local host
Get helm binary (https://github.com/helm/helm/releases/) (v2.12.x or above, but not v3.x)
helm version
helm init --client-only
helm plugin install https://github.com/rimusz/helm-tiller
helm repo add gitlab https://charts.gitlab.io/
helm repo update
Create an OpenShift project for GitLab
oc new-project gitlab

7. GitLab on OpenShift Slide 7
Installing GitLab on OpenShift
Tweak permissions
oc adm policy add-scc-to-user anyuid -z gitlab-shared-secrets
oc adm policy add-scc-to-user anyuid -z gitlab-gitlab-runner
oc adm policy add-scc-to-user anyuid -z gitlab-prometheus-server
oc adm policy add-scc-to-user anyuid -z default
Provide TLS certificates / LDAP credentials
oc create secret tls gitlab-certs --cert=tls.crt --key=tls.key
oc create secret generic gitlab-ldap-secret --from-literal=password=<ldap-password>

8. GitLab on OpenShift Slide 8
Installing GitLab on OpenShift
Compose gitlab-values.yml
global:
appConfig:
ldap:
....
edition: ce
email:
from: <from-address-for-notifications>
hosts:
domain: <my-domain> # your TLS certificates must match gitlab.<my-domain>
externalIP: <external-ip> # of gitlab.<my-domain>
ingress:
configureCertmanager: false
tls:
secretName: gitlab-certs
smtp:
....

9. GitLab on OpenShift Slide 9
Installing GitLab on OpenShift
Compose gitlab-values.yml (continued)
certmanager:
install: false
nginx-ingress:
enabled: false
registry:
enabled: false
gitlab:
sidekiq:
registry:
enabled: false
task-runner:
registry:
enabled: false
unicorn:
registry:
enabled: false

10. GitLab on OpenShift Slide 10
Installing GitLab on OpenShift
Compose gitlab-values.yml (continued)
global:
....
psql:
host: <fqdn-of-postgresql-server>
port: 5432
username: <username>
database: <database-name>
password:
secret: <secret-with-password-of-psql-username>
key: <key-within-secret>
A separate PostgreSQL database is recommended for production-ready installations.

11. GitLab on OpenShift Slide 11
Installing GitLab on OpenShift
Actual installation
helm tiller start gitlab
helm upgrade --install -f gitlab-values.yml gitlab gitlab/gitlab --version 2.5.1 --timeout 600
For a complete mapping between chart versions and versions of GitLab see
https://docs.gitlab.com/charts/installation/version_mappings.html
oc get --export -o yaml secret gitlab-rails-secret > gitlab-rails-secret.yaml
Save the secret keys created:
YOU WILL NEED THEM IN CASE OF DESASTER RECOVERY!

12. GitLab on OpenShift Slide 12
Installing GitLab on OpenShift
Issues & fixes
 Fix GitLab Runner deployment
 Configure data rentention for Prometheus
 Fix s3cmd configuration for Task Runner pods

13. GitLab on OpenShift Slide 13
Installing GitLab on OpenShift
Fix GitLab Runner deployment
oc edit deployment gitlab-gitlab-runner
Pods need server certificate provided by secret gitlab-certs created in preparational steps.
- mountPath: /home/gitlab-runner/.gitlab-runner/certs
name: volume-gitlab-certs
readOnly: true
- name: volume-gitlab-certs
secret:
defaultMode: 420
items:
- key: tls.crt
path: gitlab.<my-domain>.crt
secretName: gitlab-certs

14. GitLab on OpenShift Slide 14
Installing GitLab on OpenShift
Fix GitLab Runner deployment (continued)
Change CI_SERVER_URL to use service URL (instead of route URL)
- name: CI_SERVER_URL
value: http://gitlab-unicorn.gitlab.svc:8181

15. GitLab on OpenShift Slide 15
Installing GitLab on OpenShift
Configure data retention for Prometheus
oc edit deployment gitlab-prometheus-server
Prevent Prometheus‘ time series database to fill up its entire volume
- args:
- --config.file=/etc/config/prometheus.yml
- --storage.tsdb.path=/data
- --storage.tsdb.retention=7d
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle

16. GitLab on OpenShift Slide 16
Installing GitLab on OpenShift
Fix s3cmd configuration for Task Runner pods
oc edit configmap gitlab-task-runner
Use service URL of minio (instead of route URL) to avoid TLS issues.
host_base = gitlab-minio-svc:9000
host_bucket = gitlab-minio-svc:9000/%(bucket)
....
website_endpoint = http://gitlab-minio-svc:9000
use_https = False
oc delete pod <task-runner-pod>
Restart Task Runner pod

17. GitLab on OpenShift Slide 17
Upgrading GitLab on OpenShift
Upgrading
helm repo update
helm tiller start gitlab
helm upgrade -f gitlab-values.yml gitlab gitlab/gitlab --version <chart-version> --timeout 600
DO NOT SKIP MINOR VERSIONS WHEN UPGRADING!
11.10.4 => 11.11.x => 12.0.x => 12.1.1
Check that modifications (see issues & fixes) are still in place after each upgrade step!

18. GitLab on OpenShift Slide 18
Backup & Restore
Backup
oc rsh <gitlab-task-runner-pod>
backup-utility --backup
Backup archive is stored in GitLab‘s own installation of Minio
(e.g. 1563521329_2019_07_19_11.10.4_gitlab_backup.tar)
Various options:
 Separate S3 compatible object store with own backup strategy
 Backup persistent volume used by Minio
 Hand-crafted backup of backup archives
oc exec $TASK_RUNNER_POD -- s3cmd get s3://gitlab-backups/$BACKUP_ARCHIVE /tmp
oc cp $TASK_RUNNER_POD:/tmp/$BACKUP_ARCHIVE .
oc exec $TASK_RUNNER_POD -- s3cmd rm s3://gitlab-backups/$BACKUP_ARCHIVE
oc exec $TASK_RUNNER_POD -- sh -c "rm /tmp/*_gitlab_backup.tar"

19. GitLab on OpenShift Slide 19
Backup & Restore
Restore
oc rsh <gitlab-task-runner-pod>
backup-utility --restore -t <backup-name>
<backup-name> e.g. 1563521329_2019_07_19_11.10.4
In case backup archive is not stored in object store:
oc cp $BACKUP_ARCHIVE $TASK_RUNNER_POD:/tmp/0_gitlab_backup.tar
oc exec $TASK_RUNNER_POD -- backup-utility --restore -f /tmp/0_gitlab_backup.tar
0_gitlab_backup.tar is a work-around due to a bug in backup-utility

20. GitLab on OpenShift Slide 20
Desaster recovery
Desaster recovery
 Install the EXACT version of GitLab that was used for your latest backup
 Restore the secret keys of your lost installation
oc delete secret gitlab-rails-secret
oc create -f gitlab-rails-secret.yaml
 Do the actual restore (see previous slide)

21. GitLab on OpenShift Slide 21
Desaster recovery
Desaster recovery (continued)
 Recreate runner registration token secret
 Log in as root and access https://gitlab.<mydomain>/admin/runners
 Copy the token below Use the following registration token during setup:
 Recreate the secret gitlab-gitlab-runner-secret.
oc delete secret gitlab-gitlab-runner-secret
oc create secret generic gitlab-gitlab-runner-secret
--from-literal=runner-registration-token=<new-registration-token>
--from-literal=runner-token=""
 Restart pods
oc delete pods -l app=unicorn
oc delete pods -l app=sidekiq
oc delete pods -l app=gitlab-gitlab-runner

22. GitLab on OpenShift Slide 22
Data migration
Data migration
 Recommended way for data migration is using a S3 compatible object store and backup-utility -t …
 No luck due to TLS issues :-(
 Alternative approach: Tweak backup archive from source installation
 Extract backup archive into an empty folder
tar -xvf ../<backup-archive>
 Edit backup_information.yml
 Change line with installation type :installation_type: gitlab-helm-chart
 Repackage backup archive
tar -cvf ../<backup-archive> *
 Proceed with desaster recovery (see previous slides)

23. Thank you!

24. ConSol
Consulting & Solutions Software GmbH
St.-Cajetan-Straße 43
D-81669 Munich
Germany
Tel.: +49-89-45841-100
info@consol.de
www.consol.com
Twitter: @consol_de