Lessons Learned: Using Concourse In Production

Using Concourse in Production
- Lessons Learned -
Shingo Omura(@everpeace)
omura@chatwork.com
ChatWork, Inc.
Concourse Meetup #5 2017/03/13

© ChatWork All rights reserved.© ChatWork All rights reserved.
Outline
● About ChatWork
● Our Context From the Point of View of Infrastructure
● Our Use Case
● Good parts
● Pipeline Tips
● Small Bad parts (expect to improve)
2

© ChatWork All rights reserved.
Group Chat File Sharing
Task Management Video Conference
About ChatWork ~Group Chat for Global Teams~
3

ChatWork is growing rapidly
● 127,000 organizations
○ number of users is not opened
● 205 countries or regions
● 6 languages supported
as of 2017/02
4

Our Context
From the Point of View
of Infrastructure

New Infrastracture Project (1/2)
● Current Infra
○ EC2 based apps, deploy servers(for capistorano)
○ Jenkins servers for CI/CD
● Pain points
○ Ops team doesn’t scale
■ release always have to be done with Infra team members
○ AWS env and Jenkins are hard to sandboxing
■ part of aws resouces are managed by terraform, but not all
■ deployment flow is hard to develop and testing
6

New Infrastracture Project (2/2)
● Next Infra
○ Kubernetes and Helm with Dockerized apps
○ Concourse CI for CI/CD
● Benefits
○ Kubernetes accelarate DevOps
■ App team can fully manage their deployment cycle by themselves.
■ minikube is really helpful for local dev environemnt.
■ kubernetes team can focus on reliability of Kuberentes.
○ Concourse CI does too! ← Today’s Focus
■ reduces operational load
■ helps agile development of deployment/testing process
● Status
○ Using from new messaging backend (released the last december)
○ Current system is planned to migrate to this next infra
7

Overview of deployment system
● Concourse is deployed by concourse-aws
○ maintained by @mumoshu (my-colleague) and @everpeace (me)
● Branching model is Gitlab flow with Environment Branches
● chatwork-notify-resource for notification
staging
branch
staging environment
production environment
master branch
push
im
age
build and deploy helm package
build and deploy helm packagepush image
pull image
pull image
notify
9

Our build pipeline environment can be
split by ‘groups’
notification resource
10

Our build pipeline
test&build jobs
deploy jobs
rollback jobs
11

Good Parts
● concourse.ci is extemely well-documented
○ You can start trying concourse in 5 min.
■ virtualbox and vagrant: just ‘vagrant up’!!
■ docker-compose support!!
○ easty to write pipelines thanks to comrehensive reference
● easy to deploy & version up (thanks to concourse-aws :-P )
○ initial deploy: 3 steps
■ ‘build-amis.sh’ → edit ‘cluster.yml’ → ‘concourse-aws up’
○ version up: similar 3 steps
■ ‘build-amis.sh’(new version) → edit ‘cluster.yml’(new ami) →
‘concourse-aws up’
13

Good Parts (cont.)
● Concourse frees us from ”plugin hell”
○ all resource is provided by docker image
○ task environment can be injected by docker image too
○ no need to manage backups of CI servers!!
● Multi tenancy ‘team’ support
■ multiple team can share CI server resources
■ but isolated appropriately
■ each app team can have controll in their team
● Various authentication scheme support
■ concourse need not to have user database
■ we use github authentication
14

● easy to develop pipelines
○ Pipeline developed & tested in local env can be deployed directly
to production concourse
■ Concourse CI’s pipeline is stateless and reproductive
■ Concourse & Kubernetes both supports local env (minikube & concourse
vagrant box)
Good Parts (cont.)
15

Good Parts (cont.)
● easy to extend/custom
○ easy to develop custom resource.
■ you only need to develop 3 commands(check, in, out) whose returns json
objects.
■ language agnostic! you can choose your own language!!
○ easy to prepare task environment
■ when you need some task environment in which some toolkit is installed, you
just push docker image to any repository and specify the image to your task
definition
task.yml
---
platform: linux
image_resource:
type: docker-image
source:
repository: /yourown/image
tag: '1.1'
16

Pipeline tips: summary
● Use groups for large pipeline
● Use aggregate for running in parallel (useful for resources)
● Use “[ci skip]” keyword to commit message when
Concourse commits/push to git repo
● on_success/on_failure hook is useful for notification
● input_mapping/output_mapping is useful for shared
task definition
● use attempts for deployment task due to intermittent
network failure
● @making’s trick is helpful for build caches(sbt, ivy, maven)
18

Pipeline Tips
● Use groups for large pipeline to group many jobs
● Use aggregate for multiple resources (useful for resources)
pipeline.yml
groups:
- name: master
jobs:
- job-for-master
- name: production
jobs:
- job-for-production
pipeline.yml
plan:
- aggregate:
- get: app-repo
trigger: true
- get: tool-repo
- get: sbt-ivy-cache
those 3 get runs in parallel
19

Pipeline Tips
● Use “[ci skip]” keyword to commit message when
Concourse commits/push to git repo
○ git resource skip commits with [ci skip] keywords
○ It’s really useful when
■ back merge: “merging release branch to develop branch”
● the commit is wanted to skip CI process
■ the commit bumping versions
● when using sbt, version number is embedded to repo
20

● on_success/on_failure hook is useful for notification
Pipeline Tips
pipeline.yml
- task: deploy-write-api-to-dev-kube
file: foo/task.yml
on_success:
task: chatwork-notification
file: tasks/notify_chatwork.yml
on_failure:
task: chatwork-notification
file: tasks/notify_chatwork.yml
on_failure
on_success
21

● input_mapping/output_mapping is useful for shared
task definition
Pipeline Tips
pipeline.yml
- task: test-pull-request
file: pull-request/ci/tasks/unit.yml
input_mapping: { repo: pull-request }
- task: unit
file: master/ci/tasks/unit.yml
input_mapping: { repo: master }
ci/tasks/unit.yml
---
platform: linux
image_resource:
type: docker-image
source:
repository: yourown/toolbox
inputs:
- name: repo
run:
path: /bin/bash
args:
- repo/ci/tasks/unit.sh
22

● use attempts for deployment task due to intermittent
network failure
Pipeline Tips
pipeline.yml
...
- task: deploy-write-api-to-dev-kube
file: ..snip../deploy-to-kube-helm.yml
attempts: {{attempts}}
attempts=3
23

● @making’s trick is helpful for build caches(sbt, ivy, maven)
○ prepare own cache docker image repo (anywhere)
○ archives cache files as rootfs.tar and push it directly to
the image repo
○ related issue is now open:
Caching directories between runs of a task #230
Pipeline Tips
24

Small Bad Parts
(expect to improve)

Small Bad Parts (expect to improve)
● No fine-grained authorization
(No role based aaccess control)
○ every team member can take full controll in the team
○ ‘fly get-pipeline’ exposes all creadentials embedded in pipelines
○ We sometime want to split
■ people who can write/read pipeline
■ people who can just view logs and trigger jobs
(no rights to change pipelines but can just operate the pipeline)
○ related issues are open
■ Credential management #19
■ Individual/fine-grained access control #23
26

Small Bad Parts (expect to improve)(cont.)
● No parameterized job
○ we would like to deploy specific feature branch to shared dev
environment
○ How could do this with Concourse?? Any Idea??
○ git-multibranch-resource could achive similar thing
■ branch name convention which will be deployed to shared dev env should be
agreed
○ Perhaps `fly exec` prompts user input?
27

Small Bad Parts (expect to improve)(cont.)
● No Docker Compose in task
○ the issue is now open:
Docker Compose support in Task definitions #324
■ integration test task with app & local db containers
● FYI: various improvements are disscued in
https://github.com/concourse/design-notes/issues
28

We’re Hiring!!!
Search “ChatWork” in Wantedly
https://www.wantedly.com/companies/chatwork/projects

Lessons Learned: Using Concourse In Production

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lessons Learned: Using Concourse In Production

Similar to Lessons Learned: Using Concourse In Production (20)

Recently uploaded

Recently uploaded (20)

Lessons Learned: Using Concourse In Production