1.
1
© 2017 Pivotal
Continuous Deployment of your Application
Marcin Grzejszczak, @mgrzejszczak

2.
2
Spring Cloud developer at Pivotal
Working mostly on
● Spring Cloud Sleuth
● Spring Cloud Contract
● Spring Cloud Pipelines
About me
Twitter: @mgrzejszczak
Blog: http://toomuchcoding.com

3.
3
What problem are we trying to solve?

4.
4
The necessity to create a deployment
pipeline from scratch for each new project

5.
5
Spring Cloud Pipelines
• opinionated template of a deployment pipeline
• based on good practices from different projects
• we believe in the “Infrastructure as Code” approach
• in case of automation server going down you can recover
everything in no time
• the automation server setup should be automated too!
• you can even write tests for your pipelines

6.
6
Spring Cloud Pipelines
• we support following automation servers out of the box
• Concourse
• Jenkins using Jenkins Job DSL plugin
• Jenkins using Jenkinsfile with Blue Ocean
• logic of all steps done via Bash scripts
• you can convert the internals to suit your needs
• you can use whatever automation server you want
• supports Maven & Gradle
• Over 150 Bash tests (using https://github.com/sstephenson/bats)

7.
7
Conventions in the code - Maven

8.
8
Conventions in the code - Gradle

9.
9
Spring Cloud Pipelines - Concourse

10.
10
Concourse

11.
11
Spring Cloud Pipelines - Jenkins Job DSL

12.
12
Jenkins Job DSL

13.
13
Spring Cloud Pipelines - Jenkinsfile & Blue Ocean

14.
14
Jenkinsfile

15.
15
SPRING CLOUD PIPELINES
REPOSITORY & AUTOMATION SERVER
DEMO

16.
16
WHAT ARE WE GOING TO WORK WITH?

17.
17
Demo - flow
GITHUB-WEBHOOK GITHUB-ANALYTICS
AMQPHOOK

18.
18
Why do we deploy software?
• We’re paid for delivering business value
• Features are done when they are deployed to production
• It’s better to deploy frequently for faster feedback
• It’s better to fail the deployment pipeline as fast as possible
• Deployments should take place in a standardised, automated fashion
• Your deployment pipeline should test rollback
• That way you can perform A/B or zero downtime deployment to
production

19.
19
Spring Cloud Pipelines
• We’re paid for delivering business value
• Features are done when they are deployed to production
• It’s better to deploy frequently for faster feedback
• It’s better to fail the deployment pipeline as fast as possible
• Deployments should take place in a standardised, automated fashion
• Your deployment pipeline should test rollback
• That way you can perform A/B or zero downtime deployment to
production

20.
20
Spring Cloud Pipelines

21.
21
Problem - slow feedback
• Nobody wants to wait until the end of the pipeline to see that something is
not working fine
• We want to fail fast - even during build time
• If integration is faulty
• If our API is backward incompatible
• There should be no need to wait until end to end tests are executed

22.
22
Solution - fail fast
• We’re running unit and integration tests during build time
• To test faulty integration we use Spring Cloud Contract for HTTP / messaging
• Producer defines a contract
• From the contract
o tests are generated to see if the producer is not lying
o stubs are generated for consumers to use
• Consumer can reuse it without running the producer
• Fail at build time if integration fails (fail fast!)
• All stubs reside in Nexus / Artifactory (thus can be reused)

23.
23
CONTRACT TESTS DEMO
(BREAKING CONSUMER)

24.
24
Initial contract (producer)
Body contains username and
repository

25.
25
Generated test (producer)
Body contains username and
repository

26.
26
Expected model (consumer)

27.
27
Passing contract test (consumer)

28.
28
Breaking contract (producer)
Was username and repository
and we’ve made a breaking
change by converting those to
user and repo

29.
29
New, generated test (producer)
Was username and repository
and we’ve made a breaking
change by converting those to
user and repo

30.
30
Installing broken stubs locally (producer)
New stubs with backward
incompatible changes
installed in Maven local

31.
31
Broken contract test locally (consumer)

32.
32
Broken contract test locally (consumer)
Expected repository and
username but got repo
and user

33.
33
CONTRACT TESTS DEMO
(BREAKING PRODUCER)

34.
34
Contract for deletion of issues

35.
35
Generated test for deleting issues

36.
36
Demo - backward incompatible API change
GITHUB-ANALYTICS V1
with DELETE
@ /issues
Contracts
V1
with
DELETE @
/issues
Generated test
sends DELETE
to /issues
GITHUB-ANALYTICS V2
removed DELETE
@ /issues
Contracts
V2
removed
DELETE @
/issues
No generated
test for this
endpoint
GITHUB-ANALYTICS V2
removed DELETE
@ /issues
Contracts
V1
with
DELETE @
/issues
Generated test
sends DELETE
to /issues

37.
37
Backward incompatible changes of the API
We delete
the
contract
and the
DELETE
endpoint

38.
38
Broken build
Current
implementation
(V2) does not
support old
contracts (V1)

39.
39
Broken build - generated test from old contracts (V1)

40.
40
Broken build

41.
41
Spring Cloud Pipelines
• We’re paid for delivering business value
• Features are done when they are deployed to production
• It’s better to deploy frequently for faster feedback
• It’s better to fail the deployment pipeline as fast as possible
• Deployments should take place in a standardised, automated fashion
• Your deployment pipeline should test rollback
• That way you can perform A/B or zero downtime deployment to
production

42.
42
Spring Cloud Pipelines

43.
43
STANDARDISATION GIVES LOWER
SUPPORT COSTS AND MORE CHANCES
OF PEOPLE ROTATION

44.
44
Solution - PaaS & tooling
• Use a PaaS to standardise the way you deploy and monitor your software
• Spring Cloud Pipelines uses Cloud Foundry [1] or Kubernetes [2] as a PaaS
implementation
• For the demo purposes we’re using PCF Dev [3] or Minikube [4]
• PaaS abstracts the application governance from underlying infrastructure
• you can deploy, scale, manage applications in the same way if PaaS was
running on your laptop, Amazon, Google, Azure etc.
• Database schema upgrade is done via tools like Flyway [5] or Liquibase [6]
[1] https://www.cloudfoundry.org, [2] https://kubernetes.io/
[3] https://pivotal.io/pcf-dev, [4] https://github.com/kubernetes/minikube
[5] https://flywaydb.org/, [6] http://www.liquibase.org/

45.
45
Deploying apps with CF
https://www.cloudfoundry.org
https://pivotal.io/pcf-dev
$ cf push

46.
46
CF DEMO

47.
47
Pivotal Cloud Foundry Apps Manager
Different spaces for
different environments

48.
48
Pivotal Cloud Foundry Apps Manager
Running application in
production space

49.
49
Pivotal Cloud Foundry Apps Manager
Audit events of what
happened with your
instances State of your instances
Number of instance you
want to run
Limits for memory and disk

50.
50
Pivotal Cloud Foundry Apps Manager
Bound services to the
application. Credentials
get injected automatically

51.
51
Spring Cloud Pipelines
After the application got deployed to test
environment
• The database schema gets updated upon
application startup
• We run a handful of smoke tests to see if crucial
parts of our application are working fine
• We want to test if the app is properly packaged
• The application is surrounded by stubs - no real
integrations take place
• Spring Cloud Contract Stub Runner Boot app is
responsible for serving stubs

52.
52
Problem - rollback DB
• Deployment pipelines should test whether the application can be rolled back
• Rolling back database changes is extremely difficult
• Especially if you deploy once every 6 months (the number of changes is
gigantic)
• How can you roll back a deletion of a column?

53.
53
Solution - application rollback
• The easiest solution is… NOT TO DB ROLLBACK
• Perform only backward compatible changes (always add data)
• Or perform backward incompatible changes
in a backward compatible way [1]
• Roll back the application only (the JAR)
• The application and DB changes need to be
backward compatible
• That way you can ensure that two applications (old
/ new versions) can run simultaneously at the same time
[1] https://spring.io/blog/2016/05/31/zero-downtime-deployment-with-a-database

54.
54
Demo - backward incompatible DB change
GITHUB-ANALYTICS V1
with repository
DB
V1
with
repository
GITHUB-ANALYTICS V2
with repo
DB
V2
with
repo
GITHUB-ANALYTICS V1
with repository
DB
V2
with
repo

55.
55
BACKWARD INCOMPATIBLE
DB CHANGE DEMO

56.
56
Demo - initial DB state (Flyway)

57.
57
Demo - first run

58.
58
Demo - initial state

59.
59
Demo - backward incompatible DB change

60.
60
Demo - backward incompatible DB change

61.
61
Demo - deploying backward incompatible DB change
Old application (V1) can’t
work with new database (V2)
New application (V2) works
against new database (V2)

62.
62
Demo - errors in the app logs Old version can’t insert
data to a missing DB
column

63.
63
Spring Cloud Pipelines

64.
64

65.
65
Problem - end to end tests
• End to end tests are slow and brittle
• QA department writes an E2E for every feature we have
• E2E environment setup
• one environment shared between all applications?
• one environment per application?
• Surrounding apps should be deployed in
• production versions?
• development versions?

66.
66
Solution - don’t do E2E?
• Regardless of the time spent on QA / UAT you can still have bugs on
production
• Assuming that you ...
• embrace failure
• introduce monitoring of business key performance indicators (KPIs)
• introduce alerting over the metrics
• ensure that you can rollback on production
• … you could stop doing any end to end tests

67.
67
Spring Cloud Pipelines
• The whole step is optional
• it’s left there cause the “no e2e tests”
approach is controversial
• Deploy to stage and running e2e tests are manual
steps
• you have to wait for your turn for the env
• some manual work has to be done to purge
stale data etc.

68.
68
Spring Cloud Pipelines
• We’re paid for delivering business value
• Features are done when they are deployed to production
• It’s better to deploy frequently for faster feedback
• It’s better to fail the deployment pipeline as fast as possible
• Deployments should take place in a standardised, automated fashion
• Your deployment pipeline should test rollback
• That way you can perform A/B or zero downtime deployment to
production

69.
69
Spring Cloud Pipelines

70.
70
Problem - deployment to production
• We don’t want to deploy the application to production at night
• We want to treat a production deployment like any other deployment
• We’d like to be able to perform A/B testing and zero downtime deployment
• We’d like to easily rollback when sth goes wrong

71.
71
Solution - PaaS + SC Pipelines
• Our application has KPI monitoring in place
• Alerting are set for KPIs
• It has been tested that the application can be easily rolled back
• PaaS (CF or K8S) can take care of zero downtime deployment

72.
72
Spring Cloud Pipelines
• Deploy to prod deploys the pipeline version of the
app to production next to the current production
version
• Once deployed we tag the repo with
prod/VERSION_NUMBER
• Complete switch over allows to delete the old
instance and leave only the new one
• Rollback to blue allows to stop the new instance
(green) to route traffic only to old version (blue)

73.
73
A/B ON CF
DEMO

74.
74
Two versions running at the same time
Two versions registered
under same hostname
Old instance
New instance

75.
75
After complete switch over
One app remains

76.
76
Demo - alerts
GITHUB-WEBHOOK GITHUB-ANALYTICS
AMQPPOST POLL

77.
77
Demo - alerts
GITHUB-ANALYTICS
DELETE
POLL PUSH

78.
78
METRICS & ALERTS DEMO

79.
79
Insert some data to the service

80.
80
Grafana Satisfactory number of
issues
Threshold below which
alerts will be sent
Metric configuration

81.
81
Slack notifications

82.
82
Delete all issues

83.
83
Grafana - alert
The metric went down to 0

84.
84
Slack notifications

85.
85
Spring Cloud Pipelines

86.
86
Customizing Spring Cloud Pipelines

87.
87
Customizing Spring Cloud Pipelines
$ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/v1.0.0.M6.zip
$ unzip v1.0.0.M6.zip
$ cd spring-cloud-pipelines-v1.0.0.M6
$ git init
$ # modify the pipelines to suit your needs
$ ./gradlew customize
$ …
$ git add .
$ git commit -a -m “Initial commit”
$ git remote add origin ${YOUR_REPOSITORY_URL}
$ git push origin master

88.
88
Customizing Spring Cloud Pipelines

89.
89
Summary
• Continuous Deployment allows you to continuously deliver business value
• Spring Cloud Pipelines gives you OOB tooling to test your software via
• unit and integration testing
• contract testing
• rollback testing
• Spring Cloud Pipelines allows you to easily adjust the deployment pipeline to
suit your company’s needs
• Thanks to PaaS you can easily do A/B & zero downtime deployment

90.
90

91.
91
▪ Github Analytics: https://github.com/spring-cloud-samples/github-analytics
▪ Github Webhook: https://github.com/spring-cloud-samples/github-webhook
▪ SC-Pipelines documentation: https://cloud.spring.io/spring-cloud-pipelines/
▪ No end to end tests
• https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
• http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
▪ Prometheus on CF https://github.com/mkuratczyk/prometheus-on-PCF
▪ Prometheus for PCF Dev (Docker compose) https://github.com/vegasbrianc/prometheus
▪ Pivotal Web Services trial : https://run.pivotal.io/
▪ PCF Dev (CF on your laptop) : https://docs.pivotal.io/pcf-dev/
▪ Project Kubo (K8S & Bosh): https://pivotal.io/partners/kubo
Links

92.
92
Learn More. Stay Connected.
▪ Read the docs
http://cloud.spring.io/spring-cloud-pipelines/
▪ Talk to us on Gitter
https://gitter.im/spring-cloud/spring-cloud-pipelines
Twitter: twitter.com/springcentral
YouTube: spring.io/video
LinkedIn: spring.io/linkedin
Google Plus: spring.io/gplus

93.
93
mgrzejszczak