The document discusses many challenges faced when adopting configuration management and DevOps practices. It argues that challenges are now less technical and more cultural, with unrealistic expectations from leadership, an overemphasis on automation over culture, and failure to consider operational aspects like monitoring. It advocates focusing on people over tools, improving code quality, and learning from past experiences rather than jumping to new technologies without understanding challenges already addressed.

1. Cfgmgmt Challenges are not technical
anymore
Julien Pivotto (@roidelapluie)
Config Management Camp Ghent February 2018

2. $::user
Julien Pivotto
Consultant at inuits
@roidelapluie on irc/github/twitter
Puppet / Ansible / Terraform / mgmt

3. inuits

4. Once upon a time...
Creative Commons Attribution-ShareAlike 2.0
https://www.flickr.com/photos/lorenkerns/13991814652

5. Package
Creative Commons Attribution-ShareAlike 2.0
https://www.flickr.com/photos/halfbisqued/2353845688/

6. Config
Creative Commons Attribution 2.0
https://www.flickr.com/photos/calliope/234447967

7. Service
Creative Commons Attribution 2.0
https://www.flickr.com/photos/beaub/1795730403

8. PCS pattern
Easy to do manually
Yeah so let's do this
You say package? Here is a tarball.

9. PCS pattern (automated)
Package managers - rpm - deb
Versioned dependencies
Sanity checks
One source of truth for where files come from
Templates, reproducible config

10. Config Management
CFEngine ~25y
Puppet ~10y
Chef ~10y
Ansible ~5y
... More like this

11. Operating system abstraction
Puppet:
package{
'ntp':
ensure => installed,
}
Operating system / Package manager
independant. No bash required.

12. Then comes the zero downtime
thingie
One must be able to deploy PCS style but
without downtime
Rolling restart / upgrade
2 "easy" ways to do that

13. Built-in into our apps
Take the burden in the development process
Clusters
API versioning
Take care of data migration
Reverse proxy
e.g. Elasticsearch 6 (rolling upgrades accross
major releases)

14. Built in into the platform
Orchestration
Config management of reverse proxies

15. Reverse proxy
"dumb" reverse proxy
­ include: remove_from_rproxy.yml
­ wait_for:
host: "{{bind_address}}"
port: 8080
state: drained
­ name: stop myservice
systemd:
name: "myservice.service"
state: restarted

16. Reverse proxy
"clever" reverse proxy
Think service registry, health checks...
e.g. traefik
[consulCatalog]
endpoint = "127.0.0.1:8500"
prefix = "traefik"
$ dig +short frontend.service.consul.
182.32.12.4

17. yeah but we need httpd
because X
Still solutions: e.g. consul-templates

18. Deploying to prod
Safely
Quickly
Often

19. Cfgmgmt tools
Run every X minutes or on demand
Imperative vs declarative
One tool launches another
Event driven tools

20. CI systems
Not on your laptop
Common view on how to build and run code
Config them as code - get them stateless
Plays nicely with cfgmgmt

21. Runtime
Need version X of Y or Z of Y
How to test on those runtimes?
Containers to the rescue!
Not only docker:
lxc systemd-nspawn cri-o chroot?
or just bundle the JVM you need
oh you know everyone uses go now -- single
binaries -- everything included -- html static
files as well -- its called cloud native :)

22. Where to run it ? - on prem
Need a VM? -> Create VM
New machine? -> kickstart

23. Bare metal installation

24. Where to run it? not on prem
Need more power? -> Come on we have power
Not enough? -> Cloud has more
Wanna automate? -> terraform
resource "aws_instance" "example" {
ami = "ami­255899831"
instance_type = "t2.micro"
}

25. How to scale / distribute more
...
Coz of course all of the above is not enough
for you ...
Kubernetes
Mesos
Nomad

26. More is going on :)
Serverless .. because I do not want to compile
my golang myself :)

27. Monitoring tools
Lots have evolved to be more flexible
Chose between pull and push
The new Metrics model

28. We have been doing this for so
many years.

29. So much power
Creative Commons Attribution-ShareAlike 2.0
https://www.flickr.com/photos/spanginator

30. What did we fail??

31. DevOps: a definition
Culture
Automation
Measurement
Sharing
(Damon Edwards and John Willis, 2010 http://devopsdictionary.com/wiki/CAMS)

32. Lots of people just get the
"automation part"

33. The DevOps
Are you a devops?
Devops engineer
You know everything
Replace the wall of confusion by a devops
team of confusion

34. The expectations
You can work fast (read: day and night)
Your work is always super generic even if you
do not have the time to do it properly
No bug ofc
Autoscale and autoheal
Oh and during day and night you write doc

35. The Cloud
Oh we don't need the cloud we just bought
xxxK of hardware
Ok let's go for the cloud but do not tell anyone
Ok let's go for the cloud so we do not need ops
Ok let's go for the cloud tomorrow
Ok let's go for the cloud but let's keep our DB
internally

36. The NoOps
Because everyone knows how to tune DB,
package RPM files and
java.lang.NoSuchFieldError
Also you are not expected to take holidays

37. Bash
People still think bash is easy
And that easy is the most important thing out
there
Come and try to read my bash scripts from 3y
ago
Bash is not automation
Who needs package managers?

38. Leadership
What salespeople want
What tech leads want
What devs/ops/dba/... want
Please talk to each other!

39. The PoC
Cloud and automation help us create so called
PoC
Yeah now that there is a stupid PoC it means
you can go live tomorrow right?

40. Exceptions all over the place
Customer A wants this. OK.
Customer B wants this button in yellow. OK.
Customer C wants this other button is blue.
X stacks to manage, completely different...

41. 3rd party software
We want everything!
It must be open source free
We do not have time to contribute
Please a permissive license
Must work now. Bugs fixed now.

42. Where to find info?
Mailing lists
Groups
Blog posts
Slack
IRC
Websites
...

43. Choice of the tooling
And where to run it
State State State everywhere
Hello Stateful pods
Tools that takes configuration from REST api's
But don't understand CRUD
Still everyone is enthusiast about them

44. CI systems
Not automated = full of black magic
No one cares = Always red
Not enough resources = let's just stop those
jobs
CI servers are often in a dev environment
where thes should be considered prod

45. The environment
Let's build dev in the cloud
Have 5 services for acc on 1 server
Have the 100 prod services on 10 servers
And call it CI

46. Monitoring
Still today lots of people are not considering
monitoring before go live
Then you just get minimal technical
monitoring
How's your business doing?

47. Queing systems
Awesome technologies - yet underused in lots of
places
Do no try to do things synchronously if not needed!

48. About the data...
Databases migrations are awesome
Does not mean throw plain SQL files into
liquibase
Same migration for dev/staging/prod ..

49. Ridiculously complex install
procedure
Upon installing you must first touch those 4
files then remove that one and check by grep
that service is started correctly
Seriously? It's your software.
Update not only for security, also for bugfixes
and stability

50. Conclusion

51. Tools not toys
A 3 people team can not learn and know dozens of
new products/projects.. KVM CentOS Ubuntu
Openstack Kubernetes Gluster Foreman Puppet
Ansible Mcollective Apache Nginx Cassandra
Prometheus Icinga Terraform Go Java Python C
C++ Perl
Put people first!

52. Improve your own codebase!
You deploy more often than you thinl
Do not underestimate the time lost by badly
designed software
Take time to improve the codebase piece by
piece
Look back at 10+ years of config management
and build your tools with that in mind!

53. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact