We all hear the term "DevOps" being thrown around on a daily basis, but what does it actually mean? With a little help from everyone's favourite 80's action hero, we'll undergo a whistle-stop tour of the philosophy, culture and tooling behind this buzzword, specifically aimed at Java developers. We'll also look at a real-world case study from Instant Access Technologies Ltd, and explore the key role that DevOps has played during a successful upgrade of the epoints customer loyalty platform to support increasing traffic.
The core discussion will focus on the challenges encountered as we moved from a JVM-based monolithic app deployed into a data centre on a 'big bang' schedule, to a platform of loosely-coupled components, all being continuously deployed into the Cloud. We will conclude the talk by recommending a learning path for Java developers who are interested to learn more about DevOps.
5. Core Changes…
• Service-Oriented Architecture
– Twitter’s Story (bit.ly/1j1WbmI)
• Cloud-based deployments
– “Cloud DHARMA”, 7th May at Skillsmatter
• DevOps Culture
– Today!
@taidevcouk14/03/2014
6. What is DevOps?
• “…a software development method that stresses
communication, collaboration and integration
between developers and IT professionals.”
• “…aims to help an organization rapidly produce
software products and services”
http://en.wikipedia.org/wiki/DevOps
14/03/2014 @taidevcouk
7. What’s in a Name?
• “Development / Operations”
• Increasing cohesion between:
– Business
– Development
– Quality Assurance
– Operations
“Bu-Dev-Qa-Ops”?
14/03/2014 @taidevcouk
11. Chuck Norris doesn’t need DevOps…
…as a one-man army he codes with
one hand, tests with the other and
deploys with his beard
14/03/2014 @taidevcouk
The rest of us…
…work in teams to develop software
20. Culture is Vital
• Culture drives behaviour, drives culture…
– “Communication, simplicity, feedback, courage”
– Everyone is responsible for delivery
– Continuous experimentation and learning
• Not easy to change culture
– The hardest part of DevOps…
– …but you will learn new things
14/03/2014 @taidevcouk
21. Changing Culture
• Greenfield
– Flickr’s story (slidesha.re/sHpYV)
– “Why other people don’t get it”
• Sandro Mancuso (slidesha.re/1bcStpe)
• Enterprise
– “The Pheonix Project”
• Gene Kim et al
14/03/2014 @taidevcouk
22. Changing Culture
• Create an effective team…
• “Habits of highly effective technical teams”
– Martijn Verburg (bit.ly/1aF9SnK)
• “Patterns of Effective Teams”
– Dan North (vimeo.com/68226771)
14/03/2014 @taidevcouk
23. Chuck Norris doesn’t do iterative
development…
…all applications Chuck Norris creates
are right first time, every time
14/03/2014 @taidevcouk
The rest of us…
…need to enable agility (for both the
business and technical teams)
24. We all do Continuous Integration, right?
• Continuous Deployment/Delivery
– Jez Humble and Dave Farley
– Great InfoQ Video (bit.ly/XugWi8)
• Create a “build pipeline”
– Goal is fast feedback
• Continuous Deployment
– Production?
14/03/2014 @taidevcouk
26. Our Build Pipeline
• Component Build
– Compile
– Unit Tests (surefire)
– Integration Tests (failsafe)
• Deployment onto QA Cloud
– Python Scripts + Chef to provision
– Verify success using Python
14/03/2014 @taidevcouk
27. Our Build Pipeline
• Acceptance Tests
– Cucumber and Selenium
– Work in progress…
• Performance Tests
– Jmeter
– Jenkins Jmeter performance plugin
• Staging / Live Deployment
– Human-based conditional operation
14/03/2014 @taidevcouk
28. Gotchas
• Managing dependencies in SOA is hard, very hard
• Branching
– Gitflow, Branch Per Feature, or Mainline
• Migrating data can be challenging
– Liquibase / Flyway
– MongoDB / Solr Schema versions in data
– Wooga case study (bit.ly/1egArDC)
– Shout out to Daniel Josefsson @ Shazam for tips!
14/03/2014 @taidevcouk
29. Chuck Norris doesn’t do QA…
…Chuck Norris can test an entire
application with a single assert
(and get 110% code coverage)
14/03/2014 @taidevcouk
The rest of us…
…need high-quality automated QA
30. Automating QA
• Unit testing is essential
• Intra-component integration testing
– Spock is awesome (code.google.com/p/spock)
– Utilise embedded datastore/middleware
• Inter-component integration testing
– The hardest part of SOA…
14/03/2014 @taidevcouk
32. Automating QA
• Make it easy for everyone to execute
• Include within the build pipeline
• Make people care – fail the build!
• “Agile Testing” by Lisa Crispin, Janet Gregory
14/03/2014 @taidevcouk
33. Chuck Norris doesn’t need an OS…
…his keyboard has two keys, 0 and 1
14/03/2014 @taidevcouk
The rest of us…
…need to provision bare metal, and
also be comfortable with the OS
34. Say No To Snowflakes!
• Infrastructure as Code
– Version control everything
• Automate all provisioning
– Chef, Puppet, SaltStack, Python, AWS CLI
• Play with Vagrant (www.vagrantup.com)
– “providers” are super cool
14/03/2014 @taidevcouk
35. Thinking/Acting Like A Sysadmin
• Learn Linux fundamentals
• Diagnostic skills
– top, iotop, iostat, netstat, vmstat
– Java utils: jps, jstat, jmap, jhat
– “DevOps Troubleshooting” by K. Rankin
• Maybe grow a beard…
14/03/2014 @taidevcouk
36. Chuck Norris doesn’t fail…
…he just finds a new way in which
reality is broken
14/03/2014 @taidevcouk
The rest of us…
…should plan for failure
37. Failure
“Everything fails all the time [in the cloud]”
Werner Vogels, CTO, Amazon.com
• 21st Century Application Architecture
– www.skillsmatter.com (bit.ly/10jAdSV)
14/03/2014 @taidevcouk
38. Black Swan Theory
“…is a metaphor that describes an event that
comes as a surprise, has a major effect, and is
often inappropriately rationalized after the fact
with the benefit of hindsight”
Nassim Nicholas Taleb, The Black Swan
http://amzn.to/1gaIqxz
14/03/2014 @taidevcouk
39. Failure
Pre-Cloud we designed for success…
and handled failure.
With Cloud we design for failure…
and handle success.
Credit to John Bassil for inspiration here! (@johnbassil)
14/03/2014 @taidevcouk
40. Design for Failure
• Design patterns
– Asynchronous communication
– Timeouts / retries
– Bulkheads / circuit-breakers
• Inspiration
– Chris Richardson (slidesha.re/1ft3vsg)
– Netflix (bit.ly/1h5GMid)
14/03/2014 @taidevcouk
42. Antifragile
• The opposite of fragile?
– Robust…
– Antifragile…
• Netflix are best-in-class
– bit.ly/1gs5n3q
• System must be robust first!
14/03/2014 @taidevcouk
43. All arrays Chuck Norris creates are of
infinite size…
…as Chuck Norris knows no bounds
14/03/2014 @taidevcouk
The rest of us…
…should manage our resources and
cultivate ‘mechanical sympathy’
44. Mechanical Sympathy
• Be aware of deployment platform properties
• …especially if in the cloud
– Block Storage IOPS (100 vs 49k)
– 1Gbps Network ( < 125MB/s vs > 400MB/s)
– “Noisy neighbours”
• Monitor everything (more on this later)
14/03/2014 @taidevcouk
45. When Chuck Norris throws Exceptions…
…everybody knows about it because they
land outside of the data center
14/03/2014 @taidevcouk
The rest of us…
…should log all errors (and other vital
information for diagnostic purposes)
46. Logging
• Log pretty much everything
– Use appropriate levels
• Make comments searchable/machine readable
– Good tips (bit.ly/hweqm4)
• Use centralised logging
– Especially if in the cloud
– Logstash, Loggly, Papertrail
14/03/2014 @taidevcouk
47. Chuck Norris doesn’t worry about
application downtime…
…Chuck Norris’ production servers are
so scared they constantly ping him
14/03/2014 @taidevcouk
The rest of us…
…should monitor everything
48. Monitor All The Things!
• Infrastructure monitoring
– Nagios
– Zabbix
– Splunk
– AppDynamics
14/03/2014 @taidevcouk
55. In Summary…
• DevOps is driving Agile/XP into QA and Ops
• Faster, leaner and more effective software
– The ability to rapidly experiment is awesome!
• There are some real benefits behind the buzz
• Now is the time to step-up as a developer
14/03/2014 @taidevcouk
56. The Developer’s DevOps Action Plan
• Think about your company culture
• Explore continuous delivery
• Learn Linux basics
• Automate provisioning
• Design for failure
• Cultivate mechanical sympathy
• Improve logging/metrics
14/03/2014 @taidevcouk
57. Thanks For Listening
• Massive thanks to all the IAT team
– DevOps gurus, Jamie Clarkson and Anjani Phuyal
• Questions / comments?
– d.bryant@iatltd.com
– @taidevcouk
• Devoxx UK 12-13th June
– www.devoxx.co.uk
14/03/2014 @taidevcouk
Hinweis der Redaktion
organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizationsFocus on flow: translate business ideas to a working service efficiently by examining how fast artifacts and work flows through the development lifecycleAmplify feedback loops: quickly learn about the system by seeing through fast feedback on what happens to results of changes
James Gough’s “The benefits are more than just the tests”Mash Badar’s “TDD at Scale” (slidesha.re/19P7kzS)
Jackie Stewart, 3 times F1 world champion, coined the phrase “Mechanical Sympathy” as a term for the driver and the machine working together in harmony. This can be summarised in that a driver does not need to know how to build an engine but they need to know the fundamentals of how one works to get the best out of it.