Fed up with stop and go in your data center? Why not shift into overdrive and pull into the fast lane? Learn how AutoScout24 are building their Autobahn in the cloud to become the market leader in Europe's vehicle classified business.
Reinventing themselves by making a radical transition from monoliths to microservices, from .NET on Windows to Scala on Linux, from data center to AWS and from built by devs and run by ops to a devops mindset.
While the current stack keeps running, ever more microservices will go live as you listen to stories from the trenches.
Key takeaways from this talk includes: How to...
… become cloud native
… evolve the architecture
… create “you build it you run it” teams
… involve business people in the transformation
Created and presented together with Wolf Schleger (ThoughtWorks)
4. Baseline AutoScout24 IT
Highly optimized, but of last decade
IT platform supported growth for >6 years
Microsoft oriented stack
Enterprise IT setup - MTBF over MTTR
Proven agile and lean principles
C
13. Strategic Goals
Goals of the business side
Architectural Principles
High-Level Principles
Design and Delivery Principles
Tactical measures
Reduce Time to Market
Speed, Fast Feedback
Cost Transparency
Collect metrics to allow decisions cost vs. value.
Support Data-Driven Decisions
Listen to users and validate hypothesis.
Provide as many relevant metrics & data as possible.
You build it, you run it
Responsibility to run and maintain a product stays with the
building team. Fast feedback from live and customers helps us to
continuously improve.
Organized around Business Capabilities
Build teams around products not projects. Follow the domain and
respect bounded contexts. Inverse Conway Maneuver
Shared Nothing
Avoid shared infrastructure and tight coupling as much as
possible. Don’t create the next monolith.
Macro and Micro Architecture
Clear separation. Autonomous micro services within the rules
and constraints of the macro architecture.
AWS First
Favor AWS service over managed service, over self-hosted OSS,
over self-rolled solutions.
Data-Driven/ Metric-Driven
Collect metrics from processes and applications. Analyze, alert
and act on them.
Eliminate Accidental Complexity
Strive to keep it simple. Focus on essential complexity. No silver
bullet.
Event Sourcing and Publishing
Keep history of state changes and publish application events.
Autonomous Teams
Make fast local decisions. Be responsible. Know your
boundaries. Share findings.
Continuous Delivery
Deliver changes reliable, often and fast.
Infrastructure As Code
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.
DevOps Culture
Developers and Ops work together in collaborative teams as
engineers. No silos.
Be Bold
Go into production early. Value monitoring over tests. Recover
and learn. Optimize for MTTR not MTBF.
Security & Data Privacy
Security must be first class citizen and everybody’s concern.
Keep data-privacy in mind.
CC
Another goal
Work in progress...
25. DynamoDB as integration database?
Two kinds of coupling
Payload and connectivity
Payload is DynamoDB agnostic
DynamoDB as technical service contract
WW
26. From documents to events
Refactoring toward deeper insight
From CRUD to sync
Offline first
Writes are expensive
USD 20 over USD 500
CC
27. How many layers to estimate a price
Evolving architecture
R backend and Play application
Single Play application
Play backend and Play web server
Long feedback cycles
Frequency of change
W
CC
28. How to build autonomous teams
Do not fall back into old behaviours
Beware of Mandelbrot teams
Pager duty so that you run it
Part-time ops not working
Not all T-shapes are the same
Wolf
WW
29. Infrastructure guild
Agree on things to do
Share learnings
Delegate implementation to teams
Empty backlog should be normal
How about infrastructure product teams?
Mind the Shirky Principle CC
32. Focus sliders (cont.)
Cash stack meets shiny new stack
One company
Lights on in cash stack
Feature freeze
Where to build new features?
Ease of integration helps business people CC
34. Attributions
Blue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPG
A Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpg
http://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpg
GREG
EINRAD
Amazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952
BERGSTEIGER
Barber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpg
Wide objectives by Kivela (Own work) [Public domain]
href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpg
Transformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg
35. Attributions (cont)
Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpg
Principle of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0]
http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPG
Traffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547
Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0]
http://commons.wikimedia.org/wiki/File%3APellets.jpg
Pipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359
Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpg
Mille crêpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpg
Country Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpg
Puzzling by Bernd Gessler (Own work) [CC BY-SA 3.0] https://commons.wikimedia.org/wiki/File%3APuzzling.JPG
36. Attributions (cont)
Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg
7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by
tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpg
England rugby team 1905 by Russell & Sons (The Graphic) [Public domain or Public domain]
http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpg
Wandergeselle by Sigismund von Dobschütz [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPG
Faber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0]
http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPG
Wheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpg
GuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpg
AtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.png
Spare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg
Hinweis der Redaktion
Meetup AutoScout24
Shifting gears
How we build our services
Event Sourcing
DynamoDB as Atom Feed
From documents to events
Evolving architecture
PriceEstimation
Shared nothing?
Infrastructure
CSV
Log processing
How we organize ourselves
Autonomous teams
Infrastructure guild
Focus sliders
Printed ads in paper analogy. Bold parts are in progress
Last decade: PC-Web, Data center
Microsoft oriented stack: Team Windows, Team Linux
Availability = MTBF / (MTBF + MTTR)
Scout24 was sold.
New CEO Greg Ellis.
Are you ready for the future?
21st century internet company?
We could hire good .NET devs, but mostly from banks, insurances companies
No internet background; we need to teach
Talents that could help us where sparse
We started think about our ecosystem and the flywheel that drives it.
.NET small, slow
Linux/JVM larger, faster
“Fly at the speed of fear” - Disruptive
Japanese dragon: Flying beast
Rollercoaster: Sixflags Magic Mountain
Started Nov. 2014 with one team, now at 4 teams.
.NET/Windows -> Scala/Linux
Own Data centers -> AWS
Monolith + Swimlanes -> Microservices
Dev + Ops -> Engineers
Product: Include business, find core and add some polish -> Shiny new cut
This is for reference only.
Make your own.
We use this to guide discussions.
Poster hangs in every room.
The items on the right support items on the left.
New strategic business goals are coming. Discussions.
Organized around Business Capabilities
Build teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver
You build it, you run it
Responsibility to invent, run and maintain a product stays with the building team. Fast feedback from live and customers helps us to continuously improve.
Be Bold
Go into production early. Value semantic monitoring over tests. Recover and learn. Optimize for MTTR not MTBF.
Macro and Micro Architecture
Clear separation. Autonomous micro services within the rules and constraints of the macro architecture.
Uber Architecture - 1s
Shared Infrastucture vs. Shared Nothing
Meeting avoidance movement
Minimize coordination necessary
Availability over Shared nothing -> Monitoring, Logging, required, Macro, enables troubleshooting and correlation across service boundaries,
We share infrastructure via AWS platform
GoCD optional, convenience offering, but how to e.g. improve cycle time.
No side effects. Dashing, global shared state. -> Solution own dashing server
Fast local decisions over committees
Respect family ties
OneScout over XScout24
What to share, classified service und taxonomy. CSV as a service
How many environments: Two environments. Integration on prod. Be bold.
Fakes on dev.
CDCs as glue
Use over re-use, re-use only after hardening
Common helpers: shared library vs. copy & paste vs. OSS
Size matters: copy & paste ok for 10 classes
When to start OSS: on demand or speculatively?
Logging as OSS?
Shared lookup data as CSV via S3, not as a web service
Provisioning of CSV is part of the service.
Shared nothing
Jigsaw as lightweight as possible, not: frontend monolith, portal, behaviour
Currently one jigsaw. Could be run per service.
No shared asset pipeline. “Bring your own asset” (aka asset Brotzeit).
Autonomous teams
Microservices imply frontend integration
Services include UI
One domain
Only www, https only
Subdomains
Local storage sandboxed by origin (protocol+host+port). Breaks watchlist, last search etc.
Cookies are ok
High optimisation
Google pagespeed
Caching
Pages
are accessible via (localised) URL
could include fragments
are owned by one team
could be cacheable
Fragments
are parts of a page
don’t know the original request
should send cache headers
should behave well
Assets
Needs to have a versioned URI
Should be combined and minified
/assets/{servicename} uri
(normally: /{servicename}/*
Caching
CloudFront Caching: Caching on edge locations. Respects Cache Headers from Jigsaw.
PageSpeed Caching: Caches combined assets.
Backend Caching: Respects Cache Headers from microservices.
Design decision: No queries against DC. Data needs to be pushed into AWS
Ability to replay all events from beginning of time.
Events = All changes to a classified
Just a summary.
Skip:
SQS + S3 Inspired by ImmobilienScout24
Kinesis + S3 Feedback from AWS
Kinesis + DynomaDB Queryable, Storage costs not prohibitive (7x)
SQS + DynamoDB Queue better than LATEST or TRIM_HORIZON
Proxy + DynamoDB Changes are collected via queue table in Oracle
DynamoDB DynamoDB is global service
No Integration DB
Only the owning service writes.
Bounded contexts respected.
DynamoDB as public API of service.
Think publishing events into queue
More consumers anticipated.
DynamoDB story
Secondary global index expensive
Throughput limit by shard
DynamoDB is no document store, think about your data model and use cases.
API to integrate with existing local storage implementation lead to think about syncing changes.
Rediscovered offline first -> change/events
Add/Remove/List -> Sync -> Almost Event Sourcing
Do the right thing
API followed evolution. Don’t be afraid of breaking changes. Don’t stick to legacy design.
Timestamp vs SequenceID
Writes: DynamoDB has an asymmetric provisioning model.
Read to write costs 1:40 with eventual consistent reads
Best write minimization: Write only changes = events
How many layers to estimate a price.
Evolving architecture
R backend and Play web server, Polyglot technologies.
All in one web server / layer
Separate Play backend and Play web server, driven by:
Long feedback cycles for backend
Frequency of change of backend (weeks) and frontend (hours)
The datadevops team (aka Interdisciplinary++)
One board, one repo, R-Unit
How to build autonomous teams
Do not fall back into old behaviours
Beware of Mandelbrot teams - dev team and ops team within the devops team
Part-time ops not working
Some devs do not like ops (and the other way round) Not all Ts are shaped the same.
Pager duty enforces you build it you run it behaviour.
How not to ignore the pager, no broken window syndrom
On call rotation
Name and shame
Why infrastructure guild?
Agree on things to do
Share learnings
How we work:
Work is done in the teams
Focus work on infrastructure needed by team.
(Macro decision: Shared infrastructure: Logging, Monitoring, Security)
Teams are responsible - Subsidiarity
How to handle long running/blocked stories?
Keep delegating and resist temptation to take over
Don’t treat infrastructure story as neglected child
Empty backlog should be normal
Hurdle to create new tasks.
We want as few shared infrastructure as possible.
Shirky: Institutions will try to preserve the problem to which they are the solution.
Creates it own tasks.
Risk of stories without value or prioritisation.
“I have found an old sticky on the floor, let's implement that for two weeks!”
Focus Shift, PO missing
Focus slider - AWS
20% product + 80% platform - first 2 months
50% product + 50% platform - ideally within next 3 months
Current state: 70% product + 30 % platform
Vision: 80% product + 20% platform
War story: Beware spontaneous fluctuations in PO presence. (Compare priceestimation)
Delivery over learning
Learning extends IT-ness
Enter Tatsu 3
Change focus to delivery
Tatsu 2 continues
Focus slider - Business
Don’t split into shiny new stack and legacy.
Legacy is not used as a word anymore.
Lights on in Cash stack - AWS
Feature freeze: Where to build new features?
Ease of integration between new and old stack helps business people to switch fluently between both worlds.
Where to draw the line - Focus on technology change - Minimum viable integration
Time to market vs. fast re-platforming (N+1 systems)