Matej Konecny discusses how his team at Atlassian manages their microservices architecture. In the early days, their applications were built as WAR files running on EC2 with autoscaling and logs stored in CloudTrail, which made monitoring and incident handling difficult. They have since implemented a unified PaaS across Atlassian using Docker Compose with automated provisioning of containers and resources. This common platform enforces best practices and allows sidecars to be reused. Incident detection and communication is now automated through tools like OpsGenie, and changes are easier to track. Regular meetings cover metrics, alerts, and code health to improve oversight of services.

1. MATEJ KONECNY | SENIOR DEVELOPER | ATLASSIAN
Herding Microservices
The Atlassian Way

2. IT dev team
~ 15 microservices
3 continents
MY TEAM

3. My team's mission:
Deliver great help experience to our
customers

4. To achieve our goal,
we need to
interconnect
many services
research.archives.gov/description/1633445

5. Back in the old
days...

6. Our early days...

7. Our early days...
EC2
Applications were built as
WARs and running on EC2/
EBS. We've used
autoscaling.

8. Our early days...
EC2
Applications were built as
WARs and running on EC2/
EBS. We've used
autoscaling.
Logs
In CloudTrail. Difficult to
search for clues.

9. Our early days...
EC2
Applications were built as
WARs and running on EC2/
EBS. We've used
autoscaling.
Logs
In CloudTrail. Difficult to
search for clues.
Monitoring
Only whatever is built-in
AWS like CloudWatch.

10. Incidents
were
chaos
https://www.aair.biz

11. We treated
our services
like pets
https://www.flickr.com/photos/presidioofmonterey/43804939880

12. We now have a
unified PaaS
across all
Atlassian
PLATFORM

13. Cattle, not pets

14. Provisioning
Automatically
provision containers
AND resources.
Cattle, not pets

15. Provisioning
Automatically
provision containers
AND resources.
Technology
Built on top of Docker
Compose.
Integrates with CI/
CD.
Cattle, not pets

16. Provisioning
Automatically
provision containers
AND resources.
Technology
Built on top of Docker
Compose.
Integrates with CI/
CD.
Reuse
Sidecars can be
reused in other
services, e.g.
monitoring.
Cattle, not pets

17. Provisioning
Automatically
provision containers
AND resources.
Technology
Built on top of Docker
Compose.
Integrates with CI/
CD.
Reuse
Sidecars can be
reused in other
services, e.g.
monitoring.
Best practices
Enforce the best
practices before
deployment.
Cattle, not pets

18. INCIDENT HANDLING TODAY
Detect the
problem
Automatic
escalation to
service owner
Check the
changes globally
Rollback, turn off
feature or hotfix
Post Incident
Review
T+0 T+5 T+10 T+X Later

19. Detect problems
automatically
Have an andon
cord as a backup
DETECTION

20. INCIDENT HANDLING TODAY
Detect the
problem
Automatic
escalation to
service owner
Check the
changes globally
Rollback, turn off
feature or hotfix
Post Incident
Review
T+0 T+5 T+10 T+X Later

21. Define the service
tiers
Use tools to
communicate
ESCALATION

22. INCIDENT COMMUNICATION
DETECTION

23. HOT ticket raised
INCIDENT COMMUNICATION
DETECTION

24. HOT ticket raised
INCIDENT COMMUNICATION
DETECTION OpsGenie auto-page

25. HOT ticket raised
INCIDENT COMMUNICATION
DETECTION
PAGE ACCEPTED
OpsGenie auto-page

26. HOT ticket raised
INCIDENT COMMUNICATION
DETECTION
PAGE ACCEPTEDInvestigation
OpsGenie auto-page

27. HOT ticket raised
INCIDENT COMMUNICATION
Zoom/Slack/Statuspage
DETECTION
PAGE ACCEPTEDInvestigation
OpsGenie auto-page

28. HOT ticket raised
INCIDENT COMMUNICATION
Zoom/Slack/Statuspage
DETECTION
PAGE ACCEPTEDInvestigation
OpsGenie auto-page
Resolve

29. INCIDENT HANDLING TODAY
Detect the
problem
Automatic
escalation to
service owner
Check the
changes globally
Rollback, turn off
feature or hotfix
Post Incident
Review
T+0 T+5 T+10 T+X Later

30. 23%
incidents caused by changes

31. Find the changes
easily
CHANGES

33. INCIDENT HANDLING TODAY
Detect the
problem
Automatic
escalation to
service owner
Check the
changes globally
Rollback, turn off
feature or hotfix
Post Incident
Review
T+0 T+5 T+10 T+X Later

34. INCIDENT HANDLING TODAY
Detect the
problem
Automatic
escalation to
service owner
Check the
changes globally
Rollback, turn off
feature or hotfix
Post Incident
Review
T+0 T+5 T+10 T+X Later

35. Weekly TechOps meeting

36. Weekly TechOps meeting
Signal vs Noise
We check that the alerts
raised are meaningful.

37. Weekly TechOps meeting
Signal vs Noise
We check that the alerts
raised are meaningful.
Check KPIs
Did the service meet all
the defined Service Level
Objectives (SLO)?

38. Weekly TechOps meeting
Signal vs Noise
We check that the alerts
raised are meaningful.
Check KPIs
Did the service meet all
the defined Service Level
Objectives (SLO)?
Code health
Analyze the test coverage
and the technical debt
backlog.

39. ?
Did it help?

40. Feels
betterDid it help?

41. What's next?

42. What's next?
Knowledge silos
Automatically measure how
well each team member
knows each service to
reduce knowledge silos.

43. What's next?
Knowledge silos
Automatically measure how
well each team member
knows each service to
reduce knowledge silos.
Service costs
Gain more insights in total
cost of ownership of each
service and new feature
built.

44. What's next?
Knowledge silos
Automatically measure how
well each team member
knows each service to
reduce knowledge silos.
Service costs
Gain more insights in total
cost of ownership of each
service and new feature
built.
Runbooks
Organize the runbooks and
make them searchable.
Run more war games (fire
drills).

45. To sum up...

46. Platform
Use common
platform and define
guiding principles.
To sum up...

47. Platform
Use common
platform and define
guiding principles.
Monitoring
Collect metrics and
aggregate the logs in
central location.
To sum up...

48. Platform
Use common
platform and define
guiding principles.
Monitoring
Collect metrics and
aggregate the logs in
central location.
Standards
Enforce standards
when deploying and
developing.
To sum up...

49. Platform
Use common
platform and define
guiding principles.
Monitoring
Collect metrics and
aggregate the logs in
central location.
Standards
Enforce standards
when deploying and
developing.
Learn
Incorporate learning
from the incidents.
They are invaluable!
To sum up...

50. MATEJ KONECNY | SENIOR DEVELOPER | ATLASSIAN
Thank you!