Atlassian transitioned to becoming a more data-driven organization by forming a Product Growth Team to discover value in customer behavioral data and champion data-driven decision making. They built analytics infrastructure to collect and analyze large amounts of customer event data. Some challenges included educating people that analytics is about more than just counting, building new infrastructure to replace silos and improve data quality, and overcoming reluctance to move away from gut feelings. A/B testing was used to constantly improve products in a measurable way, conducting over 100 experiments last year. Overall, Atlassian applied the scientific method used in other fields like medicine to make their development more evidence-based and data-driven.

1. Data Science at Atlassian
The transition towards a data-driven
organisation
Dr Ilias Flaounas
Capital Markets CRC, Sydney
13 May 2015

3. Atlassian - Trivia
• Atlassian is an Australian enterprise software company
specialised in products that support project management
and collaboration.
• ~12 years old
• Latest valuation at $US3.3 billion in 2014, (x8 since 2010).
• >1.5K employees
• Headquarters in Sydney
• Best place to work in Australia
(The Australian Financial Review, 2014)

4. • ~43,000 enterprise
customers
• Tens of thousands
more own $10
starter and free
community
licenses

5. JIRA

6. JIRA

8. Key Requirements:!
• Support from upper
management!
• Some minimum
critical mass of
customers!
• Serious investment
in infrastructure,
people,
time

9. Product Growth Team
• Team was formed ~2 years ago
• Main mission: growth!
• Discover value in heaps of behavioural data!
• Champion data-driven decision making

10. Analytics Infrastructure (v.2)
~3-4TB overall (highly compressed)!
800GB per day produced (uncompressed)

12. We analyse events, i.e., user actions
Timestamp Instance User Event Attributes
1356958862 mySuperCompany John Login
1356968863 mySuperCompany Anne Login
1356978862 mySuperCompany John dashboard view
1356988862 mySuperCompany Anne dashboard view
1357884442 anotherCompany John view issue {issueID = 1234}
1357956862 mySuperCompany John create issue {issueID = 1000}
1357957862 mySuperCompany Anne view issue {issueID = 1000}
1357958862 mySuperCompany Anne comment issue {issueID = 1000}
1357959862 mySuperCompany John create issue {issueID = 1001}
… … … … …

13. The Challenges
• Build infrastructure!
- break silos!
- replace existing “solutions”!
- add instrumentation!
- improve data quality!
- deal with catch-22…

16. The Challenges
• Educate!
- “analytics is counting”!
- help people ask the
correct questions
• Build infrastructure!
- break silos!
- replace existing “solutions”!
- add instrumentation!
- improve data quality!
- deal with catch-22…

17. JIRA Agile!
Which Agile report is used the most?

18. However, counting is hard…
Report 1
Report 2
Report 3
Report 4
Report 5
Report 6
Report 7
Report 8
Report 9

20. No usage

21. No usage Low usage

22. No usage Low usage Medium usage High usage

23. … and beyond counting…

25. Predicting conversions
How many variables can you measure?

26. Top customers
none, n/a, home, self, test,
personal, atlassian, “-”, na, private,
abc, personal use, freelance,
my company,student,
testing, company, no company,
private, mycompany, a, no,
test company, noname, testcompany,
self employed, individual, tester, me,
“.” , independent, trial, myself, nocompany, xyz

27. Predicting conversions
• Baseline set at 50%
• Using only the sign-up information we have an
accuracy of 65%

28. The Challenges
• Convince
- challenge gut feelings
- demonstrate value
• Educate!
- “analytics is counting”!
- help people ask the correct
questions
• Build infrastructure
- replace existing “solutions”
- data quality
- catch-22…

29. The argument of common sense
How Common Sense Fails
Duncan Watts, 2011
κοινὴ αἲσθησις
Aristotle, 384–322 BCE

30. Source: http://www.tylervigen.com/
Correlation and causation

31. • Source: P. Chopra, The ultimate guide to A/B testing,
http://www.smashingmagazine.com, 2010

32. A/B Tests
• A/B tests allow a constant improvement of
products in a measurable and predictive way
• Last year we deployed ~118 experiments.
• The volume of customers, evaluators, visitors,
has become a limited and valuable resource

33. Subject: %%first_name%%,
assemble the
bitbucketeers!
Subject: Grab the
Bitbucketeers...4, 3, 2, 1
LAUNCH!

37. Do A/B tests slow down
development?
“Move fast with stable infra”
Mark Zuckerberg
Facebook F8, 2014
“Pauca sed matura”
i.e., “Few, but mature”
Carl Friedrich Gauss
1777-1855
“Move Fast and Break Things”“Move Fast and Break Things”

38. Instead of a Conclusion: We simply recycle the scientific method
from other fields, e.g., medicine…
Patient User
Medical doctor Product manager
Illness Bad User Experience
Diagnosis Interviews, surveys…
Treatment UX modification
Medical researchers Growth team
Observations Data mining
Medical trials A/B tests
Ethics Privacy Policy
Evidence-based
medicine
Data-driven
development

39. Reed
Johnson
Ilias
Flaounas
Stephen
Lee
Shaun
Clowes
Graeme
Smith
Tim
Garnsey
Fardin
Sarker
Theo
Voronov
Herman
Chow
Leo
Balan
John
Jiang
Rob
Sangster
Andrew
Wakeling
Martin
Smyrk
Kostas
Servis
Houda
Chehab
Lakshan
Perera
We want
YOU!

40. Thank you!
Dr Ilias Flaounas
!
ilias [dot] flaounas [at] atlassian.com