Experiments and A/B tests are at the heart of innovation at Twitter. A culture of experimentation empowers the organization to try new things with minimal downside. Twitter uses Confluence and JIRA to manage hundreds of experiments at any given time. Learn Twitter's philosophy on experimentation, how we enhance organizational learning, and uncover deeper insights that help everyone become smarter with every new experiment.

1. #atlassian

2. Scaling Product Experimentation
with JIRA and Confluence
ZANE SALIM • SENIOR TECHNICAL PM • TWITTER • @ZANES

3. Technology is wiping out
companies faster than ever

5. Companies that stop
innovating die… fast

6. How are you
innovating?

7. 1. Our approach @twitter
2. Challenges at scale
3. How we scaled
4. What got better
5. Final takeaways

8. 1. Our approach @Twitter

9. @Twitter we innovate
through experimentation

10. Anyone can launch
an experiment

11. Unleashes creativity

12. Drives user
insights at scale

13. 100s every quarter

14. Experimentation is the new black

15. … and the old black
“Negative results are just what I want. They’re just as valuable to me as
positive results. I can never find the thing that does the job best until I find
the ones that don’t.”
― Thomas Edison, Harper’s Magazine (Vol. 165, p 405)
“We must conduct research and then accept the results. If they don't stand
up to experimentation, Buddha's own words must be rejected.”
― Dalai Lama XIV, A Policy of Kindness (p 31)
“There must be some ability to experiment to advance the state of the art…
In the early days of aviation there was a great deal of experimentation and a
high death rate. We don't want that … but by the same token we can't have
a situation where no deaths are ever allowed, because that would put
innovation in a coffin too.”
– Elon Musk, L.A. Times interview (August 1, 2012)
“There’s a wonderful thing about learning as a group. A fundamental part of
that is making mistakes together. There’s no learning without trying lots of
ideas and failing lots of times.”
― Jony Ive, Telegraph interview (May 23, 2012)

16. Both refinement and exploration

17. Example: which drives more clicks?

18. Example: which drives more retention?

19. 2. Challenges at scale

20. Visibility

21. Learning

22. Collaboration

23. Rigor

24. 3. How we scaled

25. COMMON JIRA
WORKFLOW FOR
COLLABORATION

26. Lifecycle of an Experiment

27. Lifecycle of an Experiment in JIRA

28. Lifecycle of an Experiment in JIRA

29. KNOWLEDGE
MANAGEMENT IN
CONFLUENCE

30. Confluence: Knowledge Management

31. Confluence: Experiment Brief Templates

32. Confluence: Experiment Brief Templates
Experiment Brief
Experiment Write-up
Retrospective Analysis
Describe the problem
Describe the experiment
Present retrospective analysis
Define a hypothesis
Present the results
Present the results
Design the experiment
Offer take-away’s
Offer take-away’s

33. Confluence: Reviewed Experiments & Learning

34. HYPOTHESIS
TREES

35. Visualizing Focus Areas & Insights

36. 4. What got better

37. Doubled completed
experiments

38. Reduced cycle
time up to 50%

39. Learning org à
smarter experiments

40. 5. Final takeaways

41. Experimentation
drives innovation

42. Collaboration &
learning essential

43. Smart use of tools can
make you 4x more effective

44. Thank you!
ZANE SALIM • SENIOR TECHNICAL PM • TWITTER • @ZANES

45. Example: <made up example experiment>