Experimentation is at the core of today’s most successful software products, from Amazon to Google to Facebook to Netflix. These companies use A/B testing and controlled rollouts to de-risk development and measure the impact they’re making with new ideas. Join Product Manager Jamie Connolly to learn how Optimizely Full Stack enables any team to develop products like top companies--with experimentation-driven product development. -Reduce the risk of launching new features -Apply experimentation to your development process with Optimizely X Full Stack -Prove the impact of every new feature or user experience change

1. Experimentation-Driven
Product Development
Robin Pam
Sr. Product Marketing Manager
Optimizely
Product Manager
Optimizely
Jamie Connolly
With Optimizely X Full Stack

2. Robin Pam
Sr. Product Marketing Manager
Optimizely
Product Manager
Optimizely
Jamie Connolly

3. Agenda
● Why Experimentation Matters
● What Is Experimentation-Driven
Product Development?
● How to Build an Experimentation
Platform
● Full Stack Demo

4. • We are recording today’s presentation
• You will receive a copy of the slides after the webinar
• Please submit questions via the text box
Housekeeping

5. Innovation in the physical world can take decades

7. 3 YEARS LATER

9. Innovation by the hour

10. At any given point in time, there isn't just one version
of Facebook running, there are probably 10,000.
Mark Zuckerberg
“
“

12. A B

13. A
B

15. 9,800Google search experiments in 2016
www.google.com/search/howsearchworks/

16. 1,653Google search launches in 2016
www.google.com/search/howsearchworks/

18. I haven’t failed.
I’ve just found
10,000 ways that
won’t work.”
Thomas Edison
“

19. Experimentation-Driven
Product Development

20. medium.com/netflix-techblog

21. uber.com/blog/colorado

22. LAUNCHBUILD
DESIGN
PRAY
EXPERIMENT
ITERATE

23. LAUNCH
BUILD
DESIGN PRAY
EXPERIMENT
ITERATE

24. LAUNCH
BUILDDESIGN PRAY
EXPERIMENT
ITERATE

25. LAUNCHBUILDDESIGN
PRAY
EXPERIMENT
ITERATE

26. LAUNCHBUILDDESIGN PRAY
EXPERIMENT
ITERATE

27. DESIGN BUILD EXPERIMENT ITERATE
LAUNCH
PRAY

28. How to build an experimentation
platform that is fast and scales

29. // ab testing is
easy
if random() < 0.5:
var = ‘a’
else:
var = ‘b’

30. PlanOut
SixPack
Proctor
phpA/B
Convert
Airlock
Genetify
FeatureBee
Abba
AlephBet
abclub
Confidence
FluidFeatures
Gertrude
Rollout
Split
Trebuchet
Vanity
Wasabi

31. from planout.experiment import SimpleExperiment
from planout.ops.random import *
class FirstExperiment(SimpleExperiment):
def assign(self, params, userid):
params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid)
params.button_text = WeightedChoice(
choices=['Join now!', 'Sign up.'],
weights=[0.3, 0.7], unit=userid)
my_exp = FirstExperiment(userid=12)
# parameters may be accessed via the . operator
print my_exp.get('button_text'), my_exp.get('button_color')

34. medium.com/netflix-techblog

35. medium.com/netflix-techblog

36. Analytics

37. medium.com/netflix-techblog

38. medium.com/netflix-techblog

39. eng.uber.com/data-viz-intel/

40. Traffic Splitting Remote Configuration Analytics
Traffic Sampling
Audience Targeting
Mutual Exclusion
State Persistence
Multivariate Treatment
Global Holdout
QA Tool
Automation
Whitelisting
Logging
Permissions
Feature Toggles
Feature Rollouts
Kill Switch
Scheduling
Staging Environments
Audience Definitions
Webhooks
REST API
Audit Trail
Event Dispatching
Event Storage
Sessionization
Statistical Methods
Metric Definitions
User Aliasing
Segmentation
DW Integration
Notifications
Visualizations
3 key ingredients for robust experimentation

41. 1. Trust
“How can I trust the experiment was set up and tracked correctly?”
2. Process
“What are the best instrumentation practices to minimize technical debt?”
3. Maintenance
“The engineer who built this left the company… now what?”
Three key challenges with experimentation

42. Demo

43. Full Stack Use Cases
Feature Flags Performance
Optimization
Algorithms
User Experience Roll outs Pricing
$$
Marketing Channels

45. How many experiments do you run per year?
How many changes do you launch per year?

46. Your Product
Roadmap is your
Experiment Roadmap

47. Join our next webinar:
Getting Started with
Server-Side Testing
December 13, 2017
https://optimize.ly/serversidewebinar

48. Q&A
Robin Pam Jamie Connolly