While A/B test is a very known and familiar methodology for conducting experiments on production when you do that on a large scale it has many challenges in the organization level and operational level. At Wix we are practicing continuous delivery for over 4 years. Conducting A/B tests and writing feature toggles is at the core of our development process. However when doing so on a large scale, with over 1000 experiments every month, it holds many challenges and affect everyone in the company, from developers, product managers, QA, marketing and management. In this talk we will explain what is the lifecycle of an experiment, some of the challenges we faced and the effect on our development process. * How an experiment begins its life * How an experiment is defined * How do you let non technical people control the experiment while preventing mistakes * How an experiment go live, what is the lifecycle of an experiment from beginning to end * What is the difference between client and server experiments * How do you keep the user experience and not confuse them * How does it affect the development process * How can QA test an environment that changes every 9 minutes * How can support help users when every user may be part of different experiment * How can we find if an experiment is causing errors when you have millions of permutations [at least 2^(number of active experiments)] * What are the effects of always having multiple experiments on system architecture * What are the development patterns when working with AB test At Wix we have developed our 3rd generation experiment system called PETRI, which is (will be) open sourced, that helps us maintain some order in a chaotic system that keep changing. We will also explain how PETRI works, what are the patterns in conducting experiments that will have a minimal effect on performance and user experience.

1. Experimenting on Humans
Aviran Mordo
Head of Back-end Engineering
@aviranm
www.linkedin.com/in/aviran
www.aviransplace.com
Sagy Rozman
Back-end Guild master
@sagyrozman
www.linkedin.com/in/sagyrozman

3. Wix In Numbers
• Over 55M users + 1M new users/month
• Static storage is >1.5Pb of data
• 3 data centers + 3 clouds (Google, Amazon, Azure)
• 1.5B HTTP requests/day
• 900 people work at Wix, of which ~ 300 in R&D

4. 1542
(A/B Tests in 3 months)

5. Agenda
• Basic A/B testing
• Experiment driven development
• PETRI – Wix’s 3rd generation open source experiment
system
• Challenges and best practices
• How to (code samples)

6. 11:31
A/B Test

7. A
B
To B or NOT to B?

8. Home page results
(How many registered)

9. Experiment Driven
Development

10. This is the Wix editor

11. Our gallery manager
What can we improve?

12. Is this better?

13. Don’t be a loser

14. Product Experiments
Toggles & Reporting
Infrastructure

15. How do you know what is running?

16. Why so many?
If I “know” it is better, do I really
need to test it?

18. Sign-up
The theory
Choose
Template
Edit site
Publish
Premium

19. Result = Fail

20. Intent matters

21. Conclusion
• EVERY new feature is A/B tested
• We open the new feature to a % of users
○ Measure success
○ If it is better, we keep it
○ If worse, we check why and improve
• If flawed, the impact is just for % of our users

22. Start with 50% / 50% ?

24. Sh*t happens (Test could fail)
• New code can have bugs
• Conversion can drop
• Usage can drop
• Unexpected cross test dependencies

25. Minimize affected users
(in case of failure)
Gradual exposure (percentage of…)
• Language
• GEO
• Browser
• User-agent
• OS
• Company employees
• User roles
• Any other criteria you have
(extendable)
• All users

26. Not all users are equal
• First time visitors = Never visited wix.com
• New registered users = Untainted users

27. We need that
feature
…and failure
is not an
option

28. Defensive Testing

29. Adding a mobile view

30. First trial failed
Performance had to be improved

31. Halting the test results in loss of data.
What can we do about it?

32. Solution – Pause the experiment!
• Maintain NEW experience for already exposed users
• No additional users will be exposed to the NEW feature

33. PETRI’s pause implementation
• Use cookies to persist assignment
○ If user changes browser assignment is unknown
• Server side persistence solves this
○ You pay in performance & scalability

34. Decision
Keep feature Drop feature
Improve code &
resume experiment
Keep backwards compatibility for
exposed users forever?
Migrate users to another equivalent
feature
Drop it all together (users lose data/
work)

35. The road to
success

36. Reaching statistical significance
• Numbers look good but sample size is small
• We need more data!
• Expand
Control Group (A)75%
50%
25%
0%
Test Group (B)
25%
50%
75%
100%

37. Keep user experience consistent
Control
Group
(A)
Test
Group
(B)

38. Keeping persistent UX
• Signed-in user (Editor)
○ Test group assignment is determined by the user ID
○ Guarantee toss persistency across browsers
• Anonymous user (Home page)
○ Test group assignment is randomly determined
○ Can not guarantee persistent experience if changing
browser
• 11% of Wix users use more than one desktop
browser

39. There is MORE than one

40. Possible states >= 2^(# experiments)
# of active
experiment
Possible # of
states
10 1024
20 1,048,576
30 1,073,741,824
Wix has ~200 active experiments = 1.606938e+60

41. A/B testing introduces
complexity

42. Support tools
• Override options (URL parameters, cookies, headers…)
• Near real time user BI tools
• Integrated developer tools in the product

43. Define
Code
Expand Experiment
Merge
code
Close

44. Define spec
• Spec = Experiment template (in the code)
○ Define test groups
○ Mandatory limitations (filters, user types)
○ Scope = Group of related experiments (usually by product)
• Why is it needed
○ Type safety
○ Preventing human errors (typos, user types)
○ Controlled by the developer (developer knows about the context)
○ Conducting experiments in batch

45. Spec code snippet
public class ExampleSpecDefinition extends
SpecDefinition {
@Override
protected ExperimentSpecBuilder
customize(ExperimentSpecBuilder builder) {
return builder
.withOwner("OWNERS_EMAIL_ADDRESS")
.withScopes(aScopeDefinitionForAllUserTypes(
"SOME_SCOPE"))
.withTestGroups(asList("Group A", "Group B"));
}
}

46. Conducting experiment
• Experiment = “If” statement in the code
final String result =
laboratory.conductExperiment(key, fallback, new
StringConverter());
if (result.equals("group a"))
// execute group a's logic
else if (result.equals("group b"))
// execute group b's logic
// in case conducting the experiment failed -
the fallback value is returned
// in this case you would usually execute the
'old' logic

47. Upload spec
• Upload the specs to Petri server
○ Enables to define an experiment instance
{
"creationDate" : "2014-01-09T13:11:26.846Z",
"updateDate" : "2014-01-09T13:11:26.846Z",
"scopes" : [ {
"name" : "html-editor",
"onlyForLoggedInUsers" : true
}, {
"name" : "html-viewer",
"onlyForLoggedInUsers" : false
} ],
"testGroups" : [ "old", "new" ],
"persistent" : true,
"key" : "clientExperimentFullFlow1",
"owner" : ""
}

48. Start new experiment (limited population)

49. Manage experiment states

50. Ending successful experiment
1. Convert A/B Test to Feature Toggle (100% ON)
2. Merge the code
3. Close the experiment
4. Remove experiment instance

51. Experiment lifecycle
• Define spec
• Use Petri client to conduct experiment in
the code (defaults to old)
• Sync spec
• Open experiment
• Manage experiment state
• End experiment

52. Petri is more than just an A/B test
framework
Feature toggle
A/B Test
Internal testing
Personalization
Continuous
deployment
Jira integration
Experiments
Dynamic
configuration
QA
Automated
testing

53. Other things we (will) do with Petri
• Expose features internally to company employees
• Enable continuous deployment with feature toggles
• Select assignment by sites (not only by users)
• Automatic selection of winning group*
• Exposing feature to #n of users*
• Integration with Jira
* Planned feature

54. Petri is now an open source project
https://github.com/wix/petri

55. Q&A
http://goo.gl/L7pHnd
https://github.com/wix/petri
Aviran Mordo
Head of Back-end Engineering
@aviranm
www.linkedin.com/in/aviran
www.aviransplace.com
Sagy Rozman
Back-end Guild master
@sagyrozman
www.linkedin.com/in/sagyrozman

56. Credits
http://upload.wikimedia.org/wikipedia/commons/b/b2/Fiber_optics_testing.jpg
http://goo.gl/nEiepT
https://www.flickr.com/photos/ilo_oli/2421536836
https://www.flickr.com/photos/dexxus/5791228117
http://goo.gl/SdeJ0o
https://www.flickr.com/photos/112923805@N05/15005456062
https://www.flickr.com/photos/wiertz/8537791164
https://www.flickr.com/photos/laenulfean/5943132296
https://www.flickr.com/photos/torek/3470257377
https://www.flickr.com/photos/i5design/5393934753
https://www.flickr.com/photos/argonavigo/5320119828

57. Why Petri
• Modeled experiment lifecycle
• Open source (developed using TDD from day 1)
• Running at scale on production
• No deployment necessary
• Both back-end and front-end experiment
• Flexible architecture

58. PERTI Server
Your app
Laboratory
DB
Logs