Building an experimentation framework

Building an experimentation
framework for web apps

Zhi-Da Zhong
zz@etsy.com

Tuesday, July 26, 2011

About the talk
Why
What
Framework
Break / hack
Tech Details
Test design
Analysis


Why?


Questions

“What will happen if I do X”?

“Is X better than Y?”


The future
&
alternate universes
(We’re bad at those.)


Then what?


Experiments


Experiments

Try it out.


Experiments

Try it out.
Data beats speculation.


Experiments

Try different alternatives
on different people.


Which is better?

v.s.


Not a great experiment


Web apps


Front end experiments

• Layout, colors, images, copy, ...
• No functional changes
• Impact can be surprisingly high


A little more complex...

• Multipage ﬂows
• Functionality changes


Backend experiments

• Why not?
• Algorithms, architectures, batch processes, ...


The Etsy search backend

Web app
• New algorithm search()

• New RPC protocol searchA() searchB()

• New result data structure
Search Search
• New Solr trunk snapshot cluster A cluster B


DB re-architecture

• Postgres => Sharded MySQL
• Multiple experiments


Whole new features

New pages
+
New DB tables
+
New batch jobs
+
...


Not just 2 variants

• A/B/C... tests
• Multi-variate tests


Caveats

• Content not under your control
• Price tests?
• Hard-to-measure/quantify things
• Long term impact?


Other tests

• Internal users testing
• Whitelisted user testing


Opt-in experiments


Complementary techniques

• Observed/recorded testing
- show different people the same thing

• Side-by-side testing
- show each person 2 alternatives


Side by side testing


A common approach

• JS-based
• Non-techie UI
• “No IT!”
• “Designed For Marketers, By Marketers”


Our approach

• The developer is the user
• Code as conﬁguration
• An integral part of the dev process


Developer as the user

• The builder of the feature writes the test
• Not just a marketing tool


Code as conﬁg

• Simplicity
• Expressivity
• Quality
• Version => complete system state
• Revision history


Part of the dev process

Every change is an experiment!


What does it look like?


Default => Experiment => (new) Default


To add a new feature...

+ $config[‘new_search’] = array(
+ ‘enabled’ => ‘off’
+ );

function search() {
+ if ($cfg->isEnabled(‘new_search’)) {
+ return do_new_search();
+ }

// existing stuff
}


Deploy that


Now we go crazy...

function do_new_search() {
// exciting new stuff
// that might or might not work
// but we can deploy it anyway
// since it’s flagged off
}


Internal user testing

$config[‘new_search’] = array(
+ ‘enabled’ => ‘rampup’,
+ ‘rampup’ => array(
+ ‘admin’ => true
)
);


Whitelists

‘enabled’ => ‘rampup’,
‘rampup’ => array(
+ ‘whitelist’ => array('zhida'),
‘admin’ => true
)
);


Opt-in experiments

+ ‘group’ => 12345,
‘admin’ => true
)
);


A/B

+ ‘percent’ => 1.5,
‘admin’ => true
)
);


If it works...

+ ‘enabled’ => ‘on’
);


Order matters

Whitelist / Blacklist > Internal > Opt-in > Random


The framework


As easy as...


As easy as...

1. Pick a variant


As easy as...

1. Pick a variant
2. Do what it says


As easy as...

1. Pick a variant
2. Do what it says
3. Log the event


What's in a test?


Variants

• Key-value pairs
• interpreted by the app

• Name
• mostly for logging


SubjectIdProvider
function getID()

• Why?
• hashing and other selectors
• logging
• Types of subjects
• Users...but not always
• Different groups of users - sellers vs buyers, etc.
• Different ways to identify them - signed in vs signed out


Selectors

function select($subjectID) => Variant Name


Combining multiple selectors

• OR
• breaks blacklists

• AND
• breaks whitelists

• Sequence
• works!


Selector sequence

• Deﬁnes an ordering
• Returns A/B/C/... or <don't care>


Loggers

function log($testKey, $variantKey, $subjectKey)


More => better

• More data
• More ways to track
• access logs
• 3P analytics
• custom


Access log augmentation

• Apache note
• Lots of log analysis tools
• grep
• $$


3P Analytics

• Quick to start
• May be cheap
• Volume?
• Lag time?
• Flexibility / customization?


3P Analytics - how

• Custom variables
• take note of number & size limits

• Custom segments
• Canned metrics


3P Analytics - example

<script type="text/javascript">
var pageTracker = _gat._getTracker("UA-1234567-8");
pageTracker._initData();
pageTracker._setCustomVar(2, "AB", "search_test.variantC", 3);
pageTracker._trackPageview();
</script>


Our own event tracking

HTML, event
JS beacon
Web app

• HTML beacons Event log

• Hadoop
Hadoop

• Cloud Results


Break / hack
https://github.com/etsy/ab


Building on top of the
core API


Test builders

• Capture common patterns
• feature ramp ups
• opt-in experiments

• Help with test design
• weight equalization
• multivariate testing


Automatic Dispatchers

• Separate dispatching and work
• Work with components that have well-deﬁned
invocation APIs
• Deﬁne a particular level of granularity
• Feel like magic


Dispatcher example - MVC

• View dispatch
• Controller dispatch
• Spring framework, etc.


Selector Registry

• Reuse $selectorReg = array(
‘staff’ => ‘InternalUserSelector’,
• Clarity ‘whitelist’ => ‘WhitelistSelector’,
‘percent’ => ‘WeightedSelector’
• Documentation );


Randomized Selector


What does it mean?


What does it mean?

• Independent of subject attributes


What does it mean?

• Independent of other tests


What does it mean?

• Independent of other tests
• Independent of (coarse-grained) time


Persistence


Persistence

• Better experience


Persistence

• Better data


Persistence

• Better data
• Multi-part tests


Persistence

• Better data
• Multi-part tests
• ...but not forever


Ramping up/down

• Vary group sizes
• Reduce risk
• Distribute load


Persistence + Ramping

• Minimize inconsistency
• Ramping up
• Should just add people to the treatment group

• Ramping down
• Should just remove part of the treatment group


rand()

• Explicit persistence
• Cookie
• DB

• Scaling
• Maintenance


Hashing

variant = H(id)


Hashing

variant = H(id)

Persistence


Hashing

variant = H(id)

Attribute independence

Persistence


Hashing

variant = H(id)

Persistence Attribute independence


Hashing

variant = H(id)

Test independence?



Hashing

variant = H(test id, id)

Test independence



Hashing


Persistence Attribute independence Test independence


Hashing


What else?



Hashing


Weights!



Hashing

h = H(test id, id)



Hashing

h = H(test id, id)

variant = P(h, weights)



Partitioning

Hash

0 1


Partitioning

Hash

0 1
.5
Partition

Partitioning

Hash

0 A B 1
.5
Partition

Ramping up

Hash

0 A B 1
.7
Partition

Which hash function?

• MD5/SHA-256/...
• Test it!
• But be careful...


A/B + opt-in

• Need to separate the groups for analysis
• Solution: use more than 2 variants!
• Act according to variant properties
• Track by variant name


Analysis


... Conﬁdence interval ... something something
... Binomial ... blah blah ...


Conﬁdence Intervals

• How sure are we?
• What if it were random?


Binomial experiments



HT HTTT HT H H



HT HTTT HT H H
T HT HTT H HT H


Results


Dashboards


A few test design tips


Whatʼs the question?



What metrics?



What metrics?

How much better?


Who?

• Different roles
• Old vs new
• Novelty
• Habit
• Expectation


When?

• User types vary
• Activity patterns vary
• Site content might vary
• Performance might vary
• Full weeks are often a good starting point


Summary


Better living through
experimentation

• More risk taking => better product
• MTTR
• Lower stress


You can too.


Building an experimentation framework

Recommended

Recommended

More Related Content

Similar to Building an experimentation framework

Similar to Building an experimentation framework (20)

Recently uploaded

Recently uploaded (20)

Building an experimentation framework