At Sawtooth Software's 2012 Conference, our methodologists Gerard Loosschilder and Paolo Cordella presented two approaches to analyzing Menu-Based Choice modeling data on their predictive validity.

1. expect great answers
Menu-Based Choice modeling (MBC):
a practitioner’s comparison of different methodologies
Sawtooth Software Conference, March 2012
Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder

2. Menu-based choice modeling – the next big thing
Sawtooth Software has recently launched its new Menu Based
Choice modeling software. Although the idea of build-your-own
exercises has been around for a while, the launch of a new tool from
Sawtooth Software usually causes a lot of excitement and uptake of
use.
As practitioners, we at SKIM want to be ready for the avalanche of
projects, so we started to look into pros and cons of several analysis
approaches.
2

3. Look around
Menu-based choices are everywhere and
are becoming increasingly common
Menu-based choice modeling – the next big thing
3

4. ✔ ✔ ✔
Cheddar $0.50 American cheese
$ 0.75 Curly fries $1.25
Whopper $3.50 California W. $ 4.50
✔ ✔
Crispy Onions Bacon
$1.50 $1.50
Omega3 $3.75 Chicken Deli $ 3.50 French fries $1.05
Total price $ 8.50
4

5. Menu-based choice exercises are found in areas where
combining items matters
• Menu optimization in fast food/branded restaurant chains
• Telecom services bundling
• BYO computers (e.g. Dell)
• Optional features pricing optimization in automotive market
• Add-on services in the financial and insurance services industry

6. Menu-based Choice Modeling exercises deliver item-
level forecasts of performance in these markets
It can deliver:
• Demand curves on an item level among many items
• Forecast revenue and find the optimal price for all items on the menu
• Measure uptake and decide whether to add a new item to your portfolio
• Cross-effects price sensitivity and cannibalization effects
• Does decreasing the price of single items hurt full menu sales?
• Most often chosen combinations and their prices
• Suggesting which items to bundle
• Insight in budget constraints
• How many items can we stuff in a bundle before we exceed the decision
maker’s budget?
6

7. At SKIM, we’re practitioners. We would like to
understand how MBC works in our practice
In particular, we would like to better understand the analysis
procedure. At first sight, we loved the beta version of the Sawtooth
Software tool, but we wanted to investigate more.
So we developed an alternative analysis approach, and applied it to
a study into the consumer’s willingness to pay for features of a
notebook computer.
This presentation contains a comparison of results on aspects of
internal validity.
7

8. We apply the approach to a study into
consumer features of computer notebook
SKIM’s Menu Based Choice exercise
8

9. Eye scanner High quality touch-
Technical advancement has screen
brought new vistas of safety
and security and today it is
No glare screen very easy to make your Easy keys
laptops and notebooks safe
and secure with technologies
such as fingerprint readers,
Universal plug for face recognition, eye External radio w/
US, EU, UK speakers
scanners etc.
Wireless speakers On-screen keyboard
spotlight
Eye recognition ensured only you can access your laptop
through a fast and accurate scan of the retina.
The laser scanner is conveniently positioned on top of the
Gold-plated jack External battery
screen, next to the webcam indicator
3D-ready HD DVORAK keyboard
webcam
9

10. This pilot application had the following specifications:
7
26 9 3
attributes Choice tasks price levels
• 12 consumer 9 choice tasks: There are 3 price
features • 7 random tasks levels per feature,
(2 levels: On/Off) • 2 hold out tasks varied in accordance
• 12 price attributes to estimate with an orthogonal
(3 levels) predictive validity research design
• 1 notebooks core
attribute (3 levels)
• 1 none option
Sample size: 1408
10

11. There are various models to analyze MBC data:
As presented in Bryan Orme’s paper
“Menu-Based Conjoint Modeling Using Traditional Tools” :
• Exhaustive Alternatives Model
• Serial Cross Effect Model
Both models have drawbacks that we thought we could solve using
SKIM’s method:
• Choice Set Sampling Model
11

12. Exhaustive Alternatives Model
All possible ways to choose options are included Drawbacks
in the choice set. The number of possible
• This model formally recognizes and predicts the combinations grows
combinatorial outcomes of menu choices. exponentially with the
• The dependent variable is the choice of a number of options (2ⁿ
combination using a single logit-based (MNL) dichotomous choices),
model transcending into a
• All possible combinations of options are coded as problem of computational
one attribute where each level is a combination : feasibility.
• e.g. with 3 on/off options, this attribute would have
2^3=8 levels
• Price: one price attribute for each option (or one
total price attribute)
12

13. Serial Cross-effect Model
The choice of each option is modeled Drawbacks
separately Only significant
• The dependent variable is the single cross-effects should
choice of a feature be included -
meaning they have
• N different logit models predicting the to be detected
choice of option X as a function of: beforehand
• Price of option X
• All other significant cross effects
13

14. We thought of solving it by introducing a ‘hybrid’
approach: Choice Set Sampling approach
• Like in the Exhaustive Alternatives Model, we consider the full choice set
with all possible combinations of options. However:
• we code each feature and its price as separate attributes (instead of a unique
attribute with all combinations as levels);
• we use importance sampling* – we consider a random sample from the set of
all chosen combinations
• Similarly to the Serial Cross-Effect Models, we also consider whether a
respondent chose an option at various price points.
* See importance sampling Ben-Akiva and Lerman (1985)
14

15. Coding the “sampling of alternatives” approach
1. In our model there are a total of 3*2^12=12888 possible combinations. However,
“only” 4560 were chosen at least once.
Combination # Core Feature 1 Price 1 Feature 2 Price 2 Feature 3 Price 3 ... Feature 12 Price 12 Choice
1 3 2 0 1 3 2 0 ... ... ... 0
2 1 2 0 2 0 1 2 ... ... ... 1
3 2 1 1 1 3 2 0 ... ... ... 0
... ... ... ... ... ... ... ... ... ... ... ...
4560 3 2 0 1 3 2 0 ... ... ... 0
Note:
» Each feature is either included in the combination (1) or not (2)
» Option prices are alternative specific
2. We draw a random sample from this choice set.
It is basically still a single logit-based (MNL) model where the dependent variable is
the choice of the combination.
15

16. Coding the “sampling of alternatives” approach
3. Each task is codified with 33 concepts/combinations drawn from
the sub-sample, with:
• The chosen alternative in each task
• 32 combinations randomly sampled from the choice set of all chosen
combinations
CASEID Task# Concept# Core Feature 1 Price1 Feature 2 Price2 ... Response
1 1 1 1 1 2 1 3 ... 0
1 1 2 1 2 0 1 3 ... 1
1 1 3 1 2 0 2 0 ... 0
... ... ... ... ... ... ... ... ... ...
1 1 33 2 2 0 1 3 ... 0
16

17. Coding the “sampling of alternatives” approach
4. In addition, our model is “hybrid” because we add extra dummy tasks for each respondent:
• For each choice task, we add 12 dummy tasks, one per feature
• We check whether a feature has been chosen at a specific price point
• No explicit modeling of cross effects between features
CASEID Concept# Core Feature 1 Price1 Feature 2 Price2 Feature 3 Price3 ... Response
1 1 1 1 1 2 0 2 0 ... 1
1 2 1 2 0 2 0 2 0 ... 0
1 1 1 2 0 1 3 2 0 ... 0
1 2 1 2 0 2 0 2 0 ... 1
1 1 1 2 0 2 0 1 1 ... 0
1 2 1 2 0 2 0 2 0 ... 1
• This coding contains the information that respondent 1 in task 1 chooses feature 1 at price
points 1, while she does not choose feature 2 at price point 3, and so on. Therefore we
embed a price barrier in our model which amplifies accuracy in price sensitivity estimation.
17

18. Analysis steps of SKIM’s Choice Set Sampling approach
• Using this setting we run HB estimation, so we can estimate utilities
for:
• Each feature (present/not present; 12 utility values and their mirrors)
• Each price level for each feature (3*12 utility values)
• None option (nothing is chosen; one utility value)
• We build a simulator in Excel, based on either Share of Preference
(SoP) or Share of First Choice (SoFC) with which we have:
• Single feature choice prediction
• Combinations choice prediction
18

19. Serial-Cross effect model
• Using Sawtooth Software’s MBC we build 12 different models for
each feature choice
• We could not find any significant cross-effects between the
features, both using counts and aggregate logit
• We use HB estimation and we simulate:
• Single Feature Choice predictions using Draws and Point Estimates
• Combinations choice predictions using Draws, Point Estimates and
Weighted Draws.
19

20. All approaches can be used to answer the same
business question
That’s why we compare the approaches to see:
• Which approach delivers the highest validity?
And because as practitioners, we often find ourselves dealing with
demanding clients and strict deadlines, so that we don’t just need
approaches that work but that are also efficient and as easy to apply:
• Which approach is most efficient to a practitioner?
20

21. The results - Choice Set Sampling vs Serial Cross-Effect model
Which approach has the highest validity?
21

22. Results 1: Single Features Choice Predictions
The Hold-out choice tasks suggest a similar performance
Hold-out 1 Hold-out 2
R-Squared MAE R-Squared MAE
Serial HB, Point Estimates 0.991 0.9% 0.990 1.0%
Cross-Effect
Model HB, Draws 0.991 0.9% 0.992 1.1%
HB, First Choice 0.987 1.6% 0.989 1.7%
Choice Set
Sampling Model
HB, Share of Preference 0.984 1.5% 0.981 1.5%
22

23. Both approaches have a very low MAE on the hold-out tasks
60.0%
50.0%
Freq. of choice
40.0%
30.0%
20.0%
10.0%
0.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option
10 11 12
Observed Serial cross-effect model (Draws) Choice set sampling model (SoP)
Holdout task - 1
23

24. No structural consistency in errors Holdout task - 1
6.0%
5.5%
5.0%
Absolute error
4.5%
4.0%
3.5%
3.0%
2.5%
2.0%
1.5%
1.0%
0.5%
0.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option
10 11 12
Choice set sampling model (First Choice, MAE = 1.5%) Serial cross-effect model (Draws, MAE = 0.9%)
24

25. The individual hit rate is almost the same across the two
hold out tasks
• Hit rate: % of respondents for which the choice on the option was predicted correctly
• 2 holdout tasks x 1408 respondents = 2816 observations for the hit rate
100.0% 91.1% 90.9%
86.8% 86.5% 87.5% 86.6% 86.5% 89.0% 87.0% 87.2%
90.0% 84.4% 84.9%
90.7% 90.2% 88.9%
80.0% 86.4% 85.8% 86.5% 85.1% 87.0% 86.5% 86.8% 86.7%
84.1%
70.0%
60.0%
Hit rate
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option Option Option
10 11 12
Choice set sampling model (First Choice) Serial cross-effect model (Weighted Draws)
25

26. Result 2: Feature combination predictions
Both models fit individual choices of combinations
100%
90%
83.5% 86.2%
80% 86.4%
83.6%
70%
66.2% 67.7%
63.6% 67.9%
60% 66.2%
63.2%
50%
40% 41.3%
41.0%
30%
20%
10%
0%
All 12 option At least 11 At least 10 At least 9 choices At least 7 choices At least 8 choices
choices predicted choices predicted choices predicted predicted correctly predicted correctly predicted correctly
correctly correctly correctly
Choice set sampling model (First Choice) Serial cross-effects model (Weighted Draws)
26

27. So we can conclude that both approaches are viable
tools for MBC analyses
Both models
• Are able to predict accurately hold-out choice tasks on aggregate
level
• Are extremely effective to predict individual choices of single
options and combinations
So both models are viable tools for analyzing MBC data.
• But which one is the most effective for practitioners?
27

28. So both approaches work and it comes down to efficiency.
Which approach is most efficient to a practitioner?
28

29. Choice Set Sampling approach –
Benefits and Drawbacks
Benefits Drawbacks
• One model to estimate, one • Complex procedure: time
model to simulate consuming set up for estimation
• No need to make a call on • Simulations are computationally
which cross-effects to intensive
include • Simulators are not very handy
• Explicitly predicts choice of tools for clients
combinations
29

30. Serial Cross Effect Model –
Benefits and drawbacks
Benefits Drawbacks
• Dedicated software • Learning curve of
available understanding how to interpret
the significance of cross /
• Explicit inclusion of cross-
interaction effects and their
effects in the model inclusion in the model – it
• Easy simulation tools takes art and craft to build an
accurate model
• Once cross-effects are
included in the model, they
hold for all respondents
30

31. To conclude: Sawtooth Software’s Serial Cross-effect
model is the practitioner’s choice
• We would recommend using Sawtooth Software’s Serial Cross-effect
model and software package,
• After the initial learning, it’s an easy to apply and time-effective solution, thanks
to its dedicated software
• One just needs to invest in the learning curve of making the call about the
significance and meaning of interaction/cross effects
• If you want to use the Choice Set Sampling model, be prepared to invest
time to create dedicated tools
31

32. contact us or follow us online!
Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder
www.skimgroup.com | +31 10 282 3535
linkedin.com/ facebook.com/ twitter.com/ youtube.com/
skimgroup.com
company/skim skimgroup skimgroup skimvideos