1. 1
PRICING A PATENT BY SURVEYING
JaeWon Lee
BACKGROUND: REASONABLE ROYALTY
Reasonable royalty calculation in patent infringement litigation is critical in, at least, two aspects.
First, the federal statute specifies a reasonable royalty as the minimum that the patentee is entitled to be
compensated.1
Calculating reasonable royalty damages is essential in cases where non-practicing
entities (NPEs) assert patent infringement claims.2
Because NPEs, by definition, neither manufacture
nor sell products or services embodying the claimed invention, they mostly fail to receive reliefs based
on lost profits or price erosions (or, needless to say, injunction). Simply put, NPEs do not suffer those
types of injuries; they neither charge a price nor realize a profit from manufacturing or selling the
products embodying the claimed invention. As a result, NPEs are usually constrained to seek damages
based solely on reasonable royalties.3
Considering the increasing percentage of patent infringement
suits brought by the NPEs4
and difficulties of proving other type of damages,5
reasonable royalty
1
“Upon finding for the claimant the court shall award the claimant damages adequate to compensate for the infringement,
but in no event less than a reasonable royalty.” 35 U.S.C. § 284 (emphasis added). At the same time, section 284 does not
set the ceiling. Reasonable royalty can be higher than the price of infringing unit or the infringer’s net profit margin. Rite-
Hite Corp. v. Kelley Co., 56 F.3d 1538, 1555 (Fed. Cir. 1995).
2
Unlike the entities that use the patented technology in their products or services and exclude others from using the same,
the NPEs enforce patent rights against accused infringers for profit, most commonly in the form of licensing fees, but do not
manufacture products or supply services using the patented claim at issues. The NPE here refers to a broad class of entities
including individual inventors, research firms, and universities. Some NPEs are known as patent monetizing entities (PMEs)
as their business model relies on actively buying patents from others for the purpose of asserting the claims for profit. U.S.
GOV’T ACCOUNTABILITY OFFICE, INTELLECTUAL PROPERTY: ASSESSING FACTORS THAT AFFECT PATENT INFRINGEMENT
LITIGATION COULD HELP IMPROVE PATENT QUALITY 2–3 (2013), available at http://www.gao.gov/assets/660/657103.pdf.
3
Damages awards based on the lost profit and price erosion and reasonable royalty damages are not mutually exclusive. A
patentee may seek all types of damages from the defendant who has infringed the same patent at bar. Also, NPEs may seek
injunction as a remedy. However, the recourse in equity is much challenging, and the four-prong test under eBay Inc. v.
MercExchange, L.L.C., 547 U.S. 388 (2006), significantly disfavors granting injunctive awards in the NPEs’ favor.
4
A recent study reveals that in between 2007 and 2011 suits brought by the NPEs – including PMEs, likely PMEs,
individuals, research firms, or universities – have significantly increased. U.S. GOV’T ACCOUNTABILITY OFFICE, supra note
2, at 17. While the study conceded that the increase attributed to the PMEs and likely PMEs is not statistically significant, it
concluded that the decrease in suits brought by operating companies and related entities (thus, non-NPEs) is statistically
significant. Id. In total, the notable trend is that the volume of the suits brought by the NPEs is surging. See id. at 18.
Moreover, one study revealed that although the overall success rate of the patent infringement litigation is lower for the NPEs
(24.3%) than the practicing entities (34.5%), the median damages awarded to the NPEs (about $8.88 million) was
significantly higher than that for the practicing entities (about $5.35 million). CHRIS BARRY ET AL.,
PRICEWATERHOUSECOOPERS LLP, 2013 PATENT LITIGATION STUDY: BIG CASES MAKE HEADLINES, WHILE PATENT CASES
PROLIFERATE 25 (2013), available at http://www.pwc.com/en_US/us/forensic-services/publications/assets/2013-patent-
litigation-study.pdf. This study analyzed 1,856 district court patent decisions issued in between 1995 and 2012.

2. 2
estimation has become the center of the battlegrounds in most, if not all, patent infringement cases.6
Second, reasonable royalties are the predominant measure of damages, accounting for about eighty
percent of the total damages awarded throughout the last decade.7
Methodologies for calculating reasonable royalties that the Federal Circuit8
has adopted largely
fall in two categories.9
First, the analytical approach focuses on infringer’s projection of profit for the
infringing product.10
Reasonable royalties under this approach are estimated from infringer’s extra
profit realized from sales of infringing devices.11
The second, more common approach is the so-called “hypothetical negotiation” or “willing
licensor-willing licensee” approach.12
This approach presumes that the patent claims at issue are valid
and infringed. It then attempts to ascertain the royalty a patentee and an infringer would have agreed to
just before the infringement began. This is an “ex ante licensing negotiation scenario” that willing
parties would have executed a negotiated royalty payment scheme.13
Georgia-Pacific Corp. v. U.S. Plywood Corp. established the legal framework for assessing
reasonable royalties and enumerated fifteen relevant factors to consider.14
The Federal Circuit has
5
The standard for awarding lost profits is established by Panduit Corp v. Stahlin Bros. Fibre Works, Inc., 575 F.2d 1152
(6th Cir. 1978), in which the Federal Circuit has adopted and applied in numerous patent infringement cases. Panduit
provides that the patentee must demonstrate with reasonable probability that bur-for the infringement, it would not have lost
profits. Thus, the patentee must show: (1) demand for the patented product; (2) absence of acceptable non-infringing
substitutes; (3) manufacturing capability to exploit the demand; and (4) the amount of profit the patentee would have made.
Id. at 1156.
6
E.g., Rembrandt Social Media, LP v. Facebook, Inc., 22 F. Supp. 3d 585 (E.D. Va. 2013).
7
BARRY ET AL., supra note 4, at 11. Reasonable royalties constitute eighty-one percent of whole damages awards in
between 2007 and 2012 and seventy-nine percent in between 2001-2006. Id. at 11. During that period, the second most
frequently awarded damages, lost profits, constituted only about thirty percent. Because each form of damages may be
awarded on a non-exclusive base, the totals exceed one hundred percent. Damages awards based on price erosion have
become miniscule.
8
The United States Court of Appeals for the Federal Circuit has exclusive jurisdiction over patent infringement actions
appealed from the district courts. 28 U.S.C. § 1295(a)(1).
9
Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1324–25 (Fed. Cir. 2009).
10
Id. at 1324; TWM Mfg. Co. v. Dura Corp., 789 F.2d 895, 898–900 (Fed. Cir. 1986).
11
TWM Mfg., 789 F.2d at 899 (“[the expert] subtracted the infringer’s usual or acceptable net profit from its anticipated net
profit realized from sales of infringing devices”).
12
Lucent, 580 F.3d at 1324–25.
13
Id. at 1325.
14
318 F. Supp. 1116, 1120 (S.D.N.Y. 1970). Fifteen factors are: (1) royalties the patentee has received for the licensing of
the patent in suit; (2) royalty rates the licensee has paid for the use of other patents comparable to the patent in suit; (3) the
exclusivity and restrictiveness of the license; (4) patentee’s licensing policy to maintain or control patent monopoly by not
licensing the invention to others; (5) the commercial relationship between parties; (6) the effect of selling the patented
specialty in promoting sales of other products; (7) the duration of the patent and the term of the license; (8) the established
profitability of the products made under the patent; (9) advantages of the patented product over the old products; (10) the
commercial aspects of the patented invention; (11) the extent to which the infringer has used the invention; (12) the portion
of the profit customarily allowed for the use of the patent; (13) the portion of the realized profit attributable to the patent; (14)
qualified experts’ testimony; and (15) the outcome from a hypothetical arm’s-length negotiation between parties. Id.

3. 3
adopted Georgia-Pacific analysis as the “legally-required framework.”15
The test is a flexible one: not
all factors are necessarily relevant to the case at bar, nor is any particular factor dispositive.16
A royalty – whether reasonable or unreasonable, or whether structured as a running royalty or a
lump-sum payment – is a product of two distinct quantities: a royalty base and a royalty rate.17
For
example, assume that a patent at issue claims a method of manufacturing a special polymer material.
The patentee/licensor may license the patented technology to a golf ball manufacturer who uses the
method to manufacture and sell golf balls to retailers at a wholesale price of $500 for a box. The
patentee and the manufacturer may enter into a license agreement, which specifies the royalty base to be
the wholesale price and the royalty rate to be 20%. Assuming further that the manufacturer sold 1,000
boxes, the agreement requires the manufacturer to pay 000,100$1000%20500$ =×× as a royalty to the
patentee. The patentee and another manufacturer may agree to use the royalty base as the
manufacturer’s profit margin and apply the royalty rate of 30%. If this manufacturer’s profit margin is
$150 for a box and it sold 2,000 boxes, the patentee is expected to receive royalty payment of
000,90$2000%30150$ =×× .
Although these examples assumed a voluntary, willing licensor-willing licensee scenario, the
same principle applies to post-infringement reasonable royalty calculations. The royalty base represents
“the revenue pool implicated by the infringement,” and the royalty rate accounts the percentage of the
revenue pool “adequate to compensate” the patentee for the infringement.18
Both the base and the rate,
as distinct quantities, must be reasonable to reach the end result of a reasonable royalty.19
A. Royalty Rate
The first two Georgia-Pacific factors examine “royalties the patentee receives for licensing the
patent in suit” and “[t]he rates paid by the licensee for the use of other patents comparable to the patent
15
Lucent, 580 F.3d at 1335.
16
Id.
17
Cornell Univ. Hewlett-Packard Co., 609 F. Supp. 2d 279, 285 (N.D.N.Y. 2009).
18
Id.
19
Id. (“An over-inclusive royalty base including revenues from the sale of non-infringing components is not permissible
simply because the royalty rate is adjustable.” (citing Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549 n.9 (Fed. Cir. 1995)));
see Virnetx, Inc. v. Cisco Sys., 767 F.2d 1308, 1328–30 (Fed. Cir. 2014) (holding that the expert testimony on royalty base
was inadmissible and should have been excluded, whereas finding that the district court did not abuse its discretion in
permitting the testimony on royalty rate estimation). Mathematically, infinite combinations of the royalty base and rate are
possible to produce the same royalty payment. For example, 1% of $100 and 10% of $10 both result in $1 royalty. It
illustrates dangers of how easily one can manipulate the base and rate to reach the value that one conclusively has presumed
to be reasonable.

4. 4
in suit.”20
Until recently, however, the so-called “25% rule of thumb” has been widely used by the
parties and accepted (or at least passively tolerated) by the court. It assumed, as a rule of thumb, that
“25% of the value of the product would go to the patent owner and the other 75% would remain with
[the infringer/licensee],” because the infringer/licensee also has substantially contributed to the
development or realization of the commercialization of the product embodying the patented
technology.21
At first glance, automatically allocating a quarter of the product’s value to the patentee is
methodologically unsound, even considering that 25% is but a temporary, adjustable baseline rate.
Simply stated, the 25% rule of thumb’s rationale (or lack thereof) does not reach the Daubert standard.22
The Federal Circuit in Uniloc conceded that it has tolerated the use of the 25% rule.23
The
Circuit court noted that lower courts have invariably admitted royalty evidence based on the 25% rule,
because of its widespread acceptance or because its admissibility was uncontested.24
Experts, in turn,
justified the use of the rule because courts have accepted the rule as an appropriate methodology in
determining damages.25
At last, Uniloc concluded that mere widespread use of the 25% rule is not
sufficient to pass the muster under Daubert and the Federal Rules of Evidence, and consequently held
that testimony based on the 25% rule inadmissible, because “it fails to tie a reasonable royalty base to
the facts of the case at issue.”26
B. Royalty Base
Likewise, the Federal Circuit has reviewed the reasonable royalty base calculation with
heightened scrutiny, demanding a scientific methodology that reflects case-specific, real-world data.
What have particularly troubled the court are the patentee’s attempts to inflate the royalty base.
Intuitively, the maximum royalty base that the patentee could seek is the entire value of the product that
embodies the patented feature. This so-called “entire market value rule” (EMVR) permits recovery of
damages based on the value of the entire product even though such product embodies many features not
encompassed by the patent at issue.
20
Georgia-Pacific Corp. v. U.S. Plywood Corp., 318 F. Supp. 1116, 1120 (S.D.N.Y. 1970).
21
Uniloc USA Inc. v. Microsoft Corp., 632 F.3d 1292, 1318 (Fed. Cir. 2011).
22
It is quite surprising that the Federal Circuit abolished the 25% rule of thumb in 2011 in Uniloc, about two decades after
the Supreme Court decided Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993). Measuring from Kumho Tire Co.,
Ltd. v. Carmichael, 526 U.S. 137 (1999), more than a decade has passed.
23
Uniloc, 632 F.3d at 1314 (“court has passively tolerated [25% rule’s] use where its acceptability has not been the focus of
the case . . . or where the parties disputed only the percentage to be applied (i.e., one-quarter to one-third), but agreed as to
the rule’s appropriateness”) (internal citations omitted).
24
Id.
25
Id. at 1311.
26
Id. at 1315.

5. 5
The EMVR is permissible under certain circumstances. The value of the entire product may
reasonably represent the value of the patented feature if the product contains only one patented feature
or the patented feature is the single most important feature that draws a substantial consumer demand.
Assume that a product contains ten claimed inventions, one patentee holds all of them, and the accused
is found to be infringing all ten valid patents. In such a case, while it may be difficult to isolate the
estimated value of each patented feature, the patentee may be allowed to base her loyalty calculation on
the value of the entire product.
However, such a hypothetical situation is an exception rather than the norm in most of today’s
patent infringement suits where the accused products are multi-component, multi-functional systems.
For example, a smartphone contains about 250,000 patented technologies.27
Of course, many of the
quarter million patents are licensed (e.g., under reasonable and non-discriminatory licensing agreements
or by cross-licenses), have been expired or invalidated, probably should not have been issued at all, or
will likely be invalidated. The point is that most of today’s products or services consist of several, if not
hundreds or thousands of, patented invention.
As early as late nineteenth century, courts recognized the reality and risk of over-inclusiveness of
using the EMVR.28
The use of the infringing product’s entire market value as a royalty base is allowed
“only where the patented feature creates the ‘basis for customer demand’ or ‘substantially create[s] the
value of the component parts.’”29
Specifically, using the entire market value as a royalty base requires
adequate proof of following conditions: (1) the infringing components must be the basis for consumer
demand for the entire product, beyond the claimed invention; (2) the individual infringing and non-
infringing components must be sold together; and (3) the individual infringing and non-infringing
components must be analogous to a single functioning unit.30
Where the EMVR is inappropriate and exaggerates the royalty base, the royalty base must be tied
to the patented feature at issue. As required under Daubert and Federal Rules of Evidence, after all,
27
Innovation, TECHDIRT, There Are 250,000 Active Patents That Impact Smartphones; Representing One In Six Active
Patents Today, https://www.techdirt.com/blog/innovation/articles/20121017/10480520734/there-are-250000-active-patents-
that-impact-smartphones-representing-one-six-active-patents-today.shtml (last visited Apr. 2, 2015).
28
Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1336–37 (Fed. Cir. 2009) (tracing the origins of the entire market
value rule to several Supreme Court cases).
29
Uniloc, 632 F.3d at 1318 (quoting Lucent, 580 F.3d at 1336; Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549–50 (Fed.
Cir. 1995)) (emphasis added).
30
Cornell Univ. v. Hewlett-Packard Co., 609 F. Supp. 2d 279, 286–87 (N.D.N.Y. 2009).

6. 6
expert opinions on the royalty base must be sufficiently tied to the facts of the case.31
In Daubert’s
words, it must “fit.”32
Otherwise, the expert opinion is irrelevant (or marginally relevant at best) and
likely will confuse the jury.
The court in Cornell Univ. v. Hewlett-Packard Co. first apportioned the royalty base to the
“smallest salable infringing unit with close relation to the claimed invention.”33
The Federal Circuit, in
LaserDynamics, subsequently adopted the smallest salable patent-practicing unit formulation from
Cornell.34
As the two cases demonstrate, the idea is that the royalty base for the infringed feature must
be reduced, at least, to the smallest unit embodying that feature available in the market. For example, if
a claimed invention is a method for identifying the type of optical disc inserted into an optical disc drive
(ODD), the royalty base calculation must start from, at most, the ODD, not from the entire computer that
incorporates the ODD.35
Although useful in some cases, the smallest salable patent-practicing unit approach has not
provided a complete solution for the fundamental issue in determining the reasonable royalty base. That
is because even the smallest unit can still encompass a number of non-infringing components or
functions. The ODD example illustrates this point. A typical ODD certainly is manufactured based on
numerous patented technologies and incorporates more than one patented feature claimed in
LaserDynamics.36
Reading Cornell and LaserDynamics together with Lucent and Uniloc, courts require
an apportionment whenever the accused product is a multi-component product encompassing non-
infringing features.37
C. Market’s Willingness-To-Pay & Survey
While the Federal Circuit has not developed a Daubert motion-proof method of estimating
reasonable royalties, it has underscored several times that the estimation must be “tied to the facts of the
31
Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 591 (1993) (“An additional consideration under Rule 702 – and
another aspect of relevancy – is whether expert testimony proffered in the case is sufficiently tied to the facts of the case that
it will aid the jury in resolving a factual dispute.”).
32
Id.
33
Cornell, 609 F. Supp. 2d at 288.
34
LaserDynamics, Inc. v. Quanta Computer, Inc., 694 F.3d 51, 67 (Fed. Cir. 2012).
35
Id. at 56–57, 67.
36
The patent at issue in LaserDynamics, U.S. Patent No. 5,587,981 (the “’981 Patent”), cites four patents directed at optical
disk systems. More than forty U.S. patents cite the ’981 Patent.
37
Dynetix Design Solutions, Inc. v. Synopsys, Inc., No. C 11–05973 PSG, 2013 WL 4538210, at *3 (N.D. Cal. Aug. 22,
2013); see Virnetx, Inc. v. Cisco Sys., 767 F.2d 1308, 1327 (Fed. Cir. 2014) (“Where the smallest salable unit is, in fact, a
multi-component product containing several non-infringing features with no relation to the patented feature . . ., the patentee
must do more to estimate what portion of the value of that product is attributable to the patented technology.”).

7. 7
case” and reflect “footprint in the market place.”38
After Lucent and Uniloc and their progenies, parties
have increasingly sought a research tool capable of analyzing consumer behaviors in the market and
quantifying the part effect a particular feature, in isolation, can contribute to the consumer’s decision-
making process. Naturally, a market data-driven approach has emerged as an attractive tool to quantify
consumers’ willingness-to-pay a price premium for a specific feature of a multi-component/function
system.39
The underlying idea is that by aggregating individual consumer’s willingness-to-pay, the
market’s willingness-to-pay the premium can be used as a reliable estimate that captures the additional
value a particular feature contributes to the infringing product’s total value.40
The price premium that
the particular feature contributes to the infringing product becomes the royalty base (or at least a sound
starting point) for reasonable royalty calculations.41
The challenge still remains: how to isolate the
specific feature from the product and find consumer’s willingness-to-pay just for that feature –
especially where such market does not exist in reality.
A type of consumer surveys called the conjoint survey or conjoint analysis can accomplish the
task. Part I initially explains conjoint analysis in a conceptual level. Then, based on a hypo, two types
of widely applied conjoint analyses will be described in Part II. Part III identifies eight key areas where
the admissibility and credibility of survey evidence can be challenged and addresses critical legal and
practical issues relevant to conjoint analysis. Before closing the paper, Part IV briefly discusses the
interpretation of the conjoint survey results.
Although this paper starts with and focuses on the calculation of reasonable royalty damages in a
patent infringement suit, most issues and points addressed in Parts III and IV apply beyond the patent
38
Uniloc USA Inc. v. Microsoft Corp., 632 F.3d 1292, 1317 (Fed. Cir. 2011) (“To be admissible, expert testimony opinion
on a reasonable royalty must ‘carefully tie proof of damages to the claimed invention’s footprint in the market place.’”
(quoting ResQNet.com, Inc. v. Lansa, Inc., 594 F.3d 860, 869 (Fed. Cir. 2010))).
39
Shankar Iyer, IP Expert On The Push For Market-Based Evidence In High-End Patent Litigation, METRO. CORP.
COUNSEL (Oct. 23, 2014), http://www.veooz.com/news/vHdZq0R.html. Strictly speaking, the conjoint analysis does not
always rely on market data (revealed data in the market), but instead relies on survey data known as stated-preferences. See
infra Part I.A. The survey, however, has been recognized as an important and reliable tool to understand and, more
importantly, predict the market, if carefully designed and conducted accordingly.
40
See S. Christian Platt & Bob Chen, Recent Trends and Approaches in Calculating Patent Damages: Nash Bargaining
Solution and Conjoint Surveys, 86 PATENT, TRADEMARK & COPYRIGHT J. 1, 4 (2013), available at
http://www.paulhastings.com/docs/default-source/PDFs/platt_chen_insight_final.pdf; Shankar Iyer, Consumer Surveys and
Other Market-Based Methodologies in Patent Damages, A.B.A. SEC. INTELL. PROP. (Oct. 16, 2014),
http://apps.americanbar.org/litigation/committees/intellectual/articles/fall2014-0914-consumer-surveys.html.
41
This paper focuses on deriving reasonable royalty bases using conjoint analysis. As explained, reasonably royalty
calculations also require a reasonable royalty rate. The rate can be estimated, for example, by evaluating rates the licensee
has paid for the use of other patents comparable to the patent in suit and/or rates the patentee has accepted from other
licensees for the infringed patent. See Georgia-Pacific Corp. v. U.S. Plywood Corp., 318 F. Supp. 1116, 1120 (S.D.N.Y.).

8. 8
infringement litigation. Those discussions equally apply to survey evidence in general beyond the
context of the conjoint survey. It should be emphasized that using conjoint analysis, or surveys, to
quantify consumer’s willingness-to-pay for an isolated, particular feature is only the first step.42
Reasonable royalty calculations require determination of at least one other distinct term, a reasonable
royalty rate.43
I. WHAT IS CONJOINT ANALYSIS?
Conjoint analysis concerns day-to-day decisions of consumers: how and why consumers choose
one product over another. It is a survey-based market research tool that quantitatively measures trade-
offs made by consumers. The approach is particularly useful when a multitude of factors potentially
influencing consumers complicate the analysis of identifying a causal relationship between a specific
feature and consumer preferences. On one end, several features are joined together in a product, and
consumers consider the product jointly on the other end.44
Since its introduction to the field of
marketing research in 1971, academics and industry practitioners have made conjoint analysis the most
widely studied and applied form of quantitative consumer preference measurements.45
A. Survey & Stated Preference
In the free market system, when needs arise, consumers gather information about products or
services, compares any available alternatives, consider constraints, and decide which one to choose by
making trade-offs among competing alternatives.46
In the market, consumer preference is revealed as a
42
“The first step in a damages study is the translation of the legal theory of the harmful event into an analysis of the
economic impact of that event.” Comcast Corp. v. Behrend, 133 S. Ct. 1426, 1435 (2013) (citing FEDERAL JUDICIAL CENTER,
REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 432 (3d ed. 2011)) (emphasis from the Court).
43
Royalty payment can be structured either as a running royalty or a lump-sum royalty. The first method would require
determination of the numbers of units sold or used as a multiplying factor to reach the end amount the patentee is entitled.
Significantly distinguishable risks and analyses involve in two types of the royalty calculations. See Lucent, 580 F.3d at
1326–32.
44
See BRYAN K. OMRE, GETTING STARTED WITH CONJOINT ANALYSIS: STRATEGIES FOR PRODUCT DESIGN AND PRICING
RESEARCH 29 (2d. 2010), available at http://www.sawtoothsoftware.com/download/techpap/cahistory.pdf (“Marketers
sometimes have thought (or been taught) that the word conjoint refers to respondents evaluating features of products or
services CONsidered JOINTly. In reality, the adjective conjoint derives from the verb to conjoin, meaning joined together.”
(internal quotation marks omitted)).
45
Paul E. Green et al., Thirty Years of Conjoint Analysis: Reflections and Prospects, 31 INTERFACES S56, S57 (2001)
(“Conjoint analysis is, by far, the most used marketing research method for analyzing consumer trade-offs.”); see also
ANDERS GUSTAFSSON ET AL., CONJOINT MEASUREMENT – METHODS AND APPLICATIONS 3 (Anders Gustafsson et al. eds., 4th
ed. 2007) (“Based on a 2004 Sawtooth Software customer survey, the leading company in Conjoint Software, between 5,000
and 8,000 conjoint analysis projects were conducted by Sawtooth Software users during 2003.”).
46
There may be a situation where no alternative exists or the product is a necessity. Consumer behaviors, such as demand
elasticity, become markedly different under such circumstances.

9. 9
form of actual purchases, e.g., which product consumers bought out of available alternatives and/or how
much they paid for it. This is a type of information called revealed preferences or revealed data – they
are revealed in the market. Marketing professionals use these data to answer why consumers picked one
over the rest. When available, revealed preferences provide the most reliable data to study consumer
behaviors because they reflect actual decisions consumers made, as compared to those not yet been
realized in the market.47
However, benefits and availability of the revealed data are not without limit. There are practical
difficulties in gathering and using market data, including privacy issues.48
Controlled group analysis for
comparison may not be available because market data are typically obtained under uncontrolled
circumstances. Most notably, by definition, revealed data do not exist for a product that has not been
introduced to the market yet.
As a survey-based tool, conjoint analysis relies on stated preferences or stated data.49
Rather
than observing the market itself ex post, conjoint analysis explores surveys to conduct controlled
experiments with the sampled participants sampled from the target population.50
Respondents
participating in the survey usually agree to provide their demographic and product-related information
and make them available for the survey analysis. The surveyor constructs a hypothetical market and
simulates consumer preferences. There, the surveyor designs and controls experimental parameters.
Revealed preferences and stated preferences supplement, not conflict with, each other. Revealed
preferences provide practical information to the surveyors to start with. The analyst may compare the
stated data with the revealed data for the verification purposes.
47
Lisa Cameron et al., The Role of Conjoint Surveys in Reasonable Royalty Cases, THE BRATTLE GRP. (Oct. 16, 2013, 6:37
PM ET),
http://www.brattle.com/system/publications/pdfs/000/004/948/original/The_Role_Of_Conjoint_Surveys_In_Reasonable_Roy
alty_Cases.pdf?1382111156.
48
See FED. TRADE COMM’N, PROTECTING CONSUMER PRIVACY IN AN ERA OF RAPID CHANGE: RECOMMENDATIONS FOR
BUSINESSES AND POLICYMAKERS 1 (Mar. 2012), available at
https://www.ftc.gov/sites/default/files/documents/reports/federal-trade-commission-report-protecting-consumer-privacy-era-
rapid-change-recommendations/120326privacyreport.pdf. The Federal Trade Commission’s privacy framework applies to all
commercial entities that collect or use consumer data (both offline and online) that can be reasonably linked to a specific
consumer. Id. at 15–22. The Commission suggests the companies collect only the data they need to accomplish a specific
business purpose. Id. at 26. The Commission also proposes that the “data brokers,” who collect consumers’ personal
information for the purpose of reselling such information to their customers, improve transparency and give consumers
control over the company’s data practices. Id. at 68–70.
49
The power and beauty of consumer surveys in marketing field is that the researcher can design and test the market with a
new product that has not yet been introduced in the market. Modifying certain features of an existing product in a survey is
undoubtedly much cost-less and risk-less than introducing such alternatives in the market and testing market response.
Moreover, using hypothetical products in surveys allows the researchers to predict the market and suggest the roadmap for
research and development and/or investment plans.
50
See infra Part III for the discussion on the issues regarding conjoint analysis as survey evidence.

10. 10
B. Bundle of Attributes
Conjoint analysis conceptualizes products or services as bundles of attributes. Each attribute can
have one or more levels. The end product is assumed to be characterized solely by the set of attributes
with designated levels embodied in that product.
For example, a consumer who considers buying a laptop may consider following four attributes:
brand, display size, price, and storage type, to name but a few. Each attribute can have one or more
levels, and the levels can be either quantitative or qualitative. A brand may have four qualitative levels,
e.g., Apple, Dell, Lenovo, and Toshiba. Similarly, a storage type may consist of two qualitative levels,
e.g., Hard Disk Drive (HDD) and Solid State Drive (SSD). Three quantitative levels may comprise an
attribute display size, e.g., less than 13-inch, 14-to-16 inch, and 17-inch or larger. A price range may be
divided into three levels, e.g., less than $1,000, $1,000 to $1,200, and over $1,200.
A combination of attributes and corresponding levels, which referred to as a profile,
characterizes each laptop. One laptop may be an Apple laptop with 13-inch display and SSD storage,
sold at $1,299 (“Profile 00”). Another laptop may be a Lenovo laptop with 14-inch display and HDD
storage, sold at $1,099 (“Profile 01”). In theory, this example can generate up to 72 ( 2334 ××× )
profiles.
Conjoint analysis assumes that each consumer has a set of weights (or values) in units of utility
associated with each level of each attribute. They are referred to as partworth.51
Each partworth
contributes to the total value (or utility) of a product.52
Consumers are assumed to behave rationally in a
sense that they would choose the product with the maximum utility.53
Put differently, what an
individual selects is deemed to have the maximum utility to that individual among the available
alternatives. Thus, conjoint analysis is “consistent with economic and consumer-behavior theories of
approximate utility maximization.”54
Using the same laptop example, if a consumer was asked to choose one model from the 72
alternative laptops, and if Profile 00 was chosen, it is assumed that her choice has revealed or stated that
the Profile 00 gives her the maximum utility among 72 available alternatives. Comparing just two
51
Partworth and utility are used interchangeably. Strictly speaking, however, partworths are measured in units of utility.
For example, the partworth of the HDD is ten (units of utility) and that of the SDD is fifteen (units of utility).
52
The way partworth contributes to the total utility differs depending on how one models the utility function. See infra Part
III.
53
MOSHE BEN-AKIVA & STEVEN R. LERMAN, DISCRETE CHOICE ANALYSIS: THEORY AND APPLICATION TO TRAVEL DEMAND
38 (Marvin L. Manheim ed., 1985).
54
Expert Report of John R. Hauser at 11, Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK (N.D. Cal.
July 26, 2012), ECF No. 1363–1 [hereinafter Hauser Report].

11. 11
profiles (00 and 01), the consumer perceives a higher (or at least an equal) utility from having a specific
brand name and a SSD (that the Profile 00 provides) over receiving $200 discount and an extra 1-inch
display size (that the Profile 01 adds). A caveat is that her preference should be interpreted to reflect the
closed universe of choice sets – she compared only 72 profiles as represented by only four attributes.55
C. Utility Function
Most critically, conjoint analysis assumes that utility function can model consumers’ decision-
making and that the consumers behave as though having the utility function.56
This is especially so
because utility function is closely interrelated with the design of the conjoint survey and the choice of
estimation methods, which will be applied in estimating the partworth of the level of the attribute at
issue.57
Figure 1 illustrates three forms of the utility function: (1) vector model; (2) ideal point model;
and (3) partworth function model. In practice, partworth utility function is adopted most widely by the
researchers to model consumer preferences.58
This model represents the attribute utilities by a piecewise
linear curve. It is particularly useful when dealing with qualitative attributes, such as brands or discrete
functionalities because the values of each level of the qualitative attributes do not vary linearly.
Conjoint analysis is a generic term referring to a collection of methods. More than one form of
the utility function can model consumer behaviors. There are many ways to phrase questions and to
estimate parameters of the modelled utility function by applying different statistical approaches. While
many models share basic properties such as additivity and linearity, such are not universally required.59
Furthermore, no single model can describe or fit to every situation equally. A model that works well in
a certain set of data may not do so for another set of data.
55
Thus, although consumers typically associate a smaller display size with an enhanced portability and/or an increased
battery life, the survey respondent is instructed not to consider such associated effects, however closely interrelated they are
in real life. The conjoint analysis, and the survey in general, controls the risk of secondary implication by expressly
instructing the respondents to assume that all conditions other than the stated features are identical. See infra Part III.E.i.
Still, it is difficult to prove that the instructions eliminate a possibility that the participants might unconsciously have
associated the stated-features with those features unstated, but are closely interrelated.
56
The form of which the utility function takes is also the most difficult assumption to make. BEN-AKIVA & LERMAN, supra
note 53, at 45.
57
See JOHN R. HAUSER & VITHALA R. RAO, CONJOINT ANALYSIS, RELATED MODELING, AND APPLICATIONS 1, 4–16 (2002),
available at http://www.mit.edu/~hauser/Papers/GreenTributeConjoint092302.pdf .
58
Green et al., supra note 45, at S59–S61; Paul E. Green & V. Srinivasan, Conjoint Analysis in Consumer Research: Issues
and Outlook, 5 J. CONSUMER RES. 103, 105–06 (1978).
59
BEN-AKIVA & LERMAN, supra note 53, at 40–41.

12. 12
Figure 1. Three types of the utility function.60
Before jumping to Part II, it is worth mentioning briefly why marketing professionals prefer a
seemingly complex conjoint survey to a simple, direct questioning to isolate and assess the value of a
particular feature from a multi-component system.
D. Advantage & Validity
Academics recognize the unreliability of direct questioning to price a subject matter. One study
in particular shows that what a respondent says how she would value differs from how she actually
reacts. A classic example is a survey conducted on MBA students regarding their career choice.61
When asked directly prior to making decisions, they ranked salary as the sixth most important factor in
their career choice.62
However, salary was the most important factor influencing their choice as the
conjoint analysis analyzed after they actually had accepted a position.63
This study has made researchers
doubt the reliability of the data obtained from direct questioning. Academics also recognize that focus
bias (or hypothetical bias, which is similar to a leading question),64
upward bias for socially sensitive
60
Green et al., supra note 45, at S60 (originally from Green & Srinivasan, supra note 58, at 106).
61
Expert Report of Dr. V. Srinivasan (Blue-ray Players) at 7, TV Interactive Data Corp. v. Sony Corp., No. 4:10–cv–00475–
PJH (N.D. Cal. Jan. 21, 2013), ECF No. 580–1 [hereinafter Srinivasan Report] (citing David B. Montgomery, Conjoint
Calibration of the Customer/Competitor Interface in Industrial Markets, in INDUSTRIAL MARKETING: A GERMAN-AMERICAN
PERSPECTIVE 297–319 (Klaus Backhaus & David T. Wilson eds., 1985)).
62
Id.
63
Id.
64
Sentius Int’l, LLC v. Microsoft Corp., No. 5:13–cv–00825–PSG, 2015 U.S. Dist. LEXIS 8782, at *19–20 (N.D. Cal. Jan.
23, 2015). In Sentius, the survey expert recognized that the respondents might have overestimated their willingness-to-pay
for a particular feature of a product when answering direct open-ended questions. Id. at * 20. Therefore, the expert used a
calibration factor of 1.46 to adjust the respondents’ overestimation. Id.

13. 13
issues,65
and/or ambiguity66
may taint the reliability of the responses obtained from direct (and open-
ended) questioning.
Conjoint analysis handles these concerns by asking respondents to compare several alternatives
and evaluate the alternatives with respect to each other. The bundles-of-attributes description works in
two ways. On one end, respondent’s choice does not expose her preference on a particular feature. The
choice reveals (before the data is processed and analyzed) her preference only on the set of features, i.e.,
the product. On the other end, the choice set does not lead respondent’s focus to a specific feature.
Thus, the question does not unduly influence her choice.
Providing comparable alternatives also makes respondents’ decision-making process more
realistic because it resembles our day-to-day decision-making process. The bottom line is that pricing
an isolated feature from a multi-component/function system is not the way ordinary consumers make
decisions in real life. Conjoint analysis creates a hypothetical market place. If carefully designed and
executed, it can approximate the real world realistically.
The benefits conjoint analysis has provided to marketing researchers and professionals are much
more than a mere theoretical plausibility. The validity of the major assumptions underlying the theory
of conjoint analysis has been empirically tested and updated during the last several decades.67
Although
not a perfect tool, conjoint analysis has been proven to work well in practice in a wide range of
industries.
Academics have published hundreds of peer-reviewed papers on the theory and practice of
conjoint analysis.68
Product developers and marketers have applied conjoint analysis in almost every
commercial area where the consumption of goods and services occurs including healthcare and
65
Upward bias might be created because “respondents do not wish to come across as ‘cheap’” when direct questions were
asked. Srinivasan Report, supra note 61, at 7.
66
Paul E. Green & V. Srinivasan, Conjoint Analysis in Marketing: New Developments with Implications for Research and
Practice, 54 J. MARKETING 3, 9 (1990) (“‘How important is attribute X?’ is highly ambiguous because the respondent may
answer on the basis of his or her own range of experience over existing products rather than on the experimentally defined
range of the attribute levels.”).
67
An early reliability study on conjoint analysis compared its different methodological variants. David Reibstein et al.,
Conjoint Analysis Reliability: Empirical Findings, 7 MARKETING SCI. 271 (1988). Authors reached the conclusion that “the
conjoint method under a variety of methods of data collection and across a number of product categories appears to be
reliable in an absolute sense.” Id. at 284. More recently, a group of academics developed a new survey method called
“polyhedral method” for the choice-based conjoint survey. (Part II.C. explains choice-based conjoint analysis in detail.)
This method was successfully applied to support “the design of new executive education programs for a business school at a
major university.” Oliver Toubia et al., Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis, 41 J. MARKETING
RES. 116, 126–29 (2004).
68
HAUSER & RAO, supra note 57, at 2 (noting in 2002 that Professor Paul Green, a leading scholar in marketing research,
himself has contributed almost 100 articles and books on conjoint analysis).

14. 14
education.69
Conjoint analysis has been conducted in the design or development of: AT&T’s first
cellular telephone; IBM’s workstation; FedEx’s service; MasterCard features; Monsanto’s herbicide
packaging; Polaroid’s instant camera design; Marriott’s time-share units; and Ritz Carton’s hotel décor
and services.70
And the list goes on. Government also has applied conjoint analysis: it was used
successfully in New Jersey’s and New York’s EZ-Pass toll collection project, and in the design of U.S.
Navy’s benefit packages for reenlistment.71
II. TWO TYPES OF CONJOINT ANALYSIS
This Part explains two types of the conjoint analysis based on a hypothetical situation. First, a
traditional full-profile rating (or scoring) method will be described. While this method has limitations, it
remains the most common form of conjoint analysis.72
The obvious weakness is that the respondent’s
burden grows dramatically as the number of attributes and levels increases. (The laptop example in Part
I.B, even with its rather simple configurations, could generate 72 profiles.)
Next, choice-based (or discrete choice) conjoint analysis will be discussed. This form of the
conjoint analysis is more realistic and natural for the consumer’s decision-making behavior. Consumers
consider a bundle of features and make trade-offs, but they usually do not rate or rank a series of
products prior to the purchase. In addition, discrete choice analysis offers powerful benefits, including
the ability to do a better job of modeling interactions between attributes and the flexibility to incorporate
alternative-specific attributes.73
The downside is that the choice-based method generally requires more
data to estimate the model parameters. Because the analyst needs to infer the partworth based on the
information obtained from the responses of choice-based questionnaires, each response provides
substantially limited information.
69
See GUSTAFSSON ET AL., supra note 45, at 3 (“Based on a 2004 Sawtooth Software customer survey, the leading company
in Conjoint Software, between 5,000 and 8,000 conjoint analysis projects were conducted by Sawtooth Software users during
2003.”); John F. P. Bridges et al., Conjoint Analysis Applications in Health – a Checklist: A Report of the ISPOR Good
Research Practices for Conjoint Analysis Task Force, 14 VALUE IN HEALTH 403 (2011); Green et al., supra note 45, at S67;
Toubia et al., supra note 67, at 126–29.
70
See e.g., Green et al., supra note 45, at S67.
71
Id. at S68.
72
HAUSER & RAO, supra note 57, at 8.
73
OMRE, supra note 44, at 33.

15. 15
A. Hypo74
A car manufacturer, MANU, is planning to launch a new model that uses light-emitting diodes
(LEDs) on its headlight. Before investing a chunk of money on R&D and implementation, MANU
wants to know how consumers would value its new model with LED headlights. It determines that only
three features would influence consumers’ decision-making. The three attributes MANU chooses are: (1)
whether the headlights are LEDs; (2) fuel efficiency as measured by mile-per-gallon (mpg); and (3)
price. It decides to test for two levels of the fuel efficiency: 30 mpg and 35 mpg. Price levels are set at
$30,000, $32,000, and $34,000, as MANU’s new model targets in that price range.
Table 1 summarizes three attributes and each level of the corresponding attributes. The first
column indicates that, for convenience, each level of an attribute will be coded with numbers 0, 1, or 2.
Accordingly, each mode/profile can be described by a set of codes. For example, a model that does not
have LEDs (“0”), with fuel efficiency of 35 mpg (“1”), and with a price tag of $32,000 (“1”)
corresponds to (0, 1, 1). Similarly, a model with LEDs and 30 mpg, and sold at $34,000 is represented
by (1, 0, 2).
Table 1. Attributes and levels for MANU cars.
Attributes/Levels LEDs Fuel Efficiency [mpg] Price [$]
0 No 30 30,000
1 Yes 35 32,000
2 34,000
B. Full-Profile Rating Model
The combination of the attributes and levels can generate up to 12 ( 322 ×× ) profiles. In a full-
profile survey, respondents are shown a set of cards that fully describes the profile. Here, each card
describes: whether the headlights are made of LEDs; whether the fuel efficiency is 30 mpg or 35 mpg;
and the price of the car as one of the three amounts – $30,000, $32,000, or $34,000. The respondent in
74
The same fact pattern can be used in the context of a patent infringement suit, where the patentee attempts to evaluate the
value of the LED headlights. For example, an inventor INVE has a patent that claims a method of manufacturing LEDs.
MANU sells a model that INVE thinks incorporates the LEDs manufactured by the method claimed in his patent. While
MANU offers sub-models with a range of prices and other features, INVE cannot be certain about the value of the LED
headlights as all MANU models use the same LED headlights INVE thinks infringing. INVE conducts a conjoint survey
assuming that only the same three attributes characterize MANU cars and affect consumers’ purchasing decision.

16. 16
this survey is given 12 cards, each representing a profile, and asked to rate each profile in between one
and twenty with a higher-rated profile indicating a more attractive option. While the respondent saw a
full description of each profile, the data gathered is expressed as codes for the analysis. Table 2
summarizes a list of profiles and the scores the respondent rated. Stimuli column simply identifies the
profile.
Table 2. Full-profile rating result.75
Stimuli Profile Score/Rate
1 0, 0, 0 9
2 0, 0, 1 7
3 0, 0, 2 4
4 0, 1, 0 16
5 0, 1, 1 13
6 0, 1, 2 10
7 1, 0, 0 12
8 1, 0, 1 8
9 1, 0, 2 6
10 1, 1, 0 18
11 1, 1, 1 15
12 1, 1, 2 11
The respondent assigned 18 points to (1, 1, 0) and 11 points to (1, 1, 2). Points represent the
relative partworths of each profile. So, relatively speaking, to this respondent, profile (1, 1, 0) provides
her with 7 more utility than profile (1, 1, 2) does. The comparison between these two profiles makes
sense, as all conditions being equal, a more expensive car is less attractive than the cheaper one.
Comparing profiles (1, 1, 0) and (1, 0, 0), the respondent favors the former with 6 extra points. Again,
the result makes sense because, assuming other two attributes remain identical, improvement on fuel
efficiency would add some value.
75
The data set here is generated assuming a certain set of the partworth for each attribute only to illustrate how this method
works. Because the score of each profile is calculated systematically with the predetermined partworth in mind, the
parameter estimates obtained by the least squares regression fit very well, and, thus, are reliable, as the statistics in Table 3
show.

17. 17
Equation (1) takes a simple linear, additive form. It describes the relationship between the total
utility of a hypothetical vehicle model and three attributes of interest. The utility of each profile
( ofilenU Pr, ) is the sum of the partworth of all attributes constructing the profile. Unknown parameters (or
dummy variables) { Attributesn,β } are weights of independent (or explanatory) variables { AttributesX }. The
goal of the conjoint analysis is to find a set of { Attributesn,β } using given data set { ofilenU Pr, , AttributesX } with
the assumption that the data set follows equation (1).
iceicenEfficiencyEfficiencynLEDLEDnjn XXXU PrPr,,,, βββ ++= . (1)
Subscript n denotes that { ofilenU Pr, } and { Attributesn,β } are individual’s utility and weights for the
attributes. Each respondent may assign different weights on each attribute. That means each respondent
would rate differently, creating a new table of data set. Still, equation (1) can be expanded to represent
the market’s view on the partworth of each attribute in two ways. Assume that there are 100=N
respondents. One, { Attributesn,β } for each individual can be first estimated. Then, 100=N set of the
estimates can be averaged in a certain way. Two, { ofilenU Pr, , AttributesX } for all 100=N respondents can
be first aggregated and averaged. Then, { AttributesN ,β } can be estimated, which represent the market’s
(here that of the sample size of 100=N ) partworth.
Ordinary linear squares (OLS) regression is a natural and efficient means with which to estimate
{ Attributesn,β }, in this model.76
Regression analysis is a statistical process for estimating relationship
among variables. 77
First, the linear regression model takes the form similar to equation (1):
jiceiceEfficiencyEfficiencyLEDLEDj XXXY εββββ ++++= PrPr0 . (2)78
jY is a dependent variable. Total utility of the profile is represented by the points assigned to { jY }. 0β
is an intercept, i.e., a constant value that adjusts the level of the points. jε is the random error term that
76
HAUSER & RAO, supra note 57, at 12; see Paul E. Green & Yoram Wind, New way to measure consumers’ judgments, 53
HARV. BUS. REV. 107 (1975).
77
WIKIPEDIA, http://en.wikipedia.org/wiki/Regression_analysis (last visited Apr. 15, 2015).
78
Subscript n is taken out to simplify the equation.

18. 18
accounts for the collective unobservable influence of any omitted explanatory variables. Typically,
although not necessary for least squares to be used, this term is assumed to follow normal distribution.
In OLS regression analysis, { Estimatesβ } are estimated by fitting the equation to the data so that the sum of
the squared deviation of the data from the line are minimized (thus, least squares).79
The best set of
{ Estimatesβ } with the minimum deviation from the model approximates the data set most accurately.
Each response generates a data point. For example, for the 9th stimuli with a profile (1, 0, 2) and
the 11th stimuli with a profile (1, 1, 1), equation (2) becomes
9Pr0 26 εβββ +++= iceLED (3)
11Pr015 εββββ ++++= iceEfficiencyLED (4)
respectively. An individual thus has generated total 12 data points (or equations) that can be used to
estimate four unknown parameters { 0β , LEDβ , Efficiencyβ , icePrβ }.
A simple regression analysis was applied to the data set in Table 2 using a Regression Data
Analysis tool in Microsoft Excel. Table 3 summarizes the result. The Coefficients column corresponds
to the estimates of four { Estimatesβ }. Inserting the estimates, equation (2) becomes
iceEfficiencyLED XXXY Pr15.397.563.12.10ˆ −++= . (5)
Hat on Y
⌢
is used to indicate that Y
⌢
is an estimate, not a data point. The total utility of a car can be
predicted by using equation (5) (with a limitation80
). Because a higher price diminishes the total utility,
the weight of the price term has a negative sign.
In addition to parameter estimates (under Coefficients column), OLS regression results provide
an estimate of the reliability of the parameter estimates and a measure of the overall goodness of fit of
the regression model.81
The standard error of each estimate (listed in Standard Error column) shows the
magnitude of variations in the data set. The greater the variation in the data, the larger the standard error
79
SHARI SEIDMAN DIAMOND, Reference Guide on Survey Research, in REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 333–
51 (The Federal Judicial Center ed., 3d ed. 2011), available at
http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf.
80
For example, extrapolation typically generates an unreliable prediction.
81
DIAMOND, supra note 79, at 340.

19. 19
and the less reliable the regression results become.82
It should be noted that the absolute value of
standard errors might mislead the interpretation. For example, the standard error value of 1 is much
greater than that of 0.01. However, if the coefficient estimated in the first case is in the order of 100 and
the latter is in the range of 0.001, the reliability of the estimate is much higher in the first case.
Table 3. Regression Result
Regression Statistics Coefficients Standard Error t-Stat
R-squared ( 2
R ) 0.99393 0β 10.20000 0.31848 32.02728
Standard Error 0.41115 LEDβ 1.63333 0.25271 6.46327
Efficiencyβ 5.96667 0.25271 23.61071
icePrβ -3.15000 0.15924 -19.78155
T-statistics might be used to cure the potential misinterpretation. The t-statistic is defined as the
ratio of the parameter estimate to its standard error. Thus, the t-statistics for the previous example are
1
100100 = and
01.0
001.01.0 = , respectively. T-statistics can be interpreted in terms of confidence
intervals. In general, a confidence interval around any parameter estimate can be constructed such that a
95% confident interval is a range of values that one is 95% confident that the true value of parameter is
within that confident interval.83
Accordingly, when the t-statistic is less than 1.96 in magnitude, the 95%
confidence interval around the parameter must include zero, and the estimate is said to be not
statistically significant.84
Conversely, if the t-statistic is greater than 1.96 in absolute value, it can be
concluded that the true value of the estimate is unlikely to be zero, and the estimate is statistically
significant.85
The t-Stat column in Table 3 demonstrates that the estimated { Estimatesβ } in this hypo are
statistically significant. The absolute values are significantly greater than 1.96 for each and every
82
Id. at 341.
83
Id. at 342. Unlike Gaussian distribution, the 95% confidence interval of the t-distribution is not fixed to 1.96. Rather, it
varies by the degree of freedom of the t-distribution, which is equal to the difference between the number of samples and the
number of parameters to be estimated. WIKIPEDIA, http://en.wikipedia.org/wiki/Student%27s_t-
distribution#Table_of_selected_values (last visited Apr. 17, 2015). While validity of the Gaussian distribution is predicated
on a large sample size, the t-statistic applies to any sample size. See DIAMOND, supra note 79, at 343 n.82. The t-distribution
approximates the normal distribution as the sample gets large. Id.
84
Id. at 342–43.
85
Id. at 343.

20. 20
coefficient. For example, the 95% confidence interval for LEDβ is (1.03577, 2.23090) and icePrβ is (-
3.52654, -2.77346). Both intervals do not include zero.
In addition to the estimate of the reliability of the parameter estimates, the regression result also
provides a measure of the overall goodness of fit, i.e., how well the regression equation fits the data. R-
squared (or 2
R ) is a statistic that “measures the percentage of variation in the dependent variable that is
accounted for by all the explanatory variables.”86
Its value ranges from 0 (the explanatory variables
explain none of the variation of the dependent variable) to 1 (the explanatory variables explain all of the
variation of the dependent variable). While there is a no clear-cut level that makes the model
satisfactory, a high 2
R is favored.87
In this example, 2
R is very high (0.99393), which means that the
three attributes explain more than 99% of the variation in this particular respondent’s rating (and total
utility).
From Table 3 and equation (5), this respondent’s willingness-to-pay for the LED headlights in
MANU’s car can be derived. 15.3Pr −=iceβ means that the increase in price from $30,000 to $32,000
(or from $32,000 to $34,000) decreases the relative worth of the car at the rate of 3.15 units. That is,
$2,000 corresponds to the partworth of 3.15 units, and one unit of partworth represents
util
util
15.3
000,2$]/[$635 = . As the respondent’s utility with the LED headlights increase by 1.63
units, this adds 035,1$]/[$635][63.1 =× utilutil worth of value to the respondent. Thus, this
respondent’s willingness-to-pay for the LED headlights is $1,035 (when the price of the car ranges in
between $30,000 and $34,000).
A different approach would lead to the same result. Here, the task is to estimate the price
premium the respondent would be willing-to-pay for the LED headlights. Assume that the base model
has 35 mpg with a price tag at $32,000. The one model that does not have the LED headlight feature is
described by using equation (5) as
02.1315.397.52.10 =−+=NoY (6)
86
Id. at 345.
87
Id.

21. 21
because 0=LEDX , 1=EfficiencyX , and 1Pr =iceX for this alternative. Conversely, the alternative with the
LED headlight features is
iceiceYes XXY PrPr 15.38.1715.397.563.12.10 −=−++= (7)
as now 1=LEDX and 1=EfficiencyX .
Consumer’s willingness-to-pay for the particular feature is drawn by solving equations (6) and (7)
when LEDNo YY = . At that price, the consumer does not favor one alternative over the other. Solving the
equation results in 517.1Pr =iceX . Assuming that the utility for the price varies linearly in between
$32,000 ( 1Pr =iceX ) and $34,000 ( 2Pr =iceX ), 517.1Pr =iceX corresponds to $33,035. In short, the
respondent’s willingness-to-pay a price premium for the LED headlight is, again, $1,035 (over $32,000).
Assuming that the data in Table 2 represents the market data, the same result is interpreted to represent
market’s willingness-to-pay for this particular feature of the product.
C. Choice-Based or Discrete Choice Model
The choice-based model differs from the previous model because the dependent variable in
choice-based model is discrete. For example, when the question is the choice between only two
alternatives, the answer is either yes (prefer or purchase) or no. Similarly, the output for the choice task
where more than two alternatives are available is still binary – yes or no for each alternative.
The utility function in the choice-based conjoint analysis can be modeled by using a similar
approach as in the rating-based approach. (See equation (12) below.) However, as opposed to a
traditional form consisting of unknown but certain parameters, the choice-based model assumes that the
respondent’s utility is represented by random variables having a probabilistic distribution.88
The model,
thus, is called a random utility model (RUM).89
88
JORDAN J. LOUVIERE ET AL., STATED CHOICE METHODS: ANALYSIS AND APPLICATION 38 (Louviere et al. eds., 2000).
89
HAUSER & RAO, supra note 57, at 10.

22. 22
The probability of any alternative i being selected by a person n from a choice set nC depends on
the probability that the utility of the alternative i ( niU , ) is higher than any other available alternatives
from the set.90
It is expressed that:
),()|( ,, nnjnin CijUUPCiP ∈≠∀>= . (8)
Equation (8) reads: when the alternatives j is not identical to i, and both are from the choice set ( nC ), the
probability of a person n choosing the alternative i is equal to the probability that the utility of the profile
i is higher than any other available alternatives in the set nC .
The distinctive characteristic of RUM is that it models the utility function as a combination of
two parts: the systematic (or representative) component ( niV , ) and the random (or error) component
( ni,ε ) such that
ninini VU ,,, ε+= . (9)
The random component makes the utility a random variable. Inserting equation (9) into equation (8) and
rearranging it yields equation (10). Equation (11) is a simplified form of equation (10).
),()|( ,,,, nnjnininjn CijVVPCiP ∈≠∀−<−= εε (10)
)()( ,,,, njnininjn VVPiP −<−= εε (11)
Note that the probability of choosing alternative i is determined as a function of the relative
values of random components ( ninj ,, εε − ) and systematic components ( njni VV ,, − ). The systematic
component represents the levels and attributes of the alternatives with the assumption that they are linear
and additive:
iceicenEfficiencyEfficiencynLEDLEDnni XXXV PrPr,,,, βββ ++= . (12)
90
LOUVIERE ET AL., supra note 88, at 59.

23. 23
Now, the discrete choice model relies on what distributional forms or assumptions the error
terms take. For example, if the error terms are multivariate normal, the probit model is obtained.91
The
prevalent form of utility function in discrete choice model is called multinomial logit.92
Multinomial
logit model presumes that ni,ε and nj,ε are independent and identically distributed.93
Under the
assumption,
ε
εεε
−
−
=−−=< e
j eP )expexp()( . (13)94
Incorporating equations (11) and (12) to equation (13), and arranging it to a simple form, equation (11)
becomes
∑∈
=
n
nj
ni
Cj
V
V
n
e
e
iP ,
,
)( . (14)95
The way the survey question is presented in the choice-based conjoint survey differs from that of
the full-profile rating survey described in the preceding section. Here, a respondent n is given a set of
options (choice set, nC ) and asked to pick one profile. For example, assume that the choice set consists
of profiles 5, 6, 7, and 12, and further assume that the respondent n selects profile 5, in consistent with
the rating results shown in Table 2.
This decision generates three inequality constraints (or data points). By choosing profile 5 when
three other alternatives were available, the respondent n stated her preference of 5 over 6, 7, and 12.96
91
HAUSER & RAO, supra note 57, at 10; see BEN-AKIVA & LERMAN, supra note 53, at 69–70 (deriving binary probit model).
92
See Green et al., supra note 45, at S64 (“In choice-based conjoint analysis, analysts typically employ multinomial logit
models, although occasionally they use probit-based models.”).
93
LOUVIERE ET AL., supra note 88, at 44–47. This is often known as the Independence-from-Irrelevant Alternatives (IIA)
axiom. It states that “the ratio of the probabilities of choosing one alternative over another (given that both alternatives have
a non-zero probability of choice) is unaffected by the presence or absence of any additional alternatives in the choice set.” Id.
at 44. “Satisfaction of the IIA condition, however, should not be of general concern because the independence assumption is
a priori neither desirable nor undesirable, but should be accepted or rejected on empirical grounds depending on the
circumstances.” Id. at 45.
94
Id.
95
BEN-AKIVA & LERMAN, supra note 53, at 71; LOUVIERE ET AL., supra note 88, at 47.
96
Three inequality constraints are: nn UU ,6,5 > ; nn UU ,7,5 > ; and nn UU ,12,5 > .

24. 24
The probability of selecting profile 5 from this choice set is obtained by combining equation (14) and
equation (12):
nnnn
n
VVVV
V
eeee
e
P ,12,7,65
,5
)5(
+++
=
icenEfficiencynLEDnLEDnicenEfficiencynicenEfficiencyn
icenEfficiencyn
eeee
e
Pr,,,,Pr,,Pr,,
Pr,,
22 ββββββββ
ββ
++++
+
+++
= . (15)
In the choice-based conjoint analysis, each respondent is typically provided with about fifteen
choice sets, each consisting of four profiles.97
Thus, one respondent supplies 153× data points.
Typically, many more data points are required to estimate model parameters reliably in discrete choice
model as both the choice and the utility take the probabilistic forms. While a set of data for an
individual respondent can be estimated first and then aggregated with that of the others in the rating
study, the choice-based model prefers aggregating the data prior to estimating model parameters. The
natural result is that under the choice-based model, the market’s utility and willingness-to-pay, rather
than those of individuals, are obtained.
As the utility is considered as a random variable, and each choice decision is expressed as a
probability, the likelihood of the entire set of data (e.g., 153× data points for the respondent n) is the
product of the likelihoods of the individual observation as expressed in equation (16).98
Therefore, the
estimation method seeks to find the set of model parameters { nβ } that maximizes the likelihood of
generating the entire set of data. The estimation method is referred to as the maximum likelihood
estimation (MLE).
∏∏= =
=
N
n
I
i
f
n
ni
ipL
1 1
* ,
)( . (16)
Equation (16) indicates that the likelihood of each observation is multiplied across the sample:
that is, n ranges from 1 to N (the total number of the survey respondents). All answers from each
respondent are also multiplied: that is, i ranges from 1 to I (the total set of stimuli, fifteen in the hypo).
97
See Hauser Report, supra note 54, at 35–40 (sixteen choice tasks with four alternatives); Srinivasan Report, supra note 61,
at 14–15 (four options presented in each of 15 choice sets).
98
BEN-AKIVA & LERMAN, supra note 53, at 80–81, 118–20; LOUVIERE ET AL., supra note 88, at 47–50.

25. 25
nif , is a dummy variable such that 1, =nif if respondent n chose the alternative i and 0, =nif if
otherwise. Note that in equation (16), when the alternative is not chosen, that does not affect the
likelihood function because 10,
== pp nif
.
From equation (16), log likelihood function L can be written as
∑ ∑= =
==
N
n
I
i nni ipfLL 1 1 ,
*
)(lnln . (17)
As the set of { nβ } that maximizes L also maximizes *
L , finding the set of solutions that maximize
equation (17) best estimates the discrete choice problem.99
Substituting )(iPn with equation (14) gives,
∑ ∑ ∑∑ ∑
∑ = =
∈
= =
∈
−==
N
n
I
i
Cj
V
nini
N
n
I
i
Cj
V
V
ni
n
nj
n
nj
ni
eVf
e
e
fL 1 1 ,,1 1 , )ln(ln ,
,
,
. (18)
Note also that L is a function of { niV , }, which in turn are functions of the utility parameters, { nβ }.
Furthermore, it is known that L is a convex function of { nβ }.100
Thus, equation (18) can be maximized
with respect to { nβ } by using some non-linear maximization algorithms.101
A good property of convex
optimization problem is that we can always achieve globally optimal value no matter what initial
conditions are assumed, e.g., 0}{ =nβ .
Such algorithms are usually iterative.102
Typically, initial values of the { nβ } are guessed and
used in equation (12) to calculate { niV , }. The initial set of { niV , } are then used in equation (14) to
calculate { )(iPn }. These values are used in equation (17) to calculate a starting value of L . The
procedures are repeated by changing { nβ } systematically until the increase in L reaches a
predetermined level of tolerance.
99
LOUVIERE ET AL., supra note 88, at 50, 66–71.
100
Id. at 66–67.
101
Id. at 50–51, 70–71.
102
See infra Part II.D.

26. 26
The choice-based model significantly differs from the rating-based model in two aspects. First,
from the modeling perspective, the dependent variable, which represents consumer preferences, is
discrete, not continuous. Due to this discrete nature, the choice and the utility are defined in terms of
probability distributions. Second, the MLE method is applied to estimate model parameters. The choice
of estimation method also relates to the discrete and probabilistic nature of the choice problem. As the
objective of the parameter estimation is to approximate the entire data set, and the data consists of
individual probability distribution, the optimal set of parameter estimates is the one that maximizes the
likelihood of reproducing the entire data set as closely as possible.
Next section briefly describes more advanced estimation method called Hierarchical Bayes
method.103
While this method “transformed the way discrete choice studies were analyzed,” it also
applies to rating-based conjoint analysis with incremental benefits.104
D. Hierarchical Bayes (HB) Estimation Method
The approaches taken so far are examples of conventional (non-Bayesian) statistical analyses.
Under the non-Bayesian approach, the probability distribution of the data is investigated, conditioned on
the assumptions embodied in the model (or hypothesis) and its parameters. In Bayesian analysis, the
probability distribution of the parameter, given the data, is investigated.
Under the Bayes theorem, the probability of a particular hypothesis ( H ) given the data ( D ) is
)(
)(
)|(
DP
DHP
DHP
∩
= . (19)
Similarly, because
)(
)(
)|(
HP
HDP
HDP
∩
= (20)
and )()( HDPDHP ∩∩ = , rearranging equations (19) and (20) yields equation (21).
103
Experts in two recent patent infringement cases used this method to estimate market’s willingness-to-pay in choice-based
conjoint analyses. Hauser Report, supra note 54; Srinivasan Report, supra note 61.
104
OMRE, supra note 44, at 34.

27. 27
)(
)()|(
)|(
DP
HPHDP
DHP
×
= . (21)
Here, the probability of the hypothesis given the data, )|( DHP , is known as its posterior
probability.105
This is the probability of the hypothesis that reflects the data upon which the hypothesis
is based. The probability of the hypothesis, )(HP , is known as its prior probability. It describes the
analyst’s belief about the hypothesis before she saw the data. The probability of the data given the
hypothesis, )|( HDP , is known as the likelihood of the data. It is the probability of seeing that
particular collection of data, conditioned on that hypothesis about the data. Thus, equation (21) tells that
the posterior probability of the hypothesis is proportional to the product of the likelihood of the data
under that hypothesis and the prior probability of that hypothesis. The Bayesian framework provides a
formula to update the prior estimate of the probability.
The HB method analyzes the model in a hierarchical form.106
That is, the model parameters in
one level (or hierarchy) are explained in subsequent levels. Re-writing equation (10) in a simple form,
we obtain
),()( ,,,, ijVVPiP njnjnini ≠∀+>+= εε . (22)
Re-writing equation (12) in a generalized form with more than three attributes, we obtain.
nnini xV β,,
ʹ= . (23)
nix ,
ʹ , attributes of the alternative i for the n respondent is expressed in a vector form. The HB method
further assumes that individual partworth, }{ nβ , has the multivariate normal distribution, i.e.,
),(~ ∑β
ββ Normaln (24)
105
The ACA/HB Module for Hierarchical Bayes Estimation v3, SAWTOOTH SOFTWARE INC., 6 (July 2006) [hereinafter
ACA/HB Module], https://www.sawtoothsoftware.com/download/techpap/acahbtec.pdf.
106
Greg M. Allenby & Peter E. Rossi, Perspectives Based on 10 Years of HB in Marketing Research, SAWTOOTH SOFTWARE
INC., 3 (2003), http://www.sawtoothsoftware.com/download/techpap/allenby.pdf.

28. 28
where β is the mean of the distribution of individuals’ partworth and ∑β
denotes (a matrix of)
covariances of the distribution of the partworths across individuals.
Equation (22) describes the lowest level of the hierarchy. The same interpretation given in
equations (8)-(11) applies to equation (24). At the highest level, equation (24) allows for heterogeneity
among the respondents (i.e., variance within the sample). The individual partworth estimate, }{ nβ , are
linked by a common distribution, { ∑β
β , }, which represents the sample. (Note that { ∑β
β , } do
not contain individual-specific subscript, such as n.) { ∑β
β , } are known as hyper-parameters of the
model.107
Thus, in theory, estimates of individual level parameters give data, )|( DP nβ , can be obtained
by first obtaining the joint probability of all model parameters given the data,
),|()|( | ∑∏ × β
βββ n
n
nn PDP . And,
)|,},({)|( DPDP nn ∑= β
βββ
)(
),()],|()|([
)(
)()|( |
DP
PPDP
DP
PDP n
n
nn
nn
∑∑∏ ××
=
×
=
ββ
ββββ
ββ
. (25)108
Integrating equation (25) results in
∑∫ ∑ −= ββ
βββββ dddDPDP knn )|,},({)|( . (26)109
“-k” in k−β denotes “except k.” Equations (25) and (26), thus, provide an operational procedure for
estimating a specific individual’s set of partworths, }{ nβ , given all the data in the sample, D , not
merely her data, nD . As such, the Bayes theorem provides a method of bridging the analysis across the
respondents, which conjoint analysis essentially attempts to achieve.
107
Id. at 4.
108
Id.
109
Id.

29. 29
Because usually equation (26) is impossible to be solved analytically, an iterative process is used
to estimate the model parameters. The so-called “Monte Carlo Markov Chain” method provides an
efficient algorithm to HB problems. Starting with a set of initial values { mβ , mβ , ∑ mβ
, mσ }, with
0=m , each iteration consists of following four steps to generate a set of updated values, { 1+mβ , 1+mβ ,
∑ +1mβ
, 1+mσ }.
1) Using present estimates of mβ , ∑ mβ
, and mσ , generate new estimates of 1+mβ .
2) Using present estimates of mβ and ∑ mβ
, generate a new estimate of 1+mβ .
3) Using present estimates of mβ and mβ , draw a new estimate of ∑ +1mβ
.
4) Using present estimates of mβ , ∑ mβ
, and mβ , generate a new estimate of 1+mσ .110
As the four steps show, one set of parameters is re-estimated conditionally in each step, given current
values for the other three.
Compared to its traditional counterparts, the HB estimation method provides two significant
benefits. First, as Bayesian statistical analysis, HB has the advantage of using prior data to update next
parameter estimation. Using available data (e.g., utility decreases as price increases, or utility increases
as the processor gets faster and/or the storage capacity becomes larger) enhances the reliability of
conjoint analysis.111
This aspect of the HB method also makes it easier to update the next question
based on respondent’s previous answers, making the survey more efficient.112
Second, its hierarchical
form provides a more realistic estimation platform when the heterogeneity (i.e., variance within the
sample) of the data is high. The HB approach accounts for the heterogeneity present in the data set and
bridges the individuals’ responses across the sample.
110
ACA/HB Module, supra note 105, at 6.
111
Greg M. Allenby et al., Incorporating Prior Knowledge into the Analysis of Conjoint Studies, 32 J. MARKETING RES. 152
(1995); see also Hauser Report, supra note 54, at 41.
112
The underlying mechanism is similar to that of the general computer-based tests (CBTs). The next question (or choice set)
is selected from the pool of questions (or redesigned) in response to exam-takers’ (or survey respondents’) previous answers.
See generally Oliver Toubia et al., Probabilistic Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis: Theory
and Application, 26 MARKETING SCI. 596 (2007). However, adding prior information is not limited to the HB estimation
method. Traditional regression methods can also incorporate prior information for data analysis. HAUSER & RAO, supra note
57, at 12.

30. 30
So far, underlying principles of two major types of conjoint analysis rest on sound mathematical
theorems. The admissibility of the expert testimony based on conjoint analysis largely depends on the
minor premise. Expert’s major battleground is on how the survey gathers and supplies the data and the
validity of the statistical analysis rendered on the data. Part III identifies eight key areas where the
admissibility and credibility of conjoint survey evidence can be challenged.
III. CONJOINT ANALYSIS AS SURVEY EVIDENCE
Since conjoint analysis is a survey-based research tool, issues pertinent to admissibility and
credibility of the survey evidence apply to expert opinions based on conjoint analysis. Over the last half
century, the survey method has been proved to be an economical and systematic way to obtain data and
draw inferences about a large number of individuals or other units.113
A complete census of the universe
can be expensive, time-consuming, and sometimes impossible. With the increasing uses of the surveys
by academic researches, businesses, and the government, both federal and state courts have admitted
survey evidence on a variety of contexts such as: discrimination in jury panel composition; employment
discrimination; class certification; community standards for obscenity; antitrust; mass torts; consumer
perception and memory; trademark infringement; and patent infringement cases.114
One federal court in
a trademark infringement case treated the absence of a survey as the plaintiff’s failure to establish actual
consumer confusion.115
While survey evidence has been attacked as inadmissible on the theory that it is inadmissible
hearsay, the contemporary view is that the hearsay objection is unsound.116
The respondents’ answers
are either nonhearsay or admissible as exceptions to the hearsay rule as declarations of present state of
mind or under the residual exception.117
Under the Federal Rule of Evidence and Daubert, the
admissibility of survey result centers on the “validity of the techniques employed rather than [on]
relatively fruitless inquires whether hearsay is involved.”118
The key is on the quality of survey. To be admissible, surveys should generally satisfy following
foundational requirements: (1) a relevant population (or universe) was properly defined; (2) a
113
DIAMOND, supra note 79, at 367.
114
Id. at 364–67; see, e.g., Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9–
10 (N.D. Cal. June 30, 2012).
115
Morrison Entm’t Grp. v. Nintendo of Am., 56 F. App’x 782, 785 (9th Cir. 2003); DIAMOND, supra note 79, at 372
(“[S]everal courts have drawn negative inference from the absence of a survey, taking the position that failure to undertake a
survey may strongly suggest that a properly done survey would not support the plaintiff’s position.”).
116
1 PAUL C. GIANNELLI ET AL., SCIENTIFIC EVIDENCE, § 15.04[b], at 851–52 (5th ed., 2012).
117
Id.
118
FED. R. EVID. 703 advisory committee’s note.

31. 31
representative sample of that population was selected; (3) the questions were presented in a clear and
non-leading manner; (4) interview procedures were sound and unbiased; (5) the data was accurately
gathered and reported; (6) the data was analyzed in accordance with accepted statistical principles; and
(7) objectivity of the process was assured.119
This Part examines each factor in turn. The third factor
will be addressed in two parts. Issues relating to the determination of survey attributes and levels will be
examined first. Then, the focus will move to the phrasing and/or presentation of the questionnaires.
A. Relevant Population
A population or universe is a complete set or all the units of interest to the researcher. The target
population must be relevant to the questions of the survey.120
The starting point in the development of a
methodologically sound survey is identification of the appropriate population. Thus, courts have
considered the selection of proper population “as one of the most important factors in assessing the
validity of a survey as well as the weight that it should receive.”121
Leelanau Wine Cellar v. Black & Red is a trademark infringement case between competing
wineries in Michigan.122
Leelanau’s primary theory of the case was based on the secondary meaning of
the disputed mark and the resulting consumer confusion. Specifically, Leelanau alleged that Black &
Red’s “Chateau de Leelanau” mark caused consumers of the defendant’s products to mistakenly believe
that the defendant’s products were from the same source as, or were connected with, the plaintiff’s
products.123
Leelanau retained an expert to conduct a consumer survey to measure the extent to which
consumers who encounter defendant’s Chateau de Leelanau wines believed them to be the same as or
related to Leelanau Wine Cellars.124
Leelanau’s expert defined the universe as “Michigan consumers
over 21 years of age who had either purchased a bottle of wine in the $ 5 to $ 14 price range in the last
119
Leelanau Wine Cellars, Ltd. v. Black & Red, Inc., 452 F. Supp. 2d 772, 778 (W.D. Mich. 2006) (citations omitted). The
court added that, “[b]ecause almost all surveys are subject to some sort of criticism, courts generally hold that flaws in survey
methodology go to the evidentiary weight of the survey rather than its admissibility.” Id. Manual for Complex Litigation
suggests the same seven factors in assessing the admissibility of a survey. MANUAL FOR COMPLEX LITIGATION (Fourth) §
11.493 (2004). See also DIAMOND, supra note 79, at 367 (“Several critical factors have emerged that have limited the value
of some of [the] surveys: problems in defining the relevant target population and identifying an appropriate sampling frame, a
response rates that raise questions about the representativeness of the results, and a failure to ask questions that assess
opinions on the relevant issue.”).
120
DIAMOND, supra note 79, at 377 (“A survey that provides information about a wholly irrelevant population is itself
irrelevant.”).
121
Leelanau Wine Cellars, 452 F. Supp. 2d at 781.
122
Id. at 772.
123
Id. at 779.
124
Id.

32. 32
three months or who expected to purchase a bottle of wine in that price range in the three months
following the interview.”125
The court found this universe was flawed because it was significantly overbroad.126
The court
first concluded that when the dispute centers on secondary meaning of the mark, the proper universe is
the potential purchasers of defendant’s products.127
The court noted that Black & Red’s wines are
available only through their local tasting rooms and websites.128
That finding was crucial because, while
Leelanau’s universe would certainly include purchasers of the defendant’s wine, only a tiny percentage
of the respondents in Leelanau’s universe would probably purchase Black & Red’s wines in connection
with actual visits to its tasting rooms or websites.129
In another case, the court was more receptive to survey evidence. In a patent infringement suit
between competing smartphone manufacturers, Apple’s damages expert designed and conducted a
conjoint survey to determine price premium, if any, Samsung consumers would be willing to pay for the
touchscreen features associated with the Apple’s patents at issue.130
One of the arguments for
Samsung’s unsuccessful Daubert challenge was that Apple’s expert surveyed improper recent Samsung
purchasers, rather than potential Samsung purchasers.131
Rejecting Samsung’s motion to exclude the
expert opinion, the court first noted that Samsung failed to explain why recent Samsung purchasers are
not the proper universe for Apple’s survey.132
The court found that even if the category of recent
Samsung purchasers was underinclusive, they were at least members of the relevant universe of survey
participants.133
Concluding that the underinclusive population was still probative, the court stated,
“[g]enerally, underinclusiveness of a survey goes to weight, not admissibility.”134
B. Representative Sample
A sample is a subset of the population. The sample is drawn from the population for a particular
purpose of conducting the survey. Conjoint analysis quantifies sample’s preferences and estimates
sample’s partworth and willingness-to-pay. Thus, in addition to the appropriate identification of the
125
Id. at 782.
126
Id. at 782–83.
127
Id. at 782.
128
Id.
129
Id.
130
Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9–10 (N.D. Cal. June 30,
2012).
131
Id. at *9.
132
Id. at *10.
133
Id.
134
Id.; see also Microsoft Corp. v. Motorola, Inc., 905 F. Supp. 2d 1109, 1120 (W.D. Wash. 2012).

33. 33
relevant population, it is essential to select a sample that properly represents the relevant characteristics
of the population. The ultimate goal of the sample survey is “to provide information on the relevant
population from which the sample was drawn,” even though the data are incomplete as not obtained
from the population.135
Largely, there are two types of concerns – quantitative and qualitative – about the sample.
Quantitatively, the statistician wants a sample large enough to generate reliable statistics. Courts have
barred samples in some cases when the sample was too small to yield reliable statistics.136
The rule of
thumb is that a sample size exceeding thirty may provide stable statistics.137
More is better in general,
but not always so. When the sample is systematically skewed or biased, a larger sample size would
aggravate, not reduce, the systematic error (even though the standard error is inversely proportional to
the square root of the sample size).138
Qualitatively, the sample must be unbiased.139
Most survey researchers employ probabilistic
approaches in sampling.140
Probability sampling is known to maximize both the representativeness of
the survey results and the reliability of estimates obtained from the survey.141
Probability sampling
methods range from a simple random sampling method to complex multistage sampling schemes.142
In
a basic simple random sampling method, every element in the population has equal non-zero probability
135
DIAMOND, supra note 79 at 361.
136
See 1 GIANNELLI ET AL., supra note 116, at § 15.04[b], 858.
137
Id. at 845.
138
Id. n.98. The classic example of the large sample with the systematic bias is the Literary Digest’s 1936 presidential
election poll. The Literary Digest was one of the most popular magazines of that era and had a history of accurately
predicting the winners of presidential elections since 1916. It sent out 10 million straw ballots asking people who they
planned to vote for the 1936 presidential election. The magazine received 2.4 million ballots. Based on the responses, it
predicted Alfred Landon would beat Franklin D. Roosevelt 57% to 43%. As it turned out, Roosevelt won, with a whopping
margin of 62% to 37%. There were two huge problems with the poll. First, the initial sample (initial 10 million recipients)
did not correctly represent the population, i.e., voters. Literary Digest used lists of phone numbers, drivers’ registration, and
country club memberships to identify the sample. However, at the time, these luxuries were more often available to the
middle- and upper-class voters, which tended to exclude lower-income voters. On the other side, Roosevelt’s campaign
centered on reviving the economy at the height of the depression, which appealed to the majority of the lower income people.
Second, the sample chosen suffered from voluntary response (or nonresponse) bias, with a huge nonresponse rate of more
than 75%. A high nonresponse rate suggests that the voters who supported Roosevelt were less inclined to respond to the
survey. Famous Statistical Blunders in History, THE OXFORD MATH CTR.,
http://www.oxfordmathcenter.com/drupal7/node/251 (last visited Apr. 14, 2015); Case Study I: The 1936 Literary Digest
Poll, UNIV. PA. DEP’T OF MATHEMATICS, http://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html (last visited
Apr. 14, 2015).
139
1 GIANNELLI ET AL., supra note 116, at § 15.04[a], 846–47.
140
See DIAMOND, supra note 79, at 382.
141
Id. at 380 (“Probability sampling offers two important advantages over other types of sampling. First, the sample can
provide an unbiased estimate that summarizes the responses of all persons in the population from which the sample was
drawn; that is, the expected value of the sample estimate is the population value being estimated. Second, the researcher can
calculate a confidence interval that describes explicitly how reliable the sample estimate of the population is likely to be.”).
142
Id.

34. 34
of being selected in the sample.143
In stratified random sampling, the population is first divided into
mutually exclusive and exhaustive strata, and samples are selected from within these strata by basic
random sampling.144
Courts have also admitted into evidence survey results drawn from non-
probabilistic sampling.145
However, the proponent should be prepared to justify why she took a specific
non-probabilistic method to select the sample respondents in the instant case.146
With recent technological innovations, businesses and academics have increasingly used Internet
surveys for a variety of purposes. Internet survey can reduce substantially the cost of reaching potential
respondents and, at the same time, can improve the way survey is designed and presented to the
respondents. It also eliminates risks involving interviewer biases and reduces inaccuracies of data-
recording/saving as the computer program presents survey questions and collects answers automatically.
The threshold issue in evaluating an Internet survey is that the web-surfers may not fairly
represent the relevant population whose responses the survey was designed to measure.147
For example,
some Internet market research service companies maintain a huge panel of volunteers (the “panel
population”) consisting of multi-million consumers.148
Although a subset of the panel may be randomly
sampled from the panel population for conducting a specific survey, the panel population itself may not
be the product of a random selection process. The panel population likely over-represents a group of
relatively active and informative market participants.149
They may have a particular viewpoint on
subject matters and/or a motive that might bias the survey results.
Other issues concern respondent qualification and duplication. As the Internet survey is
conducted over the online, security measures must be taken to confirm that the selected sample
conforms to the purpose of the study. The procedures that Apple’s expert took in Apple v. Samsung
illustrate the concerns.150
The expert and a market research company hired by him first checked the
143
Id.
144
Id.
145
Id. at 382.
146
Id.
147
Id. at 406.
148
E.g., Hauser Report, supra note 54, at 25 (“Research now . . . maintains an invitation-only panel over 3.6 million
consumers in the United States and over 6 million panelists worldwide.”); Srinivasan Report, supra note 61, at 12 (“Optimal
Strategix Group utilized Authentic Response’s US Adults panel of approximately 3 million respondents.”); see also
DIAMOND, supra note 79, at 382 n.102.
149
One expert report notes that the panel population held by a market survey company does not accurately represent the
whole population. Srinivasan Report, supra note 61, at 12 (“This [3 million] panel is maintained to be reflective of the U.S.
Census (adults), but it is not exactly balanced with respect to the U.S. Census (adults).”). Therefore, the survey company
weighted the sample population to be a more balanced representative of the U.S. Census of adults than the un-weighted
sample. Id.
150
Hauser Report, supra note 54, at 26.

35. 35
identity of panelists by reviewing registered email addresses and their basic demographic information.151
Based on the information, they invited people who indicated that they own smartphones.152
The
invitation email included a link to actual survey, hosted on a website maintained by the market research
company affiliated with the expert.153
The email link contained an embedded identification number to
assure that only invited respondents could answer the survey and could do so only once.154
Before
starting the survey, respondents were prompted to a CAPTCHA challenge to ensure that responses were
not computer-generated.155
In addition, a high nonresponse rate can significantly bias the survey results and can be a ground
for excluding survey evidence.156, 157
Nonresponses aggravate systematic error more seriously when the
nonresponse is not random.158
The key in understanding the pattern and effect of the nonresponse in a
survey is to determine the extent to which nonrespondents differ from respondents with respect to the
dividing characteristics of these groups.159
The proponent must review the underlying raw data to investigate whether there exists
significant nonresponses and whether the nonresponses are systematic. However, mere low response
rate may not be damming.160
“Contrary to earlier assumptions, surprisingly comparable results have
151
Id.
152
Id. This initial criterion screens the pool of potential respondents by determining whether they belong to the target
population of the survey. As the target population in the survey was smartphone owners, online survey in itself does not pose
a serious underinclusive problem with respect to the availability of Internet access. “The screening questions must be drafted
so that they do not appeal to or deter specific groups within the target population, or convey information that will influence
the respondent’s answers on the main survey.” DIAMOND, supra note 79, at 385–86.
153
Hauser Report, supra note 54, at 26.
154
Id.
155
Id. at 27. CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. “A
CAPTCHA challenge refers to a program that protects websites against bots (i.e., computer-generated responses) by
generating and grading tests that humans can pass, but current computer programs cannot.” Id. n.32.
156
Methods of computing response/nonresponse rates vary. “[A]lthough response rate can be generally defined as the
number of complete interviews with reporting units divided by the number of eligible reporting units in the sample, decisions
on how to treat partial completions and how to estimate the eligibility of nonrespondents can produce differences in measures
of response rate.” DIAMOND, supra note 79, at 384 n.109.
157
See id. at 383–85 (suggesting 80% or higher response rates are desirable); 1 GIANNELLI ET AL., supra note 116, at §
15.04[b], 860–63.
158
DIAMOND, supra note 79, at 383 (“for example, persons who are single typically have three times the ‘not at home’ rate in
U.S. Census Bureau surveys as do family members”). See supra text accompanying note 138 (regarding Literary Digest’s
1936 presidential election poll).
159
See DIAMOND, supra note 79, at 383–84 (“The key is whether nonresponse is associated with systematic differences in
response that cannot be adequately modeled or assessed.”).
160
For example, Apple’s expert reported that for his conjoint survey, only 22.8% of the invitees responded. Hauser Report,
supra note 54, at 37 n.47 (“Out of the total 38,795 participants invited for the smartphone survey, 8,844 began the survey,
resulting in a response rate of 22.8%. . . . Out of the total 8,844 participants who started the smartphone survey, 604 passed
through the screening questions and qualified for the choice tasks, resulting in an incidence rate of 6.8%.”). The survey result
was not excluded.