1. THE IMPACT OF EXPERIMENTATION ON PRODUCT DEVELOPMENT IN
COMPANIES INVOLVED IN INTERACTIVE MARKETING
by
Sergey L. Sundukovskiy
H. PERRIN GARSOMBKE, Ph.D., Faculty Mentor and Chair
LARRY KLEIN, Ph.D., Committee Member
JOSE NIEVES, Ph.D., Committee Member
William A. Reed, Ph.D., Acting Dean, School of Business & Technology
A Dissertation Presented in Partial Fulfillment
Of the Requirements for the Degree
Doctor of Philosophy
Capella University
August, 2009

2. UMI Number: 3369490
Copyright 2009 by
Sundukovskiy, Sergey L.
All rights reserved
INFORMATION TO USERS
The quality of this reproduction is dependent upon the quality of the copy
submitted. Broken or indistinct print, colored or poor quality illustrations and
photographs, print bleed-through, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if unauthorized
copyright material had to be removed, a note will indicate the deletion.
______________________________________________________________
UMI Microform 3369490
Copyright 2009 by ProQuest LLC
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
_______________________________________________________________
ProQuest LLC
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106-1346

3. © Sergey L. Sundukovskiy, 2009

4. Abstract
Since its emergence, the Internet and Internet related technologies have permeated almost
all aspects of the modern life. The impact of the Internet on the day-to-day activities of its users
has been quite dramatic. However, its effect on the business community has been even more
profound. Besides yielding additional opportunities for existing businesses, the Internet has
facilitated a new era of companies that have the Internet at the center of their business model,
businesses that simply would not be able to exist without it. As such Internet-centric and
increasingly traditional businesses now rely on interactive marketing as a source of revenue,
differentiation and competitive advantage. Despite the obvious importance to many businesses,
the efficiency and improvement of interactive marketing has largely stagnated or at least
proceeds at a low pace. This study examined interactive marketing through the prism of
experimentation as a way of propelling interactive marketing forward and enabling it to keep
pace with technological advances.
Experimentation lies at the core of product development, improvement and innovation.
Active experimentation has been utilized as a product strategy in numerous business fields, but it
has been largely ignored in interactive marketing. This study examined a number of
experimentation models and their applicability to interactive marketing. In addition it focused on
the elements of interactive marketing that are conducive to experimentation.

5. Dedication
I dedicate this work to my wife Galina for her commitment, dedication and unconditional
love. Also, to my children: Aaron, Rebekah and Miriam for giving meaning to my life. Your love
and support gave me strength to see this project through.
To my parents: Valentina and Leonid, for always believing in me and forever
encouraging me to see physical, emotional and intellectual limits as imaginary lines that are
meant to be crossed.
Last, but by no means least, to my brother: Aleksey, for seeing me the way I wish I could
truly be.
iii

6. Acknowledgements
Special thanks to my friend Boris Droutman for introducing me to the subject of
experimentation and helping me generate the ideas that made this dissertation possible. Thanks
for encouraging me, listening to the things I say and hearing the things I do not say.
iv

7. Table of Contents
Dedication ...................................................................................................................................... iii
Acknowledgements ........................................................................................................................ iv
Table of Contents ............................................................................................................................ v
List of Figures ............................................................................................................................... vii
CHAPTER 1: INTRODUCTION ................................................................................................... 2
Introduction to the Problem......................................................................................................... 2
Internet Marketing ....................................................................................................................... 2
Interactive Marketing .................................................................................................................. 4
Product Development .................................................................................................................. 9
Background of the Study ........................................................................................................... 10
Purpose of the Study ................................................................................................................. 11
Research Questions ................................................................................................................... 12
Definition of Terms ................................................................................................................... 15
Assumptions and Limitations .................................................................................................... 17
Organization of the Remainder of the Study............................................................................. 19
CHAPTER 2: LITERATURE REVIEW ...................................................................................... 20
Experimentation Introduction ................................................................................................... 20
Experimentation Strategies ....................................................................................................... 21
Experimentation Models ........................................................................................................... 22
Interactive Marketing ................................................................................................................ 27
Experimentation ........................................................................................................................ 29
Product Innovation .................................................................................................................... 31
CHAPTER 3: METHODOLOGY ................................................................................................ 33
Description of the Methodology ............................................................................................... 33
Design of the Study ................................................................................................................... 36
Population and Sampling .......................................................................................................... 37
Measurement Strategy ............................................................................................................... 38
Instrumentation.......................................................................................................................... 39
Data Collection .......................................................................................................................... 40
Data Analysis Procedures.......................................................................................................... 41
v

8. Qualitative Data Display for Describing the Phenomenon ....................................................... 44
Qualitative Data Display for Explaining the Phenomenon ....................................................... 45
CHAPTER 4: DATA COLLECTION AND ANALYSIS............................................................ 46
Overall Response Analysis ....................................................................................................... 46
Research Objective One ............................................................................................................ 48
Research Objective Two ........................................................................................................... 53
Research Objective Three ......................................................................................................... 58
Research Objective Four ........................................................................................................... 61
Research Objective Five............................................................................................................ 63
Research Objective Six ............................................................................................................. 66
Validity and Bias ....................................................................................................................... 69
CHAPTER 5: RESULTS, CONCLUSIONS, AND RECOMMENDATIONS ........................... 71
Results ....................................................................................................................................... 71
Conclusions ............................................................................................................................... 74
Recommendations ..................................................................................................................... 75
Limitations of the Study ............................................................................................................ 76
REFERENCES ............................................................................................................................. 78
APPENDIX A. SURVEY TEMPLATE ....................................................................................... 82
APPENDIX B. CONCEPTUAL FRAMEWORK........................................................................ 88
APPENDIX C. INTERACTIVE MARKETING .......................................................................... 89
APPENDIX D. MARKETING ..................................................................................................... 90
APPENDIX E. EXPERIMENTATION CYCLE ......................................................................... 91
APPENDIX F. DATA ANALYSIS .............................................................................................. 92
vi

9. List of Figures
Figure F1. Response summary...................................................................................................... 92
Figure F2. Job function breakdown .............................................................................................. 92
Figure F3. Job function and experimentation experience cross-tabulation .................................. 93
Figure F4. Experimentation impact breakdown............................................................................ 93
Figure F5. Experimentation impact and department cross-tabulation .......................................... 94
Figure F6. Experimentation impact and experimentation experience cross-tabulation ................ 94
Figure F7. Experimentation levers impact breakdown ................................................................. 95
Figure F8. Experimentation levers impact and experimentation experience cross-tabulation ..... 95
Figure F9. Experimentation levers impact and job function cross-tabulation .............................. 96
Figure F10. Experimentation conditions impact breakdown ........................................................ 96
Figure F11. Experimentation conditions impact and experimentation experience cross-tabulation
....................................................................................................................................................... 97
Figure F12. Experimentation conditions impact and job function cross-tabulation ..................... 97
Figure F13. Experimentation CPA impact and experimentation overall impact cross-tabulation 98
Figure F14. Experimentation CPI impact and experimentation overall impact cross-tabulation . 98
Figure F15. Experimentation CPA impact and experimentation overall impact cross-tabulation 99
Figure F16. Experimentation upsell impact and experimentation overall impact cross-tabulation
....................................................................................................................................................... 99
Figure F17. Experimentation CPI impact and experimentation overall impact cross-tabulation100
Figure F18. Experimentation product lifecycle impact breakdown............................................ 100
Figure F19. Experimentation impact on product lifecycle and job function cross-tabulation.... 101
Figure F20. Experimentation impact on product lifecycle and experimentation experience cross-
tabulation..................................................................................................................................... 101
Figure F21. Experimentation product innovation impact breakdown ........................................ 102
Figure F22. Experimentation product innovation impact and experimentation experience cross-
tabulation..................................................................................................................................... 102
Figure F23. Experimentation product innovation impact and job function cross-tabulation ..... 103
Figure F24. Experimentation product improvement impact breakdown .................................... 103
Figure F25. Experimentation product improvement impact and job function cross-tabulation . 104
Figure F26. Experimentation product improvement impact and experimentation experience
cross-tabulation ........................................................................................................................... 104
Figure F27. Experimentation product deployment risk impact breakdown ............................... 105
vii

10. Figure F28. Experimentation product deployment risk impact and experimentation experience
cross-tabulation ........................................................................................................................... 105
Figure F29. Experimentation product deployment risk and job function breakdown ................ 106
Figure F30. Experimentation competitive standing of the company impact breakdown ........... 106
Figure F31. Experimentation competitive standing of the company impact and job function
cross-tabulation ........................................................................................................................... 107
Figure F32. Experimentation competitive standing of the company impact and experimentation
experience cross-tabulation ......................................................................................................... 107
Figure F33. Experimentation revenue impact ............................................................................. 108
Figure F34. Experimentation profit impact ................................................................................ 108
Figure F35. Experimentation market size impact ....................................................................... 108
Figure F36. Experimentation market penetration impact ........................................................... 109
Figure F37. Experimentation marketing goals impact horizontalization.................................... 110
Figure F38. Experimentation marketing levers impact horizontalization .................................. 111
Figure F39. Experimentation conditions impact horizontalization............................................. 112
Figure F40. Experimentation product lifecycle impact horizontalization .................................. 113
Figure F41. Experimentation product innovation impact horizontalization ............................... 114
Figure F42. Experimentation product improvement impact horizontalization........................... 115
Figure F43. Experimentation product deployment risk impact horizontalization ...................... 116
Figure F44. Experimentation competitive standing impact horizontalization ............................ 117
viii

11. ix

12. CHAPTER 1: INTRODUCTION
Introduction to the Problem
Internet Marketing
The introduction of the World Wide Web, enabled by the Internet, saw an explosive user
growth (Newman, 2001). Consequently both online based and traditional companies with
significant Internet presence started to realize the full potential of World Wide Web as a
marketing medium. Since the popularization of the Internet and Internet related technologies,
several new marketing paradigms have emerged that take advantage of Internet. Such emergence
was not planned and occurred spontaneously as an offshoot of traditional marketing (Mark,
2003). Internet marketing was formulated as one form of non-traditional marketing. Due to early
disappointments and lack of experience with a new non-traditional marketing medium Internet
marketing was initially used to supplement traditional marketing efforts. According to eMarketer
Research (2007), US online marketing in 2002 shrunk by 15.8%. Companies did not see Internet
marketing as a separate marketing channel, with a unique set of characteristics and considered it
the same as radio and television (Raman, 1996). Internet marketing was combined with
traditional television and radio campaigns because at first, it was not considered capable of
carrying the full weight of responsibilities.
However, over time marketers began to realize that traditional marketing campaigns
consisting of television and radio advertisements, and print media designed to attract users to
their websites had a limited success, whereas the Internet itself proved quite promising in that
regard. According to eMarketer Research (2007) online advertizing spending in 2007 had
2

13. reached $19.5 billion, projecting to reach $36.5 billion in 2011. Companies had realized that
Internet marketing was not only a strong and independent marketing medium, but also had a
number of advantages over traditional forms of marketing, such as television, radio and print
media (Mark, 2003). In order to earn its independence and be considered comparable to one of
the traditional dominant marketing channels, Internet marketing had to be evaluated against a
multitude of traditional marketing problems.
It had successfully answered a number of questions that were historically asked by
marketing science. Internet marketing showed its potential for consumer accessibility, defining
corporate identity, promoting brand awareness, enabling market segmentation and achieving
market localization and internationalization (Jebakumari, 2002). In addition, Internet marketing
capabilities such as personalization, interactivity and traceability were formulated among its
advantages over traditional dominant medium channels such as television, radio and print media.
The Internet allowed synchronous and asynchronous user behavior analysis, which in turn
allowed for previously unavailable personalization capabilities (Milley, 2000). It had the
potential to reach a vast number of World Wide Web users that otherwise would be either
difficult to reach, or in some cases completely unreachable. According to Internet World Stats
(2007), by the end of 2007 the Internet had reached 1,319,872,109 users with the largest
percentage growth accounted for by countries in Middle East, Africa and Latin America.
Since its inception, the Internet has developed into one of the strongest and mature
Marketing channels that is segmented into several sub-channels. At the present time Internet
marketing could be considered as a conglomerate marketing methodology, providing a common
umbrella for a set of diverse marketing sub-channels. The following distinct Internet sub-
marketing channels have now emerged: (a) search engine marketing (SEM); (b) e-mail
3

14. marketing; (c) affiliate marketing; (d) display marketing; and (e) social media marketing (See
Appendix C). These Internet marketing approaches, founded upon sociological, economic and
cultural marketing aspects, are now interwoven into Internet marketing fabric as a cohesive
entity. At the same time each one of these Internet marketing sub-channels has a set of unique
characteristics, based on its execution and marketing strategies. These characteristics distinguish
it from other tangential marketing approaches (Mark, 2003).
Interactive Marketing
In addition to being part of Internet marketing, social media marketing, affiliate
marketing, search engine marketing and e-mail marketing sub-channels have another aspect in
common, their interactivity. Each one of these sub-channels relies on the user to make an
interactive step directly related to the advertisement stimuli. As such these marketing sub-
channels could be placed under another umbrella called “Interactive Marketing”. In its purest
form Deighton (1996) defines “Interactive Marketing” as an ability of the computer based
system to interact with the user for marketing based purposes. In many cases however interactive
marketing consists of a multi-transactional interaction that is based on user purchase and visit
history, declared or implied preferences and demographic information consisting of age, sex,
income level and even race.
Most of the interactive marketing accomplished through the World Wide Web is enabled
by the Internet. However, it would be incorrect to consider Internet marketing as synonymous
with interactive marketing (Deighton, 1996). Based on the Veen diagram (see Appendix D) not
all Internet marketing is interactive and not all interactive marketing is conducted on the World
Wide Web or is Internet based. For instance display marketing, one of the Internet marketing
sub-channels, is not interactive in nature. In the same vain iPods, mobile devices, gaming
4

15. devices, interactive screen devices and e-books are capable of carrying a marketing message by
means of wireless or hard-connected communication. Each of these devices has its own
marketing specifics related to its size and interactivity, tracing power, display and
communication capabilities.
However, based on the Veen diagram (see Appendix D) it is evident that a great amount
of overlap exists between the Internet and interactive marketing. Chulho (1998) defined the
“Interactive Age” as an age of full interactivity between humans and machines and technology
connected humans. Similar parallels can be drawn between the “Information Age” and
“Participation Age”, coined by Schwartz (2005). Blogging and Social Networking are two recent
examples of the Interaction or Participation Age that are utilized by social media marketing.
At the heart of interactive marketing are the two main concepts of “Personalization” and
“Customer Engagement”. The interactive marketing itself could be fully defined in terms of
these concepts. More specifically, this researcher proposes an alternative definition of
“Interactive Marketing” describing it as a process of customer engagement through
personalization. In turn, Steinberg (2006) defines “Customer Engagement” as a process of
engaging potential or existing customers in inter-client or inter-customer communication by
means of the push or pull marketing model. According to Haag (2006) “Personalization” is a
process of generating unique experience for an ever finer grain segment of customers. Taken to
the logical extreme, the process of personalization is a process of targeting a single individual
with the stimuli uniquely tailored to that individual and that individual alone. The logical
conclusion of these definitions is the hypothesis that the most effective customer engagement
could be achieved through granular personalization activities.
The companies involved in interactive Internet marketing endeavor to achieve
5

16. personalization through data mining and information analysis. The data mining and analysis
constitutes the bulk of the personalization effort in both temporal and relative importance
perspectives. Ozdemiz (2002) describes personalization as an effort consisting of three major
steps: (a) data gathering; (b) data analysis; (c) personalized content generation. According to
comSource (2007) the data gathering portion of the personalization process undertaken by major
interactive marketing companies such as Yahoo, Google and Facebook has reached a significant,
earlier unseen level. According to comSource (2007), in a period of a single month in 2007
Yahoo on average collected data 2,520 times for every visitor to its site. Mark (2003) posited that
Internet marketing companies collect as much data a possible even without a specific goal in
mind. Since, it is not always clear what customer data would become useful in the future, all
available data is collected (Mark, 2003). It usually consists of webpage visits, visit history,
purchasing history, abandonment and completion history, self-declared or inferred preferences,
computer information, web browser information, self-declared or inferred demographic
information self-declared or inferred financial information, geographic information, age and sex
information (Mark, 2003). The collected data is carefully cataloged and analyzed. The purpose of
this data analysis is to determine the most appropriate marketing strategy for a particular user
and other users that fit similar characteristics. Milley (2000) describes this approach as a “Data
Empowered” marketing strategy.
The intention of any marketing strategy in general and interactive Internet marketing in
particular is to achieve a desired goal. In general interactive Internet marketing tries to achieve a
goal of conversion where the user is synchronously driven to a purchasing decision. According
to Macias (2000) even though conversion is certainly the dominant interactive marketing goal,
other goals may include customer retention, customer acquisition, lifetime value maximization
6

17. and product up-sell. Each one of these goals could be considered a question that interactive
Internet marketing is supposed to answer. Namely, interactive Internet marketing is required to
provide answers to the following questions: (a) what is the most effective customer retention
strategy? (b) what is the most effective to driver customer to a purchasing decision? (c) what is
the most effective customer acquisition strategy? and (e) what is the most effective product up-
sell strategy?
By analyzing collected user data interactive marketing researchers try to answer these
questions in a predictive manner. The predictive data analysis, which is at the core of the data
empowered marketing strategy, is accomplished through diverse data modeling techniques. Dou
(1999) described the use of the Catastrophe Theory to model online store sales. Other techniques
include the Recency-Frequency-Monetary Value (RFM) model designed to assess the customer
purchasing decision. However, these and other models have several fundamental problems. First
of all, these models are predictive and as such carry a fair amount of statistical error due to a
need to make assumptions and use guess work. Consequently, these models are subject to
Simpson’s Paradox where a lack of understanding of data granularity could lead to incorrect
conclusions. Secondly, the Internet is an open system where not all of the variables and their
effect are clearly defined (Dou, 1999). As such confounding variables become a real issue.
Thirdly, the feedback loop between data collection, data analysis and interactive Marketing
changes is quite long and inefficient (Dou, 1999).
In addition to the limitations mentioned above, predictive modeling strategies are
fundamentally unsuitable to the set of marketing tasks related to product development and risk
mitigation. The data empowered marketing strategy absolutely requires extensive amounts of
data in order to come up with a predictive marketing model. However, at the stage of new
7

18. product development, such data does not exist or is very limited. As such, predictive modeling
marketing strategies by definition are not suitable for new product development. In the past
marketers had to rely on focus groups to gain some guidance on new and existing product
development (Mark, 2003). However, the focus group method has proven to be extremely error
prone due to the Survey Paradox where people say one thing and then another. According to
Mark (2003), in many cases companies would also fail to generate a focus group of the required
diversity. As such it was not uncommon to see focus group results being skewed to a particular
group of users.
In addition, the focus group approach to product development resulted in a monolithic
product development driven by corporate executives. In this product development model,
product ideas were generated by company managers and corporate executives. Kohavi,
Longbotham, Henne and Sommerfield (2007) described this model as the “Highest Paid Person’s
Opinion” (HiPPO), where the most senior member of the team comes up with the product idea or
is responsible for choosing among multiple product options. When the product idea is selected,
the resultant product is manufactured in the monolithic fashion and subsequently demonstrated to
the focus group. One possible name for the risk profile in this type of product development is
“Risk Backloading”, where the bulk of the risk is born at the end of the product development
lifecycle. Consequently, HiPPO initiated and focus group validated product development is
extremely deficient in both risk mitigation and consumer-product fit.
The main goal of this study is to propose an alternative interactive Internet marketing
product development strategy that mitigates the shortcomings of predictive data modeling and
HiPPO-Focus Group product development. It addresses data modeling shortcomings related to
statistical uncertainly, data feedback loop and confounding variables. In addition it recommends
8

19. a more advantageous interactive Internet marketing product development approach that places
product initiative in the hands of the end-users and allows for a more advantageous risk profile
related to product innovation and product development. Even though this study treats Internet
and interactive marketing synonymously, it more specifically considers only the part of
interactive marketing that is enabled by Word Wide Web and Internet technologies. The findings
of the study should however be applicable to the interactive devices mentioned above.
Product Development
As mentioned earlier, the use of experimentation to investigate the problem has been
described by several researchers. It is possible to consider product development as a particular
type of problem and so the process of experimentation should be fully applicable to product
development. According to Thomke (1997), experimentation as a method of product
development has been utilized by a number of industries including Automotive, Pharmaceutical
and Electronics. Furthermore von Hippel (1998) posited that experimentation facilitates product
development by end-users of the product. User-driven product development allows companies to
get away from the HiPPO principal, where product decisions are made by the people who are
furthest removed from the product itself. Product users themselves consciously or
subconsciously galvanize and propel product development forward. More importantly they do so
with their actions instead of the words and in this way they eliminate the Survey Paradox.
Thomke, von Hippel, and Franke (1997) argued that employing experimentation in product
development allows companies to improve their competitive standing. Since products are shaped
by the end-users directly, the chances of product failure are greatly decreased. The
experimentation process allows controlled failures early and often when the cost of change is
minimal. By employing the experimentation approach companies reverse the risk profile from
9

20. back-loading to frontloading. Depending on the nature of the product companies may utilize
several experimentation modes: (a) simulation; (b) prototype; and (c) sample. Experimentation
modes depend on such factors as experiment result applicability, experiment cost and experiment
efficiency. According to Thomke (1997), experimentation modes constitute a continuous process
optimized based on the efficiency vs. number of experimental cycles.
Based on the applicability of experimentation to product development in other areas it
would seem logical that the process of experimentation would be pertinent to Internet marketing
and interactive marketing as well. However, very little if any empirical studies have been
conducted to explore this matter. Researchers such as Li (2003), Raman (1996), Mark (2003),
among others, have broached the subject from the predictive modeling point of view. Even if
experimentation is mentioned it lacks continuity with interactive marketing itself and is always
driven by reactive analysis.
Background of the Study
This study examines the applicability of experimentation to product development in the
confines of Internet and interactive marketing. Since, empirical evidence of experimentation in
the field of interactive marketing is sorely lacking it draws on experimentation experience from
other industries and product categories. Its background is rooted in the fields of experimentation,
product development, production innovation, Internet development, Internet marketing and
interactive marketing. The narrative of the study is centered on the product development
strategies in interactive Internet marketing. It is designed to address what can be called the
Information Age Product Development Paradox, where companies that have a significant
Internet eCommerce presence tend to ignore technology-centric product development strategies
10

21. and instead favor low-tech solutions. The paradox consists of a seeming contradiction between
relying on the technologies as the source of survival, yet disregarding them when it comes to
product development. With regards to product development, companies involved in the Internet
and the interactive marketing tend to ignore product development strategies that have been
developed in other fields. As such they face problems related to product risk, product
improvement and product innovation.
Purpose of the Study
The main purpose of the study is to examine the applicability of experimentation as the
product development strategy in the companies involved in interactive marketing. It examines
the use of experimentation in all parts of the interactive marketing product development
lifecycle: (a) idea generation; (b) idea screening; (c) concept development and testing; (d)
business analysis; (e) beta testing and market testing; (f) technical implementation; and (g)
commercialization. In addition this study examines the impact of experimentation on product
evolution as a function of user interactivity. It aims to contribute to the experimentation body of
knowledge as well as establishing an empirical base for interactive marketing experimentation
practice that is quite limited at the present time. Furthermore, this study looks at the unique
elements of interactive marketing experimentation. It attempts to define the majority of the
exogenous and endogenous interactive marketing factors and their effect on interactive
marketing goals. The secondary purpose of the study is to point out the gaps and propose
improvements in the Design of Experiments Theory, as it pertains to interactive marketing.
11

22. Research Questions
This study aims to answer six main research questions. What is the impact of
experimentation on interactive marketing goals? What are the key experimentation levers
pertaining to interactive marketing? What is the impact of experimentation on product innovation
in interactive marketing? What is the impact of experimentation on interactive marketing product
improvement? What is the impact of experimentation on interactive marketing product
development and deployment risk? What is the impact of experimentation on the competitive
standing of the companies involved in interactive marketing?
Online conversion is one of the key goals of interactive marketing companies. Even
though conversion is not the only goal, the majority of interactive marketing companies are
striving to achieve higher conversion. Even though conversion events of interactive marketing
companies are quite diverse, in essence, the conversion event is equivalent to a revenue
generating event. It is paramount to determine if the use of experimentation has a positive effect
on online conversion. This study examines a number of exogenous and endogenous variables
pertaining to interactive marketing and analyses their impact on the interactive marketing goals.
The aim of the study is not to define an exhaustive list of exogenous and endogenous factors
across all interactive marketing products and environments, but rather to determine an effective
subset usable across interactive marketing companies of different types.
A significant number of the research questions posed by this study are devoted to the
analysis of the impact of experimentation on product development lifecycle. More specifically
the study aims to examine the impact of experimentation on product innovation, product
improvement, product development and product deployment risk. Similar to the goals, the
interactive marketing companies’ products are equally as complex and as diverse. However, it is
12

23. important to understand the fundamental benefits offered by experimentation as it relates to
interactive marketing product development regardless of the product type.
Lastly, this study will examine the impact of experimentation on the competitive standing
of companies in the interactive marketing space. The aim of the study is to show that companies
in the interactive marketing space that abstain from experimentation, in relation to product
development, ignore it at their own peril and put their competitive standing and quite possibly
long term survival at risk.
Nature of the Study
Experimentation in the interactive marketing field is a social phenomenon since it is
designed to achieve the best results through social interaction. Despite the fact that the social
interaction in this context occurs between man and machine, the social context of the interaction
is preserved. As such the use of a qualitative design approach is consistent with social
constructivism and fully justified as the research approach for this dissertation. It is would have
been absolutely acceptable to employ a quantitative or mixed-method design approach since
experimentation relies heavily on statistical analysis and statistical data modeling. However, the
topic of this dissertation and the state of the empirical research in the experimentation field of
interactive marketing are more conducive to qualitative analysis. This study is theory forming in
nature and as such it is consistent with one of the strong fundamentals of qualitative analysis.
Significance of the Study
The experimentation approach can give companies involved in interactive marketing the
means to test marketing ideas without putting the majority of its traffic at risk. In the past
13

24. marketing ideas have been tested using surveys and focus groups. However, this approach is long
and inexact. It also gives rise to what can be called a “Survey Paradox”, where people say one
thing and do another (Sheffrin, 1996). The process of experimentation presents qualitative and
quantitative results and represents an innovative development in testing Internet marketing and
interactive marketing ideas. However, very little or virtually none of the empirical research
confirms positive effects of experimentation by online conversion when used as part of
interactive marketing strategies. Furthermore, due to a lack of solid theoretical research
companies involved in experimentation in the field of interactive and Internet marketing are
forced to develop a practical base behind experimentation through trial and error. Such approach
results in limited success and might lead to companies abandoning experimentation altogether. In
addition companies that do successfully pursue experimentation, in the context of interactive
marketing, quite limit their success due to poor alignment between the Business and Information
Systems designed to conduct experimentation. As such experimentation is done haphazardly,
lacks continuity and does not result in a cohesive strategy.
The significance of this study is in establishing a theoretical base behind experimentation
in the interactive and Internet marketing fields. One of the key implications of the study is the
assertion that companies that are involved in interactive and Internet marketing, outside of the
experimentation framework, negatively impact their competitive standing, increase product
development risk and reduce their likelihood of success in achieving their marketing goals.
Another essential contribution of the study is an outline of the implementation process in the
context of the experimentation framework that allows achieving cohesion between departmental
units involved in the interactive marketing experimentation. This study positions
experimentation as a product development and product improvement method designed to
14

25. optimize the marketing goals of the companies involved in interactive and Internet marketing.
Definition of Terms
Base Flow: current web site execution path with the highest fitness value. All related
flows performance is compared to the Base Flow.
Challenger Flow: variation of the Base Flow designed to achieve a higher fitness value.
Confirmation Flow: previous Base Flow designed to confirm its losing status to the
current Base Flow. The introduction of the Confirmation Flow can be classified as confirmation
testing, where results of the previous Experiment are reconfirmed. The purpose of the
confirmation testing is to make sure that the previous Base Flow had lost to the current Base
Flow in the head to head comparison and was not a result of a statistical error or impact of
unaccounted or unknown confounding variable.
Experiment: manipulation on one or more of the endogenous variables in the context of
the exogenous variables with the purpose of achieving Silo Goals. The purpose of the experiment
is to reject or accept the null hypothesis.
Experimentation Levers: collection of exogenous variables that is intrinsic to the
Experiment.
Flow: single variation of the web site execution path. Typically the experiment consists
of several Flows that manipulate a single variable for a side by side comparison.
Multivariate Experiment: experiment consisting of manipulation of multiple endogenous
variables simultaneously.
Multiple Singlevariate Experiments: Multiple experiment sets affecting a single
endogenous variable per set.
15

26. Phantom Flow: flow that does not cause a variation in behavior. It represents a virtual
duplication of any physical flow for achieving statistical significance.
Silo: area under experimentation. The Silo usually consists of a collection of endogenous
variables that can be manipulated in a particular context. In typical interactive marketing Silos
consist of landing web pages, up sell web pages, offer pages, etc.
Silo Goal: purpose of the experimentation. In the majority of interactive marketing cases
the Silo Goal is represented by conversion also known as a purchasing decision. Other Silo Goals
may include link offs, up sell actions, subscriptions, content contribution, etc. In general the
terms the goal of the experimentation is to achieve the highest fitness value.
Singlevariate Experiment: experiment consisting of manipulation of a single endogenous
variable.
View: single web page alongside the web site execution path. Typically Flow consists of
several Views that contain one or more endogenous variables that are being manipulated.
Imposed Flow: flow that allows predictive execution of the web site path without
distribution and analysis. Imposed Flow is often utilized for regulatory or testing purposes.
Silo Visit: represents a visit to one of the areas under experimentation, typically
represented by a collection of web pages.
Site Visit: represents a visit to the web site or any other Traffic Origin of the interactive
marketing company.
Traffic Origin: source where web traffic originates. Traffic Origin is often represented by
one of the interactive marketing channels such as email, search, display, affiliates, call center and
social media. Traffic Origin is one of the key exogenous variables.
Traffic Criteria: criteria that is intrinsic to the incoming web traffic, but extrinsic to the
16

27. Experiment itself. Traffic criteria are a collection of exogenous variables.
Traffic Distribution: allocation of the web traffic between Experiments and Flows
according to the distribution algorithm. Traffic Distribution between Experiments and Flows
must add up to 100% otherwise some of the incoming traffic will fail to be distributed.
Vertical: product centric collection of interactive marketing assets.
View Visit: represents a display web page or any other asset of interactive marketing such
as email, banner, etc.
Visitor: represents a unique visitor to the web site.
Assumptions and Limitations
In addition to paradigmatic and methodological assumptions this study has a number of
domain assumptions. First, it assumes the synonymous nature of following terms: A/B Testing,
Split Testing, A/B/C Testing, Singlevariate Testing, Singlevariate Multi-Experiment Testing, and
Multivariate Testing. All of these terms are aimed at describing the same phenomenon and had
evolved under a common umbrella (see Appendix C). However, it needs to be noted that the term
multivariate testing refers to the experimentation technique related to univariate testing
strategies, yet it is not synonymous with them. However, for all intent and purposes the practice
of multivariate testing as well all univariate testing techniques will be assumed to be describing a
common phenomenon. In cases where the study discusses the intricacies of a particular
experimentation approach the name of the approach will be explicitly noted. In the same vain
such terms as Internet marketing and interactive marketing are treated synonymously and
interchangeably. In cases where a distinction between these concepts needs to be made it is made
explicitly.
17

28. This study also assumes that all participants of the study are not aware of their active
participation in the study. This assumption is essential for illuminating participant bias. It is also
assumed that all participants of the study have at least cursory knowledge of interactive and
internet marketing as well as a practical understanding of experimentation as it applies to the
above mentioned concepts.
Besides listed domain assumptions this study has a number of paradigmatic and
methodological assumptions. In some respects it follows the basic assumptions of the systems
approach. It assumes objective reality that is independent of the observer. Similar to the systems
approach the key aim of experimentation is to model the real world phenomenon. As such it
employs inductive and deductive reasoning as well as verification as the basis for its conclusions.
It assumes that the results of any given experiment may be a result of casual relationships and
must be verified by further experimentation.
The methodological assumptions of the study are related to phenomenological inquiry.
According to Becker (1992), the researcher of the phenomenological study is viewed as a co-
creator of knowledge alongside the participants of the study. Even though such researcher
involvement is beneficial, in this particular case discrepancy between the participant’s and the
researcher’s knowledge of the studied phenomenon is too great. As such, the level of threat to
researcher bias, validity, generalization and repeatability of the study is too great. Consequently,
the researcher employed a combination of transcendental and experimental phenomenological
methodology that allowed the researcher to detach his experience from the experience of the
participants of the study.
18

29. Organization of the Remainder of the Study
The remainder of the study consists of four additional chapters. Chapter Two examines
literature related to interactive and Internet marketing, product development and
experimentation. The literature has been selected to describe each of the aspects of the study
independently yet leading to cohesion between experimentation, interactive marketing and
product development concepts. Chapter Three describes the design methodology that was chosen
to conduct the study. It defends the choice of research methodology by examining the
methodological fit to the research approach and the subject of the study. It also describes
elements of the study including the data collection procedures, data sampling, data collection
instruments and data coding. Chapter Four illustrates the findings of the study and undertakes a
detailed analysis of the results of the study. It consists of data analysis and data display for
describing and explaining the phenomenon of the study. Chapter Five summarizes the findings of
the study and introduces an alternative hypothesis for the findings of the study, validity and bias
as well as examines the trend in the findings of the study. It also proposes topics for future
research and the areas of inquiry in the subject area.
19

30. CHAPTER 2: LITERATURE REVIEW
Experimentation Introduction
The process of experimentation in general is at the center of scientific discovery. In fact
science or knowledge derived from experimentation is called “experimental science”. People
have utilized experimentation since ancient times with one of the classic examples of early
experimentation originating from Egypt around 2613-2589 BC. When Egyptians attempted to
build a smooth-sided pyramid they engaged in the process of experimentation. They initially
started building a pyramid at Meidum which collapsed due to angle acuteness. Based on the
result of this failed experiment Egyptians altered the angle of the Bent-Pyramid at Dahshur more
than half way through to save it from collapse, resulting in a bent shape. Subsequent smooth-
sided Egyptian pyramids have utilized the correct angle from the outset. By looking at the
experiment conducted by the Egyptians it is possible to extract several observations: (a)
experiments often fail (sometimes they are designed to fail); (b) full scale experiments can be
quite expensive; (c) conducting experiments consecutively may take a long time; and (d)
conducting experiments concurrently allows effective side by side comparison. Unbeknown to
them, this Egyptian experiment created the basis for the Design of Experiments theory. This
early experiment and subsequent application of the experiment results contained the majority of
elements founds in modern product experiments.
Thomke (1997) described experimentation as one of the forms of problem solving. In
short Thomke (1997) defined experimentation as a trial and error process. According to Thomke
(1998), the experimentation process consists of four major steps: (1) design (design consists of
20

31. coming up with an improved solution based on the previous experience from the preceding
experiment cycles); (2) build (the build step consists of modeling and constructing products to be
experimented upon); (3) run (the run step of the experimentation process consists of executing
the experiment in the real or simulated environment); and (4) analyze (analysis consists of data
mining and data investigation collected during experiment execution). Thomke (1998), pointed
out that the experimentation process changes under pressure from exogenous elements. These
exogenous elements consist of uncertain requirements, environmental and technological changes.
Experimentation Strategies
The process of conducting experimentation is not uniform. The choice of the experimentation
strategy depends on the problem it is trying to resolve. In order to understand experimentation
strategies it is important to look at what biologists define as the “fitness landscape”
(Beerenwinkel, Pachter & Sturmfels, 2007). Quite often the solution to a particular problem is
not singular. Fitness landscape consists of all possible solutions to a particular problem. A fitness
function defines the quality of the solution in relation to other solutions to the same problem. The
optimal solution to the problem is thought to have the highest fitness value.
A real world demonstration of the fitness landscape could be observed by looking at the
simple problem. For example, the way between home and the office consists of n routes. The set
of all routes constitutes a fitness landscape. Among all of the possible routes one of the routes is
the “best”, where best = f(n). Since, best is a relative term it must be defined in the context of the
fitness landscape. It is likely that the majority of people would consider the best route to
constitute shortest time. However, it is possible that some would choose a scenic route to be the
best. It is also possible to have multiple optimal solutions where the best route changes based on
exogenous elements such as time of day, day of the week and weather conditions. In that case,
21

32. the best = df(n) / dt, taking the time of the day into consideration.
Thomke (1998) defines three experimentation strategies: (a) parallel experimentation; (b)
serial experimentation with minimal learning; and (c) serial experimentation with learning. It is
possible to demonstrate these experimentation strategies by continuing with the “driving from
home to office” problem. To find out the route that yields the shortest driving time between the
home and the office and is the quickest way would require employing a parallel experimentation
strategy. However, it is impossible to accomplish this using a single driver, since it requires
taking all possible routes simultaneously. The driver of the car would need help from his friends.
If the driver of the car was determined to find the shortest route himself/herself, he/she could
employ serial experimentation with minimal learning or experimentation with learning. The
serial experimentation with minimal learning would require the driver to follow a predefined
plan of taking different routes until all possible routes where exhausted. After all routes have
been tried, the driver would have analyze the time each has taken to determine the one with the
shortest time. The serial experimentation with learning would allow the driver to avoid trying all
routes by analyzing results of initial experiments and illuminating routes that certainly would not
yield good results.
Experimentation Models
The experiments are further complicated by endogenous and the exogenous elements. It
is certainly possible to affect the experimental results by changing the endogenous elements of
the system (see Appendix E). For instance besides the different routes, changing drivers, car
type, fuel type, and the number of drivers among many other things would possible affect the
driving time. The experiments with multiple changing endogenous elements are called
multivariate. The experiments with a single endogenous variable are called univariate. It is
22

33. always possible to represent multivariate experiments as a series of univariate ones, by
temporarily freezing all but one variable. Ideally, in order to understand the impact of all
exogenous variables each of them requires a separate experiment. In other words in order to
understand the impact of the time of departure from the office on the overall travel time, the
driver needs to conduct experiments by leaving the office every hour on the hour. In the same
vain all weather conditions must be tested as well.
Based on the analysis of the experimental models it is quite obvious that the complexity
of experimentation grows exponentially based on the number of endogenous and exogenous
elements, such that Number of Experiments (E) = k * (m^n)!, where k is the number of the steps
in the process, m is the number endogenous variables and n is the number of exogenous
variables. Based on formula above it is possible to arrive at the conclusion that the chance of
guessing the option with the highest fitness value, when the number of endogenous and the
exogenous elements are beyond trivial, is infinitesimally small.
At the same time running experimentation of all combinations of endogenous and the
exogenous factors in order to determine the best possible combination yielding the highest
fitness value would constitute a full factorial experiment (Xu &Wu, 2001). However, even with a
trivial number of exogenous and endogenous factors the number of resulting experiments that
satisfies full factorial design is truly staggering. By applying the formula listed above a single
step process with 3 endogenous factors and 3 exogenous factors would result in 1.0888 E 28
number of experiments. For all intent and purposes running full factorial experiments beyond a
trivial number of factors is simply not practical.
In order to resolve the conundrum of no experimentation and full factorial
experimentation it is necessary to look at the nature of exogenous and endogenous factors. Not
all of the variables involved in the experiment affect the outcome of the experiment equally.
23

34. Depending on the nature of the experiment it is possible to find a smaller subset of variables that
have the greatest effect on the experiment. Anderson (1972) called this approach to
experimentation partial factorial. There are number of statistical methods of determining what
variables truly matter to the outcome of the experimentation. Li (2003) identified the following
partial factorial reduction models: (a) Univariate Poisson (relies on the analysis of the variable of
all of the involved variables); (b) Univariate Tobit without Log Transformation; (c) Univariate
Tobit with Log Transformation; (d) Discretized Univariate Tobit with Log Transformation; (e)
Discretized Univariate Tobit with Heteregeity; (f) Multivariate Count; (g) Multivariate Count
with Mixture; and (f) Multivariate two-state hidden Markov Chain Tobit.
Even a cursory look at the listed models allows them to be separated into two categories:
(a) univariate; and (b) multivariate. Fundamentally univariate models, where ANOVA tests are
applied in succession, are designed to ascertain the effect of the independent variables on the
dependant variables (Biskin, 1980). In the context of the experimentation univariate models are
designed to highlight the exogenous and endogenous factors that have a significant impact of the
outcome. On the other hand multivariate models, where a MANOVA test is conducted, are
designed to come up with sets of independent variables that have an impact on the dependant
variables (Huberty, 1986). Again by taking the experimentation context into account the
multivariate models are designed to highlight sets of exogenous and endogenous factors that
have a significant impact on the outcome of the experimentation. According to Huberty and
Moris (1989) the fundamental difference between multiple univariate ANOVA tests and
multivariate MANOVA tests consist in the consideration of the effects of the independent
variables on each other and their compound effect on the outcome. More specifically univariate
models tend to ignore the relationship between exogenous and endogenous factors and their
compounding effect, where multivariate models take this relationship into account.
24

35. In practical terms, going back to the “home to the office” driving example, univariate
models consider the independent impact of the weather conditions and the time of the departure
on the driving time, where multivariate models would consider these two factors in conjunction
with each other.
Even though interactive and Internet marketing are relatively new phenomena they have
generated a fair amount of popular and scientific foment. It is even fair to say that the scientific
community has been lagging behind interactive and Internet marketing practitioners who have
pushed exploration boundaries. At the same time, in recent years, interactive and Internet
marketing, as a subject of scientific inquiry, have seen an increased rate of exciting empirical
research. These studies have focused on interactive and Internet marketing from social,
physiological and physiological perspectives (Jebakumari, 2002; Milley, 2000; Macias, 2000;
Liu, 2002; Newman, 2001; Mark, 2003; Bezjian-Avery, 1997; Raman, 1996).
Unlike Internet and interactive marketing, experimentation is not a new phenomenon.
There is abundant evidence that ancient Egyptians conducted experiments during pyramid
construction. In a more recent example of experimentation James Lind, while servicing onboard
the Salisbury, conducted an experiment of using citrus to cure scurvy. One of the notable
differences between early experimentation efforts and the experiments conducted by Lind was
the use of control and treatment groups. The results of the control group were compared with the
results of treatment groups in order to confirm or reject the hypothesis of the experiment. In the
early 20th century Ronald Fisher formulated a mathematical method for designing and analyzing
experiments. He had introduced “factorial” as the term applicable to experiments involving
several factors or variables at the time (Fisher, 1935). Fisher (1926) initially used “complex
experimentation” as the term describing experimentation with multiple variables at the same
25

36. time. In more recent years researchers have focused on experimentation in the context of product
development.
Enlarged experimentation methods have been researched in the context of Electrical and
Mechanical Engineering (Wang, 1999; Hansen, 1989; Donne, 1996). It is certainly not surprising
that experimentation practice has been widely employed in industrial manufacturing since
manufacturing product commitment is quite expensive and may result in significant losses and
even, in some cases, impact on the long-term survival of the company. The product development
lifecycle often requires experimentation to be part of the product development process. The
deceptive ease of change in the Internet and interactive marketing product development has
resulted in the situation where experimentation best practices found in the industrial product
engineering are ignored. There is certainly a glaring lack of empirical research into
experimentation in the Internet and interactive marketing fields as it pertains to product
development. There are a few empirical works that have broached this subject (Dou, 1999; Li
2003; Ozdemir, 2000); however, these studies are primarily dedicated to data mining and
predictive modeling rather than experimentation as a continuous practice.
It is important to note that in recent years several researchers have focused on
interactivity in general and interactive experimentation in particular as a key driver in product
innovation (Thomke, Von Hippel & Franke, 1997; Thomke, 1998; Thomke 2001; Von Hippel
1998; Thomke; 1995). These studies have asserted that product innovation is driven by product
users themselves through interaction and experimentation. Product innovation was particularly
highlighted in these studies and was considered separately from the remaining phases of product
development lifecycle. In large, this part of this dissertation capitalizes on the mentioned studies
in the context of applying findings of above mentioned research both to Internet and interactive
26

37. marketing.
Some of the mentioned research in the areas of interactive marketing, experimentation
and product innovation is examined in more detail below.
Interactive Marketing
If the Internet timeline could be separated into three decadal stages: (a) mid 1990s –
introduction stage; (b) early 2000 – development stage; and (c) late 2000 – maturity stage; then
the Jebakumari (2002) study could classified as a study of the stages of Internet development. It
was during this time that Internet interactivity came into strong researcher and practitioner focus.
The overall purpose of the Jebakumari (2002) study was to describe interactivity in the context
of Internet marketing. Lyons (2007) offered several research questions: (a) what are the nature,
characteristics and components of interactivity? (b) what are the shortcomings of the traditional
marketing models in context of the interactive medium? (c) how is interactivity related to
comprehension?
Jebakumari (2002) examined traditional marketing and its shortcomings to explain this
interactive phenomenon. Both traditional and interactive Internet marketing were compared and
contrasted. The conclusions reached by Jebakumari (2002) were reminiscent of a similar study
conducted by Mark (2003). Jebakumari (2002) found that a number of traditional marketing
techniques were inconsistent with the interactive media and did not adequately address the
interactive audience.
A study by Milley (2000) could be attributed to the late introductory and early
development stages of Internet marketing. Miley (2002) explored, what he calls, Web-enabled
consumer marketing, its intricacies and specifics. Miley (2002) tried to formulate a theoretical
27

38. model of interactive marketing on the basis of numerous case studies, presented and analyzed in
succession. An additional focus of the study was related to the operational recommendations of
running a consumer oriented interactive web site. Miley (2002) proposed the following research
questions: (a) what is the theoretical basis of Web-enabled consumer marketing? and (b) how
should the company align its operations to be congruent with the Web-enabled consumer
marketing model?
Miley (2002) reached the conclusion that Web-enabled consumer marketing requires
analysis of behavioral user data to guide future actions and marketing decisions. He also
concluded that in order to facilitate comprehensive data analysis, interactive user data must be
both extensive and complete. As such, Web-enabled consumer marketing or interactive
marketing companies must position their human and systems resources, as well as establish
operational practices conducive to data capture.
The Raman (1996) study could be attributed to the introductory period of interactive and
Internet Marketing. Raman (1996) explored interactivity on the Web, at the time when it was
emerging phenomenon. In particular, Raman (1996) examined the desired customer exposure to
online banners. Similar to the later studies by Mark (2003) and Jebakumari (2002) that focused
on the comparison and contrast between traditional and interactive marketing, Raman (1996)
contrasted banner exposure in traditional and interactive marketing models. The Raman (1996)
study is similar to parallel study by Bezjian-Avery (1997) which attempted to define interactive
marketing and its core concepts.
Raman (1996) proposed the following research questions: (a) what are the factors
affecting the desired interactive exposure? and (b) how do the levels of interactive exposure
affect the desired advertising outcome? Raman (1996) concluded that the dominant factor
28

39. affecting desired interactive exposure is predominantly related to interactive content richness.
Additionally, Raman (1996) concluded that an interactive advertisement that speaks to the
consumer on an individual level at the same time as being pertinent and engaging has a high
chance of achieving the designed interactive outcome.
Experimentation
As mentioned earlier, the body of knowledge regarding experimentation and experiment
design is heavily focused on major engineering disciplines. The empirical research on the subject
of experimentation in interactive and Internet marketing is scarce and tangential. This study
relies on several seminal works on Design of Experiments, data modeling and product
innovation. In the area of Design of Experiments this study examined several research papers
related to the Taguchi Method. Weng (2007) presented a detailed analysis of experiment
optimization methods. These methods were compared on the bases of (a) global optimization; (b)
discontinuous object function; (c) non-differentiable function; and (d) convergence rate. Weng
(2007) found that the Taguchi Method scored extremely well in all of the compared categories.
Weng (2007) gave a detailed review of the Taguchi Method itself and its benefits over other
optimization methods. Weng (2007) also suggested several improvements to the Taguchi Method
that can be applicable to experimental design in interactive marketing.
In addition to the design of experiments in Electrical and Mechanical Engineering this
dissertation is based on another tangential topic related to data mining and data modeling in
interactive and internet marketing. It is important to note that it is impossible to conduct
experimentation without being engaged in some form of data mining and data modeling. The
experimentation is enabled by data analysis and data mining. More specifically, the
29

40. experimentation process is data analysis driven. In his essays on interactive marketing Li (2003)
examined three cases of interactive marketing. In the first essay Li (2003) described the
functionality of cross-selling services on an interactive banking web site. Li (2003) analyzed the
behavioral reasoning behind online user actions as they pertain to the purchasing of products and
services offered by the interactive banking web site. In conducting behavioral analysis Li (2003)
utilized several multivariate probit models implemented by Hierarchical Bayer framework. This
dissertation examined the applicability of the models proposed by Li (2003) in conducting
interactive real-time online experiments. In his second essay Li (2003) analyzed the browsing
behavior of users on several interactive web sites. In order to predict future browsing paths Li
(2003) utilized several Poisson and discretized tobit models. These models were compared and
contrasted in the context of their ability to accurately predict user browsing behavior. This
dissertation utilizes the modeling technique findings presented by Li (2003). In his third essay Li
(2003) analyzed purchase and conversion data from several eCommerce web sites. He used this
data to build a predictive purchase model. Li (2003) concluded that his Hierarchical Bayer
framework supplemented with hidden Markov model could accurately predict a path reflecting
user goals ultimately leading to a potential purchase. This dissertation capitalizes on the findings
of this essay during the set up and analysis of the effect of experimentation on reaching
interactive marketing goals.
Similar to the Li (2003) study, research by Dou (1999) utilized similar statistical analysis
for modeling online sales. Dou (1999) examined the applicability of the Catastrophe Theory to
modeling actual behavior and predicting potential purchasing online decisions. Dou (1999)
explored what he termed the data empowered marketing strategy, where data was mined through
tracking users to guide the interactive marketing decisions of the company. Even though Dou
30

41. (1999) did not mention this concept as interactive marketing experimentation by name, he
hypothesized that interactive marketing data can be used to alter the interactive user experience
in real time as more of the user data was collected and analyzed. Dou (1999) called this approach
adaptive marketing communication, where consumer behavior is analyzed through continuous
observation. Dou (1999) proposed that interactive marketing data can be modeled using the
Catastrophe model. He hypothesized that Catastrophe Theory is eminently suitable for this type
of analysis and predictive modeling. Dou (1999) concluded that it was indeed possible to model
and ultimately predict the browsing and purchasing behavior of users on interactive marketing
web sites.
The significance of both the Li (2003) and Dou (1999) studies is the fact that interactive
marketing data is been actively analyzed using a multitude of statistical models in the context of
interactive marketing. However, it is import to note that use of the Taguchi Method for similar
analysis has not been empirically researched. Additional computing paradigms for predictive
data modeling such as Evolutionary Computing and Genetic Algorithms have been explored by
several researchers (Ozdemir, 2002). Ozdemir (2002) argued that Evolutionary Computing offers
real potential in deriving a best fitness value. As such it holds significant promise for online data
modeling and interactive marketing experimentation.
Product Innovation
A significant portion of this study is devoted to analyzing the impact of experimentation
on the product development lifecycle in the context of interactive marketing. Even though there
are few empirical studies that directly deal with experimentation in interactive marketing,
emphasizing the web site as an interactive marketing product, there is a significant body of
31

42. empirical work that is devoted to experimentation in the context of product development. This
dissertation capitalizes on the several seminal works by Thomke and Von Hippel. Thomke
(1995) hypothesized that the mode of experimentation such as prototyping and simulation has a
significant impact on the economics of experimentation. More specifically, Thomke (1995)
proposed that the use of simulation experimentation is more economical and therefore far more
likely to be used in product development. Consequently simulation experimentation could be
viewed as a product enabler and innovation driver. Thomke (1995) presented two case studies
where experimentation was used in the design of new pharmaceutical drugs and integrated-
circuit based systems. Thomke (1995) proposed experimentation design cycles consisting of
designing, building, running and analyzing activities performed in a contiguous manner. Each
successive cycle was built taking into account the findings of the previous cycle. This study
hypothesized the applicability of this cycle in general, and the process in particular, to interactive
marketing product development (see Appendix E). Thomke (1995) found that switching between
prototyping and simulation experimentation modes significantly affected experimentation
economics, resulting in a substantial reduction in design cost.
In a seemingly unrelated study Von Hippel (1998) argued that product innovation should
be driven by the people who would benefit from the end product of innovation, end users of the
product themselves. Von Hippel (1998) described what could be called the Von Hippel paradox,
where product specialists should not be primarily responsible for product innovation, but rather
defer to product users as a source of ultimate innovation. Von Hippel (1998) described this
paradox as a shift in locus of problem-solving. In this dissertation Von Hippel’s ideas are
combined with the approach proposed by Thomke (1995), where interactive marketing product
development is driven by users through interactive experimentation.
32

43. CHAPTER 3: METHODOLOGY
Description of the Methodology
The study assessing the impact of experimentation on interactive product development
utilized a qualitative research paradigm. The choice of qualitative research methodology was
related to the nature of the topic and the innate characteristics of the field of the study.
Employing qualitative research methods makes the quality of the data of paramount importance.
Consequently, emphasis is placed on how and under what circumstances the data is collected
(Morgan & Smircich, 1980). In contrast to quantitative research methods, it is rare to see a
qualitative researcher working with large quantities of data. This is the case with the current
study as all analyzed data comes from a single organization.
Maxwell (1992) defined the qualitative research methods as theory forming. These
methods are used to generate new theories or introduce new hypotheses. Maxwell (1992) called
qualitative research a paradigm that is concerned with a “breadth first” approach as opposed to a
“depth first” as is the case with quantitative research. More specifically, qualitative research is
preoccupied with describing a phenomenon as thoroughly as possible, and forming a theory
behind it. Based on the paradigmatic characteristics provided by Maxwell (1992), the use of
qualitative research methods was consistent with the goals of the study and the state of
knowledge in the field of experimentation in the context of interactive marketing.
This study utilized a phenomenological method as one of the research methodologies
under the qualitative paradigm umbrella. The phenomenological method was first formulated by
Husserl (1983). Creswell (2007) defined the phenomenological method as a description of the
33

44. meaning for several individuals of their direct experience of a concept or a phenomenon. In this
particular case the phenomenon is a process of experimentation in the context of interactive
marketing. Husserl (1983) described the application of this phenomenological approach as an
execution of three consecutive steps. The first step consisted of adopting a phenomenological
method that encouraged the researcher to infuse quantitative data with the qualitative context that
allowed the data to be meaningful. The second step consisted of seeking out an instance where
the phenomenon can be studied in its natural context in order to distill the essence of the
phenomenon. The third and final step was described by Husserl (1983), and consisted of
describing the discovered meaning of the phenomenon.
The experimentation phenomenon in the context of interactive marketing sits well in the
phenomenological inquiry. This researcher wanted to discover meaning behind the
experimentation phenomenon in the context of interactive marketing by studying individuals
who have experienced the phenomenon first hand. Even though participants of the study have
experienced the phenomenon they are not necessarily aware of its meaning (Georgi, 2006). This
point of view is certainly consistent with the description of the experimentation phenomenon.
The participants of this study have certainly experienced the experimentation phenomenon in the
context of interactive marketing, but by and large they are not aware of its meaning and its
fundamental characteristics. By utilizing phenomenological tools as a bracketing,
horizontalization, clustering, delimiting and imaginative variation, phenomenological tools allow
the researcher to extract meaning from the experiences of the participants of the study.
The phenomenological method allows the researcher of the study to quantify his/her own
experience by supplementing findings of the study with his/her own observations and
interpretations in the context of the experience. Creswell (2007) described this type of
34

45. phenomenological method as hermeneutical. Van Manen (1990) described the researcher of the
study as one of the participants of the study. This researcher has an extensive experience with
experimentation in the context of Internet and interactive marketing. The phenomenological
method allows researcher understanding of his own experience while maintaining a strong
relationship to the topic (van Manen, 1990). However, in order to address generalization,
validation, validity and bias, the researcher must employ bracketing to distinguish his own
experience from the experience of the participants of the study. As such this researcher tried to
deemphasize his own experience. This researcher employed a combination of transcendental and
experimental phenomenology rather than hermeneutical phenomenology, where emphasis is
placed on the experience of the participants of the study and the experience of the researcher is
bracketed (Moustakas, 1994). Use of experimental phenomenology allowed the researcher to
focus on the practical application of the phenomenon rather than the philosophical side of it.
It is important to note the distinction between other qualitative methodologies such as
grounded theory or other narrative approaches and phenomenological methods. Creswell (2007)
made a distinction between narrative study and phenomenological study, where the former is
experienced by several study participants individually, as opposed to as a group in a latter case.
Even though participants of the study were selected from several groups participating in
interactive marketing experimentation, individual group representatives described an experience
of the group to which they belong. According to Cresswell (2007) phenomenological methods
place emphasis on the shared experience of the phenomenon. It is critical to understand
experimentation in interactive marketing in the context of a particular group as well as the
organization as a whole.
35

46. Design of the Study
The qualitative research paradigm employs interviews as its predominant data collection
instrument (Babour, 1998). When an interview is conducted in a purely qualitative manner, the
researcher then takes an active participation in the interview process. In that case, the researcher
is considered to be an actual instrument of the study. Babour, (1998) pointed out that participants
in a study often receive major guidance from the researcher throughout the interview process.
The qualitative research paradigm thus encourages researcher participation in order to reduce
language ambiguity and supplement the possible lack of context associated with quantitative data
collection instruments. This study however, did not employ interviews as the means of data
collection. The major concerns of the study were related to credibility, validity and bias. Since
the researcher of the study is employed by the company where the research is being conducted,
actual or potential undue influence was a paramount concern. The researcher struggled to
maintain the balance between extricating himself from the data collection process on one hand,
yet maintaining the qualitative nature of the study on the other hand.
In order to address validity, credibility and bias concepts as well as maintaining the
qualitative nature of the study, the researcher employed a research instrument used in mixed-
method research studies. More specifically this study utilized a mixed-method survey as a
research instrument. Johnson and Onwuegbuzie (2004) also described a mixed-method survey
that embodies both qualitative and quantitative aspects. Mixed-method surveys typically contain
questions found in fixed surveys. These types of questions are referred to as close-ended, where
the set of responses are limited. In addition to quantitative questions, these surveys contain
corresponding sections that allow freehand expression, allowing for a qualitative context to what
36

47. otherwise would be purely quantitative data. In contrast to close-ended questions these questions
are open-ended.
The resultant survey consisted of 15 open-ended questions and 17 close-ended questions.
In order to eliminate possible researcher influence the survey was administered over the Internet
in an anonymous fashion. In addition data was collected under false pretenses. The participants
of the study were not told that the data was being collected for the purposes of research due to
the possibility of participant bias. The survey was positioned as providing helpful feedback on
the experimentation efforts of the company in the context of interactive marketing.
Population and Sampling
The study included 23 human participants. The participants of the study work in the same
organization as the author of the study. A particular set of participants was chosen from all of the
groups involved in the interactive marketing experimentation. The technology group was
excluded from study participation, since the author of the study works for the technology group
and may exert undue influence on the participants of the study. The participants of the study
were randomly chosen from Interactive Marketing, Business Development, User Experience,
Data Analysis and Creative Design groups.
The participants from each of the mentioned business units provided information relevant
to the results of experimentation and its impact on various aspects of interactive marketing. They
were asked to elaborate on their experiences of experimentation in the context of the interactive
marketing. The participants of the study were solicited on the perceived success of
experimentation relative to its goals such as improved conversion, product innovation, product
improvement, risk mitigation and improved competitive standing.
37

48. Assuming that the chosen sources were both valid and credible, further research
credibility and validity depended only on the researcher himself/herself. One of the ways of
selecting credible and valid sources is by selecting them at random. More specifically only a
single representative of each business unit was selected at random. This type of selection method
helped reduce personal bias.
Measurement Strategy
This study surveyed interactive marketing professionals in the confines of a single
company. The study participants were selected at random to represent their interactive marketing
channel. The respondents of the study were asked to complete a mixed-method survey consisting
of 40 questions related to their experience with experimentation in the context of interactive
marketing. The research questions of the survey were designed to understand the relationship
between the corresponding dependent and independent variables. Since the number of
independent variables was too great they were grouped under common categories. For instance,
independent variables related to experimentation such as color, font, font size, and images were
grouped under a visual category. It is important to note that independent variable categories were
classified as either endogenous or exogenous. The resultant survey included five categories of
endogenous independent variables (see Appendix B) (a) visual; (b) functional; (c) positional; (d)
informational; and (e) behavioral, as well as six categories of exogenous independent variables
(see Appendix B) (a) temporal; (b) demographical; (c) seasonal; and (d) contextual.
In addition to the independent variables each of the research questions had a number of
dependent variables associated with it. These dependent variables were assigned as follows (see
Appendix B) (a) competitive standing (revenue, market size, profit, market share, and market
38

49. segmentation); (b) interactive marketing product development lifecycle (product risk, product
innovation, product improvement, product life cycle, and product targeting); and (c) interactive
marketing goals (cost per acquisition, cost per impression, cost per action, upsell, and click-
through rate).
Each research question was directly represented in the survey in the form of several
survey questions. In addition to asking participants of the study to answer research questions
directly, each of the dependent variables was investigated in isolation (see Appendix A). The
survey questions were formulated to draw a connection between independent variables in an
amortized form and the dependent variables associated with research questions. The amortized
independent variables were referred to as “Experimentation” (see Appendix A), where dependent
variables were called out in exactly the same way as they were specified in the “Conceptual
Framework” (see Appendix B).
Even though the quantitative questions were utilized alongside qualitative questions,
qualitative data was not used in drawing conclusions of the study. The point of analysis
associated with the quantitative data was to ascertain consistency between qualitative and
quantitative answers. The quantitative survey questions used single and multiple choice scales.
The qualitative survey questions utilized a measurement strategy associated with the
phenomenological method consisting of horizontalization, clustering, textual and combined
description.
Instrumentation
The survey (see Appendix A) contained 40 questions designed to solicit information
related to the experimentation efforts of the company in the context of interactive marketing. The
39

50. company where the research was conducted utilized multifaceted interactive content. More
specifically the company used email, search, social, display, internet and affiliate interactive
marketing approaches. The participants of the study were asked to fill out the survey relating to
their experience of product development through experimentation in each of the respective
interactive marketing areas. The questions of the study were crafted to meet the objectives of the
study. The close-ended questions of the study were not used in the final analysis of the study, but
rather they were designed to ensure consistency of a corresponding open-ended question as well
as to guide the user to stay in the confines of the intended question. The survey was designed and
implemented using online survey software and conducted over the Internet. The participants of
the study were invited by the CEO of the company to complete the survey via. The email
contained the link to the online survey as well as an explanation of the purpose of the survey,
ensuring that participation in the study was voluntary and anonymous.
The survey contained five major sections: (a) introduction (questions related to overall
experimentation experience); (b) interactive marketing goals (questions related to the impact of
experimentation process on the goals of various interactive marketing channels); (c) interactive
marketing product (questions related to the impact of the experimentation process on the various
aspects of product development lifecycle of various interactive marketing channels); (d)
competitive standing (questions related to the impact of experimentation process on the various
key competitive indicators); and (e) sustaining effects (questions related to the sustaining effects
of experimentation in the context of interactive marketing)
Data Collection
The data was collected via the SurveyMonkey.com web site. The initial survey was pre-
40

51. tested and modified according to the feedback from the pilot group and the mentors of the study.
The pilot group consisted of three members chosen from a pool of potential participants. The
participants of the study were given a week to complete the survey with multiple reminders sent
two days and on the day before the survey expiration period. All questions in the study were
designated as mandatory and the only two ways to exit the survey were to either to complete it or
abandon it. If the survey was abandoned in order to proceed with the survey at the future date the
participant of the study had to start the survey over again. According to the SurveyMonkey.com
statistics none of the surveys were abandoned and the effective survey completion was at 100%.
Due to employing the survey online participant anonymity was preserved. After all surveys were
completed the survey results were downloaded onto the researcher’s computer and analyzed. At
all times the survey results were protected from inadvertent or intentional disclosure. The
surveys were conducted online over secure protocol and access to the survey results was
username and password protected. When the results of the survey were downloaded to the
researcher’s computer access to the computer itself was username and password protected as
well.
Data Analysis Procedures
The data analysis procedures roughly consisted of the steps outlined by Creswell (1998)
with slight adaptation for the needs of this study. These steps consisted of (a) horizontalization
and bracketing; (b) clusters of meaning; (c) textual description; (d) composite description. It is
important to note that the usual phenomenological step of transcription was omitted since the
data was collected via an open-ended survey administered over the Internet. As such the data
transcription consisted of downloading the results of the completed surveys. The data analysis
41