1. Experimental Design
Learning Objectives
Define experimentation
Explain pros and cons of the experimental method
Become familiar with key design principles of
experiments
Define internal validity
Define external validity
2. Experiments
Manipulation of one variable to determine its
effect on other variables
e.g. the impact of insurance plan (has
insurance vs. no insurance) on auto repair
quote
e.g. the design of suspect identification
(line-up vs. one by one) on eyewitness
accuracy
8. How much the researcher manipulates and
controls the independent variables
None – observations, document analysis, case
studies*, surveys*
Little – ex post facto experiments (or quasi-
experiments)
Full control – experiments
*Surveys and case studies can be experimental depending on whether researcher manipulation
is introduced
9. Conducting an Experiment
Specify treatment variables
Specify treatment levels
Control environment
Choose experimental design
11. Experiments – Pros and Cons
Advantages Disadvantages
• Ability to manipulate IV •Artificiality of labs
• Use of control group •Non-representative
• Control of extraneous sample
variables •Expense
• Replication possible •Focus on present and
• Field experiments immediate future
possible •Ethical limitations
12. Discussion
If your group were to address your research questions
using the experimental methodology, how would you
design the experiment?
Discuss the following:
Independent variable(s)? (also called manipulation)
Dependent variable(s)? (also called outcome variables)
Confounding variable(s)? (potential alternative
explanations)
Extent of control: full experiment or ex post facto?
Would you prefer a lab or field experiment?
How would you achieve random assignment?
How do you ensure blindness (or double-blindness?)
How do you make sure it’s not just a non-experiment?
(e.g., an after-only case study)
The next set of slides is review from Session 3 – Research DesignLet’s first see if you can reconstruct from what we discussed a couple of weeks ago about experimentsWhat are the basic elements of an experiment?What does an experiment involve?An experiment is an attempt to establish cause and effectIt’s the best design available for a causal studyMany experiments have been conducted in different areas of social sciences (e.g., psychology, economics, etc). When done well, they often offer the strongest evidence supporting any given claim or argument.
Why do businesses and social scientists obsess so much about causality?Understanding causal relationships is extremely critical to problem solving. We can't solve a problem unless we understand what the root cause is! For example, if we want to know why employees have low levels of productivity, we need to find the potential causes – health issues? Motivation? Compensation? Organizational culture? We can attempt to improve productivity only after we have identified the most important causes.(However having said that, single linear causal models have its limitations - Refer to Peter Senge's systems thinking)To make a causality statement, you need strong support for your argumentWe need at least 3 pieces of evidence:1. Your independent variable correlates with your dependent variable in some way. For example, the temperature goes up, the likelihood of getting sick goes down, and vice versa. This is “negative” correlation. It’s a necessary but not sufficient condition for causality2. The independent variable must precedes the dependent variable. If you turn 30 before eating a birthday cake, it’s hard to explain why eating a cake causes you to get older!3. Control for confounding variables - You should be able to demonstrate that no other variables could possibly cause the observed changes in the dependent variable. This is hard to do with an observational study like the temperature and likelihood of getting sick. You see a pattern, but because you didn’t control other factors, you can’t really say for sure that there’s no other causes hidden somewhere.This is why “fully controlled” experiments are so highly valued in scientific studies because they provide the best way to determine causality with certainty. Unfortunately they are not always feasible, especially in the social sciences where manipulation of people is not always ethical.
Most of us are familiar with experimental strategy simply because we read the medical news all the time. A group of patients with a specific illness are given current treatment X while another group of patients are given treatment Y. Typically, a third group called control group, are given treatment Z (which is the same treatment X but patients are told it is a new treatment). If the group with treatment Y shows a statistically difference (hopefully improvement) then we say that treatment X is effective in treating the specific illness (note – the control group is needed because some people improve just because they are told they are being given a new state of the art medicine!)Note there are two key design elements here:The control group – we discussed this aboveThe pre-test – measurement of the baseline (how patients did before the treatment) is important in establishing whether significant improvement is made
Causal studies are differentiated by their ability to control and manipulate variables. Causal studies may be experiments or ex post facto studies. Experiments are studies involving the manipulation of one or more variables to determine the effect on another variable. For example, direct marketers can use split tests on mailings to test which mailing resulted in the highest response rate. Ex post facto designs are evaluations made after-the-fact based on measured variables. For example, you can create different stories about your insurance coverage and see how repair shops react differently when giving you quotes. You as a researcher play an active role in influencing and changing how the participant (i.e. repair shop owner) responds. That would make it an experiment. Because you have full control of who gets to what version of your cover-up story, you are running a real experiment.With quasi-experiments, you expect different groups of people to react/behave differently, but you have no control who is in which group. For example, if you’d like to see the impact of financial problems on marriage, ideally you’d like to run an experiment. You can cause half of your group to have financial problems (perhaps by stealing their 401K fund), and the other half not, and see which group has more marriages falling apart. Unfortunately this is not really considered an ethical thing to do and so you can’t really run this experiment, no matter how clever the design is.Your best compromise is to have a quasi-experiment. You let people decide which treatment group they end up being in – the victim groupor the control group. Then you compare their marriage quality across groups. Same thing with gender differences. You can’t assign people to the female group or the male group. Your best compromise is to go with whatever gender they are instead of mandating a sex change operation!With surveys, observations, document analysis, case studies, etc, the researchers observe and analyze the observations. They do not play an active role in influencing the study participants behaviors, reactions, or feelings. Caution: Surveys and case studies can be experimental depending on whether researcher manipulation is introduced. A clever way of conducting experiment is giving people two different versions of a questionnaire by embedding the experimental manipulation in the survey.
