This document summarizes a research study on complex problem solving using a computer simulation called Syntex. The study involved 54 students from Zhejiang University divided into 18 groups. The groups made management decisions for a simulated company each month. The researchers found differences between the "best" and "worst" groups based on company performance. Best groups increased capital and hired more employees over time compared to worst groups. The researchers also analyzed decision making processes and information gathering between the groups. They explored perspectives including the problem solving process, group interactions, and cross-cultural differences. The document discusses research design, validity, variables measured, and limitations of generalizing the results.

1. Quantitative Research Methodologies (2/6):
Research Design and Validity
Prof. Dr. Hora Tjitra & Dr. He Quan
www.SinauOnline.com

2. 2
Research Team
Research Program: Complex
Problem Solving (CPS)
Hora Tjitra, HE Quan, LIU
Manxia, SHI Wei, ZHANG Qi, YU
Zhaoyang, WANG Huiqi
Project Team Leader
Complex Problem Solving Research project
Project Team Member
The Police Bureau, Huzhou
Project Team Member
Zhongcai Group, Hangzhou
Phd. Candidate, Zhejiang University
Industry and Organization Psychology
Bachelor Degree, Zhejiang University
Industry and Organization Psychology

3. 3
Complex Problem Solving Stories
Economy
Environment
Technology
Education
Improve The
Well Being Of
Citizens in
One Nation
Politics
Social
The Moral Issue aroused by the
Biological Technology
Overexploiting The Underground
Water in Beijing City

4. 4
CPS Theoretical Framework

5. CPS Research Design
Observation System
• Group Interaction
• Decision Making Process
Syntex System Data
• Corporate Results
• Group Knowledge
• Group Decisions
Questionnaire
• Quantitative Data
Interview Coaching
• Group Reflection
SYNTEX
• Computer Business Simulation
• Managing Complex Situations
Multi-Methods
Approach
AmericaChina IndonesiaGermany France Japan
Cross-Cultural
Perspective

6. 6
Background Information and Experimental Process
Participants
Make decisions by themselves for company
every month, for example the products order
and the material purchasing.
The Task
3-persons cooperate, 54 persons divided in 18
groups, the students coming from Zhejiang
University
Program
Display

7. 7
Product
Quality
The system structure behind Syntex business simulation
Material
Storage
Purchasing
Materials
Employee
Satisfaction
Market
Demand
HR
Performance
Product
Storage
Advertisement
Salary
Pricing Policy
Production
Order
Recruitment Social
Fund
Product
Manufacture
Product Sales Company
Income
Season /
Weather
Competitor
Price / Quality
Purchasing
+
-
- -
+
+
+
+
+
+
+
+
+
Employee
Number
Oval represents the control variable, rectangle: system variable,
‘+’means active influence,’ –’mean negative influence .
Company
Assets
Maintenance
Machine /
Diesel
Leadership
HR
Work Load
Lay-off
Exchange
Resign
+
-
+
-
+
+
+
-
Number and
Assortment
Sale
PriorityMan.
Priority
and
• Bank Account
• Material Storage
• Product Storage
• Machine, Diesel

8. 8
The differences between “best” and “worst” groups
The Performance＝100*[2*Z（capital）+Z（number）+Z（satisfaction）]/4
Sorting the groups by score, selecting higher 8 groups as best groups, lower 8 groups as worst groups. ( t(14)=4.34, p=.001 )
Z=0Normal Distribution
The change of capital during 12 months:
Better groups can increase the capital
The size of the company : better groups hire
more people than worse
The change of employees’ satisfaction: both groups
can’t manage very well, worse groups are worse

9. 9
Information and Decision Making Difference
Best groupsBest groups Worst groupsWorst groups
Variables M SD M SD
Number of questions, total 124 29.76 123 20.71
Number of decisions, total 55 11.01 61 13.63
MANOVA( Repeated
Measure)
Information Decision
Condition F(1/14)=.00 p=.95 F(1/14)=1.02 p=.33
Month F(5.69/79.72)=7.06 p=.00 F(8.51/69.18)=3.67 p=.00
Interaction F(5.69/79.72)=1.31 p=.26 F(8.51/69.18)=1.29 p=.25

10. 10
The Analyzing Perspectives from the Data
AmericaChina IndonesiaGermany France Japan
3.Cross-Cultural Perspective
1.CPS Process Perspective
How do groups explore and manage the system?
What is the difference between the best and worst groups in cognitive
perspectives
2.Group Interaction
Perspective
How do group member discuss and inquire the information?
How do group member support each other?

11. 11
Variables Used in Experimentation
Dependent
Variable
Independent
Variable
Variable
…One of the antecedent conditions
manipulated by the experimenter
…The response of the organism; the variable
that measures the influence the independent
variable
…Any characteristic or phenomenon that can vary
across organisms, situations, or environments

12. 12
Three Main Research Designs
Using Experimental Approach
A design in which the influence of extraneous variables is
controlled for while the influence of the independent variable is
tested
Research design: The outline, plan, or strategy used to investigate the research problem
A research design in which an experimental procedure is applied
but all extraneous variables are not controlled
Experimental
design
Quasi-
Experimental
design
Single-case
design
A research design in which a single groups of individuals is used
to investigate the influence of a treatment condition

13. 13
Experimental Design
• the experimental and the control groups‘
posttest scores are compared to assess
the influence of the treatment condition
X
Y2
Y1
treatment measure
Posttest only design Pretest-posttest design
• the treatment effect is assesses by the
difference between the experimental and
control groups‘ pre- and posttest scores
X
Y2
Y1
treatment measurePremeasure
Y
Y

14. 14
Quasi-Experimental Design
Nonequivalent comparison
group design
Interrupted time-series
design
• The results obtained from nonequivalent
experimental and control groups are
compared
Premeasure treatment measure
Y
Y
X Y1
Y2
• A treatment effect is assessed by
comparing the pattern of pre- and posttest
scores of one group of research
participants
X Y
treatment measurePremeasure
Y Y Y

15. 15
Basic Types of Designs for Case Studies
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
single-case designs multiple-case designs
holistic
(single-
unit of
analysis)
CONTEXT
Case
Embedded
Unit of
Analysis 1
Embedded
Unit of
Analysis 1
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
embedded
(multiple
units of
analysis)
SOURCE: COSMOS Corporation

16. 16
Research Validity
... the best available approximation to the truth or falsity of
propositions, incl. propositions about cause
1. Is there a relationship between two variables ?
2. Given that there is a relationship, is it plausibly causal from one operational
variables to the other or would the same relationship have been obtained in the
absence of any treatments of any kind ?
3. Given that the relationship is plausibly causal and is reasonably known to be
from one variable to another, what are the particular cause and effect constructs
involved in the relationship ?
4. Given that there probably a causal relationship from construct A to construct B,
how generalizable is this relationship across persons, settings, and times ?

17. 17
Four Types of Validities
... Validity of the inference made about whether the
independent and dependent variables co-vary
... Validity of the inference that independent and
dependent variables are causally related
... Validity of the inference about the higher-order
constructs from the operations used to represent
them
... Validity of the inference about whether the causal
relationship holds over people, setting, treatment
variable, measurement variables, and time
Statistical Conclusion
Validity
Internal Validity
Construct Validity
External Validity

18. 18
Internal Validity
... a relationship between two variables is causal or that the absence of a relationship implies
the absence of cause.
A
Causes / Treatments /
Independent Variables
B
Effects / Outcomes /
Dependent Variables
C2
Comparison / Experimental /
Control Units
C1
Extraneous Forces /
Controlled Setting
1. Is there a relationship between A & B ?
2. Is there a causal relationship from A to B ?
3. Is there any other reason that could
also caused B apart from A ?

19. 19
Internal Validity - Threats
... Validity with which statements can be made about whether there is a causal relationship from
one variable to another in the form in which the variables were manipulated or measured.
• History
... an event which takes place between the
pretest and posttest.
• Maturation
... the respondent‘s growing older, wiser,
stronger, more experienced, and the like
between the pretest and posttest.
• Instrumentation
... a change in the measuring instruments
(e.g. observer learning effects)
• Testing
... Changes in person’s score on the
second administration of a test as a result
of previously having taken the test
... is a threat when an observed effect might be due to ...
• Regression artifact
... The tendency for extreme scores to
become less extreme on a second
assessment
• Attrition
... Some people do not show up for the
study or do not complete it
• Selection
... The choice of participants for the
various treatment groups is based on
different criteria
• Additive and interactive effects
... The combined effect of several threats
to internal validity

20. 20
... the approximate validity with which we can make generalizations about high-order
constructs from research operations.
Construct Validity of Causes and Effects
A
Independent Variables
B
Effects / Outcomes /
Dependent Variables
Causes / Treatments
C2A
Independent Variables
C2B
Effects / Outcomes /
Dependent Variables
Manipulated Group 1
Control Group
C3A
Independent Variables
C3B
Effects / Outcomes /
Dependent Variables
Placebo
Manipulated Group 2

21. 21
Construct Validity - Threats
• Inadequate Preoperational Explication of Constructs
…the choice of operations should depend on the result of a conceptual analysis of the
essential features of a construct (e.g. Aggression).
• Mono-Operation/ Method Bias
…many experiments are designed to have only one exemplar of a particular possible
cause, and some have just one measure to represent each of the possible effect
constructs.
• Reactive Self-report Changes
... it is possible that the treatments effects are due to the respondents being willing to
present themselves to experimenters in ways that would lead to favorable evaluation.
• Experimenter Effects
... which indicates that experimenter‘s attitudes/expectancies can bias the data obtained.
• Treatment Diffusion
…individuals in one treatment group receive some or all of another group’s treatment
• Restricted Generalizability Across Constructs
…sometimes treatments will affect dependent variables quiet differently, implying a positive
effect on some construct and an unintended negative effect on another

22. 22
... the presumed causal relationship can be generalized to and across alternate measures of the
cause and effect and across different types of persons, settings, and times. (1)
External Validity
... the approximate validity with which conclusions are drawn about the generalizability of a
causal relationship to and across populations of persons, settings, and times. (2)
1. Definition of the Target Populations
(persons, settings, or times)
2. Samples to Represent
These Populations
Types of Samples / Populations:
1. Target Populations
2. Formally Representative samples
that correspondent to known
populations
3. Sample Actually Achieved (in Field
Research)
4. Achieved Populations

23. 23
The categories of the external validity
... The extent to which the results of a study can be
generalized to the larger population
Population Validity
... The extent to which the results of a study can be
generalized across settings or environmental conditions
Ecological Validity
... The extent to which the results of a study can be
generalized to the larger population
Temporal Validity
... The generallizability of results across different but related
dependent variables
Outcome Validity
Treatment Variation
Validity
... The generalizability of results across variation of the
treatment

24. 24
External Validity - Threats
• Interaction of Selection and Treatment
In which categories of persons can a cause effect relationship be generalized?
• Interactions of Setting and Treatment
Can a causal relationship obtained in a factory be obtained in a military camp or an
university campus?
• Interaction of History and Treatment
To which period in the past and future can a particular causal relationship be
generalized?

25. 25
The literatures
Altschuler, L., Sussman,N.M., & Kachur, E. (2003). Assessing changes
in intercultural sensitivity among physician trainees using the
intercultural development inventory. International Journal of
Intercultural Relations, 27(4), 387–401.
Leclerc, D., & Martin, J.N. (2004).Tour guide communication
competence: French, German and American tourists'
perceptions. International Journal of Intercultural Relations(4),
28,181-200.
Palthe, J. (2004).The relative importance of antecedents to
cross-cultural adjustment: implications for managing a global
workforce. International Journal of Intercultural Relations, 28(1), 37-59.
Strohschneider, S., & Guess, D. (1999). The Fate of the Moros: A
Cross-cultural Exploration of Strategies in Complex and
Dynamic Decision Making. International Journal of Psychology,34(4),
235-252

26. @ Tjitra,2010
Thanks You
Any comments & questions
are welcome
Contact me at hora_t@sianuonline.com
26
www.SinauOnline.com