2. Adhy Kurniawan
►1987-1990 SMA 3 Semarang
►1990-1991 Fac. Of Economy, Diponegoro
University, Semarang
►S1(1991-1996) Civil Engineering Dept.
Gadjah Mada Univ.
►S3(1998-2003) Swiss Federal Institute of
Technology Lausanne (EPFL), Swiss
►Post Doct (nov.2005-sept.2006) Kyoto
University, Japan
3. My Goals for Course
►That each of you develop an
intuition for the fundamental
principles of research methodology
►That we have an enjoyable
semester learning together
4. Lecture and Homework
Homework
►Your chance to practice using the
concepts presented in class
►Teamwork vs. Individual work?
Lecture
• Presentation and discussion
5. References
►All of literature concerning: Res Met
► Marczyk, DeMatteo, Festinger. 2005,
Essentials of Research Design and
Methodology, John Wiley and Sons.
►Day and Gastel, 2006, How to write and
Publish a Scientific Report, Greenwood
Press
►Metodologi Riset, Etc.
6. List of students
► Alif Ardy Saputra, Geodesi UGM
► Anik FR, TL, ITB
► Ashri Uswatun, TFisika,UGM
► Ayi Fajarwati, TL, ITB
► Corry Agustina, Perenc Wil, TA, UGM
► Dwi Astuti, TKimia, UGM
► Elva Nur , TF, UGM
► Erika Kezia, TL, ITB
► Fitri Wijayanti, Fisika, UNS
► I Nyoman Kusuma, TF, UGM
► Ihsan Hasan, T Industri, UII,
► Ihwan Ghazali, T Industri, UAD
► Iin Lestari, TL, ITB
► M Sony Abertiawan, TL, ITB
► Maria Auliana, T Sipil, UGM
► Norma Pradipta, TArsitektur, UGM
► Satrya Alrizki, TGeofisik, ITB
► Tatag Lindu Bhakti, TFisika, UGM
7. Contents
The aims of research,
the research topic,
title and research problem,
literature review,
research design: population and sampling types, types
of quantitative research designs, validity of
conclusions, data-collecting methods and
measuring instruments in quantitative research,
qualitative research designs,
data analysis and interpretation of results,
report writing and the research proposal,
ethical consideration on research.
8. OVERVIEW OF SCIENCE AND
THE SCIENTIFIC METHOD
science can be defined as a methodological and
systematic approach to the acquisition of new
knowledge.
This definition of science highlights some of the
key differences between how scientists and
nonscientists go about acquiring new
knowledge.
Specifically, rather than relying on mere casual
observations and an informal approach to learn
about the world, scientists attempt to gain new
knowledge by making careful observations and
using systematic, controlled, and
methodical approaches (Shaughnessy &
Zechmeister, 1997).
Shaughnessy, J. J., & Zechmeister, E. B. (1997). Research methods in
psychology (4th ed.). Boston: McGraw Hill.
9. ►In addition, scientific knowledge is not
based on the opinions, feelings, or
intuition of the scientist.
►Instead, scientific knowledge is based on
objective data that were reliably obtained
in the context of a carefully designed
research study.
►In short, scientific knowledge is based
on the accumulation of empirical
evidence (Kazdin, 2003a)
Kazdin, A. E. (2003a). Methodology: What it is and why it is so important. In A.
E. Kazdin ( Ed.), Methodological issues and strategies in clinical research (3rd
ed., pp. 5–22). Washington, DC: American Psychological Association.
10. ►The defining characteristic of scientific
research is the scientific method .
►First described by the English
philosopher and scientist Roger Bacon in
the 13th century, it is still generally
agreed that the scientific method is
the basis for all scientific
investigation.
►The scientific method is best thought of
as an approach to the acquisition of new
knowledge, and this approach effectively
distinguishes science from nonscience.
11. The Scientific Method
The development of the scientific method is
usually credited to Roger Bacon, a philosopher
and scientist from 13th-century England,
although some argue that the Italian scientist
Galileo Galilei played an important role in
formulating the scientific method.
Later contributions to the scientific method were
made by the philosophers Francis Bacon and
René Descartes.
12. ►Although some disagreement exists
regarding the exact characteristics of the
scientific method, most agree that it is
characterized by the following elements:
● Empirical approach
● Observations
● Questions
● Hypotheses
● Experiments
● Analyses
● Conclusions
● Replication
13. Empirical Approach
► The scientific method is firmly based on the empirical
approach. The empirical approach is an evidence-based
approach that relies on direct observation and
experimentation in the acquisition of new knowledge (see
Kazdin, 2003a).
► In the empirical approach, scientific decisions are made based
on the data derived from direct observation and
experimentation.
► Contrast this approach to decision making with the way that
most nonscientific decisions are made in our daily lives.
► For example, we have all made decisions based on feelings,
hunches, or “gut” instinct. Additionally, we may often reach
conclusions or make decisions that are not necessarily based
on data, but rather on opinions, speculation, and a hope for
the best.
► The empirical approach, with its emphasis on direct,
systematic, and careful observation, is best thought of as the
guiding principle behind all research conducted in accordance
with the scientific method.
14. Observations
► An important component in any scientific investigation is
observation. In this sense, observation refers to two
distinct concepts—being aware of the world
around us and making careful measurements.
► Observations of the world around us often give rise to the
questions that are addressed through scientific research.
► For example, the Newtonian observation that apples fall
from trees stimulated much research into the effects of
gravity. Therefore, a keen eye to your
surroundings can often provide you with many
ideas for research studies.
15. Questions
► After getting a research idea, perhaps from making
observations of the world around us, the next step in the
research process involves translating that research idea
into an answerable question.
► The term “answerable” is particularly important in this
respect, and it should not be overlooked.
► It would obviously be a frustrating and ultimately
unrewarding endeavor to attempt to answer an
unanswerable research question through scientific
investigation.
► It is therefore important to formulate a research
question that can be answered through available
scientific methods and procedures.
16. Hypotheses
► The next step in the scientific method is coming up with a
hypothesis, which is simply an educated—and testable
—guess about the answer to your research question.
► A hypothesis is often described as an attempt by the
researcher to explain the phenomenon of interest.
► Hypotheses can take various forms, depending on the
question being asked and the type of study being
conducted.
► A key feature of all hypotheses is that each must make a
prediction.
► Remember that hypotheses are the researcher’s attempt
to explain the phenomenon being studied, and that
explanation should involve a prediction about the
variables being studied.
► These predictions are then tested by gathering and
analyzing data, and the hypotheses can either be
supported or refuted on the basis of the data.
17. ► Two types of hypotheses with which you should be
familiar are
● the null hypothesis
● and the alternate (or experimental) hypothesis.
► The null hypothesis always predicts that there will be no
differences between the groups being studied.
► By contrast, the alternate hypothesis predicts that there
will be a difference between the groups.
● For example,
● the null hypothesis would predict that the
exercise group and the no-exercise group will not
differ significantly on levels of cholesterol.
● The alternate hypothesis would predict that
the two groups will differ significantly on
cholesterol levels.
● Homework: Individual
● Please try to find one example. About the null
18. Experiments
► After articulating the hypothesis, the next step involves
actually conducting the experiment (or research
study).
► For example, if the study involves investigating the
effects of exercise on levels of cholesterol, the
researcher would design and conduct a study that would
attempt to address that question.
► As previously mentioned, a key aspect of conducting a
research study is measuring the phenomenon of interest
in an accurate and reliable manner.
► In this example, the researcher would collect data on the
cholesterol levels of the study participants by using an
accurate and reliable measurement device.
► Then, the researcher would compare the cholesterol
levels of the two groups to see if exercise had any
effects.
19. Accuracy vs. Reliability
► When talking about measurement in the context of research,
there is an important distinction between being accurate and being
reliable.
► Accuracy refers to whether the measurement is correct, whereas
reliability refers to whether the measurement is consistent.
► An example may help to clarify the distinction.
● When throwing darts at a dart board, “accuracy” refers to
whether the darts are hitting the bull’s eye (an accurate dart
thrower will throw darts that hit the bull’s eye).
● “Reliability,” on the other hand, refers to whether the darts
are hitting the same spot (a reliable dart thrower will throw
darts that hit the same spot).
● Therefore, an accurate and reliable dart thrower will
consistently throw the darts in the bull’s eye. As may be
evident, however, it is possible for the dart thrower to be
reliable, but not accurate.
● For example, the dart thrower may throw all of the darts in
the same spot (which demonstrates high reliability), but that
spot may not be the bull’s eye (which demonstrates low
accuracy).
20. Analyses
►After conducting the study and gathering the
data, the next step involves analyzing the data,
which generally calls for the use of statistical
techniques.
►The type of statistical techniques used by a
researcher depends on the design of the study,
the type of data being gathered, and the
questions being asked.
►It is important to be aware of the role of
statistics in conducting a research study.
►In short, statistics help researchers minimize
the likelihood of reaching an erroneous
conclusion about the relationship between the
variables being studied.
21. Conclusions
►After analyzing the data and determining
whether to reject the null hypothesis, the
researcher is now in a position to draw some
conclusions about the results of the study.
►For example, if the researcher rejected the null
hypothesis, the researcher can conclude that
the phenomenon being studied had an effect—
a statistically significant effect, to be more
precise.
● If the researcher rejects the null hypothesis in
our exercise-cholesterol example, the researcher
is concluding that exercise had an effect on levels
of cholesterol.
22. ►It is important that researchers
make only those conclusions that
can be supported by the data
analyses.
►Going beyond the data is a cardinal
sin that researchers must be careful
to avoid.
23. Replication
►One of the most important elements of
the scientific method is replication.
►Replication essentially means
conducting the same research study a
second time with another group of
participants to see whether the same
results are obtained.
►The same researcher may attempt to
replicate previously obtained results, or
perhaps other researchers may
undertake that task.
24. ►Replication illustrates an important point about
scientific research—namely, that researchers
should avoid drawing broad conclusions based
on the results of a single research study
because it is always possible that the results of
that particular study were an aberration.
►In other words, it is possible that the results of
the research study were obtained by chance or
error and, therefore, that the results may not
accurately represent the actual state of things.
►However, if the results of a research study are
obtained a second time (i.e., replicated), the
likelihood that the original study’s findings were
obtained by chance or error is greatly reduced.
25. ►What are the three general
goals of scientific research?
28. What Exactly is Research?
►we will focus on two of the most
common types of research—
● correlational research
● and experimental research
29. Correlational research:
► In correlational research, the goal is to determine
whether two or more variables are related. (By the way,
variables” is a term with which you should be familiar.
► A variable is anything that can take on different values,
such as weight, time, and height.)
► For example, a researcher may be interested in
determining whether age is related to weight. In this
example, a researcher may discover that age is indeed
related to weight because as age increases, weight also
increases. If a correlation between two variables is strong
enough, knowing about one variable allows a researcher
to make a prediction about the other variable.
It is important to point out, however, that a correlation—
or relationship—between two things does not necessarily
mean that one thing caused the other.To draw a cause-
and-effect conclusion,
researchers must use experimental research.
.
30. Experimental research:
► In its simplest form, experimental research involves
comparing two groups on one outcome measure to test
some hypothesis regarding causation.
► For example, if a researcher is interested in the effects of
a new medication on headaches, the researcher would
randomly divide a group of people with headaches into
two groups.
► One of the groups, the experimental group, would receive
the new medication being tested.
► The other group, the control group, would receive a
placebo medication (i.e., a medication containing a
harmless substance, such as sugar, that has no
physiological effects).
31. Experimental research:
Besides receiving the different medications, the groups
would be treated exactly the same so that the research
could isolate the effects of the medications. After
receiving the medications, both groups would be
compared to see whether people in the experimental
group had fewer headaches than people in the control
group.
Assuming this study was properly designed (and properly
designed studies will be discussed in detail in later
chapters), if people in the experimental group had fewer
headaches than people in the control group, the
researcher could conclude that the new medication
reduces headaches.
32.
33. Task
►Compose your own brief research
proposal.
►Try to determine your research
topic for MST final project
►Format:
● 1. In MS Word
● 2. In Power point
34. Task/assignment next week
►Review 1 International Publication
(Journal, Conference paper,etc)
related to Renewable energy
►Compose the summarize of your
review
►Format:
● 1. In MS Word
● 2. In Power point
35. Purpose of the research proposal
►1. To inform the reader of nature of your
proposed research.
● What is the problem?
● What is its extent?
►2. To convince the reader, especially
supervisors and reviewers, of the value
of your proposed research.
● Is this project worth the time
and money?
● Will it make a difference to the
world?
36. Purpose of the research proposal
►3. To demonstrate your expertise and
competency in a particular area of study.
● Do you have the qualifications to conduct
this research?
● Have you informed yourself of the
existing theory and data relevant to your
topic?
● Do you have the
necessary skills to
conduct the research?
37. Purpose of the research proposal
►4. To plan the research project and provide a
step-by-step guide to the tasks necessary
for its completion.
● What are the key stages of the work?
● What are the priorities?
● How do the various components fit together?
►5. To request support from individuals and
agencies who provide supervision,
oversight or funding for the research
project.
● What kinds of support does the project need?
● Are all participants properly protected?
38. Purpose of the research proposal
►6. To contract with the agencies
and individuals involved, including
supervisors, foundations and
participants in the research team.
● How will tasks be assigned and
resources expended?
● What does each contribute
to the collective endeavor?
39. First things first
1. Basics
2. Topic ideas
3. Typical methodologies
4. Common pitfalls
5. Getting started and putting it all
together
6. Questions/discussion
40. Basic steps of a research
project
►Find a topicWhat, When
►Formulate questionsWhat, Why
►Define populationWho, When
►Select design & measurementHow
►Gather evidenceHow
►Interpret evidenceWhy
►Tell about what you did and found out
41. Selecting a Research Topic
►What are some considerations
when selecting a research topic?
42. Considerations in Selecting a
Topic
● Personal interest / Passion
● Importance / Contribution to the field
● Newness / Relevance
● Feasibility
Tradeoff between rigor and practicality
Time constraints
Ethical constraints
Organizational support
Economic factors
Availability of Subjects
43. Sources of Research Topics
● Peer-reviewed journals in your field
● Personal experiences
● Work setting experiences
● Existing literature
“Recommendations for future research…”
44. Refining Your Topic
►Refinement needed for effective and efficient research
●Narrow your topic
●Identify a theoretical framework
●Specifically and unambiguously define terms
●State research questions and hypotheses
45. Refining Your Topic (cont’d)
►A literature review will help you
●See if your idea has been tried
●Include all relevant constructs
●Select instruments
●Anticipate common problems
46. Components of a Concept
Paper
● Title page
● Introduction
● Nature of the Problem
● Background and Significance of the
Problem
● Preliminary Literature Review
● Initial Research Question or
Questions
47. Components of a Concept
Paper (cont’d)
● Brief Description of Methodology
and Research Design
● Anticipated Outcomes
● Timeline
● References
51. What is a Literature Review?
• According to Creswell (2005), a
review of the literature “is a written
summary of journal articles, books
and other documents that describes
the past and current state of
information, organizes the literature
into topics and documents a need
for a proposed study.” (pp. 79)
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and EvaluatingCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating
Quantitative and Qualitative ResearchQuantitative and Qualitative Research
52. Focusing on Empirical Research
►What does Empirical Mean?
►Primary Sources
● Original Research Article
►Secondary Sources
● Newspapers
● Book chapters
● Television/Radio
● Magazines
● Wikepedia
53. Empirical Research
► All empirical research is inherently
flawed
● Limitations
1. Sampling
● Generalizability
● Representative
2. Measurement
● Measurement Error
● Social Desirability
3. Problem Identification
● Grasping the “Whole” Problem
54. Literature Reviews
►Well-written analytical narrative that
brings a reader up-to-date on what
is known on a given topic, but also
provide fresh insights that advance
knowledge
● Resolve conflicts between studies
● Identify new ways to interpret
research results
● Creating a path for future research
56. Review of Key Elements of
Previous Definition
• The LR is a summary of research:
• It is not a “list” of found research
but a coherent and articulate
account of past and current
research findings
• Suggestion: read 2 or 3 LRs in order
to become familiar with summary
styles
57. Review of Key Elements of
Previous Definition (cont’d)
• The sources typically are journal articles, books
and other documents that describe past and
present status of research in a given field:
• The LR should be exhaustive and as
current as possible.
• How many articles?
• There is no set number. As long as the search
is exhaustive and focused on the research
topic, the review will be acceptable.
58. Review of Key Elements of
Previous Definition (cont’d)
• How far back should one search?
• A reasonable and widely accepted
timeframe includes research conducted
during the past 10 years. Important
studies (i.e., studies that had a
significant impact on the field of study)
should also be mentioned even if these
go beyond the mentioned timeframe.
59. Review of Key Elements of
Previous Definition (cont’d)
• The LR should be organized:
• The review should not only be coherent, but
should organize the studies reviewed under
themes or topics.
• The reviewer is a guide and should be able
to provide readers with an in-depth and
current status of research in a given area.
• This aspect is essential for readers to
understand what the reviewer found during
the search.
60. Review of Key Elements of
Previous Definition (cont’d)
• The LR should document the need for a
proposed study:
• Studies should not duplicate research that
has been already done.
• Even in cases when research is duplicated
(replicated is the appropriate term), one is
responsible for documenting the need for
replication, e.g., need to explore the same
methodology with a different group or
population, or need to change methodology
with the same group.
61. Creswell’s 5 steps to Conduct a
Literature Review
• Step 1: Identify Key Terms or
“Descriptors”
• Extract key words from your
title (remember, you may decide
to change the title later)
• Use some of the words other
authors reported in the
literature
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
62. Step 1: Identify Key Terms or
“Descriptors” (cont’d)
• Use the “Thesaurus of ERIC
Descriptors to look for terms
that match your topic: go to
www.eric.ed.gov and in “Search”
select “Descriptors (from
Thesaurus)”
• Scan both electronic and
library journals from the past
10 years and look for key
terms in the articlesCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
63. Creswell’s 5 steps to Conduct a
Literature Review (cont’d)
• Step 2: Locate Literature
• Use academic libraries, do not limit
your search to an electronic search
of articles
• Use primary and secondary sources.
A “primary source” is research
reported by the researcher that
conducted the study. A “secondary
source” is research that
summarizes or reports findings that
come from primary sources
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
64. Step 2: Locate Literature
(cont’d)
• It is “best to report mostly
primary sources” (p. 82)
• Search different types of
literature: summaries,
encyclopedias, dictionaries and
glossaries of terms,
handbooks, statistical indexes,
reviews and syntheses, books,
journals, indexed publications,
electronic sources, abstract
series, and databases
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
65. Creswell’s 5 steps to Conduct a
Literature Review (cont’d)
• Step 3: Critically Evaluate and
Select Literature
• Rely on journal articles
published in national journals
• Prioritize your search: first
look for refereed journal
articles, then, non-refereed
articles, then books, then
conference papers,
dissertations and theses and
then papers posted to websites
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
66. Step 3: Critically Evaluate and
Select Literature (cont’d)
• Look for research articles and
avoid as much as possible
“opinion” pieces
• Blend qualitative and
quantitative research in your
review
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
67. Creswell’s 5 steps to Conduct a
Literature Review (cont’d)
• Step 4: Organize the Literature
• Create a “file” or “abstract” system
to keep track of what you read.
Each article you read should be
summarized in one page containing
Title (use APA to type the title so that
you can later copy-paste this into the
References section of your paper)
Source: journal article, book, glossary,
etc.
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
68. Step 4: Organize the Literature
(cont’d)
Research problem: one or two lines will suffice
Research Questions or Hypotheses
Data collection procedure (a description of
sample characteristics can be very handy as
well)
Results or findings of the study
• Sort these abstracts into groups of related
topics or areas which can then become the
different sections of your review
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
69. Creswell’s 5 steps to Conduct a
Literature Review (cont’d)
• Step 5: Write a Literature Review
• Types of Reviews:
Thematic Review: a theme is identified and
studies found under this theme are described.
Major ideas and findings are reported rather
than details.
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
70. Step 5: Write a Literature
Review
(cont’d)
Study-by-study Review: a detailed summary of
each study under a broad theme is provided.
Link summaries (or abstracts) using
transitional sentences. Must be organized and
flow coherently under various subheadings.
Avoid string quotations (i.e., lengthy chunks of
text directly quoted from a source)
Creswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative andCreswell, J.W. (2005) Educational Research: Planning, Conducting, and Evaluating Quantitative and
Qualitative ResearchQualitative Research
71. Preliminary Literature
Review
►This succinct review of current
literature should:
● Provide further contextual
background
● Reveal issues related to your study
● Describe similar problems in other
organizations
● Provide significance to your
approach to the study
72. Guidelines on Style, Mechanics, and
Language Usage
►Does your draft follow the logic or
idea that is presented in your intro
and title?
►Avoid overusing direct quotations,
especially long ones
►Check style manual for correct use
of citations
● (Doe, 2005); Doe (2005); (Doe & Smith,
2005); Doe and Smith (2005); (Black,
2005; Brown, 2006; Yellow, 2007)
73. Guidelines on Style, Mechanics,
and Language Usage
►Avoid using synonyms for recurring
words
● This is not creative writing and stay
consistent with terminology
Group I, Phoenix Cohort, Experimental Group
►Spell out all acronyms when first using
them
● Traditional - American Psychological
Association (APA)
● Non-traditional - Collective Efficacy (CE)
►Yes - Do NOT use contractions; No –
Don’t use contractions
►Coined terms should be set off by quotes
74. Guidelines on Style, Mechanics,
and Language Usage
►Avoid the following:
● Slang – “cool”
● Colloquialisms – “thing” >> “item” or
“feature”
● Idioms – “rise to the pinnacle” >> “to
become prominent”
►Use great care to avoid Plagiarism
75. What needs to be included in
the Literature review.
►Provides contextual background
►Reveals related issues
►Reviews similar problems elsewhere
►Provides significance to your approach
to the study
►Includes major/seminar research articles
pertaining to study
►Written in an integrated manner
►Uses peer-reviewed research
►Includes a Reference section
76. Writing Your Research
Question(s)
►Reflect the problem that the researcher
wants to investigate
►Can be formulated based on theories,
past research, previous experience, or the
practical need to make data-driven
decisions in a work environment
77. Writing Your Research
Question(s) (cont’d)
►Are vitally important because they, in
large part, dictate what type of statistical
analysis is needed, as well as what type of
research design may be employed
►A research question should address only
1 concept
►Question must be measurable
78. Types of Questions Asked
►Once you have identified the topic of study, you
will need to consider the type of question you
want answered and how it will be answered
►Two paradigms
● Quantitative Paradigm
Generally attempt to quantify variables
of interest. Questions frequently
address “how well or how much.”
79. Types of Questions Asked
● Qualitative Paradigm
“there are times when we wish to know
not how many or how well, but simply
how.” (Shulman, 1988, pg. 7)
80. Class Exercise
►Now you’re ready to formulate your own
research question(s)
►Sample questions:
● Is there a relationship between
participation in an Elluminate chat
session and course grade?
● How do 5th
grade students
experience the anticipation of
standardized testing?
81. Research Questions
► From Topic to Research Question
A good research topic asks a clear, concise question.
Asking a research question helps you keep a tight
focus on your topic.
► Tweaking Your Research Question
A good research topic is broad enough to allow you
to find plenty of material, but narrow enough to fit
within the size and time constraints of your paper.
● If your topic is either too broad or too narrow, consider
adding or eliminating the following elements:
Time Period, century, decade, future, Population
Type, age, gender, nationality, species, Geographic
Location country, state, region, Point of View
economic, social, cultural, biological
82. Assignment 2 Components
(see syllabus for details)
►Title Page
►Nature of the Problem
►Background and Significance of the
Problem
►Literature Review
►Research Questions
►References
83.
84. Topic ideas
►Online chat reference
● Types of questions
Subject? Type?
# of turnaways*
● Difference in discourse
In-person vs. chat
● Partnership studies
Similar libraries with same software
85. Topic Ideas
►E-book usage
►Usability studies of
● Online tutorial(s)
● ‘My Library” portals
►Analysis of library web sites or
library instruction sites or
pathfinders by best practices
►Student learning outcomes in LI
programs
86. Types of methodologies
►QuaLitative Measures
● Descriptive
● Numbers not the primary focus
● Interpretive, ethnographic,
naturalistic
►QuaNtitative Measures
● N for numbers
● Statistical
● Quantifiable
87. QuaLitative measures
►Content Analysis
●Analyzed course syllabi of
library use through discipline
and level (Rambler)
●Studied online tutorials,
applying best practices
recommendations (Tancheva)
88. QuaLitative Measures
►Discourse Analysis
● Analyzed student responses in writing
and discussions to a short film &
compared findings to parallel study
with LIS grad Ss (Vandergrift)
►Focus Groups
● Discussed how participants
experience & use the library (Von
Seggern & Young)
● Studied why students use the
Internet and how much time they use
it (Wilson)
89. QuaLitative Measures
►Interviews
● Studied 25 HS students’ web use for
research assignments (Lorenzen)
● Looked at what type of information first
year students need and how they go about
acquiring it (Seamans)
►Observation (obtrusive)
● Observed students as they conducted
online research & noted their activities
(Dunn)
►Observation (Unobtrusive)
● Retrieval of discarded cheat sheets to
analyze academic misconduct (Pullen et. al.)
90. QuaLitative Measures
►Think Aloud Protocols
● Studied how users navigate a library
web site (Cockrell & Jayne)
►Usability testing
● Examined students’ mental models
of online tutorials (Veldof & Beavers)
92. QuaNtitative measures
►Comparison studies
● Experimental and control groups
● Instructional methodologies (Colaric;
Cudiner & Harmon)
● Program assessment using before/after
analysis of research papers(Emmons &
Martin)
95. Ready Made Data Sets
►National Survey of Student
Engagement (Whitmire)
►College Student Experiences
Questionnaire (Kuh and Gonyea)
►The Web
● Internet Archive (Ryan, Field &
Olfman)
● Electronic journals (Dellavalle)
►Library server logs
97. Research Problem #1
A study assessing student learning
outcomes in 2 broad categories
(concepts, techniques) by examining
student research journals in 1 section
of an elective information literacy
course in fall semester.
98. Research Problem #4
A 2004 article on a library use and
services satisfaction study that used
as its measurement tool a survey
given to every nth person entering
the library building on 40 randomly
selected days throughout the school
year.
99. Research Problem #5
An outcomes assessment research
project of a 5 year old IL program in
which all incoming freshmen must
participate. Total student population on
campus is divided between 32%
freshmen to senior (or 4 year) and
68% transfer students.
101. Research Problem #2
An experimental study that proposes
a fund allocation formula for academic
library collections based on the
following:
average of overall book price + average of
overall serial prices * degree level (10 for
undergraduate to 30 for doctorate) / the number of
students enrolled in degree program as majors +
the total number of faculty in the department * three
* total number of students in program.
(OAB + OAS) * D/(Sn +(Fn*3))*Sn
N.B. Not a standard formula
102. Research Problem #3
A newspaper article you read just the
other day stated that in a recently
published study done at a major U.S.
university, researchers found that
domestic violence affects 1 in every 4
women.
103. Research Problem #4
A 2004 article on a library use and
services satisfaction study that used
as its measurement tool a survey
given to every nth person entering the
library building on 40 randomly
selected days throughout the school
year.
104. Research Problem #5
Over a one year period, researchers
studied the occurrence of turn-aways
in a virtual reference service and noted
that the significantly high occurrence of
turn-aways indicates increased need
for virtual reference service.
106. Research Problem #1
A study assessing student learning
outcomes in 2 broad categories
(concepts, techniques) by examining
student research journals in 1 section
of an elective information literacy
course in fall semester.
107. Research Problem #7
A survey of faculty found that the
majority of those interviewed interacted
most with librarians at the reference
desk. The researchers concluded that
most faculty view librarians in a
servile role.
108. Keep In Mind That
►No study is perfect
►“All data is dirty is some way or
another; research is what you do
with that dirty data” (Manuel)
►Measurement involves making
choices
109. Be Critical About Numbers
(Best 2001)
►“Every statistic is a way of summarizing
complex information into relatively
simple numbers.” (Best)
►How did the researchers arrive at these
numbers?
►Who produced the numbers and what is
their bias?
►How can key terms be defined & in how
many different ways?
110. Be Critical About Numbers
►How was the choice for the
measurement made?
►What type of sample was gathered
& how does that affect result?
►Is the statistical result interpreted
correctly?
►If comparisons are made, are they
appropriate?
►Are there competing statistics?
111. Getting Started
► Read to learn; read to analyze
● About research methodology
● Studies on similar topics
● Interesting studies
● Non-library studies
112. Getting Started
►Finding a topic needn’t be
traumatic
● Work projects Research studies
P&T overhaul
Library GO Bond Proposal Project
Library workshop trends
User repair strategies
113. Getting Started
►Data collection involves agreement
& consent
►Forge partnerships
►At some point you will need to
leave the comfort zone of reading
and literature gathering and …
118. Quantitative Methods
►A definition
● A survey or experiment that
provides as output a quantitative or
numeric description of some
fraction of the population, called
the sample.
119. Components of a survey
method
►The survey design
►The population and
sample
►The instrumentation
►Variables in the study
►Data analysis
120. The survey design
►Purpose of the survey
►The research question
►Type of survey
● Cross sectional
● Longitudinal
►Form of data collection
121. The population and sample
►Description of the population
►Sampling design
● Single stage
● Multistage
● Stratified
►Sample selection
122. The instrumentation
►The instrument (tool)
● Existing
● New
►Rating scale
● Likert scale: Rating the Items. 1-to-5 rating scale where:
1. = strongly unfavorable to the concept
2. = somewhat unfavorable to the concept
3. = undecided
4. = somewhat favorable to the concept
5. = strongly favorable to the concept
►Pilot
►Administration
● Postal survey
● email
123. Variables and analysis
►The research question
►Variable in the research
● E.g. Number of years of academic study
►The questions in the instrument
● E.g. How many years of study in a
University
As an undergraduate?
As a postgraduate?
►Data analysis
● Steps
● Bias in the data
Non-response
● Statistics, e.g. mean, standard deviation etc.
126. Randomized Controlled Trial
(RCT)
A true experiment, in which the researcher
randomly assigns some patients to at least
one maneuver (treatment) and other patients
to a placebo, or usual treatment. Key features
= the classic way to evaluate effectiveness of
drugs (or exercise, diet, counseling). Patients
are followed over time (Prospective). If
properly done, an RCT can be used to
determine cause and effect
128. The experimental design
►Type
● Pre-experimental
No control group
● Quasi-experimental
Control group, but not randomly
assigned
● Single subject design (over time)
● Pure experiment
● Repeated measures
Change groups
129. • Historical routes in anthropology
• Generates new understanding by naming and framing
concepts and themes
• Removes bias by questioning preconceived assumptions of
the social group under study
• Promotes neutrality through adoption by the researcher of
naïve stance or critical discussion, challenges pre-conceived
assumptions of both the researcher and the social group under
study
• Produces new understanding about the world, changes the
way power, culture and social interaction are understood
Overview of Qualitative Research
Design
130. •Observation (Videoed, non-participant, semi-
participant and participant observation, field notes)
•Interviews (individual and group - known as focus
groups, tape recorded and transcribed, field notes)
•Secondary data analysis (using written material
collected for purposes other than research)
•Questionnaires (unstructured, postal, interviews)
•A mixture of all four
Data Collection in Qualitative
Research
131. In qualitative research questions are open-ended.
Sometimes a check list or topic guide will be used
by the researcher to ensure all the relevant areas
are covered. This is known as semi-structured
data collection. It is used in all four methods of
data collection
Sometimes the only guide is the topic itself and
the researcher collects verbatim or naturally
occurring data. This is known as unstructured
data collection. It is used in all four methods of
data collection
Questions in Qualitative Research
132. The sampling method of choice is theoretical
sampling (queuing behaviour)
However, often this is not possible and people
resort to convenience sampling (students) and
snowball sampling (mental health in black and
ethnic minority communities)
Neither of the latter two methods are considered
strong but maybe all that can be achieved.
Research must be viable.
Sampling in Qualitative Research
133. •Read and re-read data, become engrossed in it.
•Identify themes: common, conflicting, minority
•Test themes across the data set, where are they
common, under what circumstances are they found, not
found. This sets the parameters on the interpretation and
generalisation of data
•Get more than one person to analyse the data
independently then together
•Demonstrate trustworthiness in data analysis
•Examples
•Biographical continuity
•Nursing routines as a method of managing a transient
workforce
Data Analysis in Qualitative
Research
135. References
►MSc project web pages
http://www.comp.glam.ac.uk/gis/start.asp?
whatfile=gis/gisrc/msc-proj.htm
►Creswell, J. W. (1994) Research
design : qualitative and quantitative
approaches. - Thousand Oaks,
Calif.; London : Sage Publications,
ISBN 0803952546
Hinweis der Redaktion
Welcome to Research Methodology 101. Today we would like to cover some basics concepts related to library research and also touch on a few practical matters such as putting research into practice into your library, your job.
I believe that we all can agree—no matter what type of library we work in or what type of position we hold within that library—that accountability and assessment are two governing themes of the day. We are being asked to assess and be accountable for—our services, our programs—especially our instruction programs—our collections. At the same time, ACRL IS has recently revised the Research Agenda for Library Instruction and Information Literacy. This research agenda can serve to help guide instruction librarians in assessing what they’re doing and how well they’re doing it, and how students are and are not learning.
In any sort of assessment, one does research. And research need not be a scary thing. A research project can be a lot of fun.
Note to self use timer for activity. 2?3? minutes after reading the slide.
Good. I can see that you all have some questions about these methodologies and that now you are all warmed up for this session.
Let me give you our session overview. Right now we’re going to map out some research basics and then quickly list a few topic ideas that could work into a research project within your library. After that we’ll talk a bit about the technical stuff—the research methodologies themselves. We’ll give you definitions and examples of studies that have used these methodologies. Most, if not all, of these are mentioned in the bibliography. Next. we will discuss common pitfalls found in any research study. Then we want to talk about how you can get started on your research agenda, how you can put it all together in your library setting.
Hey there, does this list of steps sound familiar with all you instruction folk out there? Or any of you who have conducted a class session on research basics? Yes! It’s the same basic plan. It was when you were writing your first research paper in high school and college and it’s the same plan now.
Finding the right topic can seem like a daunting task but we’ll show you some ways to make that step easier. After that you need to figure out just what your research focus really is, and that’s often done in the form of a question. Next, or even simultaneously, you should define your population of study. Students? Faculty? Users in your library? Which users? On to the next step of deciding your research design as well as deciding on your research instrument. You might ask yourself, “Am I going to conduct a survey? Via the web? E-mail? In person? Mail in? Will I interview people? Will I use a published measurement or scale? Will I do a pre and post test study?” Next you need to put your research plan into action by gathering your data set. Maybe you are collecting transaction logs from your web site or from your catalog or maybe you are doing classroom research so you are collecting data from your students over many semesters to do a learning outcomes assessment study. Next, you need to interpret what you have found. This step takes a little time and more than a lot of thought. Finally, you should write up your findings. Think of it as telling a story about what you did and what you found out. Simple? No? Fun? Sometimes~
Long term rewards? Priceless!
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
<number>
Since grade school many of us are familiar with the 5 W’s as a way to frame topics, situations, studies. So here’s another way to look at a research project or to figure out the boundaries of your study.
What exactly are you studying? Defining your problem or your focus is extremely important. For instance, I want to study how faculty value library instruction. I want to look at their attitudes towards not only library instruction but also towards librarians and libraries. But who exactly am I going to study? All faculty? Maybe only faculty who really value library instruction by their using our LI program over several semesters. What time period are you looking at? Most importantly, Why—what’s your point? What could someone generalize from your findings? But what does it mean to the larger community? What can we all learn from this?
Finally, how are you going to put this project, this study in place?
Let’s take a look at a variety of research topics that could easily turn into a research project. Online chat reference or virtual reference is a good example. Maybe you are interested in looking at the nature of questions you’re getting via your new service and you want to examine these by subject or by type or perhaps by transaction length. Maybe you are interested in the number of turnaways to the service because it’s busy or not staffed.
Perhaps you want to analyze the differences and similarities in discourse between in-person reference and chat or virtual reference. And when I say discourse I mean the structure and the nature of the conversation or the communication exchange between the user and the librarian.
Let’s say that one or more libraries similar to yours are all using the same virtual reference package—question point, LSSI, etc. and you want to construct a study that encompasses all those libraries VRef services.
Here’s some more opportunities for studies. Let’s say you have e-book collections from NetLibrary or Ebrary and wish to look at usage of those collections. Or you have a rough draft of one or more online tutorials or a My Library portal that you would like to test out. Let’s say that you want to apply best practices characteristics to a set of library instruction sites or library web sites or online pathfinders. Perhaps you are interested in measuring student learning outcomes in your library instruction program.
A lot of these types of projecta occur normally in libraries. It’s part of what we do. We implement and maintain services and then we want to see how those services are doing. So we measure them.
So you see, almost everything you are doing in lib in job could become the basis for a study. It’s worth emphasizing the synergy and efficiency to make your work project into a research project.
When we talk about methodologies within the context of library research, we would like to divide these measures into two different areas: Qualitative and Quantitative.
Briefly Qualitative measures is more descriptive in nature, examining in detail people and things. While numbers are not the primary focus of a qualitative study, you will find statistics within these studies. Essentially qualitative research is more interpretive and ethnographic. An ethnography describes and inteprets characteristics or behaviors of a culture or a group.
On the other hand, quantitative measures are numerically based. They include statistics and statistical measures. Quantitative measures compare, assess and quantify. These types of studies are those that frequently use those mathematical symbols and have their own language with words and phrases such as analysis of variance and statistically significant and central tendency and regression and correlation and chi square. Typically quantitative analyses or studies look at the big picture whereas qualitative studies examine groups or things in detail. Think of one as the telescope and the other as the microscope.
To give you some grounding and some encouragement in launching your own research project, let’s look as some examples of both types of studies. We’ll start with qualitative studies. According to Thomas Schwandt ”Content analysis is a generic term for many different types of textual analysis & it involves first coding then comparing, constrasting and organizing a body of data to test hypotheses”. Linda Rambler conducted a content analysis of syllabi at Penn State to determine first whether courses included library use and if so, which type of library use.
Another example of content analysis is found in Kornelia Tancheva’s study of online tutorials based on recommendations of “best practices”. By the way, Kornelia’s study builds on Nancy Dewald’s study of online tutorials. So you see you can take someone else’s study and focus yours in a different direction, adding to the body of knowledge about that area.
Another frequently used qualitative measure is discourse analysis. This is a method to analyze the process of communication. And that communication can be from a variety of sources, be it between people, found within a text, broadcast on the radio, or television, or within a film. It is concerned with the “how” of communication.Kay Vandergrift’s study in School Library Media Quaterly compares junior high school students responses to a short film, The Shooting Gallery, that is a parable on totalitarianism, with repsonses of graduate students to the same film. It’s a discourse analysis in that Vandergrift is looking at the words and phases and imagery students from both groups use to describe & respond to the film.
On the other hand, the focus group methodology comes to us from the field Marketing and is used to quickly gather ideas, views and attitudes from a group of people. It is a method noted for synergy in stimulating discussion and bouncing ideas off of many at once. Marilyn Von Seggern and Nancy Young used this method to examine how their users viewed the rapidly changing world of information and what impact those changes had on their searching for information. In another study, Rebecca Wilson used focus groups of a total of 75 students and looked at why students use the Internet and how much time they spend on the Internet.
Here’s a couple of other examples of qualitative measures. Interviews are typically conducted as a way to discover attitudes, beliefs and values. Typically interview sessions are informal where the interviewer asks open-ended questions to allow for a high degree of flexibility and responsiveness from the interviewee. Michael Lorenzen interviewed students to figure out how they used the internet in doing their research assignments and to what extent they used the Internet—more, less or the same as other information sources.
Observation has been used for a very long time as a qualitative methodology. Obtrusive studies are when the participants know that they are being observed. Unobtrusive studies are conducted without the participants knowledge that they are being observed. Kathleen Dunn used obtrusive observation—among many other types of methodologies-- in her assessment study of information literacy skills within the Cal State University System. Robert Pullen and his team took a novel approach to an unobtrusive study by picking up wadded cheat sheets from the floor or retrieving them from waste paper baskets to study how cheat sheets were used.
Think Aloud testing came about in the early 80s as a way to understand the thought processes that go into writing by asking the writers to vocalize their thoughts about what they are doing and why they are doing it as they compose. Think aloud tests have been used in library research and have involved asking a person to complete a task (like searching a database, or navigating a web page) at the same time as they vocalize their thoughts about what they are doing and their decision making processes. It’s a good way to do an early test of a prototype and uncover any misconceptions or confusions but it’s also a bit of an unnatural act for many participants. Also, the act of vocalization slows down the thought processes. Barbara Cockerell and Elaine Jayne used the TA method when they did a usability test of their library web site.
Usability tests, on the other hand, had more practical origins. These very popular tests came from product engineering as a way to see if a product does what it’s supposed to do, whether folks can figure out how to use something.
Jerilyn Veldof and Karen Beavers conducted a usabilty study to figure out what mental models students use in navigating online library tutorials.
Now let’s switch our focus to Quantitative studies. Think N for numbers, here.
Essentially, these studies, these methodologies take the big or bigger picture approach. They compare things or people, showing the differences and similarities, they count things and they survey things.
Comparison studies are often experimental in nature, using one group with no or tradition treatment and the other with the experimental treatment. Comparison studies often look at the outcomes of different types of teaching methods. Susan Colaric examined three different methods (teaching by example, instruction with conceptual models and instruction without illustrations) to figure other how students learned specific searching concepts.
Mark Emmons and Wanda Martin compared student portfolios from before and after the implementation of new course-integrated library instruction component.
Often when making these comparisons, researchers measure peoples knowledge or attitudes before and after a “treatment. Sometimes using established measures and scales as a pre and post testing instrument. For example, Anna Van Scoyoc, used Bostick’s Library Anxiety Scale as a pre and post tests to determine whether computer based or traditional library instruction reduced library anxiety.
Judit Bar-Ilian examined the number and nature of other web pages linking to Cybermetrics home page; whereas Robert Dellavalle and other analyzed the frequency, format and activity of internet reference from three science journals.
Keep in mind that you don’t always have to gather your own data; you can use pre-existing sets.
The web provides an excellent spot for data mining. Ryan, Field and Olfman used the Internet Archive to study how state government web pages evolved over time.
You might want to mine your own server logs. Nancy Turner and I looked at our catalog transaction logs to analyzed the types of repair strategies users employed when searching.
You should try to avoid some typical problems that befall researchers. One of these is found in population. First of all is your sample representative of what you are trying to study? How did you arrive at your sample? Did you not exclude those that need to be included or did you include those that shouldn’t be included. Let’s look at some of those research problems we looked at in the beginning of this session.
Note that this is an elective course so students have self-selected. They not only want to be in the class, they are probably pretty good at the topic already.
Also this study is looking at just one section of the course in one semester.
I am sure you all caught the problem of surveying only those who enter the building as opposed to survey those that use the library virtually.The random sampling of days could well skew the study. Perhaps busiest days are missed entirely. Random and representative are not always compatible.
Here it’s fairly obvious that the majority of those being assessed have not received the treatment.
It’s important to thoroughly define what you are measuring and how you are measuring it otherwise you may run into some problems.
READ SLIDE
(Note to self: Poorly defined (formula); Problems in measurement in that the average book and serial prices can hurt some areas and that it makes assumptions about the scholarly communication formats that may not be true. Also, if this is an elderly formula—as many of our funding formulas are then it’s a good time to take a look at whether this formula still fits and if not, why. Beware of unchanging measures)
READ SLIDE
(Note to self: Definition of “affect” of “domestic violence”!!!
Perhaps this ratio is based on Guessing???
How did the researchers conduct this study? Survey? Sample? Representative? How were questions formed? How about this
Q: “Have you been affected by domestic violence in your lifetime?”
A: “Oh yes, I watch the Lifetime channel for women and I have seen a lot of movies on battered women. I remember that one with Farrah Fawcett—she’s a Corpus Christi native, ya know, where she burned her husband in his bed? And then she was tried for murder? But I think she got off because he beat her so bad?)
READ SLIDE
(Note to self: Definition of group? A lot of false negatives here. Folks that ought to be included but aren’t because they don’t physically enter the library. Perhaps this campus has a large distance ed program or that gate count stats are declining whereas external web activity has skyrocketed. Perhaps the library has a doc/item delivery program so heavy users of the library’s services rarely, if ever, come in the building?
Random sample? How were the days selected? Hours? Sometimes Random is given a lot more attention than it deserves. Skewness? Could end up missing heavy use days of year.)
READ SLIDE
(Note to self: Definition problems. Significantly high compared to what? Is one significant? How about in person turnaways? Phone turnaways? E-mail?
Software problem? Networking problems?
Benchmarks please?
Hasty conclusions)
Before we get started, I’d like to briefly go over your handout. After the title page with the funny little man who has lost all hairs but one to a detestable research project, is a short activity which we’ll dive into in a moment. Following that is an annotated bibliography of many research studies. We’ve noted the gist of each study, type of instrument used, groups studied, setting, etc. to help you along. We will refer to many of these studies throughout the course of this presentation. At the end of the handout is an index of specific research methodologies, such as pre and post test, usability studies, etc. Keep in mind that this bibliogrpahy is not exhaustive by any means. It’s biased towards library instruction and more toward qualitative than quantitative studies—although both are included.
As a way of setting the tone for today’s session, please turn to the inside cover of your handout and look at the 7 descriptions of research methodologies in these studies. I’ll quickly show each of these on the screen, reading them aloud. Then what I would like you to do for the next several minutes is note any questions you have about these methologies, any problems or flaws that you note. Feel free to talk amongst yourselves. Compare notes.
READ SLIDE
(Note to self: self-selection, generalizability, sampling)
READ SLIDE
(Note to self: When did the reference desk ever mean servility? This is a good example of the problem of Transformation where researchers and/or those interpreting the data make a quantum leap in conclusions.
Similar to that children’s rhyme: Drip is a drop, drop is nature, nature is beautiful, thanks for the compliment. Or the six degrees of separation motif.