obbs reflects on her use of quantitative research methods for media literacy education research through the examination of program evaluation, hypothesis testing and scale construction.

1. A Reflection on
Quantitative Approaches to Measurement
Renee Hobbs
Media Education Lab
University of Rhode Island USA
Catholic University of Louvain
Groupe de Researche en Mediation des Savoirs (GREMS)
September 14, 2016

2. PEER-TO-PEER FILE SHARING
• CONTEXT. Media & technology are immersive cultural environments; media
institutions offer ideologies & structures that shape human perception & values
(McLuhan)
• INTERPRETATION. Audiences are active; meaning-making is variable based on
lived experience & social context (Hall)
• EMPOWERMENT & PROTECTION. Creating media and critical analyzing media
texts & institutions is a pedagogical practice (Masterman)
• SOCIAL ACTIVISM. Awareness, analysis, and reflection of media & popular culture
enable people to take action to make society more just and equitable (Freire)
• LITERACY & LEARNING. Institutions of education, communication practices &
democratic values are interconnected (Dewey)
MY THEORETICAL FRAMES

3. Hobbs, R. (2016). Exploring the roots of digital and media literacy through personal narrative.
Philadelphia: Temple University Press.

4. MAIN IDEAS FOR TODAY
Partnership Matters: The shape of media literacy research
should be determined by intersecting interests of researcher,
practitioner and institution/funder
Transdisciplinarity Matters: Media literacy researchers
should collaborate across disciplines in order to benefit in
unexpected ways from exposure to diverse paradigms
Value Matters: In developing measures, research should aim
to capture the complexity of the practice of media literacy
education

5. • Student Reporting Labs
• ML and Nonverbal Bias
• Teacher Motivations
Program Evaluation
Hypothesis Testing
Scale Construction

6. Media
Literacy
Programs
Researcher
Driven
Practitioner
Driven
Funder or
Institution Driven

7. www.studentreportinglabs.com

9. Program Assessment uses formative and summative research to guide
program developers in planning, reflection, action and evaluation. Key
characteristics include:
• Participants collaborate at every stage
• Intended to result in some action, change or improvement
Methods
• Pre-post online survey
• Interviews with teachers and mentors
• Classroom observations
• Analysis of student work samples
PEER-TO-PEER FILE SHARING
Research Design

10. 38 participating
schools

11. N = 566 HS students
Race
13% African-American
30% Caucasian
36% Hispanic/Latino
Age
Range 13 – 18
40% age 17
Gender
60% male
Class
35% low SES
Sample

12. PRODUCTION SKILLS
Non-technical skills
Technical skills
IN-CLASSROOM
ACTIVITIES
Gathering &
Synthesizing Info
Using Digital Media
Revision
MEDIA LITERACY
CIVIC
ENGAGEMENT
Sign an online
petition
Express an
opinion to news
media
Blog about an
issue
Write an opinion
letter
Gathering and Synthesizing Information
Percentage
Analyzed videos 70%
Created a story board 65%
Conducted interviews 68%
Fact-checked information 49%
Pitched a news story 54%
Discussed different points of view about
social and political issues 53%
Using Digital Media to Communicate Ideas
Used a video camera to record visuals 69%
Wrote a script 64%
Performed in front of the camera 65%
Worked behind the scenes/different roles 64%
Logged footage 43%
Edited visuals and sounds 68%
Used images/sounds to tell a story 54%
Posted videos online 38%
Engaging in Cycles of Revision & Feedback
Edited reports in response to feedback 38%
Not all students
participated in all
production activities

13. PRODUCTION SKILLS
MEDIA LITERACY
CIVIC
ENGAGEMENT
Sign an online
petition
Express an
opinion to news
media
Blog about an
issue
Write an opinion
letter
TIME 1 TIME 2 CHANGE
Communication Skills
I am confident interviewing a stranger.
2.55 2.73 +.18*
I know how to conduct interviews to gather
information.
2.33 2.86 +53*
I can compare fact and opinion. 2.88 3.29 +.41*
I know how to develop a news story pitch. 2.04 2.59 +.55*
I use the Internet to gather reliable information. 3.10 3.18 +.08
I know how to present myself on camera. 2.61 2.97 +.36*
Technical Skills
I can easily set up a tripod and camera.
2.69 3.28 +.59*
I can fix audio and video quality. 2.24 2.91 +.67*
I can do substantial video editing. 2.16 2.90 +.74*
I can plan, direct and produce a video news
report.
2.16 2.64 +.48*
I have the skills I need to make a professional
news video.
2.10 2.68 +.58*
I can use a video camera to film news reports.
N = 283, * p.<.05
2.69 3.10 +.41*
Students demonstrate
statistically significant
growth in both
communication &
technical skills

14. ATTITUDES
Intellectual Curiosity
Giving & Receiving
Feedback
Confidence in
Self-Expression
Intellectual Curiosity TIME 1 TIME 2 CHANGE
I want to learn about all sides of a
given issue.
3.08 3.31 +.23*
I am curious about ways to solve
issues in my community.
2.67 2.97 +.30*
I often find myself questioning
things that I hear or read.
3.10 3.33 +.23*
Giving and Receiving Feedback TIME 1 TIME 2 CHANGE
I am open to constructive criticism. 3.00 3.36 +.36*
I show respect for people's ideas
and feelings, even when I disagree
with them.
3.44 3.53 +.09
I motivate others to do their best. 3.05 3.18 +.13*
Confidence in Self-Expression TIME 1 TIME 2 CHANGE
It is easy for me to express my
views and opinions.
2.99 3.28 +.29*
I have a clear idea of what values
are important to me.
3.25 3.49 +.24*
I am willing to express my opinion
even if I know it is unpopular.
3.10 3.32 +.22*
N = 281, * p > .05
Students demonstrate
statistically significant
growth over time

15. What factors are associated with
intellectual curiosity?
Predictor b SE T p
Constant 1.255 0.441 2.84
In-Class Activities -0.019 0.022 -0.878 NS
Technical Skills 0.047 0.02 2.309 0.02
Confidence Self-Exp 0.345 0.049 7.06 0.001
Comm Skills 0.05 0.029 1.739 0.08
Feedback 0.288 0.054 5.348 0.001
Regression Analysis on Intellectual Curiosity
Hobbs, R. (2016). When teens create the news: Examining the impact of PBS News Hour Student Reporting Labs. Journalism
Education 5(1), 61 – 73.

16. PRODUCTION SKILLS
Communication skills
Technical skills
Giving & Receiving
Feedback
Confidence in
Self Expression
IN-CLASSROOM
ACTIVITIES
Gathering &
Synthesizing Info
Using Digital Media
Revision
Intellectual Curiosity
Hobbs, R. (2016). When teens create the news: Examining the impact of PBS News Hour Student Reporting Labs. Journalism
Education 5(1), 61 – 73.

17. Developing and Validating Measures
 Meet with funder & participating teachers to
determine their priorities
 Observe program in classrooms & interview teachers
 Examine student media work samples
 Adapt existing relevant measures from prior research
 Pilot test measures with students in one high school
 Revise measures based on results
 Gather data from multiple schools
 Analyze to identify the best results for program
evaluation purposes
 Generate new questions for future research and
practice
HOW WE DID IT
Hobbs, R. (2016). When teens create the news: Examining the impact of PBS News Hour Student Reporting Labs. Journalism
Education 5(1), 61 – 73.

18. Program evaluation of media literacy initiatives can help
advance both measurement & theory-building

19. THEMES FOR TODAY
Partnership Matters: The shape of media literacy research
should be determined by the intersecting interests of
researcher, practitioner and institution/funder

20. Nonverbal Bias
The behavior of a friendly or hostile
television interviewer can affect
viewer impressions of the interview
subject
Halo effect: positive attitudes about
the subject associated with behavior of
the interviewer
Media bias effect: negative attitudes
about the subject associated with
behavior of the interviewer
Would media literacy HS students be
more or less susceptible to
nonverbal bias?

21. Media Analysis
A one-semester course open to students
in all grades taught by a veteran media
literacy teacher.
The course emphasized the key concepts
of media literacy, and students engaged
in the process of critically analyzing
news, advertising, and film through
critical questioning.
Students completed assignments in
which they analyzed the credibility of TV
news and Internet Websites and
examined patterns in the representation
of violence in the media.
Other topics discussed in the class:
• body image and the representation of
gender in mass media
• media coverage of historical events
• presidential ad campaigns
Students learned to analyze media by
applying concepts such as purpose,
audience, point of view, media languages,
subtext, representation, and technology to
diverse forms of visual and digital
messages from contemporary and popular
culture.
Subjects
N = 88
Age range 13 – 18
48 males, 40 females
50% African American
40% Caucasian
10% Other
Quasi-Experimental
Design

22. Babad, E., Peer, A., & Hobbs, R. (2012). Media literacy and media bias: Are media literacy students less susceptible to non-
verbal judgment biases? Psychology of Popular Media Culture 1(2), 97 – 107. DOI: 10.137/a0028181
Rate your perceptions of the politician who was interviewed…
NONVERBAL
PERCEPTION STUDY

23. Babad, E., Peer, A., & Hobbs, R. (2012). Media literacy and media bias: Are media literacy students less susceptible to non-
verbal judgment biases? Psychology of Popular Media Culture.1(2), 97 – 107. DOI: 10.137/a0028181
Media literacy
students are
immune to the
media bias
effect
NONVERBAL
PERCEPTION STUDY
Rate your perceptions of the politician who was interviewed…

24. Developing and Validating Measures
 Review prior work on nonverbal bias
effect
 Theorize why media literacy students
may be immune to media bias effect
 Recruit a school & teacher to
participate in the study
 Gather data
 Analyze and report results
 Generate new questions for future
research and practice
HOW WE DID IT
Babad, E., Peer, A., & Hobbs, R. (2012). Media literacy and media bias: Are media literacy students less susceptible to non-
verbal judgment biases? Psychology of Popular Media Culture 1(2), 97 – 107. DOI: 10.137/a0028181

25. www.mindovermedia.tv

26. www.mindovermedia.tv
Theory-driven hypothesis testing can inspire
Innovation in pedagogy and curriculum

27. THEMES FOR TODAY
Partnership Matters: The shape of media literacy research
should be determined by intersecting interests of researcher,
practitioner and institution/funder
Transdisciplinarity Matters: Media literacy researchers
should collaborate across disciplines in order to benefit in
unexpected ways from exposure to diverse paradigms

28. Attention to teacher motivations
can help educators, technology
specialists and school librarians
better collaborate with teachers
to support student learning

29. SKILLS & ABILITIES
➢ Knowledge about the Computer & Internet
➢ Digital Skills
LITERACY
➢ Online Reading & New Literacies
➢ Media Production / Youth Media
➢ Coding / Programming
TEACHING WITH
➢ Technology Integration
➢ Digital Learning
➢ Blended Learning
➢ Connected Learning
TEACHING ABOUT
➢ Information Literacy
➢ Media Literacy
➢ Internet Safety & Digital Citizenship
What Digital Learning Flavors have You Tasted?

30. What Teachers Want
Students to Know and Be Able To Do
Find, comprehend and
interpret content
Gain knowledge
and information
Examine the quality of
information resources
Share ideas through
dialogue & discussion
Create, build or
make something
Reflect on expected and
unanticipated consequences
Plan and implement action
to effect social change
Critically analyze how
messages are constructed
Use digital tools
for learning

31. Genre or Format
Content and Quality
Community Connection
Texts and Audiences
Understanding Media Systems
Learner-Centered Instruction
THEORETICAL FRAMES
USED TO CONSTRUCT THE
INSTRUMENT

32. www.quiz.discovermedialiteracy.com

33. What’s Your
Digital Learning
Horoscope?
@reneehobbs #caslceca2015

36. Hobbs, R. & Tuzel, S. (2015). Teacher motivations for digital and media literacy: An examination of Turkish educators. British Journal of
Educational Technology. DOI: 10.1111/bjet.12326

37. Hobbs, R. & Tuzel, S. (2015). Teacher motivations for digital and media literacy: An examination of Turkish educators. British Journal of
Educational Technology. DOI: 10.1111/bjet.12326

38. Developing and Validating Measures
 Align German & US media literacy theoretical
concepts to identify diverse teacher
motivations
 Write items to align with each of 6 constructs
 Use cognitive pretesting to revise items
 Collect and analyze data from convenience
samples of teachers in U.S. & Germany
 Revise & eliminate items based on results
 Rename and rebrand measure as “horoscope”
 Pilot test the use of measure as a tool for
teacher PD reflection
 Validate instrument with a large sample of
teachers in Turkey
 Generate new questions for future research
and practice
HOW WE DID IT
Hobbs, R. & Tuzel, S. (2015). Teacher motivations for digital and media literacy: An examination of Turkish educators. British Journal of
Educational Technology. DOI: 10.1111/bjet.12326

39. Sensitivity to teacher motivations may contribute
to the design of PD with greater impact

40. THEMES FOR TODAY
Partnership Matters: The shape of media literacy research
should be determined by intersecting interests of researcher,
practitioner and institution/funder
Transdisciplinarity Matters: Media literacy researchers
should collaborate across disciplines in order to benefit in
unexpected ways from exposure to diverse paradigms
Value Matters: In developing measures, research should aim
to capture the complexity of the practice of media literacy
education

41. SOME CAVEATS

42. ^
research is
SOME CAVEATS

43. Community of Scholars

44. www.mediaeducationlab.com

45. SAVE THE DATE!
Summer Institute in Digital Literacy
July 23 – 28, 2017
www.mediaeducationlab.com

46. CONTACT ME
Renee Hobbs
Professor, Department of Communication Studies
Director, Media Education Lab
Harrington School of Communication and Media
University of Rhode Island USA
EMAIL: hobbs@uri.edu
TWITTER: @reneehobbs
WEB: www.mediaeducationlab.com