Learning Assessment With Serious Games

Learning Assessment
with Serious Games

Collaboratory for Interactive Learning Research (C.I.L.R.)

Dept. of Curriculum & Instruction, COEHS
Southern Illinois University Carbondale
Carbondale, Illinois, U.S.A.

Performance-based Assessment

Monitoring of students' progress in relationship to
identified learner outcomes.
The Office of Technology Assessment of the U.S. Congress


defines performance assessment as “any form of testing
that requires a student to create an answer or a product


that demonstrates his or her knowledge or skills.”
This differs from traditional testing methods which
require a student to select a single correct answer or to fill
in the blank.


Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA

Characteristics of Effective
Performance Assessment
Students should be active participants, not passive
“selectors of the single right answer."
Intended outcomes should be clearly identified and should


guide the design of a performance task.
Students should be expected to demonstrate mastery of


those intended outcomes when responding to all facets of
the task.


Students must demonstrate their ability to apply their
knowledge and skills to reality-based situations and
scenarios.


A clear, logical set of performance-based activities that
students are expected to follow should be evident.
A clearly presented set of criteria should be available to


help judge the degree of proficiency in a student response.



Cognitive Learning Theory

Knowledge is constructed. Learning is a process of creating
personal meaning from new information and prior knowledge.
Learning has social components. Group work is valuable.


People perform better when they know the goal, see models,
and know how their performance compares to the standard.


There is great variety in learning styles, attention spans,


memory, developmental paces, and intelligences.
It's important to know when to use know when to use


knowledge, how to adapt it, how to manage one's own learning.
Motivation, effort, and self-esteem affect learning and


performance.
Learning isn't necessarily a linear progression of discrete skills.





Definitions of Assessments

Formal and informal ways of eliciting evidence of intended student
 Assessment
learning


Any assessment for which the first priority in its design and practice
 Assessment for Learning (AfL)
is to serve the purpose of promoting student learning


When the evidence of learning elicited by the assessment is actually
 Formative (use of) Assessment
used to adapt teaching to meet the learning needs



Casual Games

Video game or online game targeted at a mass audience
Any type of gameplay
Fit any genre.


Simple rules


Lack of commitment required -- very little time


commitment


Need no special skills to play


Comparatively low production and distribution costs for
the producer


Played on a personal computer online in web browsers





Casual games

Based on Ludology Examples:
school of game design  Sudoku
principles
 

 Tetris, Minesweeper
 Quick, fun, engaging  Card games
 Not story-based  Casino games


Serious Games

Games for Education, Instruction & Training
Serious Play (Rieber, 1996), Serious Games (Abt, 1970)
Instructors » teachers (public school) » business trainers


Learners » students » employees




Salesman of the Month A Force More Powerful


Virtual Worlds/Complex Games

If story-based, then it is based on the
Naratology school of game design principles
The story provides the context for


interaction
The story “grounds” the experience


The storyline engages the mind/personality
Player becomes the “hero” in the plot


Objects in the environment are interact-


able and usually for a reason (plot-related)




Current Focus for Game Design

Current focus (of the industry) is in game
design
What comes after design & development?


Implementation and then evaluation
Design and evaluation (assessment) is


intertwined


Assessment methodology usually stem from


design
Game designers need to think about


assessment as they design games (NOW)



Assessment of Learning
Technology + Learner  Learning?

Computers

Students/Trainees

(Serious)
Video Games

Problems of Assessment
Can we (teachers) know what they (learners) really
learned?
UNINTENDED CONTENTS? INTENDED CONTENTS?

“How do we get into the Black Box ?”


Game Studies / Game Research

European-based  Scientific research
GAME STUDIES GAME RESEARCH

(Netherland)  Problem/scenario-based
MIT


learning
(Language/linguistics)  Pioneered by military-based
Recommendation why we
training


should study games »
 Data-driven conclusions
Recommendation:
because it is a rising social


Why we should study games
phenomenon, and is To effect change in


interesting to society
 

 Learning


Phenomenological
 Education
Ethnographical
 Performance


Feministic, etc.




Getting into the Black Box

Qualitative Analysis

Video-taping the whole event
Short (1-2 sessions)
Medium (a few weeks)


Long (a few months… even


years)


We need to double/triple the


time for data analysis
Limited generalization


Case Studies


 Ethnography


 Phenomenology

Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA

Measuring Change

Quantitative Analysis

Educators’ method:
 Attitude surveys


 Measuring effects of new
technology against
traditional teaching method
 Pre-test» Intervention»Post-
test
 Randomized testing
Better Methods?
 Require “harder” statistics!



New Research Methods

Large data sets
DATA MINING RESEARCH FRAMEWORK

Conceptual Frameworks »
 Sampling at regular intervals Test Models


Design for learning/


 Tens of thousands/millions
data points
Data mining design for assessment
Learning framework


 Analyzing for patterns in


data  Design-based learning?
Assessment framework


 Clustering of data (amateur
vs. experts)  Criterion-Reference (for


 Associative data mining games?)
 Multimedia mining  Information Trails


In Situ Data Collection

Information Trails

Tracking a Moving Object [Vehicle]

Vehicle tracking is a way of monitoring the locations,
movements, status and behavior of a vehicle or fleet of
vehicles.


This is achieved through a combination of an electronic
tracking device fitted in the vehicle, a method of
returning the data to the user and PC or web-based


software.
The data is turned into information by management
What tools in conjunction with a visual display

on computerized mapping software.

reportingcan be learned?
Driver 1 on schedule, lunch break: 2 eggs, 24
chips, 7 mushrooms and fried bread
Driver 3 helped friend move house

Southern Illinois
• Carbondale, USA

Tracking Devices
INFORMATION RECEIVED FROM DATA RECEIVED = EVIDENCE OF
TRACKING DEVICE / SOFTWARE VIOLATION » NOTIFICATION


Characteristics of Tracker
The tracker is a visual moderation and logging software product,
which enables meeting participants to keep track of their meeting
goals and the necessary steps to achieving them, as well as allowing


contributions to be anchored within the meeting log immediately.
The tracker graphically depicts the course of the current meeting
and locates it within the overall context. Tangential departures from
the meeting objectives can immediately be seen.


The course of the meeting is structured efficiently and becomes
traceable. At the end of a meeting, the minutes are already completed
and can immediately be approved by all participants.


For the most important meeting situations (such as project kick-off
meeting, information meeting, decision meeting, crisis meeting,
problem analysis meeting, etc.), the tracker offers proven agenda


templates which will ensure an efficient meeting process.


Agent Tracking

Tracking is the process of locating a moving object (or
several ones) in time.


Tracking a Tropical Depression Tracking the Position of an Object via GPS

Tracking a Moving Object [GPS]

A tracking unit is a device that determine the precise
location of an animal, person, or other asset to which it is
attached and to record the position of the asset at


The recorded location data can be stored within the
regular intervals.

tracking unit, or it may be transmitted to a central location
data base, or internet-connected computer.


This allows the asset’s location to be displayed against a
map backdrop either in real-time or when analyzing the
track later, using customized software.


Ref: http://en.wikipedia.org/wiki/GPS_tracking


What Can Be Learned
Tracking a live moving agent (sea turtle) in its habitat via a
tracking device attached to the agent


•
Ref: http://www.bonairenature.com/turtles/turtletracking.html

Christian Sebastian Loh, Ph.D. Southern Illinois Carbondale, USA

Components of Tracking

(A) PROCESS (B) TIMELINE

Southern Illinois
• Carbondale, USA

Data Processing

 Tracking the movement of Sea Turtle: Process +
Timeline

Data Flow: Data Logger » Pusher » Puller » Store
Data Mining: Data Collection » Reduction »
Visualization



Southern Illinois
• Carbondale, USA

Foraging » Trails

foraging
Animals Food Source

Path Finding
(Trails)

start

• Pirolli, P., & Card, S. K. (1999). Information foraging. Psychological Review, 106(4), 643-675.
Information Foraging
• Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. (2001, March 31-April 4). Using information scent to model user
information needs and actions on the Web. Paper presented at the SIGCHI '01, Seattle, WA.


Serious Games & Assessment

(Serious)
Video Games
? (Verification
Assessment

of Learning)

Learning
Goal
Process of
Learning

start



(Serious)
Video Games
? (Verification
Assessment

of Learning)

Information Trails


Information Nodes

Conceptually: Series of planned event hooks that are
embedded within an information ecology.
Operationally: Series of planned event hooks waiting to be


trigger by players’ avatars as they traverse a virtual game
environment. Once an event is triggered, a marker


will be placed on the trail to mark the location,
time, and outcome of the event.


Information Trails

Conceptually: A series of agent-detectable markings left by
a moving agent within an information ecology.
Operationally, it is the long track of information markings


left behind by players’ avatars as they traverse a virtual
game environment (i.e. triggered event hooks)


Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale,
Southern Illinois Learning Research
• Southern Illinois University Carbondale
Christian Sebastian Loh, Ph.D. USA

Tracking an Avatar
Avatar: a representation of self in a game environment
(An extension of self that behaves just like a live being in a
habitat)


Can a tracking device be attached on to an avatar, so as to
track the moving agent?
(Just like in the case of a vehicle or a turtle?)


Data (locations, status, behavior) about the avatar can then
be transmitted out of the game system, and be received and
stored in an external data store for later analysis


(Data logger » Push » Pull » Store)
Data can then be represented graphically, where goals and
behaviors can be scored as assessment


Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale,
Southern Illinois Learning Research
• Southern Illinois University Carbondale
Christian Sebastian Loh, Ph.D. USA


(Serious)
Video Games (Verification
Assessment

of Learning)

Data Data Visualization
data (Data Display)
Collection
Data Reduction

Southern Illinois
• Carbondale, USA

Serious Games
(with embedded
Information Nodes)
(Verification
Assessment

of Learning)

Data Collection Data Visualization
(Information Trails) data (Data Display)
Data Reduction

Southern Illinois
• Carbondale, USA

Game Development

Game Developers’ Toolkit Game Mods
(Free, New Contents)

Commercial

(Serious)
Off-The-Shelf (COTS)

Video Games Data Store
Games

Persistent Worlds
(Free Multiplayer Games)

Southern Illinois
• Carbondale, USA

Embedding Information Trails

Game Developers’ Toolkit Game Mods

(Serious)
Commercial

Video Games

Data Store
Games

Embedding Information Trails
Serious Game Developers:

into the Game Development Process

Persistent Worlds

Southern Illinois
• Carbondale, USA

In Situ Data Collection
Assessment of Digital Game-based
Researchers:

Learning using Information Trails
Game Mods

Game Developers’ Toolkit

(Serious)
Video Games
Data Store
Commercial
Games

Assessment of Learning
(Based on Learners’
Persistent Worlds

Behaviors, Decisions)

Southern Illinois
• Carbondale, USA

Data Visualization

Southern Illinois
• Carbondale, USA

Data Visualization


Learning Assessment With Serious Games

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (12)

Ähnlich wie Learning Assessment With Serious Games

Ähnlich wie Learning Assessment With Serious Games (20)

Learning Assessment With Serious Games