SlideShare ist ein Scribd-Unternehmen logo
1 von 73
Downloaden Sie, um offline zu lesen
AIED July ITS Authoring Tools Survey 1
ITS Authoring Tools: an
Overview of the state of the art
Tom Murray
University of Massachusetts &
Hampshire College, Amherst, MA
www.cs.umass.edu/~tmurray
• References in Murray 1999, IJAIED 10(1): Authoring Intelligent
Tutoring Systems: An analysis of the state of the art
AIED July ITS Authoring Tools Survey 2
OR:
Cottage industry forms as thousands
build intelligent tutoring systems in
their basements--NOT YET!
OR:
ITS construction:
How Easy Can It Be?
AIED July ITS Authoring Tools Survey 3
What is an ITS,
such that one can be “authored?”
– Any CBI system that separates content
(what) from strategy (how)
– Usually makes inferences about “what the
student knows”
– I.E. Contains a “model” of domain, strategy,
and/or student
• --> We’re talking about pretty basic ITSs
AIED July ITS Authoring Tools Survey 4
Purposes of ITS Authoring Tools
• (Caveat: Authoring shells vs. authoring tools)
• Cost-effective production of ITSs
• Decreased skill threshold for authors
• Insure good quality by content validation or
constraining the ITS to a particular model
• Allow more participation of practicing educators in
ITS design and evaluation
• Provide a test bed for evaluation of alternative
strategy or content models
AIED July ITS Authoring Tools Survey 5
How many ITS authoring tools
have been built?
29 projects
CATEGORY PROJECTS/SYSTEMS
1 Curriculum Sequencing and
Planning
DOCENT, IDE, ISD Expert, Expert CML
2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT
3 Device Simulation and
Equipment Training
DIAG, RIDES, MITT-Writer, ICAT,
SIMQUEST, XAIDA
4 Domain Expert System Demonstr8, D3 Trainer, Training Express
5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS,
XAIDA
6 Special Purpose IDLE-Tool/IMap, LAT
7 Intelligent/adaptive
Hypermedia
CALAT, GETMAS, InterBook,
MetaLinks
AIED July ITS Authoring Tools Survey 6
How many ITS authoring tools …MORE
29 + 17 =
46 systems
~ 1982 to
present
CALAT CAIRNEY
DEMONSTR8 TDK, PUPS
DOCENT Study
Eon KAFITS
ID EXPERT Electronic Trainer, ISD-Expert
IDLE-Tool IMAP, INDIE, GBS-architectures
REDEEM COCA
RIDES IMTS, RAPIDS, and see DIAG
SIMQUEST SMISLE
Smart-Trainer AT FITS
Precursor systems
• References in Murray 1999, IJAIED 10(1): Authoring Intelligent
Tutoring Systems: An analysis of the state of the art
AIED July ITS Authoring Tools Survey 7
Overview: Multiple perspectives
describing the field
• What kinds of ITSs have been authored?
• Authoring the Interface, Domain, Teaching, and
Student Models
• What Authoring/Knowledge Acquisition Methods
Have Been Used?
• How Are Authoring Systems Designed? (Design
Tradeoffs & Open Issues)
• Pragmatics and Use (Are ITS authoring systems
“real?”)
AIED July ITS Authoring Tools Survey 8
What kinds of ITSs
have been authored?
• Both pedagogy-oriented and
performance-oriented ITSs
• Seven Types of ITSs
• Tools constrain ITSs
AIED July ITS Authoring Tools Survey 9
Seven Categories of Authored ITSs
• Strengths, Limits, Variations, student perspective
• Categories 3, 4, & 6 are mostly “performance-oriented”
CATEGORY PROJECTS/SYSTEMS
1 Curriculum Sequencing and
Planning
DOCENT, IDE, ISD Expert, Expert CML
2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT
3 Device Simulation and
Equipment Training
DIAG, RIDES, MITT-Writer, ICAT,
SIMQUEST, XAIDA
4 Domain Expert System Demonstr8, D3 Trainer, Training Express
5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS,
XAIDA
6 Special Purpose IDLE-Tool/IMap, LAT
7 Intelligent/adaptive
Hypermedia
CALAT, GETMAS, InterBook,
MetaLinks
AIED July ITS Authoring Tools Survey 10
1. Curriculum Sequencing and
Planning
Systems: DOCENT, IDE, ISD Expert, Expert CML
• Basic and early historical systems
• Separates content from presentation and sequencing
• Rules, constraints, or strategies for “intelligently”
sequencing content--at the “macro level” (topic
level)
• Usually low fidelity interfaces, canned content,
simple student models
AIED July ITS Authoring Tools Survey 11
2. Tutoring Strategies
Systems: REDEEM, Eon, GTE, Smart Trainer AT
#1 above PLUS:
• Micro-level and explicit tutoring strategies
– Instructional primitives for hints, explanations, examples.
reviews, feedback…
– Instruction can have a more dialogue or conversational feel
• Some include multiple teaching strategies and meta-
strategies
• Often have low fidelity interfaces, canned content,
simple student models
AIED July ITS Authoring Tools Survey 12
Tutoring strategies category: Example
REDEEM Genetics Tutor
Content from (ToolBook based)
CAI courseware
AIED July ITS Authoring Tools Survey 13
3. Device Simulation and
Equipment Training
Systems: DIAG, RIDES, MITT-Writer, ICAT,
SIMQUEST, XAIDA
• Micro-world/simulation-based learning environments
• Most focus on equipment/device operation and
maintenance procedures
• Building the simulation is time consuming, but much
of the “tutoring” then comes for free.
AIED July ITS Authoring Tools Survey 14
Examples from RIDES Tutors
AIED July ITS Authoring Tools Survey 15
4. Domain Expert System
Systems: Demonstr8, D3 Trainer, Training Express
• Deep/runnable models of problem solving expertise
• Fine grained student diagnosis and modeling
• Building an expert system is very difficult -- but then
instruction can come “for free”
AIED July ITS Authoring Tools Survey 16
D3s Medical Tutor
AIED July ITS Authoring Tools Survey 17
Demonstr8’s Subtraction Tutor
AIED July ITS Authoring Tools Survey 18
5. Multiple Knowledge Types
Systems: CREAM-Tools, DNA, ID-Expert, IRIS,
XAIDA
• “Gagne Hypothesis:” There are different types of
knowledge --> Each has its own instructional methods
and representational formalism
• Template-like framework for decomposing content
into facts, concepts, and procedures
• Many based on instructional design theory principles
• Limited so far to relatively simple facts, concepts,
procedures
AIED July ITS Authoring Tools Survey 19
6. Special Purpose
Systems: IDLE-Tool/IMap, LEAP Authoring Tool
• Build tutors for a particular type of task
• Can provide strong authoring guidance and
constraints
• Design and pedagogical principles can be enforced
• The task, interface, and pedagogy must fit relatively
inflexibly to the given model
AIED July ITS Authoring Tools Survey 20
Example: IDLE-Tool:
Sickle Cell Counselor
AIED July ITS Authoring Tools Survey 21
7. Intelligent/Adaptive Hypermedia
Systems: CALAT, GETMAS, InterBook, MetaLinks
• Similar to Category #1 but also deals with Navigation
and (dis)orientation issues
• Accessibility and UI uniformity benefits associated
with the WWW
• Limited interactivity and learning environment
fidelity
• Potential for making inferences from large numbers of
students
AIED July ITS Authoring Tools Survey 22
Example: InterBook
AIED July ITS Authoring Tools Survey 23
Authoring the Interface, Domain,
Teaching, and Student Models
• Interface
• Domain Model
– Curriculum knowledge structures
– Simulations of Devices and Phenomena
– Expert Systems
• Teaching Model
• Student Model
AIED July ITS Authoring Tools Survey 24
1. Authoring the Interface
• Systems with interface authoring tools:
RIDES (below), Eon, SIMQUEST RIDES
AIED July ITS Authoring Tools Survey 25
Example 2
Eon’s Interface Editor
EON
AIED July ITS Authoring Tools Survey 26
2. Authoring the Domain model
Curriculum knowledge and structures
Simulations/models of devices and
phenomena
Domain Expertise models (expert system)
AIED July ITS Authoring Tools Survey 27
Authoring Curriculum
Knowledge and Structures
• Topics/KUs
• Relationships
(e.g. prerequisite)
• Knowl. Type
(concept, procedure…)
• Objectives
• Importance
• Difficulty
Eon
AIED July ITS Authoring Tools Survey 28
Example: IRIS
AIED July ITS Authoring Tools Survey 29
Example: CREAM Tools
AIED July ITS Authoring Tools Survey 30
Authoring Simulations of
Devices and Phenomena
XAIDA
AIED July ITS Authoring Tools Survey 31
Example 2 RIDES
AIED July ITS Authoring Tools Survey 32
Authoring Domain Expertise
(Expert systems): D3 Trainer
AIED July ITS Authoring Tools Survey 33
3. Authoring the Teaching Model
Example: REDEEM
AIED July ITS Authoring Tools Survey 34
Authoring the Teaching Model
Example 2: Eon
AIED July ITS Authoring Tools Survey 35
4. Authoring the Student Model
Eon’s SM
Editor
AIED July ITS Authoring Tools Survey 36
What Authoring/Knowledge
Acquisition Methods Have Been Used?
• 1. Scaffolding knowledge articulation with models
• 2. Embedded knowledge and default knowledge
• 3. Knowledge management
• 4. Knowledge visualization
• 5. Knowledge elicitation and work flow
management
• 6. Knowledge and design validation
• 7. Knowledge re-use
• 8. Automated knowledge creation
AIED July ITS Authoring Tools Survey 37
1. Scaffolding knowledge articulation
with models
• Ex. 1:
Templates:
IDLE-Tools
• Ex. 2: Ontology-Aware tools: SmartTrainer AT
AIED July ITS Authoring Tools Survey 38
REDEEM
2. Embedded knowledge and
default knowledge
AIED July ITS Authoring Tools Survey 39
3. Knowledge management
• Topics/KUs
• Lesson Objectives
• Interface objects & screens
• Exercises, examples, pictures
• Teaching Strategy actions
CALAT
AIED July ITS Authoring Tools Survey 40
4. Knowledge visualization
• LEAP-AT
AIED July ITS Authoring Tools Survey 41
5. Knowledge elicitation and
work flow management
• Author: “What do I do next?” “ Where do I start?”
• Prompts in ID-Expert and DNA: “Which of the
following describes what the student will learn: a.
What is is? B. How to do it? C. How does it work?”
• Top down vs opportunistic design
– DNA: Semi-structured interactive dialog has
prompts with choices
• REDEEM: Agenda mechanism for authoring tasks
AIED July ITS Authoring Tools Survey 42
6. Knowledge and design validation
• Opportunistic & Open ended --> more flexibility &
more errors
• Constraint-based advice:
– “The estimated time for all Lesson-2 topics exceeeds the
estimated time for Lesson-2”
– “The engine maintenance procedure has no sub-steps
defined”
– “Lesson-3 objectives include procedural and conceptual
knowledge, but there are no conceptual topics linked to
Lesson-3.”
AIED July ITS Authoring Tools Survey 43
7. Knowledge re-use
• Libraries of Content, Graphics,
Strategies, etc.
• Flexible reconfiguration of components
AIED July ITS Authoring Tools Survey 44
8. Automated knowledge creation
• Example-Based programming
– Inferring a general procedure/rule
from an example procedure/rule
DEMONSTR8
AIED July ITS Authoring Tools Survey 45
Ex.2: Automated knowledge creation
faulty
RU1 Abnormal
Outcome
A1
4
1
2
4
5
Indicator A
Indicator B
NORMAL
NORMAL
Outcome
A2
Abnormal
Outcome
B1
Abnormal
Outcome
B2
Outcome
B3
3faulty
RU2
faulty
RU3
faulty
RU1
faulty
RU2
faulty
RU3
1
6 ALWAYS
5 USUALLY
4 VERY_OFTEN
3 OFTEN AS NOT
2 SOMETIMES
1 RARELY
0 NEVER
1
6
2
3
5
DIAG
AIED July ITS Authoring Tools Survey 46
Suggestions for a
Full-Featured Authoring System
• Visual reification of conceptual and structural elements
• Assistance: design steps or agenda; constraint-based
validation
• Content reusability and object libraries
• Scriptable and customizable
• WYSIWIG editing, Opportunistic design, Easy design/test
iteration (interpreted vs compiled), Reasonable default values
– for rapid prototyping
AIED July ITS Authoring Tools Survey 47
How Are Authoring Systems Designed?
Design Tradeoffs & Open Issues
• The space of design tradeoffs
• General vs. special purpose authoring systems
• Who are the authors?
• Who should author ITS instructional
strategies?
• Meta-Level Authoring
AIED July ITS Authoring Tools Survey 48
The Space of Design Tradeoffs
Domain
Model
Tutoring
Strategy
Student
Model
Learning
Environment
Power/ Breadth
Flexibility Depth
Learnability
Usability Productivity
Fidelity
Cost
[The design space has 24 (6x4)
independent dimensions or axes.]
Domain
Model
Tutoring
Strategy
Student
Model
Learning
Environment
Power/ Breadth
Flexibility Depth
Learnability
Usability Productivity
Fidelity
Cost
[The design space has 24 (6x4)
independent dimensions or axes.]
AIED July ITS Authoring Tools Survey 49
General vs. special purpose
authoring systems
• E.G. special purpose systems: LAT and IDLE-Tool
– Greater usability, fidelity, depth -- but only for design
goals that match the tools.
– Does the “demand” balance the inflexibility?
– How to make more customizable while maintaining ease
of use?
• Types of abstraction/specialization?
– 1. Real-world tasks
– 2. Abstract tasks
– 3. Knowledge types
AIED July ITS Authoring Tools Survey 50
Abstracting ITSs for
special purpose authoring systems
• 1. Abstracting real-world tasks:
Investigate & Decide; Evidence-Based reporting;
Run an Organization
• 2. “Abstract tasks:”
Equipment operation & maintenance (RIDES);
Conversational Grammars (customer service; LAT)
• 3. Knowledge types:
Facts, concepts, procedures, principles (CREAM-
Tools, DNA, ID-Expert, XAIDA)
AIED July ITS Authoring Tools Survey 51
Who are the authors?
What level of skill & training should be expected?
• Authoring skill sets: instructional design, classroom
pragmatics, graphics/UI, domain knowledge,
knowledge engineering, script-level programming...
• IDLE, XAIDA, REDEEM: try to allow authoring by
teachers and “off the street” domain experts with
minimal training
AIED July ITS Authoring Tools Survey 52
Suggested authoring scenario
• Effort level: Building an ITS is more like writing a book
than creating a greeting card!
• Skill level: Skill level equivalent: Accounting
applications, CAD, spreadsheet macros, 3-D modeling,
advanced Photoshop…-- Special training but reasonable
• Sophistication level: Authors need to look at the big
picture and do ongoing quality assessment of what they
build
• ITSs are built by design teams, not individuals
(distributed skill sets)
AIED July ITS Authoring Tools Survey 53
Who should specify/author ITS
instructional strategies?
PROS CONS
Teachers
PRACTICAL
Practical experience Not good at articulating or
abstracting expertise
Instructional Designers
ANALYTIC
Theories are widely used in
some circles
Limited to basic knowledge types
that are easily represented
Psychologists
THEORETICAL
Know “how the mind works” Use 'first principles'—only useful
for simple knowledge structures
Educational researchers
EMPIRICAL
Empirical studies of tutoring
and classrooms
After many years still don't
agree on much
Computer scientists
(ACTUAL?!)
...end up building the systems… “Isn’t it just all common sense?”…
Domain Experts
(I.E. NO acquisition of
instructional knowledge
Experts just show how they do a
task & authoring tool infers the
instructional methods
Fixed instructional method
AIED July ITS Authoring Tools Survey 54
Meta-Level Authoring
• Custom/extensible
interface widgets
• Customizable
descriptive
vocabulary
• Pre-configured
tutoring strategies
and student models
Eon
AIED July ITS Authoring Tools Survey 55
Use & Pragmatics
(Are ITS authoring systems “real?”)
• Authoring system Use
• Authoring system Productivity
• Authoring system Evaluation
AIED July ITS Authoring Tools Survey 56
Authoring Tool Use
Examples: • XAIDA domains: equipment operation and maintenance,
algebra, medicine, computer literacy, biology
• IDLE-Tool: three informal trials with 21, 8, 8 grad student
and grade school teacher authors
1. Early prototypes and
proofs of concept
D3 Trainer, Demonstr8, DIAG, IRIS,
Expert-CML, SmartTrainer AT
2. Evaluated or used
prototypes
CREAM-Tools, DNA, Eon, GTE,
IDLE-Tool, LAT
3. Moderately evaluated
or used
ISD-Expert/Training Express,
REDEEM, SIMQUEST, XAIDA
4. Heavily used
(relatively)
IDE, CALAT, RIDES
AIED July ITS Authoring Tools Survey 57
(Relatively) Heavily Used
Authoring Tools
• Build a dozen or more ITSs
• Many ITSs used in real educational settings
• Robust enough for use independent of original design
team
• RIDES: many project spin-offs and diverse domains
• CALAT: over 300 Web-based courses (used at NTT)
AIED July ITS Authoring Tools Survey 58
Authoring Tool Productivity
• For traditional CAI: Estimated 300:1 ratio of
development to instruction time
• ITS authoring: goals and some spotty evidence
– ID-Expert’s goal: 30:1
– XAIDA’s goal 10:1; evidence of a first time user at 16:1
– KAFITS Physics tutor w/ six hours of instruction: 100:1
– CALAT: ITS development in about the same time as
traditional instruction
– REDEEM: 2:1 to segment CAI content & make intelligent
• Implication: AI Knowledge Representation does
provide ITSs with inherent efficiencies
AIED July ITS Authoring Tools Survey 59
Authoring Tool Evaluations
• Existence proofs: Usability; Productivity; Breadth
• Examples:
– XAIDA
– REDEEM
– IDLE-Tools, COCA, LAT, KAFITS, DNA
AIED July ITS Authoring Tools Survey 60
Evaluations of XAIDA
• Eight authoring field studies with average of 10
instructor participants each
• 13 studies of students using the built tutors
Data:
• Learnability: abilities assessment, self-report skills,
cognitive assessment, task-based performance
• Acceptability: open-ended questionnaire
• Productivity: use analysis
• Usability: questionnaire
AIED July ITS Authoring Tools Survey 61
XAIDA Evaluation
Valence of Comments
Across Training
0%
20%
40%
60%
80%
100%
Before
training
End of Day
1
End of Day
2
End of Day
3
After
training
Neutral
Negative
Positive
0
5
10
15
20
25
30
35
40
Before training
(22)
End Day 1 (35) End Day 2 (24) End Day 3 (34) After tra
(17
Frequency of Comments
AIED July ITS Authoring Tools Survey 62
Proficiency Using XAIDA
1
2
3
4
5
6
7
8
9
10
Before training End of Day 1 End of Day 2 End of Day 3 End of Day 4 After training
NOVICE
EXPERT
AIED July ITS Authoring Tools Survey 63
Task Time Spent
Training 2 hours
Course familiarisation 1 hour
Describing pages and sections 4 hours
Reflection points & non-computer-based
tasks
1 hour
Authoring questions 2 hours
Classifying students 15 mins
Developing teaching strategies 15 mins
Relating students to sections 15 mins
Relating students to strategies 5 mins
Total 10 hours 50 mins
REDEEM Evaluation:
• 1 SME author, 3 teacher authors, 7 “virtual students”
• Data: authoring sub-task time, variations among
authors, appreciation of added “intelligence”
Time spent by teacher practitioner:
AIED July ITS Authoring Tools Survey 64
Some Formative Evaluation Results
(And see Productivity and Use above):
• Authors’ cognitive model of the domain had a structure closer
to SME’s after tool use (XAIDA)
• Considerable difference between authors in content structure,
strategy specification, categorization of students (REDEEM)
• Teacher reactions in general positive but difficulty with
complex relationships among content pieces
• Teachers thought AI technology could simulate reasonable
teaching strategies(COCOA)
• Tools needed to give users abstract view of the content (IDLE-
Tools)
AIED July ITS Authoring Tools Survey 65
Some Formative
Evaluation Results (cont.)
• Including examples for design steps/information was very
helpful (IDLE-Tools)
• Graphic representations for knowledge elicitation much less
error-prone than text-based (LAT)
• Overestimated of the level of expertise authors would gain in
a short amount of time (LAT)
• Authors have difficulty conceptualizing non-linear, modular
content (KAFTIS)
• Comparing automated knowledge elicitation to coded-by-
hand task analysis: automated method covered most of the
domain knowledge in a small fraction of the time (DNA)
AIED July ITS Authoring Tools Survey 66
Summary
• Many types of ITSs have been “authored”
• Wide variety of knowledge acquisition and authoring
methods have been used. Too early to know when
each is most appropriate.
• Some tools have significant use and a few are in
commercial or near-commercial form
• Promising results in from evaluations of usability
and productivity, with more rigorous evaluations
just starting
• What are the foreseeable limits?
AIED July ITS Authoring Tools Survey 67
Conclusions--How Easy Can It Be?
• There are limits!
• Limited use of cookie-cutter special purpose
authoring tools-- too restrictive for most authors
• Limited ability to reduce ITS authoring to easy,
small, independent steps (recipes)
• Authors need to think about the big picture and need
skills and tools to do this
AIED July ITS Authoring Tools Survey 68
...Back to the Future
• Customizability requirements will usually lead to the
author specifying BEHAVIORS (choices, rules,
algorithms) as well as static information
• This requires ability to RUN, test, and modify these
behaviors
• This is (simple) PROGRAMMING
• Debugging skills and tools will be needed! (Tracing,
stepping, inspecting states, etc.)
AIED July ITS Authoring Tools Survey 69
------------------------------------
AIED July ITS Authoring Tools Survey 70
ITS Authoring Tools: an
Overview of the state of the art
Tom Murray
University of Massachusetts &
Hampshire College, Amherst, MA
www.cs.umass.edu/~tmurray
AIED July ITS Authoring Tools Survey 71
People #1
CALAT (&
CAIRNEY)
Kiyama, M., Ishiuchi, S., Ikeda, K., Tsujimoto, M. & Fukuhara, Y.
(1997).
CREAM-TOOLS Frasson, C., Nkambou, R., Gauthier, G., Rouane, K. (1998).
Nkambou, R., Gauthier, R., & Frasson, M.C. (1996).
D3-TRAINER Reinhardt, B., Schewe, S. (1995).
DEMONSTR8 (&
TDK, PUPS)
Blessing, S.B. (1997). Anderson, J. R. & Pelletier, R. (1991).
Anderson, J. & Skwarecki, E. (1986).
DIAG Towne, D.M. (1997).
EON (& KAFITS) Murray, T. (1998,1996).
IDLE-Tool (&
IMAP, GBS-archits)
Bell, B. (1999). Jona, M. & Kass, A. (1997).
INTERBOOK (&
ElM-Art)
Brusilovsky, P., Schwartz, E., & Weber, G. (1996).
IRIS Arruarte, A., Fernandez-Castro, I., Ferrero, B. & Greer, J. (1997).
LAT (LEAP
Authoring Tool)
Sparks, R. Dooley, S., Meiskey, L. & Blumenthal, R. (1999).
Dooley, S., Meiskey, L., Blumenthal, R., & Sparks, R. (1995).
REDEEM (&
COCA)
Major, N., Ainsworth, S. & Wood, D. (1997). Major, N.P. & Reichgelt,
H (1992).
RIDES (& IMTS,
RAPIDS, DIAG)
Munro, A., Johnson, M.C., Pizzini, Q.A., Surmon, D.S., Towne, D.M,
& Wogulis, J.L. (1997). Towne, D.M., Munro, A., (1988).
Smart Trainer
AT (& FITS)
Jin, L, Chen, W., Hayashi, Y., Ikeda, M. Mizoguchi, R. (1999); Ikeda,
M. & Mizoguchi, R. (1994)
XAIDA Hsieh, P., Halff, H, Redfield, C. (1999). Wenzel, B., Dirnberger, M.,
Hsieh, P., Chudanov, T., & Halff, H. (1998).
AIED July ITS Authoring Tools Survey 72
People #2
DNA/SMART Shute, V.J. (1998).
DOCENT (&
Study)
Winne P.H. (1991). Winne, P. & Kramer, L. (1988).
EXPERT-CML Jones, M. & Wipond, K. (1991).
GETMAS Wong, W.K. & Chan, T.W. (1997).
GTE Van Marcke, K. (1998,1992).
ID EXPERT (&
Electronic
Trainer)
Merrill, M.D., & ID2 Research Group (1998). Merrill, M. D. (1987).
IDE (& IDE
Interpreter)
Russell, D. (1988). Russell, D., Moran, T. & Jordan, D. (1988).
MetaLinks Murray, T., Condit, C., & Haaugsjaa, E. (1998).
SIMQUEST (&
SMISLE)
Jong, T. de & vanJoolingen, W.R. (1998). Van Joolingen, W.R. &
Jong, T. de (1996).
TRAINING
EXPRESS
Clancey, W. & Joerger, K. (1988).
AIED July ITS Authoring Tools Survey 73

Weitere ähnliche Inhalte

Andere mochten auch

(Civic and political education 2) murray print, dirk lange (auth.), murray pr...
(Civic and political education 2) murray print, dirk lange (auth.), murray pr...(Civic and political education 2) murray print, dirk lange (auth.), murray pr...
(Civic and political education 2) murray print, dirk lange (auth.), murray pr...Zaky Luthfi
 
Contextual jQuery
Contextual jQueryContextual jQuery
Contextual jQueryDoug Neiner
 
Curriculum evaluation models practical applications for teache
Curriculum evaluation  models   practical applications for teacheCurriculum evaluation  models   practical applications for teache
Curriculum evaluation models practical applications for teacheBELETE DAMTEW
 
Curriculum leadership chapter 5 powerpoint
Curriculum leadership chapter 5   powerpointCurriculum leadership chapter 5   powerpoint
Curriculum leadership chapter 5 powerpointlbrannan84
 
Curriculum model by nicholls
Curriculum model by nichollsCurriculum model by nicholls
Curriculum model by nichollshiba awan
 
TSL3143 Topic 2a Models of Curriculum Design
TSL3143 Topic 2a Models of Curriculum DesignTSL3143 Topic 2a Models of Curriculum Design
TSL3143 Topic 2a Models of Curriculum DesignYee Bee Choo
 
WHEELER Cyclical Model of curriculum Process
WHEELER Cyclical Model of curriculum ProcessWHEELER Cyclical Model of curriculum Process
WHEELER Cyclical Model of curriculum ProcessInternational advisers
 
Developing the curriculum chapter 5
Developing the curriculum chapter 5Developing the curriculum chapter 5
Developing the curriculum chapter 5GrigsbyB
 
Models of curriculum dvelopment
Models of curriculum dvelopmentModels of curriculum dvelopment
Models of curriculum dvelopmentjasleenbrar03
 
Curriculum models
Curriculum modelsCurriculum models
Curriculum modelsKt Mosinyi
 
Cyclical models of curriculum development
Cyclical models of curriculum developmentCyclical models of curriculum development
Cyclical models of curriculum developmentMamoona Shahzad
 
curriculum design and models
curriculum design and models curriculum design and models
curriculum design and models gaestimos
 
Linear model of Curriculum
Linear model of CurriculumLinear model of Curriculum
Linear model of CurriculumJonna May Berci
 
Tyler s model_of_curriculum_development
Tyler s model_of_curriculum_developmentTyler s model_of_curriculum_development
Tyler s model_of_curriculum_developmentAbdulrahman Al'uganda
 
Curriculum Design Models
Curriculum Design ModelsCurriculum Design Models
Curriculum Design ModelsRuth Flores
 

Andere mochten auch (20)

(Civic and political education 2) murray print, dirk lange (auth.), murray pr...
(Civic and political education 2) murray print, dirk lange (auth.), murray pr...(Civic and political education 2) murray print, dirk lange (auth.), murray pr...
(Civic and political education 2) murray print, dirk lange (auth.), murray pr...
 
Contextual jQuery
Contextual jQueryContextual jQuery
Contextual jQuery
 
Curriculum evaluation models practical applications for teache
Curriculum evaluation  models   practical applications for teacheCurriculum evaluation  models   practical applications for teache
Curriculum evaluation models practical applications for teache
 
Curriculum leadership chapter 5 powerpoint
Curriculum leadership chapter 5   powerpointCurriculum leadership chapter 5   powerpoint
Curriculum leadership chapter 5 powerpoint
 
Evaluation model
Evaluation modelEvaluation model
Evaluation model
 
Curriculum model by nicholls
Curriculum model by nichollsCurriculum model by nicholls
Curriculum model by nicholls
 
TSL3143 Topic 2a Models of Curriculum Design
TSL3143 Topic 2a Models of Curriculum DesignTSL3143 Topic 2a Models of Curriculum Design
TSL3143 Topic 2a Models of Curriculum Design
 
Decker Walker's curriculum model (1971)
Decker Walker's curriculum model (1971)Decker Walker's curriculum model (1971)
Decker Walker's curriculum model (1971)
 
WHEELER Cyclical Model of curriculum Process
WHEELER Cyclical Model of curriculum ProcessWHEELER Cyclical Model of curriculum Process
WHEELER Cyclical Model of curriculum Process
 
Developing the curriculum chapter 5
Developing the curriculum chapter 5Developing the curriculum chapter 5
Developing the curriculum chapter 5
 
Curriculum and instruction day 3
Curriculum  and instruction day 3Curriculum  and instruction day 3
Curriculum and instruction day 3
 
Models of curriculum dvelopment
Models of curriculum dvelopmentModels of curriculum dvelopment
Models of curriculum dvelopment
 
Curriculum models
Curriculum modelsCurriculum models
Curriculum models
 
Cyclical models of curriculum development
Cyclical models of curriculum developmentCyclical models of curriculum development
Cyclical models of curriculum development
 
curriculum design and models
curriculum design and models curriculum design and models
curriculum design and models
 
Linear model of Curriculum
Linear model of CurriculumLinear model of Curriculum
Linear model of Curriculum
 
Tyler s model_of_curriculum_development
Tyler s model_of_curriculum_developmentTyler s model_of_curriculum_development
Tyler s model_of_curriculum_development
 
Tyler model
Tyler modelTyler model
Tyler model
 
Taba model of curriculum
Taba model of curriculum Taba model of curriculum
Taba model of curriculum
 
Curriculum Design Models
Curriculum Design ModelsCurriculum Design Models
Curriculum Design Models
 

Ähnlich wie Aied99 a toolstalk_murray

information system analysis and design
information system analysis and designinformation system analysis and design
information system analysis and designEndalkachewYazie1
 
Instructional Design
Instructional DesignInstructional Design
Instructional Designjamalharun
 
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisation
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisationLearning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisation
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisationTore Hoel
 
Addie model of instruction
Addie model of instructionAddie model of instruction
Addie model of instructiontetenusa
 
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...Hans Põldoja
 
Rae t4 d-knowledge-economy-sa-urs-dec2017
Rae t4 d-knowledge-economy-sa-urs-dec2017Rae t4 d-knowledge-economy-sa-urs-dec2017
Rae t4 d-knowledge-economy-sa-urs-dec2017MYRA School of Business
 
Exploring Learning Ecologies: Models and Experiences So Far
Exploring Learning Ecologies: Models and Experiences So Far Exploring Learning Ecologies: Models and Experiences So Far
Exploring Learning Ecologies: Models and Experiences So Far BCcampus
 
Unifying an Introduction to Artificial Intelligence Course ...
Unifying an Introduction to Artificial Intelligence Course ...Unifying an Introduction to Artificial Intelligence Course ...
Unifying an Introduction to Artificial Intelligence Course ...butest
 
Continuous assessment design case
Continuous assessment design caseContinuous assessment design case
Continuous assessment design caseMicah Modell
 
AI-Powered Academic Writing Full Deck RV edits 12 June.pptx
AI-Powered Academic Writing Full Deck RV edits 12 June.pptxAI-Powered Academic Writing Full Deck RV edits 12 June.pptx
AI-Powered Academic Writing Full Deck RV edits 12 June.pptxVaikunthan Rajaratnam
 
Requirements for Learning Analytics
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning AnalyticsTore Hoel
 
Designing, Developing, and Deploying Automated Learning
Designing, Developing, and Deploying Automated LearningDesigning, Developing, and Deploying Automated Learning
Designing, Developing, and Deploying Automated LearningShalin Hai-Jew
 

Ähnlich wie Aied99 a toolstalk_murray (20)

Data-X-Sparse-v2
Data-X-Sparse-v2Data-X-Sparse-v2
Data-X-Sparse-v2
 
Data-X-v3.1
Data-X-v3.1Data-X-v3.1
Data-X-v3.1
 
Isd basics stc
Isd basics stcIsd basics stc
Isd basics stc
 
information system analysis and design
information system analysis and designinformation system analysis and design
information system analysis and design
 
Addie
Addie Addie
Addie
 
Instructional Design
Instructional DesignInstructional Design
Instructional Design
 
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisation
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisationLearning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisation
Learning Analytics – Opportunities for ISO/IEC JTC 1/SC36 standardisation
 
Addie model of instruction
Addie model of instructionAddie model of instruction
Addie model of instruction
 
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...
Web-Based Self- and Peer-Assessment of Teachers’ Educational Technology Compe...
 
Rae t4 d-knowledge-economy-sa-urs-dec2017
Rae t4 d-knowledge-economy-sa-urs-dec2017Rae t4 d-knowledge-economy-sa-urs-dec2017
Rae t4 d-knowledge-economy-sa-urs-dec2017
 
Exploring Learning Ecologies: Models and Experiences So Far
Exploring Learning Ecologies: Models and Experiences So Far Exploring Learning Ecologies: Models and Experiences So Far
Exploring Learning Ecologies: Models and Experiences So Far
 
Unifying an Introduction to Artificial Intelligence Course ...
Unifying an Introduction to Artificial Intelligence Course ...Unifying an Introduction to Artificial Intelligence Course ...
Unifying an Introduction to Artificial Intelligence Course ...
 
Continuous assessment design case
Continuous assessment design caseContinuous assessment design case
Continuous assessment design case
 
Chapter02 Presentation
Chapter02 PresentationChapter02 Presentation
Chapter02 Presentation
 
AI-Powered Academic Writing Full Deck RV edits 12 June.pptx
AI-Powered Academic Writing Full Deck RV edits 12 June.pptxAI-Powered Academic Writing Full Deck RV edits 12 June.pptx
AI-Powered Academic Writing Full Deck RV edits 12 June.pptx
 
Reading quiz2
Reading quiz2Reading quiz2
Reading quiz2
 
Learner Dashboards to Inform Instructional Approaches
Learner Dashboards to Inform Instructional ApproachesLearner Dashboards to Inform Instructional Approaches
Learner Dashboards to Inform Instructional Approaches
 
ICS2208 lecture6
ICS2208 lecture6ICS2208 lecture6
ICS2208 lecture6
 
Requirements for Learning Analytics
Requirements for Learning AnalyticsRequirements for Learning Analytics
Requirements for Learning Analytics
 
Designing, Developing, and Deploying Automated Learning
Designing, Developing, and Deploying Automated LearningDesigning, Developing, and Deploying Automated Learning
Designing, Developing, and Deploying Automated Learning
 

Mehr von perspegrity5

2023 IEC wisdom skills-awakening-sharable.pptx
2023 IEC wisdom skills-awakening-sharable.pptx2023 IEC wisdom skills-awakening-sharable.pptx
2023 IEC wisdom skills-awakening-sharable.pptxperspegrity5
 
2021 ESRAD wisdom skills_murray.pptx
2021 ESRAD wisdom skills_murray.pptx2021 ESRAD wisdom skills_murray.pptx
2021 ESRAD wisdom skills_murray.pptxperspegrity5
 
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdfperspegrity5
 
2021 Summary of Research on the STAGES Developmental Model
2021 Summary of Research on the STAGES Developmental Model2021 Summary of Research on the STAGES Developmental Model
2021 Summary of Research on the STAGES Developmental Modelperspegrity5
 
What's going on in later stages?—Do integral developmental theories agree?
What's going on in later stages?—Do integral developmental theories agree? What's going on in later stages?—Do integral developmental theories agree?
What's going on in later stages?—Do integral developmental theories agree? perspegrity5
 
Experiential integral ed book group slides 1
Experiential integral ed book group slides 1Experiential integral ed book group slides 1
Experiential integral ed book group slides 1perspegrity5
 
Mystical Claims and Embodied Knowledge in a Post-Metaphysical Age
Mystical Claims and Embodied Knowledge  in a Post-Metaphysical AgeMystical Claims and Embodied Knowledge  in a Post-Metaphysical Age
Mystical Claims and Embodied Knowledge in a Post-Metaphysical Ageperspegrity5
 
Authoring Tools, Complexity, Epistemic Forms, and Cognitive Development
Authoring Tools, Complexity, Epistemic Forms, and Cognitive DevelopmentAuthoring Tools, Complexity, Epistemic Forms, and Cognitive Development
Authoring Tools, Complexity, Epistemic Forms, and Cognitive Developmentperspegrity5
 
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Tools
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness ToolsBP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Tools
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Toolsperspegrity5
 
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-MurraySupporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murrayperspegrity5
 
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murrayperspegrity5
 
ODR 2013 SDSkills dashboard umass
ODR 2013 SDSkills dashboard umassODR 2013 SDSkills dashboard umass
ODR 2013 SDSkills dashboard umassperspegrity5
 
Soc comp slides_march_2012
Soc comp slides_march_2012Soc comp slides_march_2012
Soc comp slides_march_2012perspegrity5
 
Spiralectics- Tom Murray 11.10
Spiralectics- Tom Murray 11.10Spiralectics- Tom Murray 11.10
Spiralectics- Tom Murray 11.10perspegrity5
 
Holding and Promoting Beliefs: A Develpmental View.
Holding and Promoting Beliefs: A Develpmental View.Holding and Promoting Beliefs: A Develpmental View.
Holding and Promoting Beliefs: A Develpmental View.perspegrity5
 
On the development of beliefs vs. capacities: A post-metaphysical view of sec...
On the development of beliefs vs. capacities: A post-metaphysical view of sec...On the development of beliefs vs. capacities: A post-metaphysical view of sec...
On the development of beliefs vs. capacities: A post-metaphysical view of sec...perspegrity5
 

Mehr von perspegrity5 (16)

2023 IEC wisdom skills-awakening-sharable.pptx
2023 IEC wisdom skills-awakening-sharable.pptx2023 IEC wisdom skills-awakening-sharable.pptx
2023 IEC wisdom skills-awakening-sharable.pptx
 
2021 ESRAD wisdom skills_murray.pptx
2021 ESRAD wisdom skills_murray.pptx2021 ESRAD wisdom skills_murray.pptx
2021 ESRAD wisdom skills_murray.pptx
 
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf
2022 IEC Murray_wisdom skills-upside down SHARABLE.pdf
 
2021 Summary of Research on the STAGES Developmental Model
2021 Summary of Research on the STAGES Developmental Model2021 Summary of Research on the STAGES Developmental Model
2021 Summary of Research on the STAGES Developmental Model
 
What's going on in later stages?—Do integral developmental theories agree?
What's going on in later stages?—Do integral developmental theories agree? What's going on in later stages?—Do integral developmental theories agree?
What's going on in later stages?—Do integral developmental theories agree?
 
Experiential integral ed book group slides 1
Experiential integral ed book group slides 1Experiential integral ed book group slides 1
Experiential integral ed book group slides 1
 
Mystical Claims and Embodied Knowledge in a Post-Metaphysical Age
Mystical Claims and Embodied Knowledge  in a Post-Metaphysical AgeMystical Claims and Embodied Knowledge  in a Post-Metaphysical Age
Mystical Claims and Embodied Knowledge in a Post-Metaphysical Age
 
Authoring Tools, Complexity, Epistemic Forms, and Cognitive Development
Authoring Tools, Complexity, Epistemic Forms, and Cognitive DevelopmentAuthoring Tools, Complexity, Epistemic Forms, and Cognitive Development
Authoring Tools, Complexity, Epistemic Forms, and Cognitive Development
 
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Tools
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness ToolsBP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Tools
BP 2014: Supporting Deeper Deliberative Dialogue Through Awareness Tools
 
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-MurraySupporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
 
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray
2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray
 
ODR 2013 SDSkills dashboard umass
ODR 2013 SDSkills dashboard umassODR 2013 SDSkills dashboard umass
ODR 2013 SDSkills dashboard umass
 
Soc comp slides_march_2012
Soc comp slides_march_2012Soc comp slides_march_2012
Soc comp slides_march_2012
 
Spiralectics- Tom Murray 11.10
Spiralectics- Tom Murray 11.10Spiralectics- Tom Murray 11.10
Spiralectics- Tom Murray 11.10
 
Holding and Promoting Beliefs: A Develpmental View.
Holding and Promoting Beliefs: A Develpmental View.Holding and Promoting Beliefs: A Develpmental View.
Holding and Promoting Beliefs: A Develpmental View.
 
On the development of beliefs vs. capacities: A post-metaphysical view of sec...
On the development of beliefs vs. capacities: A post-metaphysical view of sec...On the development of beliefs vs. capacities: A post-metaphysical view of sec...
On the development of beliefs vs. capacities: A post-metaphysical view of sec...
 

Kürzlich hochgeladen

Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesMohammad Hassany
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptxSandy Millin
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxKatherine Villaluna
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and stepobaje godwin sunday
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationMJDuyan
 

Kürzlich hochgeladen (20)

Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming Classes
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and step
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive Education
 

Aied99 a toolstalk_murray

  • 1. AIED July ITS Authoring Tools Survey 1 ITS Authoring Tools: an Overview of the state of the art Tom Murray University of Massachusetts & Hampshire College, Amherst, MA www.cs.umass.edu/~tmurray • References in Murray 1999, IJAIED 10(1): Authoring Intelligent Tutoring Systems: An analysis of the state of the art
  • 2. AIED July ITS Authoring Tools Survey 2 OR: Cottage industry forms as thousands build intelligent tutoring systems in their basements--NOT YET! OR: ITS construction: How Easy Can It Be?
  • 3. AIED July ITS Authoring Tools Survey 3 What is an ITS, such that one can be “authored?” – Any CBI system that separates content (what) from strategy (how) – Usually makes inferences about “what the student knows” – I.E. Contains a “model” of domain, strategy, and/or student • --> We’re talking about pretty basic ITSs
  • 4. AIED July ITS Authoring Tools Survey 4 Purposes of ITS Authoring Tools • (Caveat: Authoring shells vs. authoring tools) • Cost-effective production of ITSs • Decreased skill threshold for authors • Insure good quality by content validation or constraining the ITS to a particular model • Allow more participation of practicing educators in ITS design and evaluation • Provide a test bed for evaluation of alternative strategy or content models
  • 5. AIED July ITS Authoring Tools Survey 5 How many ITS authoring tools have been built? 29 projects CATEGORY PROJECTS/SYSTEMS 1 Curriculum Sequencing and Planning DOCENT, IDE, ISD Expert, Expert CML 2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT 3 Device Simulation and Equipment Training DIAG, RIDES, MITT-Writer, ICAT, SIMQUEST, XAIDA 4 Domain Expert System Demonstr8, D3 Trainer, Training Express 5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS, XAIDA 6 Special Purpose IDLE-Tool/IMap, LAT 7 Intelligent/adaptive Hypermedia CALAT, GETMAS, InterBook, MetaLinks
  • 6. AIED July ITS Authoring Tools Survey 6 How many ITS authoring tools …MORE 29 + 17 = 46 systems ~ 1982 to present CALAT CAIRNEY DEMONSTR8 TDK, PUPS DOCENT Study Eon KAFITS ID EXPERT Electronic Trainer, ISD-Expert IDLE-Tool IMAP, INDIE, GBS-architectures REDEEM COCA RIDES IMTS, RAPIDS, and see DIAG SIMQUEST SMISLE Smart-Trainer AT FITS Precursor systems • References in Murray 1999, IJAIED 10(1): Authoring Intelligent Tutoring Systems: An analysis of the state of the art
  • 7. AIED July ITS Authoring Tools Survey 7 Overview: Multiple perspectives describing the field • What kinds of ITSs have been authored? • Authoring the Interface, Domain, Teaching, and Student Models • What Authoring/Knowledge Acquisition Methods Have Been Used? • How Are Authoring Systems Designed? (Design Tradeoffs & Open Issues) • Pragmatics and Use (Are ITS authoring systems “real?”)
  • 8. AIED July ITS Authoring Tools Survey 8 What kinds of ITSs have been authored? • Both pedagogy-oriented and performance-oriented ITSs • Seven Types of ITSs • Tools constrain ITSs
  • 9. AIED July ITS Authoring Tools Survey 9 Seven Categories of Authored ITSs • Strengths, Limits, Variations, student perspective • Categories 3, 4, & 6 are mostly “performance-oriented” CATEGORY PROJECTS/SYSTEMS 1 Curriculum Sequencing and Planning DOCENT, IDE, ISD Expert, Expert CML 2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT 3 Device Simulation and Equipment Training DIAG, RIDES, MITT-Writer, ICAT, SIMQUEST, XAIDA 4 Domain Expert System Demonstr8, D3 Trainer, Training Express 5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS, XAIDA 6 Special Purpose IDLE-Tool/IMap, LAT 7 Intelligent/adaptive Hypermedia CALAT, GETMAS, InterBook, MetaLinks
  • 10. AIED July ITS Authoring Tools Survey 10 1. Curriculum Sequencing and Planning Systems: DOCENT, IDE, ISD Expert, Expert CML • Basic and early historical systems • Separates content from presentation and sequencing • Rules, constraints, or strategies for “intelligently” sequencing content--at the “macro level” (topic level) • Usually low fidelity interfaces, canned content, simple student models
  • 11. AIED July ITS Authoring Tools Survey 11 2. Tutoring Strategies Systems: REDEEM, Eon, GTE, Smart Trainer AT #1 above PLUS: • Micro-level and explicit tutoring strategies – Instructional primitives for hints, explanations, examples. reviews, feedback… – Instruction can have a more dialogue or conversational feel • Some include multiple teaching strategies and meta- strategies • Often have low fidelity interfaces, canned content, simple student models
  • 12. AIED July ITS Authoring Tools Survey 12 Tutoring strategies category: Example REDEEM Genetics Tutor Content from (ToolBook based) CAI courseware
  • 13. AIED July ITS Authoring Tools Survey 13 3. Device Simulation and Equipment Training Systems: DIAG, RIDES, MITT-Writer, ICAT, SIMQUEST, XAIDA • Micro-world/simulation-based learning environments • Most focus on equipment/device operation and maintenance procedures • Building the simulation is time consuming, but much of the “tutoring” then comes for free.
  • 14. AIED July ITS Authoring Tools Survey 14 Examples from RIDES Tutors
  • 15. AIED July ITS Authoring Tools Survey 15 4. Domain Expert System Systems: Demonstr8, D3 Trainer, Training Express • Deep/runnable models of problem solving expertise • Fine grained student diagnosis and modeling • Building an expert system is very difficult -- but then instruction can come “for free”
  • 16. AIED July ITS Authoring Tools Survey 16 D3s Medical Tutor
  • 17. AIED July ITS Authoring Tools Survey 17 Demonstr8’s Subtraction Tutor
  • 18. AIED July ITS Authoring Tools Survey 18 5. Multiple Knowledge Types Systems: CREAM-Tools, DNA, ID-Expert, IRIS, XAIDA • “Gagne Hypothesis:” There are different types of knowledge --> Each has its own instructional methods and representational formalism • Template-like framework for decomposing content into facts, concepts, and procedures • Many based on instructional design theory principles • Limited so far to relatively simple facts, concepts, procedures
  • 19. AIED July ITS Authoring Tools Survey 19 6. Special Purpose Systems: IDLE-Tool/IMap, LEAP Authoring Tool • Build tutors for a particular type of task • Can provide strong authoring guidance and constraints • Design and pedagogical principles can be enforced • The task, interface, and pedagogy must fit relatively inflexibly to the given model
  • 20. AIED July ITS Authoring Tools Survey 20 Example: IDLE-Tool: Sickle Cell Counselor
  • 21. AIED July ITS Authoring Tools Survey 21 7. Intelligent/Adaptive Hypermedia Systems: CALAT, GETMAS, InterBook, MetaLinks • Similar to Category #1 but also deals with Navigation and (dis)orientation issues • Accessibility and UI uniformity benefits associated with the WWW • Limited interactivity and learning environment fidelity • Potential for making inferences from large numbers of students
  • 22. AIED July ITS Authoring Tools Survey 22 Example: InterBook
  • 23. AIED July ITS Authoring Tools Survey 23 Authoring the Interface, Domain, Teaching, and Student Models • Interface • Domain Model – Curriculum knowledge structures – Simulations of Devices and Phenomena – Expert Systems • Teaching Model • Student Model
  • 24. AIED July ITS Authoring Tools Survey 24 1. Authoring the Interface • Systems with interface authoring tools: RIDES (below), Eon, SIMQUEST RIDES
  • 25. AIED July ITS Authoring Tools Survey 25 Example 2 Eon’s Interface Editor EON
  • 26. AIED July ITS Authoring Tools Survey 26 2. Authoring the Domain model Curriculum knowledge and structures Simulations/models of devices and phenomena Domain Expertise models (expert system)
  • 27. AIED July ITS Authoring Tools Survey 27 Authoring Curriculum Knowledge and Structures • Topics/KUs • Relationships (e.g. prerequisite) • Knowl. Type (concept, procedure…) • Objectives • Importance • Difficulty Eon
  • 28. AIED July ITS Authoring Tools Survey 28 Example: IRIS
  • 29. AIED July ITS Authoring Tools Survey 29 Example: CREAM Tools
  • 30. AIED July ITS Authoring Tools Survey 30 Authoring Simulations of Devices and Phenomena XAIDA
  • 31. AIED July ITS Authoring Tools Survey 31 Example 2 RIDES
  • 32. AIED July ITS Authoring Tools Survey 32 Authoring Domain Expertise (Expert systems): D3 Trainer
  • 33. AIED July ITS Authoring Tools Survey 33 3. Authoring the Teaching Model Example: REDEEM
  • 34. AIED July ITS Authoring Tools Survey 34 Authoring the Teaching Model Example 2: Eon
  • 35. AIED July ITS Authoring Tools Survey 35 4. Authoring the Student Model Eon’s SM Editor
  • 36. AIED July ITS Authoring Tools Survey 36 What Authoring/Knowledge Acquisition Methods Have Been Used? • 1. Scaffolding knowledge articulation with models • 2. Embedded knowledge and default knowledge • 3. Knowledge management • 4. Knowledge visualization • 5. Knowledge elicitation and work flow management • 6. Knowledge and design validation • 7. Knowledge re-use • 8. Automated knowledge creation
  • 37. AIED July ITS Authoring Tools Survey 37 1. Scaffolding knowledge articulation with models • Ex. 1: Templates: IDLE-Tools • Ex. 2: Ontology-Aware tools: SmartTrainer AT
  • 38. AIED July ITS Authoring Tools Survey 38 REDEEM 2. Embedded knowledge and default knowledge
  • 39. AIED July ITS Authoring Tools Survey 39 3. Knowledge management • Topics/KUs • Lesson Objectives • Interface objects & screens • Exercises, examples, pictures • Teaching Strategy actions CALAT
  • 40. AIED July ITS Authoring Tools Survey 40 4. Knowledge visualization • LEAP-AT
  • 41. AIED July ITS Authoring Tools Survey 41 5. Knowledge elicitation and work flow management • Author: “What do I do next?” “ Where do I start?” • Prompts in ID-Expert and DNA: “Which of the following describes what the student will learn: a. What is is? B. How to do it? C. How does it work?” • Top down vs opportunistic design – DNA: Semi-structured interactive dialog has prompts with choices • REDEEM: Agenda mechanism for authoring tasks
  • 42. AIED July ITS Authoring Tools Survey 42 6. Knowledge and design validation • Opportunistic & Open ended --> more flexibility & more errors • Constraint-based advice: – “The estimated time for all Lesson-2 topics exceeeds the estimated time for Lesson-2” – “The engine maintenance procedure has no sub-steps defined” – “Lesson-3 objectives include procedural and conceptual knowledge, but there are no conceptual topics linked to Lesson-3.”
  • 43. AIED July ITS Authoring Tools Survey 43 7. Knowledge re-use • Libraries of Content, Graphics, Strategies, etc. • Flexible reconfiguration of components
  • 44. AIED July ITS Authoring Tools Survey 44 8. Automated knowledge creation • Example-Based programming – Inferring a general procedure/rule from an example procedure/rule DEMONSTR8
  • 45. AIED July ITS Authoring Tools Survey 45 Ex.2: Automated knowledge creation faulty RU1 Abnormal Outcome A1 4 1 2 4 5 Indicator A Indicator B NORMAL NORMAL Outcome A2 Abnormal Outcome B1 Abnormal Outcome B2 Outcome B3 3faulty RU2 faulty RU3 faulty RU1 faulty RU2 faulty RU3 1 6 ALWAYS 5 USUALLY 4 VERY_OFTEN 3 OFTEN AS NOT 2 SOMETIMES 1 RARELY 0 NEVER 1 6 2 3 5 DIAG
  • 46. AIED July ITS Authoring Tools Survey 46 Suggestions for a Full-Featured Authoring System • Visual reification of conceptual and structural elements • Assistance: design steps or agenda; constraint-based validation • Content reusability and object libraries • Scriptable and customizable • WYSIWIG editing, Opportunistic design, Easy design/test iteration (interpreted vs compiled), Reasonable default values – for rapid prototyping
  • 47. AIED July ITS Authoring Tools Survey 47 How Are Authoring Systems Designed? Design Tradeoffs & Open Issues • The space of design tradeoffs • General vs. special purpose authoring systems • Who are the authors? • Who should author ITS instructional strategies? • Meta-Level Authoring
  • 48. AIED July ITS Authoring Tools Survey 48 The Space of Design Tradeoffs Domain Model Tutoring Strategy Student Model Learning Environment Power/ Breadth Flexibility Depth Learnability Usability Productivity Fidelity Cost [The design space has 24 (6x4) independent dimensions or axes.] Domain Model Tutoring Strategy Student Model Learning Environment Power/ Breadth Flexibility Depth Learnability Usability Productivity Fidelity Cost [The design space has 24 (6x4) independent dimensions or axes.]
  • 49. AIED July ITS Authoring Tools Survey 49 General vs. special purpose authoring systems • E.G. special purpose systems: LAT and IDLE-Tool – Greater usability, fidelity, depth -- but only for design goals that match the tools. – Does the “demand” balance the inflexibility? – How to make more customizable while maintaining ease of use? • Types of abstraction/specialization? – 1. Real-world tasks – 2. Abstract tasks – 3. Knowledge types
  • 50. AIED July ITS Authoring Tools Survey 50 Abstracting ITSs for special purpose authoring systems • 1. Abstracting real-world tasks: Investigate & Decide; Evidence-Based reporting; Run an Organization • 2. “Abstract tasks:” Equipment operation & maintenance (RIDES); Conversational Grammars (customer service; LAT) • 3. Knowledge types: Facts, concepts, procedures, principles (CREAM- Tools, DNA, ID-Expert, XAIDA)
  • 51. AIED July ITS Authoring Tools Survey 51 Who are the authors? What level of skill & training should be expected? • Authoring skill sets: instructional design, classroom pragmatics, graphics/UI, domain knowledge, knowledge engineering, script-level programming... • IDLE, XAIDA, REDEEM: try to allow authoring by teachers and “off the street” domain experts with minimal training
  • 52. AIED July ITS Authoring Tools Survey 52 Suggested authoring scenario • Effort level: Building an ITS is more like writing a book than creating a greeting card! • Skill level: Skill level equivalent: Accounting applications, CAD, spreadsheet macros, 3-D modeling, advanced Photoshop…-- Special training but reasonable • Sophistication level: Authors need to look at the big picture and do ongoing quality assessment of what they build • ITSs are built by design teams, not individuals (distributed skill sets)
  • 53. AIED July ITS Authoring Tools Survey 53 Who should specify/author ITS instructional strategies? PROS CONS Teachers PRACTICAL Practical experience Not good at articulating or abstracting expertise Instructional Designers ANALYTIC Theories are widely used in some circles Limited to basic knowledge types that are easily represented Psychologists THEORETICAL Know “how the mind works” Use 'first principles'—only useful for simple knowledge structures Educational researchers EMPIRICAL Empirical studies of tutoring and classrooms After many years still don't agree on much Computer scientists (ACTUAL?!) ...end up building the systems… “Isn’t it just all common sense?”… Domain Experts (I.E. NO acquisition of instructional knowledge Experts just show how they do a task & authoring tool infers the instructional methods Fixed instructional method
  • 54. AIED July ITS Authoring Tools Survey 54 Meta-Level Authoring • Custom/extensible interface widgets • Customizable descriptive vocabulary • Pre-configured tutoring strategies and student models Eon
  • 55. AIED July ITS Authoring Tools Survey 55 Use & Pragmatics (Are ITS authoring systems “real?”) • Authoring system Use • Authoring system Productivity • Authoring system Evaluation
  • 56. AIED July ITS Authoring Tools Survey 56 Authoring Tool Use Examples: • XAIDA domains: equipment operation and maintenance, algebra, medicine, computer literacy, biology • IDLE-Tool: three informal trials with 21, 8, 8 grad student and grade school teacher authors 1. Early prototypes and proofs of concept D3 Trainer, Demonstr8, DIAG, IRIS, Expert-CML, SmartTrainer AT 2. Evaluated or used prototypes CREAM-Tools, DNA, Eon, GTE, IDLE-Tool, LAT 3. Moderately evaluated or used ISD-Expert/Training Express, REDEEM, SIMQUEST, XAIDA 4. Heavily used (relatively) IDE, CALAT, RIDES
  • 57. AIED July ITS Authoring Tools Survey 57 (Relatively) Heavily Used Authoring Tools • Build a dozen or more ITSs • Many ITSs used in real educational settings • Robust enough for use independent of original design team • RIDES: many project spin-offs and diverse domains • CALAT: over 300 Web-based courses (used at NTT)
  • 58. AIED July ITS Authoring Tools Survey 58 Authoring Tool Productivity • For traditional CAI: Estimated 300:1 ratio of development to instruction time • ITS authoring: goals and some spotty evidence – ID-Expert’s goal: 30:1 – XAIDA’s goal 10:1; evidence of a first time user at 16:1 – KAFITS Physics tutor w/ six hours of instruction: 100:1 – CALAT: ITS development in about the same time as traditional instruction – REDEEM: 2:1 to segment CAI content & make intelligent • Implication: AI Knowledge Representation does provide ITSs with inherent efficiencies
  • 59. AIED July ITS Authoring Tools Survey 59 Authoring Tool Evaluations • Existence proofs: Usability; Productivity; Breadth • Examples: – XAIDA – REDEEM – IDLE-Tools, COCA, LAT, KAFITS, DNA
  • 60. AIED July ITS Authoring Tools Survey 60 Evaluations of XAIDA • Eight authoring field studies with average of 10 instructor participants each • 13 studies of students using the built tutors Data: • Learnability: abilities assessment, self-report skills, cognitive assessment, task-based performance • Acceptability: open-ended questionnaire • Productivity: use analysis • Usability: questionnaire
  • 61. AIED July ITS Authoring Tools Survey 61 XAIDA Evaluation Valence of Comments Across Training 0% 20% 40% 60% 80% 100% Before training End of Day 1 End of Day 2 End of Day 3 After training Neutral Negative Positive 0 5 10 15 20 25 30 35 40 Before training (22) End Day 1 (35) End Day 2 (24) End Day 3 (34) After tra (17 Frequency of Comments
  • 62. AIED July ITS Authoring Tools Survey 62 Proficiency Using XAIDA 1 2 3 4 5 6 7 8 9 10 Before training End of Day 1 End of Day 2 End of Day 3 End of Day 4 After training NOVICE EXPERT
  • 63. AIED July ITS Authoring Tools Survey 63 Task Time Spent Training 2 hours Course familiarisation 1 hour Describing pages and sections 4 hours Reflection points & non-computer-based tasks 1 hour Authoring questions 2 hours Classifying students 15 mins Developing teaching strategies 15 mins Relating students to sections 15 mins Relating students to strategies 5 mins Total 10 hours 50 mins REDEEM Evaluation: • 1 SME author, 3 teacher authors, 7 “virtual students” • Data: authoring sub-task time, variations among authors, appreciation of added “intelligence” Time spent by teacher practitioner:
  • 64. AIED July ITS Authoring Tools Survey 64 Some Formative Evaluation Results (And see Productivity and Use above): • Authors’ cognitive model of the domain had a structure closer to SME’s after tool use (XAIDA) • Considerable difference between authors in content structure, strategy specification, categorization of students (REDEEM) • Teacher reactions in general positive but difficulty with complex relationships among content pieces • Teachers thought AI technology could simulate reasonable teaching strategies(COCOA) • Tools needed to give users abstract view of the content (IDLE- Tools)
  • 65. AIED July ITS Authoring Tools Survey 65 Some Formative Evaluation Results (cont.) • Including examples for design steps/information was very helpful (IDLE-Tools) • Graphic representations for knowledge elicitation much less error-prone than text-based (LAT) • Overestimated of the level of expertise authors would gain in a short amount of time (LAT) • Authors have difficulty conceptualizing non-linear, modular content (KAFTIS) • Comparing automated knowledge elicitation to coded-by- hand task analysis: automated method covered most of the domain knowledge in a small fraction of the time (DNA)
  • 66. AIED July ITS Authoring Tools Survey 66 Summary • Many types of ITSs have been “authored” • Wide variety of knowledge acquisition and authoring methods have been used. Too early to know when each is most appropriate. • Some tools have significant use and a few are in commercial or near-commercial form • Promising results in from evaluations of usability and productivity, with more rigorous evaluations just starting • What are the foreseeable limits?
  • 67. AIED July ITS Authoring Tools Survey 67 Conclusions--How Easy Can It Be? • There are limits! • Limited use of cookie-cutter special purpose authoring tools-- too restrictive for most authors • Limited ability to reduce ITS authoring to easy, small, independent steps (recipes) • Authors need to think about the big picture and need skills and tools to do this
  • 68. AIED July ITS Authoring Tools Survey 68 ...Back to the Future • Customizability requirements will usually lead to the author specifying BEHAVIORS (choices, rules, algorithms) as well as static information • This requires ability to RUN, test, and modify these behaviors • This is (simple) PROGRAMMING • Debugging skills and tools will be needed! (Tracing, stepping, inspecting states, etc.)
  • 69. AIED July ITS Authoring Tools Survey 69 ------------------------------------
  • 70. AIED July ITS Authoring Tools Survey 70 ITS Authoring Tools: an Overview of the state of the art Tom Murray University of Massachusetts & Hampshire College, Amherst, MA www.cs.umass.edu/~tmurray
  • 71. AIED July ITS Authoring Tools Survey 71 People #1 CALAT (& CAIRNEY) Kiyama, M., Ishiuchi, S., Ikeda, K., Tsujimoto, M. & Fukuhara, Y. (1997). CREAM-TOOLS Frasson, C., Nkambou, R., Gauthier, G., Rouane, K. (1998). Nkambou, R., Gauthier, R., & Frasson, M.C. (1996). D3-TRAINER Reinhardt, B., Schewe, S. (1995). DEMONSTR8 (& TDK, PUPS) Blessing, S.B. (1997). Anderson, J. R. & Pelletier, R. (1991). Anderson, J. & Skwarecki, E. (1986). DIAG Towne, D.M. (1997). EON (& KAFITS) Murray, T. (1998,1996). IDLE-Tool (& IMAP, GBS-archits) Bell, B. (1999). Jona, M. & Kass, A. (1997). INTERBOOK (& ElM-Art) Brusilovsky, P., Schwartz, E., & Weber, G. (1996). IRIS Arruarte, A., Fernandez-Castro, I., Ferrero, B. & Greer, J. (1997). LAT (LEAP Authoring Tool) Sparks, R. Dooley, S., Meiskey, L. & Blumenthal, R. (1999). Dooley, S., Meiskey, L., Blumenthal, R., & Sparks, R. (1995). REDEEM (& COCA) Major, N., Ainsworth, S. & Wood, D. (1997). Major, N.P. & Reichgelt, H (1992). RIDES (& IMTS, RAPIDS, DIAG) Munro, A., Johnson, M.C., Pizzini, Q.A., Surmon, D.S., Towne, D.M, & Wogulis, J.L. (1997). Towne, D.M., Munro, A., (1988). Smart Trainer AT (& FITS) Jin, L, Chen, W., Hayashi, Y., Ikeda, M. Mizoguchi, R. (1999); Ikeda, M. & Mizoguchi, R. (1994) XAIDA Hsieh, P., Halff, H, Redfield, C. (1999). Wenzel, B., Dirnberger, M., Hsieh, P., Chudanov, T., & Halff, H. (1998).
  • 72. AIED July ITS Authoring Tools Survey 72 People #2 DNA/SMART Shute, V.J. (1998). DOCENT (& Study) Winne P.H. (1991). Winne, P. & Kramer, L. (1988). EXPERT-CML Jones, M. & Wipond, K. (1991). GETMAS Wong, W.K. & Chan, T.W. (1997). GTE Van Marcke, K. (1998,1992). ID EXPERT (& Electronic Trainer) Merrill, M.D., & ID2 Research Group (1998). Merrill, M. D. (1987). IDE (& IDE Interpreter) Russell, D. (1988). Russell, D., Moran, T. & Jordan, D. (1988). MetaLinks Murray, T., Condit, C., & Haaugsjaa, E. (1998). SIMQUEST (& SMISLE) Jong, T. de & vanJoolingen, W.R. (1998). Van Joolingen, W.R. & Jong, T. de (1996). TRAINING EXPRESS Clancey, W. & Joerger, K. (1988).
  • 73. AIED July ITS Authoring Tools Survey 73

Hinweis der Redaktion

  1. 1. Still iformative stages, though some are comercial or precommercial 2. Overall question is how easy or cost effective can it be to build an ITS
  2. - This is dif between ITS and CBT - & dynamic generation of sequencing of instr. content
  3. ITSs are difficult and expensive to build
  4. -xamples: over representatin of systems am most familiar with (including the tools developed in our lab at UMass) does not reflect on quality of the tool.
  5. Many systems listed are only the latest in a lineage of attempts. So there have been many systems. Lets hope we have learned something!
  6. - dangerous to talk about other people’s systems! Errors and omissions - everything I say about a given system or category of systems is partly wrong or a simplification - alomost unapoligeticlly mention that I will use a dispropotional number of figures from the Eon authoringsystem developed at UMass. - bag of tricks vs. a shelf of tools - more an overview and look to the future than a critique - difficulty in coming with categories. - see my IJAIED paper for more details (**give URL ***) - you might be surprised, you might be dissapointed!
  7. Defiing characteristic: separates content from strategy (skip this slide?? But not title?).
  8. - no system does everything;
  9. - rest of categories build open these basics Student perspeive is like CAI
  10. - agin student persopective like CAI
  11. - operation and component identification are farily generic and ubiquitous tasks. - Perf. Montoiring and feedback tend to be straightforward: that is not the wing pressure cutoff valve; you should have checked the temperature first. - diagnosis is harder and authoring not as far along
  12. - in the “device simulation” categories we see how having a consistent underlyingrepresentation can lead to instructin for free. - Systems in this category capitalkize on this fact even more. - examples: facts are taught with mneumonic devices and drill and practice; concepts are taught with examples and anslogies; proecedrues are taught one step at a time. - for student these are similar to Multiple teachign strategies category: maindifference is for he author,sincle authoring is both easier and leff flexible. - mention “transaction shells” in Merril’s Transaction Theory
  13. - Takes the ideas just discussed a step further: rather than having templates for knowledge tpyes, create them for specific types of tasks. - more of a production line model - examples: - I’ll say more about special pruose vs generla purpose systm later template for nvestigaand decide Les LAThas conversational grammars ; for to train telephone customer service people
  14. -
  15. - sequencing PLUS navigation isses
  16. - most ITS authr tools don’t allow the author to build the interface design highly interactive screens from scratch - most mune because its what COTS authoring systems do well - but important.. - Items on screen are objects; animation and other visual effexcts are achievd by attaching properties of the object, such as its location, bitmap, or color, to dynamic simulation parameters.
  17. Visual and textual (next slide) method
  18. Note: multi knowledge types; objective type; difculty and abstraction level - separating the learning objectives from the knowledge - Merril’s Content types and - Uses Gagnes taxonomy of learning objectives
  19. - A connection has to be made between the abstract world of topics and concepts to the concrete world of the screens and digital resrouces that illustrate or (make them contrete) - separating the domain relationships (part-of) from the learning pedagogical relationshis (prerequisite); - Capabilities C, Objectives ), resources R, pedagogy P - Resorce model; Ax abstraction, CP particular case; An analogy; - Resouce types inmiddle; resource linking tool on right
  20. - graphic objects (e.g. pump) have alternate graphics representations. The representaiont (open or closed valve) is set according to an author defined ate variable. The author then defines rules like if linePressure > 3000 then control vallve position - open. - Theoryof operations in XAIDA--aircraft maintenance domain
  21. - - constraint based and event based e simulain vent processing -more complicted simulaitns than XAIDA,but harder to author
  22. - most Atools odn’t allow for authoring the teaching strategy… -- this is a structured interview tecnique
  23. - like authorware or icaon authors - more powerful andlfexible bt less usable
  24. - only system with customizable student model
  25. Authoring is: design, knowledge acquisition, and fabrication,
  26. - many of the tools involve helping authors visualize. - helping authors visualize conversational grammars
  27. - interactive knowlsdge elicitation - stepping the author through the design process by asking questions - previously we had VISUAL and Strucural organization; this is PROCEDURAL scaffolding; next slide is CONSTEAIN-BASED
  28. We’ll revisit scriptable and customizable later SKIP THIS?
  29. - so far descibed a wide variety of features and teqniques. - to AT has all of them. - each has strenghts a weaknesses - sometimes due to amount of effort or attention payed to certain components, but aslo due to philosophical differences.
  30. - strong in one area tends to be weak in some other
  31. - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility - like chandrasakaren’s abstract task types
  32. - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility
  33. - some think we can built systems an order of magnitude more powerful than traditional CAI with an order of magnitude less effort. I am hopeful yet skeptical. Seems only possible for template-based systems. - should be able to xpect authors to have some degree of training, sophisitcation, and dedication
  34. - who is going to tell these machines how to teach? (other suggestions: students, market analysis, legislative committees…) SKIP?
  35. - leave themost idfficult authoringto the experts. - one way to deal with the tradeoffs between usabilty and power - use a general purpose AT to build special purpose ones
  36. - category 1: toy domains - category 4: next slide
  37. Rovust: usu. included user documentatin, some level of suport etc. - not sure how mnay were built with SIMQUEST or Trainins express (and lwhether they were used)
  38. - Xaida figure: including trianing time - Demonstr8 informal study: moel tracing multi-column addition or subtractin tuutor built in < 20 minutes. - Summary: inconclusive estimates but all indications are that non-model tracing tutors can be authored in same time as traditioanl multimedia instruction - since building an ITS presumably takes additional effort for knowledge engineering (to acquire a model of the domain and/or teaching strategies) over and above traditional CAI, we must assume that there are real efficiencies involved in the modular and reusable nature the knowledge in ITSs. & generation of content on the fly -
  39. - “Existence proofs,” (see authoring tool USE above ) as alternative to evaluation - how do you evaluate something like an authoring tool? - no clear methods or metrics, but …
  40. - mostly the physical characteristcs part of the XAIDA shell Brenda wenzel Henry Half and colleagues
  41. -- to give an idea of the types of results researchers are getting. The individual results are suggestive only, as most have not been reproduced.