Fourth International Workshop on Teaching Analytics @ECTEL 2016

1. “Analysis of Human-to-Human
Tutorial Dialogues:
Insights for Teaching Analytics”
Irene-Angelica Chounta, Bruce M. McLaren
Carnegie Mellon University
Patricia Albacete, Pamela Jordan, Sandra Katz
University of Pittsburg

2. Dialogue as a means for learning
• We aim to develop an adaptive tutorial dialogue
system, guided by a student model that will support
students in learning physics

3. Research questions
• RQ1: What makes tutorial dialogue successful?
– Teachers’ adapt the level of discussion to the
student’s “zone of proximal development”
(Vygotsky)
• RQ2: How tutorial dialogues adapt to different
student characteristics and prior knowledge?
– Level of Control/ Level of Specificity (van de Pol)
– Contingent Tutoring (Pino-Pasternak)
– Cognitive complexity (Nystrand, Graesser)

4. Research questions
• RQ1: What makes tutorial dialogue successful?
– Teachers’ adapt the level of discussion to the
student’s “zone of proximal development”
(Vygotsky)
• RQ2: How tutorial dialogues adapt to different
student characteristics and prior knowledge?
– Level of Control/ Level of Specificity (van de Pol)
– Contingent Tutoring (Pino-Pasternak)
– Cognitive complexity (Nystrand, Graesser)
Research Objective:
What makes some tutor’s help
generous or stingy, easy or
challenging, straightforward or
“cognitively complex”? Level of Support
(LOS)

5. An example would be nice….
RQ: What minimum acceleration must the
climber have in order for the rope not to
break while she is rappelling down the
cliff? (You do not have to come up with a
numerical answer. Just solve for "a"
without any substitution of numbers.)
Chip: a = f / m
T: what's f ?
Chip : f = mg
T: just mg ? how many forces act ont he
climber ?
Chip : mg + T
T: is mg down or up ?
Chip : down and T is up
T: ok so now solve for a again plugging in T
and mg
RQ: What minimum acceleration must the
climber …….
Dale: 500/55 kg=a m/s^2
T: I don't agree - that's the acceleration
that just the pull from the rope would
produce (well once the units are
straightened out it would be). Think a little
more. What is the general rule for finding
acceleration from forces?
Dale : F/m=a
T: and what is the F there?
Dale : tension?
T: No.. the F in F=ma is always the net force
on the object (or group of objects). The
vector sum of all the forces on the object. I
prefer to say "Sum of F= ma" because it's
easier to get it right. So.. if she is sliding
down and the rope is just short of breaking,
what is the *net* force on her?
High performer

6. [How] can we group the features of dialogic
discourse to differentiate and operationalize the
“levels of support” (LOS)?
• Analyze human-to-human tutorial dialogues
• Build a coding scheme to operationalize Level of
Support
The mechanics of tutorial discussions

7. Method of the study
3 human-to-human dialogues on
Physics / 1 per overall learning
gains level [low/medium/high]
Level of Control
[3-step scale]
Question Categories
[18 types]
Level of
Specificity
[3-step scale]
Contingent
Tutoring
[binary]
LOS
Coding
Scheme

8. Coding scheme - Application
• 4 coders
• 3 dialogues [low/medium/high]
• 19 tutor turns
• Introduction to the coding scheme
• Rating handbook & template

9. Coding scheme - Results
Dimension Fleiss’ Kappa p-value
Level of control 0.404 4.13e-11
Question category 0.395 0
Level of specificity 0.141 0.0245
Contingency 0.0764 0.415
Lessons learned:
Still unclear how teachers effectively
regulate the level of support

10. Coding scheme: Lessons learned
• Not easy to interpret the goal of the intervention
• One intervention, multiple goals
• Crucial features:
– New content
– Feedback Information / Information meant to push student
forward
– Degree of detail

11. Coding scheme
Adaptation and Evaluation
Before After
Level of Control Information related to
student’s answer
(Backward/Forward)
Hints Provision
Question Category Question Category
Level of Specificity Feedback on Correctness
Information related to
feedback
Contingency Contingency

12. Application and Evaluation
• 10 human-to-human tutorial dialogues
(Physics) – 3 High, 3 Low, 4 Medium
• 2 raters per dialogue
• The raters were given a tutorial on the coding
scheme and detailed instructions
Dimensions Cohen's kappa
D1. Information Provision (B) 0.871
D1. Information Provision (F) 0.843
D2 –Hints Provision 0.843
D3-Feedback on correctness 0.826
D4-Information related to feedback 0.764

13. So you coded it. Now what?
• Provide appropriate, adaptive dialogic support:
What is appropriate for whom?
– guidelines for feedback provision
• Use dialogue-support mechanisms to inform
teachers:
– Concepts-coverage and content-contribution
– Provide hints vs. provide information
– Give away answer vs. set challenge

14. Chip & Dale: a use case example
Dale
Not so high performer…
Loves to ride his bike out in the sun!
Not good background knowledge in Physics
Chip
High Performer!
Loves to study!
Good background knowledge in Physics
Teacher: During the arrow`s flight, how does its horizontal
velocity change (increases, decreases, remains the same,
etc.)? Remember that you can ignore air resistance."
Student: decreases

15. Chip & Dale: a use case example
Dale
Not so high performer…
Loves to ride his bike out in the sun!
Not good background knowledge in Physics
Chip
High Performer!
Loves to study!
Good background knowledge in Physics
Teacher: During the arrow`s flight, how does its horizontal
velocity change (increases, decreases, remains the same,
etc.)? Remember that you can ignore air resistance."
Student: decreases
High Performers: Is there something
that could cause velocity to change?
What can this be?
Teacher’s Support:
- do not give away the answer, give
some time to the student to
construct the correct answer
- do not provide concrete
information
- provide hints
Low Performers : No, this is not right.
If there is nothing to cause velocity to
change (for example, some force),
then the velocity will remain the
same. Please list all forces that are
applied on the arrow while it is in
flight.
Teacher’s Support:
- provide explicit information
regarding background knowledge
- provide explicit instruction on next
steps
-use simple language

16. Future work
• Let the experts tell! Great time for a study 
– author dialogic support for various student types
– Provide alternatives to existing tutorial dialogues
– Ask for teachers’ input: what do you prefer?
what would you say?
why?

17. The end.
If you want to know more, get in touch!
ichounta@cs.cmu.edu
(plus cat pictures!)