2021_06_30 «Increasing Student Interaction with Formal Languages using Programmed Instruction». Cliff Shaffer

1. Increasing Student Interaction
with Formal Languages using
Programmed Instruction
Mostafa Mohammed and Cliff Shaffer
Department of Computer Science
Virginia Tech

2. OpenDSA
2
OpenDSA is an eTextbook system
▪ Content
▪ Prose and other features of a textbook (sections, TOC,
index, etc)
▪ Visualizations: slideshows
▪ Auto-graded exercises: T/F, MCQ, proficiency
▪ Project started in Summer 2011
▪ Content for Data Structures, Algorithms, Formal Languages,
Translator/Compilers
▪ Infrastructure
▪ Generate books, store interaction clickstreams and course
data
▪ Ruby/Rails, MySQL database, LTI support

3. Goal
3
▪ Implement a new book for Formal Languages
▪ Address inadequate engagement and interaction
▪ Evaluate on-the-spot if the student understands or not the
concepts taught.
▪ Provide feedback on student responses
▪ Provide many auto-graded exercises and proficiency
exercises

4. Goal
4
▪ The new book should satisfy higher levels of the
Algorithm Visualization Engagement Taxonomy
▪ 1- No viewing (minimize)
▪ 2- Viewing (Visualizations)
▪ 3- Responding (answer questions about content)
▪ 4- Changing (proficiency exercises: demonstrate
algorithm behavior)
▪ 5- Constructing (proficiency exercises: build machines)
▪ 6- Presenting

5. Why Formal Languages?
5
▪ Formal Languages course is a mix
▪ Programming-like assignments
▪ Heavy mathematics

6. Why Formal Languages?
6
▪ Formal Languages course is a mix
▪ Programming-like assignments
▪ Heavy mathematics
▪ This means that FLA has challenges found in previous
studies
▪ Students should spend enough time to practice (similar to
recursion)
▪ Students should read and understand mathematical prose
(similar to Algorithm analysis)

7. Why Formal Languages?
7
▪ Formal Languages course is a mix
▪ Programming-like assignments
▪ Heavy mathematics
▪ This means that FLA has challenges found in previous
studies
▪ Students should spend enough time to practice (similar to
recursion)
▪ Students should read and understand mathematical prose
(similar to Algorithm analysis)
▪ Two key tools that we use:
▪ Programmed Instruction
▪ OpenFLAP

8. Programmed Instruction
8
▪ Graded sequence of controlled steps
▪ Students read small sentences
▪ Students must answer a related question
▪ Students can not skip the questions

9. Programmed Instruction Support in OpenDSA
9
▪ Modified OpenDSA slideshows to provide PI support
▪ Add a satisfaction criterion to advance to next slide
▪ Restrict the “forward” and “last slide” buttons
▪ Add questions space
▪ Store student's progress on server

10. Programmed Instruction Support in OpenDSA
10
▪ Auto-generating frames questions

11. OpenFLAP
11
● JFLAP drawbacks inspired us to develop OpenFLAP
○ JFLAP cannot be integrated into online eTextbooks
○ JFLAP lacks autograded exercises
● OpenFLAP allows
○ creating exercises,
○ auto-assessing, and
○ reporting the result to an LMS

12. OpenFLAP Exercises
12
▪ Auto-Graded Exercises

13. OpenFLAP Exercises
13
▪ Auto-Graded Exercises

14. OpenFLAP Exercises
14
▪ Proficiency Exercises

15. Implementation and Evaluation
15
▪ We developed the OpenDSA FLA textbook in three
stages
1. Standard prose-based ebook developed from prior
coursenotes
2. Replaced much prose with slideshows and replaced
paper-based homeworks with auto-graded exercises
3. Replaced slideshows with Programmed Instruction versions
▪ We taught students from each book and collected
exams results and satisfaction survey data

16. Stage 2: Visualizations book
16
▪ The book includes
▪ Proficiency Exercises
▪ Auto-Graded Exercises
▪ Visualizations
(slideshows)
▪ Static figures

17. Stage 3: Programmed Instruction book
17
▪ Implemented the library files to create the Frames
structure inside OpenDSA
▪ Refactored the Visualizations Formal Languages
book to be a series of framesets.
▪ adding questions that students must answer
▪ adding comments for students in correct/incorrect answers
▪ Store student progress in framesets

18. 18
Programmed Instruction book

19. Evaluation Protocol
19
● Spring 2018 (Control Group)
○ 44 students
○ Class notes
○ Some visualizations for computability theory
○ Supplemented with Linz FLA book
○ Use JFLAP simulator
○ Thirteen weekly paper-based homework sets
○ Three exams

20. Evaluation Protocol
20
▪ Spring 2019, Fall 2019, and Spring 2020
(Visualization Groups)
▪ 271 students
▪ Use the visualizations ebook
▪ Use OpenFLAP
▪ Fall 2020 and Spring 2021 (PI Groups)
▪ 135 students
▪ Use the PI ebook
▪ This book inherited all exercise from the visualization book

21. Research Questions
21
▪ What feedback do students give regarding their
experience with the Programmed Instruction book?
▪ How does performance on exams compare for the
three treatments (control, visualizations, and
Programmed Instruction)?
▪ How is performance on exams affected by attempting
to solve Programmed Instruction questions, or
spending more time studying from the books?

22. Student Satisfaction
22
▪ Book type preference
▪ 70% of students
preferred to have a
Programmed Instruction
eTextbook,
▪ 23% preferred to have
slideshows only, and
▪ 7% preferred the
traditional text.

23. Discussion
23
▪ Students found the exercises beneficial
▪ some of them asked for more exercises
▪ Students found the PI ebook more engaging and solving the
questions helped them test their understanding
▪ Students with negative comments indicate that some
changes would make the book more valuable:
▪ provide the correct answer after a few wrong attempts.
▪ have the information presented in summary, then add Frames with
questions for a detailed presentation
▪ display new questions in place of answered questions

24. Student Performance
24
▪ Comparing the overall exam grades

25. Student Performance
25
▪ Compare
grades per
concept
(Regular
Languages)

26. Student Performance
26
▪ Compare
grades per
concept
(Context-Free
Languages)

27. Student Performance
27
▪ Comparing grades per concept (Pumping Lemma)

28. Discussion
28
▪ The Visualization group students has better performance than
the control group in topics with exercises and visualizations
▪ In the remaining topics, students did not score better than the control
group
▪ PI frames and questions enhanced the understanding of the
contents and helped students to score better in most of the
Formal Languages topics.
▪ We need to pay more attention to topics that Programmed
Instruction ebook could not help the students to score better
▪ Language Hierarchy, Computability and Countability topics
▪ Programmed Instruction ebook has border effect on students
grades as it covers all course topics

29. Students Interactions with the eTextbooks
29
▪ Spent time in book (in hours)

30. COVID19 EFFECT
30
▪ Students during our study took the course exams in
different modalities
Semester
Modality
Book
Midterm 1 Midterm 2 Final
Spring 2018 Face to Face Traditional
Spring 2019, Fall 2019 Face to Face
Visualizations
Spring 2020 Face to Face Online
Fall 2020, Spring 2021 Online Programmed Instruction

31. COVID19 EFFECT
31
▪ Compare all groups in the theory part of the course
across 4 groups
▪ Control group (Spring 2018).
▪ Face to Face group.(Spring 19 and Fall 19).
▪ Partly Virtual group(Spring 2020)
▪ Programmed Instruction group

32. COVID19 EFFECT
32
▪

33. Conclusions
33
▪ Visualization eTextbook significantly improved students scores
in all topics covered by the visualizations and the exercises
▪ Programmed Instruction ebook further significantly improved
student scores in most topics covered
▪ eTextbook and tools were found to be more engaging than the
control group materials (traditional online textbook)
▪ Majority of surveyed students report that OpenFLAP exercises
are helpful for them to learn the material
▪ Majority gave positive feedback on PI concept
▪ Some students requested adding more exercises