Slides from our EC-TEL'18 Paper presentation. Full paper is available at https://dx.doi.org/10.1007/978-3-319-98572-5_19 Abstract: This paper presents a method for automated content analysis of students’ messages in asynchronous discussions written in Portuguese. In particular, the paper looks at the problem of coding discussion transcripts for the levels of cognitive presence, a key construct in a widely used Community of Inquiry model of online learning. Although there are techniques to coding for cognitive presence in the English language, the literature is still poor in methods for others languages, such as Portuguese. The proposed method uses a set of 87 different features to create a random forest classifier to automatically extract the cognitive phases. The model developed reached Cohen’s κ of .72, which represents a “substantial” agreement, and it is above the Cohen’s κ threshold of .70, commonly used in the literature for determining a reliable quantitative content analysis. This paper also provides some theoretical insights into the nature of cognitive presence by looking at the classification features that were most relevant for distinguishing between the different phases of cognitive presence.

1. Automated Analysis of Cognitive Presence
in Online Discussions Written in Portuguese
Valter Neto, Vitor Rolim, Rafael Ferreira, Vitomir Kovanovic, Dragan
Gaševic, Rafael Dueire Lins, and Rodrigo Lins

2. Online and BlendedLearning

4. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher
education. The internet and higher education, 2(2-3), 87-105.

5. Cognitive
Presence
Garrison, D. R., Anderson, T., &
Archer, W. (2001). Critical
thinking, cognitive presence,
and computer conferencing in
distance education. American
Journal of distance
education, 15(1), 7-23.

6. How to automatically identify
cognitive presence?

9. Method
Kovanovic, V., Joksimovic, S., Waters, Z., Gaševic, D., Kitto, K., Hatala, M.,
Siemens, G.: Towards automated content analysis of discussion transcripts:
A cognitive presence case. In: Proceedings of the Sixth International
Conference on Learning Analytics & Knowledge (LAK’16). pp. 15–24. LAK
’16, ACM, New York, NY, USA (2016)

10. Dataset
• Biology undergraduate level course
• 215 students
•Discussions over four weeks
• Question-answering discussion

11. Dataset

12. Data preprocessing
•Step 1: Divide the data into training and test datasets
(75% and 25%)

13. Data preprocessing
•Step 2: SMOTE algorithm

14. Data preprocessing
•Step 3: Feature extraction
• LIWC (Linguistic Inquiry and Word Count) tool
• Word counts which are indicative of different psychological
processes
• Coh-Metrix
• Metrics of text coherence
• Discussion context features
• Number of replies
• Message Depth
• Similarity to previous/next message
• Word embedding similarity
• Number of named entities

15. Data preprocessing
•Step 4: Random forest classifier
•Step 5: Optimization of the classifier

16. Data preprocessing
•Step 6: Feature importance

18. Discussion
• Higher levels of cognitive presence were related to
messages that are:
• Longer, with more words and sentences;
• Complex, with complex words (words bigger than 6
letters) and longer sentences
• With less lexical diversity, lexical diversity and givenness
• Use more third-person singular pronouns
• Use fewer question marks

19. Future directions of work
•What is the performance of the classifier for different
cohorts of students?
•How do social, cognitive and teaching presences
interact with each other?
•How can we aid instructors to create a environment
more productive discussions?
•How cognitive and social presences could be used for
self-regulated learning?

20. Automated Analysis of Cognitive Presence in
Online Discussions Written in Portuguese
Questions?