Mlearn conference 2019 delft

1. Evaluating the Usability of a
Study Support Mobile App in
Higher Education
Mohammad Khalil,
Jacqueline Wong,
Martine Baars,
Farshida Zafar, &
Barbara Wasson
mLearn Conference, Delft
18.09.2019

2. Background & Motivation
◉ Self-study is an important factor in student success (Wijnen et
al., 2017)
◉ Self-study entails self-regulated learning (SRL) and use of
effective learning strategies
◉ Self-testing can be use for self-monitoring as part of SRL (Baars,
2014)
2
Jacqueline Wong, 18.09.19, Delft

3. Background & Motivation
◉ Smartphones account for 75% of mobile devices used in
educational context (Wu et al., 2012)
◉ Growing number of apps used for learning
◉ Few studies examined the usability of self-study apps!
◉ Research on usability for self-study apps can help to:
○ Improve apps and enhance user experience
○ Carry out finer research
○ Support teacher instructions based on the difficulties students are
facing
○ Implement support on app use for self-regulated learning, and in
turn, improve learning performance
3

4. “
4
Usability is the extent
to which a system, product or service can be used by
specified users to achieve specified goals
with effectiveness, efficiency, and satisfaction
in a specified context of use”
-ISO FDIS 9241-210
Jacqueline Wong, 18.09.19, Delft

5. What is the EUR App?
Jacqueline Wong, 18.09.19, Delft

6. 6
Eur App
● App developed in 2016
● Made by the students for
the students!
● Led by a lecturer in the
law faculty - Farshida
Zafar
● Formative assessment
tool with a focus on
gamification

7. 1. Teacher Perspective
7
● Teachers have control over the content.
● Teachers can drop in 250 multiple choice questions divided over categories
● Teachers can reference to a literature or video for feedback
● Quick changes can be done ‘on the fly’
● Simple descriptive analytics (successful students and percentage of
answers)
Jacqueline Wong, 18.09.19, Delft

8. 2. Student-face design
8
● Four game modalities (practice, compete, test, and battle)
● Availability of instant feedback in practice mode
● The app focuses on knowledge testing and entertaining
● Gamification elements are available (i.e., leaderboard, time
restrictions...etc.)
Jacqueline Wong, 18.09.19, Delft

9. 9
Eur App - Screenshots
Jacqueline Wong, 18.09.19, Delft

10. 10
Jacqueline Wong, 18.09.19, Delft

11. The Research
Study
Jacqueline Wong, 18.09.19, Delft

12. 12
Research Questions
◉ How often did the students use the app
and which mode was mostly used for
studying in the course?
◉ What were the students’ perceived
usability and enjoyment of the app?
Jacqueline Wong, 18.09.19, Delft

13. Sample Profile
◉ Survey sent to 800 students
◉ Year 1 Bachelor students from Erasmus University
◉ Course “Introduction to constitutional and administrative law”
◉ Response rate was 10.12%
◉ Female : Male (65% : 36%)
13

14. Criteria used for evaluation
14
◉ Learnability: How easy is the application to learn and
start working on.
◉ Efficiency: How efficient is the system in terms of
productivity
◉ Memorability: How memorable is the application when
a user leaves it for a while
◉ Errors: The application should have a low error profile
without catastrophic bugs
◉ Satisfaction: How pleasant is the application to use
and are the users satisfied of its functionalities?
◉ Enjoyment: Closely related to motivation by which we
examine whether students were enjoyed using the
application.
◉“It was easy for me to learn how to use
the app and navigate through it”
◉I was able to use the app without
frustration.
◉I remembered how to use the app when I
leave it for a while
◉The app gave error messages when I
made a mistake
◉I was satisfied with the app as a study
tool
◉I enjoyed using the practice mode of the
game app
Nielsen (1994) and
Mun & Hwang (2003)

15. How often did the students use the app and
which mode was mostly used for studying in the
course?
15
RQ1

16. What were the students’ perceived
usability and enjoyment of the app?
16
Criteria Cronbach’s a Mean
Learnability (4 items) .702 5.59
Efficiency (3 items) .716 5.43
Memorability (2 items) 4.31
Errors (2 items) 4.80
Satisfaction (4 items) .818 5.75
Enjoyment (5 items) .707 5.26
RQ2

17. Results
Mode usage
The practice and test
modes were the most
used ones. The battle
and compete modes
were not as highly used.
App usage
Around 50% of the
students used the app
once a week to practice.
60% of the students used
it to assess their
knowledge
Highest usability
criteria
Satisfaction (e.g., app is
user-friendly) was the
highest criterion of the
usability testing
Lowest usability criterions
Memorability (e.g.,
remembering how the app
works, app recalling same
page…) was the lowest criterion
of the usability testing
Other facts
Efficiency and enjoyment
of the app were
perceived on average by
the students
17

18. Study key outcomes
◉ Students used the app as more of a diagnostic tool rather than
a study tool
◉ Perhaps, students prefer modalities that provide feedback
rather than models with gamification
◉ According to Nielsen (1994), the application can be considered
“Stable”
◉ The gamification models of the app should be improved
18

19. Any questions ?
Thanks!
19
Mohammad Khalil
mohammad.khalil@uib.no
Jacqueline Wong
wong@essb.eur.nl
Martine Baars Farshida Zafar Barbara Wasson

20. Any questions ?
You can find me at:
◉ wong@essb.eur.nl
◉ mohammad.khalil@uib.no
Thanks!
20
Mohammad
Khalil,
Jacqueline
Wong,
Martine Baars,
Farshida Zafar, &
Barbara Wasson
You can find me at:
◉ wong@essb.eur.nl
◉ mohammad.khalil@uib.no