1. Lies, damned lies and statistics:
an evaluation of learning styles in AEH
Elizabeth FitzGerald (née Brown)
e.j.fitzgerald@open.ac.uk
Institute of Educational Technology

2. Introduction
• Adaptive educational hypermedia (AEH)
• Personalisation mechanisms
• Learning styles for user profiling
• Experimental methodologies
• Case studies: WHURLE and DEUS
• Conclusions and discussion

3. Adaptive hypermedia/AEH
• Main ways of adapting hypermedia:
– at the link level;
– at content level;
– via structural adaptation
– or presentational adaptation
• User modelling is common in educational applications and
tends to use either prior knowledge or preferred learning style

4. Why use learning styles?
• Apparently:
– Students learn better when using preferences in
which they're successful
– Students will be better learners when they can
expand these preferences
– When the learning experience accommodates
various preferences, more students will be
successful

5. Examples of learning styles
Reflector Activist
Theorist Pragmatist

6. Learning style models
• Dunn + Dunn
• MBTI (Myers-Briggs Type Inventory)
• Kolb’s experiential model / Honey + Mumford
• Riding's CSA (Cognitive Styles Analysis)
• Herrmann's brain dominance theory
• Multiple intelligences (7 or 9?)
• Biggs SPQ (Surface Processing Questionnaire)
• Field dependence vs field independence
• VAKT (Visual/Auditory/Kinaesthetic/Tactile)
• Wholist/holist vs analytic
• Which is best and how should it be used?

7. Experimental methodologies
• How can we evaluate the learning experience?
• Which methods are most suitable?
– Quantitative
– Qualitative
• Random controlled trials
• Statistical tools
… but we are dealing with people, who are
inherently complicated!

8. Case studies
• Two systems investigated:
– WHURLE: a revision guide, used with undergraduates
– DEUS: a web-based e-learning system, used with primary school
children
• Two different learning styles looked at:
– WHURLE: visual/verbal
– DEUS: global/sequential
• I wanted to find out if these user models were
beneficial, from a quantitative perspective

9. WHURLE experimental study
• User trials carried out with an online revision guide for a
taught module
• Over 200 university students involved
• Used a visual-verbal approach, investigating 2 variables:
– Visual and verbal environments
– Visual-verbal learning style of students
• Feedback/evaluation via assessment data,
questionnaires, interviews and log files

10. WHURLE revision guide:
system architecture
Chunks
+ + Lesson Plan
Links User Model
Adaptation Filter
Skin
Display Engine
The Title
Some text some text
some text some more
text some more text. Text
text text Some text some
text some text some
more text some more
text. Text text text.
Some text some text
some text some more
text some more text. Text
text text Some text some
text some text some
more text some more
text. Text text text.
Virtual Document

11. Learning styles in WHURLE
• Lesson plan produced for visual, verbal and no preference
users
• Chunks created: mix of visual, verbal, no preference or
universal
• Students filled in a learning styles questionnaire during
first log-in
• Users then randomly assigned to matched group,
mismatched group or neutral group

12. Student information
• Mostly 2nd/3rd year undergraduates
• Average age was 21, gender ratio of 3.6 males:1 female
• Out of 221 students who logged on at least once:
– 105 were visual
– 105 were bimodal (no preference)
– 11 were verbal

13. Screenshots
No preference
Visual environment Verbal environment
environment

14. What were we investigating?
• To see if matching or mismatching would make a
difference
• To see if there were any differences between students
with different learning styles
• To see if there were any differences between students
who used the different environments

15. Main findings of the study
• Matching or mismatching made no difference to student
performance
• No difference between students with different learning
styles
• No difference between students who used the different
environments

16. Statistical results
Hypothesis: Statistical significance:
F(4,210)=0.66, p=0.62,
H1: matched students will do
Wilks’ Lambda=0.98,
significantly better
partial eta squared=0.1
F(4,210)=0.66, p=0.62,
H2: mismatched students will do
Wilks’ Lambda=0.98,
significantly worse
partial eta squared=0.1
F(2,106)=0.46, p=0.63,
H3: one type of learning style is
Wilks’ Lambda=0.99,
more beneficial
partial eta squared=0.01
F(4,210)=0.59, p=0.67,
H4: one type of learning
Wilks’ Lambda=0.98,
environment is more beneficial
partial eta squared=0.01

17. Secondary findings
• No correlation between amount of use of the system and
student performance
• Qualitative data suggests students found it an enjoyable and
useful resource
• All students interviewed agreed that personalisation was
important
• Phase II, n=144. Testing mode choice (visual/verbal) and
switching behaviours when users allowed free reign via
analysis of web log data.
User choice and switching seemingly not driven by learning
style but by other factors (time available; preparation for
verbal task; boredom etc).

18. Conclusions from WHURLE trial
• Personalising for visual-verbal learning style does not
seem to have much educational benefit
• However, many students studying for Computer Science
degrees seem to be visual learners
• Students feel that personalisation in web-based learning
is important
• BUT… there are problems with visual/verbal learning
style, also with the type of users involved in the trial

19. Visual/verbal learning style
• Number of problems:
– Majority of people are mostly visual*
– Difficulty of determining granularity of a visual image
– Visual representations susceptible to bias/subjectivity
– Visual and verbal information equivalence
– Dual coding theory
• Different learning style selected for next trial:
global/sequential
* to a greater or lesser extent**
** depending on what’s being learned

20. The problem with participants
• Too many studies are tested on “pet students”
• Users for such studies should be based in the
“real world”
• Resultant data will be:
– More realistic
– More widely applicable
• As a consequence, AEH will become more useful

21. AEH use in schools
• School children can be termed “real” users
• Only limited amount of research done with schools
• Bajraktarevic, 2003: 2 studies involving GCSE pupils
– ILASH: summarising and questioning strategies supported by link and
text adaptation in user interface
– Global and sequential learning styles as adaptation mechanism
• Neglect of subsequent school-based research

22. Overview of DEUS case study
• Investigation into matching and mismatching pupils’ learning
styles with a web-based AEH system
• Initial pilot study and software testing by target users
prior to study
• Case study: Glapton Primary School, Nottingham:
– Mixed school, non-denominational
– Around 300 pupils on roll, aged from 3-11 years old
– Mostly white British background with English as first language
– Number of pupils with learning difficulties/disabilities
close to national average
– Attainment on entry to school is slightly below national expectations

23. Pilot study
• Few learning style models in AEH used with children
• Development of modified questionnaire to assess
visual/verbal and sequential/global learning styles
• Collected data on 2 occasions – 10-day gap
• Main findings:
– Visual/verbal style seemed fairly stable but skewed
– Sequential/global style was more changeable but normally distributed

24. Distribution of visual/verbal learning style
preferences in 10-11 year old children
14
Number of pupils 12
10 Time 1
Time 2
8
6
4
2
0
11 9 7 5 3 1 -1 -3 -5 -7 -9 -11
Learning style
visual verbal

25. Distribution of sequential/global learning style
preferences in 10-11 year old children
18
16
Time 1
Number of pupils 14
Time 2
12
10
8
6
4
2
0
11 9 7 5 3 1 -1 -3 -5 -7 -9 -11
Learning style
sequential global

26. Experimental design: overview
Pre-test
(assess domain knowledge and learning style)
Pupils assigned into experimental groups
2 hours of intervention
(including structured learning using
workbook exercises)
Post-test
(assess domain knowledge and learning style)

27. The creation of DEUS
• Previous work with WHURLE and visual/verbal learning
style
• New system conceptually similar to WHURLE: user model,
lesson plans and chunks
• Coded in PHP/XHTML and mySQL
• Learning styles adaptation mechanism contained within
lesson plan and reflected in navigation
• Range of multimedia used

28. DEUS login screen

31. Using DEUS
• Personalised login
• Topic choice in
negotiation with
teacher/school
• Navigation tailored
to global or
sequential,
matched or
mismatched
• Workbooks used
for tasks

32. Data analysis: overview
• 82 participants, aged 9-11 years old, over 3 weeks
• Assigned into 1 of 4 groups:
Environment Sequential Global
Style
Sequential ms mms m = matched
mm = mismatched
Global mmg mg
• Smallest group size = 15 pupils
• Assessment via pre-test and post-test quizzes
• Knowledge construction aided by specific tasks in
workbooks that matched the environment
(sequential/global)

33. Main hypotheses
• H0 – there will be no statistically significant difference in
knowledge gained between users from different
experimental groups
• H1 – matched pupils will do significantly better than
mismatched pupils
• H2 – mismatched pupils will do significantly worse than
matched pupils
• H3 – one type of learning style is better than another
• H4 – one type of learning environment is better than
another

34. Summary of main findings
• No difference between knowledge gains of
matched/mismatched pupils
• No difference between pupils with different learning
styles
• No difference between pupils using different
environments
• Also no difference between SEN/non-SEN pupils
• Hypotheses 1-4 rejected; revert to null hypothesis

35. Additional investigations: browsing time
• H0 – there will be no statistically significant difference in
amount of browsing time between users from different
experimental groups
• H1 – browsing times are different in matched versus
mismatched users
• H2 – browsing times are different for different learning styles
• H3 – browsing times are different for different learning
environments
• H4 – browsing times are different for SEN pupils compared to
non-SEN pupils.

36. Findings: browsing time
• No difference in browsing time between matched or
mismatched groups
• No difference also between pupils with different learning
styles
• No difference between SEN and non-SEN pupils
• Learning environment DID make a difference
– Global took less time to browse than sequential
– Only by about 15 minutes though
(3¾ hours compared to 4 hours)
– Supports H3 hypothesis

37. Correlations
• No correlation between knowledge gained and browsing
time
• Significant negative correlation (r=-0.364, p<0.001)
between post-test score and browsing time – attributed
to lack of motivation
• Significant positive correlation (r=0.555, p<0.001)
between number of pages visited and browsing time

38. Conclusions and discussion
• No particular benefit to using sequential/learning styles
as personalisation mechanism in AEH
• Outcomes of this study were based on “real users”
• Flaws of the study:
– Reliability and validity of modified questionnaire
– Dynamic nature of sequential/global learning style
– The Hawthorne effect
• Two overarching problems:
– Learning styles as a concept
– The process of learning: a “wicked problem”

39. Future considerations
• A fundamental need for high-quality research:
– Solid experimental design (see Robson, 2002)
– Address the limitations/controversies of any aspect of
the study (theoretical/methodological etc)
– Use “real” participants – not just those who are
convenient
– Statistical analysis and effect size

40. In this class there is a serialist pragmatist kinaesthetic learner
(who is also field-dependent, not to mention his MBTI) primarily
a convergent thinker, high on logico-mathematical intelligence
but low on linguistic intelligence, sitting next to a holist, reflector,
primarily visual and field-independent... who is also chronically
shy (no-one mentions that).
Even assuming that such things can be assessed with some
validity and reliability, which is itself far from clear — what are
you going to do about it? There are after all thirty other students
in the class, each of whom could be described in similar terms.
And two-thirds of them are female, and one-third male (two of
whom are gay).
Five of the class are from ethnic minorities, two are dyslexic, one
is visually impaired, and three are clinically depressed (although
only one of them knows it). Six are "mature" students — at least,
they are chronologically over 25.

41. Thanks for listening
Thanks and acknowledgements to:
Tim Brailsford; Tony Fisher; Amir Pourabdollah;
Adam Moore; Cees van der Eijk, Sue Jones and Shaaron Ainsworth.
Also to all those students and pupils who participated in the user trials.

42. References
• Brown, E. J.; Brailsford, T. J.; Fisher, T. and Moore, A. (2009). Evaluating learning style personalization in adaptive
systems: quantitative methods and approaches. IEEE Transactions on Learning Technologies (Special Issue on
Personalization) 2(1) pp. 10–22. http://oro.open.ac.uk/30224
• Brown, E. J. (2008) PhD thesis: The Use of Learning Styles in Adaptive Hypermedia.
http://etheses.nottingham.ac.uk/577/
• Brown, E., T. Fisher and T. Brailsford (2007) Real users, real results: examining the limitations of learning styles
within AEH. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (HT07), Manchester, UK,
10-12 Sept 2007, p57-66. [Awarded Theodore Holm Nelson Prize for Best Newcomer Paper]
http://oro.open.ac.uk/29998
• Brown, E., T. Brailsford, T. Fisher and C. van der Eijk (2007) Revealing the hidden rationality of user browsing
behaviour. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (HT07), Manchester, UK,
10-12 Sept 2007, pp85-94. http://oro.open.ac.uk/29999
• Brown, E., T. Brailsford, T. Fisher, A. Moore and H. Ashman (2006) Reappraising cognitive styles in adaptive web
applications. Proceedings of the 15th International World Wide Web Conference (WWW2006), Edinburgh, UK, 22-26
May 2006. http://oro.open.ac.uk/30002
• Coffield, F., D. Moseley, E. Hall and K. Ecclestone (2004) Learning styles and pedagogy in post-16 learning: A
systematic and critical review. Learning & Skills Research Centre.
• Coffield, F., D. Moseley, E. Hall and K. Ecclestone (2004) Should we be using learning styles? What research has to
say to practice. Learning & Skills Research Centre.
• Robson, C. (2002) Real World Research: A Resource for Social Scientists and Practitioner-researchers. Oxford, UK,
Blackwell Publishing.
• Weibelzahl, S. (2005). Problems and pitfalls in the evaluation of adaptive systems. in Adaptable and Adaptive
Hypermedia Systems. S. Y. Chen and G. D. Magoulas. Hershey, PA, IRM Press: 285-299.
• YouTube video: “Learning Styles Don't Exist” - Prof. Daniel Willingham: http://tinyurl.com/esteem-ls (7 mins long)