Uptake and Usage of Virtual Learning Environments: Findings from a longitudinal multi institutional student usage survey
1. Uptake and Usage of Virtual
Learning Environments: Findings
from a longitudinal multi institutional
student usage survey
Robert Cosgrave, Eamon
Costello, Tom Farrelly, Nuala
Harding, Theresa Logan-
Phelan, Claire McAvinia,
Fiona O'Riordan, Damien
Raftery, Angelica Risquez
EdTech 2012, May 31st-June 1st, NUI Maynooth
2. Virtual Learning Environments are now
core infrastructure in most tertiary
institutions.
But we don't know as much as we
should about how they are used, and
for what...
6. Higher Education Times
University of Leitrim has
best VLE!
President Gloats!
Researchers sacked!
No use of findings for PR
All published results to be anonymous by Institution.
Institutions letter coded A,B,C…
7. Three myths of the VLE:
"The choice of VLE will influence how I teach"
"Using a VLE will reduce attendance"
"If everything happens online, the face to face
teaching will be redundant."
8. Meet the Data
• 15,385 Individual Student Responses
• 12 Institutions
• 22 Survey Instances
• Data collection since 2008
• 78 questions (including subquestions)
9. How to slice it?
• Blackboard vs. Moodle
• Demographics
• Disability
• Time
10. What do you use the VLE for?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Keep track of Get copies of Get other Online Online quizzes Submit
class times lecture notes course discussions assignments
material about the
course
%
2008/09 2011/12
11. If you do not use [VLE], why not?
30%
25%
20%
15%
10%
5%
0%
My Lecturers don’t The information on I can’t access it I find it difficult to I don’t have
use it it isn’t useful use access to the web
%
2008/09 2011/12
12. I am comfortable with using computers
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
16-23 24-35 36-50 50+ 16-23 24-35 36-50 50+
2009/09 2011/12
Strongly agree Agree Neutral Disagree Strongly disagree
13. What is the most important thing you would
Lecturer use to seebig issue
like is a improved
More use of
Audio/Multimedia/
Podcasts
Better Usage by
Lecturers
More Reliable
Easier to Use
14. When asked what prevents them getting most
of their VLE...
"Answer is simple...The system as usual is perfect. What is
holding us back is the lecturers...I feel each lecturer
should be trained more"
"I think that if lecturers don't use Moodle effectively
(uploading class notes, including research resources,
case studies, discussions, etc.), Moodle wouldn't be a
useful or helpful resource for students'"
"All lecturers appear to have a different method of using
[VLE] and this should be standard"
15. Digital Literacy, Usability, Support
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
I am I have I think systems The VLE is The VLE is I can get
comfortable adequate like the VLE easy to use reliable adequate help
with using access (ie are helpful and support to
computers computers, use the VLE
bandwidth) to
the web
outside college
%
2008/09 2011/12
16. "I like the freedom of having access to info day or night."
60%
50%
40%
30%
20%
10%
0%
07:00 to 09:00 09:00 to 12:00 12:00 to 14:00 14:00 to 18:00 18:00 to 22:00 22:00 to 07:00
%
2008/09 2011/12
17. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
H
om
e
W
or
kp
la
ce
O
pe
n
Ac
ce
ss
La
b s
D
ep
a rtm
en
ta
lL
ab
s
2008/09
La
%
pt
op
s
In on
te
r ne
Ca
m
2011/12
tC pu
af s
és
ou
ts
i de
ca
m
pu
s
From where do you access the VLE
M
ob
ile
De
vi
c e
18. Things are getting better
My lecturers make good use of the VLE
100%
80%
60%
40%
20%
0%
2008/09 2011/12
Disagree/Strongly Disagree Agree/Strongly Agree
19. Students overwhelmingly like VLE's
Systems like VLE's are helpful
100%
80%
60%
40%
20%
0%
2008/09 2011/12
Disagree/Strongly Disagree Agree/Strongly Agree
20. 90%
80%
70%
60%
50%
"I find it makes note-taking less hard and
means I can concentrate more in the
40%
lecture…"
30%
20%
10%
0%
It mainly repeats It adds to what is It helps to clarify Using the VLE Using the VLE
what is covered in covered in class what has been makes it easier for helps me
class covered in class me to learn understand how
well I am doing
%
2008/09 2011/12
22. Getting notes on [VLE] makes me less likely to
go to lectures
50%
40%
30%
20%
10%
0%
Strongly Agree Agree Neutral Disagree Strongly Disagree
23. "I believe extra notes online aid the learning of
students and not deter them [students] from
coming to class"
"Sometimes lecturers don't use it to its full
potential…some are reluctant to put lecture
notes up in the flawed attempt to make
people come to class but if people don't want
to be there they just won't come either way"
24. Three myths of the VLE?
"The choice of VLE will influence how I teach"
No. You're only using it for content distribution anyway, and
there's no evidence of any VLE specific trends in usage.
"Using a VLE will reduce attendance"
Students disagree.
"If everything happens online, the face to face teaching will
be redundant."
Students disagree. They say it adds to what is covered in
class.
25. Three myths of the VLE?
"The choice of VLE will
influence how I teach"
"Using a VLE will
reduce attendance“
"If everything happens
online, the face to face
teaching will be
redundant."
26. So, how are we doing?
Our VLE's are doing OK!
They solve basic problems well
They are becoming more widely used
We need to help lecturers get off the bottom
rung where they need to
27. Next Steps
• Keep running the student survey
• Complement with a staff survey
• Consider system derived metrics and
learning analytics type data
• Integrate with National Strategy regarding
student engagement
28. Longitudinal Studies never end...
New institutions welcome.
Contact any of the authors to get involved.
Get involved anytime.
Plenty of work to go around!
29. Dissemination
Cosgrave, Robert ; McAvinia, Claire ;Risquez, Angelica ;Logan-Phelan, Theresa (2008): Uptake
and usage of virtual learning environments: findings from a multi institutional student usage
survey. Second Annual Conference of the National Academy for Integration of Research,
Teaching and Learning. WIT, 14-15th November
Cosgrave, Robert ; McAvinia, Claire ;Risquez, Angelica ;Logan-Phelan, Theresa; Farelly, Tom;
Harding, Nuala; Cooper, Rosemary; Vaughan, Noreen; Palmer, Marion (2009): Usage and
uptake of virtual learning environments and technology assisted learning tools: Findings from a
multi-institutional, multi year comparative study. EdTech 2009, May NCI
Cosgrave et al, (2011) Usage and uptake of Virtual Learning Environments and Technology
Assisted Learning Tools: Findings from a multi-institutional, multi-year, comparative study, The
All Ireland Journal of Teaching and Learning in Higher Education (AISHEJ)
http://journals.sfu.ca/aishe/index.php/aishe-j/index
Publication pending
EDIN publication (June 2013)
Emerging Issues III – From Capacity Building to Sustainability
Hinweis der Redaktion
Presenters: roughly Nuala Intro/Myths/Backgound/Meet the Data, then Damien Data slice to Students feel the VLE is a genuine learning aid, then Angelica Myths/Summary/Wrap up Claire McAv & Theresa also for questions
VLE's are core infrastructure. Allowing for costs of support, licenses, staff, helpdesk, major investment on VLE's in Ireland. Higher Education and now also used in secondary schools. Educational technologists talk about new technologies – VLE is seen as a common platform/baseline technology
Like a lot of things, it all started in Cork. NAIRTL funding Rob Cosgrave. If conference was in Cork he would be here but passport is out of date!! But not much use having single institution data. Much more useful to be able to compare different institutions, and everyone had the same problem and questions.
People are involved because the information is useful in guiding their own management of the VLE, not because they wrote a grant People contribute data, and work on the analysis according to their level of interest, time, capability Many hands make light work! Sharing in the spirit of collegeality and fair usage. Currently under the umbrella of ILTA Strategy on Research.
Because we had to share data between multiple, competing institutions, there were complexities - data protection and so on, to be adhered to. Row level data had to be shared for the data to be of use, but no student identifiers, emails etc were to be shared. Doug Belsow – social media putting social back
Why? Otherwise, institutions who do not have heavily resourced VLE, who maybe have the most to gain, will be blocked from being involved because it ‘looks bad’ We get into a whole PR / Benchmarking world and lose the benefits. Institutions who put in data can see who is who, so they know who to talk to, but publication and circulation of that information is strictly limited. One could, if industrious, figure out who is who from the authors, data (Cluedo joke – NUIM, in the library, with a webserver…) Also data protection issues – individual identifiers (Students IDs, eMails) are not pooled. Each institution remains legal custodian of it’s own data.
Here's three myths about VLE's which our data will touch on...Keep these in mind as we look at the data [Suggest keep in intro for Nuala before slice slide - Damien]
Fairly large dataset by past standards and growing all the time. Many participating institutions have run the survey 2/3 times, others only once. Image from http://www.c0d3m0nk3y.com/gallery/communication-node.html licenced for reuse. Cool!
Lots of way to cut the data, but today, where data allows, we're focusing on time. We have a number of institutions where we have two surveys, one from an early 08/09 group, and one in a later, 11/12 group. This allows us to compare like for like in two different time groups (with just over 4000 students in each group), so when I put up charts with two time bars, that the subset we're seeing. We're not going to talk, ever, about which institutions are in what data due to the confidentiality issues. [HANDOVER POINT - Nuala to Damien]
So, this year, for the first time, we're starting to get data from enough institutions on a long enough time base to talk about changes over time as well as gross patterns we discussed last time. Getting copies of lecture notes and other materials is the big usage modality. So is this VLE thing just a souped up content management system? Is that bad? Use of a VLE as a simple transmission systems solves a big problem - paper management. If it's transmitting information that can then be discussed in class instead of simply read out, that can still be pedagogically positive. Basic usage can still be a very good thing, let's not knock a system that solves a simple problem well, just because groovy constructivist things like online discussion remain relatively low (and probably much lower in reality, as this is a sample, not a census!) Usage of everything is up, but submit assigments seems to be a notable one upshifting. We think this is driven by integration of systems like turnitin to VLE's, so making it a more fluid process to submit material online.
Some respondents didn't use the VLE, and here's why - lecturers don't use it! Note the numbers here a relatively small, so everything except lecturer non use is trivial. The decrease in that over time is small, but encouraging. Qualitative data supports this - when asked what prevents them from getting the most out of their VLE 42% said it was lecturer non use or poor use that prevented them using VLEs effectively (compared with 49% in 2008) Note the scale, going up to 30% not 100%.
Digitial divide issues that were front of mind in '08 are fading away - see the change in those agreeing with the statement between the early and late groups - especially in the 50+. There will always be some who are not comfortable. Of course, comfortable here is self defined. Some people aren't comfortable using a machine they didn't hand build, others feel 'comfortable' because they can do facebook.
Talk about barriers to access - lecturer non use In context of results the satisfaction question
inconsistent and poor lecturer usage comes up as a theme in comments. Even students who say they find the VLE hard to use, often remark in comments about topics specific to poor or inconsistent lecturer usage
q9 Other Barriers have reduced over time. This shows the 'agree' and 'strongly agree' % for the statements shown. You can see conventional digital literacy issues around computer use are basically gone (note - among respondants - again with the self selecting sample - but it was self selecting in both years) bandwidth and access is muchimproved,usability, reliability - but no big change in the help and support area, where it's only a strong half.... Qualitatively students are saying a major barrier was access e.g. systems down; password problems; broadband issues (36% of qualitative responses related to access issues; compared to 38% in 2008) Navigation was also cited by many as a barrier (up this time at 18%; compared with 10% in 2008) - navigation refers to students difficulty in finding and accessing resources Student quote 'In the second semester i took me ages to find my modules as the first semester ones are still on the home page with a two of my new modules but the rest are under 'my sites' which can get confusing' Ease of use is often around lecturer usage rather than system
The help and support issue is interesting, as increasingly people are accessing the system across a broad range of hours, with big growth in access outside of conventional office hours - a shift towards the 24 hour university. Again, system logs would be great for this kind of data - superior to survey.
The other 'pattern of access' shift in in devices. Access devices and locations have changed. Everything is up, but mobile devices is the big shift since 08. We expect to have to grow further in '13 and on, and will drop some items like internet cafe's. This is an area where we could really use system derived data on accessing devices - but noting that system won't see offline access, like people mailing docs to a kindle to read them offline.
rising...higher agreement on positive lecture great positive improvement...but why? Moving from early majority to late majority Could be just older lecturers retiring off. Could be upskilling over time, helped along by VLE's getting easier to use, and our wonderful professional development programmes.
q9c [Damien - Rob, some postiive student quotes if you want: I think blackboard is a fantastic resource It informs you of what you need to cover in the modules and keeps you on the right track, without it one would be lost. Access to lecturers, classmates and learning materials are the most useful]
Students seem to feel the VLE is a genuine learning aid. This shows the agree+strongly agree % for some questions on that - now the year comparison isn't so important here, you can ignore that as the shifts are quite small. But look, three quarters of students say it helps to clarify what is covered in class. Close to 70% say it helps them to learn. Almost half feel it helps them understand how well they are doing. This is good news - it tells us that students, or at least the ones who care to tell us, like this thing and feel it is of benefit. That said, 40% agreed that it mainly repeats what is in class, but that isn't so awful either.
q10a, 10b, 10f Changes over time not so singificant here, but a substantial group finds the VLE makes it easier to contact their classmates, even in the facebook age. Small groups find it helps them contact their lecturers. We can be happy with that - if these people might otherwise be lurkers or it empowers them to engage with the lecturer, super. A high positive response isn't always the right answer.
mythbuster - excuse number one Lecturers cite declining attendance as the a common reason not to engage with VLE's. Students, at least those who responded, said it didn't matter. And even if it did, so what? If you can cover the transmission stuff in the VLE, maybe you can do something else with the lecture (the flip!) So 62% disagreed or strongly disagreed that getting the notes made them less likely to go to lectures, vs 21% ayes. q10e
Qualitative data
Here's three myths about VLE's which our data will touch on...
1.No. You're only using it for content distribution anyway, and there's no evidence of any VLE specific trends in usage. 2.Students disagree. 3.Students disagree. They say it adds to what is covered in class. Image – disney.com
Added value cannot be underestimated Mission critical – there would be uproar if removed Assessment submission – turnitin – promoting academic integrity
Review and keep running the student survey - review every year. Small tweaks needed, but keep to a minimum to ensure continuity. As we move on, we find things that seemed like good questions in 2008 aren't anymore, and assumptions implicit in the questions, which we didn't notice at the time, become obvious and need to be unpacked (for example, the full time, module ased course assumption) National Strategy – evidence of student engagement We designed a staff survey in 08/09 but never got in going. Time to revisit this over the summer and add the staff perspective Survey data is very limited - poor responses rates, self selecting samples and so on. One way to address that is to supplement it with system derived data. We would need to derive common metrics which can be drawn from different VLE's in a consistent and comparable way. That could be a big job.
Finally, our project is open, and we always welcome new participants. Join us, we don't bite. Please feel free to approach any of us about coming into the workgroup. We are very informal, don't bite etc.
