2. Social surveys in EMMA
Chiara M Ferrari,
Group Director, social research department at
Ipsos Public Affairs, Milano
part of the Ipsos Group
3. Participants Profiling: HOW IT WORKS
REGISTRATION FORM
Email __________
Password_________
Age__________
Gender___________
______
Please complete your registration
answering to a short questionnaire:
http://research.ipsosinteractive.com/mrIWeb/mrIWeb.dll?I
.Project=S1036981&Id=101&i.User1=39253324&i.User2=CIB
eCCBd
INDIVIDUAL Link matched to the
users ID , which redirects to the
IPSOS platform
REGISTRATION FORM
Thank you!
Your registration is
completed.
The registration data are recorded in
the EMMA SERVER . Each
student/batch of data gets an
individual and unique ID which will
allow to connect the data from
different sources: surveys, tracking and
learning analytics
Data on expectations will
be stored in the IPSOS
SERVER and will receive the
same individual and unique
ID, to allow connection and
subsequent analysis
QUESTIONNAIRE ON EXPECTATIONS
4. •Email
•Gender
•Age
•Employment status:
• Working full time
• Working part time
• Looking for a work
• In full time education
• Retired
• Other “don’t work”
•Education Level: (ISCED LEVEL)
What is the highest level of education that you have completed?
Level 0 - [Pre-primary education]
Level 1 - [Primary education or first stage of basic education]
Level 2 - [Lower secondary or second stage of basic education]
Level 3 - [(Upper) secondary education]
Level 4 - [Post-secondary non-tertiary education]
Level 5 - [First stage of tertiary education] University/college
Level 6 - [Second stage of tertiary education] Doctorate level
• Country of birth/Country of residence
• Mother tongue/language used in studies/language of
choice for EMMA
Participants Profiling: WHAT WE COLLECT
5. Expectation questionnaire optional and one-off, a red toolbar which is there
during the entire user navigation and reminds the student he /she should complete
the survey
6. Specific questions about the EMMA offer
how did they hear about EMMA
what are the main reasons to participate, e.g..
• experience a MOOC
• look and experience the design of the MOOC
• gain knowledge in a new domain
• keep up to date with my domain
• want a certificate
personal learning objectives single course vs. wider path of studies (e.g. several courses in connection) vs. focus on a
specific topic / theme vs. general info to gain general knowledge on a topic / theme vs. …
Previous experience with e-learning (as a learner)
How much time they expect to invest in this MOOC
In which of the learning activities they expect to take part
In which of the interactions they expect to participate
How much time they have available for this MOOC
From what location (home, university, work, library, ...)
On which support (PC, laptop, tablet, mobile…)
Expectations from EMMA users
8. Was this in context of your formal education?
Have you had any previous experience with e-learning?
What is your overall level of satisfaction with e-learning so far?
90% 10%
Very satisfied = 29%
E-learning experience
9. What are the reasons why you decided to register on EMMA?.
Reasons for visiting EMMA
10. Mini questionnaire -> optional one for each MOOC, it is entered freely by the teacher
as a simple unit (see the picture below). Teachers can decide when and where they
submit it
16. 16
YOUR EXPERIENCE WITH THE MOOC E-LEARNING
The course in general
The audiovisual material
Recordings of expert interviews
Masterclasses on specific topics
Interaction with other learners
Interaction with teacher/tutor
Learning activities and assignments
The opportunity to keep up with the domain “e-learning” in a flexible manner
12%
38%
25%
12%
38%
38%
12%
25%
25%
38%
25%
38%
12%
12%
12%
12%
12%
12%
38%
50%
50%
50%
62%
62%
38%
50%
Strongly like Like Neutral Dislike Strongly dislike Didn't use/ does not apply
Base casi: totale rispondenti
Please rate from "strongly dislike" to "strongly like" how you would rate your experience
with this MOOC
17. 17
50%
38%
12%
Fully in English
Fully in Dutch
Partly in English and partly in
Dutch
TRANSLATION FUNCTIONALITY
Base casi: totale rispondenti
a) Let’s now discuss the translation functionality. Did you follow the course…?
b) How much do you use the translation functionality? I used it…
25%
75%
Always
Sometimes
Not at all
c) How much do you agree or disagree with the following statements
about the translation functionality?
It’s effective
It needs more languages to be really
effective
It’s totally new and different
It enhances the overall value of the
EMMA platform
It makes EMMA a truly European
experience
50%
50%
50%
100%
50%
50%
50%
50%
50%
Completely agree
Fairly agree
Neither agree nor disagree
Fairly disagree
Completely disagree
18. 18
TASKS AND ASSIGNMENTS
Base casi: totale rispondenti
How much do you agree or disagree with the following statements about the tasks and the assignments
which were requested?
They are a good way to experience the course
They are too time consuming
I found them educational
They are engaging
They motivated me
They are a good way to create a collaborative activity
They are a good way to improve individual learning
I received the expected feedback
I liked receiving feedback from other participants
I liked giving feedback to other participants
If I was left behind with the tasks, I was still motivated to continue with the course
If I was left behind with the tasks, I wanted to quit the course
12%
12%
12%
12%
12%
12%
12%
25%
25%
38%
38%
38%
25%
50%
12%
12%
25%
12%
25%
12%
12%
12%
25%
12%25%
12%
38%
12%
12%
12%
12%
12%
25%
25%
12%
12%
12%
12%
12%
38%
25%
38%
38%
25%
25%
25%
62%
62%
50%
38%
38%
Completely agree Fairly agree Neither agree nor disagree Fairly disagree Completely disagree Does not apply
19. 19
ADVICE
Verbatim
I engaged in this MOOC but there is no subtitle in English.
Part of the course was in Dutch and not in English as expected. The very first video about what is e-learning was in Dutch with no
subtitles. There were too many videos in Dutch so even if you activate the subtitles, it is not easy to follow as there is too much text and
what is being said is not well translated. There were some bugs with the activities so after the week 2/3, I just left it as for me, if the
technology supporting the course is not reliable, then, it is a bit pointless. I guess that this platform needs a lot of improvements as well
as the course design. From the moment you accept any participant, just make sure that all the material is accessible in proper English
and that the activities can be made without any technical hassle. Overall, navigating into the platform is not intuitive, it takes too long. I
also felt that there was too much text and that some resources was out of date. Another thing, the platform is not supported well by all
the browsers
When I started the course, the automatic translation was not working well and some functions were not working well. This was
commented by the participants, but the facilitators were not reaction on this in the first days. After two days it was still not working, I
somehow stopped the course, because I simply forget to check in ....
cursus eigenlijk niet gevolgd (andere prioriteiten)
so bad because mi mother tongue is spanish for that reason i dont understood nothing
The course is not available after the end date which is a significant weakness for those pushed for time and who can't complete the
course in the timescale advised. I wasn't able to complete the course and the platform doesn't tell you this.
Timing was niet ideaal. Start tijdens paasvakantie, daarna een weekje conge en dan was het moeilijk om nog op te volgen!
Een enkele expert zou moeten werken aan haar presentatietechniek. Zij komt ondanks haar hoog opleidingsniveau tamelijk moeizaam
aan een vlotte redenering.
In your opinion, which are the weaker aspects of the course? Why?
20. 20
YOUR POINT OF VIEW
Is there anything you would like to report or highlight, after having analyzed the data?
Should you re-run this same MOOC, is there anything you would do differently? Please explain
_____________________________________________________________________________________________________________________________________
_____________________________________________________________________________________________________________________________________
_____________________________________________________________________________________________________________________________________
_____________________________________________________________________________________________________________________________________
21. Exit questionnaire -> optional and and one-off, automatically embedded at the end of
each slide in all units of the last lesson of each course
22. 65% 66% 57% 61% 73% 71%
It's engaging It's immediate
It's self-explaining It's rich in offer
It offers cultural diversity It offers flexibility in learning paths
disagree
9% 9% 22% 13%
7% 11%
The EMMA platform is … (% of agreement)
23. 35%
46%
13%
5% 1% Extremely good
Fairly good
Neither good not bad
Fairly bad
Extremely bad
Evaluation of the experience
1%3%3%
3%
3%
3%
4%
4%
4%
4%
6%
9%
9%
11%
11%
21%
Sport
Arts
Education
Law and commerce
Marketing
Creative writing
Social Science
Sciences
Literature
Languages
Economy/Finances/taxes
Computer /web programming
Communication culture
History
Health/medical school
Food and wine
Areas of interest for the future
24. Learning Analytics in EMMA
Kairit Tammets, PhD
Senior researcher, head of the studies
Centre for Educational Technology
School of Digital Technologies
Tallinn University
25. Why Learning Analytics?
• LA allows to understand:
– how MOOC participants learn, engage with the
content and with the other participants;
– how MOOC designers and facilitators can improve
their course designs and pedagogies
• LA was implemented in EMMA as an
additional evaluation tool;
26. Why Learning Analytics? (2)
• Current approaches of learning analytics in
MOOCs focus on:
– Calculation of drop-out rates;
– Prediction of drop-out;
– Clustering on participants based on their online
behavior
• EMMA approach:
– To establish feedback loop between facilitators and
participants in the form of dashboard;
– To encourage evidence-based course design and
pedagogical interventions
27. Learning Analytics methodology
• What are the research questions?
• What data is needed for answering research
questions?
• How this data can be collected?
• How to meaningfully present data?
• How to design feedback loop?
• How to learn from the process?
28. How Learning Analytics work?
• Tracking tool – tracks and collects users
traces;
• Learning Record Store – stores learning
experiences (xAPI statements) and sends them
to dashboard application (Learning Locker);
• Dashboard – visualization of the interactions
31. EMMA dashboards
• Learner view:
– Progress based on access of materials and
submissions of assignments;
– Overview of usage of the learning resources;
– Social network analysis based on comments and
posts;
– Activity stream;
– Participants – enrolled MOOC participants
35. EMMA dashboards
• Facilitator view:
– Overview of different interactions – accessed
materials, time spent on activities, submissions
and posts
– Overview of the usage of the learning resources;
– Social network analysis based on comments and
posts;
– Activity stream
– Participants – enrolled MOOC participants
40. Addition to dashboard
• Additional analysis:
– Cluster of the participants:
• Enrolled;
• Observers;
• Contributors;
• Active
– In-depth overview of usage of the course content
– Additional inquiries by the MOOC facilitators
42. What next?
• Beyond the dashboards – what happens after
using the dashboards, how MOOC designers
improve their pedagogies, technologies and
media
• Combining survey data and LA data;
• Translation of the numbers for contributing to
European MOOC model