2013 07 05 (uc3m) lasi emadrid jgzubia deusto learning analytics primeras experiencias weblab deusto

1. Learning analytics and remote
experimentation
Universidad de Deusto (Bilbao, Spain)
IQS ‐ Universidad Ramon Llull (Barcelona, Spain)
Aristos Campus Mundus

2. Presentation
• International Campus of Excellence: Aristos
Campus Mundus
• University of Deusto – DeustoTech : Pablo
Orduña, Aitor Almeida, Mari Luz Güenaga,
Susana Romero y Javier García Zubía
(zubia@deusto.es)
• IQS – Ramón Llull University: Jordi Cuadros y
Lucinio González

3. Agenda
• What is a remote laboratory?
• Remote lab examples. Data mining.
• Case of study: Analysis of opinion surveys of
users of WebLab‐Deusto
• Case of study: Analysis of learning outcomes
with VISIR
• Conclusion and future work

4. What is a remote lab?
• It allows the user to experiment out of the lab,
as being there.
• Internet are the hands and the eyes of the
user: student, teacher, citizens.
• The experience can be worse than in the lab,
but it can be similar, or even it can be better
because the user controls more the
experiment. But, what about the feeling and
the inmersion?

5. What is a remote lab?
• Advantages: organizational, academical,….
• It is cheap because it can be shared among
different institutions.
• It is always available.
• It is powerful because the teacher can add all that
he wants
• The teacher can track everything done by the
user.
• Disadvantages.

6. Examples of remote labs
• WebLab‐Deusto (Spain), VISIR (Sweden), iLAB
(USA), labShare (Australia), etc.
• WebLab‐Deusto offers experiments with basic
analog circuits, basic digital circuits,
microcontrollers, robots, incubator,
Archimedes’ principle…..
• Weblab‐Deusto also offers the system to
schools, etc.

7. Examples of remote labs
• WebLab‐Deusto Robot:
• Move, upload, program…

8. Examples of remote labs
• VISIR allows to mount circuits and analyze them

9. Data mining in WebLab‐Deusto

10. Data mining in WebLab‐Deusto
• Tracking: who, when, where, how long, web
browser, IP, what device, experiment, state of the
queue, Facebook…. And the most important
• What commands were executed and with what
results
• The teacher can reproduce the session to see and
analyze the profile of the user: knowledge and
learning
• Since 2009 more than 55 000 sessions.

11. Data mining in WebLab‐Deusto
• Is it useful? Is it usual?

12. Analysis of the opinion survey
• At the end of the semester the students
answer to a survey with 18 questions.
• The first survey was designed on 2003 with 15
questions.
• In the year 2008‐2009 the current survey was
designed merging the previous survey with
the results of the eMERGE project (Bordeaux)
and Ma&Nickerson.

13. Analysis of the opinion survey
Q1. WebLab helps me with the subject, the concepts, the exercises, projects, etc
Q2. Using the WebLab, I fell that it is real and it is not a simulation
Q3. It is a good idea to extend this WebLab to all the students
Q4. I have enjoyed using the WebLab
Q5. WebLab is easy to use
Q6. The quality of the WebCam is good
Q7. The different I/O devices (switches, buttons, etc.) are easy to use
Q8. I don’t have problems with the assigned time
Q9. The I/O devices implemented are well selected
Q10. Even being far from the WebLab, I have felt that I control it
Q11. I would like to use the WebLab in others subjects
Q12. I am satisfied with the WebLab
Q16. The user’s manuals are good and clear.
Q17. I have been motivated by the WebLab to learn more about the subject
Q18. The WebLab is a high quality software (access, management, availability, etc.)

14. Analysis of the opinion survey
• Is it correct?
• Is it useful and meaningful?
• Does it contain information?
• For what is it being used by WebLab‐Deusto?
• Learning analytics

15. Analysis of the opinion survey
• X‐O:
– Experience
• The students use the weblab in a regular way in two
different subjects since 2008 to 2012
– LP: Programmable logic (VHDL&CPLD) (123 surveys) Third
year.
– DE: Electronics design (VHDL&FPGA) (73 surveys) Fifth year.
– Opinion survey
• 18 questions, Likert pseudo‐scale with 5 levels

16. Analysis of the opinion survey
• Fiability
– Cronbach’s alfa
• Construct Validity
– Is there an internal consistency?
• Item analysis
– Homogeneity index

17. Analysis of the opinion survey
• FIABILITY
• Cronbach’s alfa
• Result for the survey: 0.78
• It is a medium to high value

18. Analysis of the opinion survey
• Construct validity
• Designed substructure (by me)
– UTILITY: Q01, Q03, Q11, Q12, Q17
– USABILITY: Q04, Q05, Q07, Q08, Q09, Q16, Q18
– INMERSION: Q02, Q06, Q10
– OTHERS: Q13, Q14, Q15

19. Analysis of the opinion survey
• FIABILITY
– UTILITY α = 0,74
– USABILITY α = 0,64
– INMERSION α = 0,40
– OTHERS α = 0,27
• Looking to the last two values it seems that
these two factors do not exist

20. Analysis of the opinion survey
Q02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16 Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15
Q02 1,00
Q06 0,20 1,00
Q10 0,30 0,07 1,00
Q04 0,25 ‐0,07 0,19 1,00
Q05 0,24 0,11 0,29 0,19 1,00
Q07 0,21 0,20 0,21 0,10 0,39 1,00
Q08 0,15 0,09 0,19 0,21 0,08 0,14 1,00
Q09 0,17 0,15 0,26 0,13 0,29 0,37 0,15 1,00
Q16 0,32 0,21 0,25 0,15 0,34 0,15 0,09 0,16 1,00
Q18 0,32 0,26 0,36 0,35 0,27 0,18 0,07 0,28 0,37 1,00
Q01 0,43 0,21 0,34 0,37 0,41 0,19 0,11 0,24 0,35 0,58 1,00
Q03 0,27 0,05 0,20 0,27 0,38 0,12 0,09 0,21 0,22 0,44 0,46 1,00
Q11 0,04 ‐0,03 0,31 0,37 0,16 0,19 0,05 0,13 0,20 0,36 0,36 0,28 1,00
Q12 0,25 0,17 0,46 0,34 0,39 0,37 0,15 0,33 0,29 0,59 0,61 0,44 0,43 1,00
Q17 0,25 0,00 0,12 0,50 0,07 0,06 0,14 0,07 0,22 0,31 0,38 0,18 0,41 0,27 1,00
Q13 0,05 ‐0,08 0,04 0,10 0,13 0,09 0,10 ‐0,01 ‐0,01 0,02 0,02 0,08 0,17 0,13 0,05 1,00
Q14 0,08 0,09 0,12 ‐0,07 0,12 0,03 0,06 0,19 ‐0,02 0,13 0,04 0,13 ‐0,01 0,08 ‐0,12 0,35 1,00
Q15 0,02 0,03 ‐0,01 ‐0,06 ‐0,07 ‐0,01 0,05 ‐0,09 ‐0,04 ‐0,14 ‐0,10 ‐0,10 ‐0,04 ‐0,18 0,01 ‐0,13 ‐0,09 1,00

21. Analysis of the opinion survey
• Items‐factor correlations
Q02 Q06 Q10 Q04 Q05 Q07 Q08 Q09 Q16
r_IN 0,71 0,68 0,64 0,17 0,31 0,31 0,2 0,28 0,38
r_US 0,42 0,23 0,44 0,56 0,63 0,57 0,47 0,58 0,56
r_UT 0,33 0,1 0,4 0,53 0,37 0,26 0,15 0,26 0,35
r_ZZ 0,08 0,02 0,09 0 0,13 0,07 0,11 0,08 ‐0
Q18 Q01 Q03 Q11 Q12 Q17 Q13 Q14 Q15
r_IN 0,46 0,47 0,25 0,15 0,42 0,17 ‐0 0,14 0,02
r_US 0,61 0,56 0,43 0,36 0,61 0,35 0,11 0,1 ‐0,1
r_UT 0,62 0,76 0,62 0,74 0,75 0,68 0,13 0,02 ‐0,1
r_ZZ 0,05 0 0,09 0,08 0,07 ‐0 0,78 0,79 0,16

22. Analysis of the opinion surveys
• Factorial analysis.
Three factors
Four factors
Five factors
Factor 1 Factor 2 Factor 3 Factor
Q01 0,50 0,27 0,58 4
Q02 0,17 0,10 0,65 3
Q03 0,40 0,29 0,34 4
Q04 0,75 0,09 0,07 1
Q05 0,08 0,61 0,31 2
Q06 -0,28 0,11 0,65
Q07 -0,05 0,75 0,08 2
Q08 0,15 0,26 0,03
Q09 0,01 0,70 0,09 2
Q10 0,27 0,50 0,21 2
Q11 0,69 0,28 -0,05 1
Q12 0,46 0,57 0,32 4
Q16 0,15 0,13 0,64 3
Q17 0,74 -0,09 0,17 1
Q18 0,46 0,27 0,55 4

23. Analysis of the opinion surveys
• Identified factors
– Q04, Q11, Q17 ENJOYMENT
– Q02, Q16 INMERSION?
– Q01, Q03, Q12, Q18 CLIENT SATISFACTION
– Q05, Q07, Q09, Q10 USABILITY
• Excluded questions
– Q06, Q08, Q13, Q14 and Q15

24. Analysis of the opinion surveys
• Fiability of the survey excluding Q06, Q08,
Q13, Q14 y Q15: 0,83
• Fiability of the new identified model
– ENJOYMENT α = 0,69
– ¿INMERSION? α = 0,49
– CLIENT SATISFACTION α = 0,81
– USABILITY α = 0,71

25. Analysis of the opinion surveys
Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18 Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15
Q04 1,00
Q11 0,37 1,00
Q17 0,50 0,41 1,00
Q02 0,25 0,04 0,25 1,00
Q16 0,15 0,20 0,22 0,32 1,00
Q01 0,37 0,36 0,38 0,43 0,35 1,00
Q03 0,27 0,28 0,18 0,27 0,22 0,46 1,00
Q12 0,34 0,43 0,27 0,25 0,29 0,61 0,44 1,00
Q18 0,35 0,36 0,31 0,32 0,37 0,58 0,44 0,59 1,00
Q05 0,19 0,16 0,07 0,24 0,34 0,41 0,38 0,39 0,27 1,00
Q07 0,10 0,19 0,06 0,21 0,15 0,19 0,12 0,37 0,18 0,39 1,00
Q09 0,13 0,13 0,07 0,17 0,16 0,24 0,21 0,33 0,28 0,29 0,37 1,00
Q10 0,19 0,31 0,12 0,30 0,25 0,34 0,20 0,46 0,36 0,29 0,21 0,26 1,00
Q06 ‐0,07 ‐0,03 0,00 0,20 0,21 0,21 0,05 0,17 0,26 0,11 0,20 0,15 0,07 1,00
Q08 0,21 0,05 0,14 0,15 0,09 0,11 0,09 0,15 0,07 0,08 0,14 0,15 0,19 0,09 1,00
Q13 0,10 0,17 0,05 0,05 ‐0,01 0,02 0,08 0,13 0,02 0,13 0,09 ‐0,01 0,04 ‐0,08 0,10 1,00
Q14 ‐0,07 ‐0,01 ‐0,12 0,08 ‐0,02 0,04 0,13 0,08 0,13 0,12 0,03 0,19 0,12 0,09 0,06 0,35 1,00
Q15 ‐0,06 ‐0,04 0,01 0,02 ‐0,04 ‐0,10 ‐0,10 ‐0,18 ‐0,14 ‐0,07 ‐0,01 ‐0,09 ‐0,01 0,03 0,05 ‐0,13 ‐0,09 1,00

26. Analysis of the opinion surveys
• items‐factor CORRELATIONS
Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18
r_DI 0,78 0,77 0,82 0,23 0,24 0,47 0,31 0,44 0,43
r_IN 0,25 0,15 0,28 0,8 0,82 0,48 0,3 0,33 0,43
r_SC 0,42 0,45 0,36 0,4 0,39 0,83 0,72 0,83 0,81
r_US 0,22 0,29 0,12 0,33 0,33 0,43 0,33 0,56 0,4
Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15
r_DI 0,18 0,15 0,14 0,27 ‐0 0,16 0,13 ‐0,1 ‐0
r_IN 0,36 0,22 0,2 0,34 0,25 0,15 0,02 0,03 ‐0
r_SC 0,45 0,27 0,33 0,43 0,22 0,13 0,08 0,12 ‐0,2
r_US 0,72 0,71 0,7 0,64 0,19 0,2 0,09 0,16 ‐0,1

27. Analysis of the opinion surveys
• Homogeneity index
Q04 Q11 Q17 Q02 Q16 Q01 Q03 Q12 Q18
0,58 0,59 0,54 0,52 0,53 0,75 0,57 0,75 0,70
Q05 Q07 Q09 Q10 Q06 Q08 Q13 Q14 Q15
0,57 0,46 0,47 0,56 0,19 0,21 0,12 0,08 ‐0,1
Q1. WebLab helps me in the subject concept, exercises, projects, etc
Q12. I am satisfied with the WebLab
Q18. The WebLab is a high quality software (access, management, availability, etc.)

28. Analysis of the opinion surveys
• Enjoyment: Q04, Q11, Q17 (196 surveys). 75%

29. Analysis of the opinion surveys
• Inmersion?: Q02, Q16. 92%

30. Analysis of the opinion surveys
• Client satisfaction: Q01, Q03, Q12, Q18. 97%

31. Analysis of the opinion surveys
• Usability: Q05, Q07, Q09, Q10. 97%

32. Analysis of the opinion surveys
• Does the subject/matter influence the results?

33. Analysis of the opinion surveys
• Kolmogorov‐Smirnov test for the two subjects

34. Analysis of the opinion surveys
• Mann‐Whitney test

35. Analysis of the opinion surveys
• Does the year influence the results?

36. Analysis of the opinion surveys
• Kruskal‐Wallis test

37. Analysis of the opinion surveys

38. Learning evaluation
• O‐X‐O: Around 50 students of two different and
similar groups …. Analog electronics. 2012‐2013
– Pre‐test
• 10 questions, multiple choice with 4 answers
– Activity/Treatment: 2 weeks
• 2 working sessions with VISIR in the classroom
• 1 reviewing session
– Post‐test
• 10 questions, multiple choice with 4 answers
• The same questions but in different order

39. Learning evaluation

40. Learning evaluation
• Fiability
– Cronbach’s alfa
• Construct validity
– Content
– Construct, internal consistency?
• Items analysis
– Difficulty index
– Homogeinity index

41. Learning evaluation
• Cronbach’s alfa
• Result for the survey: 0,52
• It is a low value, but the results of the next
test are good. It means that the survey it is
not very good, so it should be improved.

42. Learning evaluation: Questions 1‐5
P01 P02 P03 P04 P05
a 70 32 23 31 32
b 8 29 2 23 2
c 22 37 73 22 51
d 6 7 8 30 21
P01 P02 P03 P04 P05
a 34 12 12 12 13
b 6 12 0 6 1
c 9 25 38 9 25
d 5 5 4 27 15
P01 P02 P03 P04 P05
a 36 20 11 19 19
b 2 17 2 17 1
c 13 12 35 13 26
d 1 2 4 3 6
ALL THE TESTS
PRE‐TEST
POST‐TEST

43. Learning evaluation: Question 2
• In the circuit with two resistors of the same
value, the voltage in each resistor is:
A: it is equal, B: it is the half of E in each resistor
C: serial or paralell D: it is null

44. Learning evaluation: Question 5
• Which of the following circuits measures the
voltage in R1?

45. Learning evaluation: Questions 6‐10
P06 P07 P08 P09 P10
a 46 9 35 18 10
b 17 19 18 41 18
c 34 57 3 27 10
d 9 21 50 20 67
P06 P07 P08 P09 P10
a 17 7 15 9 6
b 10 8 7 15 10
c 21 20 3 18 6
d 6 19 29 12 31
P06 P07 P08 P09 P10
a 29 2 20 9 4
b 7 11 11 26 8
c 13 37 0 9 4
d 3 2 21 8 36
ALL THE TESTS
PRE‐TEST
POST‐TEST

46. Learning evaluation: Question 7
• The total value of the resistor is:
A: close to 1 kohm B: close to 100 ohm
C: close to 10 kohm D: power the circuit

47. Learning evaluation: Question 8
• Which of the following circuits measures the
total resistance?

48. Learning evaluation
• Difficulty index
P01 P02 P03 P04 P05
0,61 0,28 0,64 0,21 0,28
P06 P07 P08 P09 P10
0,41 0,50 0,34 0,37 0,60

49. Learning evaluation
• Difficulty index. The test is too difficult.
• We should redesign it.
Difficulty index Interpretation Recommended number of items
< 0,25 Very difficukt 10% (1) 1
[0,25; 0,45) Difficult 20% (2) 5
[0,45; 0,55) Normal 40% (4) 1
[0,55; 0,75] Easy 20% (2) 3
> 0,75 Very easy 10% (1) 0

50. Learning evaluation
• Homogeneity index
P01 P02 P03 P04 P05
0,43 0,38 0,24 0,50 0,38
P06 P07 P08 P09 P10
0,47 0,53 0,50 0,49 0,43

51. Learning evaluation
• Question 3: Which of the following circuits
measures the current in R1?

52. Learning evaluation

53. Learning evaluation

54. Learning evaluation

55. Learning evaluation
• Total punctuation
pre post
0246810

56. Learning evaluation
• The difference between groups is statistically
significant:
– Paired t‐test, p < 0,001
Paired t-test
data: dfParelles[, 3] and dfParelles[, 2]
t = 4.9189, df = 51, p-value = 9.474e-06
– Wilcoxon test (rangos con signo), p < 0,001
Wilcoxon signed rank test with continuity correction
data: dfParelles[, 3] and dfParelles[, 2]
V = 825, p-value = 1.886e-05

57. Conclusion
• The survey designed to know the opinion of
the users is useful, but it is clearly and easily
improvable.
• The use of VISIR has a positive effect in the
students’ learning with basic electronics.

58. Future work
• Remote Labs + Learning Analytics
• Improve and redesign the survey for the users.
• Analyse the inmersion concept when using
remote labs.
• Extend the analysis of the VISIR effect in the
students’ learning.
• Better explotation of the available data: user
profiles, use profiles, session profiles
• Massive use of remote labs in MOOCs.
• Data mining.

61. Who is copying?

65. Future work
• New network at spanish level fostering
collaborations between groups working on LA
in Spain
• First meeting at some point in autumn to be
established
• 100% Open (contact snola@deusto.es)
• http://www.snola.deusto.es/

66. Learning analytics and remote
experimentation
Universidad de Deusto (Bilbao, Spain)
IQS ‐ Universidad Ramon Llull (Barcelona, Spain)
Aristos Campus Mundus