2013 03-14 (educon2013) emadrid urjc mining student repositories to gain learning analytics an experience report

1. Mining student repositories to gain learning
analytics
An experience report
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a
{grex,jgb}@gsyc.urjc.es
GSyC/LibreSoft, Universidad Rey Juan Carlos, Madrid, Spain
Berlin, Germany, March 14th, 2013
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

2. c 2013 Gregorio Robles, Jes´s M. Gonz´lez-Barahona
u a
All figures are ours, except when the original source is specified.
Some rights reserved. This presentation is distributed under the
“Attribution-ShareAlike 3.0” license, by Creative Commons, available at
http://creativecommons.org/licenses/by-sa/3.0/
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

3. Summary: What this talk is about
Engineering students often have to deliver small computer
programs in many engineering courses
Instructors have to evaluate these assignments according to
the learning goals and their quality, but ensure as well that
there is no plagiarism
We report the experience of using mining software repositories
techniques
Related efforts (please, see paper)
Procedure
Tools
Links and ideas
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

4. Context
3rd-year Telecommunication Engineering students at URJC
Course on multimedia networks
Assignments: small programs that exchange multimedia
set-up and content information using standardized network
protocols: SIP, SDP, RTP, UDP, IP...
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

5. Technologies used by students
Python
git: distributed versioning system
pep8 (a script that checks if the code follows coding
guidelines)
wireshark: network protocol analyzer
Scope. The assignment includes:
Program with communication among clients and servers using
standardized protocols
A live capture with the result of a scenario
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

6. 2. Preprocess
Cloning of the repository
Checking if the files with the assignment exist are have been
correctly named
Checking if the style guide has been followed (with pep8)
Evaluating the quality of the code (in our case, Pylint)
Retrieving of the git log and analysis (analyzed with
CVSAnalY)
Analysis of the wireshark network exchange capture
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

7. 3. Plagiarism detection 4. Functional assessment
A note about ”plagiarism” Black box testing
We use the non-open-source Domain-specific
free web service MOSS In computer networks:
(from Stanford University) standards-oriented
Figure: Black-box testing. Source:
goo.gl/3e1Fq
Figure: Plagiarism
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

8. 5. Post-process 6. Personalized exam
Final grades for the Three type of questions:
assignment are calculated Code snippets (own and
Creates file with feedback external)
for the student, with input Black box questions
information from all the Questions about specific
steps scenarios
Instructors get a report of Personalized exams take from 10
the whole process, including to 20 minutes and can be done
assignments suspicious of simultaneously by many students.
plagiarism, and errors
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

9. 6. Personalized exam (example question)
Figure: Personalized exams
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

10. Experience report
Students Instructors
Welcome feedback and We promised ourselves
become better in following complete automatizing...
assignments really not possible
Automatizing allows to We have a lot of data... but
better understand standards maybe not enough of the
one we want!
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

11. Conclusions/summary
Not fully automatable, but scalable (continuous evaluation
possible)
Offers large and good feedback to students
Domain of the program is very important!
Group assessment could be easily introduced
Discussion: we have a wealth of data, but not always the one
we would really like
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics

12. Mining student repositories to gain learning
analytics
An experience report
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a
{grex,jgb}@gsyc.urjc.es
GSyC/LibreSoft, Universidad Rey Juan Carlos, Madrid, Spain
Berlin, Germany, March 14th, 2013
Gregorio Robles, Jes´s M. Gonz´lez Barahona
u a Mining student repositories to gain learning analytics