1. Programming Exercises Evaluation Systems:
An Interoperability Survey
Ricardo Queirós and José Paulo Leal
Faculty of Sciences of University of Porto
4th International Conference on Computer Supported Education (CSEDU’12)
Porto, Portugal
April 16, 2012

2. Introduction Related Work Interoperability Survey Synthesis Conclusion
Outline
1 Introduction
2 Related Work
3 Interoperability Survey
4 Synthesis
5 Conclusion
Queirós & Leal PES: An Interoperability Survey 16-04-2012 2 / 18

3. Introduction Related Work Interoperability Survey Synthesis Conclusion
xxx
Context
Learning programming requires solving programming exercises.
Manual assessment of exercises:
Time consuming - teachers need to assess a large number of
exercises (e.g. large classes)
Error prone - hinders the consistency and accuracy of assessment
results
Existent automatic evaluation systems are
too generic
focused on simple domains (e.g. quizzes)
Appearance of
Programming Exercises Evaluation Systems (PES)
Queirós & Leal PES: An Interoperability Survey 16-04-2012 3 / 18

4. Introduction Related Work Interoperability Survey Synthesis Conclusion
xxx
Programming Exercises Evaluation Systems
Automatic tools to
grade students’ programming exercises
give feedback on the quality of students’ solutions
Used on different learning scenarios:
Curricular (e.g. practical classes, assignments and exams)
Competitive (e.g. programming contests)
IOI - for secondary school students
ACM-ICPC - for university students
IEEExtreme - for IEEE student members
Examples:
AutoGrader, BOSS, Hustoj, Mooshak, WEB-CAT, etc.
Queirós & Leal PES: An Interoperability Survey 16-04-2012 4 / 18

5. Introduction Related Work Interoperability Survey Synthesis Conclusion
xxx
Existing surveys coverage
Several surveys enumerates and compares PES features...
how the analysis of the code is made
how the tests are defined
how grades are calculated
...but neglects the interoperability feature. Organized at two levels:
content
communication
Queirós & Leal PES: An Interoperability Survey 16-04-2012 5 / 18

6. Introduction Related Work Interoperability Survey Synthesis Conclusion
xxx
Existing surveys coverage
Content - how a programming exercise is described
as a Learning Object
composed by assets such as statement, test cases, feedback, etc.
communication - how it should be shared among systems:
Learning
Management
System
LO reference, Evaluatio n repo rt
1 4
Learner reference,
attemp t
Programming
Evaluation
System 3
LO referenc e 2 LO
Learning
Objects
Repository
Queirós & Leal PES: An Interoperability Survey 16-04-2012 6 / 18

7. Introduction Related Work Interoperability Survey Synthesis Conclusion
xxx
Project
Work description
An interoperability survey on existing PES
Main goal
to gather information on the interoperability features of existent PES
Benefits
fills the gap found in most surveys on PES interoperability features
helps in the PES selection to integrate in e-Learning environments
Queirós & Leal PES: An Interoperability Survey 16-04-2012 7 / 18

8. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Evolution of assessment tools
Evolution of # systems
Evolution of features
Early Assessme nt Syst ems (19 60 - 19 85) Tool Oriente d Systems (1 985 - 2000) Web-Oriented Syst ems (2000 - ... )
Web based interfaces
Command-line interfaces (manual operation of
scripts)
Multi-languages (Prolog, SQL, FORTRAN)
Support for few languages (e.g. C, JAVA)
Richer and incremental feedback
Punched cards
Limited feedback
Course/student administration facilities
Support for one language (e.g. Algol)
Content management
Sophisticated testing approaches
No feedback
Static analysis
Automatic test generation
No administration facilities
Student automated testing
Plagiarism detection
Grading-support system
Service-oriented
Competitive learning (e.g. contests)
Integration with LMS
Queirós & Leal PES: An Interoperability Survey 16-04-2012 8 / 18

9. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Recent surveys
Five surveys:
Douce et al. (2005)
Methodology: details features of PES organized in 3 generations
Trend: evaluation of GUI programs, meta-testing (evaluation of the
students’ tests), SOA and use of interoperability standards
Kirsti AlaMutka (2005)
Methodology: organizes PES in dynamic and static evaluators
Trend: content and communication standardization.
Liang et al. (2009)
Methodology: details dynamic and static analysis methods of PES
Trends: security, algorithms for automatic generation of test data
and content standardization
Ihantola et al. (2010)
Methodology: discuss PES (2006-2010) features (e.g. tests
definition, resubmission policies and security)
Trends: integration with LMS and assessment of web applications
Romli et al. (2010)
Methodology: approaches for test data generation
Trends: test data generation techniques, interoperability and security
Queirós & Leal PES: An Interoperability Survey 16-04-2012 9 / 18

10. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Interoperability analysis
Survey on PES interoperability features
Approach for the selection of
the sample - 15 tools
based on its effective use (exercises and users management)
selected tools: AutoGrader, BOSS2, CourseMaker, CTPracticals,
DOMJudge, EduComponents, GAME, HUSTOJ, Moe, Mooshak,
Peach3, Submit!, USACO, Verkkoke, Web-CAT
the criteria - 3 facets
based on surveys trends and our background
selected facets: programming exercises, users and assessment results
Queirós & Leal PES: An Interoperability Survey 16-04-2012 10 / 18

11. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Interoperability analysis
These 3 facets are synchronized with PES main objective
to evaluate a user’s attempt to solve a programming exercise and
produce an assessment result.
For each facet each PES is evaluated by 3 interoperability maturity levels:
Level 0 - manual configuration of data
Level 1 - data import/export
Level 2 - services invocation
Queirós & Leal PES: An Interoperability Survey 16-04-2012 11 / 18

12. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Programming Exercises
Programming Exercises facet
Level 0 - manual configuration of exercises
Level 1 - import/export of exercises
Level 2 - integration with repository services
Systems Level 0 Level 1 Level 2
AutoGrader F - -
BOSS2 F - -
CourseMaker F - -
CTPraticals F - -
DOMJudge F - -
EduComponents F - -
GAME F - -
HUSTOJ F P -
Moe F P -
Mooshak F F F
Peach3 F P -
Submit! F - -
USACO F - -
Verkkoke F F -
Web-CAT F F P
All systems support the configuration of exercises
6 tools export exercises; 3 bidirectional support; few systems use
exercises formats
2 tools support communication with repositories through SOA
Queirós & Leal PES: An Interoperability Survey 16-04-2012 12 / 18

13. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Users
Users facet
Level 0 - manual configuration of users;
Level 1 - import/export of users;
Level 2 - integration with user directories services (authentication)
and AMS (authorization)
Systems Level 0 Level 1 Level 2
AutoGrader F F P
BOSS2 F - -
CourseMaker F - -
CTPraticals F F P
DOMJudge F F P
EduComponents F F P
GAME F - -
HUSTOJ F - -
Moe F - -
Mooshak F F P
Peach3 F - -
Submit! F - -
USACO F F -
Verkkoke F F -
Web-CAT F F -
All systems support the manual configuration of users
8 tools allow the import/export of users in non-standard formats
5 tools communicates with authentication services (LDAP)
Queirós & Leal PES: An Interoperability Survey 16-04-2012 13 / 18

14. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Assessment results
Assessment results facet
Level 0 - visualization of evaluation results
Level 1 - export of assessment results
Level 2 - integration with LMS
Systems Level 0 Level 1 Level 2
AutoGrader F F P
BOSS2 F - -
CourseMaker F - -
CTPraticals F F P
DOMJudge F F -
EduComponents F F P
GAME F - -
HUSTOJ F - -
Moe F - -
Mooshak F F -
Peach3 F F -
Submit! F - -
USACO F - -
Verkkoke F F P
Web-CAT F F -
All systems present the evaluation results to users
The majority allows its exportation in non-standard formats
4 systems support the communication with LMS
Queirós & Leal PES: An Interoperability Survey 16-04-2012 14 / 18

15. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Interoperability facets coverage
Mooshak, Web-CAT, Verkkoke offer the best interoperability levels
Half of the systems studied did not reach 50% of the maturity rate
There are a lot to do regarding PES interoperability
Mooshak 83%
Web-CAT 72%
Verkkoke 72%
EduComponents 67%
CTPracticals 67%
AutoGrader 67%
DOMJudge 61%
Systems
Peach3 50%
USACO 44%
Moe 39%
HUSTOJ 39%
Submit! 33%
GAME 33%
CourseMaker 33%
BOSS2 33%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
% of maturity
Queirós & Leal PES: An Interoperability Survey 16-04-2012 15 / 18

16. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Interoperability features coverage
There is no specific trend on interoperability features
Nevertheless, programming exercises facet presents the lower results
Need to standardize:
the description of programming exercises
the communication of PES with other systems (e.g. repositories)
100%
90%
80% 43% 44%
53%
70%
% of coverage
60%
50%
not covered
40%
covered
30% 56% 55%
46%
20%
10%
0%
programming exercises user assessment results
Interoperability facets
Queirós & Leal PES: An Interoperability Survey 16-04-2012 16 / 18

17. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Conclusion
This work presents an interoperability survey on PES
15 tools were surveyed based on 3 interoperability trend facets
programming exercises
users
assessment results
Issues detected on most PES regarding interoperability
the lack of standards to
describe programming exercises and assessment results
communicate with other e-Learning systems (e.g. LMS, AMS, LOR)
Survey benefits
is the first interoperability study on existing PES
can base the
selection of existing PES in e-Learning environments
design of interoperability requirements for the creation of new PES
Queirós & Leal PES: An Interoperability Survey 16-04-2012 17 / 18

18. Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1
Questions
Thanks for your attention
Ricardo Queirós
ricardo.queiros@eu.ipp.pt
Queirós & Leal PES: An Interoperability Survey 16-04-2012 18 / 18