SlideShare ist ein Scribd-Unternehmen logo
1 von 39
Downloaden Sie, um offline zu lesen
THE OPENSUBMIT PROJECT
How to grade 1200 code submissions
Dr. PeterTröger, Operating Systems Group,TU Chemnitz


(with Frank Feinbube, Bernhard Rabe, Kai Fabian)
https://github.com/troeger/opensubmit
PART I: PURPOSE
PART II:TECHNOLOGY
BACKGROUND
• University researcher and teacher, 13 years of experience
• Courses from 5 to 6000 participants
• Winter 2015/16: 

Embedded programming course with 300 students, 

~1200 code submissions to grade
• OpenSubmit developed + used at 

Hasso Plattner Institute Potsdam andTU Chemnitz
• Replaced Roundup-based solution (issue tracker)
3
THE ENVIRONMENT
• Homework assignments („Übungsblätter“)
• Often more important than the lecturing
• Students develop text, equations or code
• Written / printed paper, also e-mail and file transfer
• Tutors provide comments (and grade)
• May impact final course grade
4
5
Electronic
Submission
PROBLEM 1: FORMATS
• Students are way too creative
• C code as scanned picture or .docx
• Visual Studio / Eclipse project folder mess
• Archive formats you never heard about (.s7z) or don’t want (.rar)
• Endless dependencies on third-party header files
• Complete repos or third-party sources included
6
PROBLEM 2: REPRODUCIBILITY
• „Must be your fault, it definitely worked on my laptop.“
• „I have this weird compiler message, can you fix it?“
• „I submitted the wrong file, can you replace foo.c with bar.cpp?“
• „We rely on strangelib.so. It’s written in NoTeS/readme.docx.zip“ 

• Tasks with special hardware (e.g. OpenCL)
• Test machine on the evening before deadline …
7
PROBLEM 3: GRADING
• Coordinate work of the teaching team
• Keep students informed about grades and comments
• Handle re-submissions and submission corrections
• Manage database of final scores
• Deal with cheaters
• Copying of code on test machines
• Stack-Overflow’ing
8
MOODLE?
• Moodle (and most other LMSs) are all-inclusive
• Online teaching, participant management, grading books,
collaboration, file management, calendar, ….
• Course owners need training. Seriously?
• Assignments are just a small part.
• Plugins for large PHP projects. Not fun.
• OpenSubmit is focussing on assignments only.
9
OPENSUBMIT PRINCIPLES
• I) Minimalism
• Don’t let teaching policies live in code
• Assignment rules vary widely in different institutions and groups
• Example:When and how form student groups.
• Create a tool for humans to implement these policies
• Reduced student interface.
• Clear workflows for teachers.
10
STUDENT UI
11
OPENSUBMIT PRINCIPLES
• II) Teacher-driven development (TDD)
• Develop capabilities ‚on-the-fly‘, focus on needs of correctors
• People sometimes call it user-centric, agile or design thinking
• Your development process must fit to that
12
winter

term
summer

term
winter

term
summer

term
winter

term
winter

term
TEACHER UI
13
EXAMPLE:
DUPLICATE DETECTION
• Good students don’t cheat, bad student cheat badly
• Solvable with half-smart MD5 calculation on student file uploads
• Results in duplicate report for correctors (minimalism policy)
• Added in the last semester in 2 weeks
• Future versions will rely on 

fuzzy content comparison
14
OPENSUBMIT PRINCIPLES
• III) Assignment validation
• Compile and validate
student file uploads
• Different tests and 

tests machines per assignment
• Direct feedback, chance for
withdrawal before deadline
• Dedicated full test for grading
after deadline
15
OpenSubmit
Web
Students
OpenSubmit
Exec
OpenSubmit
Exec
OpenSubmit
Exec
VALIDATION SCRIPT
• Something written by the
assignment creator
• Executor daemon downloads the
student submission and this script
• Must run the (compiled) student
code and print debug output
• STDOUT shown to student, 

error code as success indicator
• Full test works the same way
16
OpenSubmit
Web
Students
OpenSubmit
Exec
OpenSubmit
Exec
OpenSubmit
Exec
STUDENT UI
17
TEACHER UI
18
ASSIGNMENT
• Relates to a course
• Has a grading scheme, start time, hard deadline and description link
• May have a soft deadline
• May include compilation of file upload
• May have a validation script executing the student code
• May have a full test script executing the student code
• Has grading scheme as collection of arbitrary grading strings
19
SUBMISSION
• A submission is handed in by a single student
• Can declare group members, no group management (minimalism)
• Submission and results show up for all of them
• Whole submission can be explicitly withdrawn, not deleted
• Grading and grading notes visible when tutors triggered notification

• Status from student and tutor perspective looks different
20
STATES
21
CURRENT FEATURES
• Social login (SAML, OAuth, OpenID)
• Simple student front-end
• Compilation, validation and full test
on assignment-specific test machines
• Output of compilation and validation
scripts shown to students
• E-mail notification on state changes
• History of assignment file uploads
• Central overview of grading progress
• Rich tutor support: 

Archive preview, duplicate detection,
full tests, filtering
• All students notified at-once after
grading is completely finished
• Excel-style grading table
• (Usable) course archive download
22
1200 SUBMISSIONS?
• OpenSubmit supports teachers grading workflow
• Trivial work sharing in the teaching team
• Bad cheaters are already identified
• All student code is proven to work (validation script)
• Triggering an extra test is a mouse click away
• Progress alway visible
23
AUTO GRADING?
• University teachers would love that, especially with MOOCs
• Option 1: Students develop (blindly) against a test suite
• Option 2:Analyze code AST to derive some score
• Option 3: Check for OS-visible behaviors on execution
• Option 4: ???
• Very hard to generalize.Also a legal problem.
• Validation script concept allows all of them.
24
PART I: PURPOSE
PART II:TECHNOLOGY
OPENSUBMIT-WEB
• Django web application
• Started with Django 1.3, followed
all migrations up to recent 1.9
• Python 2.7 (sorry Martin),
mod_wsgi,Apache 2.4,
PostgreSQL
• Varying set of third-party code,
including JavaScript libs
• Some experiences over the years
26
OpenSubmit
Web
Students
OpenSubmit
Exec
OpenSubmit
Exec
OpenSubmit
Exec
EXPERIENCES:
THIRD-PARTY CODE
• Pro: Other people solve your existing problem
• Authentication, templating,API management, fancy UI, crypto, testing, …
• They fix the really bad bugs for you
• Con: Other people create your new problem
• Your environment is not their environment
• Even when you fix it, the pull request may take some time
• You must be willing to understand their code, too
27
EXPERIENCES:
THIRD-PARTY CODE
• Development vs. integration trade-off
• The maintenance problem is always there, but in different flavors
• Example: django-reversion vs. own model implementation
• Example: Django REST framework vs. custom HTTP API
• Lessons learned
• Popularity and age are good indicators (djangopackages.com)
• Avoid solutions with „advanced magic“
28
EXPERIENCES:

DATA MIGRATION
• Database migration needed on model changes
• South was ok for that, Django>=1.7 support is even better
• They handle model migration, but data migration is your problem
• Lessons learned
• Create migration-friendly data models.You have only one try.
• Data attributes may later become 1:N relationships.
• Put more in queries and less in foreign keys (Django reverse lookup)
29
EXPERIENCES:
WEB AUTHENTICATION
• I hate separate accounts.
• There is only social login in this project.
• All universities anyway have their own single sign-on.
• python-social has pluggable backends and seamless Django integration
• Lessons learned
• If you change your authentication code base, hell breaks lose.
• Identity mapping based on e-mail no longer helps.
30
OPENSUBMIT-EXEC
• Python 3 script on test machines
• Download of validator and student 

code from predefined web host
• Called by cron, or manually for testing
• Isolation of student code through 

dedicated (virtual) machines
• Timeout for deadlocking student code, always report a result
• Handling of archive obscurities on the test machine, not on web server
31
OpenSubmit
Web
OpenSubmit
Exec
OpenSubmit
Exec
OpenSubmit
Exec
Students
EXPERIENCES:
PUSHVS. PULL
• Executors ask for jobs via HTTP
• Pulling high frequency cron job
• Push-receiving daemon would reduce 

latency, but increases complexity
• Trivial to operate, good enough 

for the scale of this application
• No inbound connectivity needed, outbound connectivity restricted
• Load peaks shortly before assignment deadline, handled by dynamicVM creation
32
OpenSubmit
Web OpenSubmit
Exec
OpenSubmit
Exec
OpenSubmit
ExecStudents
REGRETTABLETHINGS
• Some early design mistakes hunt you forever
• No central state transition (!) logic
• View-driven development, instead of API focus
• Half-baked configuration management
• Missing consideration of installation maintenance
• Missed chance for on-the-fly manual writing
• Ignorance of the async job queue problem.
33
THINGS DONE RIGHT
• Ignorance of performance issues (until they show up)
• Ignorance of PEP-8 (with 1,5 developers)
• Ignorance of test coverage for non-security stuff
• Minimalism policy
• Trivial UI for most users, complex UI for power users
• Making terrible code public.
• Python, Django, PostgreSQL.They just do the job.
34
DJANGO IS STILL AWESOME
• Fulfilled its promises from day one
• Fantastic documentation
• Powerful ORM query features
• Django Admin is the core of the teacher backend
• Stupidity protection layer (e.g. XSS, input sanitization)
• Healthy feature addition / deprecation ratio
• Trustworthy deprecation policy, no surprises on updates
35
BIKE-SHEDDING
• „… disproportionate weight to trivial issues“ [Wikipedia]
• In combination with users that see you as vendor, this is annoying
• Explain your prioritization policy
• Be the representative for all silent (student) users
• Remain grateful that they contribute ‚something‘
• There are books about that.
• It’s open source. Let them fork and see what happens.
36
FUTURE STEPS
• More test coverage (for the GitHub badge)
• More documentation (for the users)
• Test machines withVagrant / Docker / libvirt
• „Hyper-agile cloud-scale load management“
• UI homogenization with (Django) Grappelli
• LTI provider
• Improved grading with sophisticated in-browser preview
37
CONCLUSION
• OpenSubmit brings the KISS principle to learning management
• Focus on one problem, and do this right
• Developing your own tools for daily work is fun, but …
• Consider that you will create 

technical debt.
• If you want people to use it, 

you need advertising.
38
39
OpenSubmit Web
• Python + Django
• PostgreSQL
• python-social
Students
OpenSubmit Exec
OpenSubmit Exec
OpenSubmit Exec
• Python + Linux
https://github.com/troeger/opensubmit
Student
Code
Validation
Script

Weitere ähnliche Inhalte

Was ist angesagt?

Dependable Systems - Introduction (1/16)
Dependable Systems - Introduction (1/16)Dependable Systems - Introduction (1/16)
Dependable Systems - Introduction (1/16)Peter Tröger
 
Dependable Systems -Reliability Prediction (9/16)
Dependable Systems -Reliability Prediction (9/16)Dependable Systems -Reliability Prediction (9/16)
Dependable Systems -Reliability Prediction (9/16)Peter Tröger
 
Dependable Systems - System Dependability Evaluation (8/16)
Dependable Systems - System Dependability Evaluation (8/16)Dependable Systems - System Dependability Evaluation (8/16)
Dependable Systems - System Dependability Evaluation (8/16)Peter Tröger
 
Dependable Systems - Hardware Dependability with Redundancy (14/16)
Dependable Systems - Hardware Dependability with Redundancy (14/16)Dependable Systems - Hardware Dependability with Redundancy (14/16)
Dependable Systems - Hardware Dependability with Redundancy (14/16)Peter Tröger
 
Fault Tolerance System
Fault Tolerance SystemFault Tolerance System
Fault Tolerance Systemprakashjjaya
 
Software Fault Tolerance
Software Fault ToleranceSoftware Fault Tolerance
Software Fault ToleranceAnkit Singh
 
Fault tolearant system
Fault tolearant systemFault tolearant system
Fault tolearant systemarvinthsaran
 
Fault Tolerance System
Fault Tolerance SystemFault Tolerance System
Fault Tolerance SystemEhsan Ilahi
 
Fault tolerance techniques for real time operating system
Fault tolerance techniques for real time operating systemFault tolerance techniques for real time operating system
Fault tolerance techniques for real time operating systemanujos25
 
Fault tolerance techniques tsp
Fault tolerance techniques tspFault tolerance techniques tsp
Fault tolerance techniques tspPradeep Kumar TS
 
Reverse Engineering of Software Architecture
Reverse Engineering of Software ArchitectureReverse Engineering of Software Architecture
Reverse Engineering of Software ArchitectureDharmalingam Ganesan
 
Fault avoidance and fault tolerance
Fault avoidance and fault toleranceFault avoidance and fault tolerance
Fault avoidance and fault toleranceJabez Winston
 
Real Time Systems
Real Time SystemsReal Time Systems
Real Time Systemsleo3004
 
Ivv workshop model-based-testing-of-nasa-systems
Ivv workshop model-based-testing-of-nasa-systemsIvv workshop model-based-testing-of-nasa-systems
Ivv workshop model-based-testing-of-nasa-systemsDharmalingam Ganesan
 

Was ist angesagt? (20)

Dependable Systems - Introduction (1/16)
Dependable Systems - Introduction (1/16)Dependable Systems - Introduction (1/16)
Dependable Systems - Introduction (1/16)
 
Dependable Systems -Reliability Prediction (9/16)
Dependable Systems -Reliability Prediction (9/16)Dependable Systems -Reliability Prediction (9/16)
Dependable Systems -Reliability Prediction (9/16)
 
Dependable Systems - System Dependability Evaluation (8/16)
Dependable Systems - System Dependability Evaluation (8/16)Dependable Systems - System Dependability Evaluation (8/16)
Dependable Systems - System Dependability Evaluation (8/16)
 
Dependable Systems - Hardware Dependability with Redundancy (14/16)
Dependable Systems - Hardware Dependability with Redundancy (14/16)Dependable Systems - Hardware Dependability with Redundancy (14/16)
Dependable Systems - Hardware Dependability with Redundancy (14/16)
 
Fault tolerance
Fault toleranceFault tolerance
Fault tolerance
 
Fault Tolerance System
Fault Tolerance SystemFault Tolerance System
Fault Tolerance System
 
Software Fault Tolerance
Software Fault ToleranceSoftware Fault Tolerance
Software Fault Tolerance
 
Fault tolearant system
Fault tolearant systemFault tolearant system
Fault tolearant system
 
Fault Tolerance System
Fault Tolerance SystemFault Tolerance System
Fault Tolerance System
 
Fault tolerance techniques for real time operating system
Fault tolerance techniques for real time operating systemFault tolerance techniques for real time operating system
Fault tolerance techniques for real time operating system
 
Fault tolerance techniques tsp
Fault tolerance techniques tspFault tolerance techniques tsp
Fault tolerance techniques tsp
 
SRE Tools
SRE ToolsSRE Tools
SRE Tools
 
Sw testing
Sw testingSw testing
Sw testing
 
Voting protocol
Voting protocolVoting protocol
Voting protocol
 
Reverse Engineering of Software Architecture
Reverse Engineering of Software ArchitectureReverse Engineering of Software Architecture
Reverse Engineering of Software Architecture
 
Fault avoidance and fault tolerance
Fault avoidance and fault toleranceFault avoidance and fault tolerance
Fault avoidance and fault tolerance
 
Bangalore march07
Bangalore march07Bangalore march07
Bangalore march07
 
Fault tolerance techniques
Fault tolerance techniquesFault tolerance techniques
Fault tolerance techniques
 
Real Time Systems
Real Time SystemsReal Time Systems
Real Time Systems
 
Ivv workshop model-based-testing-of-nasa-systems
Ivv workshop model-based-testing-of-nasa-systemsIvv workshop model-based-testing-of-nasa-systems
Ivv workshop model-based-testing-of-nasa-systems
 

Andere mochten auch

теоретич обоснование обновленное
теоретич обоснование обновленноетеоретич обоснование обновленное
теоретич обоснование обновленноеТатьяна Гришина
 
Pragmatic Works Fast Start Offerings
Pragmatic Works Fast Start OfferingsPragmatic Works Fast Start Offerings
Pragmatic Works Fast Start OfferingsCasey Lucas
 
Образовательная политика (Селюков р.в.)
Образовательная политика (Селюков р.в.)Образовательная политика (Селюков р.в.)
Образовательная политика (Селюков р.в.)Беленкова Татьяна
 
региональная образовательная политика
региональная образовательная политикарегиональная образовательная политика
региональная образовательная политикаБеленкова Татьяна
 
ウェブラジコン!
ウェブラジコン!ウェブラジコン!
ウェブラジコン!Junya Kono
 
El trabajo de un científico
El trabajo de un científicoEl trabajo de un científico
El trabajo de un científicoBelen Laborde
 
Pecha kucha del currículo
Pecha kucha del currículoPecha kucha del currículo
Pecha kucha del currículoanselmo31
 
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)Casey Lucas
 
Presentación Mooc Introducción a la Metodología A+S
Presentación Mooc   Introducción a la Metodología A+SPresentación Mooc   Introducción a la Metodología A+S
Presentación Mooc Introducción a la Metodología A+SCiiet UdeSantiago
 
Judielene Obrero Interior Design Portfolio
Judielene Obrero Interior Design PortfolioJudielene Obrero Interior Design Portfolio
Judielene Obrero Interior Design PortfolioLakeland Cathy Wolters
 

Andere mochten auch (16)

Slik17 Pitch
Slik17 PitchSlik17 Pitch
Slik17 Pitch
 
теоретич обоснование обновленное
теоретич обоснование обновленноетеоретич обоснование обновленное
теоретич обоснование обновленное
 
Pragmatic Works Fast Start Offerings
Pragmatic Works Fast Start OfferingsPragmatic Works Fast Start Offerings
Pragmatic Works Fast Start Offerings
 
Venture Capital World Summit 2015 Alan Thomas
Venture Capital World Summit 2015 Alan ThomasVenture Capital World Summit 2015 Alan Thomas
Venture Capital World Summit 2015 Alan Thomas
 
Образовательная политика (Селюков р.в.)
Образовательная политика (Селюков р.в.)Образовательная политика (Селюков р.в.)
Образовательная политика (Селюков р.в.)
 
региональная образовательная политика
региональная образовательная политикарегиональная образовательная политика
региональная образовательная политика
 
аналитическая справка ч.2
аналитическая справка ч.2аналитическая справка ч.2
аналитическая справка ч.2
 
ウェブラジコン!
ウェブラジコン!ウェブラジコン!
ウェブラジコン!
 
Pecha kucha
Pecha kuchaPecha kucha
Pecha kucha
 
El trabajo de un científico
El trabajo de un científicoEl trabajo de un científico
El trabajo de un científico
 
Pecha kucha del currículo
Pecha kucha del currículoPecha kucha del currículo
Pecha kucha del currículo
 
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)
Growing Up Hybrid -- Accelerating Digital Transformation (Cloud)
 
Presentación Mooc Introducción a la Metodología A+S
Presentación Mooc   Introducción a la Metodología A+SPresentación Mooc   Introducción a la Metodología A+S
Presentación Mooc Introducción a la Metodología A+S
 
Judielene Obrero Interior Design Portfolio
Judielene Obrero Interior Design PortfolioJudielene Obrero Interior Design Portfolio
Judielene Obrero Interior Design Portfolio
 
Construindo uma-carreira-open-source
Construindo uma-carreira-open-sourceConstruindo uma-carreira-open-source
Construindo uma-carreira-open-source
 
Playful Scrum learning
Playful Scrum learningPlayful Scrum learning
Playful Scrum learning
 

Ähnlich wie OpenSubmit - How to grade 1200 code submissions

Dig1108C Lesson 1 Fall 2014
Dig1108C Lesson 1 Fall 2014Dig1108C Lesson 1 Fall 2014
Dig1108C Lesson 1 Fall 2014David Wolfpaw
 
Texas Rangers to the rescue: turning your VLE into an exam centre
Texas Rangers to the rescue: turning your VLE into an exam centreTexas Rangers to the rescue: turning your VLE into an exam centre
Texas Rangers to the rescue: turning your VLE into an exam centreBlackboardEMEA
 
Fyp slide presentation muiz
Fyp slide presentation muizFyp slide presentation muiz
Fyp slide presentation muizMuhd Mu'izuddin
 
Webinar: How We Evaluated MongoDB as a Relational Database Replacement
Webinar: How We Evaluated MongoDB as a Relational Database ReplacementWebinar: How We Evaluated MongoDB as a Relational Database Replacement
Webinar: How We Evaluated MongoDB as a Relational Database ReplacementMongoDB
 
Experience in teaching devops
Experience in teaching devopsExperience in teaching devops
Experience in teaching devopsLen Bass
 
Improving the Design of Existing Software
Improving the Design of Existing SoftwareImproving the Design of Existing Software
Improving the Design of Existing SoftwareSteven Smith
 
lecture01_Introduction.pdf
lecture01_Introduction.pdflecture01_Introduction.pdf
lecture01_Introduction.pdfMarlonMagtibay2
 
Agileand saas davepatterson_armandofox_050813webinar
Agileand saas davepatterson_armandofox_050813webinarAgileand saas davepatterson_armandofox_050813webinar
Agileand saas davepatterson_armandofox_050813webinarRoberto Jr. Figueroa
 
Building a custom cms with django
Building a custom cms with djangoBuilding a custom cms with django
Building a custom cms with djangoYann Malet
 
Software Engineering - chp1- software dev methodologies
Software Engineering - chp1- software dev methodologiesSoftware Engineering - chp1- software dev methodologies
Software Engineering - chp1- software dev methodologiesLilia Sfaxi
 
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital CompetencesWeb-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital CompetencesHans Põldoja
 
Lessons Learned from Continuous Delivery
Lessons Learned from Continuous DeliveryLessons Learned from Continuous Delivery
Lessons Learned from Continuous DeliveryMandi Walls
 
Lessons learned after 190M lessons served
Lessons learned after 190M lessons servedLessons learned after 190M lessons served
Lessons learned after 190M lessons servedRicardo Bánffy
 

Ähnlich wie OpenSubmit - How to grade 1200 code submissions (20)

EURO Conference 2015 - Automated Timetabling
EURO Conference 2015 - Automated TimetablingEURO Conference 2015 - Automated Timetabling
EURO Conference 2015 - Automated Timetabling
 
Dig1108C Lesson 1 Fall 2014
Dig1108C Lesson 1 Fall 2014Dig1108C Lesson 1 Fall 2014
Dig1108C Lesson 1 Fall 2014
 
Texas Rangers to the rescue: turning your VLE into an exam centre
Texas Rangers to the rescue: turning your VLE into an exam centreTexas Rangers to the rescue: turning your VLE into an exam centre
Texas Rangers to the rescue: turning your VLE into an exam centre
 
Fyp slide presentation muiz
Fyp slide presentation muizFyp slide presentation muiz
Fyp slide presentation muiz
 
Webinar: How We Evaluated MongoDB as a Relational Database Replacement
Webinar: How We Evaluated MongoDB as a Relational Database ReplacementWebinar: How We Evaluated MongoDB as a Relational Database Replacement
Webinar: How We Evaluated MongoDB as a Relational Database Replacement
 
BbWorld 2010 notes
BbWorld 2010 notesBbWorld 2010 notes
BbWorld 2010 notes
 
Mini Project.pptx
Mini Project.pptxMini Project.pptx
Mini Project.pptx
 
Experience in teaching devops
Experience in teaching devopsExperience in teaching devops
Experience in teaching devops
 
Improving the Design of Existing Software
Improving the Design of Existing SoftwareImproving the Design of Existing Software
Improving the Design of Existing Software
 
lecture01_Introduction.pdf
lecture01_Introduction.pdflecture01_Introduction.pdf
lecture01_Introduction.pdf
 
Agileand saas davepatterson_armandofox_050813webinar
Agileand saas davepatterson_armandofox_050813webinarAgileand saas davepatterson_armandofox_050813webinar
Agileand saas davepatterson_armandofox_050813webinar
 
Building a custom cms with django
Building a custom cms with djangoBuilding a custom cms with django
Building a custom cms with django
 
DITEC - Software Engineering
DITEC - Software EngineeringDITEC - Software Engineering
DITEC - Software Engineering
 
Lecture1.ppt
Lecture1.pptLecture1.ppt
Lecture1.ppt
 
Software Engineering - chp1- software dev methodologies
Software Engineering - chp1- software dev methodologiesSoftware Engineering - chp1- software dev methodologies
Software Engineering - chp1- software dev methodologies
 
Web-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital CompetencesWeb-based Self- and Peer Assessment of Teachers Digital Competences
Web-based Self- and Peer Assessment of Teachers Digital Competences
 
Itec410 lec01
Itec410 lec01Itec410 lec01
Itec410 lec01
 
Lec 01 introduction
Lec 01   introductionLec 01   introduction
Lec 01 introduction
 
Lessons Learned from Continuous Delivery
Lessons Learned from Continuous DeliveryLessons Learned from Continuous Delivery
Lessons Learned from Continuous Delivery
 
Lessons learned after 190M lessons served
Lessons learned after 190M lessons servedLessons learned after 190M lessons served
Lessons learned after 190M lessons served
 

Mehr von Peter Tröger

WannaCry - An OS course perspective
WannaCry - An OS course perspectiveWannaCry - An OS course perspective
WannaCry - An OS course perspectivePeter Tröger
 
Cloud Standards and Virtualization
Cloud Standards and VirtualizationCloud Standards and Virtualization
Cloud Standards and VirtualizationPeter Tröger
 
Distributed Resource Management Application API (DRMAA) Version 2
Distributed Resource Management Application API (DRMAA) Version 2Distributed Resource Management Application API (DRMAA) Version 2
Distributed Resource Management Application API (DRMAA) Version 2Peter Tröger
 
Design of Software for Embedded Systems
Design of Software for Embedded SystemsDesign of Software for Embedded Systems
Design of Software for Embedded SystemsPeter Tröger
 
Humans should not write XML.
Humans should not write XML.Humans should not write XML.
Humans should not write XML.Peter Tröger
 
Dependable Systems - Hardware Dependability with Diagnosis (13/16)
Dependable Systems - Hardware Dependability with Diagnosis (13/16)Dependable Systems - Hardware Dependability with Diagnosis (13/16)
Dependable Systems - Hardware Dependability with Diagnosis (13/16)Peter Tröger
 
Verteilte Software-Systeme im Kontext von Industrie 4.0
Verteilte Software-Systeme im Kontext von Industrie 4.0Verteilte Software-Systeme im Kontext von Industrie 4.0
Verteilte Software-Systeme im Kontext von Industrie 4.0Peter Tröger
 
Operating Systems 1 (12/12) - Summary
Operating Systems 1 (12/12) - SummaryOperating Systems 1 (12/12) - Summary
Operating Systems 1 (12/12) - SummaryPeter Tröger
 
Operating Systems 1 (8/12) - Concurrency
Operating Systems 1 (8/12) - ConcurrencyOperating Systems 1 (8/12) - Concurrency
Operating Systems 1 (8/12) - ConcurrencyPeter Tröger
 
Operating Systems 1 (9/12) - Memory Management Concepts
Operating Systems 1 (9/12) - Memory Management ConceptsOperating Systems 1 (9/12) - Memory Management Concepts
Operating Systems 1 (9/12) - Memory Management ConceptsPeter Tröger
 
Operating Systems 1 (7/12) - Threads
Operating Systems 1 (7/12) - ThreadsOperating Systems 1 (7/12) - Threads
Operating Systems 1 (7/12) - ThreadsPeter Tröger
 
Operating Systems 1 (1/12) - History
Operating Systems 1 (1/12) - HistoryOperating Systems 1 (1/12) - History
Operating Systems 1 (1/12) - HistoryPeter Tröger
 
Operating Systems 1 (2/12) - Hardware Basics
Operating Systems 1 (2/12) - Hardware BasicsOperating Systems 1 (2/12) - Hardware Basics
Operating Systems 1 (2/12) - Hardware BasicsPeter Tröger
 
Operating Systems 1 (11/12) - Input / Output
Operating Systems 1 (11/12) - Input / OutputOperating Systems 1 (11/12) - Input / Output
Operating Systems 1 (11/12) - Input / OutputPeter Tröger
 
Operating Systems 1 (3/12) - Architectures
Operating Systems 1 (3/12) - ArchitecturesOperating Systems 1 (3/12) - Architectures
Operating Systems 1 (3/12) - ArchitecturesPeter Tröger
 
Operating Systems 1 (6/12) - Processes
Operating Systems 1 (6/12) - ProcessesOperating Systems 1 (6/12) - Processes
Operating Systems 1 (6/12) - ProcessesPeter Tröger
 
Operating Systems 1 (5/12) - Architectures (Unix)
Operating Systems 1 (5/12) - Architectures (Unix)Operating Systems 1 (5/12) - Architectures (Unix)
Operating Systems 1 (5/12) - Architectures (Unix)Peter Tröger
 
Operating Systems 1 (4/12) - Architectures (Windows)
Operating Systems 1 (4/12) - Architectures (Windows)Operating Systems 1 (4/12) - Architectures (Windows)
Operating Systems 1 (4/12) - Architectures (Windows)Peter Tröger
 

Mehr von Peter Tröger (18)

WannaCry - An OS course perspective
WannaCry - An OS course perspectiveWannaCry - An OS course perspective
WannaCry - An OS course perspective
 
Cloud Standards and Virtualization
Cloud Standards and VirtualizationCloud Standards and Virtualization
Cloud Standards and Virtualization
 
Distributed Resource Management Application API (DRMAA) Version 2
Distributed Resource Management Application API (DRMAA) Version 2Distributed Resource Management Application API (DRMAA) Version 2
Distributed Resource Management Application API (DRMAA) Version 2
 
Design of Software for Embedded Systems
Design of Software for Embedded SystemsDesign of Software for Embedded Systems
Design of Software for Embedded Systems
 
Humans should not write XML.
Humans should not write XML.Humans should not write XML.
Humans should not write XML.
 
Dependable Systems - Hardware Dependability with Diagnosis (13/16)
Dependable Systems - Hardware Dependability with Diagnosis (13/16)Dependable Systems - Hardware Dependability with Diagnosis (13/16)
Dependable Systems - Hardware Dependability with Diagnosis (13/16)
 
Verteilte Software-Systeme im Kontext von Industrie 4.0
Verteilte Software-Systeme im Kontext von Industrie 4.0Verteilte Software-Systeme im Kontext von Industrie 4.0
Verteilte Software-Systeme im Kontext von Industrie 4.0
 
Operating Systems 1 (12/12) - Summary
Operating Systems 1 (12/12) - SummaryOperating Systems 1 (12/12) - Summary
Operating Systems 1 (12/12) - Summary
 
Operating Systems 1 (8/12) - Concurrency
Operating Systems 1 (8/12) - ConcurrencyOperating Systems 1 (8/12) - Concurrency
Operating Systems 1 (8/12) - Concurrency
 
Operating Systems 1 (9/12) - Memory Management Concepts
Operating Systems 1 (9/12) - Memory Management ConceptsOperating Systems 1 (9/12) - Memory Management Concepts
Operating Systems 1 (9/12) - Memory Management Concepts
 
Operating Systems 1 (7/12) - Threads
Operating Systems 1 (7/12) - ThreadsOperating Systems 1 (7/12) - Threads
Operating Systems 1 (7/12) - Threads
 
Operating Systems 1 (1/12) - History
Operating Systems 1 (1/12) - HistoryOperating Systems 1 (1/12) - History
Operating Systems 1 (1/12) - History
 
Operating Systems 1 (2/12) - Hardware Basics
Operating Systems 1 (2/12) - Hardware BasicsOperating Systems 1 (2/12) - Hardware Basics
Operating Systems 1 (2/12) - Hardware Basics
 
Operating Systems 1 (11/12) - Input / Output
Operating Systems 1 (11/12) - Input / OutputOperating Systems 1 (11/12) - Input / Output
Operating Systems 1 (11/12) - Input / Output
 
Operating Systems 1 (3/12) - Architectures
Operating Systems 1 (3/12) - ArchitecturesOperating Systems 1 (3/12) - Architectures
Operating Systems 1 (3/12) - Architectures
 
Operating Systems 1 (6/12) - Processes
Operating Systems 1 (6/12) - ProcessesOperating Systems 1 (6/12) - Processes
Operating Systems 1 (6/12) - Processes
 
Operating Systems 1 (5/12) - Architectures (Unix)
Operating Systems 1 (5/12) - Architectures (Unix)Operating Systems 1 (5/12) - Architectures (Unix)
Operating Systems 1 (5/12) - Architectures (Unix)
 
Operating Systems 1 (4/12) - Architectures (Windows)
Operating Systems 1 (4/12) - Architectures (Windows)Operating Systems 1 (4/12) - Architectures (Windows)
Operating Systems 1 (4/12) - Architectures (Windows)
 

Kürzlich hochgeladen

General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and stepobaje godwin sunday
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationMJDuyan
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxraviapr7
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice documentXsasf Sfdfasd
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxAditiChauhan701637
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxDr. Santhosh Kumar. N
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxKatherine Villaluna
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 

Kürzlich hochgeladen (20)

General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and step
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive Education
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptx
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice document
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptx
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptx
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 

OpenSubmit - How to grade 1200 code submissions

  • 1. THE OPENSUBMIT PROJECT How to grade 1200 code submissions Dr. PeterTröger, Operating Systems Group,TU Chemnitz 
 (with Frank Feinbube, Bernhard Rabe, Kai Fabian) https://github.com/troeger/opensubmit
  • 2. PART I: PURPOSE PART II:TECHNOLOGY
  • 3. BACKGROUND • University researcher and teacher, 13 years of experience • Courses from 5 to 6000 participants • Winter 2015/16: 
 Embedded programming course with 300 students, 
 ~1200 code submissions to grade • OpenSubmit developed + used at 
 Hasso Plattner Institute Potsdam andTU Chemnitz • Replaced Roundup-based solution (issue tracker) 3
  • 4. THE ENVIRONMENT • Homework assignments („Übungsblätter“) • Often more important than the lecturing • Students develop text, equations or code • Written / printed paper, also e-mail and file transfer • Tutors provide comments (and grade) • May impact final course grade 4
  • 6. PROBLEM 1: FORMATS • Students are way too creative • C code as scanned picture or .docx • Visual Studio / Eclipse project folder mess • Archive formats you never heard about (.s7z) or don’t want (.rar) • Endless dependencies on third-party header files • Complete repos or third-party sources included 6
  • 7. PROBLEM 2: REPRODUCIBILITY • „Must be your fault, it definitely worked on my laptop.“ • „I have this weird compiler message, can you fix it?“ • „I submitted the wrong file, can you replace foo.c with bar.cpp?“ • „We rely on strangelib.so. It’s written in NoTeS/readme.docx.zip“ 
 • Tasks with special hardware (e.g. OpenCL) • Test machine on the evening before deadline … 7
  • 8. PROBLEM 3: GRADING • Coordinate work of the teaching team • Keep students informed about grades and comments • Handle re-submissions and submission corrections • Manage database of final scores • Deal with cheaters • Copying of code on test machines • Stack-Overflow’ing 8
  • 9. MOODLE? • Moodle (and most other LMSs) are all-inclusive • Online teaching, participant management, grading books, collaboration, file management, calendar, …. • Course owners need training. Seriously? • Assignments are just a small part. • Plugins for large PHP projects. Not fun. • OpenSubmit is focussing on assignments only. 9
  • 10. OPENSUBMIT PRINCIPLES • I) Minimalism • Don’t let teaching policies live in code • Assignment rules vary widely in different institutions and groups • Example:When and how form student groups. • Create a tool for humans to implement these policies • Reduced student interface. • Clear workflows for teachers. 10
  • 12. OPENSUBMIT PRINCIPLES • II) Teacher-driven development (TDD) • Develop capabilities ‚on-the-fly‘, focus on needs of correctors • People sometimes call it user-centric, agile or design thinking • Your development process must fit to that 12 winter
 term summer
 term winter
 term summer
 term winter
 term winter
 term
  • 14. EXAMPLE: DUPLICATE DETECTION • Good students don’t cheat, bad student cheat badly • Solvable with half-smart MD5 calculation on student file uploads • Results in duplicate report for correctors (minimalism policy) • Added in the last semester in 2 weeks • Future versions will rely on 
 fuzzy content comparison 14
  • 15. OPENSUBMIT PRINCIPLES • III) Assignment validation • Compile and validate student file uploads • Different tests and 
 tests machines per assignment • Direct feedback, chance for withdrawal before deadline • Dedicated full test for grading after deadline 15 OpenSubmit Web Students OpenSubmit Exec OpenSubmit Exec OpenSubmit Exec
  • 16. VALIDATION SCRIPT • Something written by the assignment creator • Executor daemon downloads the student submission and this script • Must run the (compiled) student code and print debug output • STDOUT shown to student, 
 error code as success indicator • Full test works the same way 16 OpenSubmit Web Students OpenSubmit Exec OpenSubmit Exec OpenSubmit Exec
  • 19. ASSIGNMENT • Relates to a course • Has a grading scheme, start time, hard deadline and description link • May have a soft deadline • May include compilation of file upload • May have a validation script executing the student code • May have a full test script executing the student code • Has grading scheme as collection of arbitrary grading strings 19
  • 20. SUBMISSION • A submission is handed in by a single student • Can declare group members, no group management (minimalism) • Submission and results show up for all of them • Whole submission can be explicitly withdrawn, not deleted • Grading and grading notes visible when tutors triggered notification
 • Status from student and tutor perspective looks different 20
  • 22. CURRENT FEATURES • Social login (SAML, OAuth, OpenID) • Simple student front-end • Compilation, validation and full test on assignment-specific test machines • Output of compilation and validation scripts shown to students • E-mail notification on state changes • History of assignment file uploads • Central overview of grading progress • Rich tutor support: 
 Archive preview, duplicate detection, full tests, filtering • All students notified at-once after grading is completely finished • Excel-style grading table • (Usable) course archive download 22
  • 23. 1200 SUBMISSIONS? • OpenSubmit supports teachers grading workflow • Trivial work sharing in the teaching team • Bad cheaters are already identified • All student code is proven to work (validation script) • Triggering an extra test is a mouse click away • Progress alway visible 23
  • 24. AUTO GRADING? • University teachers would love that, especially with MOOCs • Option 1: Students develop (blindly) against a test suite • Option 2:Analyze code AST to derive some score • Option 3: Check for OS-visible behaviors on execution • Option 4: ??? • Very hard to generalize.Also a legal problem. • Validation script concept allows all of them. 24
  • 25. PART I: PURPOSE PART II:TECHNOLOGY
  • 26. OPENSUBMIT-WEB • Django web application • Started with Django 1.3, followed all migrations up to recent 1.9 • Python 2.7 (sorry Martin), mod_wsgi,Apache 2.4, PostgreSQL • Varying set of third-party code, including JavaScript libs • Some experiences over the years 26 OpenSubmit Web Students OpenSubmit Exec OpenSubmit Exec OpenSubmit Exec
  • 27. EXPERIENCES: THIRD-PARTY CODE • Pro: Other people solve your existing problem • Authentication, templating,API management, fancy UI, crypto, testing, … • They fix the really bad bugs for you • Con: Other people create your new problem • Your environment is not their environment • Even when you fix it, the pull request may take some time • You must be willing to understand their code, too 27
  • 28. EXPERIENCES: THIRD-PARTY CODE • Development vs. integration trade-off • The maintenance problem is always there, but in different flavors • Example: django-reversion vs. own model implementation • Example: Django REST framework vs. custom HTTP API • Lessons learned • Popularity and age are good indicators (djangopackages.com) • Avoid solutions with „advanced magic“ 28
  • 29. EXPERIENCES:
 DATA MIGRATION • Database migration needed on model changes • South was ok for that, Django>=1.7 support is even better • They handle model migration, but data migration is your problem • Lessons learned • Create migration-friendly data models.You have only one try. • Data attributes may later become 1:N relationships. • Put more in queries and less in foreign keys (Django reverse lookup) 29
  • 30. EXPERIENCES: WEB AUTHENTICATION • I hate separate accounts. • There is only social login in this project. • All universities anyway have their own single sign-on. • python-social has pluggable backends and seamless Django integration • Lessons learned • If you change your authentication code base, hell breaks lose. • Identity mapping based on e-mail no longer helps. 30
  • 31. OPENSUBMIT-EXEC • Python 3 script on test machines • Download of validator and student 
 code from predefined web host • Called by cron, or manually for testing • Isolation of student code through 
 dedicated (virtual) machines • Timeout for deadlocking student code, always report a result • Handling of archive obscurities on the test machine, not on web server 31 OpenSubmit Web OpenSubmit Exec OpenSubmit Exec OpenSubmit Exec Students
  • 32. EXPERIENCES: PUSHVS. PULL • Executors ask for jobs via HTTP • Pulling high frequency cron job • Push-receiving daemon would reduce 
 latency, but increases complexity • Trivial to operate, good enough 
 for the scale of this application • No inbound connectivity needed, outbound connectivity restricted • Load peaks shortly before assignment deadline, handled by dynamicVM creation 32 OpenSubmit Web OpenSubmit Exec OpenSubmit Exec OpenSubmit ExecStudents
  • 33. REGRETTABLETHINGS • Some early design mistakes hunt you forever • No central state transition (!) logic • View-driven development, instead of API focus • Half-baked configuration management • Missing consideration of installation maintenance • Missed chance for on-the-fly manual writing • Ignorance of the async job queue problem. 33
  • 34. THINGS DONE RIGHT • Ignorance of performance issues (until they show up) • Ignorance of PEP-8 (with 1,5 developers) • Ignorance of test coverage for non-security stuff • Minimalism policy • Trivial UI for most users, complex UI for power users • Making terrible code public. • Python, Django, PostgreSQL.They just do the job. 34
  • 35. DJANGO IS STILL AWESOME • Fulfilled its promises from day one • Fantastic documentation • Powerful ORM query features • Django Admin is the core of the teacher backend • Stupidity protection layer (e.g. XSS, input sanitization) • Healthy feature addition / deprecation ratio • Trustworthy deprecation policy, no surprises on updates 35
  • 36. BIKE-SHEDDING • „… disproportionate weight to trivial issues“ [Wikipedia] • In combination with users that see you as vendor, this is annoying • Explain your prioritization policy • Be the representative for all silent (student) users • Remain grateful that they contribute ‚something‘ • There are books about that. • It’s open source. Let them fork and see what happens. 36
  • 37. FUTURE STEPS • More test coverage (for the GitHub badge) • More documentation (for the users) • Test machines withVagrant / Docker / libvirt • „Hyper-agile cloud-scale load management“ • UI homogenization with (Django) Grappelli • LTI provider • Improved grading with sophisticated in-browser preview 37
  • 38. CONCLUSION • OpenSubmit brings the KISS principle to learning management • Focus on one problem, and do this right • Developing your own tools for daily work is fun, but … • Consider that you will create 
 technical debt. • If you want people to use it, 
 you need advertising. 38
  • 39. 39 OpenSubmit Web • Python + Django • PostgreSQL • python-social Students OpenSubmit Exec OpenSubmit Exec OpenSubmit Exec • Python + Linux https://github.com/troeger/opensubmit Student Code Validation Script