First prize for Engineering Design at ISTE 2011

1. The Essay Scoring Tool - TEST B.E Project presentation Submitted by: Abhinav Gupta 201/CO/03 Danish Contractor 233/CO/03 Gaurav Singh 238/CO/03 Himanshu Mehrotra 241/CO/03 Under the guidance of: Dr. Shampa Chakraverty COE Dept. NSIT Date of presentation: 1 st June 2007 NSIT, Delhi

2. PRIOR WORK NSIT, Delhi

3. Overview of the Software NSIT, Delhi Student Essay TEST Essay TEST Training Essays INPUTS Spelling & Grammatical Checks Corpus Facts Feedback to student Score OUTPUTS

4. Scoring Parameters NSIT, Delhi Scoring Engine Quality of Content Global Coherence Factual Accuracy Local Coherence

5. SINGULAR VALUES (K) RETAINED NSIT, Delhi

7. LOCAL COHERENCE – Good Essays Average variance from gold standard - 0.0219 NSIT, Delhi

8. LOCAL COHERENCE – Other Essays NSIT, Delhi Average variance from gold standard - 0.212

9. LOCAL COHERENCE- Combined Essays NSIT, Delhi Series 1 : Good essays Series 2 : Other Essays

10. LOCAL COHERENCE - MARKING SCHEME NSIT, Delhi

11. LOCAL COHERENCE - MARKS NSIT, Delhi

12. CONTENTS-ESSAYS TO BE MARKED NSIT, Delhi

13. CONTENT – Good Essays NSIT, Delhi

14. CONTENT – Other Essays NSIT, Delhi

15. CONTENT - COMBINED SERIES 1 : GOOD ESSAYS SERIES 5: OTHER ESSAYS NSIT, Delhi

16. CONTENT-NORMALIZED MARKS NSIT, Delhi

18. GOOD STRUCTURED ESSAY NSIT, Delhi

19. AVERAGELY STRUCTURED ESSAY NSIT, Delhi

20. BADLY STRUCTURED ESSAY NSIT, Delhi

21. GLOBAL COHERENCE MARKS NSIT, Delhi

22. Fact Evaluation Module NSIT, Delhi TEST Fact Evaluation Module Topic Specific Keywords List of Essays Correct Facts List Incorrect Facts List Individual Essay Reports & Scores N X 1 Score Matrix (For Internal use by TEST)

23. Fact Evaluation No. of facts matched:4 No. of Incorrect Facts matched:1 SCORE: 0.8 NSIT, Delhi

24. Breakup of Essay Scores NSIT, Delhi

25. Human scores v/s TEST scores NSIT, Delhi

28. COMPARISON OF TEST WITH OTHER AES TOOLS PEG IEA E-Rater TEST Evaluation parameters Essay length, Complexity of sentence and word length Similarity with gold standard Lexical complexity, Vocabulary, Essay organization and many more.. Similarity with gold standard, Essay organization,Fact Accuracy. Feedback No Yes Yes Yes Essay content checking No Yes Yes Yes Fact checking No No Yes Yes Training phase Time consuming & inexpensive Time consuming & inexpensive Time consuming & expensive Time consuming & inexpensive Language of essays English English English Hindi Performance Correlation of 0.87 with human raters Correlation of 0.85 with human raters. Correlation of 0.87 with human raters. Correlation of 0.7652 with human raters.

34. Q & A ? NSIT, Delhi

38. Overview of the Software

41. Training corpus of gold standard essay and other articles, essays on the same topic + Essay under evaluation Term-document matrix (M) (After Singular-value decomposition) Three matrices – T,S and D (T=Term matrix, S=Singular-values matrix and D=document matrix) Dimensionality reduction and preserving only 2 largest dimensions in S gives S-improved (Multiplying T, S-improved and D) New Term by Document matrix LSA: Steps involved

42. LSA Example Titles of Some Technical Memos • c1: Human machine interface for ABC computer applications • c2: A survey of user opinion of computer system response time • c3: The EPS user interface management system • c4: System and human system engineering testing of EPS • c5: Relation of user perceived response time to error measurement • m1: The generation of random, binary, ordered trees • m2: The intersection graph of paths in trees • m3: Graph minors IV: Widths of trees and well- quasi- ordering • m4: Graph mino rs : A survey

43. LSA Example : Term by document matrix

44. LSA Example: After SVD

45. LSA Example: Results Similarity between documents: C1 and C2 = 0.91 (high) C1 and C3 = 1.00 (very-high) C1 with C5 = 0.85(high) C2 with C3 = 0.91 (high) C1 and M1 = -0.85 (low) M1 and M2 = 1.00 (very-high) M2 and M3 = 1.00 (very-high) C2 and C3 = 0.91 (high)

46. Local Coherence Estimation What is Coherence? Each sentence in an essay is connected to previous sentences. The degree of this connection measures the coherence of the sentence pairs. Coherence estimation using LSA: By comparing vectors for two adjoining segments of text in a semantic space, LSA measures degree of semantic relatedness between the segments.

52. Local Coherence Module NSIT, Delhi The reduced term-document Matrix after LSA Evaluation Essay column number in term-document matrix Score on Local Coherence Feedback to Student Local Coherence Module

53. Local Coherence Results NSIT, Delhi

54. Content Evaluation Module NSIT, Delhi Set of Domain Specific Golden Standard Essays Set of Essays to be evaluated Essay Content Evaluation Module Normalized scores on basis of Content

55. Content Evaluation Results NSIT, Delhi

56. Content Evaluation Normalized Results NSIT, Delhi

57. Global Coherence Module NSIT, Delhi Golden Standard Essays Global Coherence Evaluation Module Feedback Score Evaluation Essay(s)

58. Global Coherence Evaluation Effect of K NSIT, Delhi