SlideShare ist ein Scribd-Unternehmen logo
1 von 14
Downloaden Sie, um offline zu lesen
An Ontology for Open Rubric Exchange on the Web
Brian Panulla
Principal
Panulla Information Systems
4521 NE 27th
Avenue
Portland, OR 97211 USA
bpanulla@gmail.com
Megan Kohler
Instructional Designer, e-Learning Institute
The Pennsylvania State University
12 Borland Building
University Park, PA USA
msm26@psu.edu
Abstract
While the Internet has given educators access to a steady supply of Open Educational
Resources, the educational rubrics commonly shared on the Web are generally in the form of
static, non-semantic presentational documents or in the proprietary data structures of
commercial content and learning management systems.
With the advent of Semantic Web Standards, producers of online resources have a new
framework to support the open exchange of software-readable datasets. Despite these
advances, the state of the art of digital representation of rubrics as sharable documents has not
progressed.
This paper proposes an ontological model for digital rubrics. This model is built upon the
Semantic Web Standards of the World Wide Web Consortium (W3C), principally the
Resource Description Framework (RDF) and Web Ontology Language (OWL).
Keywords: rubric, ontology, open educational resources, Semantic Web
About The Authors
Brian Panulla has worked to build a better Web in higher education since 1994, primarily as
professional staff at Pennsylvania State University. He holds a B.S. in Mathematical Science
from Penn State. His current areas of interest include distributed intelligent systems, Rich
Internet Applications, information visualization and sonification, and resource-oriented
information systems. He currently resides in Portland, Oregon USA.
Megan Kohler brings a lifelong love of the arts to her role as Instructional Designer for the
e-Learning Institute in the College of Arts and Architecture at Pennsylvania State University.
She holds a B.A. in Theater and Communication Studies and an M.S. in Instructional
Technology from Bloomsburg University. Her principle responsibilities within the Institute
include project management and assisting faculty in the analysis, design, development, and
implementation of online and blended courses.
Benefits of Rubrics
Much attention has been paid in recent years to the effectiveness of rubrics in fostering a
more level, efficient and scalable educational experience. Rubrics serve as “the criteria for a
piece of work, or ‘what counts’” (Goodrich, 1997) during the assessment or evaluation of a
student work, activity, or artifact. They can be a powerful aid to educators in communicating
their expectations to learners.
Traditional educational assessments, such as quizzes, seek to evaluate a student’s retention of
knowledge rather than a “student’s ability to apply skills and knowledge to real-world
problems.” (Marzano). This effect becomes more pronounced in high enrolling courses
regardless of the delivery method (online, blended, or face-to-face) due to difficulties of
scaling up these methods.
Rubrics offer a unique opportunity to replace these traditional assessment methods with
projects and creative assignments. These authentic assessment activities (Rocco) enable
students to learn through direct application of concepts. Tierney & Simon found rubrics
“especially useful in assessment for learning because they contain qualitative descriptions of
performance criteria that work well within the process of formative evaluation.” By
understanding the benefits of rubrics from both the student and instructor perspective we can
begin to examine the individual elements of rubrics and how we can best utilize them in
education.
Assessments based on students’ knowledge retention from lectures and course readings often
provide limited guidance or focus on what information is most important. In this format
nearly all of the information may appear to have a significant value depending on an
individual perspective or interpretation. Rubrics have the ability to clarify the learning
objectives and guide students toward meeting the predetermined objectives through
meaningful activities.
Rubrics also enable instructors to more clearly communicate their expectations to students by
identifying the criteria required to obtain a certain level of success. Andrade (2000) posited
that we “often expect students to just know what makes a good essay, a good drawing, or a
good science project, so we don't articulate our standards for them.” By not establishing
criteria or standards for an assignment, students are forced to push blindly forward seeking
unidentified milestones that may not even exist until the first project is graded.
In a similar manner, rubrics also have the ability to normalize subjective assignments. If no
criterion has been identified for students, they often associate or establish their own standards
based on previous experiences. “In comparing criteria mentioned by students, I found that
students with no experience with rubrics tended to mention fewer and more traditional
criteria. Students who had used rubrics tended to mention the traditional criteria, plus a
variety of other criteria—often the criteria from their rubrics.” (Andrade, 2000) When
presented with any type of evaluative task students strive to establish some type of construct
to inform their creation process which may or may not align with the instructors intent. In a
similar manner, students will have different perceptions of what type of criteria is required
for a passing grade. Through the inclusion of rubrics, students are provided with a set of
standardized goals, which enables them to focus on the creation of the project rather than
trying to determine the instructor’s intent.
Once a consistent and reliable set of criteria has been established, the students can then utilize
the rubrics as a form of self-assessment. Students can learn to evaluate their work while
referencing the rubric. This enables them to identify the gaps or weak areas of their projects
prior to submitting it for an instructor review, thereby instilling valuable self evaluation
skills. Instructional rubrics provide students with more informative feedback about their
strengths and areas in need of improvement than traditional forms of assessment do.
(Andrade, 2000) The use of rubrics enable students to achieve the standards set forth by the
instructor, in addition to learning how to review and revise their own work.
In addition to providing educational benefits for students, rubrics also have the ability to
significantly impact an instructors ability to grade assessments. Once an instructor has
established a scoring rubric, it can be duplicated and adapted to address variances in
assignments. This allows assignments with similar attributes to share a base set of standards,
yet allowing for the individual differences to be addressed. For scoring rubrics to fulfill their
educational ideal, they must first be designed or modified to reflect greater consistency in
their performance criteria descriptors. (Tierney & Simon) Rubrics can also serve as a
standard measure across course sections, ensuring all instructors or Teaching Assistants
assign grades based on a shared standard.
While a main purpose of rubrics is to set forth an established and standardized set of criteria
to benefit students, rubrics also have the ability to significantly impact an instructors’
effectiveness when teaching high-enrollment courses. Rubrics can assist instructors in the
following ways:
• Ensure a consistent grading scale across all course sections, enabling all instructors or
Teaching Assistants to assign grades based on a shared standard.
• Allow the instructor to efficiently review a student’s work by select the appropriate
level of accomplishment, while respecting the instructors limited time.
• Reduce grading time by assigning the standardized criteria to students projects which
enables the instructor to provide additional customized responses to the submissions
• Provide scalability by decreasing grading time enabling instructors to take on a
greater number of students with less of an impact on their teaching load.
Rubrics and the Internet
With all the benefits to the educational process provided by rubrics, it should come as no
surprise that a wide variety of rubrics can be found on the Internet. Many educational
institutions, from primary schools to universities, publish collections of rubrics created or
used by their faculty.
This quantity, however, is not of equal quality: “The most accessible rubrics, particularly
those available on the Internet, contain design flaws that not only affect their instructional
usefulness, but also the validity of their results.” (Tierney & Simon) In many cases, Internet
rubrics provide only rough scoring guidelines, and lack the specificity of feedback to provide
real value to the learning experience, thereby failing both the educator and the learner.
In addition, most available rubrics are in a presentational format – typically a visual
representation meant for human consumption. Examples include:
• Documents – Microsoft Word, Portable Document Format (PDF)
• Spreadsheets – Microsoft Excel
• Non-semantic Web pages – HyperText Markup Language (HTML)
While generally quite usable, presentational format rubrics are ill suited for inclusion as part
of an interactive software system. Such systems will be unable to use these rubrics to mediate
grading activities until it has been converted to the system’s own internal format.
Software tools to facilitate creation of rubrics are also plentiful. A survey of these tools
reveals that most produce rubrics in proprietary semantic representations that are not open
and transferable between systems. These representations are different for each system and are
used internally to manage the storage and retrieval of the data and metadata. On-line tools
such as RubiStar (ALTEC, 2010) and the Rubric Machine (Warlick & The Landmark Project,
2010) are examples of this. While both systems are fairly easy to use and provide large
libraries of existing rubrics, they provide only limited output formats for documents. Anyone
wishing to transfer the rubrics they create with these tools to another system, such as an LMS,
must re-create the rubric in the new system or transfer it in a presentational format (e.g. PDF,
HTML). In addition, some of the more advanced systems, such as ANGEL Grading Rubrics
(ANGEL Learning, 2010) and Rubrix (Discovery Software, 2010) charge significant
licensing fees for their use, making a proprietary data model even less desirable.
What is needed is an open, non-proprietary way of describing rubrics for both machine and
human use.
Enter The Semantic Web
In order to meet the requirement of an open representation for rubrics in software systems, we
must have a means of describing the entities, components and relationships present in the
education knowledge domain. This set of ideas forms the conceptualization of the domain.
By explicitly describing this conceptualization in a formal way we create an ontology.
(Gruber, 1993)
While many frameworks exist for specifying ontologies, two complementary frameworks
have been by the World Wide Web Consortium (W3C) as Semantic Web Standards. These
frameworks are the Resource Description Framework (RDF) and Web Ontology Language
(OWL). OWL 2.0 (World Wide Web Consortium [W3C], 2009) represents the state of the art
for the creation of open ontologies and is a superset of RDF (W3C “RDF Concepts and
Abstract Syntax”, 2004). Information described in OWL can be consumed and manipulated a
large number of software packages and libraries.
By using OWL to describe our ontology we can also leverage the growing number of existing
RDF and OWL vocabularies to describe aspects of our data model not central to the rubric
domain. Some excellent examples of existing ontologies that complement our rubric ontology
are:
• Friend-Of-A-Friend (FOAF) - a vocabulary for describing people, such as creators
and users of rubrics (Brickley & Miller, 2010)
• Dublin Core Metadata Initiative (DCMI) - a popular system for describing metadata
for digital assets (DCMI, 2010)
• Creative Commons - for denoting the rights granted by the creator of digital assets to
their users (Creative Commons, 2010)
The ePortfolio Ontology (Wang, 2009) is another good example of a complementary
ontology. It models the various entities in a student ePortfolio system, including a minimal
Rubric entity. We can use this Rubric as an integration point to allow systems built on that
model to leverage rubrics created according to our ontology.
While a thorough introduction to OWL and RDF is beyond the scope of this paper, we will
touch on some aspects of OWL and RDF in the next section when necessary to shed light on
the rubric ontology implementation.
Overview of The Rubric Ontology
The key entities in the rubric ontology are: Rubric, Criterion, Level, and Category. Two more
entities, Scope and Scoring, indicate the intended application of the rubric by users and
software systems. Figure 1 shows a simple UML class diagram detailing the basic
relationships between these key entities.
Rubric
The central entity of the rubric ontology is naturally the Rubric. A review of the literature
around use of rubrics has identified at least three distinct subtypes of rubrics:
• Analytic – analytic rubrics break down the assessment or evaluation of a work into
discrete criteria. These criteria are generally tied back to the learning objectives of the
unit or course. (Rocco) Each criterion is further broken down into levels of
achievement, from minimal to exemplary.
• Holistic – Holistic rubrics are used to assess “the whole of a process, performance, or
product” (Rocco). This type of rubric is used when criteria overlap or are otherwise
hard to isolate.
• Primary Trait – Similar to the analytic rubric, the primary trait rubric “describes in
detail what is required for performance” (Rocco). Rather than breaking down to
discrete levels, a free-form evaluation or assessment is made for each criterion. A
variant, the scoring guide rubric, is found in several sources, including (Stevens and
Levi)
Analytic rubrics are by far the most common subtype in online rubric repositories. They also
appear to be the most regular form of rubric in structure. The only significant variation found
was whether the number of levels per criterion was uniform (making the rubric a regular grid
or table) or variable (some criteria possessing a finer or rougher gradation of scoring).
In contrast, holistic rubrics varied widely in appearance. Some seemed to be nearly identical
to analytic rubrics, differing only in their wording and intended use (Rocco), while others
were very different. (Bargannier, Mertler).
Property Description
title Title of rubric
description Free-text description of intended use.
hasScope See Scope and Scoring below.
hasScoring See Scope and Scoring below.
hasCriteria Set of criteria/categories (mainly Analytic Rubrics).
Table 1: Properties of Rubric class
Criterion
The main building block of the analytic and primary trait rubrics is the Criterion (plural
Criteria). Each Criterion represents a focused “part of the task”. (Stevens and Levi). In the
grid form of the analytic rubric, each row of the grid corresponds to a criterion. While we
chose Criterion as the canonical name for this element, it is called a dimension in at least one
source (Stevens and Levi). In our opinion this is merely a symbolic variation and not a
semantic difference between conceptualizations.
Property Description
title Title of criterion
description Free-text description of meaning of criterion.
weight Weighting factor for scored rubrics
hasLevels Set of levels (mainly Analytic Rubrics).
Table 2: Properties of Criterion class
Level
The Level is the main component of the Rubric class in a holistic rubric, and of Criteria in
analytic rubrics. The set of Levels should prescribe the range of assessment outcomes, from a
low achievement (e.g. “poor” or “incomplete”) to high achievement (e.g. “exemplary”). The
columns of a tabular analytic rubric generally correspond with the Level entities. Similarly,
the rows of a holistic rubric will generally map to Levels.
Property Description
benchmark Text describing characteristics of this degree of
achievement. Can have one or more per level.
quality A qualitative description of this degree of
achievement. Used for column headers in tabular
rubrics
score The points awarded for achieving this level.
feedback Pre-defined feedback text to be relayed to the learner;
may include guidance and suggestions for
improvement or development.
Table 3: Properties of Level class
Category
Categories are simple containers to aggregate multiple sequenced Criteria. We generally
found categories in large or complex analytic rubrics. Categories can also play a useful role
in software user interfaces based on rubrics. The Faculty Self-Assessment Tool from Penn
State (Panulla, Rocco and McQuiggan, 2008) places the Criteria within a given Category in
the same section of an accordion control (See Figure 2).
Property Description
title Title of category.
hasCriteria Set of criteria/categories (mainly Analytic Rubrics).
Table 4: Properties of Category class
Scope
The Scope of a rubric is meant to indicate what the rubric’s creator intended to assess or
evaluate. The current ontology defines four distinct scopes:
• Individual – used by a teacher or educator to assess or evaluate the work of a single
individual learner.
• Team – used by a teacher or educator to assess or evaluate the work of a group or
team. Team rubrics are commonly found in Problem-Based Learning (PBL)
environments.
• Peer – used by one individual learner to evaluate or assess the work of another
individual learner. Peer rubrics are also common in PBL environments; teammate
evaluations may play a role in an overall course participation grade.
• Self – used by an individual to assess his or her own learning or development.
Specifying the intended Scope of a rubric can provide two powerful advantages:
• Search systems may filter results from a database of rubrics, eliminating those created
for a different purpose than the one desired.
• Software systems may use the Scope to identify how it should present the rubric to the
user in different contexts. For example, a secure online Peer-Review system with
knowledge of course group assignments may allow students to securely and privately
evaluate the other members of their team via a rubric, while preventing other students
from seeing or affecting those evaluations.
Scoring
The Scoring attribute of a rubric allows creators to create both scored and unscored rubrics.
While scored rubrics are likely to be the norm, educators sometimes find it desirable to assess
a student’s learning progress without having it directly impact their grade. It is expected that
software systems displaying unscored rubrics should still provide qualitative feedback to the
learner.
Artificial Entities
As with any computerized representation of an information model, there are several entities
introduced into the rubric ontology by the idiosyncrasies of underlying technology.
It can be difficult to maintain closed, explicitly ordered lists of items in the present version of
OWL. While the RDF Schema specification (W3C “RDF Vocabulary Description
Language”, 2004) does define several Collection classes for aggregation, these classes do not
provide semantics for restricting their contents to given classes of entities.
To get around this problem and strictly model lists of Criteria and Levels we have defined
two additional entities: CriteriaList and LevelList, based off the RDF List Collection.
Class Description
CriteriaList Contains Criterion and Category instances in explicit order.
Category instances contain an additional CriteriaList,
resulting in a tree structure of Categories and Criteria.
LevelList Contains Level instances in explicit order.
Table 5: List classes
Future Goals
Holistic Rubrics
The model of holistic rubrics is somewhat incomplete. While a holistic rubric may be
modeled superficially using the existing entities in the ontology, a more thorough explanation
of the use and intent of holistic rubrics may ultimately lead to a different data model.
Improved List Modeling
The use of RDF Lists in OWL ontologies results in the ontology being validated as OWL-
Full (W3C, 2009). Reasoning over OWL Full ontologies cannot be guaranteed to be finite or
even efficient. Consequently, it is a best practice to avoided models that push into the OWL-
Full realm. It may be possible to replace the RDF Lists-derived classes in the rubric ontology
with an alternate model that allows validation as OWL DL without a significant loss of
compatibility or semantic expressiveness.
Implementations
At present, three systems are under development at Pennsylvania State University that can
create and/or consume rubrics described according to this ontology:
• The Faculty Self-Assessment tool (Panulla, Rocco and McQuiggan, 2008) was the
prototype application in which many of the ideas captured in the current version of the
ontology were developed.
• The Assignment Studio Rubric module for Drupal (Bailey and Ollendyke, 2009) is
used at Penn State University’s College of Arts and Architecture to manage course
activities in several resident and hybrid courses. An updated version of this module
for Drupal 7 that incorporates this rubric ontology is currently in the design phase.
• A Rubric Builder Rich Internet Application (RIA) is currently under development for
Spring 2011. The Builder provides a rich, easy to use interface that captures many of
the best practices of rubric design and guides the user to produce better rubrics.
Figures
Fig. 1 – Simplified Rubric UML model
Figure 2: Categories as UI navigational elements
References
Andrade, H. G. (2000). Using rubrics to promote thinking and learning. Educational
Leadership 57 (5): 13–18.
ALTEC. (2010). RubiStar. Retrieved September 10, 2010, from http://rubistar.4teachers.org/
ANGEL Learning. (2010). Grading Rubrics. Retrieved September 10, 2010, from
http://www.angellearning.com/products/lms/rubrics.html
Bailey, K and B. Ollendyke. Rubric module for Drupal. Retrieved September 10, 2010 from
http://drupal.org/project/rubric
Brickley, D. and Miller, L. (2010). FOAF Vocabulary Specification 0.98. Retrieved from
http://xmlns.com/foaf/spec/
Creative Commons. (2010). Describing Copyright in RDF. Retrieved September 10, 2010,
from http://creativecommons.org/ns
DCMI. (2010) Dublin Core Metadata Initiative. Retrieved from
http://dublincore.org/schemas/rdfs/
Discovery Software. (2010). Rubrix. Retrieved September 10, 2010, from http://rubrix.com/
Goodrich, H. (1997). Understanding rubrics. Educational Leadership 54 (4): 14–17.
Gruber, T. (1993). A translation approach to portable ontology specifications. Knowledge
Acquisition, 5(2), 199-220.
Marzano, R., D. Pickering, and J. McTighe (1993). Assessing Student Outcomes:
Performance Assessment Using the Dimensions of Learning Model. Alexandria, VA:
ASCD.
Panulla, B. J., S. A. Rocco, and C. McQuiggan (2008). Faculty Self-Assessment: Preparing
for Online Teaching. Retrieved September 10, 2010 from
https://weblearning.psu.edu/FacultySelfAssessment/
Rocco, S. A. (2007). Assessment and evaluation in the online environment. New Directions
in Adult and Continuing Education, 119, 75-86.
Stevens, D. D., and Levi, A. J. (2005). Introduction to Rubrics. Sterling, Virginia: Stylus.
Tierney, R., and Simon, M. (2004). What's still wrong with rubrics: focusing on the
consistency of performance criteria across scale levels. Practical Assessment, Research &
Evaluation, 9(2).
Warlick, D. & The Landmark Project. (2010). The Rubric Machine. Retrieved September 10,
2010, from http://landmark-project.com/rubric_builder/
Wang, S. (2009) E-Portfolios for Integrated Reflection. Issues in Informing Science and
Information Technology. Volume 6, 2009. Retrieved from
http://iisit.org/Vol6/IISITv6p449-460Wang630.pdf
World Wide Web Consortium. (2004). Resource Description Framework (RDF): Concepts
and Abstract Syntax. Retrieved from http://www.w3.org/TR/rdf-concepts/
World Wide Web Consortium. (2004). RDF Vocabulary Description Language 1.0: RDF
Schema. Retrieved from http://www.w3.org/TR/rdf-schema/
World Wide Web Consortium. (2009). OWL 2 Web Ontology Language Document
Overview. Retrieved from http://www.w3.org/TR/2009/REC-owl2-overview-20091027/

Weitere ähnliche Inhalte

Was ist angesagt?

Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPeter Brusilovsky
 
Evaluation of online learning
Evaluation of online learningEvaluation of online learning
Evaluation of online learningAlaa Sadik
 
A learning design toolkit for creating effective learning activities
A learning design toolkit for creating effective learning activitiesA learning design toolkit for creating effective learning activities
A learning design toolkit for creating effective learning activitiesgrainne
 
2015 11-17 Venia Legendi Kairit Tammets
2015 11-17 Venia Legendi Kairit Tammets2015 11-17 Venia Legendi Kairit Tammets
2015 11-17 Venia Legendi Kairit Tammetsifi8106tlu
 
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, TurkeyAn insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkeystrehlst
 
2015-11-19 Venia Legendi: Vladimir Tomberg
2015-11-19 Venia Legendi: Vladimir Tomberg2015-11-19 Venia Legendi: Vladimir Tomberg
2015-11-19 Venia Legendi: Vladimir Tombergifi8106tlu
 
The User Side of Personalization: How Personalization Affects the Users
The User Side of Personalization: How Personalization Affects the UsersThe User Side of Personalization: How Personalization Affects the Users
The User Side of Personalization: How Personalization Affects the UsersPeter Brusilovsky
 
E-learning research methodological issues
E-learning research methodological issuesE-learning research methodological issues
E-learning research methodological issuesgrainne
 
Conole Jisc Lxp
Conole Jisc LxpConole Jisc Lxp
Conole Jisc Lxpgrainne
 
Bridging the gap: e-learning research
Bridging the gap: e-learning researchBridging the gap: e-learning research
Bridging the gap: e-learning researchgrainne
 
DialogPlus toolkit
DialogPlus toolkitDialogPlus toolkit
DialogPlus toolkitgrainne
 
Lamar coehd regional_research_presentation_final_3-23-12
Lamar coehd regional_research_presentation_final_3-23-12Lamar coehd regional_research_presentation_final_3-23-12
Lamar coehd regional_research_presentation_final_3-23-12Lamar University
 
Partnerships to PLCs - Dr. Michael Johanek, University of Pennsylvania
Partnerships to PLCs - Dr. Michael Johanek, University of PennsylvaniaPartnerships to PLCs - Dr. Michael Johanek, University of Pennsylvania
Partnerships to PLCs - Dr. Michael Johanek, University of PennsylvaniaJoe Mazza, Ed.D.
 
Personality Assessment using Twitter Tweets
Personality Assessment using Twitter TweetsPersonality Assessment using Twitter Tweets
Personality Assessment using Twitter TweetsNadeem Ahmad Ch
 
LRT Talks 2013-03-12 CETIS
LRT Talks 2013-03-12 CETISLRT Talks 2013-03-12 CETIS
LRT Talks 2013-03-12 CETISMark Stubbs
 
Human Interfaces to Artificial Intelligence in Education
Human Interfaces to Artificial Intelligence in EducationHuman Interfaces to Artificial Intelligence in Education
Human Interfaces to Artificial Intelligence in EducationPeter Brusilovsky
 
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Callejaifi8106tlu
 
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
Learning Analytics in Education:  Using Student’s Big Data to Improve TeachingLearning Analytics in Education:  Using Student’s Big Data to Improve Teaching
Learning Analytics in Education: Using Student’s Big Data to Improve TeachingRafael Scapin, Ph.D.
 
Usability Analysis of Educational Information Systems from Student’s Perspective
Usability Analysis of Educational Information Systems from Student’s PerspectiveUsability Analysis of Educational Information Systems from Student’s Perspective
Usability Analysis of Educational Information Systems from Student’s PerspectiveNadeem Ahmad Ch
 

Was ist angesagt? (20)

Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning Programming
 
Evaluation of online learning
Evaluation of online learningEvaluation of online learning
Evaluation of online learning
 
A learning design toolkit for creating effective learning activities
A learning design toolkit for creating effective learning activitiesA learning design toolkit for creating effective learning activities
A learning design toolkit for creating effective learning activities
 
2015 11-17 Venia Legendi Kairit Tammets
2015 11-17 Venia Legendi Kairit Tammets2015 11-17 Venia Legendi Kairit Tammets
2015 11-17 Venia Legendi Kairit Tammets
 
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, TurkeyAn insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
An insight into Educational Data Mining at Muğla Sıtkı Koçman University, Turkey
 
2015-11-19 Venia Legendi: Vladimir Tomberg
2015-11-19 Venia Legendi: Vladimir Tomberg2015-11-19 Venia Legendi: Vladimir Tomberg
2015-11-19 Venia Legendi: Vladimir Tomberg
 
The User Side of Personalization: How Personalization Affects the Users
The User Side of Personalization: How Personalization Affects the UsersThe User Side of Personalization: How Personalization Affects the Users
The User Side of Personalization: How Personalization Affects the Users
 
E-learning research methodological issues
E-learning research methodological issuesE-learning research methodological issues
E-learning research methodological issues
 
Conole Jisc Lxp
Conole Jisc LxpConole Jisc Lxp
Conole Jisc Lxp
 
Bridging the gap: e-learning research
Bridging the gap: e-learning researchBridging the gap: e-learning research
Bridging the gap: e-learning research
 
DialogPlus toolkit
DialogPlus toolkitDialogPlus toolkit
DialogPlus toolkit
 
Lamar coehd regional_research_presentation_final_3-23-12
Lamar coehd regional_research_presentation_final_3-23-12Lamar coehd regional_research_presentation_final_3-23-12
Lamar coehd regional_research_presentation_final_3-23-12
 
Partnerships to PLCs - Dr. Michael Johanek, University of Pennsylvania
Partnerships to PLCs - Dr. Michael Johanek, University of PennsylvaniaPartnerships to PLCs - Dr. Michael Johanek, University of Pennsylvania
Partnerships to PLCs - Dr. Michael Johanek, University of Pennsylvania
 
Personality Assessment using Twitter Tweets
Personality Assessment using Twitter TweetsPersonality Assessment using Twitter Tweets
Personality Assessment using Twitter Tweets
 
Addictive links: Adaptive Navigation Support in College-Level Courses
Addictive links: Adaptive Navigation Support in College-Level CoursesAddictive links: Adaptive Navigation Support in College-Level Courses
Addictive links: Adaptive Navigation Support in College-Level Courses
 
LRT Talks 2013-03-12 CETIS
LRT Talks 2013-03-12 CETISLRT Talks 2013-03-12 CETIS
LRT Talks 2013-03-12 CETIS
 
Human Interfaces to Artificial Intelligence in Education
Human Interfaces to Artificial Intelligence in EducationHuman Interfaces to Artificial Intelligence in Education
Human Interfaces to Artificial Intelligence in Education
 
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
2016-05-31 Venia Legendi (CEITER): Adolfo Ruiz Calleja
 
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
Learning Analytics in Education:  Using Student’s Big Data to Improve TeachingLearning Analytics in Education:  Using Student’s Big Data to Improve Teaching
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
 
Usability Analysis of Educational Information Systems from Student’s Perspective
Usability Analysis of Educational Information Systems from Student’s PerspectiveUsability Analysis of Educational Information Systems from Student’s Perspective
Usability Analysis of Educational Information Systems from Student’s Perspective
 

Andere mochten auch

Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014
Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014 Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014
Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014 Jane Cockburn
 
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?The Semantic Web – A Vision Come True, or Giving Up the Great Plan?
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?Martin Hepp
 
De Empreendedor à Empresário
De Empreendedor à EmpresárioDe Empreendedor à Empresário
De Empreendedor à EmpresárioFernando Tomé
 
Ux och design som konverterar del 1
Ux och design som konverterar   del 1Ux och design som konverterar   del 1
Ux och design som konverterar del 1Wipcore
 
Scottish Public Opinion Monitor - May 2013
Scottish Public Opinion Monitor - May 2013Scottish Public Opinion Monitor - May 2013
Scottish Public Opinion Monitor - May 2013Ipsos UK
 
Big Data World presentation - Sep. 2014
Big Data World presentation - Sep. 2014Big Data World presentation - Sep. 2014
Big Data World presentation - Sep. 2014Wing Yuen Loon
 
Actielijst 201304
Actielijst 201304Actielijst 201304
Actielijst 201304Kees Dekker
 
The Tablet Project: Charts
The Tablet Project: ChartsThe Tablet Project: Charts
The Tablet Project: ChartsNewsworks
 
淺談HTTP發展趨勢與SPDY
淺談HTTP發展趨勢與SPDY淺談HTTP發展趨勢與SPDY
淺談HTTP發展趨勢與SPDYBilly Yang
 
Tutorial avanzado wordle
Tutorial avanzado wordleTutorial avanzado wordle
Tutorial avanzado wordleAníbal Rossi
 

Andere mochten auch (15)

Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014
Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014 Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014
Design Thinking for Advanced Manufacturing _ Industry Recommendations_Dec 2014
 
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?The Semantic Web – A Vision Come True, or Giving Up the Great Plan?
The Semantic Web – A Vision Come True, or Giving Up the Great Plan?
 
De Empreendedor à Empresário
De Empreendedor à EmpresárioDe Empreendedor à Empresário
De Empreendedor à Empresário
 
Ux och design som konverterar del 1
Ux och design som konverterar   del 1Ux och design som konverterar   del 1
Ux och design som konverterar del 1
 
RASPBIAN JESSIE WITH PIXEL
RASPBIAN JESSIE WITH PIXELRASPBIAN JESSIE WITH PIXEL
RASPBIAN JESSIE WITH PIXEL
 
Scottish Public Opinion Monitor - May 2013
Scottish Public Opinion Monitor - May 2013Scottish Public Opinion Monitor - May 2013
Scottish Public Opinion Monitor - May 2013
 
1 spattern matching using biometric techniques
1 spattern matching using biometric techniques1 spattern matching using biometric techniques
1 spattern matching using biometric techniques
 
Big Data World presentation - Sep. 2014
Big Data World presentation - Sep. 2014Big Data World presentation - Sep. 2014
Big Data World presentation - Sep. 2014
 
Actielijst 201304
Actielijst 201304Actielijst 201304
Actielijst 201304
 
Van Godtsenhoven Karen
Van Godtsenhoven KarenVan Godtsenhoven Karen
Van Godtsenhoven Karen
 
The Tablet Project: Charts
The Tablet Project: ChartsThe Tablet Project: Charts
The Tablet Project: Charts
 
淺談HTTP發展趨勢與SPDY
淺談HTTP發展趨勢與SPDY淺談HTTP發展趨勢與SPDY
淺談HTTP發展趨勢與SPDY
 
CUE one pager
CUE one pagerCUE one pager
CUE one pager
 
Top 7 Features in an LMS
Top 7 Features in an LMSTop 7 Features in an LMS
Top 7 Features in an LMS
 
Tutorial avanzado wordle
Tutorial avanzado wordleTutorial avanzado wordle
Tutorial avanzado wordle
 

Ähnlich wie An ontologyforopenrubricexchangeontheweb

Graded Assessment – Myth Or Fact Ppt Jan 2k10
Graded Assessment – Myth Or Fact Ppt Jan 2k10Graded Assessment – Myth Or Fact Ppt Jan 2k10
Graded Assessment – Myth Or Fact Ppt Jan 2k10KeithH66
 
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...guest3f9d24
 
A review of classroom observation techniques used in postsecondary settings..pdf
A review of classroom observation techniques used in postsecondary settings..pdfA review of classroom observation techniques used in postsecondary settings..pdf
A review of classroom observation techniques used in postsecondary settings..pdfErin Taylor
 
Feedback as dialogue and learning technologies: can e-assessment be formative?
Feedback as dialogue and learning technologies: can e-assessment be formative?Feedback as dialogue and learning technologies: can e-assessment be formative?
Feedback as dialogue and learning technologies: can e-assessment be formative?Centre for Distance Education
 
Increasing Course Revision Efficacy
Increasing Course Revision EfficacyIncreasing Course Revision Efficacy
Increasing Course Revision EfficacySteven McGahan
 
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...Sheila Sinclair
 
Ch. 11 designing and conducting formative evaluations
Ch. 11 designing and conducting formative evaluationsCh. 11 designing and conducting formative evaluations
Ch. 11 designing and conducting formative evaluationsEzraGray1
 
Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Tom Kohntopp
 
Student Centered Assessment
Student Centered AssessmentStudent Centered Assessment
Student Centered Assessmentjoanne chesley
 
Assessment of Information Literacy Learning
Assessment of Information Literacy LearningAssessment of Information Literacy Learning
Assessment of Information Literacy LearningJohan Koren
 
Assessment, feedback, evaluation, & grading
Assessment, feedback, evaluation, & gradingAssessment, feedback, evaluation, & grading
Assessment, feedback, evaluation, & gradingfusonline
 
Safdar -power point slide show--tests vs. portfolios
Safdar -power point slide show--tests vs. portfoliosSafdar -power point slide show--tests vs. portfolios
Safdar -power point slide show--tests vs. portfoliosnrsafdar
 
Improving Research Productivity of Science Technology & Engineering Students ...
Improving Research Productivity of Science Technology & Engineering Students ...Improving Research Productivity of Science Technology & Engineering Students ...
Improving Research Productivity of Science Technology & Engineering Students ...Felipe De Oca
 
Introduction to DIFFERENTIATION through computer assisted instruction
Introduction to DIFFERENTIATION through computer assisted instructionIntroduction to DIFFERENTIATION through computer assisted instruction
Introduction to DIFFERENTIATION through computer assisted instructionMarianne McFadden
 
Competency based learning: State of the U.S. K-12 market
Competency based learning: State of the U.S. K-12 marketCompetency based learning: State of the U.S. K-12 market
Competency based learning: State of the U.S. K-12 marketNewSchools Ignite
 
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docxfelicidaddinwoodie
 
Using Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning NeedsUsing Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning NeedsDreamBox Learning
 
Providing the Spark for CCSS
Providing the Spark for CCSSProviding the Spark for CCSS
Providing the Spark for CCSSKristen Wheat
 

Ähnlich wie An ontologyforopenrubricexchangeontheweb (20)

Graded Assessment – Myth Or Fact Ppt Jan 2k10
Graded Assessment – Myth Or Fact Ppt Jan 2k10Graded Assessment – Myth Or Fact Ppt Jan 2k10
Graded Assessment – Myth Or Fact Ppt Jan 2k10
 
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...
E:\T&La Special Projects\Computer Engineering & Applied Science\Grade...
 
A review of classroom observation techniques used in postsecondary settings..pdf
A review of classroom observation techniques used in postsecondary settings..pdfA review of classroom observation techniques used in postsecondary settings..pdf
A review of classroom observation techniques used in postsecondary settings..pdf
 
Feedback as dialogue and learning technologies: can e-assessment be formative?
Feedback as dialogue and learning technologies: can e-assessment be formative?Feedback as dialogue and learning technologies: can e-assessment be formative?
Feedback as dialogue and learning technologies: can e-assessment be formative?
 
ADOVH Validity and Reliability of Online Assessments.pdf
ADOVH Validity and Reliability of Online Assessments.pdfADOVH Validity and Reliability of Online Assessments.pdf
ADOVH Validity and Reliability of Online Assessments.pdf
 
Increasing Course Revision Efficacy
Increasing Course Revision EfficacyIncreasing Course Revision Efficacy
Increasing Course Revision Efficacy
 
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...
A Comparative Study Of Competency-Based Courses Demonstrating A Potential Mea...
 
Ch. 11 designing and conducting formative evaluations
Ch. 11 designing and conducting formative evaluationsCh. 11 designing and conducting formative evaluations
Ch. 11 designing and conducting formative evaluations
 
Prof. David Scott
Prof. David ScottProf. David Scott
Prof. David Scott
 
Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)Assessment Matters Newsletter_November 2015 (3)
Assessment Matters Newsletter_November 2015 (3)
 
Student Centered Assessment
Student Centered AssessmentStudent Centered Assessment
Student Centered Assessment
 
Assessment of Information Literacy Learning
Assessment of Information Literacy LearningAssessment of Information Literacy Learning
Assessment of Information Literacy Learning
 
Assessment, feedback, evaluation, & grading
Assessment, feedback, evaluation, & gradingAssessment, feedback, evaluation, & grading
Assessment, feedback, evaluation, & grading
 
Safdar -power point slide show--tests vs. portfolios
Safdar -power point slide show--tests vs. portfoliosSafdar -power point slide show--tests vs. portfolios
Safdar -power point slide show--tests vs. portfolios
 
Improving Research Productivity of Science Technology & Engineering Students ...
Improving Research Productivity of Science Technology & Engineering Students ...Improving Research Productivity of Science Technology & Engineering Students ...
Improving Research Productivity of Science Technology & Engineering Students ...
 
Introduction to DIFFERENTIATION through computer assisted instruction
Introduction to DIFFERENTIATION through computer assisted instructionIntroduction to DIFFERENTIATION through computer assisted instruction
Introduction to DIFFERENTIATION through computer assisted instruction
 
Competency based learning: State of the U.S. K-12 market
Competency based learning: State of the U.S. K-12 marketCompetency based learning: State of the U.S. K-12 market
Competency based learning: State of the U.S. K-12 market
 
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx
1DavisP-EL-7003-82DavisP-EL-7003-8Develop Engaging Learn.docx
 
Using Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning NeedsUsing Data to Drive Personalized Math Learning Needs
Using Data to Drive Personalized Math Learning Needs
 
Providing the Spark for CCSS
Providing the Spark for CCSSProviding the Spark for CCSS
Providing the Spark for CCSS
 

Kürzlich hochgeladen

Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17Celine George
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphNetziValdelomar1
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17Celine George
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationMJDuyan
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17Celine George
 
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...Nguyen Thanh Tu Collection
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice documentXsasf Sfdfasd
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational PhilosophyShuvankar Madhu
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 

Kürzlich hochgeladen (20)

Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a Paragraph
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive Education
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17
 
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
 
The Singapore Teaching Practice document
The Singapore Teaching Practice documentThe Singapore Teaching Practice document
The Singapore Teaching Practice document
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational Philosophy
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 

An ontologyforopenrubricexchangeontheweb

  • 1. An Ontology for Open Rubric Exchange on the Web Brian Panulla Principal Panulla Information Systems 4521 NE 27th Avenue Portland, OR 97211 USA bpanulla@gmail.com Megan Kohler Instructional Designer, e-Learning Institute The Pennsylvania State University 12 Borland Building University Park, PA USA msm26@psu.edu Abstract While the Internet has given educators access to a steady supply of Open Educational Resources, the educational rubrics commonly shared on the Web are generally in the form of static, non-semantic presentational documents or in the proprietary data structures of commercial content and learning management systems. With the advent of Semantic Web Standards, producers of online resources have a new framework to support the open exchange of software-readable datasets. Despite these advances, the state of the art of digital representation of rubrics as sharable documents has not progressed. This paper proposes an ontological model for digital rubrics. This model is built upon the Semantic Web Standards of the World Wide Web Consortium (W3C), principally the Resource Description Framework (RDF) and Web Ontology Language (OWL). Keywords: rubric, ontology, open educational resources, Semantic Web
  • 2. About The Authors Brian Panulla has worked to build a better Web in higher education since 1994, primarily as professional staff at Pennsylvania State University. He holds a B.S. in Mathematical Science from Penn State. His current areas of interest include distributed intelligent systems, Rich Internet Applications, information visualization and sonification, and resource-oriented information systems. He currently resides in Portland, Oregon USA. Megan Kohler brings a lifelong love of the arts to her role as Instructional Designer for the e-Learning Institute in the College of Arts and Architecture at Pennsylvania State University. She holds a B.A. in Theater and Communication Studies and an M.S. in Instructional Technology from Bloomsburg University. Her principle responsibilities within the Institute include project management and assisting faculty in the analysis, design, development, and implementation of online and blended courses.
  • 3. Benefits of Rubrics Much attention has been paid in recent years to the effectiveness of rubrics in fostering a more level, efficient and scalable educational experience. Rubrics serve as “the criteria for a piece of work, or ‘what counts’” (Goodrich, 1997) during the assessment or evaluation of a student work, activity, or artifact. They can be a powerful aid to educators in communicating their expectations to learners. Traditional educational assessments, such as quizzes, seek to evaluate a student’s retention of knowledge rather than a “student’s ability to apply skills and knowledge to real-world problems.” (Marzano). This effect becomes more pronounced in high enrolling courses regardless of the delivery method (online, blended, or face-to-face) due to difficulties of scaling up these methods. Rubrics offer a unique opportunity to replace these traditional assessment methods with projects and creative assignments. These authentic assessment activities (Rocco) enable students to learn through direct application of concepts. Tierney & Simon found rubrics “especially useful in assessment for learning because they contain qualitative descriptions of performance criteria that work well within the process of formative evaluation.” By understanding the benefits of rubrics from both the student and instructor perspective we can begin to examine the individual elements of rubrics and how we can best utilize them in education. Assessments based on students’ knowledge retention from lectures and course readings often provide limited guidance or focus on what information is most important. In this format nearly all of the information may appear to have a significant value depending on an individual perspective or interpretation. Rubrics have the ability to clarify the learning objectives and guide students toward meeting the predetermined objectives through meaningful activities. Rubrics also enable instructors to more clearly communicate their expectations to students by identifying the criteria required to obtain a certain level of success. Andrade (2000) posited that we “often expect students to just know what makes a good essay, a good drawing, or a good science project, so we don't articulate our standards for them.” By not establishing criteria or standards for an assignment, students are forced to push blindly forward seeking unidentified milestones that may not even exist until the first project is graded. In a similar manner, rubrics also have the ability to normalize subjective assignments. If no criterion has been identified for students, they often associate or establish their own standards based on previous experiences. “In comparing criteria mentioned by students, I found that students with no experience with rubrics tended to mention fewer and more traditional criteria. Students who had used rubrics tended to mention the traditional criteria, plus a variety of other criteria—often the criteria from their rubrics.” (Andrade, 2000) When presented with any type of evaluative task students strive to establish some type of construct to inform their creation process which may or may not align with the instructors intent. In a similar manner, students will have different perceptions of what type of criteria is required for a passing grade. Through the inclusion of rubrics, students are provided with a set of standardized goals, which enables them to focus on the creation of the project rather than trying to determine the instructor’s intent.
  • 4. Once a consistent and reliable set of criteria has been established, the students can then utilize the rubrics as a form of self-assessment. Students can learn to evaluate their work while referencing the rubric. This enables them to identify the gaps or weak areas of their projects prior to submitting it for an instructor review, thereby instilling valuable self evaluation skills. Instructional rubrics provide students with more informative feedback about their strengths and areas in need of improvement than traditional forms of assessment do. (Andrade, 2000) The use of rubrics enable students to achieve the standards set forth by the instructor, in addition to learning how to review and revise their own work. In addition to providing educational benefits for students, rubrics also have the ability to significantly impact an instructors ability to grade assessments. Once an instructor has established a scoring rubric, it can be duplicated and adapted to address variances in assignments. This allows assignments with similar attributes to share a base set of standards, yet allowing for the individual differences to be addressed. For scoring rubrics to fulfill their educational ideal, they must first be designed or modified to reflect greater consistency in their performance criteria descriptors. (Tierney & Simon) Rubrics can also serve as a standard measure across course sections, ensuring all instructors or Teaching Assistants assign grades based on a shared standard. While a main purpose of rubrics is to set forth an established and standardized set of criteria to benefit students, rubrics also have the ability to significantly impact an instructors’ effectiveness when teaching high-enrollment courses. Rubrics can assist instructors in the following ways: • Ensure a consistent grading scale across all course sections, enabling all instructors or Teaching Assistants to assign grades based on a shared standard. • Allow the instructor to efficiently review a student’s work by select the appropriate level of accomplishment, while respecting the instructors limited time. • Reduce grading time by assigning the standardized criteria to students projects which enables the instructor to provide additional customized responses to the submissions • Provide scalability by decreasing grading time enabling instructors to take on a greater number of students with less of an impact on their teaching load. Rubrics and the Internet With all the benefits to the educational process provided by rubrics, it should come as no surprise that a wide variety of rubrics can be found on the Internet. Many educational institutions, from primary schools to universities, publish collections of rubrics created or used by their faculty. This quantity, however, is not of equal quality: “The most accessible rubrics, particularly those available on the Internet, contain design flaws that not only affect their instructional usefulness, but also the validity of their results.” (Tierney & Simon) In many cases, Internet rubrics provide only rough scoring guidelines, and lack the specificity of feedback to provide real value to the learning experience, thereby failing both the educator and the learner.
  • 5. In addition, most available rubrics are in a presentational format – typically a visual representation meant for human consumption. Examples include: • Documents – Microsoft Word, Portable Document Format (PDF) • Spreadsheets – Microsoft Excel • Non-semantic Web pages – HyperText Markup Language (HTML) While generally quite usable, presentational format rubrics are ill suited for inclusion as part of an interactive software system. Such systems will be unable to use these rubrics to mediate grading activities until it has been converted to the system’s own internal format. Software tools to facilitate creation of rubrics are also plentiful. A survey of these tools reveals that most produce rubrics in proprietary semantic representations that are not open and transferable between systems. These representations are different for each system and are used internally to manage the storage and retrieval of the data and metadata. On-line tools such as RubiStar (ALTEC, 2010) and the Rubric Machine (Warlick & The Landmark Project, 2010) are examples of this. While both systems are fairly easy to use and provide large libraries of existing rubrics, they provide only limited output formats for documents. Anyone wishing to transfer the rubrics they create with these tools to another system, such as an LMS, must re-create the rubric in the new system or transfer it in a presentational format (e.g. PDF, HTML). In addition, some of the more advanced systems, such as ANGEL Grading Rubrics (ANGEL Learning, 2010) and Rubrix (Discovery Software, 2010) charge significant licensing fees for their use, making a proprietary data model even less desirable. What is needed is an open, non-proprietary way of describing rubrics for both machine and human use. Enter The Semantic Web In order to meet the requirement of an open representation for rubrics in software systems, we must have a means of describing the entities, components and relationships present in the education knowledge domain. This set of ideas forms the conceptualization of the domain. By explicitly describing this conceptualization in a formal way we create an ontology. (Gruber, 1993) While many frameworks exist for specifying ontologies, two complementary frameworks have been by the World Wide Web Consortium (W3C) as Semantic Web Standards. These frameworks are the Resource Description Framework (RDF) and Web Ontology Language (OWL). OWL 2.0 (World Wide Web Consortium [W3C], 2009) represents the state of the art for the creation of open ontologies and is a superset of RDF (W3C “RDF Concepts and Abstract Syntax”, 2004). Information described in OWL can be consumed and manipulated a large number of software packages and libraries. By using OWL to describe our ontology we can also leverage the growing number of existing RDF and OWL vocabularies to describe aspects of our data model not central to the rubric domain. Some excellent examples of existing ontologies that complement our rubric ontology are:
  • 6. • Friend-Of-A-Friend (FOAF) - a vocabulary for describing people, such as creators and users of rubrics (Brickley & Miller, 2010) • Dublin Core Metadata Initiative (DCMI) - a popular system for describing metadata for digital assets (DCMI, 2010) • Creative Commons - for denoting the rights granted by the creator of digital assets to their users (Creative Commons, 2010) The ePortfolio Ontology (Wang, 2009) is another good example of a complementary ontology. It models the various entities in a student ePortfolio system, including a minimal Rubric entity. We can use this Rubric as an integration point to allow systems built on that model to leverage rubrics created according to our ontology. While a thorough introduction to OWL and RDF is beyond the scope of this paper, we will touch on some aspects of OWL and RDF in the next section when necessary to shed light on the rubric ontology implementation. Overview of The Rubric Ontology The key entities in the rubric ontology are: Rubric, Criterion, Level, and Category. Two more entities, Scope and Scoring, indicate the intended application of the rubric by users and software systems. Figure 1 shows a simple UML class diagram detailing the basic relationships between these key entities. Rubric The central entity of the rubric ontology is naturally the Rubric. A review of the literature around use of rubrics has identified at least three distinct subtypes of rubrics: • Analytic – analytic rubrics break down the assessment or evaluation of a work into discrete criteria. These criteria are generally tied back to the learning objectives of the unit or course. (Rocco) Each criterion is further broken down into levels of achievement, from minimal to exemplary. • Holistic – Holistic rubrics are used to assess “the whole of a process, performance, or product” (Rocco). This type of rubric is used when criteria overlap or are otherwise hard to isolate. • Primary Trait – Similar to the analytic rubric, the primary trait rubric “describes in detail what is required for performance” (Rocco). Rather than breaking down to discrete levels, a free-form evaluation or assessment is made for each criterion. A variant, the scoring guide rubric, is found in several sources, including (Stevens and Levi) Analytic rubrics are by far the most common subtype in online rubric repositories. They also appear to be the most regular form of rubric in structure. The only significant variation found was whether the number of levels per criterion was uniform (making the rubric a regular grid or table) or variable (some criteria possessing a finer or rougher gradation of scoring).
  • 7. In contrast, holistic rubrics varied widely in appearance. Some seemed to be nearly identical to analytic rubrics, differing only in their wording and intended use (Rocco), while others were very different. (Bargannier, Mertler). Property Description title Title of rubric description Free-text description of intended use. hasScope See Scope and Scoring below. hasScoring See Scope and Scoring below. hasCriteria Set of criteria/categories (mainly Analytic Rubrics). Table 1: Properties of Rubric class Criterion The main building block of the analytic and primary trait rubrics is the Criterion (plural Criteria). Each Criterion represents a focused “part of the task”. (Stevens and Levi). In the grid form of the analytic rubric, each row of the grid corresponds to a criterion. While we chose Criterion as the canonical name for this element, it is called a dimension in at least one source (Stevens and Levi). In our opinion this is merely a symbolic variation and not a semantic difference between conceptualizations. Property Description title Title of criterion description Free-text description of meaning of criterion. weight Weighting factor for scored rubrics hasLevels Set of levels (mainly Analytic Rubrics). Table 2: Properties of Criterion class Level The Level is the main component of the Rubric class in a holistic rubric, and of Criteria in analytic rubrics. The set of Levels should prescribe the range of assessment outcomes, from a low achievement (e.g. “poor” or “incomplete”) to high achievement (e.g. “exemplary”). The columns of a tabular analytic rubric generally correspond with the Level entities. Similarly, the rows of a holistic rubric will generally map to Levels. Property Description benchmark Text describing characteristics of this degree of achievement. Can have one or more per level. quality A qualitative description of this degree of achievement. Used for column headers in tabular
  • 8. rubrics score The points awarded for achieving this level. feedback Pre-defined feedback text to be relayed to the learner; may include guidance and suggestions for improvement or development. Table 3: Properties of Level class Category Categories are simple containers to aggregate multiple sequenced Criteria. We generally found categories in large or complex analytic rubrics. Categories can also play a useful role in software user interfaces based on rubrics. The Faculty Self-Assessment Tool from Penn State (Panulla, Rocco and McQuiggan, 2008) places the Criteria within a given Category in the same section of an accordion control (See Figure 2). Property Description title Title of category. hasCriteria Set of criteria/categories (mainly Analytic Rubrics). Table 4: Properties of Category class Scope The Scope of a rubric is meant to indicate what the rubric’s creator intended to assess or evaluate. The current ontology defines four distinct scopes: • Individual – used by a teacher or educator to assess or evaluate the work of a single individual learner. • Team – used by a teacher or educator to assess or evaluate the work of a group or team. Team rubrics are commonly found in Problem-Based Learning (PBL) environments. • Peer – used by one individual learner to evaluate or assess the work of another individual learner. Peer rubrics are also common in PBL environments; teammate evaluations may play a role in an overall course participation grade. • Self – used by an individual to assess his or her own learning or development. Specifying the intended Scope of a rubric can provide two powerful advantages: • Search systems may filter results from a database of rubrics, eliminating those created for a different purpose than the one desired. • Software systems may use the Scope to identify how it should present the rubric to the user in different contexts. For example, a secure online Peer-Review system with knowledge of course group assignments may allow students to securely and privately
  • 9. evaluate the other members of their team via a rubric, while preventing other students from seeing or affecting those evaluations. Scoring The Scoring attribute of a rubric allows creators to create both scored and unscored rubrics. While scored rubrics are likely to be the norm, educators sometimes find it desirable to assess a student’s learning progress without having it directly impact their grade. It is expected that software systems displaying unscored rubrics should still provide qualitative feedback to the learner. Artificial Entities As with any computerized representation of an information model, there are several entities introduced into the rubric ontology by the idiosyncrasies of underlying technology. It can be difficult to maintain closed, explicitly ordered lists of items in the present version of OWL. While the RDF Schema specification (W3C “RDF Vocabulary Description Language”, 2004) does define several Collection classes for aggregation, these classes do not provide semantics for restricting their contents to given classes of entities. To get around this problem and strictly model lists of Criteria and Levels we have defined two additional entities: CriteriaList and LevelList, based off the RDF List Collection. Class Description CriteriaList Contains Criterion and Category instances in explicit order. Category instances contain an additional CriteriaList, resulting in a tree structure of Categories and Criteria. LevelList Contains Level instances in explicit order. Table 5: List classes
  • 10. Future Goals Holistic Rubrics The model of holistic rubrics is somewhat incomplete. While a holistic rubric may be modeled superficially using the existing entities in the ontology, a more thorough explanation of the use and intent of holistic rubrics may ultimately lead to a different data model. Improved List Modeling The use of RDF Lists in OWL ontologies results in the ontology being validated as OWL- Full (W3C, 2009). Reasoning over OWL Full ontologies cannot be guaranteed to be finite or even efficient. Consequently, it is a best practice to avoided models that push into the OWL- Full realm. It may be possible to replace the RDF Lists-derived classes in the rubric ontology with an alternate model that allows validation as OWL DL without a significant loss of compatibility or semantic expressiveness. Implementations At present, three systems are under development at Pennsylvania State University that can create and/or consume rubrics described according to this ontology: • The Faculty Self-Assessment tool (Panulla, Rocco and McQuiggan, 2008) was the prototype application in which many of the ideas captured in the current version of the ontology were developed. • The Assignment Studio Rubric module for Drupal (Bailey and Ollendyke, 2009) is used at Penn State University’s College of Arts and Architecture to manage course activities in several resident and hybrid courses. An updated version of this module for Drupal 7 that incorporates this rubric ontology is currently in the design phase. • A Rubric Builder Rich Internet Application (RIA) is currently under development for Spring 2011. The Builder provides a rich, easy to use interface that captures many of the best practices of rubric design and guides the user to produce better rubrics.
  • 11. Figures Fig. 1 – Simplified Rubric UML model
  • 12. Figure 2: Categories as UI navigational elements
  • 13. References Andrade, H. G. (2000). Using rubrics to promote thinking and learning. Educational Leadership 57 (5): 13–18. ALTEC. (2010). RubiStar. Retrieved September 10, 2010, from http://rubistar.4teachers.org/ ANGEL Learning. (2010). Grading Rubrics. Retrieved September 10, 2010, from http://www.angellearning.com/products/lms/rubrics.html Bailey, K and B. Ollendyke. Rubric module for Drupal. Retrieved September 10, 2010 from http://drupal.org/project/rubric Brickley, D. and Miller, L. (2010). FOAF Vocabulary Specification 0.98. Retrieved from http://xmlns.com/foaf/spec/ Creative Commons. (2010). Describing Copyright in RDF. Retrieved September 10, 2010, from http://creativecommons.org/ns DCMI. (2010) Dublin Core Metadata Initiative. Retrieved from http://dublincore.org/schemas/rdfs/ Discovery Software. (2010). Rubrix. Retrieved September 10, 2010, from http://rubrix.com/ Goodrich, H. (1997). Understanding rubrics. Educational Leadership 54 (4): 14–17. Gruber, T. (1993). A translation approach to portable ontology specifications. Knowledge Acquisition, 5(2), 199-220. Marzano, R., D. Pickering, and J. McTighe (1993). Assessing Student Outcomes: Performance Assessment Using the Dimensions of Learning Model. Alexandria, VA: ASCD. Panulla, B. J., S. A. Rocco, and C. McQuiggan (2008). Faculty Self-Assessment: Preparing for Online Teaching. Retrieved September 10, 2010 from https://weblearning.psu.edu/FacultySelfAssessment/ Rocco, S. A. (2007). Assessment and evaluation in the online environment. New Directions in Adult and Continuing Education, 119, 75-86. Stevens, D. D., and Levi, A. J. (2005). Introduction to Rubrics. Sterling, Virginia: Stylus. Tierney, R., and Simon, M. (2004). What's still wrong with rubrics: focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2). Warlick, D. & The Landmark Project. (2010). The Rubric Machine. Retrieved September 10, 2010, from http://landmark-project.com/rubric_builder/
  • 14. Wang, S. (2009) E-Portfolios for Integrated Reflection. Issues in Informing Science and Information Technology. Volume 6, 2009. Retrieved from http://iisit.org/Vol6/IISITv6p449-460Wang630.pdf World Wide Web Consortium. (2004). Resource Description Framework (RDF): Concepts and Abstract Syntax. Retrieved from http://www.w3.org/TR/rdf-concepts/ World Wide Web Consortium. (2004). RDF Vocabulary Description Language 1.0: RDF Schema. Retrieved from http://www.w3.org/TR/rdf-schema/ World Wide Web Consortium. (2009). OWL 2 Web Ontology Language Document Overview. Retrieved from http://www.w3.org/TR/2009/REC-owl2-overview-20091027/