11. Language A Literature External Assessment: Performance assessment Diploma Programme, Language A Literature, curriculum review report – update on progress November 2009 Standard Level –assessment component and weighting Higher Level –assessment component and weighting Paper 1 : Literary analysis (1 ½ hours) Literary analysis of one unseen text in response to guided questions 20% Paper 1: Literary analysis: unseen commentary (2 hours) Literary analysis of one unseen text 20% Paper 2 : Essay (1 ½ hours) An essay based on at least two works studied in part 3 25% Paper 2: Essay (2 hours) An essay based on at least two works studied in part 3 25% Written assignment A reflective statement and literary essay on one part 1 work Reflective statement 300-400 words Essay 1200-1500 words 25% Written assignment A reflective statement and literary essay on one part 1 work Reflective statement 300-400 words Essay 1200-1500 words 25%
There are many theories on how to develop a good examination paper. The type of examination paper you create depends on the specific needs of the examining body, school, organization or nation - their philosophy, their classification/statistical needs, if any, and the needs of the students and the institutions in which they would like to continue their education, or the jobs into which they would like to enter. It also depends on the subject and what is suitable to test the skills and knowledge for that subject. There are many aspects of the examination to take into consideration, each one playing off the other, working with the other to create the perfect examination paper for a certain situation. I was asked what the essential ingredients for an excellent Literature paper in an international context would be. We may, therefore be able to make a few assumptions on the needs and philosophy of the organization giving the exam and the students taken them, given the examination is to take place in an international context and would therefore have certain requirements, for example - the questions will need to be applicable to any culture it is taken in and the skills and concepts learned must be transferable as well as innately promote intercultural awareness and contemplation from the content of the examination. In this presentation I would like to look at different decisions that would need to be made when designing an examination, important concepts which would apply to any examination and give an example as to how these concepts might come together to create a strong literature examination in an intercultural context.
There are quite a few aspects to consider when putting together an examination paper. One of those is the type of skills you wish to assess in the examination. This can be determined by the subject and also by what the results of the examination will be used for. Presently there are two main types of assessment used today. They are psychometric assessment and performance assessment. In both types of assessment questions and tasks are carefully chosen and put together in order to create an examination which will fulfils the needs of the examining body. It is important that they try to get the right balance between the construction of the exam and the reliability of the marking. However they go about this in two different ways, and each method produces different results, both positive and negative depending on what the examination results will be used for.
Psychometric testing is a form of standardized test, usually multiple choice, used to test, what was often seen as innate qualities in a person - such as intelligence, aptitude and capability. It was later used also to test academic achievement. It is valued as an assessment tool due to its ability to provide accurate measurements and precise results for student ranking. Questions can be carefully chosen to isolate a certain skill or type of knowledge which could be used to differentiate between students. This type of exam offers a high level of accuracy of measurement and as exams are automatically marked a high level of equity. Needless to say this method of assessment is also cost and time effective as there is no need for an external examiner.
Performance assessment differs quite dramatically from psychometric assessment. Instead of demonstrating knowledge by choosing from among a list of possible answers the students are required to actually use the skills and knowledge they have learned in the classroom to demonstrate their abilities. The types of assessment used are varied and can include many tasks such as "problem solving, essays, project work and examinations” and push the students to use and demonstrate a wide range of abilities and skills, not just regurgitating knowledge.
In the DP Assessment principles and practice document, a document for an international education organization, it states, “It is often stated that we now live in a knowledge society. This should not be taken to mean that the acquisition and retention of factual information is of prime importance. The explosion of information in recent times makes it impossible for individuals to achieve mastery of knowledge in many areas. The more valued academic skills of today are in accessing, ordering sifting, synthesizing and evaluating information, and creatively constructing knowledge (16).” Performance assessment is often preferred as an assessment option because it can adequately account for this change in society and the skills needed to succeed in this developing society.
Performance assessment incorporates and allows to assess skills such as Bloom’s higher-order cognitive skills in order to adapt to this different way of gathering and using information and knowledge. This type of assessment does have its drawbacks, however, as it requires the professional judgment of an examiner who will mark the exams and determine the level of achievement. This can be time consuming and costly, whereas psychometric testing is relatively cost and time effective with a high reliability of results. There is also the consequence that with performance assessment, whereas the construct validty of the examination may be maintained, the scores will be less reliable as different marker's results/opinions may vary. However, levels of reliability can be raised with the use of a detailed markscheme and high levels of quality control.
Another important aspect of examinations, is their effect on teaching and learning. The effect on teaching of examinations is often called Washback or Backwash. This can be positive or negative. Examinations can be powerful entities, especially high-stakes examinations where the futures of the students and the teachers will be effected. Teachers’ positions in some countries may rely on the results of their students, and the students' futures, which direction they follow in school and after may also depend on the outcomes of these exams. Therefore the nature of the exam will effect how the teacher teaches the necessary information in the classroom. For example, if the exam is a multiple choice examination, testing a narrowly defined construct or concept, then the teaching might reflect this, leaving the students with limited knowledge and skills. However, if the examination is based on performance assessment, proliferate with varying assessment types and skill use, then the teacher would be rewarded for teaching a proliferation of skills and knowledge, and practice in using these skills, which would result in a more well-rounded and skilled student.
Another important aspect of preparing an examination paper is the design. The question of whether an exam is well constructed is often a question of balance. I have already alluded in some cases to the concepts of validity and reliability. It is important to carefully construct the examination to have a good balance of these two concepts. There are several types of validity and reliability, however, simply put, a high level of validity occurs when the construct and questions assess what the exam has been developed to assess, and reliability occurs when the measurement of the assessment results are accurate and can be reproduced in another similar environment. These concepts are an important aspect to any examination, especially a high-stakes examination. The DP Assessment principles and practices document states, “Validity and reliability are widely regarded as essential characteristics of any assessment system, particularly a high-stakes one where the outcome is of great importance to the student or the teacher (8).” It must be mentioned here that performance assessment requires the use of professional judgement and therefore marker reliability may be reduced if measure are not put into place to preserve this. For example, by arefully choosing markers, having a detailed markscheme, and a strong moderation system. These definitions broaden when speaking about psychometric testing and performance assessment. When discussing psychometrics, validity refers to identifying the single construct being tested in the examination and reliability “how consistently the different items in the test behave, in terms of the correlation of student responses given to these different items (8).” Validity and reliability are defined differently when discussing performance assessment. Important to validity in this case is also what the results of the examination will be used for and how they will be perceived as well as the washback effect on teaching and learning. With performance assessment, the scope of validity is greatly widened to include the wide range of skills and knowledge assessed, the way it is assessed and even the results of the assessment on teaching.
We have looked at several different major aspects involved in putting together an examination: the type of examination, whether it be psychometric or performance, the effect of negative or positive washback and the internal construction of the exam in terms of validity and reliability. After looking at the various aspects of putting together an examination I think we can make an assumption as to the ingredients that would make an excellent literature examination paper in an international context. It seems that a clear idea of the goal of the examination is needed, if this goal is to develop students with higher-order transferable skills, then an examination within the parameters of performance assessment is most desirable. After the skills that need to be assessed have been decided, then a well constructed exam, with questions that target and adequately assess the students ability to use the skills they have learned in class is required. It should therefore be able to demonstrate a positive effect on teaching and learning and not a negative effect. And finally, it should be possible to mark the results with a high level of reliability and ensure the trust of the institutions and organizations which will be using the results, and the students who will be relying on the results to shape their future.
The IBO is an example of such an organization that would give a literature examination in an international context. It is an international organization catering to schools around the world. They aim to promote international and intercultural understanding among the students taking the IB examinations. The DP examinations are high-stakes examinations, recognized by universities internationally, and therefore will have a high degree of washback. It is interesting to see how they have chosen certain assessment concepts in order to create a strong literature examination.
This is the outline of the IB’s Language A Literature examination. We can see right away that this is an example of performance assessment. Students are asked to use a range of skills and knowledge in order to analyze texts, some unseen, others studied in class. If we look at Paper 1 we can see that the question necessitates that students use skills acquired in class on unseen texts. They are not being asked to reiterate what they learned about a text they have already read, but to apply skills of literary analysis to interpret the texts and to discuss aspects such as content, technique, style, structure, theme and language. This is a good example of promoting the use of higher-order skills and performance assessment to assess the students. Paper 2 looks at yet another aspect of the skills the students should have learned during the literature course. In this case they are asked to look at texts all from the same literary genre but to focus on recognisable techniques or literary conventions applicable to that genre, therefore assessing another area of knowledge learned in the classroom.
The written assignment again tests the students in yet a different area. They are asked to write a reflective statement and an essay of up to 1500 words. What is interesting about the written statement is that preparation for this begins explicitly in the classroom and is carried out in stages. Teachers are asked to lead the students in discussion and address certain cultural and contextual aspects of the texts studied. The texts studied for this part of the examination are also works in translation. They will be texts from countries with different languages and cultures, thereby giving another dimension to, and support, intercultural understanding and contemplation.
Stage 2 consists of asking students to look back at the oral discussion and write a short reflective statement. This will be turned in with the completed statement.
Stage 3 requires students to write, using prompts from the teacher, for about 50 minutes on each of the texts studied. They will choose one of these texts to develop into an essay which will be marked externally. Stage 4 consists of writing the essay. What is really interesting about this assignment is that it sets out the skills the students are to be able to use in the examination, and therefore reflects back positively on the teaching and learning. The teacher is required to teach a wide range of skills so that the students will be knowledgeable and comfortable with using these skills for the examination. This is true for the entire examination. This literature exam has positive washback on teaching and learning. As we can see there is also a high level of construct validity due to the carefully chosen questions, and the fact that they were careful to design the exam to test the high level skills they want to assess. The IB also brings a high level of reliability to the assessment with detailed mark schemes, careful selection of examiners and a detailed moderation process.
The development of an examination paper is a mutli-faceted production. There are many aspects and much theoretical knowledge which goes into this, and this highlights the need to keep up to date of current theories. However, it is not just knowledge of the varying theories that is required, but a clear idea of the goal of the examination. We have seen how assessment can influence the teaching and learning in schools, and therefore shape the learning experience of students and the way they learn. We have also looked at the importance of concepts such as reliability and validity and how the importance of these changes with respect to the type of examination desired, whether it be psychometric assessment or performance assessment. In an international context we decided which of these aspects were the most important, and looked at the International Baccalaureate Organization literature examination as an example. We explored how they used the various aspects of putting together an examination paper in an international context, and created an excellent literature examination paper.
![Ingredients? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)