There are a number of important elements to criterion referenced standards based assessment, and while they are represented here as a series of steps, in practice you often complete these concurrently rather than sequentially. Whatever order you complete them in, you need to consider:
Criteria and standards are not something you develop once and use forever; you'll want to tweak them over time, based on student feedback, questions from markers and from your own experience as you see how others interpret them. For example, if you find most of your feedback to students relates to the need to use the appropriate style for the assessment type, however you haven't explicitly included this as a criterion, you would update that for the next session you use the assessment.
A good place to start when developing criteria and standards, is to review the subject learning outcomes in relation to the assessment task.
Throughout a process of consultation, and often accreditation, a subject's learning outcomes have been identified as the key knowledge and skills that our students need to master. These often build progressively over the course, so we need to ensure:
Review your subject's learning outcomes and assessment tasks and ask yourself:
Once you are happy that the assessment tasks reflect the subject's learning outcomes, you can start developing your marking criteria.
The first step in developing marking criteria is to identify the skills, knowledge and application to professional practice that the learning outcomes articulate. An easy way to do this is to create a table with a column for the outcome and one each for the skills, knowledge and application. This exammple below demonstrates how the breakdown of the outcome into skills, knowledge and application might look.
|Subject Learning Outcome||Skills||Knowledge/Topics||Application to practice|
|be able to design and review assessment tasks that align with subject and course learning outcomes, enhance student learning and promote academic integrity||analyse, design, research, develop arguments||know about: taxonomies, design process, valid research in learning design,||application to own professional practice|
|be able to justify their practices with reference to scholarly literature and CSU policies.||identify valid literature, conduct searches, cite in-text and in bibliography,||taxonomies, design processes, valid research in learning designings||relate the literature to their own practice|
How to change subject learning outcomes
What can you do if the subject learning outcomes DO NOT articulate the knowledge, skills and application to be learned in the subject? If you wish to change the subject learning outcomes, the following steps will guide you through the process.
So how do we check students' achievement of the subject learning outcomes? By developing marking criteria which relate to the specific task at hand and the knowledge, skills and application you expect the student to demonstrate through the assessment task – which should link back to the subject learning outcomes.
While a task description states what students will hand in, criteria are about what you will prioritise when you assess what they submit. Telling students what you value ahead of time builds a powerful partnership and shared responsibility.
Criteria are the elements you will use to evaluate a student's work. They should not simply restate the assessment tasks or elements, but articulate the learning and what you are giving value to – that is, you need to make your expectations explicit.
This PowerPoint on writing criteria and standards may help.
Sources to assist you in developing criteria could include:
Once you've written your criteria, you then need to develop the performance standards.
Performance standards describe the expectations you have of students' work for different grade levels. Standards help you differentiate criteria in ways that are qualitatively and quantitatively distinct and they help build student ownership because students understand what they have to do.
Write standards descriptors for each criterion, for each passing grade
Taxonomies are useful in articulating types or levels of performance, however they are not a generic substitute for the process, and much thought needs to go into differentiating meaningful standards. Some Taxonomy resources that may help include:
Another useful resource from the Association of American Colleges and Universities can also help to guide development of language in rubrics. See Value Rubric.
Review your standards descriptors and ask:
Think carefully about exactly what you want to see from your students and communicate this via the criteria and standards.
Another consideration is how you will assign marks using the criteria and standards you develop.
You now need to think carefully about how to convert achievement against criteria and standards into a 'mark' or grade, in a way that is valid and reliable. To be valid, the outcome needs to accurately reflect the quality of the work; the holistic professional judgement we often talk about comes into play here. To be reliable, the process you use to assign grades or marks should ensure the same data will always produce the same result. This is especially important if there are multiple markers involved.
While using standards and criteria provides a more consistent approach to marking student work, professional judgement still plays a key role in assigning marks or grades. Often, a piece of student work contains elements across multiple levels of performance and we need to use our professional judgement to decide how that should be combined to produce a final mark or grade for the student.
Broadly, there are two approaches to ascribing grades under CRSBA.
The first approach assigns a weighting to each criteria and awards marks for each standard of performance. You can add these together or average them to get a final mark for the assessment.
|Criterion 1 (40%)||Description (35-40 marks)||Description (30-34marks)||Description (25-29 marks)||Description (20-24marks)||Description (0-19 marks)|
|Criterion 2 (40%)||Description (35-40 marks)||Description (30-34marks)||Description (25-29 marks)||Description (20-24marks)||Description (0-19marks)|
|Criterion 3 (20%)||Description (17.5-20 marks)||Description (15-17 marks)||Description (12.5-14.5 marks)||Description (10-12marks)||Description (0-9marks)|
You need to ensure the weightings you apply don't place too much or too little emphasis on each criterion. For example if you weight quality of writing fairly low, would a barely literate response still be eligible to achieve a pass grade? Conversely, if you weight it too high would those students who write well, but who don't necessarily understand the content, achieve a higher grade than they deserve?
Some criticise rubrics for their atomistic approach to assessment, however this doesn't need to be applied rigidly to your marking. While your criteria and standards make it clear to students what you are expecting them to do to achieve each grade, you need to decide – on balance and against those criteria – what level of performance the student's work is at. This is where you use your professional judgement to grade holistically.
|Criterion 1 ||Description ||Description ||Description ||Description ||Description |
|Criterion 2||Description ||Description ||Description ||Description ||Description |
|Criterion 3||Description ||Description ||Description ||Description ||Description |
In the above example, if you value each criterion equally, you can 'average' or combine the grades to assign a Credit as the final grade, and award a mark within the Credit band (65-74%). The final mark or grade needs to make sense given the grades highlighted in the rubric; for example, assigning a Pass grade for this assignment doesn't make sense based on the information provided. Sometimes, you may find a student response covers most but not all of the elements described in your criteria, for example they may have missed one of the areas you asked them to look at or have completed it to a different standard. A simple way to demonstrate this is to highlight the words that are relevant, for example:
Accurately identify and describe the structural, language and visual features of a range of types of text.
A range of complex structural, language and visual features are consistently and accurately identified across all text types. The features are clearly described with consistent control of complex meta-language.
|A range of complex structural language and visual features are consistently and accurately identified across different text types. The features are described using complex meta-language that is sustained across most text types.||A range of simple and more complex structural, language and visual features are accurately identified for most types of text. |
Few errors in identification are evident. The features are described using more complex meta-language.
|Simple structural, language or visual features relevant to the text type areidentified. Some errors in identification are evident for some text types. |
The features are described in basic terms with the use of appropriate meta-language.
|Analysis of text content reflects critical literacy practices.||Text content is analysed using a range of sophisticated critical literacy practices. These have been skillfully synthesised in the discussion.||Text content is analysed using more sophisticated critical literacy practices.The practices are coherently synthesised in the discussion.||Text content is analysed using a number of simple critical literacy practices. |
These are used discretely in the discussion.
|Text content is described rather than analysed. There is limited use of criticalliteracy practices resulting in a more superficial discussion of the|
This is a useful way to provide specific feedback to students about what they have done well and areas for improvement. As in the previous example, you would use your professional judgement to decide a holistic final mark/grade, which makes sense taking into account the student's performance across all the marking criteria.
Now you're ready to publish your assessment tasks and marking criteria and standards. As you use them throughout a session, keep notes on what works and any issues you'd like to improve upon for next time.
As you mark against your criteria and standards, keep notes for yourself so you can remember what needs to be tweaked for next time. We're all so busy that it's easy to forget the problems; keep note of:
Make sure you also capture feedback from other markers working on your subject as their insights will also help you improve your criteria and standards. Remember, there is no such thing as the perfect criteria or standards; you will continue to tweak these over time as your confidence and understanding of this new skill grows. Yes, this tweaking goes on all the time – it is part of a continuous improvement cycle even when the skill is not new. Just keep in mind, one of the key benefits of criterion referenced standards based assessment is the self-assessment and improvement that students can undertake when presented with well-written criteria and standards. Try to think of the criteria and standards as a communication tool: they communicate to students (and markers) what you are looking for when you mark their work, and once it has been marked it gives students a clear indication of where they can improve.
Even if you won't be teaching this subject again until next year, reflecting on what worked and what didn't will help you develop new criteria and standards for the subjects you have next session. Your reflections will also be valuable to whoever inherits the subject next time it is offered. If you need assistance, involve others working in the subject, discipline or course, or call on an Educational Designer.
Watch this video to see how Jonathan Howard developed and improved the assessment task and rubric for BIO327. The slides Jonathan refers to are available here.
You may like to use this blank template to start writing your own criteria and standards, and refer to the PowerPoint to help you get started. If you would like to review your final marking rubric this Self-evaluation form provides a number of questions that will help.
With the approach advocated here, we are trying to avoid unhelpful, general descriptions that provide little guidance to students (or markers), and ensure there are informative descriptions to distinguish among performance at each level.
For example, consider these descriptors for one criterion within a marking rubric:
|Understanding of the topic||Evidence of an exceptionally high understanding||Evidence of a thorough understanding||Evidence of a good understanding||Evidence of basic understanding||Poor understanding|
Can you see the problems with this rubric?
We can address these issues by providing clear and specific text in the criterion that links back to the learning outcome using action, content and context to format the descriptor. We can also extend the description in the standards to clearly indicate what is required for each grade. This removes the potential for confusion and misinterpretation by both students and markers.
Using one of the learning outcomes for a subject we have reworked the above criterion and standards to more clearly articulate what is being assessed and the standard required to achieve each grade level.
Learning outcome that links to this criterion - Be able to understand the different frameworks through which sociologists and human geographers interpret issues of inequality in society.
|Tutorial postings demonstrate knowledge and understanding of major theories of inequality and the forms of inequality in society. (10 marks)||The (3) tutorial postings show the students familiarity with tutorial topics and the student can evaluate the relevant theories associated with inequality in society when discussing the topic question.||The (3) tutorial postings show the students familiarity with tutorial topics and the student can compare the relevant theories associated with inequality in society when discussing the topic question.||The (3) tutorial postings show the students familiarity with tutorial topics and the student can interpret the relevant theories associated with inequality in society when discussing the topic question.||The (3) tutorial postings show the students familiarity with tutorial topics and the student can describe the relevant theories associated with inequality in society when discussing the topic question.||No Tutorial Forum postings are made and /or do not show student engagement with online tutorial discussion.|
Information and workshops to help guide you are available here. You can also find some examples to guide you on the Example Rubrics page. If you would like to review your final marking rubric the following Self-evaluation form provides a number of questions that will help. For more information contact the QLT Assessment Leaders.
The following checklist is one tool that can help you determine whether your assessment task meets quality. Click on the image to access a template for your own use. Click here for an alternate assessment quality checklist.
A tip on checking whether all subject learning outcomes have been covered by assessment in the subject is to:
Click here for an example of the process
Ensure the rationale for each assessment task is clearly linked to the learning outcomes being assessed. Tips for checking include:
The criteria are aligned to the learning outcomes being assessed.
This example shows the colour coded alignment of the component parts of the learning outcome and the criterion. Green = action (verb); Red = content; Blue = context or environment
Critical analysis of the applicationof the identified political behaviour including the strategies and tactics employed.
be able to critically analyse the application of political strategies in organisations
The pass level standard is adequate to meet the standard expressed in the learning outcomes. The level of the learning outcome must be met by the pass level standard. For example, if the outcome requires analysis, the pass level standard must also require analysis. Here's an example:
Learning outcomes for the task
be able to critically analyse a database design and apply normalisation theory and techniques
Analysis identifies key PKs and functional dependencies and draws out relevant 3NF relations. Creates an ERD that represents identified 3NF relations and includes the related entities, PKs, attributes, relationships, cardinalities and optionalities.
The Online Moderation System also asks questions related to assessment quality. The following questions need to be answered by subject convenors and moderators so understanding what we look for in Quality Assessment means that these questions are easier to answer.
3. Are all Learning Outcomes assessed by the assessment tasks?
4. Is it clear to students what they need to submit for their assessments (eg format, style, due dates, elements to include)
5. Is it clear to students what their performance will be assessed against?
6. Please comment on the assessment tasks chosen for this subject, are they:
7. Are there any issues in the Subject Outline that should be fixed before the subject is offered again in another session?
8. Are there any issues in the Subject Outline that need to be fixed now before the Subject Outline is published for this session?
You may like to refer to the QA checklist to ensure everything has been covered, however you are not required to complete or submit the checklist.
9. Is the tone of the communication with students in the Subject Outline appropriate?
That is, not too formal, no excessive use of jargon and written in plain English.
More help can be found on the Online Moderation System help site.
Assessments that are in the form of exams are also required to include marking criteria and standards in the subject outline. So the same approach outlined in Criteria to assess tasks and Performance standards for each criterion sections above can be applied to exams.
You might like to consider if an exam is actually the best (and most authentic) assessment tool to use to evaluate a student's performance against the subject learning outcomes. For example, rather than an exam asking students to label the parts of the body, it may be more useful (and authentic) to apply this knowledge in a case study that asks the student to identify the interconnected tissues in the shoulder and explain the impact an injury on tissue 'x' would have on the others.
If you decide that an exam is the most useful and authentic assessment tool to use, you then need to think carefully about the learning outcomes for the subject and how you will assess these within the exam. The following slides prepared by Dr Alison Matthews give you a guide on how to prepare effective exam questions - including multiple choice (MCQ), short answer, and essay questions - and show you how to incorporate higher order thinking into MCQ.