Developing quality assessment for your subject

Throughout a process of consultation, and often accreditation, a subject's learning outcomes have been identified as the key knowledge and skills that our students need to master. These often build progressively over the course, so we need to ensure:

1. Our teaching and learning activities are geared towards helping students learn these knowledge and skills; and
2. We need to know they have mastered the knowledge and skills before they move onto the next step, whether that's a new session or a professional workplace.
   

What you need to do:

Review your subject's learning outcomes and assessment tasks and ask yourself:

1. Do the learning outcomes articulate the knowledge, skills and application to be learned in the subject?
2. Do the assessment tasks give students an opportunity to show they have met the subject's learning outcomes? That is, does the task make sense in light of the subject learning outcomes?
3. Is each outcome covered in at least one assessment task?
4. Is the assessment task clearly described (is it written in plain English)? Does it include clear guidance as to the required content and format? Does the name of the task match the description (e.g. an essay task that then asks for an executive summary and other subheadings should be called a report)?
5. Can I make the assessment tasks more authentic? That is, how would a graduate apply this knowledge in the professional workplace and do the tasks I have chosen simulate such knowledge application?

Once you are happy that the assessment tasks reflect the subject's learning outcomes, you can start developing your marking criteria.

The first step in developing marking criteria is to identify the skills, knowledge and application to professional practice that the learning outcomes articulate. An easy way to do this is to create a table with a column for the outcome and one each for the skills, knowledge and application. This exammple below demonstrates how the breakdown of the outcome into skills, knowledge and application might look.

So how do we check students' achievement of the subject learning outcomes? By developing marking criteria which relate to the specific task at hand and the knowledge, skills and application you expect the student to demonstrate through the assessment task – which should link back to the subject learning outcomes.

While a task description states what students will hand in, criteria are about what you will prioritise when you assess what they submit. Telling students what you value ahead of time builds a powerful partnership and shared responsibility.

Criteria are the elements you will use to evaluate a student's work. They should not simply restate the assessment tasks or elements, but articulate the learning and what you are giving value to – that is, you need to make your expectations explicit.

What you need to do:

1. Identify the 4-5 main elements you are looking for in students' responses, these should match the skills, knowledge and application you identified in your learning outcomes in the previous section. For example:
   1. Application of theory to a relevant example
   2. Analysis
   3. Knowledge of topic
   4. Professional presentation, etc.
2. Flesh these out more to explain what it is you are looking for, using the action, content and context approach. See examples below.
   1. Describes and contextualises professional communication within health and aged care settings.
   2. Evaluates professional practice within the contexts of the literature and research.
   3. The Current Position describes the management approach and status of the species.
   4. Report is presented to a professional standard including a title page, table of contents and headings.
   5. Evaluates traditional and contemporary strategic planning methods for application in a given context.
   6. Justifies the application of the chosen method to solve the strategic planning challenge.
3. Now review your criteria using the following questions to guide you. Ask yourself:
   1. How well do the criteria describe what is important for students to demonstrate?
   2. Have I used language from the learning outcomes in my criteria?
   3. How well do my existing criteria link to the subject learning outcomes?
   4. Am I assessing anything that is outside the scope of the learning outcomes, and therefore potentially irrelevant to a student's achievement in this subject?
   5. Are there any value descriptions, such as 'adequate' or 'satisfactory' – these are subjective terms and do not clearly describe what is valued in the criteria.

This PowerPoint on writing criteria and standards may help.

Sources to assist you in developing criteria could include:

1. Accreditation requirements
2. The level of the subject (AQF)
3. The research literature
4. Peak bodies (Professional Associations)
5. Your own professional experience
6. Your knowledge of the topic & students
7. Example rubrics for similar disciplines or assessment type.

Once you've written your criteria, you then need to develop the performance standards.

Performance standards describe the expectations you have of students' work for different grade levels. Standards help you differentiate criteria in ways that are qualitatively and quantitatively distinct and they help build student ownership because students understand what they have to do.

What you need to do:

Write standards descriptors for each criterion, for each passing grade

1. It's best to start with one level, for example a such as a Pass level; what is the minimum standard you expect students to demonstrate their competence in this assessment? Write a Pass descriptor for each criterion that describes evidence of achievement of the subject Learning Outcome associated with that assessment.

You now need to think carefully about how to convert achievement against criteria and standards into a 'mark' or grade, in a way that is valid and reliable. To be valid, the outcome needs to accurately reflect the quality of the work; the holistic professional judgement we often talk about comes into play here. To be reliable, the process you use to assign grades or marks should ensure the same data will always produce the same result. This is especially important if there are multiple markers involved.

While using standards and criteria provides a more consistent approach to marking student work, professional judgement still plays a key role in assigning marks or grades. Often, a piece of student work contains elements across multiple levels of performance and we need to use our professional judgement to decide how that should be combined to produce a final mark or grade for the student.

Broadly, there are two approaches to ascribing grades under CRSBA.

Using Marks and weightings

The first approach assigns a weighting to each criteria and awards marks for each standard of performance. You can add these together or average them to get a final mark for the assessment.

For example:

You need to ensure the weightings you apply don't place too much or too little emphasis on each criterion. For example if you weight quality of writing fairly low, would a barely literate response still be eligible to achieve a pass grade? Conversely, if you weight it too high would those students who write well, but who don't necessarily understand the content, achieve a higher grade than they deserve?

Holistic Marking

Some criticise rubrics for their atomistic approach to assessment, however this doesn't need to be applied rigidly to your marking. While your criteria and standards make it clear to students what you are expecting them to do to achieve each grade, you need to decide – on balance and against those criteria – what level of performance the student's work is at. This is where you use your professional judgement to grade holistically.

In the above example, if you value each criterion equally, you can 'average' or combine the grades to assign a Credit as the final grade, and award a mark within the Credit band (65-74%). The final mark or grade needs to make sense given the grades highlighted in the rubric; for example, assigning a Pass grade for this assignment doesn't make sense based on the information provided. Sometimes, you may find a student response covers most but not all of the elements described in your criteria, for example they may have missed one of the areas you asked them to look at or have completed it to a different standard. A simple way to demonstrate this is to highlight the words that are relevant, for example:

This is a useful way to provide specific feedback to students about what they have done well and areas for improvement. As in the previous example, you would use your professional judgement to decide a holistic final mark/grade, which makes sense taking into account the student's performance across all the marking criteria.

What you need to do

1. Carefully consider the best way to ascribe grades – holistic marking or using marks and weightings. It often comes down to personal choice, teaching experience and discipline as to which is the best approach.
2. Show your assessment task description, criteria, standards and grading approach to a colleague or others in your subject team and ask for their feedback. Can they understand what you are asking for, or perhaps you need to clarify some areas.

Now you're ready to publish your assessment tasks and marking criteria and standards. As you use them throughout a session, keep notes on what works and any issues you'd like to improve upon for next time.

Improving your criteria & standards and reflecting on the lessons learned

As you mark against your criteria and standards, keep notes for yourself so you can remember what needs to be tweaked for next time. We're all so busy that it's easy to forget the problems; keep note of:

1. Any criterion that doesn't seem to apply to the work students have actually submitted – do you need to change the language so students understand it better, or perhaps the criterion isn't relevant to the task you've set and needs to go? Do you need to revise the task description to make it clearer?
2. Times when you can't decide which level of performance or standards descriptor is the most appropriate – do you need to add more guidance within your descriptors? Or perhaps there isn't enough differentiation between the levels (e.g. between pass and credit) and they need to be rewritten?
3. Any vague or subjective language (good, adequate, high quality etc.) – ask yourself what are the specific knowledge and skills you are looking for, can you make this more explicit in the standards descriptors? This will give students a clearer understanding of your expectations, as well as enable a more consistent application of the standards between markers.
4. Criteria or standards that are interpreted differently by different markers – do you need to tighten up the language so your intent is clear? Do you need to have more discussions with the subject team prior to marking? Are students also confused as to what you mean?
5. Feedback that you find yourself giving over and over again – perhaps this requirement wasn't made clear in the task description, marking criteria and standards, or needs more focus in your teaching?
6. Any student work you think represents a 'typical' response at each level (pass through to high distinction). Read over these when you start revising your criteria and standards and write a list of what makes them a pass or credit or high distinction response. These elements will help you see what it is that you value in the students' work, and once you can identify them it makes it much easier to share this knowledge with your students.

Make sure you also capture feedback from other markers working on your subject as their insights will also help you improve your criteria and standards. Remember, there is no such thing as the perfect criteria or standards; you will continue to tweak these over time as your confidence and understanding of this new skill grows. Yes, this tweaking goes on all the time – it is part of a continuous improvement cycle even when the skill is not new. Just keep in mind, one of the key benefits of criterion referenced standards based assessment is the self-assessment and improvement that students can undertake when presented with well-written criteria and standards. Try to think of the criteria and standards as a communication tool: they communicate to students (and markers) what you are looking for when you mark their work, and once it has been marked it gives students a clear indication of where they can improve.

Even if you won't be teaching this subject again until next year, reflecting on what worked and what didn't will help you develop new criteria and standards for the subjects you have next session. Your reflections will also be valuable to whoever inherits the subject next time it is offered. If you need assistance, involve others working in the subject, discipline or course, or call on an Educational Designer.

Watch this video to see how Jonathan Howard developed and improved the assessment task and rubric for BIO327. The slides Jonathan refers to are available here.

You may like to use this blank template to start writing your own criteria and standards, and refer to the PowerPoint to help you get started. If you would like to review your final marking rubric this Self-evaluation form provides a number of questions that will help.

With the approach advocated here, we are trying to avoid unhelpful, general descriptions that provide little guidance to students (or markers), and ensure there are informative descriptions to distinguish among performance at each level.

For example, consider these descriptors for one criterion within a marking rubric:

Can you see the problems with this rubric?

1. The language is vague, making the rubric mysterious and disempowering for the student, e.g. how does the student differentiate between 'good', 'thorough' or 'exceptionally high'?
2. The standards are too general, it is hard to know exactly what the marker is looking for, and in terms of feedback, the student cannot see what needs to be fixed.
3. The criterion is too generic and may or may not relate to the subject's learning outcomes.
4. The validity of standards is also questionable; should a student pass (ultimately a graduate) who has only demonstrated 'basic understanding'?

We can address these issues by providing clear and specific text in the criterion that links back to the learning outcome using action, content and context to format the descriptor. We can also extend the description in the standards to clearly indicate what is required for each grade. This removes the potential for confusion and misinterpretation by both students and markers.

Using one of the learning outcomes for a subject we have reworked the above criterion and standards to more clearly articulate what is being assessed and the standard required to achieve each grade level.

Learning outcome that links to this criterion - Be able to understand the different frameworks through which sociologists and human geographers interpret issues of inequality in society.

Information and workshops to help guide you are available here. You can also find some examples to guide you on the Example Rubrics page. If you would like to review your final marking rubric the following Self-evaluation form provides a number of questions that will help. For more information contact the QLT Assessment Leaders.

What do we look for in Quality Assessment?

1. All subject learning outcomes are covered by the assessment tasks in the subject.
2. Each assessment task is linked to at least one subject learning outcome.
3. The rationale for each assessment task is clearly linked to the learning outcomes being assessed.
4. The criteria are aligned to the learning outcomes being assessed.
5. The pass level standard is adequate to meet the standard expressed in the learning outcomes.

The following checklist is one tool that can help you determine whether your assessment task meets quality. Click on the image to access a template for your own use. Click here for an alternate assessment quality checklist.

A tip on checking whether all subject learning outcomes have been covered by assessment in the subject is to:

1. Copy learning outcomes into a Word document
2. For each task check the rationale for the subject outcomes that have been linked and highlight the learning outcomes in your Word document
3. Then check the task – can you see that this learning outcome is covered by the task? If so, cross it out in the Word document
4. Repeat for each task in the subject outline.

Click here for an example of the process

Ensure the rationale for each assessment task is clearly linked to the learning outcomes being assessed. Tips for checking include:

1. Does the rationale state the purpose of the task?
2. Does the rationale identify the skills and knowledge that students need to demonstrate?
3. Does the rationale clearly link to the learning outcomes being assessed?
4. [EXTRA: For a scaffolded task, does the rationale state how this task links to other assessment tasks - e.g. builds into next task, uses information from previous task]

The criteria are aligned to the learning outcomes being assessed.
This example shows the colour coded alignment of the component parts of the learning outcome and the criterion. Green = action (verb); Red = content; Blue = context or environment

The pass level standard is adequate to meet the standard expressed in the learning outcomes. The level of the learning outcome must be met by the pass level standard. For example, if the outcome requires analysis, the pass level standard must also require analysis. Here's an example:

Learning outcomes for the task

be able to critically analyse a database design and apply normalisation theory and techniques

Pass:

Analysis identifies key PKs and functional dependencies and draws out relevant 3NF relations. Creates an ERD that represents identified 3NF relations and includes the related entities, PKs, attributes, relationships, cardinalities and optionalities.

The Online Moderation System also asks questions related to assessment quality. The following questions need to be answered by subject convenors and moderators so understanding what we look for in Quality Assessment means that these questions are easier to answer.

3. Are all Learning Outcomes assessed by the assessment tasks?

4. Is it clear to students what they need to submit for their assessments (eg format, style, due dates, elements to include)

5. Is it clear to students what their performance will be assessed against?

1. Are marking criteria and standards, including descriptions of the levels of performance required to achieve each passing grade, provided for EVERY assessment task?
2. Are the marking criteria and performance standards explicit?
3. Are the marking criteria and performance standards easy to understand?
4. Does each assessment item clearly state the referencing requirements for the task (where relevant), and is this reflected in the marking criteria?
5. Are the marking criteria and performance standards measurable?
6. Are the marking criteria and performance standards able to be applied consistently by all markers and students?
7. Is the pass standard at a sufficient level to meet the standard required by the learning outcome?

6. Please comment on the assessment tasks chosen for this subject, are they:

1. Relevant to the subject?
2. Appropriate to the course level (AQF 7, 8, 9) and year of study?
3. Are the assessment tasks problematic requiring attention before delivery this session?

7. Are there any issues in the Subject Outline that should be fixed before the subject is offered again in another session?

8. Are there any issues in the Subject Outline that need to be fixed now before the Subject Outline is published for this session?
You may like to refer to the QA checklist to ensure everything has been covered, however you are not required to complete or submit the checklist.

9. Is the tone of the communication with students in the Subject Outline appropriate?
That is, not too formal, no excessive use of jargon and written in plain English.

More help can be found on the Online Moderation System help site.

Assessments that are in the form of exams are also required to include marking criteria and standards in the subject outline. So the same approach outlined in Criteria to assess tasks and Performance standards for each criterion sections above can be applied to exams.

You might like to consider if an exam is actually the best (and most authentic) assessment tool to use to evaluate a student's performance against the subject learning outcomes. For example, rather than an exam asking students to label the parts of the body, it may be more useful (and authentic) to apply this knowledge in a case study that asks the student to identify the interconnected tissues in the shoulder and explain the impact an injury on tissue 'x' would have on the others.

If you decide that an exam is the most useful and authentic assessment tool to use, you then need to think carefully about the learning outcomes for the subject and how you will assess these within the exam. The following slides prepared by Dr Alison Matthews give you a guide on how to prepare effective exam questions - including multiple choice (MCQ), short answer, and essay questions - and show you how to incorporate higher order thinking into MCQ.

1. How to construct effective essay questions
2. Multiple choice questions: improve your tests with MCQ