Designing exams

What is the purpose of the exam?

Like other assessment tasks, examinations can be used to gauge both student learning, and the efficacy of the learning process. So, we must ensure that the cognitive skills examined during the examination are constructively aligned to the skills required by the subject learning outcomes.

The primary aim with examination questions is to provide students with a platform to demonstrate their knowledge and their ability to apply the knowledge in a context relevant to the subject learning outcomes.

What exam format/method will allow the students to demonstrate the relevant learning outcomes?

What type of response is required?

A fixed response can be explained as an objective response where students select the correct response to a question, or supply a word or short phrase to answer a question or complete a statement. For example, multiple choice, true-false, matching or fill-in-the-blank questions.

A constructed response is a subjective narrative that challenges students to create an original answer. For example: short answer, long answer, essay, or performance test items.

Have you chosen the best test format for evaluating cognitive ability to meet the learning outcomes?

The verbs in the learning outcome can provide direction towards the choice of the question type. Some verbs such as identify, list, and select clearly indicate that students need to select the response. If the question is written in such a way that a student has to reason in order to select a correct response, student actions such as analyse or compare could be included in the selection of answers. Generally, verbs such as analyse, apply, interpret, compare, infer and predict indicate that a student should construct a response.

For instance, if the subject learning outcome is expecting the student to be able to synthesis information; then multiple choice questions (MCQ) would not serve as an appropriate assessment tool. Instead, a long answer question with clarity and emphasis on the importance of the student’s ability to synthesis information should be given. However, having MCQ as an assessment does not always mean that the questions are meant to evaluate lower order cognitive skills. MCQ type questions can be constructed to assess higher order cognitive skills like analyse, compare, judge etc.

To gain a greater understanding of potential student actions for different question types, you could review Bloom's or SOLO Taxonomy.

Exam format examples

There are many different exam formats. Some examples are:

• Automatically Marked exams - including multiple choice, fill in the blank and matching question types.
• Short answer exams.
• Essay exams.
• Problem or case-based/scenario exams.
• Oral exams where students may be interviewed and need to justify their answers or demonstrate their clinical reasoning.
• Practical/Performance exams where students physically demonstrate skills and knowledge in a controlled environment such as a simulation or role-play or practical skills exam.
• Group exams where students collaborate on a task to create a product or solution to a problem.
• Computational or calculation exams.
• Open-book and take-home exams are usually based around higher order thinking skills. They may require synthesising research, forming recommendations, evaluating contexts or justifying decisions.

Mapping Questions

You may want to map the questions against the subject learning outcomes, knowledge level, degree of difficulty, etc to gain an understanding of coverage of the outcomes and whether the right level of challenge has been achieved.

The exam may also combine different formats such as multiple choice and case based questions.

Have you considered these factors?

Use this list to sense check your exam.

• How much time you have to set question papers and mark.
• How much the the student has to cover the content in a set time.
• Number of students taking the exam.
• Diversity of students taking the exam.
• What kind of technology is used for exam delivery.

Reliability, validity and bias

We also talk about exams having the following characteristics of:

• Reliability - is demonstrated when an exam produces outcomes that are consistent over time and reflect a students actual ability (Banta and Palomba, 2015). Reliable tests are not too long, and have clear instructions and marking guidelines.
• Validity - means that the test has assessed what it aimed to assess (Banta and Palomba, 2015). This means it is aligned to the subject learning outcomes.
• Free from bias - There are two types of bias:
  • Construct validity bias - Does the exam assess what it is intended to assess (the subject learning outcomes)?
  • Content validity bias - Does the exam favour one group of students over another group?

How to deter cheating

Another aspect of designing an exam is to consider strategies to deter cheating.

• Set questions that require students to make use of their subject material rather than simply locating or rewriting information.
• Require students to submit workings, calculations, proofs or justifications for their answers.
• Use open ended questions that may have multiple solutions.
• Require student responses to be contextualised to their own experience.
• Never reuse whole exams from previous sessions. Even small changes to case studies or questions can be effective.
• Talk to students about why academic integrity matters. Discuss strategies that demonstrate academic integrity.
• Make explicit and clear your expectations about student behaviour in exams and what is allowed.
• Have clear exam instructions that have been discussed or communicated with students.
• Communicate to students about any plagiarism detecting strategies that might be used such as Turnitin.
• Help students feel confident in taking a exam through scaffolding in the subject and use of a practice exam with feedback.