Exam wrappers

This description of "exam wrappers" is taken from an online newsletter hosted by Stanford University and edited by Rick Reis: Tomorrow's Professor. For more information, visit http://mailman.stanford.edu/mailman/listinfo/tomorrows-professor. 

What are Exam Wrappers?

Exam wrappers are short activities that direct students to review their performance (and the instructor's feedback) on an exam with an eye toward adapting their future learning. Exam wrappers ask students three kinds of questions: How did they prepare for the exam? What kind of errors did they make on the exam? What could they do differently next time? Each of the question types is discussed next.

1. How students prepare for the exam. Asking students to reflect on how they prepared for the exam forces them to confront the choices, explicit or implicit, they made about their studying. This prompts students to consider issues such as whether they studied enough or sufficiently in advance. Similarly, asking students which of various study strategies they employed (e.g., reviewing notes, solving practice problems, rereading the textbook) highlights that there are many options they could have taken. It also presents various study strategies that students might not have even considered and thus suggests some new possibilities for how they might prepare differently next time.

2. What kind of errors students made. Once they have received a grade, students do not always think carefully about their performance on an exam. If they did well, they might mark it as a success without much further thought; if they did poorly, there's strong temptation to leave the painful event behind. Thus, the second set of questions posed in exam wrappers is designed to encourage students to analyze their performance in greater depth, giving students something constructive to do with feedback and graded exam offers. One way to do this is to identify the critical components or stages of the tasks on the exam and have students estimate their degree of difficulty (e.g., how many points they lost) with each component. For example, did they read the question carefully, did they have trouble "setting up" the problem, did they fail to understand the concepts involved, or did they make mistakes on the required arithmetic or algebra? Focusing students' reflection at this level informs their analysis of their own performance. Moreover, the labels for the different possibilities provide a concrete language for students to use when assessing their own performance. Note that this part of the exam wrapper is a natural place for instructors to tailor the questions to their own needs. Instructors may also want to adapt these labels to include specific misconceptions or difficulties that have been revealed in their course by past students. Or, they may want to include more general issues that impact students' exam performance. For instance, an instructor I worked with recently was concerned about test anxiety adversely affecting her students' exam performance, so she incorporated this issue into her exam wrapper.

3. How students should study for the next exam. Students can use their responses to the first and second types of exam-wrapper questions to think about how they should approach the next exam. A key goal of the third type of question exam-wrapper question is to help students see the association between their study choices and their exam performance so they can better predict what study strategies will be effective in the future. One way to do this task is to ask students to look back at their responses to the first two parts of the wrapper and then to list specific ways they might prepare differently for the next exam to improve their performance. Another option is to prompt students to attribute their various difficulties (from part two of the wrapper) to specific study strategies (from part one) they did or did not employ. Rather than merely telling students to "study harder" or "do more practice problems before the exam", this third type of exam-wrapper question helps students discover effective study strategies on their own. In effect, exam wrappers are asking students to give their future selves advice.

Why Exam Wrappers Work

The earlier section on design considerations described five constraints that a metacognitive intervention should satisfy to be practical and effective. Understanding the ways in which exam wrappers satisfy these constraints helps explain why they work.

1. Impinge minimally on class time. Exam wrappers require only a few minutes and are completed at a time when students arguably are somewhat distracted anyway. Moreover, instructors can eliminate even this minimal impact on class time by giving exam wrappers as part of homework.

2. Be as easily completed by students within the time they are willing to invest. With typically only one page of questions, none of which requires much writing, exam wrappers take relatively little student time.

3. Be easily adaptable. As we have seen, exam wrappers include three main question types, and these question types can be applied to almost any course, as long as it has exams. In addition, this general approach can be applied to any type of graded assignment.
4. Be repeatable yet flexible. The core questions being in an exam wrapper do not diminish in value when asked repeatedly. At the same time, it is easy to adjust the details of exam wrappers so as to keep the exercise fresh. Instructors can easily vary the specific content of exam wrapper questions, add new questions, and tailor the questions to their particular instructional situation.

5. Exercise the skills instructors want their students to learn. The reflections required to complete an exam wrapper leads students to assess their own strengths and weaknesses, evaluate their performance, identify which strategies work for them, and generate appropriate adjustments. These are key metacognitive skills that many instructors want to promote.

How to Use Exam Wrappers

Here I describe a basic recipe for how to use exam wrappers, along with variations and options instructors may find useful.

Step1: Students prepare for and take the exam using their typical study strategies. No special intervention is need for this first exam.

Step 2: The instructor gives students the exam wrapper instrument when the graded exams are returned and asks students to complete the exam wrapper as soon as possible upon seeing their exam performance. Ideally, this is done right then in class and need only take 10 minutes of class time. But there are other possibilities. Students can complete the exam wrapper as homework, submitting their responses by a specified deadline. Or students can complete the exam wrappers online as a nongraded assignment (e.g., within a course management system or online instructional environment or with an online survey system).

Step 3: The instructor collects the exam wrappers. Although exam wrappers are not graded activities, it is important to collect them because (a) the exam wrapper will need to be returned to students at a later point (see Step 4) and this prevents them from getting lost, and (b) the instructional team may want to review students' responses to gain insight into their students' learning that they otherwise might not be able to obtain. In particular, the instructor or teaching assistants can skim students' responses to see whether there are patterns in how students analyzed their strengths and weaknesses or in how students described their approach to studying for the exam. For example, instructors may be surprised to learn the amount of time students spent studying (either how much or how little), when students chose to start their studying (e.g., 2 a.m. the night before), or how students spent their study time (e.g., memorizing formulae rather than solving practice problems). The instructional team can also consider the additional instructional support students said they would like to receive and possibly provide something along these lines. A wide variety of adjustments may be suggested based on what the instructional team learns from the exam wrappers about students? strengths and weaknesses, and study strategies.

Step 4: At the time when students should begin studying for the next exam, the instructor returns the completed exam wrappers (from the previous exam) to students. The idea here is that students review their own recommendations for how to study more effectively, given their own past experiences, strengths, and weaknesses. Depending on the class format and the time available, there are many variations on this step that instructors can use.

Step 5 (optional, but desirable): Repeat steps 2 through 4 for subsequent exams.

When an instructor provides exam wrappers regularly across multiple exams, students get repeated practice in applying the skills of self-regulated learning. This helps students build a habit of mind to monitor their own learning, reflect on their study strategies, and make appropriate adjustments. For reasons mentioned earlier, it can be useful to include nontrivial variation in the structure of subsequent wrappers while still prompting the desired metacognitive processes. For example, an exam wrapper used for an exam later in the semester can be streamlined compared to the first exam wrapper. Having the wrapper still gets students to engage in reflection and analysis, but fading the scaffolding encourages students to take on more of the responsibility for this process.

These five steps are easy to implement, take relatively little time, and are very flexible. The metacognitive practice from using wrappers in a course offers substantial benefits. And when multiple instructors do this across different courses, students can learn metacognitive skills in multiple contexts, thereby increasing their likelihood of transferring the skills to new learning situations in the future. This is exactly what we did at Carnegie Mellon University, implementing wrappers in several introductory math and science courses, as described in the next section.

Techniques for formative assessment

Here are a couple more tools you can use for formative assessment (activities that provide you with information about how to design future learning activities).

Analogy Prompt
Occasionally, when you are eliciting information from students about how well they understand the material you are teaching, get them to complete this sentence: "(a designated concept, principle, or process) is like ___ because ___."

3-minute Pause
The 3-minute Pause gives students an opportunity to stop and reflect on new knowledge recently acquired. Get them to complete the following statements:

• I changed my attitude about ... because ...
• I became more aware of ...
• I was surprised by ...
• I felt ...
• I empathized with ...

Ticket to Leave
This is a closing activity where students respond in writing or verbally to short assignments before they leave the room.

Formative assessment: index card summaries

Distribute index cards in class if your are teaching face-to-face, or set up a bulletin board discussion if you are teaching online.

Instructions to students:
Side 1 / Question 1: Based on our study of (topic), summarise one of the main ideas you have encountered as a single sentence statement.
Side 2 / Question 2:  In one sentence, identify something about (topic) that you do not yet fully understand.

Collect and read the material and use it to provide students with extra learning resources, or to guide their future learning.

Web-based tools for teaching

There are a range of tools available that facilitate collaborative and group work on the web. Perhaps surprisingly, they are often free. Here are some I have come across recently. These can be used in online classes, but are equally useful in supporting face-to-face teaching.

Titan Pad

http://titanpad.com/

This a tool that provides for multiple writers and editors on the same document. Updates happen in real time, there is an infinite undo button, and each contributor gets his or her own colour.

Crocodoc

https://crocodoc.com/

Crocodoc isn’t free, but you can sign up for an evaluation account. Crocodoc is a tool designed to convert Microsoft Office and PDF documents to HTML5. These documents can be embedded in any web or mobile application using iFrame or Javascript. The application has a range of commenting and mark-up tools, so that multiple authors can comment, highlight, and drawing on the document in creation.

iResponse

http://www.itate-solutions.com/edu-solutions/iresponse-classroom-responder-system/

iResponse is two iPhone/iPad apps – one for the teacher and one for the students. It allows students to use their mobile devices as classroom clickers, responding to multiple-choice, true-false, yes-no, and survey questions. The teacher can view the class results on a Mac. It’s a pretty nifty Mac application that means that institutions can get students to use their phones and tablets to replace the hand-held classroom response devices that have been around for several years.

VotApedia

http://www.urvoting.com/

VotApedia is an audience response system that makes use of mobile phones. You can set up surveys, questionnaires, quizzes, anonymous texting, identified texting, and ranking, e.g. of conference presentations. It's a great tool: free, and invented up the road from me by the CSIRO.

Lino It

http://en.linoit.com/

Lino It is a “free sticky and canvas service that requires nothing but a web browser”. Those who have access can post a sticky note to their canvas from anywhere at any time. The canvas can be shared by several people.

Wallwisher

http://wallwisher.com/

Wallwisher is similar to Lino It. It’s a free shared web-based notice board that allows you to make noticeboards, teach, bookmark sites, discuss things, moderate discussions, brainstorm, send messages, plan events, keep notes, make lists, watch and share videos, and collect feedback.

Webspiration

http://www.mywebspiration.com/

This one isn’t free, but you get a 30-day trial of the application, which is a tool to help students to write. It looks pretty good.

Quizlet

http://quizlet.com/

A tool to make online quizzes.

Quia

http://www.quia.com/

Another tool for making online quizzes.

Voicethread

http://voicethread.com/

This certainly isn’t free, but it looks like a great tool. Here’s what they say they do:

“With VoiceThread, group conversations are collected and shared in one place from anywhere in the world. All with no software to install.

“A VoiceThread is a collaborative, multimedia slide show that holds images, documents, and videos and allows people to navigate slides and leave comments in 5 ways - using voice (with a mic or telephone), text, audio file, or video (via a webcam). Share a VoiceThread with friends, students, and colleagues for them to record comments too.

“Users can doodle while commenting, use multiple identities, and pick which comments are shown through moderation. VoiceThreads can even be embedded to show and receive comments on other websites and exported to MP3 players or DVDs to play as archival movies.”

 If you are using something along these lines, post a comment and a link.

Formative assessment in action

University teachers tend not to be explicit about the formative assessment techniques they use. In fact, there are many who don't use the term at all when describing their practice.

Formative assessment is any learning activity that will assist the teacher to identify deficiencies in student learning to date, design future learning tasks to address these deficiencies, and provide students with feedback that will assist them to improve their performance in summative assessment activities. Formative assessment may be marked, but that mark will not count towards a final grade. It's one of the most useful strategies employed by effective teachers in any part of the sector. It helps teachers to monitor their own effectiveness, and to monitor the progress of their students.

So, if you've ever reached the end of a teaching session - a lecture, a tutorial, a seminar, a class, or some other form of lesson - and wondered whether your students have actually grasped the points you intended them to learn, you need to introduce some formative assessment. Here are a few techniques that will give you feedback on your teaching and on how well students have learnt the lesson you were teaching. You might discover that some quite unexpected things are happening in your class - both inspirational and horrifying. It's always best to know.

1. The Minute Paper: Towards the end of the session, ask students to spend five minutes answering two questions:
• What was the most important thing you learnt during this class?
• What important question remains unanswered?
2. The Muddiest Point: Towards the end of the session, or at the end of a section, ask students to spend five minutes answering the question "What was the muddiest point in ___ ?".
3. RSQC2: Students write brief statements that recall, summarize, question, connect, and comment on key points from a session.
4. Pro & Con Grid: Students list the pros/cons, costs/benefits, or advantages/disadvantages of an issue or question in a two-column format.

Collect the papers on occasion, and respond to them at the beginning of your next session with the students, in whatever way seems appropriate. At other times, simply invite students to make brief comments themselves at the beginning of the next class before moving on.

Variations on these techniques can be found in a range of different texts.

Directed Paraphrasing

This is a technique I encountered in an old booklet called Class Assessment Techniques: a handbook for faculty, by K. Patricia Cross and Tom Angelo, and have adapted slightly for this posting.

The activity is designed to test "the degree to which students have understood and internalized a given lecture or reading by [evaluating] their ability to summarize and restate the lesson in their own words". That sounds a bit bland, but think further on this later comment by the authors: "much of a student's eventual success depends on his or her ability to first internalize rather specialized and often complex information and then effectively communicate it". Are you the kind of person who understands something better once you have taught it to someone else? Me too.

This is a very authentic learning activity which mirrors the kinds of skills that students will need once they enter the workplace.

So here's what you do.

1. Select a point in your course - perhaps after a major reading assignment, lecture, or seminar.
2. Decide on an appropriate audience for the students, and how much speaking time they will need. Choose a non-expert audience. For example, if they are psychology students, they might be asked to describe a counselling technique to a group of teachers; if they are accountants, they might be asked to describe a change in the tax law to a client; and if they are medical students, they might be asked to describe a surgical technique to the parents of a child about to undergo an operation.
3. Provide the audience with feedback sheets. Each feedback sheet should include the following:
• an overall rating presented as a set of boxes to tick:
• confusing
• mininally understandable
• adequately outlined
• excellent explanation
• space for members of the audience to record comments about the best aspects of the presentation
• space for members of the audience to record comments about the worst aspects of the presentation
4. If you have time, for each presentation, put the speaker in a group with his or her audience, and give them time to talk about the best and worst aspects of the presentation. You might need to provide guidance on the best way to provide useful feedback without destroying confidence. Alternatively, you as teacher can collect these evaluation sheets, go through and summarize them, add your own comments, and return them to the speaker.

Make sure that students do present the material in the role they have been allocated (and that they don't merely read out a pre-prepared script from a cheatsheet or a PowerPoint slide). Encourage them to use metaphor and examples, to draw on the board or on a piece of paper, and to check understanding with their audience to ensure that the main points are getting across.

Further reading:

Cross, K. Patricia & Angelo, Thomas A. (1988). Classroom Assessment Techniques: a handbook for faculty. Michigan: National Center for Research to Improve Postsecondary Teaching and Learning.

I found a copy on the web at http://eric.ed.gov/PDFS/ED317097.pdf, courtesy of my University's libary database subscriptions, but more recent and much more beautiful editions can be purchased from all the usual places.

The Mini-conference

In a traditional face-to-face teaching environment where students are assessed partly on an oral presentation, the lecturer usually puts up a list of the available spots and students (or groups of students) sign up for a spot.  Some go for the earliest spots, so that they can get it over and done with early. Others sign up for later spots, so that they can see what others do before them. Either way, marks have to be tweaked to take the schedule into account.

Here’s an alternative, suggested to me by my colleague James Meek: the mini-conference.

Instead of holding oral presentations every week for the 13 weeks of the semester, bring the students together towards the end of the semester for a whole day (or two), and run the day like a mini academic conference, complete with plenaries, discussion panels, streamed presentations, and opportunities for informal networking.

This works equally well for traditional semester-long courses, intensively-delivered courses, online courses (except here you run it like an online conference, e.g. Follow the Sun: Online Learning Futures Festival), and short courses.