The American Journal of Surgery 185 (2003) 264 –267 Association for surgical education A learning prescription permits feedback on feedback Jay B. Prystowsky, M.D.a,b,*, Debra A. DaRosa, Ph.D.a a Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA Department of Surgery, Northwestern University Medical School, 201 E. Huron St., Galter 10-105, Chicago, IL 60611, USA b Manuscript received September 3, 2002; revised manuscript November 1, 2002 Presented at the 22nd Annual Meeting of the Association of Surgical Education, Baltimore, Maryland, April 4 – 6, 2002. Abstract Background: Students consistently identified inadequate feedback as a deficiency in our third-year clerkship. Methods: We asked students to solicit one faculty and one resident every 2 weeks for written feedback on a “feedback prescription pad.” Each prescription requested four comments: two things the student did well and two things the student needs to improve. Students rated feedback using a five-point scale. A three-point categorization scheme was employed to assess the quality of feedback. Results: Students’ rating of feedback improved significantly compared with a previous time period (3.5 ⫾ 1.2 versus 2.6 ⫾ 1.2, P ⬍0.01). Interrater reliability of our categorization scheme was high (kappa ⱖ0.75, P ⬍0.01) and demonstrated that only 10% of comments were specific enough to qualify as effective feedback. Conclusions: Feedback prescription pads were a simple method to facilitate feedback. Although students appreciated feedback, most feedback was inadequate. Faculty development programs to enhance student feedback should be a priority of clinical medical education. © 2003 Excerpta Medica Inc. All rights reserved. Keywords: Feedback; Surgical education; Clerkship; Faculty development Life-long learning is an important educational objective for healthcare professionals. To achieve this goal, learners must be able to establish their own educational goals and accurately critique themselves [1]. Beginning learners typically do not understand what they need to know and are unskilled at evaluating their own progress. Teacher critique, or feedback, is therefore an essential component of learning especially for beginners. Feedback provides students with criteria against which their performance is measured and enables them to compare their own self-assessment with the assessments of their teachers [2]. The feedback process gives students clear direction about their educational objectives and their progress towards attaining those objectives [3]. Clinical expertise requires a complex set of knowledge, skills, and attitudes. Clinical expertise is often best demonstrated, rather than described. The nature of clinical medicine is that the most fruitful educational experiences often occur at the bedside, in the operating room, or in the clinic. In these set- * Corresponding author. Tel.: ⫹1-312-695-1414; fax: ⫹1-312-6951462. E-mail address: jprystowsky@nmff.org tings, there is a unique opportunity to provide trainees with insight as to what they actually did and the consequences of their actions. The feedback process emphasizes the intended and actual results of the trainee’s actions, and thereby provides stimulus for change [4]. Thus, the importance of feedback in clinical medical education cannot be overemphasized. Unfortunately, it is common in health professions education that feedback is poorly accomplished or even absent [3,5]. Trainees may receive summative evaluations or grades, but direct observation of a trainee by a teacher followed by specific, objective, and timely feedback to the trainee, is an uncommon feature of most health professions educational programs. In our own clerkship, student observations consistently identified inadequate feedback as a critical deficiency in our clerkship. To address this concern, we initiated a program using “feedback prescriptions” in order to increase the frequency and quality of feedback given to students by faculty and residents. Our research questions were the following: (1) Does this program improve students’ perception of the feedback they receive? (2) What is the quality of student feedback provided by faculty and residents? 0002-9610/03/$ – see front matter © 2003 Excerpta Medica Inc. All rights reserved. doi:10.1016/S0002-9610(02)01358-2 J.B. Prystowsky and D.A. DaRosa / The American Journal of Surgery 185 (2003) 264 –267 265 Table 1 Examples of written feedback to students from faculty and residents Fig. 1. Feedback prescription. Things that student did well Things that student needs to improve “Excellent job pre-rounding and organizing patient info in the morning” (3) “Asked for relevant feedback every day we worked together and then noticeably improved on identified learning issues on subsequent days” (3) “Good patient work-ups in the office” (2) “Great with patients and staff” (2) “Fun to work with” (1) “Need to learn how to place foley using sterile technique” (3) “Talk him into going into surgery; he has ideas to go into medicine” (1) Methods During four consecutive 6-week surgery clerkships, we asked each student to solicit one faculty and one resident every 2 weeks to provide written feedback on a feedback prescription pad (Fig. 1). Two-ply prescription pads were given to each student at the beginning of the clerkship. They were designed to fit easily into the pockets of the students’ laboratory coats. Students, faculty, and residents were all oriented to the program. It was emphasized to all groups that comments should be formative, not evaluative. We asked faculty and residents not to provide “grades” to students. Instead, we made it clear that they should provide written reinforcement for positive behaviors and suggestions for improvement of deficiencies. We made a point to ask faculty and residents to be as specific as possible. We also informed students that feedback comments would not be used in the grading process. Each prescription asked for four comments from a faculty or resident: two things that the student did well and two things the student needs to improve. Students were asked to keep one copy of each prescription and submit the other copy to the clerkship coordinator by the end of the clerkship. Student feedback prescriptions were recorded. A threepoint categorization scheme was employed to assess the nature of student feedback: 1 ⫽ useless feedback, 2 ⫽ general compliment or criticism, and 3 ⫽ specific suggestion for improvement or reinforcement. Four faculty rated all comments in a blinded fashion. Descriptive statistics were used and interrater reliability was assessed using Cohen’s kappa. At the conclusion of the clerkship, using a five-point Likert-type scale (1 ⫽ strongly disagree, to 5 ⫽ strongly agree), students were asked to respond to the following statement: “I received sufficient feedback during the clerkship regarding my progress.” Student’s t test was used to assess a difference in student responses from an identical “Thinking up a plan for his patient as he sees them in the clinic then comparing his plan to what we actually do” (3) “Work on oral round presentations” (2) “Keep reading” (2) “No deficiencies; Honors student” (1) “Singing voice” (1) 1 ⫽ useless feedback; 2 ⫽ general compliment or criticism; 3 ⫽ specific suggestion for improvement or reinforcement. period during the previous year when there was no formal student feedback program. Results Forty-six of a possible 52 students (88%) obtained at least one prescription ([mean ⫾ SD] 4.04 ⫾ 1.95; range 1 to 8). There were 74 faculty prescriptions that included 296 evaluable comments that were provided by 18 faculty. The interrater reliability of our categorization scheme for faculty comments was kappa ⫽ 0.82 (P ⬍0.01). There were 114 resident prescriptions that included 456 evaluable comments that were provided by 42 residents. The interrater reliability for resident comments was kappa ⫽ 0.75 (P ⬍0.01). There was no significant difference in the type of comments provided by faculty and residents. Of 296 faculty comments, 55 (19%) were useless, 208 (70%) were general compliments or criticisms, and 33 (11%) were specific. Of 456 resident comments, 90 (20%) were useless, 327 (71%) were general compliments or criticisms, and 39 (9%) were specific. Examples of all three types of feedback are listed in Table 1. There were 72 specific comments that were made. Approximately 63% (45 of 72) of specific comments related to “things the student needs to improve” whereas 37% (27 of 72) related to “things the students did well.” This finding suggests that faculty and residents were more likely to be specific when providing negative comments to students as opposed to positive comments. In response to the statement concerning the adequacy of feedback, rating of the clerkship during the study period was significantly higher than the rating during the previous year prior to the establishment of the student feedback program (3.5 ⫾ 1.2 versus 2.6 ⫾ 1.2; t ⫽ 3.5, P ⬍0.01). 266 J.B. Prystowsky and D.A. DaRosa / The American Journal of Surgery 185 (2003) 264 –267 Comments Feedback in clinical medical education is critical to effective learning. There are multiple reports that feedback accelerates and facilitates learning in medical education [6,7]. Stillman et al [8,9] found that students who received feedback regarding their interviewing skills, performed significantly better than students who did not receive feedback. Scheidt et al [10] observed similar findings among students who received faculty critiques of their communication skills compared with students who received no feedback or those who used self-guided critiques. Wigton et al [11] also observed that students who received feedback regarding their diagnostic skills, improved these skills more rapidly than those students who did not receive feedback. Learners value constructive feedback. Wolverton and Bosworth [12] surveyed family practice residents perceptions about effective teaching and observed that residents ranked teacher provision of constructive feedback second only to clinical competence. Black and Harden [13] surveyed students undergoing an objective structured clinical examination and found that students preferred to receive feedback during, rather than after the examination. Students understand that without feedback, mistakes can go uncorrected and bad habits may develop. Conversely, positive behavior may not be reinforced and can be lost. Finally, without adequate feedback, the importance of examinations may be overemphasized. Nonetheless, feedback in health professions education is typically inadequate or absent. Irby [2] reported that students at the University of Washington School of Medicine rated their teachers lowest in terms of their “providing direction and feedback.” There may be striking differences in the perceptions of feedback by faculty and trainees. Collins et al [14] surveyed faculty and found that 79% thought they were assessing their trainees’ skills on rounds, whereas only 46% of trainees agreed with that perception. Stritter et al [15] also observed that teachers consistently thought they provided more and better feedback than their students felt they received. There are many potential barriers to effective feedback in clinical medical education [1,3]. Faculty may not have had good models of feedback that they can emulate. Both faculty and trainees may have had poor experiences with feedback or they may fear that negative feedback may damage their relationship. Faculty may not feel qualified to provide feedback based on inadequate or brief interactions with trainees. Faculty may experience time constraints that prevent them from spending appropriate time with trainees to provide constructive feedback. Faculty may not understand the elements of effective feedback or its importance in clinical medical education. Health professions education has often relied on large group teaching in which case it is difficult to provide helpful feedback to students. In short, there are numerous reasons why our medical education environment does not consistently foster effective student feedback. Ende [5] has provided several guidelines for giving feedback. Feedback should be given by faculty who demonstrate an interest in the accomplishments of their trainees. Optimal feedback should be based on first-hand information and should be given in a timely fashion. Feedback should be specific and provided in descriptive, non-evaluative language. Although praise should be used to encourage student accomplishment, feedback should be used to clarify goals and expectations, and encourage students to improve their performance. Prior to the inception of our feedback prescription program, our students consistently rated our provision of feedback as one of our clerkship’s worst attributes. These observations stimulated us to devise a system to improve our feedback system. We observed that prescription feedback forms were feasible and effective in increasing the frequency of student feedback. Student participation was high. Although we provided faculty and residents with orientation to this initiative, we did not institute an extensive faculty development program on the elements of effective feedback. Not surprisingly, the overall quality of feedback that we observed was poor. Only about 10% of comments were satisfactory in terms of their specificity or utility. Despite these shortcomings, there was significant improvement in the ratings by students of the feedback they received compared with a previous time when there was no formal feedback program. In short, students hunger for any type of feedback regardless of quality. We also found that a simple three-point categorization scheme of student feedback proved reliable and assisted us in classifying the quality of our student feedback. It clearly demonstrated that although feedback was given by faculty and appreciated by students, its educational impact was not maximized. Taken together, these observations lead us to believe that delivery of effective feedback should be a prominent subject for faculty development so that students may receive quality feedback. In conclusion, feedback is both necessary and valuable in health professions education. We found that the feedback prescription pad was a simple, inexpensive, and feasible method that facilitated student feedback by faculty and residents. Although students appreciated the feedback they received, we observed that most faculty and resident feedback to students was inadequate. Faculty development programs to enhance the delivery of effective student feedback should be a priority of clinical medical education. References [1] Westberg J, Jason H. Providing constructive feedback. Boulder: Johnson Printing, 1991. [2] Irby DM. Clinical teaching and the clinical teacher. J Med Educ 1986;61:35– 45. [3] Sachdeva AK. Use of effective feedback to facilitate adult learning. J Cancer Educ 1996;11:106 –18. J.B. Prystowsky and D.A. DaRosa / The American Journal of Surgery 185 (2003) 264 –267 [4] Nadler DA. Feedback and organization development: using databased methods. Reading: Addison-Wesley, 1977. [5] Ende J. Feedback in clinical medical education. JAMA 1983;250: 777– 81. [6] Hammond KR. Computer graphics as an aid to learning. Science 1971;172:903– 8. [7] Skeff KM. Evaluation of a method for improving the teaching performance of attending physicians. Am J Med 1983;75:465–70. [8] Stillman PL, Sabers DL, Redfield BM. The use of paraprofessionals to teach interviewing skills. Pediatrics 1976;57:769 –74. [9] Stillman PL, Sabers DL, Redfield BM. Use of trained mothers to teach interviewing skills to first year medical students: a follow-up study. Pediatrics 1977;58:165–9. 267 [10] Scheidt PC, Lazoritz S, Ebbeling WL, et al. Evaluation of a system providing feedback to students on videotaped patient encounters. J Med Educ 1986;61:585–90. [11] Wigton RS, Kashinath DP, Hoellerich VL. The effect of feedback in learning clinical diagnosis. J Med Educ 1986;61:816 –22. [12] Wolverton S, Bosworth M. A survey of resident perceptions of effective teaching behaviors. Fam Med 1985;17:106 – 8. [13] Black NMI, Harden RM. Providing feedback to students on clinical skills by using the objective structured clinical examination. Med Educ 1986;20:48 –52. [14] Collins GG, Cassie J, Daggett C. The role of the attending physician in clinical training. J Med Educ 1978;53:429 –31. [15] Stritter FT, Haines JD, Grimes DA. Clinical teaching reexamined. J Med Educ 1975;50:876 – 82.