Mind your language: on novices' interactions with error messages

Reviewer: Keith Harrow

Anyone who has taught an introductory programming course knows that students are often baffled by the error messages they receive from compilers or integrated development environments (IDEs). On the one hand, this statement seems so obvious that it requires no further discussion. On the other hand, as the authors of this paper note, even if we all believe the statement to be true, it is nice to have some formal verification of this phenomenon in terms of human evaluation. Even more useful would be some recommendations on what can be done to help alleviate the situation. This paper focuses on how an error message influences what a student does next. The authors worked with students using the DrRacket programming language at Brown University, Northeastern University, and Worcester Polytechnic Institute. They used a number of investigative techniques: they logged students' programming sessions and were able to see the error message, where the student edited in response, and how much editing the student did before resubmitting; they talked to the students about their debugging; and they conducted tests to see how well the students understood the vocabulary used in the error messages. The results are sobering. First, many students took the error message and highlighting too literally, as if the compiler knew what the mistake was and how to fix it, and was thus giving the student a hint on how to proceed. Students thought the IDE was saying "edit here" and "look inside here for the problem." A number of students debugged by guessing what the mistake was-that is, by simply trying alternatives-without understanding either the underlying problem or how to fix it. Many students did not understand the vocabulary used in the error messages, even though the DrRacket development team had tried to be sensitive to this issue. Perhaps the most useful feature of the paper is a series of recommendations to instructors and those involved with IDEs. First, the error message should not propose a solution-just diagnose the problem. (Of course, this is exactly the opposite of what students want!) The error message should not lead the student, because of misinterpretation, to an incorrect edit. Specific recommendations include: simplify the vocabulary in the error messages (this is especially important for introductory students); be explicit about inconsistencies (for example, if a function use does not match the definition, show both parts and not just one); and help students match terms to code fragments (for example, a function body). Some of the recommendations make more sense in the authors' environment using the IDE that their team created. For example, they recommend that professors consider the specific IDE when designing lectures, and treat error messages and highlighting as integral parts of the course. This may be feasible in their limited student environment, but it is harder to do in a class where students might be using different IDEs. Although the paper focuses on pedagogic IDEs, the authors also discuss related work with other Java and C++ IDEs. To summarize: "This paper describes a series of formal studies on novice students' understanding of vocabulary and source highlighting." It addresses a problem that all programming instructors face, and it suggests some interesting ways to study that problem. Online Computing Reviews Service