Mutation testing measures a test suite’s ability to detect bugs by inserting bugs into the code and seeing if the tests behave differently. Mutation testing has recently seen increased adoption in industrial and open-source software but sees limited use in education. Some instructors use manually-constructed mutants to evaluate student tests and provide general automated feedback. Additional tutoring requires more intensive instructor interaction such as in office hours, which requires substantial resources at scale. Prior work suggests that students benefit from frequent, actionable feedback, and our work focuses on the challenge of leveraging automation to give students high-quality quality feedback when they need it.

We deployed an automated hint system that provides instructor-written hints related to mutants that student-written tests do not detect. We evaluated our hint system in a controlled experiment across four assignments in two introductory programming courses, comprising 4,122 students. We also analyzed student test suite revisions and conducted a mixed-methods analysis of student hint ratings and comments collected by the automated hint system.

We observed a small, statistically significant increase in the mean number of mutants detected by students who received hints (experiment group) compared to those who did not (control group). In 25% of instances where students received a hint, they detected the mutant in a single revision to their test suite. We conclude with recommendations based on our analysis as a starting point for instructors who wish to deploy this type of automated feedback.