Students' Use of GitHub Copilot for Working with Large Code Bases
This program is tentative and subject to change.
Large language models (LLMs) are already heavily used by professional software engineers. An important skill for new university graduates to possess will be the ability to use such LLMs to effectively navigate and modify a large code base. While much of the prior work related to LLMs focuses on students learning to program, less work has focused on the student experience and trust in using these tools, especially while working with large code bases. In this study, we taught students about various Github Copilot features, including Copilot chat, in an upper-division software engineering course and asked students to add a feature to a large code base using Copilot. Our analysis revealed a new interaction pattern that we call textit{one-shot prompting}, in which students ask Copilot to implement the entire feature at once and spend the next few prompts asking Copilot to debug the code or asking Copilot to regenerate its incorrect response. Finally, students reported significantly more trust in the code comprehension features than code generation features of Copilot, perhaps due to the presence of trust affordances in the Copilot chat that are absent in the code generation features. Our study contributes to our understanding of how students interact with LLMs for working with large code bases, provides implications for educators and researchers, and suggests design recommendations for existing LLMs.