Gauging a student’s understanding of course concepts, at an arbitrary point during a course, can be challenging. Standardized exams offer only a snapshot of performance rather than a deep understanding of progress. However, with Large Language Models (LLMs) now deployed at scale in CS1 courses, we can track multiple attempts from each student for every homework problem. This data provides insights into how students learn and deploy concepts over time, presenting a unique opportunity to rethink how we track changes in individual student knowledge. Traditional Knowledge Tracing (KT) methods often lack explainability and are computationally expensive. In contrast, our framework leverages an LLM to identify student progress on labeled, problem-level concepts from a student homework code submission. Our initial results show that the student’s knowledge state can be dynamically updated. This knowledge state can then be used to provide more targeted, effective feedback and create tailored study materials.