Understanding task relevance is important for identifying engagement patterns and shifts in focus during learning activities. By analyzing these conversations, educators gain insights into student behaviors, pinpoint when students are most engaged, and identify potential learning gaps. This paper investigates the characteristics and trends of task- relevant dialogue when middle school students develop AI-based chatbot in a science learning context. We use a keyword-based approach to identify on-task and off-task utterances and further divide the on-task utterances to AI talk and Science talk. We visualize the trends among 30 dialogue sessions that reflect students’ focus on specific topics over time. Our results show that as the activities progressed, students’ discussion on AI topics decreased but discussion on science topics increased. This analysis provides an explainable and contextual-relevant approach to understand dialogue topic shifts and student engagement when building AI artifacts, which help inform the design of effective interventions.