As researchers investigate the role of AI in education, questionnaires have become a prominent method of assessing AI literacy. They tend to include questions on technical AI concepts, though the relationship between students’ technical understanding of AI and ability to effectively use AI tools has been left largely unexplored. My study seeks to address this gap using an embedded mixed-methods approach, with quantitative analysis of survey data supported by qualitative analysis of focus group interviews. There was no significant relationship within this sample of undergraduates between their perceived technical knowledge of AI and perceived ability to effectively use AI. Further, the significance of relationships between survey responses did not align with the results of the confirmatory factor analysis originally used to construct the survey. Implications of technical knowledge in question phrasing and student demographics may have impacted student responses.