What happens when AI tools empower students instead of replacing their critical thinking? This is the question a Willamette University faculty-student research team, led by Assistant Professor of Computer Science Lucas Cordova, has set out to answer.
Cordova and his team gave subjects one of two chatbots to interact with for their study: OpenAI’s ChatGPT and a customized version with learning outcomes and guardrails baked into the answers.
“The idea is that instead of talking to an all-knowing entity that will just give students the answers, they chat with a large language model that prompts them to think more deeply about their responses,” says Ben Webster BS’27, one of the student researchers on the team.
“We found that the students using our customized version performed around 25% better on assignments given to them,” says Cordova. “When we reviewed the chat transcripts, we found that the plain ChatGPT veered off in different directions and provided information that wasn’t helpful or that caused confusion where our specialized chatbot did not.”
The team published their findings in the paper “Not All Chatbots Teach: Evidence for Pedagogical Design in AI-Assisted Technical Education.” The team of students, including Webster, Teo Mendoza BS’27, and Derec Gregory BS’27 presented the paper at the Association for Computing Machinery’s Annual Conference on Cybersecurity and Information Technology Education through funding from the Computing Research Association. Students Kayle Megginson BS/MS’25 and Sam Holmes BS’27 also contributed to the project as co-authors.
“Not All Chatbots Teach” is part of a larger assessment tool Cordova is developing called Pedalogical — a play on the words pedagogy and logic. He’s already implemented into his computer science classes at Willamette and is piloting the program with other instructors.
“We designed the tool in such a way that educators could use it for any subject or education level,” Cordova says. “It supports much more than chatting. Instructors can develop tailor-made multiple choice or matching pair questions. It doesn’t replace instructors, but acts as a complement to support their work.”
Cordova hopes to further personalize the learning experience for each student. “We’re building a model that can assess where each student is starting from and adapt the learning experience in response,” Cordova says. “It helps instructors shape feedback, practice, and assessment to match their course goals while meeting students where they are."
His team’s research aims to show that with the proper guardrails and datasets, the power behind generative AI can be harnessed.
“We can teach students to evaluate what they get back from chatbots to determine whether it’s high quality information or not,” Cordova says. “It also makes the process more efficient for students and instructors. The learning outcomes are built in, so the guidance stays aligned with the course.”
The project puts the interdisciplinary human-centered computing teaching and learning offered at Willamette into practice.
“What stands out about our approach is our holistic vision as Willamette students,” shares Webster. “Others are looking in this direction, but our vision is uniquely human-centered.”
“I try to instill in my students the belief that we're just at the brink of what generative AI can do,” says Cordova.
