ChatGPT Can Pass Engineering Homework, But Struggles With Deeper Thinking
A recent study has demonstrated that while ChatGPT excels in solving structured math problems, it faces challenges with reasoning-based tasks. This sheds light on the importance for educators to reconsider assessment strategies that prioritize genuine understanding over mere academic success shortcuts.
Research: The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own. Image Credit: Anggalih Prasetya / Shutterstock
Assessment of ChatGPT's Performance
Conducted by researchers in the Department of Aerospace Engineering at the University of Illinois Urbana-Champaign, the study evaluated ChatGPT's capabilities in comparison to human students in an undergraduate control systems course over a semester.
The findings revealed that while ChatGPT scored an A on conventional math assignments, it exhibited peculiar responses. However, when faced with complex problems requiring critical reasoning skills, its performance dropped to a D. This stark difference underscored the limitations of the AI model in handling higher-order thinking tasks.
Educational Implications
The study suggested that by relying solely on ChatGPT, a student could feasibly achieve a passing grade without truly grasping the subject matter. This approach, though, might lead to a superficial understanding, as indicated by the disparity between ChatGPT's performance in different types of questions.
Caution in Using ChatGPT
While ChatGPT demonstrates speed and accuracy in solving structured problems, caution is advised in its application. The AI's tendency to provide incorrect or irrelevant responses, including the use of technical terms not covered in the course, raises concerns about its reliability.
Moreover, the study highlighted that while ChatGPT showed some capacity to learn from corrections, its overall progress remained limited. The premium version of the software was noted to offer enhanced analytical capabilities, but the free version was preferred for this study to align with typical student resources.
Conclusion
The research emphasized the importance of designing assessments that foster genuine understanding and critical thinking skills. By acknowledging the benefits and limitations of AI tools like ChatGPT, educators can tailor their approaches to encourage meaningful learning experiences for students.
This work was supported by the Grants for Advancement of Teaching in Engineering program at The Grainger College of Engineering, University of Illinois, Urbana-Champaign.
