ChatGPT earns B in college engineering course, U. Illinois study finds
AI lacks ‘deep understanding’ of material, researchers find. ChatGPT managed to pass a college-level engineering course at the University of Illinois at Urbana-Champaign, earning a B grade with what researchers described as “minimal effort.”
Study Details
The research conducted by doctoral student Gokul Puthumanaillam and Professors Timothy Bretl and Melkior Ornik involved programming a robot to complete an undergraduate aerospace engineering course. Their findings were published in a paper titled “The Lazy Student’s Dream: ChatGPT Passing an Engineering Course on Its Own”. The paper revealed that ChatGPT achieved a B-grade performance (82.24%), which approached but did not exceed the class average of 84.99%.
According to the study, ChatGPT excelled in structured assignments but faced limitations in open-ended projects. The evaluation process involved 115 course deliverables under a ‘minimal effort’ protocol to simulate realistic student usage patterns. Assessments included multiple-choice questions, Python programming tests, and detailed analytical writing assignments.
The project’s website states that ChatGPT performed well in methodology description and result presentation but struggled with critical analysis and design justification.
Challenges Faced by AI
Researchers noted that while AI, including ChatGPT, excels at pattern recognition, it lacks a deep understanding of course material. This limitation becomes evident in handling complex questions that require applying learned material to new situations.
Professor Ornik highlighted the challenge faced by AI in responding to ‘out-of-distribution’ tasks, which are significantly different from previously encountered tasks. The study emphasized the need for more training data to improve AI’s performance on such complex issues.
Broader Implications
While AI technology shows promise in certain domains, the study also raised concerns about its impact on the labor market. Professor Ornik mentioned shifts in programming and software development jobs due to AI automation but emphasized that fields tied to human experience and emotions, such as humanities and art, remain beyond AI’s capabilities.
As more students and professors admit to using AI for course work, concerns about the implications of AI in education continue to grow.
