Students Outperform ChatGPT in Accounting Exams
A recent study conducted by researchers from Brigham Young University (BYU) and 186 other universities found students to have fared better at accounting exams than ChatGPT, OpenAI's chatbot product. Despite this, the researchers' assessment of ChatGPT's performance was positive, noting that it has the potential to revolutionize the way people learn and teach.
In the study, students achieved an overall average of 76.7%, surpassing ChatGPT's score of 47.4%. While ChatGPT outperformed the student average in 11.3% of the questions, it struggled with tax, financial, and managerial assessments, possibly due to mathematical challenges. ChatGPT performed well on accounting information systems and auditing.
According to the study, ChatGPT did better on true/false questions (68.7%) and multiple-choice questions (59.5%) but had difficulty answering short-answer questions (28.7% to 39.1%). The researchers noted that ChatGPT encountered difficulty with higher-order questions and sometimes provided incorrect answers while providing authoritative written descriptions.
The study showed that ChatGPT often provided explanations for its answers, even if they were incorrect. In some cases, it selected the incorrect multiple-choice answer despite providing correct descriptions. The researchers also flagged concerns about ChatGPT making up facts and generating nonsensical mathematical errors, such as adding instead of subtracting or dividing numbers incorrectly.
To better understand how models like ChatGPT could factor into education, lead study author David Wood, a BYU professor of accounting, recruited over 300 co-authors from 186 educational institutions in 14 countries for the research. They contributed 25,181 classroom accounting exam questions. Additionally, undergraduate BYU students provided ChatGPT with 2,268 textbook test bank questions covering AIS, auditing, financial accounting, managerial accounting, and tax.
The study highlights an ongoing debate about how AI models like ChatGPT should be used in education. While the study shows that students still outperform AI in accounting exams, the researchers believe the technology has great potential to enhance education.
