Study Finds ChatGPT Struggles with Math, Unlikely to Replace Human Accountants
Despite its reputation for being adept at storytelling, behavioral learning, and other creative tasks, the recent study conducted by Brigham Young University (BYU) found that ChatGPT, the fastest-growing and most prominent AI language platform, often struggles with mathematical processes, which makes it unlikely to replace human accountants anytime soon.
The study, led by accounting professor David Wood, sought to test ChatGPT's aptitude in completing accounting exams in comparison to actual accounting students. The researchers submitted 25,181 questions on information systems, auditing, financial accounting, managerial accounting, and taxes from 186 educational institutions in 14 countries. They also fed 2,268 more textbook test bank questions into the repository. The questions were presented in different formats with varying levels of difficulty, including multiple choice, true/false, and written response prompts.
According to the results of the study, the students outperformed ChatGPT by over 30%, scoring an average of 76.7% to ChatGPT's 47.4%. The AI language model only outperformed the students on 11.3% of the questions, particularly those concerning auditing and accounting information systems. The chatbot also performed better on multiple choice and true/false questions, scoring 59.5% and 68.7%, respectively. However, it underperformed significantly on short answer questions, scoring only between 28.7% and 39.1%.
"It's not perfect; you're not going to be using it for everything," said Jessica Wood, a BYU freshman who participated in the study. "Trying to learn solely by using ChatGPT is a fool's errand."
Despite concerns about ChatGPT's potential for aiding students in cheating, its struggles with math suggest a limited range of applications in the accounting field. Human accountants possess a breadth of knowledge and intuition that AI language platforms like ChatGPT cannot replicate. Thus, rather than being a threat, the study suggests that ChatGPT can serve as a tool to improve the teaching and learning process for faculty and students.
