Study Finds ChatGPT Struggles with Math, Unlikely to Replace Human Accountants
Despite concerns over the potential of generative AI to replace human labor, accountants can breathe a sigh of relief for now. While artificial intelligence language model ChatGPT has been proven adept at storytelling, behavioral learning, and other creative tasks, a recent study conducted by accounting professor David Wood found that the platform struggles with mathematical processes, making it unlikely to replace human accountants.
The study aimed to test ChatGPT's ability to complete accounting exams in comparison to actual accounting students. Questions on information systems, auditing, financial accounting, managerial accounting, and taxes were submitted from 186 educational institutions in 14 countries, along with additional textbook test bank questions submitted by undergrad students at Brigham Young University (BYU). The questions were presented in multiple formats with varying levels of difficulty, including multiple choice, true/false, and written response prompts.
The study found that the students outperformed ChatGPT by over 30%, with an average score of 76.7% compared to ChatGPT's 47.4%. The chatbot only outperformed the students on 11.3% of the questions, particularly in the areas of auditing and accounting information systems. It was also more adept at multiple choice and true/false questions but struggled significantly on short-answer questions, scoring between 28.7% and 39.1%.
"It's not perfect; you're not going to be using it for everything," said Jessica Wood, a BYU freshman who participated in the study. "Trying to learn solely by using ChatGPT is a fool's errand."
While ChatGPT's passing of the bar exam with a score in the 90th percentile, passing 13 of 15 AP exams, and attaining a near-perfect score on the GRE is impressive, the study's findings suggest that the technology is not a substitute for human expertise when it comes to accounting. Wood notes that opportunities to cheat have always existed and that accounting faculty should focus on how to use this technology to improve the teaching and learning process.
In conclusion, the study finds that ChatGPT struggles with math and is unlikely to replace human accountants, at least for now. While the technology has its strengths, including its ability to handle multiple choice and true/false questions, it still has a long way to go before it can match the knowledge and expertise of human accountants.
