Study finds ChatGPT struggles with math, unlikely to replace human accountants
The use of artificial intelligence (AI) language model ChatGPT has raised concerns about its potential to help students cheat on coursework and exam material. However, a recent study has shown that the platform struggles to understand mathematical processes, which may relieve accountants' concerns over job security.
The study, led by Brigham Young University accounting professor David Wood, sought to test ChatGPT's aptitude in completing accounting exams in comparison to actual accounting students. The study submitted 25,181 questions on information systems, auditing, financial accounting, managerial accounting, and taxes from 186 educational institutions in 14 countries. Furthermore, 2,268 more textbook test bank questions were fed into the study's repository by undergraduate students at BYU, presented in various formats with varying levels of difficulty.
The study found that students scored higher than ChatGPT, outpacing the chatbot by over 30%. The students scored an average of 76.7% while ChatGPT scored only 47.4%. The chatbot only outperformed the students on 11.3% of the questions, particularly on those concerning auditing and accounting information systems. ChatGPT was also more adept at answering multiple choice and true/false questions, scoring 59.5% and 68.7% on each respective format. However it underperformed significantly on short answer questions, only scoring between 28.7% and 39.1%.
Given this, ChatGPT is unlikely to replace human accountants. "It’s not perfect; you’re not going to be using it for everything," says Jessica Wood, a BYU freshman who participated in the study. "Trying to learn solely by using ChatGPT is a fool's errand."
