ChatGPT is no CPA: Popular chatbot can't pass accounting test
Recent studies reveal that ChatGPT (Chatbot Generative Pre-trained Transformer), the revolutionary AI technology is not an accountant. Despite its remarkable accomplishments and cutting edge technology, it has trouble understanding math and makes up facts to cover up its mistakes. Researchers indicate that ChatGPT has limitations in its ability to understand and compute accounting principles and questions. These findings reveal opportunities to improve the chatbot and reshape how professors approach teaching accounting.
ChatGPT's Performance in Accounting Exams
The research focused on accounting exams, and it was conducted in 186 educational institutions in 14 countries. Participants made up 25,181 accounting exam questions covering different accounting fields such as accounting information systems, auditing, financial accounting, managerial accounting, and tax. The questions were presented in different formats, including true/false, multiple choice, and written responses. The AI technology was also fed 2,268 textbook test bank questions by undergrad students at BYU.
Students scored higher than ChatGPT, which scored 47.4%, while students averaged 76.7%. The bot's performance scores varied depending on the questions. ChatGPT performed reasonably well, scoring above students in 11.3% of tasks, particularly in auditing and accounting information systems. However, the AI technology performed poorly in tax, financial, and managerial assessments. The study found that ChatGPT had trouble with the math required for these questions, such as random errors in adding and subtracting numbers or dividing figures incorrectly.
ChatGPT Test Results Depend on Question Type
The study found that ChatGPT performed better on true/false questions, with 68.7% correct answers, and multiple-choice questions, with 59.5% correct responses. ChatGPT did not perform as well in short-answer questions, scoring between 28.7% and 39.1%.
To compensate for its lack of knowledge, ChatGPT used authoritatively written explanations for incorrect answers or provided the same answer in different ways. However, the study revealed that relying solely on ChatGPT to learn accounting is a "fool's errand."
Limitations of ChatGPT and Its Potential
ChatGPT is relatively new, and AI updates can help it better understand and answer future accounting-related questions. The study authors indicated that the chatbot can reshape how professors approach teaching and learning of accounting principles. The revolution of ChatGPT in the accounting field is an opportunity to assess the value-added information for students in learning accounting and creating awareness that ChatGPT is not a substitute for human accountants.
It's worth mentioning that the AI program admits its limitations concerning generating incorrect or nonsensical answers, being sensitive to input phrasing, being excessively verbose, or failing to ask clarifying questions for ambiguous queries. The developers of ChatGPT are continually working on improving the model to make it more effective and safer for public use.
