ChatGPT less than 50% Accurate in Accounting Exams, Finds Study
A global study conducted by the American Accounting Association (AAA) has found that accounting students’ performance is significantly better than ChatGPT in answering accounting questions. The study evaluated ChatGPT’s ability to perform on accounting-specific content by presenting it with over 25,000 assessment questions from 187 institutions worldwide. Then, researchers cross-referenced the results with the performance of accounting students on the same assessments. The study found that students scored an average of 76.7%, while ChatGPT scored only 47.4% across all assessments, including topics such as audit, financial accounting, management accounting, and tax. The study concludes that “This study provides important insights into the current capabilities of AI compared to human performance in an accounting-specific context. It highlights the limitations of an AI chatbot trained on general material.”
The results of the study raise concerns over the practical application of AI-powered chatbots, as while they possess ample knowledge, they lack practical skills that students have to excel in exams. Additionally, the study found that Chat GPT outperformed students on 11.3% of assessments and performed better on topics such as audit, true/false, and multiple-choice questions. But it struggled with short-answer questions and questions that require working out, highlighting the superiority of the human brain over AI in various disciplines.
However, the current gap between human and AI performance will likely close with further advancements in AI technology. According to the study, a model trained on one trillion parameters is expected to become available in 2023, which will lead to even more sophisticated AI models. Emma Rawson, technical officer at the Association for Accounting Technicians, warns that this is an area that the profession and professional bodies must keep an eye on and adapt accordingly.
The AAA study also encourages educators to prepare for an AI-powered future, warning of issues such as overreliance on technology hampering student learning ability and short-circuiting the learning process through cheating. The study acknowledged the potential power of ChatGPT in the world of accounting, arguing that it could provide the much-needed stimulus to reimagine accounting education practices.
