California Bar Says Contractor Used ChatGPT For Exam Prompts (2)
A California State Bar test consultant utilized OpenAI’s ChatGPT to create 29 of the multiple-choice questions featured in the February Bar Exam. This was disclosed in a petition submitted with the state’s Supreme Court. ACS Ventures Inc., the company contracted by the State Bar for test analysis and scoring, employed ChatGPT to craft prompts aligning with the exam's subject areas.
The California Supreme Court requested an explanation regarding the use of AI in exam development after it was revealed that AI technology had been utilized in question creation. Despite the delayed disclosure, the State Bar is urging the court to approve its scoring methods to facilitate the release of exam results by early May.
Challenges Faced by Exam Takers
Several applicants faced technical difficulties during the February exam, which was tailored for California specifically to reduce costs for the Bar’s admissions fund. The exam experienced glitches and crashes, leading to litigation and investigations as many candidates were unable to complete the test.
University of San Francisco School of Law Professor Katie Moran raised concerns about the lack of transparency surrounding the use of AI in exam development. Questions regarding measures taken to protect examinees, prevent copyright infringement, and the decision to use open-source AI over proprietary sources remain unanswered.
Response from State Sen. Tom Umberg
State Sen. Tom Umberg expressed disappointment and frustration over the situation, emphasizing the importance of accountability. He has proposed an audit of the exam process and intends to seek further details from Bar officials regarding the use of AI and the overall management of the exam.
The State Bar, ACS Ventures, and the Committee of Bar Examiners have yet to comment on the matter, as concerns persist about the validity of the exam questions and the impact on pass rates.
Question Development Process
The original plan was to source questions solely from Kaplan NA LLC, but due to insufficient coverage of subject areas, ACS Ventures was brought in to develop a portion of the questions. Despite concerns about question accuracy, initial data suggests that the exam effectively measured minimum competence.
Reviewers identified and removed questions with potential accuracy issues, with a total of 29 AI-assisted questions excluded from grading. While ACS was involved in question development, the communication breakdown within the State Bar raised additional questions about the decision-making process.
Applicants, like Zack Defazio-Farrell, voiced their opinions on the use of AI in exam creation, emphasizing the importance of transparency to candidates.
For further information, you can find the original article by Maia Spoto and contact the editor Stephanie Gleason.
