Bots Battle for Supremacy in Legal Reasoning – Part Five: Reigning ...
The champion of legal reasoning was determined to be Orion ChatGPT-4.5 in March 2025. This was reported in Part Three of the Battle of the Law Bots series. It defeated prior champ, Omni ChatGPT-4o. It had been AI legal reasoning champion of the world for a month after defeating several other Google Gemini and OpenAI models. This was reported in Parts One and Two of the series. Then 4.5 Orion was released in March and defeated Omni in reasoning tests.
The Battle Continues
The next month, April 2024, ChatGPT-o3 was released. I pitted o3 against 4o expecting Omni to win. Surprisingly, little o3 defeated Omni in a convincing manner. This was reported in Part Four. This win qualified ChatGPT-o3, which I nicknamed Scorpio, to challenge Orion. That is what we do here in Part Five of the AI legal reasoning bot battles.
Meet the Contenders
OpenAI in late April 2025 described the current champ, Orion, as follows: GPT-4.5 is OpenAI’s latest and most advanced language model, introduced as a research preview. It emphasizes enhanced pattern recognition, creative insight generation, and emotional intelligence, aiming to deliver more natural and reliable responses. GPT-4.5 is available as a research preview to Plus, Pro, and Team users.
OpenAI described GPT-o3, Scorpio, in this manner: GPT-o3 is designed to handle complex reasoning tasks with enhanced capabilities. o3 excels in step-by-step logical reasoning, making it adept at solving intricate problems in mathematics, science, and programming.
The Mythological Showdown
According to legend, Orion’s hunting prowess led to his downfall when he faced the scorpion. Just like in Greek mythology, Orion and Scorpio are set to engage in a battle for supremacy in legal reasoning, as envisioned in the AI bot battles.
Although I admittedly favor little Scorpio, I have gone out of my way to keep this a fair fight as I will explain next. First of all, to make sure neither had any inside information and this was a closed book exam. I picked two Bar Exam essay questions from July 2024 Californian Bar Exam. One was a UCC sales type question involving baseball cards and another was an attorney ethics question involving settlement of an unrelated case. I made sure both models did not previously know these essay questions and could not browse or research. I also make sure neither had seen the model answers provided for each. The training of both models preceded the July 2024 test. I gave them both the same test and instructions (included with the questions) and the same the same general guidance instructions.
As an extra precaution, I gave them both the test at the same time to prevent internal instructions carryover or second taker advantage. (I used my OpenAI Team account and hit the send button simultaneously.) Of course, I only provided them with the model answers later when I asked for critiques. Again, I made very sure neither model had seen any answers prior to the test.
