5 AI bots took our tough reading test. One was smartest — and it ...
All of the most popular artificial intelligence chatbots have the ability to upload and summarize documents, from legal contracts to an entire book. The tech promises to give you a kind of speed-reading superpower. But do any of the bots really understand what they’re reading?
To figure out which AI tools you can trust as a reading assistant, I held a competition. I challenged five bots to read four very different types of writing and then tested their comprehension. The reading spanned the liberal arts, including a novel, medical research, legal agreements and speeches by President Donald Trump.
To judge the AI tools’ summaries and analysis, I gathered a panel of experts — including the original authors of the book and scientific reports. All told, I asked 115 questions about the assigned reading to ChatGPT, Claude, Copilot, Meta AI and Gemini.
Some of the AI responses were astoundingly good. Others were so clueless they sounded like “Seinfeld’s” George Costanza. All the bots barring one made up — or “hallucinated” — information, a persistent AI problem. But facts were only one part of the challenge; my questions also challenged the AI to provide analysis, such as recommending improvements to the contracts and spotting factual problems in Trump’s speeches.
Full comparison:
A comparison of GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, Llama 4, and Copilot: Claude won overall, having the most consistent answers and no hallucinations. Read more here.
