Google's AI overviews are still not to be trusted | heise online
Since the spring, Google searches have been spitting out AI overviews in more and more countries. But the hallucinatory oddities are only partially diminishing.
Google's AI Overviews: Generated AI image from search results – presented as if it were real.
Image: Todd Fong / Midjourney / heise online
On the plane to Tokyo, I want to find out what options there are for getting from Narita Airport to the city by train. The airplane WLAN seems to classify me as a US citizen – or at least as someone who Google considers to be a person who can be given access to the AI Overviews introduced in the first regions in the spring. So when I ask for an "airliner train" (because that's how I mistakenly remembered the term "Skyliner") in the Japanese capital, the AI overview created by Google's generative Gemini system, which is supposed to summarize web results as intelligently as possible, appears right at the top.
What it says (see screenshot) sounds pretty good: there are "different ways" to get to Tokyo by "airliner train", including the "Skyliner", the "Narita Express" and the "Jodan Skyflyer Ultra Express". Below this, there are a few statements including links to what the different trains mean.
While I am familiar with the "Skyliner" and "Narita Express" as competing means of rail transportation, the "Jodan Skyflyer Ultra Express" makes me wonder. Has the airport launched a new special train? The name sounds very Japanese. So I do what I always do – and google it. As it turns out, the term apparently only appears on one website: the blog of a Japan fan called Todd Fong, who gives travel tips and presents his photos.
In the blog post, he describes a trip on a "flying train" of the same name in a short story-like text. "For anyone who has been living under a rock for the last few months: The Skyflyer Ultra Express is the world's first public transportation system that flies from station to station. It's basically a flying train that travels to Tokyo, Takanawa Gateway, Kawasaki and Yokohama stations." The story is rounded off with some relatively pretty mid-journey AI images, which Fong uses to accompany his post. It is only in the last paragraph that you can read what the story is really about: "Can we be honest here? The story you have just read is part of my "Illusions of Japan" series. These are works that are part true and part fiction – about people, places and events in Japan." In other words: Google's AI overview has fallen into the fake trap, or rather does not seem to "read" texts to the end.
Image: Screenshot / heise online
The confusing thing about this case is the fact that the AI overviews not only missed completely obvious signs of fiction (more than a self-outing is not possible), but also seem to give a high trust score to almost unknown websites. The fact that the search term obviously only appears once in the index (at least in the context of my Google cookie personalization) also casts doubt on the algorithm.
Google's Approach with AI Overviews
It was probably smart of Google to initially reduce the display of AI overviews after their introduction in the summer. But you shouldn't make the mistake of thinking that the company will abandon the function. It is too lucrative for that. In the search business, it has a promising future because it allows users to stay on Google itself for longer instead of accessing the web via a direct link from the search engine, as was previously the case.
Even before the Gemini era, Google had been integrating more and more content directly into its own offerings under and above its own advertising over the last decade. Weather or exchange rates? Google does it itself. News? Are summarized. Cooking recipes or song lyrics? Sure, we'll give them to you straight away.
According to studies, AI overviews together with Google's own content already account for up to 75 percent of page content on mobile devices and around 67 percent on desktops. It is increasingly unlikely that many people will continue to scroll to the so-called organic search results.
Challenges with Generative AI
Generative AI is known to work with probabilities. This results in a complex network of language understanding that not even many AI researchers understand because it is far too complicated internally. It is clear that you can never trust the output 100 percent, because you can train it as much as you like, hallucinations will still occur. This is due to the basic system of generative AI.
For its AI overviews, which are a critical application, Google therefore uses Retrieval Augmented Generation (RAG), in which the generative AI can make use of current search data. However, these links are often not clicked or refer to untrustworthy content.
Future of AI Overviews
There was much rejoicing in the AI scene about Google's clever move to sign an exclusive contract with Reddit for training and RAG, but even this partnership had its challenges. Google boss Sundar Pichai is already promising even more AI in search next year. The AI Overviews have quickly become one of the most popular search functions of all time, reaching a billion people.
Concerns and Conclusion
There is already a certain counter-current among users and professional content creators regarding the reliability of AI overviews. It is essential to address the challenges and limitations of generative AI to avoid misinformation and accidents that may arise from trusting unverified content.
As we move forward with AI technology, it is crucial to prioritize accuracy and reliability to ensure the safety and integrity of information shared online.
