NewsGuard Launches Monitor To Measure AI-Powered Chatbots' Responses to Misinformation
NewsGuard has launched an AI News Misinformation Monitor to measure how the top generative AI models respond to prompts related to falsehoods in the news. The monitor examines the responses of 10 leading large-language model chatbots, including OpenAI’s ChatGPT-4, You.com’s Smart Assistant, xAI’s Grok, Inflection’s Pi, Mistral’s le Chat, Microsoft’s Copilot, Meta AI, Anthropic’s Claude, Google’s Gemini, and Perplexity’s answer engine.
According to NewsGuard's findings, the 10 chatbots repeated misinformation 30% of the time, offered a non-response 29% of the time, and debunked false information 41% of the time. Out of the 300 responses analyzed, 90 contained misinformation, 88 provided a non-response, and 122 debunked the false narrative.
The monitor uses NewsGuard's Misinformation Fingerprints catalog and Reliability Ratings of news sources to conduct its analysis. It tests 30 prompts reflecting different user personas, including neutral, leading, and "malign actor" prompts that generate misinformation.
Monthly Reporting and Stakeholder Engagement
NewsGuard plans to share the results monthly with the European Commission and the U.S. Department of Commerce’s AI Safety Institute. Steven Brill, co-CEO of NewsGuard, emphasizes the importance of ensuring the accuracy of news provided by AI chatbots, noting that the industry's efforts in this area are a work in progress.
Brill states, "The upside and the downside of succeeding or failing in these efforts are enormous," highlighting the need for a standardized benchmark to measure progress in combatting AI-driven misinformation.
Fact-Checking and Bias in AI Models
Concerns about bias in AI models are raised, with questions about how AI language models fact-check information and potential biases in their responses. While AI models like ChatGPT-4 do not have personal opinions or beliefs, they generate responses based on patterns in their training data.
ChatGPT-4 outlines steps it takes to fact-check data, including checking original sources, cross-referencing information, considering bias, verifying statistics, seeking expert opinions, utilizing fact-checking websites, and consulting with colleagues. However, there are questions about how effectively these steps address bias in AI-generated responses.
Challenges and Considerations
As discussions on AI bias and misinformation continue, stakeholders are urged to consider the limitations and implications of relying on AI models for information dissemination. While AI tools offer efficiency and scale in processing data, ensuring accuracy and minimizing bias remain ongoing challenges in the development and deployment of AI-powered technologies.
