Meta's latest artificial intelligence (AI) giant model (LLM) Lamar 4...
Meta's latest artificial intelligence (AI) giant model (LLM) Lamar 4, unveiled on the 5th (local time), has been embroiled in suspicions of inflating performance. As rumors spread, Meta began to evolve, saying that the theory of performance manipulation was not true.
Overview of Lamar 4
Lamar4 is Meta's latest LLM. According to Meta's description, Lamar4 is divided into Behemoth, Maverick, and Scout. Meta has unveiled its two models, Maverick and Scout, to LM Arena, a platform that compares and evaluates various LLM performances with open source, with the exception of Behemoth currently undergoing learning.
Behemoth is a giant model with 2 trillion parameters, boasting the most parameters of any llama ever unveiled. This is more than the 1.7600 of OpenAI's GPT 4, which was launched in March 2023. Maverick and Scout are known to have been created using Behemoth. Maverick is known to be superior to GPT-4o and Gemini 2.0 Flash. Meta explained that the small model scout was optimized for inference processing and achieved superior performance compared to Gemma 3 and Gemini 2.0 Flashlight.
Controversies and Responses
After Meta's disclosure, some AI researchers and developers began to raise doubts about Lamar's performance. Meta's revealed Maverick and developers' actual available Maverick have seen a "clear" performance difference.
According to TechCrunch, rumors that Meta inflated the performance of Lamar 4 were first raised on Chinese social media. In a user's post that he quit the company in protest of Meta's inflated benchmark performance, the user argued that it was the same reason that Joel Pinot, the general manager of Meta AI research, resigned on the 2nd.
Some developers pointed to the difference between the Maverick seen on their X and the version they downloaded and used directly from the LM Arena. The LM Arena version uses more emojis and gives lengthy answers. He also pointed out that if the performance verification model and the actual distribution model are different, it may be difficult to accurately predict the performance, which may cause difficulties for developers.
As the controversy spread, Meta rushed to evolve. Ahmad Aldale, Meta Generating AI's general manager, emphasized, "I've also heard the claim that I trained (LLM) on the test set, which is not true, and Meta won't do that either." A test set is a dataset used for performance evaluation, and if the suspicion is true, it is to solve the problem by looking at the test paper in advance.
