Meta Likely to Release Llama 4 Early Next Year, Pushing Towards ...
AI models are getting better at reasoning — of sorts. OpenAI’s o1, for instance, has levelled up enough to earn a cautious nod from Apple. Meanwhile, Kai-Fu Lee’s 01.AI is also making waves with Yi-Lightning, claiming to outpace GPT-4o on reasoning benchmarks. With China’s models catching up fast, Meta is also stepping up Llama’s game.
The Big Question:
Can Meta bring Llama’s reasoning closer to the likes of GPT-4o and o1? Manohar Paluri, VP of AI at Meta, told AIM that the team is exploring ways for Llama models to not only “plan” but also evaluate decisions in real time and adjust when conditions change. This iterative approach, using techniques like ‘Chain of Thought,’ supports Meta’s vision of achieving “autonomous machine intelligence” that can effectively combine perception, reasoning, and planning.
Yann LeCun's Vision:
Meta AI chief Yann LeCun believes that autonomous machine intelligence or AMI, also known as “friend” in French, systems can truly help people in their daily lives. This involves developing systems that can understand cause and effect, and model the physical world. This might also be an alternative term for AGI or ASI, which OpenAI is obsessed with achieving.
Reasoning in AI:
Paluri highlighted that reasoning in AI, particularly in “non-verifiable domains”, requires breaking down complex tasks into manageable steps, which allows the model to dynamically adapt. For example, planning a trip involves not only booking a flight but also handling real-time constraints like weather changes, which may mean rerouting to alternative transportation.
Meta's Innovations:
Recently, Meta unveiled Dualformer, a model that dynamically switches between fast, intuitive thinking and slow, deliberate reasoning, mirroring human cognitive processes and enabling efficient problem-solving across tasks like maze navigation and complex maths.
Llama's Secret Sauce:
Meta said that it leverages self-supervised learning during its training to help Llama learn broad representations of data across domains, which allows for flexibility in general knowledge. RLHF (reinforcement learning with human feedback), which currently powers GPT-4o and majority of other models today, focuses on refining behavior for specific tasks, ensuring that the model aligns with practical applications.
Llama 4, When?
Meta CEO Mark Zuckerberg, in a recent interview with AI influencer Rowan Cheung, said the company has already started pre-training for Llama 4. Zuckerberg added that Meta has set up compute clusters and data infrastructure for Llama 4, expecting it to be a major advancement over Llama 3. Meta’s VP of product, Ragavan Srinivasan, at Meta’s Build with AI Summit, hinted at releasing “Next gen” Llama models by 2025.
Quantisation of LLMs:
Meta recently introduced quantized versions of its Llama 3.2 models, enhancing on-device AI performance with faster inference speeds, reduced model size, and decreased memory usage.
