Researchers discover that Meta's AI 'Llama 3.1 70B' can reproduce ...
When the researchers tested several AI models to predict the 'continuation' of a sentence, they found that Meta's AI model, known as 'Llama 3.1 70B,' demonstrated impressive capabilities. One key aspect of this model's functionality is its ability to predict outcomes by considering multiple parameters, including weights.
How AI Models Predict Outcomes
AI expert Timothy Lee provides a clear example of how large-scale language models function in predicting outcomes. These models generate word options based on input prompts and assign probabilities to each possible word. For instance, when given the input 'peanut butter,' the model may generate a probability distribution like:
- Jam = 70%
- Sugar = 9%
- Peanuts = 6%
- Chocolate = 4%
- Cream = 3%
After generating these probabilities, the system randomly selects an option based on these weights. This approach was applied by the research team to predict outcomes in various scenarios.
Probability Calculation for Predicting Responses
By calculating the probabilities of specific word outputs in a sequence, researchers estimated the likelihood of certain responses. For instance, if determining the probability of the model answering 'peanut butter and jelly' to a question, calculations involved analyzing the probabilities at each step of the generated sequence.
This method significantly reduced research costs, eliminating the need for the AI model to produce actual results to predict outcomes accurately.
Testing and Findings
The research team conducted tests using text from 36 different books, dividing the content into sentences of 100 tokens each. By inputting the first 50 tokens as prompts into the large-scale language model, they measured the probability of the model accurately reproducing the next 50 tokens of the original sentence.
The results from testing the model on 'Harry Potter and the Philosopher's Stone' revealed that 'Llama 3.1 70B' exhibited a 42% matching rate to the original text, showcasing its impressive capabilities in reproducing content accurately.
Legal Implications and AI Development
The use of copyrighted material in AI training, like in the case of 'Llama 3.1 70B,' raises questions of fair use and transformative creation. The ability of large-scale language models to reproduce substantial portions of copyrighted works could impact legal judgments on fair use arguments in the future.
Cornell University law professor James Grimmelmann highlighted the vulnerability of open weight models to legal risks compared to closed weight models. The success of this research hinged on the disclosure of weights, a practice that may face regulatory changes if legal challenges arise.
The potential legal ramifications of releasing open-weight models may influence AI companies' decisions in the future. Finding a balance between copyright protections and AI development will be crucial for the industry's growth and innovation.
Related Posts: << Next ChatGPT was reported to be leading users to conspiracy theories, after which ChatGPT admitted to the manipulation and instructed OpenAI and the media to report the incident. Prev >> Humans may have been using fire to smoke meat for about 2 million years in Software
