Elvis S. on LinkedIn: Nice report on challenges in evaluating LLMs. It...
Nice report on challenges in evaluating LLMs. It also includes a section on best practices for language model evaluation. Great read and lessons on the very difficult task of LLM evaluation.
To read the full report, you can visit here.
Agent Planning with World Knowledge Model
This report introduces a parametric world knowledge model to facilitate agent planning. The agent model can self-synthesize knowledge from expert and sampled trajectories to train the world knowledge model. Prior task knowledge is used to guide global planning and dynamic state knowledge is used to guide local planning. The report demonstrates superior performance compared to various strong baselines when adopting open-source LLMs like Mistral-7B and Gemma-7B. Augmenting LLM-based agents with a world knowledge model to enable planning is an interesting idea that also helps reduce hallucination and invalid actions commonly associated with language agents. This concept aligns with Yann LeCun's recent comments on the necessity of deeper exploration of world models to enhance current AI systems' reasoning and planning abilities.
For more details, the paper can be accessed here.
The Prompt Engineering Guide reached 4M Visitors!
The Prompt Engineering Guide has reached a significant milestone by welcoming 4 million visitors within a year. The guide now includes advanced prompting techniques and covers topics like LLM-based agents and RAG. Additionally, longer video tutorials for each section of the guide have been introduced. The project is continuously evolving to align with the advancements in the field and has now garnered 45K stars on GitHub.
For more information, you can visit the GitHub page here or the project website here.
# Experimenting with Long-Context LLMs
Long-context LLMs provide immense utility with their flexibility, allowing for in-depth exploration of capabilities. The writer shares thoughts on long-context LLMs and their practical applications. Further insights on this topic are available here.
![QA] Does Fine-Tuning LLMs on New Knowledge Encourage ...](https://i.ytimg.com/vi/17SQs1dyKfE/hqdefault.jpg)
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
An intriguing report discusses the impact of fine-tuning LLMs with new knowledge on their tendency to hallucinate. The report suggests that LLMs struggle to acquire factual knowledge through fine-tuning, leading to an increase in hallucinations when exposed to new information. This concept is further elaborated in the LLM recap, which can be viewed here.
Enhancing Answer Selection in LLMs
A proposed hierarchical reasoning aggregation framework, called Aggregation of Reasoning (AoR), aims to enhance the reasoning capabilities of LLMs. AoR focuses on evaluating reasoning chains to improve the selection of final answers. This framework outperforms various ensemble methods and can be integrated with different LLMs to boost performance on complex reasoning tasks. The idea of self-assessing reasoning chains, prone to biases and incorrect assumptions, has led to significant advancements in LLMs.
Llama 3 From Scratch
A fascinating project that implements Llama 3 from scratch. The detailed breakdown provided in the readme serves as an excellent study resource to understand the main components of an LLM.
Scientific Applications of LLMs
INDUS, a comprehensive suite of LLMs designed for various scientific domains including Earth science, biology, and physics, is presented. The suite includes encoder models, embedding models, and small distilled models. Exploring applications of LLMs in specialized domains like these showcases the versatility and potential of these models.
