LiveBench is an open LLM benchmark using contamination-free test data
Are you ready to honor the amazing women who are making significant strides in the field of AI? Nominate the inspiring leaders for VentureBeat’s Women in AI Awards today before June 18. Learn More
Introducing LiveBench
A team consisting of Abacus.AI, New York University, Nvidia, the University of Maryland, and the University of Southern California has unveiled a groundbreaking new benchmark known as LiveBench. This benchmark addresses the limitations present in existing benchmarks by providing contamination-free test data. Unlike traditional benchmarks, LiveBench offers a general-purpose LLM benchmark that ensures test data remains uncontaminated, preventing models from being artificially inflated in performance due to exposure during training.
What is LiveBench?
LiveBench serves as a standardized test designed to evaluate the performance of AI models, particularly Large Language Models (LLMs). It offers a set of tasks and metrics against which LLMs can be measured, allowing researchers and developers to benchmark their models effectively. The benchmark incorporates a diverse range of challenging tasks spanning various categories such as math, coding, reasoning, language, instruction following, and data analysis.
Key Features of LiveBench
The LiveBench benchmark integrates frequently updated questions sourced from recent data, automatically scoring answers based on objective ground-truth values. With 960 questions currently available and newer, more challenging inquiries being released monthly, LiveBench ensures a continuous flow of fresh test data to minimize the risk of contamination.
The Impact of LiveBench
LiveBench has been instrumental in evaluating numerous closed-source and open-source models ranging from 500 million to 110 billion tokens in size. The benchmark has revealed that even the top-performing models achieve less than 60 percent accuracy, highlighting the need for robust evaluation standards in the AI industry.
Comparing LiveBench to Existing Benchmarks
LiveBench's creators have compared its performance with established LLM benchmarks such as Chatbot Arena and Arena-Hard, showcasing similar trends in model evaluation. However, LiveBench emphasizes the importance of minimizing biases and ensuring a fair assessment of model capabilities.
Accessing LiveBench
Developers can access LiveBench's code on GitHub and its datasets on Hugging Face. This open-source benchmark is available for anyone to use and contribute to, with plans for ongoing updates and expansions to enhance LLM evaluation capabilities.
