Introducing HELMET: Holistically Evaluating Long-context Language Models

Back to Articles Introducing HELMET: Holistically Evaluating Long-context Language Models Published April 16, 2025 Update on GitHub Upvote 42 +36 Howard Yen hyen Follow princeton-nlp Tianyu Gao gaotianyu1350 Follow princeton-nlp Minmin Hou houminmin Follow Intel Ke Ding kding1 Follow Intel Daniel...

Back to Articles Introducing HELMET: Holistically Evaluating Long-context Language Models Published April 16, 2025 Update on GitHub Upvote 42 +36 Howard Yen hyen Follow princeton-nlp Tianyu Gao gaotianyu1350 Follow princeton-nlp Minmin Hou houminmin Follow Intel Ke Ding kding1 Follow Intel Daniel Fleischer danf Follow Intel Moshe Wasserblat moshew Follow Intel Danqi Chen cdq10131 Follow princeton-nlp Evaluating long-context language models is challenging but important Existing evaluations overly rely on synthetic tasks Crafting diverse, controllable, and reliable evaluation for LCLMs Key improvements over existing benchmarks LCLMs still have a long way to go on real-world tasks Diverse evaluation is needed for assessing long-context abilities Models degrade with increasing lengths and task complexity Using HELMET for future developments How to run HELMET Faster development Quick comparison with existing models Looking ahead Acknowledgements Citation Contact: hyen@cs.princeton.edu Paper: https://arxiv.org/abs/2410.02694 Website: https://princeton-nlp.github.io/HELMET Code & Data: https://github.com/princeton-nlp/HELMET Since we first released HELMET last October, there has been more development on long-context language models than ever before, and we are thrilled to see the adoption of HELMET by the community, such as Microsoft's Phi-4 and AI21's Jamba 1.6. After the initial release, we have added more models to our evaluation suite and conducted additional analyses. We are excited to share our new results and present HELMET at ICLR 2025! In this blog, we will describe the construction of HELMET, our key findings, and how practitioners can use HELMET to differentiate between various LCLMs in future research and applications. Finally, we will conclude with a quickstart guide for using HELMET with HuggingFace. Evaluating long-context language models is challenging but important From summarizing numerous legal documents to learning new tasks on the fly, long-context language models (LCLMs) have immense potential to change the way we use and interact with language models. Language models have been limited by their context window, which is around 2K to 8K tokens (e.g., ChatGPT, Llama-2/3). Recently, model developers have been constantly increasing the context window of their models, with recent models like GPT-4o, Claude-3, and Gemini-1.5 supporting context windows of up to millions of tokens. Figure 1: Existing benchmarks show counterintuitive trends, such as smaller models outperforming larger ones (e.g., Llama-3.1 8B > 70B). However, with longer context windows, previous natural language benchmarks (e.g., Scrolls) are no longer suitable for evaluating LCLMs. Consequently, perplexity and synthetic tasks (e.g., needle-in-a-haystack) emerged as the most popular evaluation metrics for recent LCLMs, but they often do not reflect real-world performance. Model developers may also evaluate on other arbitrary datasets, which complicates model comparisons. Furthermore, existing benchmarks for LCLMs may show confusing and counterintuitive results, making it difficult to understand the strengths and weaknesses of different models (Figure 1). In this work, we propose HELMET (How to Evaluate Long-Context Models Effectively and Thoroughly), a comprehensive benchmark for evaluating LCLMs that improves upon existing benchmarks in several ways—diversity, controllability, and reliability. We evaluate 59 recent LCLMs and find that it is crucial to evaluate models across diverse applications to understand their capabilities and frontier LCLMs are still limited on complex tasks. Existing evaluations overly rely on synthetic tasks With the development of LCLMs across both industry and the open-source community, it is crucial to have a reliable method for evaluating and comparing these models. However, current models are often evaluated on different benchmarks (Table 1). Table 1: Model developers often evaluate on different sets of datasets. ♭: Base models. NQA: NarrativeQA, Qspr: Qasper, QALT: QuALITY, SQALT: SQuALTY. A common practice for evaluating long-context language models is to use perplexity or synthetic tasks, such as needle-in-a-haystack (NIAH). However, recent works have shown that perplexity does not correlate well with downstream performance (Fang et al., 2024). In Figure 2, we show that synthetic tasks like NIAH do not correlate with real-world performance, but the more complex synthetic tasks achieve higher correlation with real-world tasks. Figure 2: Simple synthetic tasks, such as NIAH, do not correlate well with downstream tasks, such as summarization or generation with citations. More complex variants (e.g., RULER MV) achieve higher correlation. Among the existing benchmarks with realistic applications, such as ZeroScrolls (Shaman et al., 2023), LongBench (Bai et al., 2024), and InfiniteBench (Zhang et al., 2024), there are still crucial limitations: Insufficient coverage of downstream tasks: often focused on specific domains Inadequate lengths for testing frontier LCLMs: older QA datasets are often limited to

Source: Hugging Face — Published April 16, 2025 at 00:00 UTC — Category: Open Source