In short: Ai2 (Allen Institute for AI) published a new analysis about how to decide which benchmarks actually help when you compare language models. The main idea? Not every test is equally useful when you run small-scale experiments and then scale up, and there’s a simple way to measure that reliability. (allenai.org)

What Ai2 announced

Ai2 introduces the idea of measuring the relationship between signal and noise to evaluate which benchmarks are reliable for design and scaling decisions. They published a blog post with the results, the data (900,000 evaluations) and the code so others can reproduce it. (allenai.org)

What is SNR and why does it matter?

SNR means signal-to-noise ratio. Here: signal is how well a benchmark separates models from each other (that is, how much useful spread there is in scores), and is the random variability that appears during training and can hide those differences.