For years the community has asked: can large language models do real science, not just answer textbook questions? Anthropic published a technical study that looks for that answer with BioMysteryBench, a benchmark designed for complex bioinformatics tasks on real data.
Qué es BioMysteryBench y por qué importa
BioMysteryBench is a set of 99 bioinformatics problems created by experts from real or minimally processed data (WGS, scRNA-seq, metagenomics, ChIP-seq, Hi-C, methylation, proteomics, and metabolomics). Each question comes with a validation notebook that shows the signal exists in the data, even if finding it from scratch can be hard.
The key idea is to measure research tasks that reflect real work: reading data, installing tools (pip, conda), querying databases like NCBI and Ensembl, writing and running analyses, and justifying conclusions. It’s not just about knowing the answer; it’s about reproducing the scientific process.
Diseño del experimento y métricas técnicas
- 99 questions in total; 76 were solved by at least one human and are therefore considered human-solvable. 23 were marked human-difficult after quality controls (4 questions were withdrawn because of formulation issues).
- Each problem was attempted five times by each model to measure not only accuracy but robustness. A problem solved 5/5 indicates a reproducible strategy; 1/5 suggests a fragile path or luck.
- Models were run inside containers with access to canonical tools and the ability to install additional software and access online resources.
This design lets you evaluate two dimensions: capability (can it get the answer?) and reliability (does it get it consistently?).
Resultados clave
- Recent generations of Claude improve consistently. Models like Opus 4.6, Opus 4.7 and Mythos show jumps in performance and robustness.
- Mythos solved up to 30% of the problems the human panel couldn’t solve. That doesn’t mean it’s always right, but it does find signals humans missed.
- For Opus 4.6: on the human-solvable questions, 86% of the problems the model manages to solve are solved at least 4/5 times. In the human-difficult set, that percentage falls to 44% and fragile wins (solves 1-2 out of 5) rise from 9% to 44%.
- Sonnet 4.6 shows a similar drop: 75% → 22% in reliably solved problems, and an increase in fragile wins.
- Mythos improves reliability on the “solvable” set: it gets 94% of its hits as reliable (>=4/5) in that group.
Numerical conclusion: overall accuracy doesn’t tell the whole story. The difference between solving something always and solving it sometimes is the most revealing narrative.
Cómo resuelve Claude: dos estrategias interesantes
-
Aggregated knowledge: Claude leverages its structural and meta-analytic knowledge base (information learned from hundreds of thousands of papers and resources) to tie together pieces that a human would have to obtain through meta-analysis or manual searches.
-
Layered approach: when the model is uncertain, it tries multiple methods and looks for convergence across lines of evidence. It’s a kind of in-silico experiment: it tests alternative routes and combines results.
Sometimes these strategies let it solve tasks a human panel could not. Other times, the model’s prior knowledge introduces biases and errors when its pretraining information doesn’t apply exactly to the observed data.
Ejemplos y lecciones prácticas
- Claude can find sequence patterns or signals that the original human strategy didn’t detect. That’s reminiscent of historical discoveries where a repeated observation revealed a biological function (like the TATA sequence in promoters).
- In some cases, the model’s prior knowledge worked against it: if the model trusts incorrect prior information too much, its result can fail on concrete datasets.
Should you use Claude for bioinformatics research today? Yes, but as an assistant: it speeds up searching, tests hypotheses and suggests routes. It doesn’t replace experimental validation or human expert judgment.
Limitaciones y riesgos técnicos
- For questions that neither humans nor models solve, there’s no way to know if the problem is impossible or just very hard. The validation notebooks reduce but don’t eliminate this uncertainty.
- Reliability on hard problems remains a challenge: many hits in the human-difficult set are fragile (1-2 of 5), which means the reproducibility of the reasoning path isn’t guaranteed.
- Pretraining and model biases can lead to false positives; you must always review and validate with additional analyses and, when possible, experiments.
Cómo encaja esto con otros esfuerzos
While Anthropic published these results, Genentech and Roche launched CompBioBench, a similar benchmark of 100 tasks designed with synthetic and augmented data to have exact answers. Their results agree: Claude Opus 4.6 reaches high figures (81% overall, 69% on difficult questions). The convergence across teams suggests frontier models are already useful collaborators in bioinformatics.
Reflexión final
BioMysteryBench represents an important step: it moves from exam-like tests to measuring real workflows in computational science. Models like Claude no longer just answer; they investigate, combine evidence and, in some cases, uncover signals that are hard for humans.
Does this mean AI will replace researchers? Not yet. It means AI is an increasingly capable tool that can speed up analysis, suggest alternative routes and maybe be the first to solve some puzzles. The responsibility remains human: designing experiments, validating results and deciding what to follow up in the lab.
Fuente original
https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench