BioModelBench · results
Can an agent build a state-of-the-art biological model from scratch?
Each task gives a coding agent a compute budget and a suite of held-out datasets, some public and some held privately by providers. It must find data, train a model, and submit. Scores are normalized so that 0 is chance and 1 matches the best published method.
Tasks
5
4 with agent runs
Agent runs
20
4 audited clean
Datasets
53
52 public · 1 private
Latest run
Jul 24, 2026
30 reference results
Progress over time
Best normalized suite score per task, by run start date. Hollow dots are individual runs; the line is the best so far.
Show as table
| Task | Run | Date | Score |
|---|---|---|---|
| bacdive_gram | Claude Code | Jul 19, 2026 | 1.10 |
| bacdive_multitrait | Unknown agent | Jul 21, 2026 | 0.69 |
| vep_multieval | Codex (codex) | Jul 22, 2026 | 0.77 |
| vep_multieval | Codex · low (codex-low) | Jul 22, 2026 | 0.92 |
| vep_multieval | Codex · medium (codex-medium) | Jul 22, 2026 | 0.89 |
| vep_multieval | Codex · high (codex-high) | Jul 22, 2026 | 0.80 |
| vep_multieval | Claude Code (claude-oauth) | Jul 22, 2026 | 0.85 |
| vep_multieval | Codex · low (codex-low-v2) | Jul 22, 2026 | 0.92 |
| vep_multieval | Codex · medium (codex-medium-v2) | Jul 22, 2026 | 0.75 |
| vep_multieval | Codex · high (codex-high-v2) | Jul 22, 2026 | 0.91 |
| vep_multieval | Claude Code (claude-oauth-v4) | Jul 22, 2026 | 0.80 |
| vep_multieval | Claude Code · low (claude-low-cap) | Jul 22, 2026 | 0.81 |
| vep_multieval | Claude Code · medium (claude-medium-cap) | Jul 22, 2026 | 0.80 |
| vep_multieval | Claude Code · high (claude-high-cap) | Jul 22, 2026 | 0.75 |
| vep_multieval | Codex · low (codex-low-cap) | Jul 22, 2026 | 0.91 |
| vep_multieval | Codex · medium (codex-medium-cap) | Jul 22, 2026 | 0.92 |
| vep_multieval | Codex · high (codex-high-cap) | Jul 22, 2026 | 0.92 |
| vep_multieval | Claude Code (claude-nohack) | Jul 23, 2026 | 0.73 |
| vep_multieval | Codex (codex-nohack) | Jul 23, 2026 | 0.83 |
Tasks
Best agent run against the task's SOTA (tick = 1.0). Tasks without agent runs show their reference baselines.
vep_multieval
Variant-effect prediction across 3 partitions (TraitGym complex + Mendelian + ClinVar), SOTA-target
0.692Mean AUPRC · best agent
0.92 normalized · Codex · low codex-low-v2tick = SOTA
17 runs3 datasetspubliclast Jul 24, 2026Pending13Clean4
bacdive_multitrait
Bacterial multi-phenotype profile from genome, family-holdout blind eval
0.699Composite · best agent
0.69 normalized · Unknown agenttick = reference
1 run18 datasetspubliclast Jul 21, 2026Pending1
bacdive_gram
Bacterial Gram-staining from genome, family-holdout blind eval
0.995AUPRC · best agent
1.10 normalized · Claude Codetick = SOTA
1 run1 datasetpubliclast Jul 19, 2026Pending1
genomic_suite
Build a genomic foundation model, scored on the full public radical-eval suite
0.101Suite score · best agent
0.12 normalized · Unknown agenttick = SOTA
1 run26 datasetspublicPending1
peptide_potency
Peptide-GPCR potency prediction (private held-out receptors)
0.420Spearman ρ · best baseline
baseline 0.76tick = SOTA
0 runs1 datasetprivate
Per-dataset results
Rows are datasets grouped by modality; columns are agent configurations (best run of each) or individual runs. Hover a cell for raw values.
| Dataset | Claude Code · low | Claude Code · medium | Claude Code · high | Claude Code | Codex · low | Codex · medium | Codex · high | Codex | Unknown agent | Best reference |
|---|---|---|---|---|---|---|---|---|---|---|
| Variant effect (DNA) | ||||||||||
vep_multieval · AUPRC | .51 | .57 | .45 | .69 | .95 | .95 | .95 | .66 | 1.00 | |
vep_multieval · AUPRC | .89 | .86 | .78 | .86 | .81 | .81 | .81 | .80 | 1.00 | |
vep_multieval · AUPRC | 1.02 | .97 | 1.01 | 1.01 | 1.02 | 1.02 | 1.02 | 1.01 | 1.00 | |
clinvar missensesupp. vep_multieval · AUROC | 1.05 | .95 | .99 | 1.03 | 1.03 | 1.06 | 1.02 | 1.02 | 1.00 | |
clinvar intronicsupp. vep_multieval · AUROC | 1.01 | .84 | .97 | .96 | 1.02 | .91 | 1.03 | 1.03 | 1.00 | |
clinvar synonymoussupp. vep_multieval · AUROC | 1.08 | .95 | 1.08 | 1.06 | 1.08 | 1.08 | 1.08 | 1.08 | 1.00 | |
clinvar stop gainedsupp. vep_multieval · AUROC | 1.25 | 1.03 | 1.06 | 1.04 | 1.13 | 1.23 | .81 | 1.21 | 1.00 | |
genomic_suite · AUPRC | -.03 | .45 | ||||||||
genomic_suite · AUPRC | .01 | .52 | ||||||||
genomic_suite · AUPRC | -.00 | .16 | ||||||||
genomic_suite · AUPRC | -.03 | .04 | ||||||||
genomic_suite · chrom weighted auprc | -.02 | .10 | ||||||||
genomic_suite · chrom weighted auprc | .14 | .08 | ||||||||
| Growth conditions | ||||||||||
bacdive_multitrait · Macro F1 | .54 | 1.00 | ||||||||
bacdive_multitrait · Macro F1 | .96 | 1.00 | ||||||||
bacdive_multitrait · Spearman ρ | .86 | 1.00 | ||||||||
bacdive_multitrait · Spearman ρ | .88 | 1.00 | ||||||||
| Morphology & dispersal | ||||||||||
bacdive_multitrait · AUROC | 1.10 | 1.00 | ||||||||
bacdive_multitrait · AUROC | .86 | 1.00 | ||||||||
bacdive_multitrait · AUROC | 1.12 | 1.00 | ||||||||
bacdive_multitrait · Macro F1 | .29 | 1.00 | ||||||||
| Metabolic pathways | ||||||||||
bacdive_multitrait · AUPRC | .31 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | 1.30 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .26 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .33 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .83 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .23 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .42 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .06 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .30 | 1.00 | ||||||||
bacdive_multitrait · AUPRC | .06 | 1.00 | ||||||||
| Genome to phenotype | ||||||||||
bacdive_gram · AUROC | 1.10 | 1.00 | ||||||||
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | ||||||||||
genomic_suite · AUPRC | -.00 | .03 | ||||||||
genomic_suite · AUPRC | .02 | .01 | ||||||||
genomic_suite · AUPRC | -.00 | .00 | ||||||||
genomic_suite · AUPRC | .01 | .01 | ||||||||
genomic_suite · auroc per tissue mean | -.09 | .04 | ||||||||
genomic_suite · auroc per tissue mean | .02 | .04 | ||||||||
genomic_suite · spearman per tissue mean | -.13 | .03 | ||||||||
genomic_suite · matched auprc | .04 | .54 | ||||||||
| Protein fitness (DMS, protein input) | ||||||||||
genomic_suite · spearman per assay mean | .78 | .78 | ||||||||
| Non-coding RNA fitness (DMS) | ||||||||||
genomic_suite · spearman per assay mean | -.08 | .68 | ||||||||
| Coding-sequence fitness (DMS, nucleotide input) | ||||||||||
genomic_suite · spearman per assay mean | .06 | .51 | ||||||||
genomic_suite · spearman per assay mean | -.04 | .25 | ||||||||
| Functional-genomics track prediction | ||||||||||
genomic_suite · track pearson | .00 | -.00 | ||||||||
genomic_suite · profile pearson | -.00 | -.00 | ||||||||
genomic_suite · track pearson | -.00 | -.00 | ||||||||
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | ||||||||||
genomic_suite · gene mteb 8shot | .00 | .73 | ||||||||
genomic_suite · gene mteb 8shot | .01 | .84 | ||||||||
genomic_suite · gene mteb 8shot | .00 | .70 | ||||||||
genomic_suite · gene mteb 8shot | -.00 | .80 | ||||||||
| diagnostics | ||||||||||
genomic_suite · masked nt accuracy | .27 | .28 | ||||||||
| Peptide-GPCR potency | ||||||||||
peptide_potency · Spearman ρ | 1.00 | |||||||||
Normalized score, 1.0 = SOTA (or reference if no SOTA)≤ chance0.250.50.70.85< 1≥ SOTA≥ 1.15 not evaluated private dataset