Bacterial Gram-staining from genome, family-holdout blind eval
About this task
Predict Gram-staining status (positive vs. negative) for held-out bacterial families from genome assembly alone. Agent gets an NCBI assembly accession + phylum for each strain and — for the training strains — a Gram label. Nothing more granular than phylum is shipped (no genus, no species, no strain name), so the agent must extract features from the actual genome sequence rather than looking the answer up by name. Test strains come from families not represented in the training set. Feature choice is on the agent: gene predictions, protein-family counts, KEGG/COG abundances, genome FM embeddings (GenomeOcean, Evo 2, Nucleotide Transformer), lipopolysaccharide biosynthesis-pathway hits, tRNA/rRNA counts, whole-genome k-mer profiles — anything extractable from the assembly.
Leaderboard
Ranked by suite score (normalized, 1.0 = SOTA). Dataset cells show normalized values; hover for raw values and CIs.
| # | Run | Integrity | |||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Codeinf.20260719T113801 · claude-opus-4-8[1m] | 0.995 | 1.10 | 1.10 | Pending | $3.10 | Jul 19, 2026 |
| SOTAprotein-family-profile_classifier | — | 1.00 | 1.00 | — | — | ||
| Refcodon-usage_+_tetranucleotide | — | 0.78 | 0.78 | — | — | ||
| Refclass_prior (always Gram-) | — | 0.00 | 0.00 | — | — |
* partial suite score (some core datasets not evaluated). ≈ approximate reference value. “inf.” = agent / effort inferred from the run id or trace (no run_meta.json).
Per-dataset results
Every run on every dataset, against chance, the reference baseline and SOTA. Filters above apply.
Suite datasets
From suite.yaml. Normalized score = (value − chance) / (SOTA − chance); without SOTA the reference is used.
| Dataset | Visibility | Metric | Chance | Reference | SOTA | Source |
|---|---|---|---|---|---|---|
| gram family holdout | public | AUROC | 0.500 | 0.850 codon-usage_+_tetranucleotide | 0.950 protein-family-profile_classifier | Madin et al. 2020 (BacDive-derived), family-disjoint test |
Prompt
Exactly what the agent receives.
Show the full prompt
BioModelBench — Bacterial Gram-staining (bacdive_gram)
Task
For each row in test.parquet, predict gram_prob ∈ [0, 1]: the
probability the strain is Gram-positive. The label is binary
(Gram-positive = 1, Gram-negative = 0).
Test strains are drawn from bacterial families not present in the training set. You have the assembly accession + phylum for each strain — that's it. No genus, no species, no strain name.
What's shipped
train.parquet
Columns:
assembly_accession(string, e.g.GCF_000005845.2orGCA_…) — NCBI assembly accession.phylum(string) — coarse taxonomy.label(int, 0/1) — 1 = Gram-positive, 0 = Gram-negative.
test.parquet
Columns: assembly_accession, phylum. No label.
Compute
Modal container: A10G GPU (24 GB VRAM), 8 CPUs, 96 GB RAM. Python 3.11
with torch / transformers / peft / xgboost / lightgbm / scikit-learn / pandas / pyarrow / numpy / pyBigWig / pysam / biopython / requests
pre-installed. pip install <x> works.
Budget: ~10 hours of GPU compute. Use it. Even after you produce a
valid answer.parquet, keep iterating until the budget is exhausted.
The runner will relaunch your session with a continuation prompt if you
exit early — don't finalize prematurely.
Deliverables
answer.parquet— columnsassembly_accession, gram_prob, one row per test strain,gram_prob ∈ [0, 1].method.md(< 500 words).training_manifest.json— every URL / API / dataset touched.
What the grader does
Reads your answer.parquet, joins on assembly_accession against the
hidden Gram-staining labels, computes AUPRC / AUROC / Brier / coverage.
Reference bar
Test-set base rate: ~0.34 Gram-positive. Published Gram predictors on
family-holdout land around AUPRC 0.7–0.9. Exact reference-baseline bar
will be baked into grade.py before this task is treated as production.