Claude Code

Pending

20260719T113801harness: Claude Codemodel: claude-opus-4-8[1m]effort: default(inferred)

AUPRC

0.995

task headline

Suite score

1.10

1.0 = SOTA

Wall clock

23 min

of 10 h budget

Compute

1xH100

23 min GPU

Agent cost

$3.10

summed from trace

Started

Jul 19, 2026

finished Jul 19, 2026

Results by dataset

Raw metric, 95% CI where reported, and the normalized score against the dataset's SOTA (or reference).

Dataset	Metric	Value	95% CI	Normalized	Chance	Ref.	SOTA	n · coverage
gram family holdout Genome to phenotype	AUROC	0.996	—	1.10	0.500	0.850	0.950	393 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
gram family holdout	0.995	0.996	0.009

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
No post-run audit yet.
pending
training manifest
training_manifest.json present
pass
trace
agent_log.jsonl available for the trace judge
pass
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: bacdive_gram v1
Started: Jul 19, 2026 11:38 UTC
Finished: Jul 19, 2026 12:00 UTC
Sessions: 2
Tool calls: 87
Turns: 46
Artifacts: method.md, agent_log.jsonl, training_manifest.json

Method

method.md, written by the agent.

Method — Bacterial Gram-stain prediction (bacdive_gram)

Approach

Gram status is a genome-encoded phenotype: Gram-negatives (diderms) build an outer membrane with LPS; Gram-positives (monoderms) build a thick peptidoglycan wall with teichoic acids and LPXTG-anchored surface proteins. Because the test families are unseen, I predict from conserved biological markers in the genome content, not from taxon identity.

Pipeline

Genome retrieval. For all 2704 train+test accessions I fetched assembly metadata (assembly name, GC%, genome length, contig count, gene count) from the NCBI Datasets API, then downloaded RefSeq proteomes (_protein.faa.gz) from the NCBI genome FTP. For 47 unannotated assemblies I called genes with pyrodigal.
Marker features (HMMER). Using pyhmmer, I searched each proteome (dom E-value < 1e-5, score > 20) against 20 curated Pfam HMMs:
- Gram-negative / outer-membrane: LpxC, LpxK (lipid A), Omp85/BamA, LptD, LptC, LolA (LPS transport & OM assembly), OmpA, OMP β-barrel, KdsB, LPS heptosyltransferase, Wzz/Wzy.
- Gram-positive / cell-wall: Sortase, LPXTG Gram-pos anchor, WTA glycerophosphotransferase.
- Shared: transpeptidase/transglycosylase PBPs, SLT. Per genome I record hit counts and best bit-scores, plus engineered gn_core/gp_core presence sums.
Auxiliary features. GC%, genome length, contig count, coding density, proteome size, mean protein length, 20-dim amino-acid composition, and the provided phylum (one-hot). No genus/species/family/organism-name was ever used as a feature.
Model. A diversified ensemble — LightGBM on all features (5 seeds), LightGBM on markers+phylum only (5 seeds, most family-robust), and an L2 logistic regression — blended 0.45/0.30/0.25, then isotonic-calibrated.
Validation. 5-fold family-grouped cross-validation (families held out, mirroring the test split; family recovered via Entrez taxonomy for grouping only). Isotonic calibration fit on grouped out-of-fold predictions.

Results (family-grouped CV)

AUPRC 0.990, AUROC 0.994, Brier 0.022. An ablation shows a markers-only model reaches AUPRC 0.989 — performance stems from family-generalizable biology, not memorized taxonomy. Within the mixed Firmicutes phylum, AUROC ≈ 0.90 (LPS markers identify diderm Negativicutes). The markers also corrected a mislabeled test strain (Agitococcus, labeled Firmicutes but genomically Gammaproteobacteria → correctly predicted Gram-negative).

Output

answer.parquet (assembly_accession, gram_prob), 393 rows, full coverage, mean predicted Gram-positive rate 0.34 (matches the stated base rate).

Training manifest

Declared data sources and filtering.

NCBI Datasets API v2 — genome dataset_report
Resolve assembly_name, GC%, genome length, contig count, protein-coding gene count for each assembly accession.
https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report
NCBI genome FTP (assembly proteomes / genomes)
Download <acc>_<asm>_protein.faa.gz (RefSeq annotated proteomes); fallback <acc>_<asm>_genomic.fna.gz + pyrodigal gene calling for unannotated assemblies.
https://ftp.ncbi.nlm.nih.gov/genomes/all/
EBI InterPro — Pfam HMM profiles
Fetch 20 marker HMMs (LPS/outer-membrane vs cell-wall/anchor) for hmmsearch.
https://www.ebi.ac.uk/interpro/wwwapi/entry/pfam/<PFAM_ACC>?annotation=hmm
NCBI Entrez efetch (taxonomy)
Retrieve family rank per tax_id — used ONLY for family-grouped cross-validation, NOT as a model feature.
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=taxonomy

Raw training_manifest.json

{
  "task": "bacdive_gram_v0 — bacterial Gram-stain prediction (family-holdout)",
  "inputs_provided": [
    "train.parquet",
    "test.parquet"
  ],
  "external_resources": [
    {
      "name": "NCBI Datasets API v2 — genome dataset_report",
      "url": "https://api.ncbi.nlm.nih.gov/datasets/v2/genome/dataset_report",
      "use": "Resolve assembly_name, GC%, genome length, contig count, protein-coding gene count for each assembly accession."
    },
    {
      "name": "NCBI genome FTP (assembly proteomes / genomes)",
      "url": "https://ftp.ncbi.nlm.nih.gov/genomes/all/",
      "use": "Download <acc>_<asm>_protein.faa.gz (RefSeq annotated proteomes); fallback <acc>_<asm>_genomic.fna.gz + pyrodigal gene calling for unannotated assemblies."
    },
    {
      "name": "EBI InterPro — Pfam HMM profiles",
      "url": "https://www.ebi.ac.uk/interpro/wwwapi/entry/pfam/<PFAM_ACC>?annotation=hmm",
      "use": "Fetch 20 marker HMMs (LPS/outer-membrane vs cell-wall/anchor) for hmmsearch.",
      "pfam_accessions": [
        "PF03331",
        "PF01103",
        "PF04453",
        "PF03968",
        "PF00691",
        "PF13505",
        "PF04413",
        "PF02348",
        "PF03548",
        "PF07161",
        "PF02606",
        "PF04932",
        "PF13103",
        "PF06835",
        "PF04203",
        "PF00746",
        "PF04464",
        "PF01464",
        "PF00905",
        "PF00912"
      ]
    },
    {
      "name": "NCBI Entrez efetch (taxonomy)",
      "url": "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=taxonomy",
      "use": "Retrieve family rank per tax_id — used ONLY for family-grouped cross-validation, NOT as a model feature."
    }
  ],
  "software": [
    "pyhmmer (HMMER3)",
    "pyrodigal (Prodigal)",
    "lightgbm",
    "scikit-learn",
    "pandas",
    "requests"
  ],
  "notes": "Model features are strictly genome-sequence-derived (marker-gene HMM hits, GC/size/coding-density, amino-acid composition) plus the provided phylum. Organism names / species / genus / family were never used as predictive features."
}

Agent trace

Compact trace, 102 KB. Loaded on demand.