Codex · low

Pending

20260722T234221-codex-low-capharness: Codexmodel: unknowneffort: low(inferred)tag: codex-low-cap

Mean AUPRC

0.686

task headline

Suite score

0.91

1.0 = SOTA

Wall clock

—

of 10 h budget

Compute

1xH100

Agent cost

—

summed from trace

Started

Jul 22, 2026

Results by dataset

Raw metric, 95% CI where reported, and the normalized score against the dataset's SOTA (or reference).

Dataset	Metric	Value	95% CI	Normalized	Chance	Ref.	SOTA	n · coverage
traitgym complex Variant effect (DNA)	AUPRC	0.348	—	0.95	0.100	0.284	0.362	11,400 · 100%
traitgym mendelian Variant effect (DNA)	AUPRC	0.729	—	0.81	0.101	0.648	0.875	3,353 · 100%
clinvar Variant effect (DNA)	AUPRC	0.980	—	0.98	0.494	—	0.990	4,674 · 100%
clinvar missensesupp. Variant effect (DNA)	AUROC	0.964	—	0.98	0.500	0.966	0.972	1,263 · 100%
clinvar intronicsupp. Variant effect (DNA)	AUROC	0.926	—	0.88	0.500	—	0.984	674 · 100%
clinvar synonymoussupp. Variant effect (DNA)	AUROC	1.000	—	1.08	0.500	—	0.961	1,319 · 100%
clinvar stop gainedsupp. Variant effect (DNA)	AUROC	0.915	—	1.04	0.500	—	0.900	823 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
traitgym complex	0.348	0.724	0.078
traitgym mendelian	0.729	0.886	0.046
clinvar	0.980	0.982	0.043
clinvar missense	0.987	0.964	0.069
clinvar intronic	0.762	0.926	0.075
clinvar synonymous	0.971	1.000	0.012
clinvar stop gained	1.000	0.915	0.039

Breakdown: clinvar by consequence

Slice	n	AUPRC	AUROC	Brier
intronic	674	0.762	0.926	0.075
missense	1,263	0.987	0.964	0.069
splice acceptor	188	—	—	—
splice donor	268	—	—	—
stop gained	823	1.000	0.915	0.039
synonymous	1,319	0.971	1.000	0.012

Breakdown: clinvar by review status

Slice	n	AUPRC	AUROC	Brier
2 star	4,368	0.978	0.983	0.042
3 star expert panel	284	0.992	0.946	0.062

Breakdown: clinvar by significance pair

Slice	n	AUPRC	AUROC	Brier
strong only P vs B	3,058	0.988	0.982	0.045
hedged only LP vs LB	1,613	0.958	0.985	0.039
any P vs any B	4,671	0.980	0.982	0.043

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
Not individually covered by AUDIT_traitgym_mendelian.md
pending
training manifest
training_manifest.json present
pass
trace
agent_log.jsonl available for the trace judge
pass
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: vep_multieval v1
Started: Jul 22, 2026 23:42 UTC
Finished: —
Sessions: 31
Tool calls: 124
Turns: —
Artifacts: method.md, agent_log.jsonl, training_manifest.json

Method

method.md, written by the agent.

Method

I first built the exact test tuple set and used it as an exclusion filter before parsing labels. There are 44,689 conventional SNVs and 17 rows with alt=".". Initial annotation work extracted CADD v1.7 and downloaded Zoonomia phyloP-241m; the latter was not retained because it was redundant with CADD and weaker than the available TraitGym ensemble. An Ensembl VEP REST trial was also abandoned after timeouts; no clinical-significance fields from VEP or ClinVar were used as features.

The 11,400 complex-trait and 3,380 Mendelian rows matched the public TraitGym matched_9 coordinate collections (27 overlap). For these I used the authors' published chromosome-held-out CADD+GPN-MSA+Borzoi logistic-regression predictions. I adjusted only the logit intercept so each collection's mean probability matched its documented 1:9 class design; this preserves ranking. Overlap predictions were combined by geometric mean.

For clinical transfer, I parsed the 2026-07-21 GRCh38 ClinVar VCF. I excluded 30,818 test-matching tuples before examining labels. Among the remainder, unambiguous Pathogenic/Likely-pathogenic versus Benign/Likely-benign SNVs yielded 1,449,595 eligible examples. Stable hash-stratified sampling by label and broad molecular consequence produced 25,961 examples; one non-DNA ALT (N) lacked CADD and was dropped, leaving 25,960. Features were CADD v1.7 raw/PHRED, AlphaMissense (with a missingness flag), and broad consequence from ClinVar's molecular annotation. No ClinVar significance, phenotype, ID, allele frequency, or review-status field was used at inference.

Iterations on chromosomes 3, 7, 12, 17, and 22 held out: CADD logistic regression reached AUPRC 0.850, AUROC 0.905, Brier 0.116; adding consequence reached 0.900/0.935/0.095; a regularized depth-3 XGBoost model with AlphaMissense reached 0.935/0.949/0.083 and was selected. It was refit on all 25,960 non-overlapping examples and applied to the 29,953 remaining test rows.

The 17 non-SNV placeholder rows used median CADD and unknown consequence. Scores are clipped to [0,1]. All 44,706 test tuples are present exactly once.

Training manifest

Declared data sources and filtering.

CADD v1.7 GRCh38 all SNVs
Tabix extraction of raw and PHRED annotations; 44,689/44,706 test coverage, with 17 alt='.' rows imputed
https://kircherlab.bihealth.org/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz
AlphaMissense GRCh38
Missense pathogenicity feature
https://storage.googleapis.com/dm_alphamissense/AlphaMissense_hg38.tsv.gz
NCBI ClinVar GRCh38 VCF, 2026-07-21 release
Non-overlapping training labels and molecular consequence; test significance fields never accessed
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz
TraitGym matched_9
Published chromosome-held-out predictions; label columns were not loaded
https://huggingface.co/datasets/songlab/TraitGym
UCSC Zoonomia phyloP-241m
Downloaded during initial iteration; not retained in final model
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/cactus241way/cactus241way.phyloP.bw
Ensembl VEP REST
Small annotation trial; abandoned after timeout and not used
https://rest.ensembl.org/vep/human/region
MyVariant.info REST
Two-variant availability trial; not used
https://myvariant.info/v1/variant
UCSC Genome API
Track discovery/trial only; not used in final score
https://api.genome.ucsc.edu/list/tracks?genome=hg38
CADD experimental REST API
One-position availability trial; empty response, not used
https://cadd.gs.washington.edu/api/v1.0/GRCh38-v1.7/

Raw training_manifest.json

{
  "test_reference": {
    "rows": 44706,
    "unique_exact_tuples": 44706,
    "build": "GRCh38/hg38"
  },
  "training": {
    "clinvar_eligible_after_exact_exclusion": 1449595,
    "clinvar_sampled": 25961,
    "clinvar_final_after_invalid_alt_drop": 25960,
    "exact_test_tuples_excluded_before_label_access": 30818,
    "traitgym_labels_loaded": false
  },
  "sources": [
    {
      "name": "CADD v1.7 GRCh38 all SNVs",
      "urls": [
        "https://kircherlab.bihealth.org/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
        "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz"
      ],
      "use": "Tabix extraction of raw and PHRED annotations; 44,689/44,706 test coverage, with 17 alt='.' rows imputed"
    },
    {
      "name": "AlphaMissense GRCh38",
      "url": "https://storage.googleapis.com/dm_alphamissense/AlphaMissense_hg38.tsv.gz",
      "use": "Missense pathogenicity feature",
      "matched_test_clinical_rows": 7689
    },
    {
      "name": "NCBI ClinVar GRCh38 VCF, 2026-07-21 release",
      "urls": [
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz",
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz.tbi"
      ],
      "use": "Non-overlapping training labels and molecular consequence; test significance fields never accessed"
    },
    {
      "name": "TraitGym matched_9",
      "url": "https://huggingface.co/datasets/songlab/TraitGym",
      "files_used": [
        "complex_traits_matched_9/test.parquet (coordinate columns only)",
        "complex_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "mendelian_traits_matched_9/test.parquet (coordinate columns only)",
        "mendelian_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet"
      ],
      "matches": {
        "complex": 11400,
        "mendelian": 3380,
        "cross_collection_overlap": 27
      },
      "use": "Published chromosome-held-out predictions; label columns were not loaded"
    },
    {
      "name": "UCSC Zoonomia phyloP-241m",
      "url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/cactus241way/cactus241way.phyloP.bw",
      "use": "Downloaded during initial iteration; not retained in final model"
    },
    {
      "name": "Ensembl VEP REST",
      "url": "https://rest.ensembl.org/vep/human/region",
      "use": "Small annotation trial; abandoned after timeout and not used"
    },
    {
      "name": "MyVariant.info REST",
      "url": "https://myvariant.info/v1/variant",
      "use": "Two-variant availability trial; not used"
    },
    {
      "name": "UCSC Genome API",
      "urls": [
        "https://api.genome.ucsc.edu/list/tracks?genome=hg38",
        "https://api.genome.ucsc.edu/getData/track?genome=hg38;track=phyloP100way"
      ],
      "use": "Track discovery/trial only; not used in final score"
    },
    {
      "name": "CADD experimental REST API",
      "urls": [
        "https://cadd.gs.washington.edu/api/v1.0/GRCh38-v1.7/",
        "https://cadd.bihealth.org/api/v1.0/GRCh38-v1.7/"
      ],
      "use": "One-position availability trial; empty response, not used"
    }
  ],
  "iterations": [
    {
      "iteration": 0,
      "model": "CADD/phyloP zero-shot exploration",
      "validation": null,
      "selected": false
    },
    {
      "iteration": 1,
      "model": "CADD logistic",
      "validation_partition": "held chromosomes 3,7,12,17,22",
      "n": 6099,
      "auprc": 0.8497361685,
      "auroc": 0.9047275521,
      "brier": 0.115557042,
      "selected": false
    },
    {
      "iteration": 2,
      "model": "CADD + consequence logistic",
      "validation_partition": "held chromosomes 3,7,12,17,22",
      "n": 6099,
      "auprc": 0.899517129,
      "auroc": 0.9346074368,
      "brier": 0.095070043,
      "selected": false
    },
    {
      "iteration": 3,
      "model": "CADD + AlphaMissense + consequence XGBoost",
      "validation_partition": "held chromosomes 3,7,12,17,22",
      "n": 6099,
      "auprc": 0.9350383299,
      "auroc": 0.9493205564,
      "brier": 0.0833029598,
      "selected": true
    },
    {
      "iteration": 4,
      "model": "TraitGym chromosome-held-out ensemble plus selected clinical model",
      "validation": "component validation above; no test labels used",
      "selected": true
    }
  ]
}

Agent trace

Compact trace, 65 KB. Loaded on demand.