Codex · medium

Pending

20260722T205100-codex-medium-v2harness: Codexmodel: unknowneffort: medium(inferred)tag: codex-medium-v2

Mean AUPRC

0.626

task headline

Suite score

0.75

1.0 = SOTA

Wall clock

—

of 10 h budget

Compute

1xH100

Agent cost

—

summed from trace

Started

Jul 22, 2026

Results by dataset

Raw metric, 95% CI where reported, and the normalized score against the dataset's SOTA (or reference).

Dataset	Metric	Value	95% CI	Normalized	Chance	Ref.	SOTA	n · coverage
traitgym complex Variant effect (DNA)	AUPRC	0.260	—	0.61	0.100	0.284	0.362	11,400 · 100%
traitgym mendelian Variant effect (DNA)	AUPRC	0.688	—	0.76	0.101	0.648	0.875	3,353 · 100%
clinvar Variant effect (DNA)	AUPRC	0.930	—	0.88	0.494	—	0.990	29,953 · 100%
clinvar missensesupp. Variant effect (DNA)	AUROC	0.963	—	0.98	0.500	0.966	0.972	7,861 · 100%
clinvar intronicsupp. Variant effect (DNA)	AUROC	0.921	—	0.87	0.500	—	0.984	4,368 · 100%
clinvar synonymoussupp. Variant effect (DNA)	AUROC	0.966	—	1.01	0.500	—	0.961	8,155 · 100%
clinvar stop gainedsupp. Variant effect (DNA)	AUROC	0.822	—	0.80	0.500	—	0.900	5,762 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
traitgym complex	0.260	0.646	0.089
traitgym mendelian	0.688	0.903	0.063
clinvar	0.930	0.939	0.106
clinvar missense	0.986	0.963	0.094
clinvar intronic	0.712	0.921	0.083
clinvar synonymous	0.790	0.966	0.080
clinvar stop gained	1.000	0.822	0.160

Breakdown: clinvar by consequence

Slice	n	AUPRC	AUROC	Brier
3 prime utr	445	0.016	0.858	0.111
5 prime utr	170	0.484	0.812	0.278
initiator codon	163	0.997	0.849	0.293
intronic	4,368	0.712	0.921	0.083
missense	7,861	0.986	0.963	0.094
other	133	0.641	0.862	0.172
splice acceptor	1,231	0.999	0.871	0.112
splice donor	1,649	1.000	0.951	0.123
stop gained	5,762	1.000	0.822	0.160
synonymous	8,155	0.790	0.966	0.080

Breakdown: clinvar by review status

Slice	n	AUPRC	AUROC	Brier
2 star	28,058	0.925	0.939	0.105
3 star expert panel	1,764	0.982	0.925	0.131
unknown	124	0.920	0.940	0.102

Breakdown: clinvar by significance pair

Slice	n	AUPRC	AUROC	Brier
strong only P vs B	20,145	0.953	0.942	0.112
hedged only LP vs LB	9,790	0.876	0.942	0.095
any P vs any B	29,935	0.930	0.939	0.106

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
Not individually covered by AUDIT_traitgym_mendelian.md
pending
training manifest
training_manifest.json present
pass
trace
agent_log.jsonl available for the trace judge
pass
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: vep_multieval v1
Started: Jul 22, 2026 20:51 UTC
Finished: —
Sessions: 101
Tool calls: 120
Turns: —
Artifacts: method.md, agent_log.jsonl, training_manifest.json

Method

method.md, written by the agent.

Method

I treated the output as a transferable functional-impact probability with two annotation branches, not as a lookup of test labels. All matching and deduplication used the exact hg38 (chrom,pos,ref,alt) tuple.

Iterations. (1) I tested MyVariant annotations, but coverage was negligible and this baseline was discarded. (2) I retrieved CADD v1.7 raw/PHRED scores and obtained consequence-only annotations from ClinVar; an Ensembl VEP+CADD/SpliceAI trial was abandoned because of throughput and is not used. Unreliable point-API responses prompted a switch to local allele-specific UCSC CADD bigWigs. (3) For variants matching the two public TraitGym coordinate sets, I added only published zero-shot annotations: CADD raw, GPN-MSA absLLR, Borzoi L2, primate phastCons, and 241-mammal phyloP. I did not load benchmark labels into modeling and did not use the downloaded chromosome-trained LR predictions. I percentile-ranked each annotation, formed a literature-informed weighted rank ensemble, reranked it, and mapped it to the known 10% benchmark prior. (4) I added AlphaMissense plus conservative consequence priors. (5) The final clinical branch replaced hand-set CADD mappings with measured consequence-specific calibrators.

For calibration training I downloaded the 2026-07-21 ClinVar GRCh38 VCF and constructed the test tuple set first. The missense set had 223,433 eligible unambiguous benign/likely-benign versus pathogenic/likely-pathogenic SNVs; 8,070 exact test overlaps and 12,504 variants lacking AlphaMissense were excluded, leaving 202,859 examples. A logistic AlphaMissense calibrator achieved AUROC 0.9614, AUPRC 0.9343, and Brier 0.0724 on held-out chromosomes 19–22/X, then was refit on all leakage-free rows. A broader aggregation started from 1,482,054 eligible SNVs, excluded 30,656 exact test overlaps, and retained 1,451,398. From it I sampled 13,305 consequence/class-stratified examples for CADD calibration. Held-out consequence AUROCs ranged 0.789–0.921. Missense scores blend AlphaMissense (78%) and CADD (22%); other clinical consequences use their measured monotone CADD calibrator.

No clinical significance, review-status, TraitGym label, PIP, MAF, or LD field from any submitted test variant was used. Final scores are clipped to [0,1]; all 44,706 rows are covered.

Training manifest

Declared data sources and filtering.

CADD v1.7 GRCh38 API
test-time zero-shot raw and PHRED annotation
https://cadd.gs.washington.edu/api/v1.0/GRCh38-v1.7/
UCSC AlphaMissense hg38 allele tracks
allele-specific zero-shot missense annotation
https://hgdownload.soe.ucsc.edu/gbdb/hg38/alphaMissense/a.bw
NCBI ClinVar GRCh38 weekly VCF, 2026-07-21
leakage-filtered calibration labels and molecular consequence
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz
songlab/TraitGym
coordinate matching and label-free zero-shot annotations
https://huggingface.co/datasets/songlab/TraitGym
Exploratory APIs and tracks not used in final model
https://myvariant.info/v1/variant

Raw training_manifest.json

{
  "created_utc": "2026-07-22",
  "test_exclusion_key": [
    "chrom",
    "pos",
    "ref",
    "alt"
  ],
  "sources": [
    {
      "name": "CADD v1.7 GRCh38 API",
      "urls": [
        "https://cadd.gs.washington.edu/api/v1.0/GRCh38-v1.7/",
        "https://cadd.bihealth.org/api/v1.0/GRCh38-v1.7/",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/cadd1.7/a.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/cadd1.7/c.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/cadd1.7/g.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/cadd1.7/t.bw"
      ],
      "purpose": "test-time zero-shot raw and PHRED annotation",
      "labels_used": false
    },
    {
      "name": "UCSC AlphaMissense hg38 allele tracks",
      "urls": [
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/alphaMissense/a.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/alphaMissense/c.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/alphaMissense/g.bw",
        "https://hgdownload.soe.ucsc.edu/gbdb/hg38/alphaMissense/t.bw"
      ],
      "purpose": "allele-specific zero-shot missense annotation",
      "labels_used": false
    },
    {
      "name": "NCBI ClinVar GRCh38 weekly VCF, 2026-07-21",
      "urls": [
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz",
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz.tbi"
      ],
      "purpose": "leakage-filtered calibration labels and molecular consequence",
      "dedup": {
        "missense_eligible_before": 223433,
        "missense_exact_test_removed": 8070,
        "missense_missing_score_removed": 12504,
        "missense_final": 202859,
        "all_snv_eligible_before": 1482054,
        "all_snv_exact_test_removed": 30656,
        "all_snv_final": 1451398,
        "cadd_calibration_stratified_sample": 13305
      }
    },
    {
      "name": "songlab/TraitGym",
      "urls": [
        "https://huggingface.co/datasets/songlab/TraitGym",
        "https://huggingface.co/api/datasets/songlab/TraitGym/tree/main/complex_traits_matched_9?recursive=true&expand=false&limit=1000",
        "https://huggingface.co/api/datasets/songlab/TraitGym/tree/main/mendelian_traits_matched_9?recursive=true&expand=false&limit=1000"
      ],
      "configs": [
        "complex_traits_matched_9",
        "mendelian_traits_matched_9"
      ],
      "files_touched_each_config": [
        "test.parquet",
        "preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "preds/all/CADD.plus.RawScore.parquet",
        "preds/all/GPN-MSA_absLLR.plus.score.parquet",
        "preds/all/Borzoi_L2_L2.plus.all.parquet",
        "preds/all/phastCons-43p.plus.score.parquet",
        "preds/all/phyloP-241m.plus.score.parquet"
      ],
      "purpose": "coordinate matching and label-free zero-shot annotations",
      "labels_used": false,
      "trained_LR_prediction_used": false
    },
    {
      "name": "Exploratory APIs and tracks not used in final model",
      "urls": [
        "https://myvariant.info/v1/variant",
        "https://rest.ensembl.org/vep/human/region",
        "https://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bigWigToBedGraph",
        "https://zenodo.org/records/19701997",
        "https://github.com/goodfire-ai/evee-manuscript"
      ],
      "labels_used": false
    }
  ],
  "iterations": [
    {
      "iteration": 1,
      "method": "MyVariant exploratory annotation",
      "status": "discarded_low_coverage",
      "validation": null
    },
    {
      "iteration": 2,
      "method": "CADD plus consequence baseline",
      "status": "superseded",
      "validation": null
    },
    {
      "iteration": 3,
      "method": "Add zero-shot TraitGym rank ensemble",
      "status": "retained",
      "validation": null
    },
    {
      "iteration": 4,
      "method": "Add leakage-safe AlphaMissense calibration and conservative consequence priors",
      "status": "superseded_in_part",
      "validation": {
        "split": "ClinVar chromosomes 19-22,X",
        "n": 37526,
        "auroc": 0.9614245905009449,
        "auprc": 0.9343103771613556,
        "brier": 0.07240013496929594
      }
    },
    {
      "iteration": 5,
      "method": "Replace hand-set non-missense mappings with leakage-safe consequence-specific CADD calibrators; retain AlphaMissense blend",
      "status": "final",
      "validation": {
        "split": "ClinVar chromosomes 19-22,X",
        "sample_n": 13305,
        "per_consequence_auroc": {
          "intronic": 0.9092838196286473,
          "missense": 0.920517913070048,
          "nonsense": 0.802048611111111,
          "other": 0.8790772719344148,
          "splice": 0.7888305071252288,
          "synonymous": 0.8742630094847477,
          "utr": 0.8651635720601237
        }
      }
    }
  ]
}

Agent trace

Compact trace, 79 KB. Loaded on demand.