Codex · medium

Pending

20260722T065456-codex-mediumharness: Codexmodel: unknowneffort: medium(inferred)tag: codex-medium

Mean AUPRC

0.671

task headline

Suite score

0.89

1.0 = SOTA

Wall clock

—

of 10 h budget

Compute

1xH100

Agent cost

—

summed from trace

Started

Jul 22, 2026

Results by dataset

Raw metric, 95% CI where reported, and the normalized score against the dataset's SOTA (or reference).

Dataset	Metric	Value	95% CI	Normalized	Chance	Ref.	SOTA	n · coverage
traitgym complex Variant effect (DNA)	AUPRC	0.341	—	0.92	0.100	0.284	0.362	11,400 · 100%
traitgym mendelian Variant effect (DNA)	AUPRC	0.673	—	0.74	0.101	0.648	0.875	3,353 · 100%
clinvar Variant effect (DNA)	AUPRC	0.998	—	1.02	0.494	—	0.990	29,953 · 100%
clinvar missensesupp. Variant effect (DNA)	AUROC	0.998	—	1.06	0.500	0.966	0.972	7,861 · 100%
clinvar intronicsupp. Variant effect (DNA)	AUROC	0.943	—	0.91	0.500	—	0.984	4,368 · 100%
clinvar synonymoussupp. Variant effect (DNA)	AUROC	0.893	—	0.85	0.500	—	0.961	8,155 · 100%
clinvar stop gainedsupp. Variant effect (DNA)	AUROC	0.992	—	1.23	0.500	—	0.900	5,762 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
traitgym complex	0.341	0.717	0.078
traitgym mendelian	0.673	0.924	0.053
clinvar	0.998	0.998	0.016
clinvar missense	0.999	0.998	0.017
clinvar intronic	0.482	0.943	0.054
clinvar synonymous	0.514	0.893	0.007
clinvar stop gained	1.000	0.992	0.001

Breakdown: clinvar by consequence

Slice	n	AUPRC	AUROC	Brier
3 prime utr	445	0.167	0.994	0.002
5 prime utr	170	0.871	0.989	0.047
initiator codon	163	1.000	1.000	0.007
intronic	4,368	0.482	0.943	0.054
missense	7,861	0.999	0.998	0.017
other	133	0.601	0.958	0.041
splice acceptor	1,231	1.000	0.997	0.004
splice donor	1,649	1.000	0.998	0.003
stop gained	5,762	1.000	0.992	0.001
synonymous	8,155	0.514	0.893	0.007

Breakdown: clinvar by review status

Slice	n	AUPRC	AUROC	Brier
2 star	28,058	0.998	0.998	0.014
3 star expert panel	1,764	0.997	0.990	0.034
unknown	124	0.999	0.999	0.047

Breakdown: clinvar by significance pair

Slice	n	AUPRC	AUROC	Brier
strong only P vs B	20,145	0.999	0.999	0.017
hedged only LP vs LB	9,790	0.994	0.997	0.012
any P vs any B	29,935	0.998	0.998	0.015

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
Not individually covered by AUDIT_traitgym_mendelian.md
pending
training manifest
training_manifest.json present
pass
trace
agent_log.jsonl available for the trace judge
pass
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: vep_multieval v1
Started: Jul 22, 2026 06:54 UTC
Finished: —
Sessions: 1
Tool calls: 59
Turns: —
Artifacts: method.md, agent_log.jsonl, training_manifest.json

Method

method.md, written by the agent.

Method

I built one consequence-aware ensemble. All keys were normalized as (chrom, pos, ref, alt) on hg38, and the 44,706 test keys were materialized before any labeled data work.

Iteration 1. I tested direct sparse lookup of CADD v1.7 from its indexed 81-GB hg38 SNV file. Single-site lookup succeeded, but sustained remote Tabix access was unstable (only 285 records recovered), so this was discarded.

Iteration 2. I lifted coordinates to hg19 solely for batched MyVariant lookup (reverse-complementing the 37 negative-strand mappings). This supplied CADD, conservation, gnomAD frequency, AlphaMissense, REVEL, ClinPred, MetaRNN, BayesDel, PrimateAI, VEST4, FATHMM-XF, MutationAssessor, SIFT, and PolyPhen. Ambiguous/unmapped positions were left missing; XGBoost handles missing values natively.

Iteration 3. The 11,400 complex-trait and 3,380 Mendelian rows matched public TraitGym coordinate tables. I used the authors' chromosome-held-out CADD+GPN-MSA+Borzoi predictions (not labels). Their rankings correspond to reported AUPRC 0.362 and 0.648. I shifted logits without labels so each matched-9 partition averaged its known 0.10 prevalence. Twenty-seven variants occur in both partitions; their two probabilities were averaged.

Iteration 4. For clinical transfer/calibration I compiled two balanced training samples: 20,000 variants from songlab/clinvar_vs_benign after excluding 2,797 test tuples, and 20,000 from the 2026-07-21 NCBI ClinVar VCF after excluding 30,657 test tuples. Deduplication between sources left 39,408 labeled variants. No exact test tuple entered training. A chromosome-held-out XGBoost probe used 18 numeric annotations plus 14 consequence indicators. Validation improved from CADD-only AUROC/AUPRC/Brier 0.9163/0.9017/0.1112, to CADD+consequence+AF 0.9806/0.9766/0.0526, to the final model 0.9938/0.9940/0.0260. The final clinical model was refit on all 39,408 rows.

Scores are clipped to [1e-6, 1-1e-6]; every test row is covered.

Training manifest

Declared data sources and filtering.

UCSC hg38-to-hg19 LiftOver chain
annotation-coordinate lookup only
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/liftOver/hg38ToHg19.over.chain.gz
MyVariant.info batch API
CADD/dbNSFP/gnomAD annotations for test and leakage-filtered training variants
https://myvariant.info/v1/variant
CADD v1.7 GRCh38 whole-genome SNVs
discarded sparse-Tabix feasibility probe
https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz
TraitGym
songlab ClinVar datasets
overlap audit; clinvar_vs_benign also supplied leakage-filtered missense labels
https://huggingface.co/datasets/songlab/clinvar
NCBI ClinVar GRCh38 VCF, 2026-07-21 snapshot
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz
GPN-MSA genome-wide scores/catalog (discovery only)
https://huggingface.co/datasets/songlab/gpn-msa-hg38-scores

Raw training_manifest.json

{
  "created_utc": "2026-07-22",
  "test": {
    "path": "test.parquet",
    "rows": 44706,
    "unique_exact_tuples": 44706,
    "assembly": "hg38"
  },
  "resources": [
    {
      "name": "UCSC hg38-to-hg19 LiftOver chain",
      "url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/liftOver/hg38ToHg19.over.chain.gz",
      "purpose": "annotation-coordinate lookup only",
      "test_mapped_unique": 44688,
      "negative_strand": 37
    },
    {
      "name": "MyVariant.info batch API",
      "url": "https://myvariant.info/v1/variant",
      "metadata_url": "https://myvariant.info/v1/metadata/fields",
      "purpose": "CADD/dbNSFP/gnomAD annotations for test and leakage-filtered training variants",
      "labeled_rows_sent": 40000,
      "test_rows_sent": 44706
    },
    {
      "name": "CADD v1.7 GRCh38 whole-genome SNVs",
      "url": "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
      "index_url": "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz.tbi",
      "documentation": "https://cadd.bihealth.org/api",
      "purpose": "discarded sparse-Tabix feasibility probe",
      "records_recovered_before_discard": 285
    },
    {
      "name": "TraitGym",
      "dataset_url": "https://huggingface.co/datasets/songlab/TraitGym",
      "github_url": "https://github.com/songlab-cal/TraitGym",
      "configs_touched": [
        "complex_traits",
        "mendelian_traits",
        "complex_traits_full",
        "mendelian_traits_full"
      ],
      "matched_test_rows": {
        "complex_traits": 11400,
        "mendelian_traits": 3380,
        "overlap_between_partitions": 27
      },
      "labels_used_for_training": false,
      "prediction_files": [
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD+GPN-MSA_LLR+Borzoi_L2_L2.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD+GPN-MSA_LLR+Borzoi_L2_L2.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD.LogisticRegression.chrom.parquet"
      ]
    },
    {
      "name": "songlab ClinVar datasets",
      "urls": [
        "https://huggingface.co/datasets/songlab/clinvar",
        "https://huggingface.co/datasets/songlab/clinvar_vs_benign",
        "https://huggingface.co/datasets/songlab/deprecated-clinvar-missense"
      ],
      "purpose": "overlap audit; clinvar_vs_benign also supplied leakage-filtered missense labels",
      "clinvar_vs_benign_rows": 50164,
      "exact_test_tuples_removed": 2797,
      "eligible_after_filter": 47367,
      "balanced_sample_annotated": 20000
    },
    {
      "name": "NCBI ClinVar GRCh38 VCF, 2026-07-21 snapshot",
      "url": "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz",
      "index_url": "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz.tbi",
      "eligible_unambiguous_biallelic_snv_rows": 1482344,
      "exact_test_tuples_removed": 30657,
      "eligible_after_filter": 1451687,
      "balanced_sample_annotated": 20000
    },
    {
      "name": "GPN-MSA genome-wide scores/catalog (discovery only)",
      "urls": [
        "https://huggingface.co/datasets/songlab/gpn-msa-hg38-scores",
        "https://github.com/songlab-cal/gpn"
      ],
      "downloaded_for_training": false
    }
  ],
  "training": {
    "exact_test_tuple_filter_applied_before_sampling": true,
    "input_sample_rows": 40000,
    "cross_source_duplicates_removed": 592,
    "final_unique_train_rows": 39408,
    "model": "XGBoost consequence-aware annotation ensemble"
  },
  "iterations": [
    {
      "iteration": 1,
      "method": "CADD v1.7 sparse remote Tabix",
      "status": "discarded: remote sparse retrieval unstable",
      "validation": null
    },
    {
      "iteration": 2,
      "method": "CADD-only XGBoost clinical probe",
      "validation": {
        "split": "held-out chromosomes",
        "n": 16318,
        "auroc": 0.9162979581,
        "auprc": 0.9017132831,
        "brier": 0.1111918613
      }
    },
    {
      "iteration": 3,
      "method": "CADD + consequence + allele-frequency clinical probe",
      "validation": {
        "split": "held-out chromosomes",
        "n": 16318,
        "auroc": 0.9805874611,
        "auprc": 0.9765876687,
        "brier": 0.0526372977
      }
    },
    {
      "iteration": 4,
      "method": "full clinical annotation ensemble",
      "validation": {
        "split": "held-out chromosomes",
        "n": 16318,
        "auroc": 0.9938157935,
        "auprc": 0.9939686496,
        "brier": 0.0260111317
      }
    },
    {
      "iteration": 5,
      "method": "TraitGym chromosome-held-out CADD+GPN-MSA+Borzoi predictions",
      "validation": {
        "complex_traits_external_auprc": 0.362,
        "mendelian_traits_external_auprc": 0.648
      }
    }
  ]
}

Agent trace

Compact trace, 45 KB. Loaded on demand.