topo-confidence — selective prediction via prefill residual-stream geometry on Qwen-2.5-1.5B

The decomposition triangle

F-2's signal factors into three additive pieces: a directional component (PC1 mean shift + PC9 trim) that saturates at the 2-feature ceiling, plus a per-problem second-order spectral component that lifts above it. Anchor tier in bold.

Probe	OOF AUROC	Δ vs DoM 0.7679
1-d DoM (matched protocol)	0.7679	—
2-feat (PC1, PC9)	0.7856	+1.77 pp
Full 1536-d L2-reg (best C=0.001)	0.7847	+1.68 pp ≈ 2-feat
Top-20 log-eigvals of PC1-residualized cov	0.7928	+2.49 pp

The full 1536-d L2-reg ceiling sits at 0.7847 ≈ 2-feat 0.7856 within fold noise — so the directional ceiling at L19 saturates at two features. The cov-spectrum lift to 0.7928 is therefore genuinely second-order, not an under-regularized linear-directional artifact. Unsupervised PCA-PC1 recovers DoM (cosine 0.9216) — the dominant correctness direction is unsupervised-identifiable.

What was overturned

Truncation-artifact retraction. The original headline — ABC-44 AUROC 0.796 on MATH-500 — ran against labels generated with max_new_tokens=256. The 1.5B model only emitted 104/500 correct answers under that truncation, vs 243/500 (48.6%) at 1024 tokens. Every downstream cross-scale and cross-benchmark claim derived from the 256-tok labels was 2.3× off. Full graveyard at PROJECT_RECORD.md §1d.

Persistent homology ≠ topology. PH features on raw L19 clouds match a rank-matched Gaussian null (real 0.690 vs null 0.693; F-10). On PC1-residualized clouds, PH strengthens against the matched-cov null in the wrong direction by 12.4 pp (real 0.6884 vs null 0.7628, gap −0.074) — F-10 holds (FE880, 2026-05-02).
Cross-scale «7B is a stronger verifier» — 256-tok artifact. At 1024 tokens, 7B→1.5B transfer 0.717 vs 1.5B self 0.719: no asymmetry.
Fixed-vector steering. The correctness axis rotates through generation; cos(prefill_DoM, final_DoM) = 0.046. Final-token-fit vectors injected at position 15 are geometrically random.
CoE-60 > single-layer DoM — on 1024-tok labels, +0.004 in favor of DoM (cross-domain transfer). CoE is redundant.
Monotonic prefill compute gating. The B-bucket (K=8-recoverable) lives at middle prefill score, not bottom. Negative-sequence-length gating is Pareto-dominant.

v1 archive (April 2026) — click to expand

Historical snapshot. The panels below reflect the project's v1 "persistent homology of hidden-state token clouds predicts correctness" framing. The 0.796 MATH-500 AUROC headline was a max_new_tokens=256 truncation artifact; current findings supersede this section. Preserved here as a record of the framing the program started from.

0.796

v1 AUROC (MATH-500, 256-tok — superseded)

+0.057

vs best baseline (256-tok)

0 R→W

at tau=0.3 gating (256-tok)

v1 framing: 44 topological features from a single forward pass on the prompt. No generation, no sampling, no output parsing. The only single-pass method that competed with 32-pass self-consistency baselines on mathematical reasoning — until the 1024-tok rebuild showed the comparator was truncation-confounded.

How it worked (v1)

Step 1

Forward pass

→

Step 2

Extract hidden states

→

Step 3
PCA + Ripser (PH)

→

Step 4

44 topo features

→

Step 5

P(correct)

Each prompt's hidden states at the final transformer layer formed a point cloud in R^d. Persistent homology extracted topological structure — connected components (H0) and loops (H1). Layer-wise dynamics (cosines, PCA spectrum, SVD ratios) captured how representations evolve across depth. Together: 44 features. F-10 (PH = Gaussian null) overturned this framing.

44 features in 3 tiers (v1, 256-tok)

9 features

0.704

Persistent homology + geometry. H0/H1 entropy, lifetimes, centroid distances, token norms.

30 features

0.761

Layer dynamics. Inter-layer cosines, PCA spectrum ratios, SVD decay.

5 features

0.732

Depth-2 cross-tier products. Interactions between PH and layer features.

Combined A+B+C = 0.796 AUROC (256-tok). The 256-tok rebuild at 1024 tokens showed the topology-specific signal sits at the matched-cov Gaussian null.

Cross-benchmark results (v1, 256-tok)

All numbers below are 256-tok labels — superseded by the 1024-tok rebuild.

Configuration	Accuracy	Topo AUROC	Best Baseline	Gap
Qwen2.5-1.5B × MATH-500	104/500 (20.8%) — truncation	0.796	vote_margin 0.767	+0.057
Qwen2.5-7B × MATH-500	348/500 (69.6%)	0.739	first_token 0.637	+0.102
Qwen2.5-1.5B × GSM8K	871/1319 (66.0%)	0.615	neg_entropy 0.741	-0.126

Transparency note: Initial AUROC was reported as 0.948. Phase 6.5 (April 2026) found three confounds: answer-extraction false negatives, PCA holdout leakage, and the max_new_tokens=256 truncation. The 1024-tok rebuild later showed the correct 1.5B baseline is 48.6% (243/500), not 20.8%. The PH-specific signal sits at the matched-cov Gaussian null; the surviving signal is a single L19 prefill direction (DoM).

Get started (v1 reference impl)

The topo_confidence Python package (v0.2.0) is the v1-framing reference implementation. The headline result it computes (AUROC 0.796 on MATH-500) is the 256-tok number, kept for reproducibility of the historical claim. Current findings live in the pathway11_h100/ directory of the GitHub repo.

$ pip install -e .

from topo_confidence import TopoConfidence

tc = TopoConfidence("Qwen/Qwen2.5-1.5B-Instruct")
tc.calibrate(calibration_prompts, calibration_labels)
confidences = tc.predict_confidence(["What is 2+2?", "Prove the Riemann Hypothesis"])