From 0791296370fd9265e07b324bdebe511c28a5952d Mon Sep 17 00:00:00 2001 From: MuMDIA Date: Thu, 23 Jul 2026 13:44:16 +0200 Subject: [PATCH 1/7] chore: clean-slate declutter (prune scripts, dead config, consolidate docs) - scripts/: keep only the 10 engine-invoked sidecars + DIA-NN recipe helpers; remove ~30 one-off diagnostic/analysis/probe scripts, the exploratory notebook, and feature_registry.yaml (consumed only by the removed diagnostics) - config: remove dead/unwired surface (threads; extract.{k_select, max_fragment_charge, scan_scale, scan_window_mode} + ScanWindowMode; digest.decoy.{ratio, source} + DecoySource; search_seed.precursor_tol_ppm; rt_im_train.tolerance_regime + ToleranceRegime; FeatureSet::Custom; MatcherKind::Naive; CompetitionMode::from_token). Kept DiannShift / None guards and the MBR config surface as documented hooks. - docs (local, gitignored): consolidated sensitivity_plan/ into rewritten plan.md + CLAUDE.md; also removed ~60GB of gitignored run outputs/notebooks/ scratch from the working tree (preserved the DIA-NN library under lib/). cargo test --workspace green (105 tests). Co-Authored-By: Claude Opus 4.8 --- feature_registry.yaml | 1922 ----------------- rust/mumdia/crates/mumdia-core/src/config.rs | 113 +- .../crates/mumdia/src/stages/extract.rs | 7 +- .../crates/mumdia/src/stages/features.rs | 2 +- scripts/analyze_entrapment.py | 60 - scripts/benchmark_report.py | 768 ------- scripts/build_funnel.py | 201 -- scripts/candidate_diagnostics.py | 537 ----- scripts/combine_mutdec.py | 78 - scripts/compare_diann.py | 58 - scripts/conflict_features.py | 334 --- scripts/copy_irt.py | 31 - scripts/deeplc_calibrate.py | 79 - scripts/empirical_reanalyse.py | 98 - scripts/entrapment_corrected.py | 128 -- scripts/entrapment_holdout.py | 105 - scripts/feature_ablation.py | 352 --- scripts/feature_auc.py | 31 - scripts/feature_audit.py | 723 ------- scripts/ft_calibrate_eval.py | 124 -- scripts/ft_epoch_eval.py | 139 -- scripts/import_diann_decoys.py | 72 - scripts/localization.py | 285 --- scripts/make_mutation_decoys_fasta.py | 60 - scripts/nn_semisupervised_rescore.ipynb | 567 ----- scripts/normalize_output.py | 295 --- scripts/patch_irt.py | 46 - scripts/peak_selection_model.py | 193 -- scripts/plot_ids.py | 235 -- scripts/reference_apex_topk.py | 470 ---- scripts/rt_anchor.py | 122 -- scripts/rt_anchor_extract.py | 136 -- scripts/search_space_manifest.py | 564 ----- .../01_workflow_and_gap_analysis.md | 224 -- sensitivity_plan/03_feature_evaluation.md | 439 ---- .../04_peak_and_peptide_competition.md | 305 --- sensitivity_plan/05_experiment_matrix.md | 199 -- .../06_agent_implementation_backlog.md | 372 ---- sensitivity_plan/ARCHITECTURE_MAP.md | 634 ------ sensitivity_plan/BENCHMARK_GUIDE.md | 291 --- sensitivity_plan/FEATURE_REGISTRY.md | 648 ------ sensitivity_plan/IMPLEMENTATION_STATUS.md | 188 -- sensitivity_plan/NEXT_STEPS.md | 132 -- sensitivity_plan/README.md | 91 - 44 files changed, 4 insertions(+), 12454 deletions(-) delete mode 100644 feature_registry.yaml delete mode 100644 scripts/analyze_entrapment.py delete mode 100644 scripts/benchmark_report.py delete mode 100644 scripts/build_funnel.py delete mode 100644 scripts/candidate_diagnostics.py delete mode 100644 scripts/combine_mutdec.py delete mode 100644 scripts/compare_diann.py delete mode 100644 scripts/conflict_features.py delete mode 100644 scripts/copy_irt.py delete mode 100644 scripts/deeplc_calibrate.py delete mode 100644 scripts/empirical_reanalyse.py delete mode 100644 scripts/entrapment_corrected.py delete mode 100644 scripts/entrapment_holdout.py delete mode 100644 scripts/feature_ablation.py delete mode 100644 scripts/feature_auc.py delete mode 100644 scripts/feature_audit.py delete mode 100644 scripts/ft_calibrate_eval.py delete mode 100644 scripts/ft_epoch_eval.py delete mode 100644 scripts/import_diann_decoys.py delete mode 100644 scripts/localization.py delete mode 100644 scripts/make_mutation_decoys_fasta.py delete mode 100644 scripts/nn_semisupervised_rescore.ipynb delete mode 100644 scripts/normalize_output.py delete mode 100644 scripts/patch_irt.py delete mode 100644 scripts/peak_selection_model.py delete mode 100644 scripts/plot_ids.py delete mode 100644 scripts/reference_apex_topk.py delete mode 100644 scripts/rt_anchor.py delete mode 100644 scripts/rt_anchor_extract.py delete mode 100644 scripts/search_space_manifest.py delete mode 100644 sensitivity_plan/01_workflow_and_gap_analysis.md delete mode 100644 sensitivity_plan/03_feature_evaluation.md delete mode 100644 sensitivity_plan/04_peak_and_peptide_competition.md delete mode 100644 sensitivity_plan/05_experiment_matrix.md delete mode 100644 sensitivity_plan/06_agent_implementation_backlog.md delete mode 100644 sensitivity_plan/ARCHITECTURE_MAP.md delete mode 100644 sensitivity_plan/BENCHMARK_GUIDE.md delete mode 100644 sensitivity_plan/FEATURE_REGISTRY.md delete mode 100644 sensitivity_plan/IMPLEMENTATION_STATUS.md delete mode 100644 sensitivity_plan/NEXT_STEPS.md delete mode 100644 sensitivity_plan/README.md diff --git a/feature_registry.yaml b/feature_registry.yaml deleted file mode 100644 index 0e7ecf8..0000000 --- a/feature_registry.yaml +++ /dev/null @@ -1,1922 +0,0 @@ -total_features: 360 -n_families: 17 -features: - rt_error_abs: - family: minimal - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - rt_error_rel: - family: minimal - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_matched_fragments: - family: minimal - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - coelution_run: - family: minimal - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - log_apex_intensity: - family: minimal - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - frag_corr: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - frag_cosine: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - spectral_angle: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - coelution_mean: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - coelution_best: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_coelution_above: - family: minimal - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - charge: - family: minimal - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - peptide_length: - family: minimal - level: candidate - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_proteins: - family: minimal - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - library_norm_manhattan: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - library_rmsd: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - xcorr_coelution: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - xcorr_shape: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - sum_b_intensity: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - sum_y_intensity: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - diff_by_intensity: - family: rich - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_b_ions: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_y_ions: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - weighted_mass_error: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - mean_mass_error: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - isotope_corr: - family: rich - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - ms1_isom1_ratio: - family: rich - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - log_mono_ms1: - family: rich - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - has_ms1: - family: rich - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - log_sn: - family: rich - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_observations: - family: rich - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - base_width_rt: - family: rich - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - seed_score: - family: rich - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - seed_identified: - family: rich - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - matched_fraction: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - profile_cos: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - ref_corr: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - best_ref_corr: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - low_frag_coel: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - evidence: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - contrast_min: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - resid_corr: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - coel_clean: - family: rich - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - shadow_frac: - family: rich - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - peak_contested_frac: - family: extended-extra (psms-derived) - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - n_charge_states: - family: extended-extra (cross-candidate) - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - charge_multi_flag: - family: extended-extra (cross-candidate) - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - cross_charge_intensity_log: - family: extended-extra (cross-candidate) - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features.rs - spectrum_cosine_matched: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectrum_cosine_sqrt: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectrum_cosine_log: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_angle@similarity: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - dropped_collision: true - spectral_angle_sqrt: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_angle_matched: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - pearson_intensity_matched: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - pearson_intensity_log: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spearman_intensity: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spearman_intensity_matched: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - kendall_tau_intensity: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - dot_product_raw: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - dot_product_norm: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - library_recall_intensity: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - manhattan_sim: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - manhattan_sqrt: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - rmsd_norm: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - mae_norm: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - mse_log: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - mae_weighted_pred: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - abs_diff_q3: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - max_positive_residual: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - chebyshev_dist: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - minkowski_p3: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - bray_curtis: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - bray_curtis_sqrt: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - canberra: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - canberra_matched: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - wave_hedges: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - chi_square_pearson: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - chi_square_symmetric: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - divergence_distance: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - bhattacharyya_coef: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - hellinger: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - squared_chord: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - harmonic_mean_sim: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - jaccard_presence: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - dice_presence: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - intensity_weighted_pearson: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - regression_slope: - family: similarity - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - gini_diff: - family: similarity - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - wasserstein_mz: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - footrule_norm: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - rank_overlap_top3: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - top1_frag_match: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - top1_predicted_observed: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - frac_top3_predicted_observed: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - count_strong_predicted_absent: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - frac_predicted_absent: - family: similarity - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - pearson_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_angle_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_fullwindow: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - stein_scott_weighted_dot: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - log_dot_product: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_log_evidence: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - scribe_score: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - log_dot_product_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_log_evidence_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - scribe_score_area: - family: similarity - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_high_ordinal: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_robust_trim1: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_robust_trim2: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - cosine_robust_trim3: - family: similarity - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/similarity.rs - spectral_entropy_similarity: - family: entropy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - weighted_spectral_entropy_similarity: - family: entropy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - spectral_entropy_similarity_sqrt: - family: entropy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - spectral_entropy_similarity_topk: - family: entropy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - spectral_entropy_similarity_area: - family: entropy - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - jensen_shannon_divergence: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - jeffreys_divergence: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - kl_obs_pred: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - kl_pred_obs: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - cross_entropy_obs_pred: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - obs_spectrum_entropy: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - pred_spectrum_entropy: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - entropy_diff: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - entropy_ratio: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - obs_normalized_entropy: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - normalized_entropy_diff: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - residual_spectrum_entropy: - family: entropy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - entropy_weight_obs: - family: entropy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/entropy.rs - frag_ref_corr_mean: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_obsweighted: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_min: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_std: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_sq_mean: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_topk_weighted: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - n_frag_ref_corr_above_0_9: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frac_frag_ref_corr_above_0_8: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_ref_corr_mean_full: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - full_vs_peak_corr_gain: - family: coelution - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_weighted: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_min: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_median: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_std: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_frac_negative: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_hi: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - pairwise_coelution_lo: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - coelution_hi_lo_contrast: - family: coelution - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - coelution_corr_entropy: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_shape_mean: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_shape_min: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_shape_std: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_mean_abs: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_std: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_iqr: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_frac_zero: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_max_abs: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - xcorr_lag_entropy: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - ref_xcorr_lag_mean: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - ref_xcorr_shape_mean: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - observed_sum_vs_template_corr: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_loo_ref_corr_mean: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frag_loo_ref_corr_min: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - frac_frags_apex_aligned: - family: coelution - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - top3_frag_ref_corr: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - by_cross_coelution: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - by_cross_lag_mean: - family: coelution - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - charge_cross_coelution: - family: coelution - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/coelution.rs - explained_variance_ref: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - profile_residual_fraction: - family: interference - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - n_interfered_fragments: - family: interference - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - corrected_vs_raw_cos: - family: interference - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - corrected_vs_raw_ratio: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - ifs_removed_count: - family: interference - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - ifs_removed_intensity_frac: - family: interference - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - ifs_corr_gain: - family: interference - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - ifs_retained_frac: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - matched_frac_after_ifs: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - peak_to_full_area_ratio_profile: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - peak_to_full_area_ratio_frag_mean: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - peak_to_full_area_ratio_weighted: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - out_of_peak_intensity_frac: - family: interference - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - profile_corr_full_vs_peak_delta: - family: interference - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - frac_frag_ref_corr_below_0_5: - family: interference - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - explained_apex_intensity_frac: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - apex_purity: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - interference_apex_residual_fraction: - family: interference - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - dominant_frag_ref_corr: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - explained_variance_ratio: - family: interference - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - second_component_fraction: - family: interference - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - profile_second_peak_ratio: - family: interference - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - n_competing_peaks_in_window: - family: interference - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - matched_pred_intensity_fraction: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - top_pred_frag_matched: - family: interference - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/interference.rs - gaussian_fit_r2: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - gaussian_cosine: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - emg_fit_improvement: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_prominence: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - profile_peak_snr: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - fwhm_seconds: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - fwhm_to_window_ratio: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - width_at_10pct: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - width_ratio_10_50: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - hwhm_asymmetry: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - tailing_factor_usp: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - asymmetry_factor_10pct: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_sharpness: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_curvature: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_to_boundary_ratio: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_dominance: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - zigzag_index: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - jaggedness: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - roughness_2nd_deriv: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - n_local_maxima: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - modality: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - rt_skewness: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - rt_excess_kurtosis: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - rt_std_seconds: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - mean_mode_offset: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - fraction_area_within_fwhm: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - triangle_area_similarity: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - baseline_fraction: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - peak_completeness: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - apex_centering_offset: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - intensity_score: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - total_xic_log: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_fwhm_cv: - family: chromatographic - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_fwhm_mean: - family: chromatographic - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_apex_rt_dispersion: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_apex_rt_dispersion_weighted: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_apex_offset_from_profile_mean: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_gaussianity_mean: - family: chromatographic - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_gaussianity_weighted: - family: chromatographic - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - frag_zigzag_mean: - family: chromatographic - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - sumtrace_unweighted_gaussian_r2: - family: chromatographic - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - reference_profile_rt_entropy_peak: - family: chromatographic - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - reference_profile_rt_entropy_ratio: - family: chromatographic - level: peak - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs - median_abs_frag_ppm: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - signed_mean_frag_ppm: - family: mass_accuracy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - ppm_std: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - ppm_iqr: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - ppm_range: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - max_abs_frag_ppm: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - intensity_weighted_abs_ppm: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - intensity_weighted_signed_ppm: - family: mass_accuracy - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - intensity_weighted_ppm_std: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - lib_weighted_abs_ppm: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - frac_frag_within_half_tol: - family: mass_accuracy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - high_ppm_intensity_frac: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - ppm_intensity_anticorr: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - mass_error_mz_trend: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - mean_abs_mz_error_da: - family: mass_accuracy - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - mass_evidence_gauss: - family: mass_accuracy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - mass_log_evidence: - family: mass_accuracy - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs - n_matched_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - n_matched_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - frac_matched_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - frac_matched_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_count_balance: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_intensity_ratio: - family: ion_series - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_ratio_agreement: - family: ion_series - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_ratio_consistency: - family: ion_series - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - longest_b_run: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - longest_y_run: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - longest_run_max: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - longest_run_frac_length: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - series_coverage_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - series_coverage_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - sequence_coverage: - family: ion_series - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - series_gap_fraction: - family: ion_series - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_complement_count: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_complement_mz_consistency: - family: ion_series - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_complement_coelution: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - ordinal_intensity_concordance_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - ordinal_intensity_concordance_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - series_coelution_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - series_coelution_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - spectral_angle_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - spectral_angle_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - pearson_b: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - pearson_y: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - cosine_charge1: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - cosine_charge2: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - charge_corr_balance: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - mean_matched_ordinal_norm: - family: ion_series - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_ion_contiguous_intensity: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - by_ion_contiguous_lib_frac: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - both_series_present: - family: ion_series - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs - ms1_isotope_cosine_apex: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_spectral_angle_apex: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_chi2_apex: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_manhattan_apex: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_ratio_1_0: - family: ms1 - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_ratio_2_0: - family: ms1 - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_plus1_ratio_dev: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_plus2_ratio_dev: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_minus_one_fraction: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - iso_overlap_flag: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - log_ms1_mono: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_total_isotope_log: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - has_ms1_signal: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_apex_entropy_3: - family: ms1 - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_m1_entropy_contribution: - family: ms1 - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_ms2_time_corr: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_ms2_envelope_time_corr: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_iso_coelution: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_ms2_apex_rt_delta: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_iso_ratio_stability: - family: ms1 - level: precursor - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_mono_gaussianity: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_ms2_fwhm_ratio: - family: ms1 - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_corr_xic: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_envelope_over_time_corr: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - ms1_isotope_xic_shape_consistency: - family: ms1 - level: precursor - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/ms1.rs - rt_error_signed: - family: rt - level: candidate - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_error_abs@rt: - family: rt - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - dropped_collision: true - rt_error_squared: - family: rt - level: candidate - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_error_signed_norm_gradient: - family: rt - level: run - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_error_abs_norm_gradient: - family: rt - level: run - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - observed_rt_raw: - family: rt - level: candidate - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - predicted_rt_raw: - family: rt - level: candidate - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - observed_rt_fraction: - family: rt - level: run - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - predicted_rt_fraction: - family: rt - level: run - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_error_over_peak_width: - family: rt - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_error_over_fwhm: - family: rt - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - rt_diff_profile_apex: - family: rt - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - predicted_rt_in_gradient: - family: rt - level: run - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/rt.rs - log_seed_hyperscore: - family: novel - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - seed_hyperscore_per_matched: - family: novel - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - seed_identified@novel: - family: novel - level: candidate - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - dropped_collision: true - peptide_length@novel: - family: novel - level: candidate - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - dropped_collision: true - precursor_charge: - family: novel - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - charge_is_2: - family: novel - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - charge_is_3: - family: novel - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - charge_is_4plus: - family: novel - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - precursor_mass: - family: novel - level: precursor - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - log_total_matched_intensity: - family: novel - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - n_matched_frags: - family: novel - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - n_predicted_frags: - family: novel - level: fragment - direction: neutral - source_file: rust/mumdia/crates/mumdia/src/stages/features/novel.rs - frag_corr_peakmax: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - frag_cosine_peakmax: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - spectral_angle_peakmax: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - frag_corr_matched_nz: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - frag_cosine_matched_nz: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - peakmax_apex_gain: - family: nonzero - level: fragment - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - n_frag_present_inpeak: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - frac_frag_present_inpeak: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - coelution_mean_bothpos: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - coelution_mean_summpos: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - ref_corr_nz: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - profile_cos_nz: - family: nonzero - level: fragment - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs - rank_corr_vs_apex_mean: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - rank_corr_vs_apex_std: - family: order_consistency - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - rank_corr_adjacent_mean: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - kendall_vs_apex_mean: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - top1_frag_persistence: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - top2_order_persistence: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - argmax_frag_entropy: - family: order_consistency - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - self_cosine_vs_apex_mean: - family: order_consistency - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs - n_peak_scans: - family: peak_scans - level: peak - direction: higher_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/peak_scans.rs - peak_window_degenerate: - family: peak_scans - level: peak - direction: lower_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/peak_scans.rs - frag_apex_rt_std: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - frag_apex_rt_mad: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - frag_apex_max_dev: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - frag_apex_mean_dev: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - frag_apex_agree_frac: - family: apex_dispersion - level: peak_group - direction: higher_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - precursor_frag_apex_delta: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_symmetry: - family: apex_dispersion - level: peak_group - direction: target_0.5 - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_tailing: - family: apex_dispersion - level: peak_group - direction: target_1.0 - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_n_local_maxima: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_shoulder_score: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_fwhm_scans: - family: apex_dispersion - level: peak_group - direction: ambiguous - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - peak_truncation: - family: apex_dispersion - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - apex_frac_of_window: - family: apex_dispersion - level: peak_group - direction: ambiguous - source_file: rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs - frag_mass_err_median: - family: mass_uncertainty - level: peak_group - direction: target_0 - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frag_mass_err_abs_median: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frag_mass_err_std: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frag_mass_err_iqr: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frag_mass_err_max_abs: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frag_mass_err_range: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - effective_frag_count: - family: mass_uncertainty - level: peak_group - direction: higher_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - evidence_concentration: - family: mass_uncertainty - level: peak_group - direction: lower_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frac_top3_pred_observed: - family: mass_uncertainty - level: peak_group - direction: higher_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs - frac_top5_pred_observed: - family: mass_uncertainty - level: peak_group - direction: higher_is_better - source_file: rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs diff --git a/rust/mumdia/crates/mumdia-core/src/config.rs b/rust/mumdia/crates/mumdia-core/src/config.rs index f7176cf..dc8a772 100644 --- a/rust/mumdia/crates/mumdia-core/src/config.rs +++ b/rust/mumdia/crates/mumdia-core/src/config.rs @@ -31,32 +31,17 @@ impl Default for DecoyStrategy { } } -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum DecoySource { - /// MuMDIA generates decoys; the seed engine is told not to. - Library, - SearchEngine, -} -impl Default for DecoySource { - fn default() -> Self { - DecoySource::Library - } -} - /// Fragment-matcher backend for search-seed and extract (fragindex_spec). /// Default `Fragindex` (log-bin CSR matcher): on narrow-window DIA it is ~1.95x /// faster in search-seed and ~1.26x in extract with essentially unchanged IDs /// (HYE B_01: peptides -0.1%); `Bucketed` is the previous `Library::page_search` /// path (retained for A/B and for the AIF full-range-window case, where the -/// predicate difference shifts IDs more); `Naive` is the band-join reference -/// (equivalence tests only). +/// predicate difference shifts IDs more). #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum MatcherKind { Bucketed, Fragindex, - Naive, } impl Default for MatcherKind { fn default() -> Self { @@ -78,20 +63,6 @@ impl Default for Enzyme { } } -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum ToleranceRegime { - /// MVP-conservative: fixed ppm tolerances (PLAN.md Section 10). - Fixed, - /// v1: learned per-run percentile-driven optimizer (PLAN.md Section 8.4). - LearnedPercentile, -} -impl Default for ToleranceRegime { - fn default() -> Self { - ToleranceRegime::Fixed - } -} - #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum CalibrationMethod { @@ -111,7 +82,6 @@ pub enum FeatureSet { /// MVP feature set (PLAN.md Section 10). Minimal, Rich, - Custom, /// Minimal + Rich + the extended battery (DIA-NN / OpenSWATH / AlphaDIA / /// MS2Rescore / OktoberFest analogs + novel families) from the per-family /// modules in `stages/features/`. Superset, opt-in; the classifier picks the @@ -222,18 +192,6 @@ impl Default for PeakClaim { } } -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum ScanWindowMode { - PeakWidthDerived, - Fixed, -} -impl Default for ScanWindowMode { - fn default() -> Self { - ScanWindowMode::PeakWidthDerived - } -} - // --------------------------------------------------------------------------- // Per-stage config // --------------------------------------------------------------------------- @@ -249,16 +207,10 @@ where #[serde(default, deny_unknown_fields)] pub struct DecoyConfig { pub strategy: DecoyStrategy, - pub source: DecoySource, - pub ratio: u32, } impl Default for DecoyConfig { fn default() -> Self { - Self { - strategy: t(), - source: t(), - ratio: 1, - } + Self { strategy: t() } } } @@ -373,7 +325,6 @@ impl Default for PredictFragConfig { #[serde(default, deny_unknown_fields)] pub struct SearchSeedConfig { pub fdr_seed: f64, - pub precursor_tol_ppm: f64, pub fragment_tol_ppm: f64, /// Max reported PSMs per spectrum (wide-window DIA, PLAN.md Stage S). pub report_psms: usize, @@ -398,7 +349,6 @@ impl Default for SearchSeedConfig { fn default() -> Self { Self { fdr_seed: 0.01, - precursor_tol_ppm: 20.0, fragment_tol_ppm: 20.0, report_psms: 5, min_matched_peaks: 4, @@ -412,7 +362,6 @@ impl Default for SearchSeedConfig { #[derive(Clone, Debug, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct RtImTrainConfig { - pub tolerance_regime: ToleranceRegime, pub calibration_method: CalibrationMethod, pub q_train: f64, /// Percentile of |obs - calibrated_pred| residuals for the RT window. @@ -459,7 +408,6 @@ pub struct RtImTrainConfig { impl Default for RtImTrainConfig { fn default() -> Self { Self { - tolerance_regime: t(), calibration_method: t(), q_train: 0.01, p_rt: 0.95, @@ -481,8 +429,6 @@ impl Default for RtImTrainConfig { #[derive(Clone, Debug, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct ExtractConfig { - pub scan_window_mode: ScanWindowMode, - pub scan_scale: f64, pub fixed_scan_window: usize, pub frag_tol_ppm: f64, pub prec_tol_ppm: f64, @@ -535,12 +481,8 @@ pub struct ExtractConfig { /// window), so the elution profile drops to zero between peaks and the /// features-stage boundary calling is not misled by interpolated gaps. pub emit_window_grid: bool, - /// tier-(c) k-select cap: exact scores run on at most this many per cell. - pub k_select: usize, /// m/z bucket size (power of two). pub bucket_size: usize, - /// Max fragment charge to probe (deconvolution loop bound). - pub max_fragment_charge: i32, /// How a shared observed peak's intensity is apportioned among co-isolated, /// co-eluting candidates that all match it (see [`PeakClaim`]). pub peak_claim: PeakClaim, @@ -606,8 +548,6 @@ pub struct ExtractConfig { impl Default for ExtractConfig { fn default() -> Self { Self { - scan_window_mode: ScanWindowMode::Fixed, // MVP-conservative - scan_scale: 2.2, fixed_scan_window: 3, frag_tol_ppm: 20.0, prec_tol_ppm: 20.0, @@ -627,9 +567,7 @@ impl Default for ExtractConfig { apex_count_window: 1, // no rolling smoothing by default (opt-in; window 5 // cuts AIF apex misassignment, median |dRT| 131s->9s) emit_window_grid: true, // zero-filled window-grid chromatograms - k_select: 50, bucket_size: 8192, - max_fragment_charge: 1, peak_claim: PeakClaim::None, emit_contested_features: false, peak_claim_margin: 2.0, @@ -791,20 +729,6 @@ pub enum CompetitionMode { MarginGated, } -impl CompetitionMode { - /// Parse a CLI/token spelling. - pub fn from_token(s: &str) -> Option { - match s.to_ascii_lowercase().replace('-', "_").as_str() { - "winner_take_all" | "winner" => Some(Self::WinnerTakeAll), - "none" => Some(Self::None), - "features_only" | "features" => Some(Self::FeaturesOnly), - "unique_evidence" | "unique" => Some(Self::UniqueEvidence), - "margin_gated" | "margin" => Some(Self::MarginGated), - _ => None, - } - } -} - #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum CompeteGroupBy { @@ -1085,7 +1009,6 @@ impl Default for RescoreConfig { #[serde(default, deny_unknown_fields)] pub struct Config { pub rng_seed: u64, - pub threads: Option, pub digest: DigestConfig, pub peptidoforms: PeptidoformsConfig, pub predict_frag: PredictFragConfig, @@ -1103,7 +1026,6 @@ impl Default for Config { fn default() -> Self { Self { rng_seed: 0, - threads: None, digest: t(), peptidoforms: t(), predict_frag: t(), @@ -1165,36 +1087,6 @@ impl Config { .into(), )); } - // Warn (not fail) when a declared-but-unimplemented knob is set away from - // its default: it silently has no effect, which otherwise misleads tuning. - let d = Self::default(); - let mut dead = Vec::new(); - if self.search_seed.precursor_tol_ppm != d.search_seed.precursor_tol_ppm { - dead.push("search_seed.precursor_tol_ppm"); - } - if self.rt_im_train.tolerance_regime != d.rt_im_train.tolerance_regime { - dead.push("rt_im_train.tolerance_regime"); - } - if self.extract.k_select != d.extract.k_select { - dead.push("extract.k_select"); - } - if self.extract.max_fragment_charge != d.extract.max_fragment_charge { - dead.push("extract.max_fragment_charge"); - } - if self.extract.scan_scale != d.extract.scan_scale { - dead.push("extract.scan_scale"); - } - if self.digest.decoy.source != d.digest.decoy.source { - dead.push("digest.decoy.source"); - } - if self.digest.decoy.ratio != d.digest.decoy.ratio { - dead.push("digest.decoy.ratio"); - } - for k in &dead { - eprintln!( - "config warning: `{k}` is set but not implemented in the engine; it has no effect" - ); - } Ok(()) } @@ -1237,7 +1129,6 @@ mod tests { let back: Config = serde_json::from_str(&j).unwrap(); assert_eq!(back.digest.min_len, 5); assert_eq!(back.features.set, FeatureSet::Minimal); - assert_eq!(back.rt_im_train.tolerance_regime, ToleranceRegime::Fixed); assert_eq!(back.search_seed.top_n_peaks, 300); } diff --git a/rust/mumdia/crates/mumdia/src/stages/extract.rs b/rust/mumdia/crates/mumdia/src/stages/extract.rs index 887a8d8..c856ffb 100644 --- a/rust/mumdia/crates/mumdia/src/stages/extract.rs +++ b/rust/mumdia/crates/mumdia/src/stages/extract.rs @@ -760,12 +760,7 @@ pub fn run(p: ExtractParams) -> Result<(u64, u64)> { info!(materialized = acc.len(), "extract: candidates with evidence"); // Cascade + apex per candidate. - let scan_window = match p.cfg.scan_window_mode { - mumdia_core::config::ScanWindowMode::Fixed => p.cfg.fixed_scan_window, - // MVP: data-derived mode approximated by fixed default (v1 optimizer later) - mumdia_core::config::ScanWindowMode::PeakWidthDerived => p.cfg.fixed_scan_window, - } - .max(1); + let scan_window = p.cfg.fixed_scan_window.max(1); // psms_extracted columns let (mut cid_c, mut apexrt_c, mut apexint_c) = (Vec::new(), Vec::new(), Vec::new()); diff --git a/rust/mumdia/crates/mumdia/src/stages/features.rs b/rust/mumdia/crates/mumdia/src/stages/features.rs index 8dac336..e09f6d1 100644 --- a/rust/mumdia/crates/mumdia/src/stages/features.rs +++ b/rust/mumdia/crates/mumdia/src/stages/features.rs @@ -205,7 +205,7 @@ pub const RICH_EXTRA: &[&str] = &[ /// The ordered active feature list for the configured set. pub fn active_features(set: FeatureSet) -> Vec { let mut v: Vec = MINIMAL_FEATURES.iter().map(|s| s.to_string()).collect(); - if matches!(set, FeatureSet::Rich | FeatureSet::Custom | FeatureSet::Extended) { + if matches!(set, FeatureSet::Rich | FeatureSet::Extended) { v.extend(RICH_EXTRA.iter().map(|s| s.to_string())); } if matches!(set, FeatureSet::Extended) { diff --git a/scripts/analyze_entrapment.py b/scripts/analyze_entrapment.py deleted file mode 100644 index 8920a4e..0000000 --- a/scripts/analyze_entrapment.py +++ /dev/null @@ -1,60 +0,0 @@ -"""Entrapment-FDR analysis. The sample is pure E. coli; the library also contains -human (entrapment) peptides. Any human identification is false by construction, -so it measures the TRUE FDR independent of decoys and of DIA-NN. - -FDR_true(ecoli) ~= n_human_ids * (n_ecoli_lib / n_human_lib) / n_ecoli_ids -(false IDs are assumed to hit E. coli and human in proportion to library size). - -Usage: python analyze_entrapment.py [q] -""" -import sys -import re -import pandas as pd - -scored = sys.argv[1] -Q = float(sys.argv[2]) if len(sys.argv) > 2 else 0.01 - -# library sizes (E. coli vs human-only target peptides) -pep = pd.read_parquet("out_full/pep_ent.parquet") -is_hum = pep.protein.str.contains("_HUMAN") & ~pep.protein.str.contains("_ECOLI") -n_hum_lib = int(is_hum.sum()) -n_eco_lib = int((~is_hum).sum()) -ratio = n_eco_lib / n_hum_lib - -def strip(pf): - return re.sub(r"\[[^\]]*\]", "", str(pf).replace("DECOY_", "")) - -s = pd.read_parquet(scored) -t = s[(s.label == "target") & (s.q_value <= Q)].copy() -prot = "protein_group" if "protein_group" in t.columns else "protein" -t["human"] = t[prot].str.contains("_HUMAN") & ~t[prot].str.contains("_ECOLI") -t["strip"] = t.peptidoform.map(strip) - -eco = t[~t.human]; hum = t[t.human] -n_eco = eco.strip.nunique(); n_hum = hum.strip.nunique() -fdr = n_hum * ratio / max(1, n_eco) - -# Contaminant-aware correction: real handling contaminants (keratin/albumin/Hb/ -# actin/Ig) inside the spike-in proteome are genuinely present, so counting them -# as false over-estimates the FDR. Report the corrected FDR excluding them. -import re as _re -_CRAP = _re.compile(r"K[12][CH]\d|K22E|K1H\d|K2M\d|KRT\d|KRHB|ALBU|HB[ABDGE]\d?_|" - r"ACT[BGSC]_|TBB\d|TBA\d|TRY[PB]|IGHG|IGKC|IGLC", _re.I) -hum_noncontam = hum[~hum[prot].map(lambda p: bool(_CRAP.search(str(p))))] -n_hum_nc = hum_noncontam.strip.nunique() -fdr_nc = n_hum_nc * ratio / max(1, n_eco) - -# decoy TDA FDR for comparison -d = s[(s.label == "decoy") & (s.q_value <= Q)] -tda = len(d) / max(1, len(t)) - -# DIA-NN concordance of the E. coli IDs -dn = pd.read_csv("out_diann/report.tsv", sep="\t") -diann = set(dn[dn["Q.Value"] <= 0.01]["Stripped.Sequence"].unique()) -ov = len(set(eco.strip) & diann) - -print(f"library: ecoli_lib={n_eco_lib} human_lib={n_hum_lib} ratio(eco/hum)={ratio:.3f}") -print(f"@q<={Q}: ecoli_ids={n_eco} human(entrapment)_ids={n_hum}") -print(f" ENTRAPMENT true FDR (ecoli) ~= {fdr*100:.1f}% [reported TDA decoy FDR = {tda*100:.2f}%]") -print(f" contaminant-corrected true FDR (cRAP excluded, false human seqs {n_hum}->{n_hum_nc}) ~= {fdr_nc*100:.1f}%") -print(f" ecoli IDs in DIA-NN: {ov} ({ov/max(1,n_eco)*100:.1f}% concordance) | ecoli_ids as %of DIA-NN = {n_eco/len(diann)*100:.1f}%") diff --git a/scripts/benchmark_report.py b/scripts/benchmark_report.py deleted file mode 100644 index facaa1e..0000000 --- a/scripts/benchmark_report.py +++ /dev/null @@ -1,768 +0,0 @@ -#!/usr/bin/env python -"""Self-contained HTML benchmark report for MuMDIA sensitivity diagnostics. - -Implements sensitivity_plan backlog P7.1 (spec 02 Section 8, spec 05). Assembles -the diagnostics this project already produces into a single, portable HTML file: - - * identification-loss waterfall (candidate audit metrics + stage counts); - * candidate-audit stratification (rejection reason by charge / entrapment); - * reference-apex top-K peak recall (self, and reference if present); - * feature-family ablation (per model, most useful first); - * empirical entrapment FDP (parsed from the holdout harness stdout). - -Non-invasive: it only reads existing JSON / CSV / Parquet outputs and writes one -HTML file with all CSS inlined and every chart embedded as a base64 PNG data URI. -There are no external assets and no network access, so the report opens correctly -straight from disk. Any missing input renders as "not provided" for its section. - -Deterministic: analysis and charts are reproducible for the same inputs. Only the -header timestamp varies (fix it with --stamp). Charts require matplotlib; if it is -unavailable the tables still render and the chart panels note the omission. - -Interpreter: C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe - (pyarrow, pandas, numpy, matplotlib). - -Usage: - python benchmark_report.py --out report.html - [--audit-metrics candidate_audit.parquet.metrics.json] - [--audit candidate_audit.parquet] - [--topk topk.json] - [--ablation ablation_dir_or_csv] - [--entrapment entrapment_stdout.txt] - [--title "..."] [--stamp "2026-07-17 12:00"] -""" - -from __future__ import annotations - -import argparse -import base64 -import csv -import datetime as _dt -import html -import io -import json -import os -import re -import sys - -# --------------------------------------------------------------------------- # -# matplotlib is optional at runtime: charts are omitted gracefully if absent. -try: - import matplotlib - - matplotlib.use("Agg") - import matplotlib.pyplot as plt - - _HAVE_MPL = True -except Exception: # pragma: no cover - defensive - _HAVE_MPL = False - -# Fixed PNG metadata so byte output does not carry a matplotlib version string or -# creation time, keeping identical inputs byte-reproducible across runs. -_PNG_META = {"Software": "mumdia-benchmark-report"} - -# Canonical ordering of known rejection reasons, following the pipeline ladder -# (spec 02 Section 8). Unknown reasons are appended in first-seen order. -_REASON_ORDER = [ - "NOT_IN_SEARCH_SPACE", - "NO_CANDIDATE", - "NOT_GENERATED", - "NO_FRAGMENT_TRACES", - "NO_VALID_FRAGMENTS", - "NO_PEAK_GROUP", - "RT_PRUNED", - "PEAK_NOT_SELECTED", - "WRONG_VARIANT", - "OUTCOMPETED_PEAK", - "OUTCOMPETED_PEPTIDE", - "OUTCOMPETED_PEPTIDOFORM", - "OUTCOMPETED_LOCALIZATION", - "OUTCOMPETED_DUPLICATE", - "OUTCOMPETED_TARGET_DECOY", - "FAILED_PRECURSOR_FDR", - "FAILED_PEPTIDE_FDR", - "FAILED_PROTEIN_FDR", - "NOT_REPORTED", - "REPORTED", -] -_REASON_RANK = {r: i for i, r in enumerate(_REASON_ORDER)} - - -# --------------------------------------------------------------------------- # -# small helpers -def esc(x) -> str: - return html.escape("" if x is None else str(x)) - - -def fmt_int(x) -> str: - try: - return f"{int(round(float(x))):,}" - except (TypeError, ValueError): - return esc(x) - - -def fmt_pct(x, digits: int = 1) -> str: - try: - return f"{float(x) * 100:.{digits}f}%" - except (TypeError, ValueError): - return esc(x) - - -def fmt_num(x, digits: int = 4) -> str: - try: - return f"{float(x):.{digits}g}" - except (TypeError, ValueError): - return esc(x) - - -def load_json(path): - if not path or not os.path.isfile(path): - return None - with open(path, "r", encoding="utf-8") as fh: - return json.load(fh) - - -def reason_sort_key(reason: str): - return (_REASON_RANK.get(reason, len(_REASON_ORDER)), str(reason)) - - -def fig_to_data_uri(fig) -> str: - buf = io.BytesIO() - fig.savefig(buf, format="png", dpi=110, bbox_inches="tight", - facecolor="white", metadata=_PNG_META) - plt.close(fig) - b64 = base64.b64encode(buf.getvalue()).decode("ascii") - return f"data:image/png;base64,{b64}" - - -def not_provided(msg: str = "not provided") -> str: - return f'

{esc(msg)}

' - - -def html_table(headers, rows, align_right_from: int = 1) -> str: - """Render a responsive table. Columns from `align_right_from` are right-aligned.""" - th = "".join( - f'= align_right_from else ""}">{esc(h)}' - for i, h in enumerate(headers) - ) - body = [] - for row in rows: - tds = "".join( - f'= align_right_from else ""}">{c}' - for i, c in enumerate(row) - ) - body.append(f"{tds}") - return ( - '
' - f"{th}" - f'{"".join(body)}' - "
" - ) - - -def img_panel(uri: str, caption: str = "") -> str: - cap = f'
{esc(caption)}
' if caption else "" - return f'
{esc(caption)}{cap}
' - - -# --------------------------------------------------------------------------- # -# Section 2: identification-loss waterfall -def section_waterfall(metrics): - if not metrics: - return not_provided() - - wf = metrics.get("waterfall") or {} - parts = [] - - # stage-count funnel - stage_rows = [] - for key, label in ( - ("search_space", "Search space (candidates)"), - ("extracted", "Extracted"), - ("competed", "Competed"), - ("reported", "Reported"), - ): - if key in metrics and metrics[key] is not None: - stage_rows.append((label, fmt_int(metrics[key]))) - if metrics.get("trace_recall") is not None: - stage_rows.append(("Trace recall (extracted / search space)", - fmt_pct(metrics["trace_recall"], 3))) - if metrics.get("q_threshold") is not None: - stage_rows.append(("q threshold", fmt_num(metrics["q_threshold"]))) - if metrics.get("run_id"): - stage_rows.append(("Run id", esc(metrics["run_id"]))) - if stage_rows: - parts.append("

Stage counts

") - parts.append(html_table(["Stage", "Value"], stage_rows)) - - if not wf: - parts.append("

Waterfall

") - parts.append(not_provided("no waterfall block in metrics")) - return "".join(parts) - - ordered = sorted(wf.items(), key=lambda kv: reason_sort_key(kv[0])) - total = sum(v for _, v in ordered) or metrics.get("search_space") or 1 - - # table - wf_rows = [ - (esc(reason), fmt_int(count), fmt_pct(count / total, 2)) - for reason, count in ordered - ] - parts.append("

Earliest-loss waterfall

") - parts.append(html_table(["Rejection reason", "Candidates", "% of total"], wf_rows)) - - # chart (log scale: the earliest bucket dominates by orders of magnitude) - if _HAVE_MPL: - labels = [r for r, _ in ordered] - counts = [c for _, c in ordered] - fig, ax = plt.subplots(figsize=(8.2, max(2.2, 0.55 * len(labels) + 1.0))) - ypos = list(range(len(labels)))[::-1] - bars = ax.barh(ypos, counts, color="#3f6fb0", edgecolor="#26456f") - ax.set_yticks(ypos) - ax.set_yticklabels(labels, fontsize=9) - ax.set_xscale("log") - ax.set_xlabel("candidates (log scale)") - ax.set_title("Identification-loss waterfall") - for rect, c in zip(bars, counts): - ax.text(rect.get_width() * 1.05, rect.get_y() + rect.get_height() / 2, - f"{c:,}", va="center", ha="left", fontsize=8, color="#333") - ax.margins(x=0.18) - ax.grid(axis="x", linestyle=":", alpha=0.4) - parts.append(img_panel(fig_to_data_uri(fig), - "Candidates lost at each stage (log-scaled).")) - else: - parts.append(not_provided("matplotlib unavailable: chart omitted")) - - return "".join(parts) - - -# --------------------------------------------------------------------------- # -# Section 3: candidate-audit stratification -def _crosstab(df, index_col, col_col, col_order=None, col_fmt=str): - import pandas as pd - - ct = pd.crosstab(df[index_col], df[col_col]) - # order rows by the pipeline ladder - ct = ct.reindex(sorted(ct.index, key=reason_sort_key)) - if col_order is not None: - cols = [c for c in col_order if c in ct.columns] - cols += [c for c in ct.columns if c not in cols] - else: - cols = sorted(ct.columns, key=lambda c: (str(type(c)), c)) - # reindex (not ct[cols]) so a list of booleans is read as labels, not a mask - ct = ct.reindex(columns=cols) - headers = ["Rejection reason"] + [col_fmt(c) for c in ct.columns] + ["Total"] - rows = [] - for reason, series in ct.iterrows(): - vals = [fmt_int(v) for v in series.tolist()] - rows.append([esc(reason)] + vals + [fmt_int(series.sum())]) - totals = ["Total"] + [f"{fmt_int(ct[c].sum())}" for c in ct.columns] - totals += [f"{fmt_int(ct.values.sum())}"] - rows.append(totals) - return html_table(headers, rows) - - -def section_stratification(audit_path): - if not audit_path or not os.path.isfile(audit_path): - return not_provided() - try: - import pyarrow.parquet as pq - except Exception as exc: # pragma: no cover - defensive - return not_provided(f"pyarrow unavailable: {exc}") - - schema = pq.read_schema(audit_path) - want = [c for c in ("rejection_reason", "charge", "entrapment_label", - "target_decoy_label") if c in schema.names] - if "rejection_reason" not in want: - return not_provided("no rejection_reason column in audit parquet") - - df = pq.read_table(audit_path, columns=want).to_pandas() - n = len(df) - parts = [f'

{fmt_int(n)} candidate rows.

'] - - if "charge" in df.columns: - charges = sorted(c for c in df["charge"].dropna().unique()) - parts.append("

Rejection reason by charge

") - parts.append(_crosstab(df, "rejection_reason", "charge", - col_order=charges, col_fmt=lambda c: f"charge {int(c)}")) - - if "entrapment_label" in df.columns: - parts.append("

Rejection reason by entrapment label

") - parts.append(_crosstab(df, "rejection_reason", "entrapment_label", - col_order=[False, True], - col_fmt=lambda c: f"entrapment={bool(c)}")) - - if "target_decoy_label" in df.columns: - parts.append("

Rejection reason by target / decoy

") - parts.append(_crosstab(df, "rejection_reason", "target_decoy_label", - col_order=["target", "decoy"], - col_fmt=str)) - - return "".join(parts) - - -# --------------------------------------------------------------------------- # -# Section 4: top-K peak recall -_TOPK_KEYS = [ - ("frac_rank1", "Selected apex is peak rank-1"), - ("frac_top3", "Selected apex in top-3"), - ("frac_top5", "Selected apex in top-5"), - ("frac_top10", "Selected apex in top-10"), - ("frac_no_peak", "Selected apex in no enumerated peak"), -] - - -def section_topk(topk): - if not topk: - return not_provided() - - parts = [] - params = topk.get("params") or {} - ctx_rows = [] - for key, label in ( - ("n_candidates_total", "Candidates total"), - ("n_candidates_selected", "Candidates selected"), - ("n_candidates_processed", "Candidates processed"), - ("n_no_chrom_or_empty", "No chromatogram / empty"), - ): - if topk.get(key) is not None: - ctx_rows.append((label, fmt_int(topk[key]))) - if params.get("rt_tol_s") is not None: - ctx_rows.append(("RT tolerance (s)", fmt_num(params["rt_tol_s"]))) - if params.get("top_frags") is not None: - ctx_rows.append(("Top fragments", fmt_int(params["top_frags"]))) - if ctx_rows: - parts.append("

Context

") - parts.append(html_table(["Item", "Value"], ctx_rows)) - - self_m = topk.get("self") - if not self_m: - parts.append(not_provided("no 'self' block in topk json")) - return "".join(parts) - - rows = [] - for key, label in _TOPK_KEYS: - if key in self_m and self_m[key] is not None: - rows.append((label, fmt_pct(self_m[key], 2))) - for key, label in ( - ("mean_peaks_per_candidate", "Mean peaks / candidate"), - ("median_peaks_per_candidate", "Median peaks / candidate"), - ("frac_ge2_peaks", "Candidates with >= 2 peaks"), - ("n_apex_matched_to_peak", "Apexes matched to a peak"), - ("denominator", "Denominator"), - ): - if key in self_m and self_m[key] is not None: - val = fmt_pct(self_m[key], 2) if key == "frac_ge2_peaks" else ( - fmt_num(self_m[key]) if "peaks_per" in key else fmt_int(self_m[key])) - rows.append((label, val)) - parts.append("

Self peak recall

") - parts.append(html_table(["Metric", "Value"], rows)) - - # cumulative top-K bar chart - if _HAVE_MPL: - bar_keys = [("frac_rank1", "rank-1"), ("frac_top3", "top-3"), - ("frac_top5", "top-5"), ("frac_top10", "top-10")] - xs = [lbl for k, lbl in bar_keys if self_m.get(k) is not None] - ys = [self_m[k] for k, _ in bar_keys if self_m.get(k) is not None] - if ys: - fig, ax = plt.subplots(figsize=(6.0, 3.2)) - bars = ax.bar(xs, ys, color="#4c9a63", edgecolor="#2f6b40") - ax.set_ylim(0, 1.0) - ax.set_ylabel("fraction of candidates") - ax.set_title("Selected apex within cumulative top-K") - for rect, y in zip(bars, ys): - ax.text(rect.get_x() + rect.get_width() / 2, y + 0.02, - f"{y * 100:.1f}%", ha="center", va="bottom", fontsize=9) - ax.grid(axis="y", linestyle=":", alpha=0.4) - parts.append(img_panel(fig_to_data_uri(fig), - "Cumulative fraction where the selected apex is " - "among the top-K enumerated peaks.")) - - # reference (peak-oracle) block, if present - ref = topk.get("reference") - if isinstance(ref, dict) and ref: - ref_rows = [] - for k in ("reference_apex_in_top_1", "reference_apex_in_top_3", - "reference_apex_in_top_5", "reference_apex_in_top_10"): - if k in ref and ref[k] is not None: - ref_rows.append((k.replace("_", " "), fmt_pct(ref[k], 2))) - for k, v in ref.items(): - if not k.startswith("reference_apex_in_top_"): - ref_rows.append((esc(k), fmt_num(v) if isinstance(v, float) else fmt_int(v))) - parts.append("

Reference-apex peak oracle

") - parts.append(html_table(["Metric", "Value"], ref_rows)) - else: - parts.append('

Reference (DIA-NN) apex block not present; ' - "self analysis only.

") - - return "".join(parts) - - -# --------------------------------------------------------------------------- # -# Section 5: feature-family ablation -def _resolve_ablation_csvs(path): - if not path: - return [] - if os.path.isfile(path) and path.lower().endswith(".csv"): - return [path] - if os.path.isdir(path): - preferred = os.path.join(path, "feature_ablation.csv") - found = [] - if os.path.isfile(preferred): - found.append(preferred) - for name in sorted(os.listdir(path)): - full = os.path.join(path, name) - if full != preferred and name.lower().endswith(".csv") and os.path.isfile(full): - found.append(full) - return found - return [] - - -def _to_int(x): - try: - return int(float(x)) - except (TypeError, ValueError): - return None - - -def section_ablation(path): - csvs = _resolve_ablation_csvs(path) - if not csvs: - if path: - return not_provided(f"no ablation CSV found at {esc(path)}") - return not_provided() - - rows = [] - for cpath in csvs: - with open(cpath, "r", encoding="utf-8", newline="") as fh: - for r in csv.DictReader(fh): - rows.append(r) - if not rows: - return not_provided("ablation CSV(s) contained no rows") - - parts = [f'

Source: {esc(", ".join(os.path.basename(c) for c in csvs))}

'] - - models = [] - for r in rows: - m = r.get("model") or "model" - if m not in models: - models.append(m) - - for model in models: - mrows = [r for r in rows if (r.get("model") or "model") == model] - # most useful first: most negative delta_vs_full (removal hurts most) - mrows.sort(key=lambda r: (_to_int(r.get("delta_vs_full")) is None, - _to_int(r.get("delta_vs_full")) if - _to_int(r.get("delta_vs_full")) is not None else 0)) - headers = ["Feature family", "Baseline IDs", "New IDs", - "delta vs full", "Recommendation"] - trows = [] - for r in mrows: - delta = _to_int(r.get("delta_vs_full")) - delta_s = f"{delta:+,}" if delta is not None else esc(r.get("delta_vs_full")) - rec = r.get("recommendation") or "" - rec_cls = { - "KEEP": "rec-keep", "HARMFUL": "rec-harm", - "REDUNDANT_BUT_INFORMATIVE": "rec-info", "REDUNDANT": "rec-red", - }.get(rec, "") - rec_html = f'{esc(rec)}' if rec else "" - trows.append([ - esc(r.get("feature_family")), - fmt_int(r.get("baseline_identifications")), - fmt_int(r.get("new_identifications")), - delta_s, - rec_html, - ]) - parts.append(f"

Model: {esc(model)}

") - parts.append(html_table(headers, trows)) - return "".join(parts) - - -# --------------------------------------------------------------------------- # -# Section 6: empirical entrapment FDP -def section_entrapment(path): - if not path or not os.path.isfile(path): - return not_provided() - with open(path, "r", encoding="utf-8", errors="replace") as fh: - text = fh.read() - - rows = [] - m = re.search(r"E\.?coli targets\s*=\s*(\d+)", text) - if m: - rows.append(("E. coli targets", fmt_int(m.group(1)))) - m = re.search(r"human train-neg\s*=\s*(\d+)", text) - if m: - rows.append(("Human train-negatives", fmt_int(m.group(1)))) - m = re.search(r"test-null\s*=\s*(\d+)", text) - if m: - rows.append(("Held-out test null", fmt_int(m.group(1)))) - m = re.search(r"ratio\s*=\s*([\d.]+)", text) - if m: - rows.append(("Library-size ratio", fmt_num(m.group(1)))) - for m in re.finditer( - r"held-out E\.?coli stripped seqs @\s*(\d+)%[^:]*:\s*(\d+)", text - ): - rows.append((f"Held-out E. coli stripped seqs @ {m.group(1)}%", - fmt_int(m.group(2)))) - m = re.search( - r"at shipped q<=\s*(\d+)%:\s*E\.?coli\s*=\s*(\d+),\s*true FDR" - r"[^=]*=\s*([\d.]+)%", - text, - ) - if m: - rows.append((f"E. coli at shipped q <= {m.group(1)}%", fmt_int(m.group(2)))) - rows.append((f"True FDR on held-out null (q <= {m.group(1)}%)", - f"{m.group(3)}%")) - - parts = [] - if rows: - parts.append(html_table(["Metric", "Value"], rows)) - else: - parts.append(not_provided("no recognizable entrapment metrics in the file")) - parts.append("
Raw entrapment output" - f"
{esc(text.strip())}
") - return "".join(parts) - - -# --------------------------------------------------------------------------- # -# Section 7: limitations footer -def section_limitations(rendered): - absent = [name for name, ok in rendered.items() if not ok] - items = [ - "Diagnostics are computed on a single dataset (the E. coli / HYE example); " - "no family, threshold, or component decision should be made from one dataset " - "or one favourable subset (spec 03 Section 9, spec 05 Section 7).", - "The identification-loss waterfall collapses all extraction losses to " - "NO_PEAK_GROUP at artifact resolution; the in-extract audit sidecar is needed " - "to separate NO_FRAGMENT_TRACES / NO_VALID_FRAGMENTS / PEAK_NOT_SELECTED / " - "RT_PRUNED.", - "Top-K self recall measures peak-selection opportunity only; the reference " - "(DIA-NN) peak-oracle metric requires a reference report and is shown only " - "when a reference block is present.", - "Feature-family ablation is cross-validated on one dataset; a gain that flips " - "sign across models or datasets is not a keep. Rerun with both models and a " - "second dataset before acting.", - "The entrapment held-out FDP is the accept / reject gate; a sensitivity gain " - "that inflates empirical FDP must not be retained.", - ] - parts = ["
    "] - for it in items: - parts.append(f"
  • {esc(it)}
    • ") - parts.append("
") - if absent: - pretty = ", ".join(esc(a) for a in absent) - parts.append(f'

Sections not rendered (input absent): ' - f"{pretty}.

") - return "".join(parts) - - -# --------------------------------------------------------------------------- # -_CSS = """ -:root { color-scheme: light dark; } -* { box-sizing: border-box; } -body { - margin: 0; padding: 0 0 4rem 0; - font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, - Arial, sans-serif; - line-height: 1.5; color: #1c2530; background: #f4f6f9; -} -.container { max-width: 1040px; margin: 0 auto; padding: 1.5rem; } -header.report { - background: #26456f; color: #fff; padding: 1.6rem 1.5rem; border-radius: 0 0 10px 10px; -} -header.report h1 { margin: 0 0 .3rem 0; font-size: 1.55rem; } -header.report .stamp { opacity: .85; font-size: .9rem; } -section { - background: #fff; border: 1px solid #e2e7ee; border-radius: 10px; - padding: 1.2rem 1.3rem; margin: 1.2rem 0; box-shadow: 0 1px 2px rgba(0,0,0,.04); -} -section > h2 { - margin: 0 0 .4rem 0; font-size: 1.2rem; color: #26456f; - border-bottom: 2px solid #eef1f5; padding-bottom: .4rem; -} -section > h2 .sec-no { - display: inline-block; min-width: 1.6rem; color: #7a8aa0; font-weight: 600; -} -h3 { font-size: 1rem; margin: 1rem 0 .4rem 0; color: #33445c; } -p.desc { margin: .2rem 0 .8rem 0; color: #55627a; font-size: .92rem; } -p.np { - color: #8a94a6; font-style: italic; background: #f7f8fb; border: 1px dashed #d7dde7; - padding: .5rem .7rem; border-radius: 6px; display: inline-block; -} -p.muted { color: #6b7688; font-size: .88rem; } -.table-wrap { overflow-x: auto; -webkit-overflow-scrolling: touch; margin: .3rem 0 .6rem; } -table { border-collapse: collapse; width: 100%; font-size: .9rem; } -th, td { - text-align: left; padding: .4rem .6rem; border-bottom: 1px solid #eef1f5; - white-space: nowrap; -} -th { background: #f0f3f8; color: #33445c; font-weight: 600; position: sticky; top: 0; } -td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; } -tbody tr:hover { background: #f7f9fc; } -figure.chart { margin: .8rem 0; text-align: center; } -figure.chart img { - max-width: 100%; height: auto; border: 1px solid #e2e7ee; border-radius: 8px; - background: #fff; -} -figcaption { color: #6b7688; font-size: .82rem; margin-top: .35rem; } -.toc { font-size: .92rem; } -.toc a { color: #26456f; text-decoration: none; } -.toc a:hover { text-decoration: underline; } -.rec { font-size: .78rem; padding: .1rem .45rem; border-radius: 10px; font-weight: 600; } -.rec-keep { background: #dff3e4; color: #1f6b39; } -.rec-harm { background: #fde3e0; color: #9c2b20; } -.rec-info { background: #e5eefb; color: #26456f; } -.rec-red { background: #eef0f4; color: #66707f; } -details { margin-top: .6rem; } -summary { cursor: pointer; color: #26456f; font-weight: 600; } -pre { - overflow-x: auto; background: #0f1826; color: #d7e0ee; padding: .8rem; - border-radius: 8px; font-size: .82rem; line-height: 1.45; -} -footer.report { color: #6b7688; font-size: .8rem; text-align: center; padding: 1rem; } -@media (prefers-color-scheme: dark) { - body { background: #10151d; color: #d6dde8; } - section { background: #1a212c; border-color: #2a3341; box-shadow: none; } - section > h2 { color: #9db9e6; border-bottom-color: #2a3341; } - h3 { color: #c3cede; } - th { background: #232c39; color: #c3cede; } - th, td { border-bottom-color: #2a3341; } - tbody tr:hover { background: #212a37; } - p.np { background: #202836; border-color: #2f3a49; color: #8a94a6; } - p.desc, p.muted, figcaption { color: #97a3b6; } - figure.chart img { border-color: #2a3341; } -} -""" - - -def build_html(title, stamp, sections): - """sections: list of (num, anchor, name, description, body_html).""" - toc = " · ".join( - f'{num}. {esc(name)}' - for num, anchor, name, _desc, _body in sections - ) - blocks = [] - for num, anchor, name, desc, body in sections: - desc_html = f'

{esc(desc)}

' if desc else "" - blocks.append( - f'
' - f'

{num}.{esc(name)}

' - f"{desc_html}{body}
" - ) - return ( - "\n" - '' - '' - f"{esc(title)}" - f"" - f'

{esc(title)}

' - f'
Generated {esc(stamp)}
' - f'
' - f'{"".join(blocks)}' - '
MuMDIA sensitivity benchmark report. ' - "All diagnostics are read-only over existing artifacts.
" - "
" - ) - - -# --------------------------------------------------------------------------- # -def main(): - ap = argparse.ArgumentParser( - description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter - ) - ap.add_argument("--out", required=True, help="output HTML file") - ap.add_argument("--audit-metrics", default=None, - help="candidate_audit.parquet.metrics.json (waterfall)") - ap.add_argument("--audit", default=None, - help="candidate_audit.parquet (stratification)") - ap.add_argument("--topk", default=None, help="topk.json (peak recall)") - ap.add_argument("--ablation", default=None, - help="feature-family ablation CSV or directory") - ap.add_argument("--entrapment", default=None, - help="entrapment holdout stdout text file") - ap.add_argument("--title", default="MuMDIA Sensitivity Benchmark Report") - ap.add_argument("--stamp", default=None, - help="fixed header timestamp (default: current local time)") - args = ap.parse_args() - - stamp = args.stamp or _dt.datetime.now().strftime("%Y-%m-%d %H:%M:%S") - - metrics = load_json(args.audit_metrics) - topk = load_json(args.topk) - - # Section 1 body: run metadata + input inventory - inputs = [ - ("Audit metrics (JSON)", args.audit_metrics), - ("Candidate audit (Parquet)", args.audit), - ("Top-K peak recall (JSON)", args.topk), - ("Feature ablation (CSV/dir)", args.ablation), - ("Entrapment stdout (text)", args.entrapment), - ] - meta_rows = [] - if metrics and metrics.get("run_id"): - meta_rows.append(("Run id", esc(metrics["run_id"]))) - meta_rows.append(("Report generated", esc(stamp))) - for label, path in inputs: - present = bool(path) and os.path.exists(path) - status = "present" if present else ("missing" if path else "not provided") - shown = esc(path) if path else "—" - meta_rows.append((label, f"{shown} ({status})")) - header_body = html_table(["Item", "Value"], meta_rows, align_right_from=99) - - # build sections - wf_body = section_waterfall(metrics) - strat_body = section_stratification(args.audit) - topk_body = section_topk(topk) - abl_body = section_ablation(args.ablation) - ent_body = section_entrapment(args.entrapment) - - rendered = { - "Identification-loss waterfall": bool(metrics), - "Candidate-audit stratification": bool(args.audit and os.path.isfile(args.audit)), - "Top-K peak recall": bool(topk), - "Feature-family ablation": bool(_resolve_ablation_csvs(args.ablation)), - "Empirical entrapment FDP": bool(args.entrapment and os.path.isfile(args.entrapment)), - } - lim_body = section_limitations(rendered) - - sections = [ - (1, "run", "Run metadata", - "Report inputs and provenance. Each downstream section renders only when " - "its input is provided.", header_body), - (2, "waterfall", "Identification-loss waterfall", - "Where DIA-NN-only precursors are lost, from candidate audit metrics " - "(spec 02 Section 8).", wf_body), - (3, "stratification", "Candidate-audit stratification", - "Earliest rejection reason grouped by charge and entrapment label.", - strat_body), - (4, "topk", "Top-K peak recall", - "How often the selected apex is the strongest peak, and (with a reference) " - "whether the reference apex is within the top-K peaks.", topk_body), - (5, "ablation", "Feature-family ablation", - "Cross-validated contribution of each feature family, most useful first " - "(largest identification drop when removed).", abl_body), - (6, "entrapment", "Empirical entrapment FDP", - "Held-out entrapment identification count at a genuinely controlled FDP " - "(the accept / reject gate).", ent_body), - (7, "limitations", "Limitations", - "What is single-dataset or not yet measured.", lim_body), - ] - - doc = build_html(args.title, stamp, sections) - out_dir = os.path.dirname(os.path.abspath(args.out)) - if out_dir: - os.makedirs(out_dir, exist_ok=True) - with open(args.out, "w", encoding="utf-8") as fh: - fh.write(doc) - - n_rendered = sum(1 for v in rendered.values() if v) - print(f"[written] {args.out} ({os.path.getsize(args.out):,} bytes)") - print(f"[sections] {n_rendered + 2}/7 rendered " - f"(run metadata + limitations always render)") - for name, ok in rendered.items(): - print(f" {'rendered ' if ok else 'not-prov '} {name}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/build_funnel.py b/scripts/build_funnel.py deleted file mode 100644 index 2da7618..0000000 --- a/scripts/build_funnel.py +++ /dev/null @@ -1,201 +0,0 @@ -"""Build a per-precursor / per-peptide LOSS FUNNEL attributing every DIA-NN -identification to the earliest MuMDIA stage where we lose it, on the current-best -E. coli pipeline (DIA-NN predicted library + shift decoys + GPU-fine-tuned -multitask DeepLC iRT + rich features). - -Because the MuMDIA library IS DIA-NN's predicted library, library/fragment -prediction is (near) 100% coverage; the funnel isolates the DOWNSTREAM losses: - NOT_IN_LIB -> NO_EVIDENCE (not extracted even gate-off) - -> GATE_LOSS (extracted gate-off, rejected by the extract gate) - -> FDR_LOSS (extracted with gate, ranked below 1% FDR) - -> FOUND (target q <= 0.01, gate-on) - -Two chains feed it: gate-ON (reported IDs) and gate-OFF (maximal extraction, to -separate "no evidence at all" from "gate rejected it"). Diagnostics from DIA-NN -(RT, quantity, CScore, Ms1.Profile.Corr, Spectrum.Similarity) and from our own -features (frag_corr, coelution, ref_corr, rt_error, in-window) are attached per -row for the downstream deep-dive. - -Usage: python build_funnel.py - e.g. python build_funnel.py out_dl ftgate ftnogate out_dl/funnel -""" -import re -import sys -import numpy as np -import pandas as pd - -Q = 0.01 - - -def dn_to_pf(modseq): - s = str(modseq).replace("(UniMod:4)", "[Carbamidomethyl]").replace("(UniMod:35)", "[Oxidation]") - return s - - -def unmapped(modseq): - return "(UniMod:" in dn_to_pf(modseq) - - -def strip_pf(s): - return re.sub(r"\[[^\]]*\]", "", str(s).replace("DECOY_", "")) - - -def main(): - D, gate, nogate, outpref = sys.argv[1:5] - - # ---- DIA-NN reference (precursor level); run from project root ---- - dn = pd.read_csv("out_diann/report.tsv", sep="\t") - dn = dn[dn["Q.Value"] <= Q].copy() - dn = dn.rename(columns={"Precursor.Quantity": "dn_quantity", "CScore": "dn_cscore", - "Ms1.Profile.Corr": "dn_ms1corr", "Spectrum.Similarity": "dn_specsim", - "Mass.Evidence": "dn_massev"}) - dn["pf"] = dn["Modified.Sequence"].map(dn_to_pf) - dn["mod_unmapped"] = dn["Modified.Sequence"].map(unmapped) - dn["charge"] = dn["Precursor.Charge"].astype(int) - dn["rt_sec"] = dn["RT"] * 60.0 - dn["rt_start_sec"] = dn["RT.Start"] * 60.0 - dn["rt_stop_sec"] = dn["RT.Stop"] * 60.0 - dn["key"] = list(zip(dn["pf"], dn["charge"])) - dn["stripped"] = dn["Stripped.Sequence"] - - # ---- our combined ft library (targets): key -> candidate_id ---- - lib = pd.read_parquet(f"{D}/lib_precursors_ft.parquet", - columns=["candidate_id", "peptidoform", "charge", "label", "predicted_irt", "protein"]) - libt = lib[lib.label == "target"].copy() - libt["key"] = list(zip(libt["peptidoform"], libt["charge"].astype(int))) - key2cand = dict(zip(libt["key"], libt["candidate_id"])) - - # ---- seed (confident) ---- - seed = pd.read_parquet(f"{D}/seed_{gate}.parquet", columns=["candidate_id", "label", "spectrum_q"]) - seed_conf = set(seed[(seed.label == "target") & (seed.spectrum_q <= Q)].candidate_id) - - # ---- run windows (RT) ---- - rw = pd.read_parquet(f"{D}/rw_{gate}.parquet", columns=["candidate_id", "rt_lo", "rt_hi"]) - cand2lo = dict(zip(rw.candidate_id, rw.rt_lo)) - cand2hi = dict(zip(rw.candidate_id, rw.rt_hi)) - - # ---- extraction: gate-off (any evidence) and gate-on ---- - px_ng = pd.read_parquet(f"{D}/psms_{nogate}.parquet", - columns=["candidate_id", "apex_rt", "n_matched_fragments", "coelution_run"]) - extracted_ng = set(px_ng.candidate_id) - cand2apex = dict(zip(px_ng.candidate_id, px_ng.apex_rt)) - px_g = pd.read_parquet(f"{D}/psms_{gate}.parquet", columns=["candidate_id"]) - extracted_g = set(px_g.candidate_id) - - # ---- features gate-off (gate quantities for the GATE_LOSS deep-dive) ---- - fcols = ["candidate_id", "frag_corr", "frag_cosine", "n_matched_fragments", "coelution_run", - "coelution_mean", "matched_fraction", "ref_corr", "profile_cos", "rt_error_abs", - "log_apex_intensity", "isotope_corr", "log_sn", "seed_score"] - feat_ng = pd.read_parquet(f"{D}/feat_{nogate}.parquet", columns=fcols).set_index("candidate_id") - - # ---- scored q-values: gate-on (reported) and gate-off ---- - sc_g = pd.read_parquet(f"{D}/scored_{gate}.parquet", columns=["candidate_id", "label", "q_value", "peptide_q_value"]) - cand2q_g = dict(zip(sc_g[sc_g.label == "target"].candidate_id, sc_g[sc_g.label == "target"].q_value)) - sc_ng = pd.read_parquet(f"{D}/scored_{nogate}.parquet", columns=["candidate_id", "label", "q_value"]) - cand2q_ng = dict(zip(sc_ng[sc_ng.label == "target"].candidate_id, sc_ng[sc_ng.label == "target"].q_value)) - - # ---- assemble per-precursor rows ---- - rows = [] - for r in dn.itertuples(index=False): - cand = key2cand.get(r.key) - in_lib = cand is not None - ext_ng = in_lib and (cand in extracted_ng) - ext_g = in_lib and (cand in extracted_g) - qg = cand2q_g.get(cand, np.nan) if in_lib else np.nan - qng = cand2q_ng.get(cand, np.nan) if in_lib else np.nan - found = in_lib and (qg <= Q) - lo = cand2lo.get(cand, np.nan) if in_lib else np.nan - hi = cand2hi.get(cand, np.nan) if in_lib else np.nan - in_window = in_lib and not np.isnan(lo) and (lo <= r.rt_sec <= hi) - f = feat_ng.loc[cand] if (in_lib and cand in feat_ng.index) else None - - if not in_lib: - bucket = "1_NOT_IN_LIB" - elif not ext_ng: - bucket = "2_NO_EVIDENCE" - elif not ext_g: - bucket = "3_GATE_LOSS" - elif not found: - bucket = "4_FDR_LOSS" - else: - bucket = "5_FOUND" - - rows.append(dict( - stripped=r.stripped, pf=r.pf, charge=r.charge, mod_unmapped=r.mod_unmapped, - bucket=bucket, candidate_id=(int(cand) if in_lib else -1), - in_lib=in_lib, seeded=(in_lib and cand in seed_conf), - extracted_nogate=ext_ng, extracted_gate=ext_g, - q_gate=qg, q_nogate=qng, found=found, - rt_in_window=in_window, our_apex_rt=(cand2apex.get(cand, np.nan) if ext_ng else np.nan), - rt_lo=lo, rt_hi=hi, - dn_rt_sec=r.rt_sec, dn_rt_start_sec=r.rt_start_sec, dn_rt_stop_sec=r.rt_stop_sec, - dn_quantity=r.dn_quantity, dn_cscore=r.dn_cscore, dn_ms1corr=r.dn_ms1corr, - dn_specsim=r.dn_specsim, dn_massev=r.dn_massev, - our_frag_corr=(float(f.frag_corr) if f is not None else np.nan), - our_frag_cosine=(float(f.frag_cosine) if f is not None else np.nan), - our_n_matched=(float(f.n_matched_fragments) if f is not None else np.nan), - our_coelution_run=(float(f.coelution_run) if f is not None else np.nan), - our_coelution_mean=(float(f.coelution_mean) if f is not None else np.nan), - our_matched_fraction=(float(f.matched_fraction) if f is not None else np.nan), - our_ref_corr=(float(f.ref_corr) if f is not None else np.nan), - our_profile_cos=(float(f.profile_cos) if f is not None else np.nan), - our_rt_error_abs=(float(f.rt_error_abs) if f is not None else np.nan), - our_log_apex=(float(f.log_apex_intensity) if f is not None else np.nan), - our_log_sn=(float(f.log_sn) if f is not None else np.nan), - )) - pp = pd.DataFrame(rows) - pp.to_csv(f"{outpref}_per_precursor.csv", index=False) - - # ---- collapse to peptide level: best (max) bucket reached across precursors ---- - order = {"1_NOT_IN_LIB": 1, "2_NO_EVIDENCE": 2, "3_GATE_LOSS": 3, "4_FDR_LOSS": 4, "5_FOUND": 5} - pp["brank"] = pp.bucket.map(order) - pep = pp.loc[pp.groupby("stripped")["brank"].idxmax()].copy() # best stage reached per peptide - pep.to_csv(f"{outpref}_per_peptide.csv", index=False) - - # ---- summary ---- - with open(f"{outpref}_summary.txt", "w") as fh: - def out(*a): - print(*a); print(*a, file=fh) - out("=== FUNNEL (precursor level, DIA-NN @1% =", len(pp), "precursors) ===") - vc = pp.bucket.value_counts().sort_index() - cum = 0 - for b in sorted(order, key=order.get): - n = int(vc.get(b, 0)); cum += n - out(f" {b:16s} {n:6d} ({100*n/len(pp):5.1f}%) cum {cum}") - out("") - out("=== FUNNEL (peptide level, DIA-NN @1% =", len(pep), "peptides) ===") - vcp = pep.bucket.value_counts().sort_index() - for b in sorted(order, key=order.get): - n = int(vcp.get(b, 0)) - out(f" {b:16s} {n:6d} ({100*n/len(pep):5.1f}%)") - out("") - out("=== per-bucket DIA-NN abundance / quality (precursor medians) ===") - for b in sorted(order, key=order.get): - g = pp[pp.bucket == b] - if len(g) == 0: - continue - out(f" {b:16s} n={len(g):5d} dn_quant_med={g.dn_quantity.median():.3e} " - f"cscore_med={g.dn_cscore.median():.3f} ms1corr_med={g.dn_ms1corr.median():.3f} " - f"specsim_med={g.dn_specsim.median():.3f} rt_in_window={100*g.rt_in_window.mean():.0f}% " - f"seeded={100*g.seeded.mean():.0f}%") - out("") - out("=== GATE_LOSS sub-diagnosis (why the gate rejected; gate needs frag_corr>=0.5, presence>=3, coel>=2) ===") - gl = pp[pp.bucket == "3_GATE_LOSS"] - if len(gl): - out(f" n={len(gl)} frag_corr<0.5: {100*(gl.our_frag_corr<0.5).mean():.0f}% " - f"n_matched<3: {100*(gl.our_n_matched<3).mean():.0f}% " - f"coelution_run<2: {100*(gl.our_coelution_run<2).mean():.0f}%") - out(f" frag_corr med={gl.our_frag_corr.median():.3f} BUT DIA-NN specsim med={gl.dn_specsim.median():.3f} " - f"(high specsim + low our frag_corr => our frag_corr computation, not missing signal)") - out("") - out("=== FDR_LOSS: would gate-off scoring have found them? ===") - fl = pp[pp.bucket == "4_FDR_LOSS"] - if len(fl): - out(f" n={len(fl)} q_gate med={fl.q_gate.median():.3f} " - f"would be <=1% at gate-off q: {100*(fl.q_nogate<=Q).mean():.0f}% " - f"rt_in_window={100*fl.rt_in_window.mean():.0f}%") - print("\nwrote", f"{outpref}_per_precursor.csv", f"{outpref}_per_peptide.csv", f"{outpref}_summary.txt") - - -if __name__ == "__main__": - main() diff --git a/scripts/candidate_diagnostics.py b/scripts/candidate_diagnostics.py deleted file mode 100644 index 9aeee5a..0000000 --- a/scripts/candidate_diagnostics.py +++ /dev/null @@ -1,537 +0,0 @@ -#!/usr/bin/env python -"""Per-candidate diagnostic bundle export (sensitivity_plan backlog P7.2). - -Implements the individual-candidate diagnostic packet described in -sensitivity_plan/02_sensitivity_diagnostic_plan.md section 9. For a selected -list of candidate_ids the script reads the run artifacts (non-invasively, never -writing back to them) and produces, per candidate: - - fragments.png overlaid fragment chromatograms (MS2 left axis, MS1 - right axis via twinx, apex and reference RT markers) - predicted_vs_observed.png predicted vs observed-at-apex intensity per fragment - candidate.json metadata, per-fragment summary, and all comp features - -A top-level index.txt lists the exported candidates and their key fields. - -Design notes: - - Deterministic: fixed Agg backend, sorted candidate order, sorted fragment - order, no random state. - - Bounded memory: chrom / comp / psms / scored are read with a pyarrow - dataset filter (predicate pushdown) restricted to the requested - candidate_ids, so the full tables are never materialized. - - Robust to missing MS1 traces, missing comp, and missing scored inputs. - -Interpreter: C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe -""" - -from __future__ import annotations - -import argparse -import json -import os -import re -import sys - -import matplotlib - -matplotlib.use("Agg") # headless, deterministic rendering - -import matplotlib.pyplot as plt -import numpy as np -import pyarrow.dataset as ds - - -# --------------------------------------------------------------------------- # -# Fragment-name parsing and ordering -# --------------------------------------------------------------------------- # - -_FRAG_RE = re.compile(r"^(?P[a-zA-Z]+)(?P\d+)(?:\^(?P\d+))?$") - - -def is_ms1_name(name: str) -> bool: - """Return True for MS1 precursor-isotope trace rows (ms1_mono, ms1_iso1, ...).""" - return str(name).lower().startswith("ms1") - - -def frag_sort_key(name: str): - """Deterministic, human-readable ordering key for fragment names. - - MS2 ions sort by series letter, ordinal, then fragment charge. Names that do - not parse fall back to a lexical key placed after parsed ions. - """ - m = _FRAG_RE.match(str(name)) - if not m: - return (2, str(name), 0, 0) - series = m.group("series").lower() - ordinal = int(m.group("ordinal")) - charge = int(m.group("charge")) if m.group("charge") else 1 - return (0, series, ordinal, charge) - - -# --------------------------------------------------------------------------- # -# IO helpers -# --------------------------------------------------------------------------- # - -def load_filtered(path: str, cand_ids, columns=None): - """Load only the rows for cand_ids from a parquet file (predicate pushdown). - - Returns a pandas DataFrame, or None if the path is falsy. Missing columns - are tolerated: only columns present in the file schema are requested. - """ - if not path: - return None - dataset = ds.dataset(path, format="parquet") - if columns is not None: - available = set(dataset.schema.names) - columns = [c for c in columns if c in available] - if "candidate_id" not in columns: - columns = ["candidate_id"] + columns - table = dataset.to_table( - filter=ds.field("candidate_id").isin(list(cand_ids)), - columns=columns, - ) - return table.to_pandas() - - -def index_by_candidate(df): - """Return {candidate_id: first-row dict} for quick per-candidate lookup.""" - out = {} - if df is None: - return out - for row in df.to_dict("records"): - cid = int(row["candidate_id"]) - if cid not in out: # keep first occurrence for determinism - out[cid] = row - return out - - -def as_float_array(seq): - """Coerce a nullable list column value to a 1-D float ndarray (empty if None).""" - if seq is None: - return np.empty(0, dtype=float) - arr = np.asarray(list(seq), dtype=float) - return arr - - -def value_at_rt(rt, intensity, target_rt): - """Observed intensity at the scan nearest target_rt, or None if unavailable.""" - rt = as_float_array(rt) - intensity = as_float_array(intensity) - if rt.size == 0 or intensity.size == 0 or target_rt is None: - return None - n = min(rt.size, intensity.size) - idx = int(np.argmin(np.abs(rt[:n] - float(target_rt)))) - return float(intensity[idx]) - - -def count_nonzero(intensity): - intensity = as_float_array(intensity) - if intensity.size == 0: - return 0 - return int(np.count_nonzero(intensity > 0.0)) - - -def json_default(obj): - """Make numpy scalars / arrays JSON-serializable.""" - if isinstance(obj, (np.integer,)): - return int(obj) - if isinstance(obj, (np.floating,)): - v = float(obj) - return v if np.isfinite(v) else None - if isinstance(obj, (np.bool_,)): - return bool(obj) - if isinstance(obj, np.ndarray): - return obj.tolist() - return str(obj) - - -def clean_scalar(v): - """Normalize a scalar for JSON: NaN/inf -> None, numpy -> python.""" - if v is None: - return None - if isinstance(v, (np.integer,)): - return int(v) - if isinstance(v, (np.floating, float)): - v = float(v) - return v if np.isfinite(v) else None - if isinstance(v, (np.bool_,)): - return bool(v) - return v - - -# --------------------------------------------------------------------------- # -# Reference RT table -# --------------------------------------------------------------------------- # - -def parse_ref_rt_file(path): - """Parse a candidate_id,rt reference table. Returns {candidate_id: rt}. - - Accepts an optional header line and comma or whitespace separation. Lines - that do not parse as (int, float) are skipped. - """ - ref = {} - if not path: - return ref - with open(path, "r", encoding="utf-8") as fh: - for line in fh: - line = line.strip() - if not line or line.startswith("#"): - continue - parts = re.split(r"[,\t ]+", line) - if len(parts) < 2: - continue - try: - cid = int(float(parts[0])) - rt = float(parts[1]) - except ValueError: - continue # header or malformed line - ref[cid] = rt - return ref - - -# --------------------------------------------------------------------------- # -# Candidate id selection -# --------------------------------------------------------------------------- # - -def parse_candidate_list(arg_value, arg_file): - ids = [] - if arg_value: - for tok in re.split(r"[,\s]+", arg_value.strip()): - if tok: - ids.append(int(tok)) - if arg_file: - with open(arg_file, "r", encoding="utf-8") as fh: - for line in fh: - for tok in re.split(r"[,\s]+", line.strip()): - if tok and not tok.startswith("#"): - ids.append(int(tok)) - # de-duplicate, preserve first-seen order, then sort for determinism - seen = set() - uniq = [] - for i in ids: - if i not in seen: - seen.add(i) - uniq.append(i) - return sorted(uniq) - - -# --------------------------------------------------------------------------- # -# Plotting -# --------------------------------------------------------------------------- # - -def plot_fragments(cand_id, frag_rows, ms1_rows, apex_rt, ref_rt, - ms1_scalars, title, out_path): - """Overlaid fragment chromatograms with MS1 on a separate twin axis.""" - fig, ax = plt.subplots(figsize=(11, 6)) - - cmap = plt.get_cmap("tab20") - handles, labels = [], [] - - # MS2 fragment traces on the left axis. - for i, row in enumerate(frag_rows): - rt = as_float_array(row["rt"]) - inten = as_float_array(row["intensity"]) - if rt.size == 0 or inten.size == 0: - continue - n = min(rt.size, inten.size) - color = cmap(i % 20) - (line,) = ax.plot(rt[:n], inten[:n], color=color, linewidth=1.2, - label=row["frag_name"]) - handles.append(line) - labels.append(row["frag_name"]) - - ax.set_xlabel("Retention time (s)") - ax.set_ylabel("MS2 fragment intensity") - ax.set_title(title) - - # MS1 traces on a separate twin axis (never normalized into MS2 scaling). - ax2 = None - if ms1_rows: - ax2 = ax.twinx() - ms1_cmap = plt.get_cmap("Dark2") - for j, row in enumerate(ms1_rows): - rt = as_float_array(row["rt"]) - inten = as_float_array(row["intensity"]) - if rt.size == 0 or inten.size == 0: - continue - n = min(rt.size, inten.size) - color = ms1_cmap(j % 8) - (line,) = ax2.plot(rt[:n], inten[:n], color=color, linewidth=1.4, - linestyle="--", label=row["frag_name"]) - handles.append(line) - labels.append(row["frag_name"]) - ax2.set_ylabel("MS1 precursor-isotope intensity") - elif ms1_scalars: - # No MS1 traces; show available MS1 XIC scalars as points at the apex. - ax2 = ax.twinx() - ms1_cmap = plt.get_cmap("Dark2") - for j, (name, val) in enumerate(sorted(ms1_scalars.items())): - if val is None or apex_rt is None: - continue - color = ms1_cmap(j % 8) - pt = ax2.scatter([apex_rt], [val], color=color, marker="D", s=40, - zorder=5, label=name + " (scalar)") - handles.append(pt) - labels.append(name + " (scalar)") - ax2.set_ylabel("MS1 precursor-isotope intensity (scalar)") - - # Apex and reference RT markers. - if apex_rt is not None: - vl = ax.axvline(apex_rt, color="black", linewidth=1.4, linestyle="-", - label="apex_rt") - handles.append(vl) - labels.append("apex_rt") - if ref_rt is not None: - vr = ax.axvline(ref_rt, color="red", linewidth=1.4, linestyle=":", - label="reference_rt") - handles.append(vr) - labels.append("reference_rt") - - if handles: - ax.legend(handles, labels, fontsize=7, ncol=2, loc="upper right", - framealpha=0.9) - - fig.tight_layout() - fig.savefig(out_path, dpi=120) - plt.close(fig) - - -def plot_predicted_vs_observed(frag_summ, title, out_path): - """Grouped bar chart: predicted intensity vs observed-at-apex per fragment.""" - names = [f["name"] for f in frag_summ] - pred = np.array([f["predicted_intensity"] or 0.0 for f in frag_summ], float) - obs = np.array( - [(f["observed_apex_intensity"] or 0.0) for f in frag_summ], float - ) - - fig, ax = plt.subplots(figsize=(max(8, 0.5 * len(names) + 2), 6)) - if names: - x = np.arange(len(names)) - width = 0.4 - # Normalize each series to its own max so predicted (relative) and - # observed (raw counts) shapes are comparable on one axis. - pred_n = pred / pred.max() if pred.max() > 0 else pred - obs_n = obs / obs.max() if obs.max() > 0 else obs - ax.bar(x - width / 2, pred_n, width, label="predicted (norm)", - color="#4C72B0") - ax.bar(x + width / 2, obs_n, width, label="observed at apex (norm)", - color="#DD8452") - ax.set_xticks(x) - ax.set_xticklabels(names, rotation=60, ha="right", fontsize=8) - ax.set_ylabel("Relative intensity (per-series max = 1)") - ax.legend(fontsize=8) - else: - ax.text(0.5, 0.5, "no MS2 fragments", ha="center", va="center") - ax.set_title(title) - fig.tight_layout() - fig.savefig(out_path, dpi=120) - plt.close(fig) - - -# --------------------------------------------------------------------------- # -# Per-candidate bundle -# --------------------------------------------------------------------------- # - -FEATURE_SKIP = { - "candidate_id", "label", "base_peptide_id", "peptidoform", "protein", - "apex_rt", "precursor_mz", -} - - -def build_bundle(cand_id, chrom_df, psms_row, comp_row, scored_row, ref_rt, - out_dir): - """Write fragments.png, predicted_vs_observed.png and candidate.json. - - Returns a summary dict used for index.txt. - """ - cdir = os.path.join(out_dir, str(cand_id)) - os.makedirs(cdir, exist_ok=True) - - # Split chrom rows into MS2 fragments and MS1 traces, sorted deterministically. - rows = chrom_df.to_dict("records") if chrom_df is not None else [] - frag_rows = [r for r in rows if not is_ms1_name(r["frag_name"])] - ms1_rows = [r for r in rows if is_ms1_name(r["frag_name"])] - frag_rows.sort(key=lambda r: frag_sort_key(r["frag_name"])) - ms1_rows.sort(key=lambda r: str(r["frag_name"])) - - # Metadata, tolerant of missing psms / scored rows. - def g(row, key, default=None): - if row is None or key not in row: - return default - return clean_scalar(row[key]) - - peptidoform = g(psms_row, "peptidoform") or g(scored_row, "peptidoform") \ - or g(comp_row, "peptidoform") - charge = g(psms_row, "charge") - if charge is None: - charge = g(scored_row, "charge") - if charge is None: - charge = g(comp_row, "charge") - label = g(psms_row, "label") or g(scored_row, "label") or g(comp_row, "label") - protein = g(psms_row, "protein") or g(scored_row, "protein") \ - or g(comp_row, "protein") - apex_rt = g(psms_row, "apex_rt") - if apex_rt is None: - apex_rt = g(comp_row, "apex_rt") - q_value = g(scored_row, "q_value") - n_matched = g(psms_row, "n_matched_fragments") - - # MS1 XIC scalars from psms (fallback plotting + JSON record). - ms1_scalars = {} - for key in ("ms1_isom1", "ms1_mono", "ms1_iso1", "ms1_iso2"): - val = g(psms_row, key) - if val is not None: - ms1_scalars[key] = val - - # Per-fragment summary (MS2 fragments only). - frag_summ = [] - for r in frag_rows: - frag_summ.append({ - "name": r["frag_name"], - "frag_mz": clean_scalar(r.get("frag_mz")), - "predicted_intensity": clean_scalar(r.get("predicted_intensity")), - "observed_apex_intensity": value_at_rt( - r.get("rt"), r.get("intensity"), apex_rt), - "n_nonzero_points": count_nonzero(r.get("intensity")), - }) - - charge_str = "" if charge is None else "+%d" % int(charge) - title = "%s%s %s (candidate %d)" % ( - peptidoform or "?", charge_str, label or "?", cand_id) - - plot_fragments( - cand_id, frag_rows, ms1_rows, apex_rt, ref_rt, ms1_scalars, title, - os.path.join(cdir, "fragments.png"), - ) - plot_predicted_vs_observed( - frag_summ, title, os.path.join(cdir, "predicted_vs_observed.png"), - ) - - # Feature values from comp (all columns except identity/context fields). - features = {} - if comp_row is not None: - for k, v in comp_row.items(): - if k in FEATURE_SKIP: - continue - features[k] = clean_scalar(v) - - bundle = { - "candidate_id": cand_id, - "peptidoform": peptidoform, - "charge": None if charge is None else int(charge), - "label": label, - "protein": protein, - "apex_rt": apex_rt, - "reference_rt": ref_rt, - "q_value": q_value, - "n_matched_fragments": None if n_matched is None else int(n_matched), - "n_fragments_chrom": len(frag_rows), - "n_ms1_traces": len(ms1_rows), - "ms1_scalars": ms1_scalars, - "fragments": frag_summ, - "features": features, - } - with open(os.path.join(cdir, "candidate.json"), "w", encoding="utf-8") as fh: - json.dump(bundle, fh, indent=2, default=json_default) - - return { - "candidate_id": cand_id, - "peptidoform": peptidoform or "?", - "charge": "" if charge is None else int(charge), - "label": label or "?", - "q_value": q_value, - "apex_rt": apex_rt, - "n_matched_fragments": n_matched, - "n_fragments": len(frag_rows), - "protein": protein or "?", - } - - -# --------------------------------------------------------------------------- # -# Main -# --------------------------------------------------------------------------- # - -def main(argv=None): - ap = argparse.ArgumentParser( - description="Export per-candidate diagnostic bundles (sensitivity_plan " - "P7.2, spec 02 section 9).", - ) - ap.add_argument("--chrom", required=True, help="chrom.parquet path") - ap.add_argument("--psms", required=True, help="psms.parquet path") - ap.add_argument("--comp", default=None, help="comp.parquet path (optional)") - ap.add_argument("--scored", default=None, - help="scored.parquet path (optional)") - ap.add_argument("--candidates", default=None, - help="comma-separated candidate_ids") - ap.add_argument("--candidates-file", default=None, - help="file with candidate_ids (comma/whitespace/newline)") - ap.add_argument("--out", default="candidate_diag", - help="output directory (default: candidate_diag)") - ap.add_argument("--ref-rt-file", default=None, - help="reference RT table: candidate_id,rt per line") - args = ap.parse_args(argv) - - cand_ids = parse_candidate_list(args.candidates, args.candidates_file) - if not cand_ids: - ap.error("no candidate_ids given (use --candidates or --candidates-file)") - - os.makedirs(args.out, exist_ok=True) - ref_rts = parse_ref_rt_file(args.ref_rt_file) - - # Bounded reads: only rows for the requested candidate_ids. - chrom_df = load_filtered(args.chrom, cand_ids) - psms_idx = index_by_candidate(load_filtered(args.psms, cand_ids)) - comp_idx = index_by_candidate(load_filtered(args.comp, cand_ids)) - scored_idx = index_by_candidate(load_filtered(args.scored, cand_ids)) - - # Group chrom rows per candidate once. - chrom_by_cand = {} - if chrom_df is not None and len(chrom_df): - for cid, grp in chrom_df.groupby("candidate_id"): - chrom_by_cand[int(cid)] = grp - - summaries = [] - for cid in cand_ids: - sub = chrom_by_cand.get(cid) - if sub is None: - print("warning: no chrom rows for candidate_id %d" % cid, - file=sys.stderr) - summ = build_bundle( - cid, - sub, - psms_idx.get(cid), - comp_idx.get(cid), - scored_idx.get(cid), - ref_rts.get(cid), - args.out, - ) - summaries.append(summ) - print("wrote %s" % os.path.join(args.out, str(cid))) - - # Top-level index. - index_path = os.path.join(args.out, "index.txt") - header = ["candidate_id", "peptidoform", "charge", "label", "q_value", - "apex_rt", "n_matched_fragments", "n_fragments", "protein"] - with open(index_path, "w", encoding="utf-8") as fh: - fh.write("\t".join(header) + "\n") - for s in summaries: - qv = "" if s["q_value"] is None else "%.6g" % s["q_value"] - ar = "" if s["apex_rt"] is None else "%.4f" % s["apex_rt"] - nm = "" if s["n_matched_fragments"] is None else str( - int(s["n_matched_fragments"])) - fh.write("\t".join([ - str(s["candidate_id"]), - str(s["peptidoform"]), - str(s["charge"]), - str(s["label"]), - qv, - ar, - nm, - str(s["n_fragments"]), - str(s["protein"]), - ]) + "\n") - print("wrote %s" % index_path) - - -if __name__ == "__main__": - main() diff --git a/scripts/combine_mutdec.py b/scripts/combine_mutdec.py deleted file mode 100644 index 38107fd..0000000 --- a/scripts/combine_mutdec.py +++ /dev/null @@ -1,78 +0,0 @@ -"""Combine a target library with a DIA-NN-predicted MUTATION-decoy library into a -single MuMDIA library (contiguous candidate_id sorted by precursor m/z). Decoys are -labeled 'decoy' and DECOY_-prefixed. Unlike shift decoys (which reuse target -intensities on shifted m/z), these decoys have DIA-NN-predicted fragments for a -mutated sequence, so they experience the same chimeric interference as targets. - -Usage: python combine_mutdec.py -""" -import sys -import numpy as np -import pandas as pd - - -def main(): - tp, tf, dp, df_, op, of = sys.argv[1:7] - tprec = pd.read_parquet(tp); tfrag = pd.read_parquet(tf) - dprec = pd.read_parquet(dp); dfrag = pd.read_parquet(df_) - # Match the decoy (length, charge) distribution to the targets so no feature - # (e.g. peptide_length) trivially separates target from decoy -> valid null. - import re - sstrip = lambda s: re.sub(r"\[[^\]]*\]", "", str(s).replace("DECOY_", "")) - plen = lambda s: len(re.sub(r"\[[^\]]*\]", "", str(s))) - tprec = tprec.copy(); dprec = dprec.copy().reset_index(drop=True) - # Drop decoys whose stripped sequence also occurs among targets (shared peptides - # would be present in the sample -> false decoys). - tset = set(tprec.peptidoform.map(sstrip)) - before = len(dprec) - dprec = dprec[~dprec.peptidoform.map(sstrip).isin(tset)].reset_index(drop=True) - if before != len(dprec): - print(f"dropped {before - len(dprec)} decoys sharing a target stripped sequence") - tprec["_len"] = tprec.peptidoform.map(plen); dprec["_len"] = dprec.peptidoform.map(plen) - rng = np.random.RandomState(0) - tc = tprec.groupby(["_len", "charge"]).size() - keep = [] - for (L, z), n in tc.items(): - pool = dprec.index[(dprec._len == L) & (dprec.charge == z)].to_numpy() - if pool.size: - keep.append(rng.choice(pool, min(int(n), pool.size), replace=False)) - dprec = dprec.loc[np.concatenate(keep)].drop(columns="_len").reset_index(drop=True) - tprec = tprec.drop(columns="_len") - dfrag = dfrag[dfrag["candidate_id"].isin(set(dprec["candidate_id"]))].reset_index(drop=True) - tprec["label"] = "target" - dprec["label"] = "decoy" - dprec["peptidoform"] = "DECOY_" + dprec["peptidoform"].astype(str) - dprec["protein"] = "DECOY_" + dprec["protein"].astype(str) - - # offset decoy candidate_ids so target/decoy ids are disjoint before merge - off = int(tprec["candidate_id"].max()) + 1 - dprec = dprec.copy(); dfrag = dfrag.copy() - dprec["candidate_id"] = dprec["candidate_id"].astype(np.int64) + off - dfrag["candidate_id"] = dfrag["candidate_id"].astype(np.int64) + off - - allp = pd.concat([tprec, dprec], ignore_index=True) - allp = allp.sort_values("precursor_mz", kind="mergesort").reset_index(drop=True) - old2new = {old: new for new, old in enumerate(allp["candidate_id"].tolist())} - allp["candidate_id"] = np.arange(len(allp), dtype=np.uint32) - - allf = pd.concat([tfrag, dfrag], ignore_index=True) - allf["candidate_id"] = allf["candidate_id"].map(old2new).astype(np.uint32) - allf = allf.sort_values("candidate_id", kind="mergesort").reset_index(drop=True) - - allp["candidate_id"] = allp["candidate_id"].astype(np.uint32) - allp["peptidoform_id"] = allp["peptidoform_id"].astype(np.uint32) - allp["base_peptide_id"] = allp["base_peptide_id"].astype(np.uint32) - allp["charge"] = allp["charge"].astype(np.int32) - allp["predicted_irt"] = allp["predicted_irt"].astype(np.float32) - allp["n_fragments"] = allp["n_fragments"].astype(np.int32) - allf["predicted_intensity"] = allf["predicted_intensity"].astype(np.float32) - allf["ordinal"] = allf["ordinal"].astype(np.int32) - allf["frag_charge"] = allf["frag_charge"].astype(np.int32) - - allp.to_parquet(op, index=False); allf.to_parquet(of, index=False) - nt = int((allp.label == "target").sum()); nd = int((allp.label == "decoy").sum()) - print(f"targets={nt} decoys={nd} total_prec={len(allp)} total_frag={len(allf)}") - - -if __name__ == "__main__": - main() diff --git a/scripts/compare_diann.py b/scripts/compare_diann.py deleted file mode 100644 index b6f1c0c..0000000 --- a/scripts/compare_diann.py +++ /dev/null @@ -1,58 +0,0 @@ -"""Compare MuMDIA (Rust MVP) identifications against DIA-NN on the same run. - -Run with the conda env that has pandas + pyarrow, e.g.: - /c/Users/robbi/anaconda3/envs/py312_mumdia/python.exe scripts/compare_diann.py - -Compares unique stripped peptide sequences at 1% FDR and their overlap. -""" -import re -import sys -import pandas as pd - -OURS = "out_run/psms_scored.parquet" -DIANN = "out_diann/report.tsv" -Q = 0.01 - -def strip_mods(pf: str) -> str: - return re.sub(r"\[[^\]]*\]", "", pf) - -def load_ours(): - df = pd.read_parquet(OURS) - t = df[(df["label"] == "target") & (df["peptide_q_value"] <= Q)] - peps = set(t["peptidoform"].map(strip_mods)) - return peps, len(t) - -def load_diann(): - df = pd.read_csv(DIANN, sep="\t") - # DIA-NN 1.9.x: precursor-level run q is 'Q.Value'; peptide sequence is - # 'Stripped.Sequence'; precursor id is 'Precursor.Id'. - qcol = "Q.Value" if "Q.Value" in df.columns else "Global.Q.Value" - d = df[df[qcol] <= Q] - peps = set(d["Stripped.Sequence"].unique()) - prec = d["Precursor.Id"].nunique() if "Precursor.Id" in d.columns else len(d) - pg = None - for c in ("Protein.Group", "Protein.Ids"): - if c in d.columns: - pg = d[c].nunique() - break - return peps, prec, pg - -def main(): - ours, ours_psm = load_ours() - diann, diann_prec, diann_pg = load_diann() - inter = ours & diann - union = ours | diann - print("=" * 60) - print(f"MuMDIA target peptides @1% FDR : {len(ours)} (PSMs {ours_psm})") - print(f"DIA-NN peptides @1% FDR : {len(diann)} (precursors {diann_prec}, protein groups {diann_pg})") - print("-" * 60) - print(f"overlap (shared peptides) : {len(inter)}") - print(f"MuMDIA-only : {len(ours - diann)}") - print(f"DIA-NN-only : {len(diann - ours)}") - print(f"Jaccard : {len(inter)/len(union):.3f}") - if ours: - print(f"fraction of MuMDIA IDs in DIA-NN: {len(inter)/len(ours):.3f}") - print("=" * 60) - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/conflict_features.py b/scripts/conflict_features.py deleted file mode 100644 index bacae3a..0000000 --- a/scripts/conflict_features.py +++ /dev/null @@ -1,334 +0,0 @@ -"""Cross-candidate interference and ambiguity features (non-invasive pass). - -Implements sensitivity_plan backlog items P2.1 (fragment claimant index), -P2.2 (peak-group conflict graph), P2.3 (conflict features) and P5.5 (candidate -ambiguity) as a single pass over existing MuMDIA artifacts. It reads the -per-candidate PSM table and the per-transition chromatogram table, builds a -bounded fragment claimant index, and writes one row of conflict features per -candidate to `conflict.parquet`, joinable into rescoring by `candidate_id`. - -The engine is not modified. This is a first-pass ("fix later") implementation: -correctness, bounded memory, and a working smoke test are prioritised over -completeness. All computation is deterministic (stable sorts, sorted iteration -where floats are summed; no RNG is used). - -Two candidates "claim" the same fragment when - * their fragment m/z agree within `--frag-tol-ppm`, AND - * their candidate apex RT agree within `--rt-window-s`, AND - * their precursor m/z agree within `--mz-precursor-tol` (co-isolatable). - -Only the scalar chromatogram columns (candidate_id, frag_mz, -predicted_intensity) are read; the large list columns (rt, intensity) are never -loaded, which keeps memory bounded. - -House style: literal prose, no em-dashes, no hardcoded secrets. -""" - -import argparse -import math -import sys -import time - -import numpy as np -import pyarrow as pa -import pyarrow.parquet as pq - - -def log(msg): - print(msg, file=sys.stderr, flush=True) - - -def load_psms(path, max_candidates): - """Load per-candidate metadata. Returns a dict of numpy arrays keyed by a - compact 0..C-1 candidate index, plus the sorted original candidate ids.""" - cols = ["candidate_id", "apex_rt", "precursor_mz", "charge", "label", - "base_peptide_id"] - t = pq.read_table(path, columns=cols) - cid = t.column("candidate_id").to_numpy() - # Stable sort by candidate_id for reproducible compact indexing. - order = np.argsort(cid, kind="stable") - cid = cid[order] - apex_rt = t.column("apex_rt").to_numpy()[order] - prec_mz = t.column("precursor_mz").to_numpy()[order] - charge = t.column("charge").to_numpy()[order] - label = np.asarray(t.column("label").to_pylist(), dtype=object)[order] - base_pep = t.column("base_peptide_id").to_numpy()[order] - - if max_candidates is not None and max_candidates < len(cid): - cid = cid[:max_candidates] - apex_rt = apex_rt[:max_candidates] - prec_mz = prec_mz[:max_candidates] - charge = charge[:max_candidates] - label = label[:max_candidates] - base_pep = base_pep[:max_candidates] - - is_decoy = np.array([str(x).lower() == "decoy" for x in label], dtype=bool) - return { - "uids": cid, # sorted original candidate ids (== compact order) - "apex_rt": apex_rt.astype(np.float64), - "prec_mz": prec_mz.astype(np.float64), - "charge": charge.astype(np.int32), - "is_decoy": is_decoy, - "base_pep": base_pep.astype(np.int64), - } - - -def load_prelim(path, uids): - """Load prelim_score from the comp table, aligned to the compact index.""" - t = pq.read_table(path, columns=["candidate_id", "prelim_score"]) - cid = t.column("candidate_id").to_numpy() - score = t.column("prelim_score").to_numpy() - prelim = np.full(len(uids), np.nan, dtype=np.float64) - comp = np.searchsorted(uids, cid) - inb = comp < len(uids) - ok = np.zeros(len(cid), dtype=bool) - ok[inb] = uids[comp[inb]] == cid[inb] - prelim[comp[ok]] = score[ok] - return prelim - - -def load_fragments(path, uids, batch_size=1_000_000): - """Stream the scalar chromatogram columns and map each transition to the - compact candidate index. Fragments of candidates outside `uids` are dropped - (this happens when --max-candidates subsets the candidate set).""" - pf = pq.ParquetFile(path) - cols = ["candidate_id", "frag_mz", "predicted_intensity"] - comp_chunks, mz_chunks, pint_chunks = [], [], [] - for batch in pf.iter_batches(columns=cols, batch_size=batch_size): - cid = batch.column("candidate_id").to_numpy(zero_copy_only=False) - mz = batch.column("frag_mz").to_numpy(zero_copy_only=False) - pint = batch.column("predicted_intensity").to_numpy(zero_copy_only=False) - comp = np.searchsorted(uids, cid) - inb = comp < len(uids) - keep = np.zeros(len(cid), dtype=bool) - keep[inb] = uids[comp[inb]] == cid[inb] - if not keep.any(): - continue - comp_chunks.append(comp[keep].astype(np.int32)) - mz_chunks.append(mz[keep].astype(np.float64)) - pint_chunks.append(pint[keep].astype(np.float64)) - if not comp_chunks: - empty_i = np.zeros(0, dtype=np.int32) - empty_f = np.zeros(0, dtype=np.float64) - return empty_i, empty_f, empty_f - return (np.concatenate(comp_chunks), - np.concatenate(mz_chunks), - np.concatenate(pint_chunks)) - - -def build_bin_index(frag_bin, frag_rt): - """Group fragments by m/z bin, sorted by candidate apex RT within each bin. - Returns the sort order and a dict bin_value -> (start, end) into the sorted - arrays. Within [start, end) the RT array is ascending, so RT-window - neighbours are a range scan rather than a full scan.""" - order = np.lexsort((frag_rt, frag_bin)) # primary bin, secondary rt - sbin = frag_bin[order] - bin_index = {} - if len(sbin): - uniq, starts = np.unique(sbin, return_index=True) - ends = np.append(starts[1:], len(sbin)) - for b, s, e in zip(uniq.tolist(), starts.tolist(), ends.tolist()): - bin_index[b] = (s, e) - return order, bin_index - - -def softmax_entropy(scores): - """Shannon entropy (nats) of the softmax over `scores`. Deterministic given - a fixed input order. Returns 0.0 for a single element.""" - if len(scores) <= 1: - return 0.0 - s = np.asarray(scores, dtype=np.float64) - if not np.all(np.isfinite(s)): - s = np.nan_to_num(s, nan=np.nanmin(s) if np.any(np.isfinite(s)) else 0.0) - z = s - s.max() - w = np.exp(z) - tot = w.sum() - if tot <= 0: - return 0.0 - w = w / tot - nz = w > 0 - return float(-np.sum(w[nz] * np.log(w[nz]))) - - -def main(): - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--psms", required=True) - ap.add_argument("--chrom", required=True) - ap.add_argument("--comp", default=None, - help="optional comp.parquet with prelim_score for ambiguity features") - ap.add_argument("--out", default="conflict.parquet") - ap.add_argument("--frag-tol-ppm", type=float, default=20.0) - ap.add_argument("--rt-window-s", type=float, default=30.0) - ap.add_argument("--mz-precursor-tol", type=float, default=0.5) - ap.add_argument("--max-candidates", type=int, default=None, - help="cap the candidate set (both queries and claimant " - "universe); for fast smoke tests over the full chrom") - args = ap.parse_args() - - t0 = time.time() - log(f"[conflict] loading psms {args.psms}") - meta = load_psms(args.psms, args.max_candidates) - uids = meta["uids"] - C = len(uids) - log(f"[conflict] {C} candidates") - - prelim = None - if args.comp: - log(f"[conflict] loading prelim_score {args.comp}") - prelim = load_prelim(args.comp, uids) - - log(f"[conflict] streaming fragments {args.chrom}") - fc, fmz, fpint = load_fragments(args.chrom, uids) - nfrag = len(fc) - log(f"[conflict] {nfrag} transitions retained") - - # Per-fragment candidate-level RT and precursor m/z (co-isolation keys). - frt = meta["apex_rt"][fc] - fpmz = meta["prec_mz"][fc] - - # ppm-scaled log bins. Two m/z within frag_tol_ppm differ by ~one bin, so - # queries also scan neighbour bins (b-1, b+1). - log_step = math.log1p(args.frag_tol_ppm * 1e-6) - fbin = np.floor(np.log(fmz) / log_step).astype(np.int64) - - order, bin_index = build_bin_index(fbin, frt) - sbin = fbin[order] - srt = frt[order] - smz = fmz[order] - spmz = fpmz[order] - scid = fc[order] - spint = fpint[order] - - ppm = args.frag_tol_ppm * 1e-6 - rtw = args.rt_window_s - mztol = args.mz_precursor_tol - - n_frags = np.zeros(C, dtype=np.int64) - claim_sum = np.zeros(C, dtype=np.float64) - claim_max = np.zeros(C, dtype=np.int64) - contested_cnt = np.zeros(C, dtype=np.int64) - contested_int = np.zeros(C, dtype=np.float64) - total_int = np.zeros(C, dtype=np.float64) - group_sets = [set() for _ in range(C)] - - log("[conflict] scanning claimant index") - for p in range(len(sbin)): - b = sbin[p] - rt_i = srt[p] - mz_i = smz[p] - pmz_i = spmz[p] - c_i = scid[p] - tol_abs = mz_i * ppm - rt_lo = rt_i - rtw - rt_hi = rt_i + rtw - others = None - for nb in (b - 1, b, b + 1): - se = bin_index.get(nb) - if se is None: - continue - s, e = se - seg_rt = srt[s:e] - lo = s + int(np.searchsorted(seg_rt, rt_lo, side="left")) - hi = s + int(np.searchsorted(seg_rt, rt_hi, side="right")) - if hi <= lo: - continue - mzs = smz[lo:hi] - pmzs = spmz[lo:hi] - cids = scid[lo:hi] - m = (np.abs(mzs - mz_i) <= tol_abs) & \ - (np.abs(pmzs - pmz_i) <= mztol) & \ - (cids != c_i) - if m.any(): - if others is None: - others = set() - others.update(int(x) for x in cids[m]) - cc = 0 if others is None else len(others) - n_frags[c_i] += 1 - claim_sum[c_i] += cc - if cc > claim_max[c_i]: - claim_max[c_i] = cc - total_int[c_i] += spint[p] - if cc > 0: - contested_cnt[c_i] += 1 - contested_int[c_i] += spint[p] - group_sets[c_i].update(others) - - log("[conflict] reducing per-candidate features") - with np.errstate(invalid="ignore", divide="ignore"): - nf = n_frags.astype(np.float64) - safe_nf = np.where(nf > 0, nf, 1.0) - claimant_count_mean = claim_sum / safe_nf - contested_frac = contested_cnt / safe_nf - unique_cnt = n_frags - contested_cnt - unique_frac = unique_cnt / safe_nf - safe_int = np.where(total_int > 0, total_int, 1.0) - shared_intensity_frac = contested_int / safe_int - # Candidates with no fragments get zeroed features (documented default). - claimant_count_mean[nf == 0] = 0.0 - contested_frac[nf == 0] = 0.0 - unique_frac[nf == 0] = 0.0 - shared_intensity_frac[total_int == 0] = 0.0 - conflict_group_size = np.array([len(g) for g in group_sets], dtype=np.int64) - - out = { - "candidate_id": uids.astype(np.uint32), - "claimant_count_mean": claimant_count_mean, - "claimant_count_max": claim_max, - "contested_fragment_count": contested_cnt, - "contested_fragment_frac": contested_frac, - "unique_fragment_count": unique_cnt, - "unique_fragment_frac": unique_frac, - "conflict_group_size": conflict_group_size, - "shared_intensity_frac": shared_intensity_frac, - } - - if prelim is not None: - log("[conflict] computing candidate ambiguity (P5.5)") - base_pep = meta["base_pep"] - is_decoy = meta["is_decoy"] - margin_alt = np.full(C, np.nan, dtype=np.float64) - margin_dec = np.full(C, np.nan, dtype=np.float64) - n_comp = np.zeros(C, dtype=np.int64) - comp_entropy = np.zeros(C, dtype=np.float64) - for c in range(C): - grp = sorted(group_sets[c]) # deterministic order - n_comp[c] = len(grp) - if not grp: - continue - self_score = prelim[c] - alt = [prelim[o] for o in grp if base_pep[o] != base_pep[c] - and np.isfinite(prelim[o])] - dec = [prelim[o] for o in grp if is_decoy[o] and np.isfinite(prelim[o])] - if alt: - margin_alt[c] = self_score - max(alt) - if dec: - margin_dec[c] = self_score - max(dec) - scores = [self_score] + [prelim[o] for o in grp] - comp_entropy[c] = softmax_entropy(scores) - out["margin_to_best_alt_peptide"] = margin_alt - out["margin_to_best_decoy"] = margin_dec - out["n_competitors_within_group"] = n_comp - out["competitor_score_entropy"] = comp_entropy - - table = pa.table(out) - pq.write_table(table, args.out) - - # Summary - mean_group = float(conflict_group_size.mean()) if C else 0.0 - mean_contested = float(contested_frac.mean()) if C else 0.0 - dt = time.time() - t0 - log("[conflict] done") - print(f"conflict.parquet written to {args.out}") - print(f"candidates: {C}") - print(f"transitions scanned: {nfrag}") - print(f"mean conflict_group_size: {mean_group:.4f}") - print(f"mean contested_fragment_frac: {mean_contested:.4f}") - if args.max_candidates is not None: - print(f"NOTE: --max-candidates={args.max_candidates} limits both the " - f"query set and the claimant universe; counts are lower bounds.") - print(f"runtime_s: {dt:.1f}") - - -if __name__ == "__main__": - main() diff --git a/scripts/copy_irt.py b/scripts/copy_irt.py deleted file mode 100644 index 98db86e..0000000 --- a/scripts/copy_irt.py +++ /dev/null @@ -1,31 +0,0 @@ -"""Copy calibrated predicted_irt from a source precursor table into a target -precursor table, matching by peptidoform. Avoids re-running DeepLC calibration -when only the fragment set changed. - -Usage: python copy_irt.py -""" -import sys -import numpy as np -import pyarrow as pa -import pyarrow.parquet as pq - - -def main(): - src_path, dst_path = sys.argv[1], sys.argv[2] - src = pq.read_table(src_path).to_pydict() - irt_by_pf = {} - for pf, irt in zip(src["peptidoform"], src["predicted_irt"]): - irt_by_pf[pf] = irt - - dst = pq.read_table(dst_path) - pforms = dst.column("peptidoform").to_pylist() - new = np.array([irt_by_pf.get(pf, 0.0) for pf in pforms], dtype=np.float32) - miss = int((new == 0.0).sum()) - idx = dst.schema.get_field_index("predicted_irt") - dst = dst.set_column(idx, "predicted_irt", pa.array(new, pa.float32())) - pq.write_table(dst, dst_path) - print(f"copied iRT for {len(pforms)} rows ({miss} unmatched)") - - -if __name__ == "__main__": - main() diff --git a/scripts/deeplc_calibrate.py b/scripts/deeplc_calibrate.py deleted file mode 100644 index 4a40e73..0000000 --- a/scripts/deeplc_calibrate.py +++ /dev/null @@ -1,79 +0,0 @@ -"""Per-run DeepLC calibration (PLAN.md Stage B / Section 5 improvement): calibrate -the DeepLC multitask model to this run's observed RTs (from confident seed PSMs), -then write calibrated predicted_irt into the library precursor table. - -Targets are predicted+calibrated; shift-decoys inherit their target's calibrated -RT (their peptidoform is `DECOY_`). - -Usage: python deeplc_calibrate.py -""" -import sys -import numpy as np -import pyarrow as pa -import pyarrow.parquet as pq - - -def agg(a): - a = np.asarray(a, dtype=np.float64) - return a.mean(axis=1) if a.ndim == 2 else a - - -def main(): - lib_in, seed_path, lib_out = sys.argv[1:4] - import deeplc - from psm_utils import PSM, PSMList - - lib = pq.read_table(lib_in) - pform = lib.column("peptidoform").to_pylist() - label = lib.column("label").to_pylist() - - # Reference: confident target seed PSMs (peptidoform + observed RT). - seed = pq.read_table(seed_path).to_pydict() - ref = {} - for i in range(len(seed["peptidoform"])): - if seed["label"][i] == "target" and seed["spectrum_q"][i] <= 0.01: - ref[seed["peptidoform"][i]] = seed["observed_rt"][i] - ref_psms = PSMList(psm_list=[ - PSM(peptidoform=pf, retention_time=rt, spectrum_id=str(k)) - for k, (pf, rt) in enumerate(ref.items()) - ]) - print(f"calibration reference: {len(ref_psms)} confident seed peptides") - - # Unique target peptidoforms to predict+calibrate. - uniq = [] - seen = set() - for pf, lb in zip(pform, label): - if lb == "target" and pf not in seen: - seen.add(pf) - uniq.append(pf) - print(f"predicting {len(uniq)} unique target peptidoforms") - - cal = {} - chunk = 200_000 - for s in range(0, len(uniq), chunk): - batch = uniq[s:s + chunk] - preds = agg(deeplc.predict_and_calibrate(batch, ref_psms)) - for pf, v in zip(batch, preds): - cal[pf] = float(v) - print(f" {min(s + chunk, len(uniq))}/{len(uniq)}") - - # Assign: targets from cal; decoys from their DECOY_-stripped target. - new_irt = np.empty(len(pform), dtype=np.float32) - miss = 0 - for i, (pf, lb) in enumerate(zip(pform, label)): - key = pf[6:] if pf.startswith("DECOY_") else pf - if key in cal: - new_irt[i] = cal[key] - else: - new_irt[i] = 0.0 - miss += 1 - print(f"assigned iRT; {miss} rows without a prediction") - - idx = lib.schema.get_field_index("predicted_irt") - lib = lib.set_column(idx, "predicted_irt", pa.array(new_irt, pa.float32())) - pq.write_table(lib, lib_out) - print(f"wrote calibrated library: {lib_out}") - - -if __name__ == "__main__": - main() diff --git a/scripts/empirical_reanalyse.py b/scripts/empirical_reanalyse.py deleted file mode 100644 index df1c61a..0000000 --- a/scripts/empirical_reanalyse.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Empirical transition reanalysis (a first, DIA-NN-`--reanalyse`-style pass). - -From a first-pass rescored result, learn how reliably each PREDICTED-RANK transition -is actually observed in-peak among confident target IDs, then reweight (or prune) -the library fragments by that rank reliability. The transform depends only on a -transition's predicted-intensity RANK within its candidate, not on its identity, so -it is applied SYMMETRICALLY to targets and decoys and cannot by itself distort the -target-decoy null. The user then re-extracts/re-rescores with the new library. - -This mirrors DIA-NN's empirical library: weak/interfered low-rank transitions are -down-weighted or dropped, high-rank signature transitions kept. - -Usage: - python empirical_reanalyse.py - [--q 0.01] [--mode reweight|prune] [--prune-thr 0.25] - -Caveat (leakage): rank reliability is learned globally from all confident IDs here. -The rigorous version learns it out-of-fold / from other runs (see the audit's P3); -this global version is a usable first pass, not the leakage-free final. -""" -import re -import sys - -import numpy as np -import pyarrow.dataset as pds -import pyarrow.parquet as pq -import pyarrow as pa - -strip = lambda p: re.sub(r"\[[^\]]*\]", "", str(p)) - - -def arg(flag, default): - return sys.argv[sys.argv.index(flag) + 1] if flag in sys.argv else default - - -def main(): - scored_p, chrom_p, lib_in, lib_out = sys.argv[1:5] - q_conf = float(arg("--q", "0.01")) - mode = arg("--mode", "reweight") - prune_thr = float(arg("--prune-thr", "0.25")) - - # Confident target candidates from the first pass. - sc = pds.dataset(scored_p).to_table(columns=["candidate_id", "label", "q_value"]).to_pandas() - conf = set( - sc.candidate_id[(sc.label == "target") & (sc.q_value <= q_conf)].astype(int) - ) - if not conf: - raise SystemExit("empirical_reanalyse: no confident target IDs at the given q") - - # Library fragments; rank per candidate by predicted intensity (0 = strongest). - lf = pds.dataset(lib_in).to_table().to_pandas() - lf = lf.sort_values(["candidate_id", "predicted_intensity"], ascending=[True, False]) - lf["rank"] = lf.groupby("candidate_id").cumcount() - - # Observed-in-peak set from the chromatograms: a (candidate_id, frag_name) whose - # XIC has any nonzero intensity is "present". - ch = pds.dataset(chrom_p).to_table(columns=["candidate_id", "frag_name", "intensity"]).to_pandas() - ch = ch[ch.candidate_id.astype(int).isin(conf)] - present = set() - for cid, fn, inten in zip(ch.candidate_id, ch.frag_name, ch.intensity): - arr = np.asarray(inten, dtype=float) - if arr.size and arr.max() > 0.0: - present.add((int(cid), str(fn))) - - # Rank reliability = fraction of confident candidates whose rank-r transition is - # present in-peak (denominator = confident candidates that HAVE a rank-r transition). - conf_lf = lf[lf.candidate_id.astype(int).isin(conf)] - max_rank = int(conf_lf["rank"].max()) - retention = np.ones(max_rank + 1) - for r in range(max_rank + 1): - rows = conf_lf[conf_lf["rank"] == r] - if len(rows) == 0: - continue - obs = sum((int(c), str(n)) in present for c, n in zip(rows.candidate_id, rows.name)) - retention[r] = obs / len(rows) - print("rank reliability (fraction observed in-peak among confident IDs):") - for r in range(min(max_rank + 1, 15)): - print(f" rank {r:2d}: {retention[r]:.3f}") - - # Apply symmetrically by rank to the WHOLE library (targets + decoys). - rk = lf["rank"].to_numpy().clip(0, max_rank) - rel = retention[rk] - if mode == "prune": - keep = rel >= prune_thr - lf = lf[keep].copy() - print(f"prune: kept {int(keep.sum())}/{len(keep)} transitions (rank reliability >= {prune_thr})") - else: - lf = lf.copy() - lf["predicted_intensity"] = (lf["predicted_intensity"].to_numpy() * rel).astype(np.float32) - print(f"reweight: scaled {len(lf)} predicted intensities by rank reliability") - - lf = lf.drop(columns=["rank"]).sort_values("candidate_id").reset_index(drop=True) - pq.write_table(pa.Table.from_pandas(lf, preserve_index=False), lib_out) - print(f"wrote empirical library -> {lib_out} ({len(lf)} fragments)") - - -if __name__ == "__main__": - main() diff --git a/scripts/entrapment_corrected.py b/scripts/entrapment_corrected.py deleted file mode 100644 index bef3665..0000000 --- a/scripts/entrapment_corrected.py +++ /dev/null @@ -1,128 +0,0 @@ -"""Contaminant-aware entrapment true-FDR measurement. - -The entrapment design spikes the human proteome into a pure-E. coli search and -treats every human ID as false. That over-counts: real handling contaminants -(keratins, albumin) and abundant proteins genuinely present are corroborated by -an independent engine (DIA-NN) on the same file. This script re-measures E. coli -at a true 1% FDR while EXCLUDING demonstrably-real human peptides from (a) the -training negatives and (b) the FDR false count, under three definitions: - raw - every human ID is false (original, pessimistic) - crap - exclude cRAP-style contaminant proteins (keratin/albumin/Hb/actin/...) - diann - additionally exclude human sequences DIA-NN also identifies (present) - -Usage: python entrapment_corrected.py [tag] -Operates POST-HOC on a shipped rescorer's scores (fixed), recomputing E. coli at -true 1% FDR under each accounting definition. The `raw` column reproduces the -shipped baseline; `crap`/`diann` show the count once contaminants are removed -from the false population. (An earlier retrain sweep confirmed excluding -contaminants from the TRAINING negatives is near-neutral, since they are a tiny -fraction of the ~40k human negatives; the accounting is the first-order fix.) -""" -import json -import re -import sys - -import numpy as np -import pandas as pd - -COMP = sys.argv[1] -TAG = sys.argv[2] if len(sys.argv) > 2 else COMP - -# cRAP-style contaminant entry-name tokens (UniProt _HUMAN naming in the protein col). -CRAP = re.compile( - r"K[12][CH]\d|K22E|K1H\d|K2M\d|KRT\d|KRHB|ALBU|HB[ABDGE]\d?_|" - r"ACT[BGSC]_|TBB\d|TBA\d|TRY[PB]|IGHG|IGKC|IGLC|CO\dA\d|CASA|CASB|CASK", - re.I, -) - - -def strip(pf): - return re.sub(r"\[[^\]]*\]", "", str(pf)).replace("DECOY_", "") - - -# DIA-NN-corroborated human sequences on the same raw file (independent-engine presence). -dn = pd.read_csv("out_diann/report_ent.tsv", sep="\t") -pc = "Protein.Names" if "Protein.Names" in dn.columns else "Protein.Ids" -dn = dn[dn["Q.Value"] <= 0.01] -dn_hum = set( - dn[dn[pc].astype(str).str.contains("_HUMAN") & ~dn[pc].astype(str).str.contains("_ECOLI")][ - "Stripped.Sequence" - ] -) - -# library ratio N_ecoli_lib / N_human_lib (contaminant fraction of the 343k-peptide -# human library is negligible, so the ratio is held constant across definitions). -pep = pd.read_parquet("out_full/pep_ent.parquet", columns=["protein"]) -ish = pep.protein.str.contains("_HUMAN") & ~pep.protein.str.contains("_ECOLI") -RATIO = int((~ish).sum()) / int(ish.sum()) - -c = pd.read_parquet(COMP) -c["strip"] = c.peptidoform.map(strip) -c["is_decoy"] = c.label.eq("decoy") -pcol = "protein_group" if "protein_group" in c.columns else "protein" -c["is_human"] = c[pcol].str.contains("_HUMAN") & ~c[pcol].str.contains("_ECOLI") -c["crap"] = c.is_human & c[pcol].map(lambda p: bool(CRAP.search(str(p)))) -c["diann_corr"] = c.is_human & c.strip.isin(dn_hum) -SCORE = c.score.to_numpy(float) -dec = c.is_decoy.to_numpy() -hum = c.is_human.to_numpy() - - -def ent_q(sc, is_entrap, is_real, ratio): - order = np.argsort(-sc, kind="stable") - ne = nr = 0 - fdr = np.ones(len(sc)) - for rank, i in enumerate(order): - if is_entrap[i]: - ne += 1 - elif is_real[i]: - nr += 1 - fdr[rank] = ratio * ne / max(1, nr) - q = np.ones(len(sc)) - qmin = 1.0 - for rank in range(len(sc) - 1, -1, -1): - qmin = min(qmin, fdr[rank]) - q[order[rank]] = qmin - return q - - -def contam_mask(acct): - if acct == "raw": - return np.zeros(len(c), bool) - if acct == "crap": - return c.crap.to_numpy() - return (c.crap | c.diann_corr).to_numpy() - - -def eco_at_true1(acct): - contam = contam_mask(acct) - is_entrap = hum & ~contam & ~dec # valid false population (exclude human decoys) - is_real = ~dec & ~hum # E. coli targets - q = ent_q(SCORE, is_entrap, is_real, RATIO) - gate = q <= 0.01 - eco = c.loc[is_real & gate, "strip"].nunique() - leak = c.loc[is_entrap & gate, "strip"].nunique() - return eco, leak - - -print(f"=== {TAG} | ratio(eco/hum)={RATIO:.3f} | rows={len(c)} | " - f"human PSMs={int(hum.sum())} cRAP={int(c.crap.sum())} DIANN-corr={int(c.diann_corr.sum())} ===") - -# Framing A: hold the SHIPPED reported gate fixed (q_value<=0.01); the E. coli -# count is unchanged, but the TRUE FDR there drops once contaminants are not -# counted as false -> shows the shipped count is conservative. -if "q_value" in c.columns: - gate = c.q_value.to_numpy() <= 0.01 - eco_fixed = c.loc[(~dec) & (~hum) & gate, "strip"].nunique() - print(f" [A] at shipped gate (q<=0.01): E.coli={eco_fixed} true FDR by accounting:") - for a in ("raw", "crap", "diann"): - contam = contam_mask(a) - false_h = c.loc[(hum & ~contam & ~dec) & gate, "strip"].nunique() - print(f" acct={a:6}: true FDR = {false_h*RATIO/max(1,eco_fixed)*100:4.2f}% (false human seqs={false_h})") - -# Framing B: re-threshold to a genuine true 1% under each accounting (score sweep) -# -> how many E. coli when the FDR budget is spent against the CORRECTED false set. -print(" [B] re-thresholded to true 1% FDR (score sweep):") -for a in ("raw", "crap", "diann"): - eco, leak = eco_at_true1(a) - print(f" acct={a:6}: E.coli @ true 1% = {eco:5d} (human leak: {leak})") diff --git a/scripts/entrapment_holdout.py b/scripts/entrapment_holdout.py deleted file mode 100644 index 94e16fc..0000000 --- a/scripts/entrapment_holdout.py +++ /dev/null @@ -1,105 +0,0 @@ -"""Held-out entrapment validation harness. - -The entrapment-trained scorer uses the human proteome as negatives, but training, -FDR, and reporting on the SAME human population overfits (research leak). This -harness splits the human entrapment proteins into two disjoint halves by a stable -protein hash, trains the classifier on E. coli targets (positive) vs human-TRAIN -(negative), and then evaluates FDR/sensitivity against the UNSEEN human-TEST null. -That held-out E. coli count at 1% is the honest number. - -Usage: - python entrapment_holdout.py [--folds 3] [--q 0.01] - -Requires an env with scikit-learn + pyarrow (py312_mumdia). Deterministic. -""" -import hashlib -import re -import sys - -import numpy as np -import pyarrow.dataset as pds -from sklearn.ensemble import HistGradientBoostingClassifier - -strip = lambda p: re.sub(r"\[[^\]]*\]", "", str(p)) - - -def arg(flag, default): - return sys.argv[sys.argv.index(flag) + 1] if flag in sys.argv else default - - -def half(protein: str) -> int: - # Stable (non-randomized) split of a protein string into half 0/1. - h = hashlib.md5(str(protein).encode("utf-8")).hexdigest() - return int(h, 16) & 1 - - -def ent_q(score, is_entrap, is_real, ratio): - order = np.argsort(-score, kind="stable") - ne = nr = 0 - fdr = np.ones(len(score)) - for rank, i in enumerate(order): - if is_entrap[i]: - ne += 1 - elif is_real[i]: - nr += 1 - fdr[rank] = ratio * ne / max(1, nr) - q = np.ones(len(score)) - qmin = 1.0 - for rank in range(len(score) - 1, -1, -1): - qmin = min(qmin, fdr[rank]) - q[order[rank]] = qmin - return q - - -def main(): - path = sys.argv[1] - q_cut = float(arg("--q", "0.01")) - t = pds.dataset(path).to_table().to_pandas() - pcol = "protein_group" if "protein_group" in t.columns else "protein" - dec = t.label.eq("decoy").to_numpy() - human = (t[pcol].str.contains("_HUMAN") & ~t[pcol].str.contains("_ECOLI")).to_numpy() - ecoli_tgt = (~dec) & (~human) - hh = t[pcol].map(half).to_numpy() - train_neg = human & (hh == 0) # human-TRAIN negatives - test_neg = human & (hh == 1) # human-TEST null (unseen) - - meta = {"candidate_id", "label", "peptidoform", "protein", "protein_group", - "base_peptide_id", "charge", "q_value", "peptide_q_value", "pg_q_value", - "global_q_value", "score", "prelim_score", "source"} - feats = [c for c in t.columns if c not in meta and np.issubdtype(t[c].dtype, np.number)] - X = np.nan_to_num(t[feats].to_numpy(np.float64), posinf=0.0, neginf=0.0) - - # Train ONLY on E. coli targets (pos) vs human-TRAIN (neg); human-TEST + decoys - # are never seen in training. - tr = ecoli_tgt | train_neg - y = np.where(ecoli_tgt, 1, 0)[tr] - if len(np.unique(y)) < 2: - raise SystemExit("entrapment_holdout: need both E.coli targets and human-train negatives") - m = HistGradientBoostingClassifier(random_state=0, early_stopping=False) - m.fit(X[tr], y) - score = m.predict_proba(X)[:, 1] - - # Held-out evaluation: null = human-TEST (unseen). Library-size ratio correction. - ratio = int(ecoli_tgt.sum()) / max(1, int(test_neg.sum())) - q = ent_q(score, test_neg, ecoli_tgt, ratio) - gate = q <= q_cut - eco_heldout = t.loc[ecoli_tgt & gate, "peptidoform"].map(strip).nunique() - - # For contrast: FDR at the shipped q_value cutoff, measured on the held-out null. - shipped = None - if "q_value" in t.columns: - g2 = t.q_value.to_numpy() <= q_cut - eco_s = t.loc[ecoli_tgt & g2, "peptidoform"].map(strip).nunique() - leak = int((test_neg & g2).sum()) - shipped = (eco_s, ratio * leak / max(1, int((ecoli_tgt & g2).sum())) * 100) - - print(f"=== held-out entrapment ({path}) ===") - print(f" E.coli targets={int(ecoli_tgt.sum())} human train-neg={int(train_neg.sum())} " - f"test-null={int(test_neg.sum())} ratio={ratio:.3f}") - print(f" held-out E.coli stripped seqs @ {q_cut:.0%} (unseen null): {eco_heldout}") - if shipped: - print(f" at shipped q<= {q_cut:.0%}: E.coli={shipped[0]}, true FDR on held-out null = {shipped[1]:.2f}%") - - -if __name__ == "__main__": - main() diff --git a/scripts/feature_ablation.py b/scripts/feature_ablation.py deleted file mode 100644 index 7ab1d3f..0000000 --- a/scripts/feature_ablation.py +++ /dev/null @@ -1,352 +0,0 @@ -#!/usr/bin/env python -"""Grouped feature-family ablation for MuMDIA (spec 03 Sections 4-5, backlog P4.3). - -Estimates how much each feature family contributes to sensitivity, with strict -leakage guards, on the comp.parquet feature table: - - * grouped cross-validation with the precursor (peptidoform + charge) held whole - inside a single fold, so no row of a precursor trains a model that then scores - another row of the same precursor; - * imputation and standardization fit inside each training fold only; - * target-decoy q-values computed from out-of-fold scores; the metric is the - number of target rows passing an empirical FDP threshold; - * two model families (L2 logistic regression, HistGradientBoosting). - -Ablations reported per family: full-minus-family (does removing it drop targets?) -and minimal-baseline-plus-family (does adding it to a small baseline help?). - -Reads Parquet only; deterministic (all hashing / seeds fixed). -""" - -from __future__ import annotations - -import argparse -import csv -import hashlib -import json -import os -import sys -import warnings - -import numpy as np -import pyarrow.parquet as pq - -warnings.filterwarnings("ignore") # silence sklearn convergence chatter - -SEED = 0 - -# Columns that must never enter the feature matrix (ids, labels, targets, -# leakage-prone scores). `charge` is treated as meta per the task spec. -META_COLUMNS = { - "candidate_id", "label", "base_peptide_id", "peptidoform", "protein", - "apex_rt", "precursor_mz", "prelim_score", "charge", - "q_value", "peptide_q_value", "pg_q_value", "global_q_value", - "score", "protein_group", "source", -} - - -# --------------------------------------------------------------------------- # -def load_registry(path): - """Parse feature_registry.yaml (name -> family) without a YAML dependency. - - The file is a flat two-space-indented mapping under `features:`; each feature - name is a quoted key at indent 2 and `family:` is a quoted value at indent 4. - """ - fam = {} - cur = None - in_features = False - with open(path, "r", encoding="utf-8") as fh: - for line in fh: - if line.rstrip("\n") == "features:": - in_features = True - continue - if not in_features: - continue - if line and not line[0].isspace() and line.strip(): - # a new top-level key ends the features block - break - stripped = line.strip() - if not stripped or stripped.startswith("#"): - continue - indent = len(line) - len(line.lstrip(" ")) - if indent == 2 and stripped.endswith(":"): - cur = stripped[:-1].strip().strip('"').strip("'") - elif indent == 4 and stripped.startswith("family:") and cur is not None: - val = stripped.split(":", 1)[1].strip().strip('"').strip("'") - fam[cur] = val - return fam - - -def stable_fold(key, folds): - h = hashlib.sha1(key.encode("utf-8")).hexdigest() - return int(h, 16) % folds - - -def target_ids_at_fdp(scores, is_target, fdp): - """Count target rows passing `fdp` using target-decoy q-values. - - q at a score threshold = (n_decoys + 1) / max(n_targets, 1); ties share the - group-end value; q-values are monotonised from the least confident end. - """ - order = np.argsort(-scores, kind="mergesort") - s = scores[order] - t = is_target[order].astype(np.int64) - d = 1 - t - cum_t = np.cumsum(t) - cum_d = np.cumsum(d) - fdr = (cum_d + 1.0) / np.maximum(cum_t, 1) - n = len(s) - # tie handling: every row in an equal-score group gets the group-end fdr - q = np.empty(n) - i = 0 - while i < n: - j = i - while j + 1 < n and s[j + 1] == s[i]: - j += 1 - q[i:j + 1] = fdr[j] - i = j + 1 - qv = np.minimum.accumulate(q[::-1])[::-1] - return int(((qv <= fdp) & (t == 1)).sum()) - - -# --------------------------------------------------------------------------- # -def build_model(kind): - from sklearn.ensemble import HistGradientBoostingClassifier - from sklearn.linear_model import LogisticRegression - - if kind == "logreg": - # lbfgs must be allowed to converge: on the full ~355 collinear standardized - # features it needs ~850 iterations, and an under-converged fit produces - # degenerate scores (0 targets at 1% FDP). It stops early once tol is met, so - # a high cap costs nothing on the small ablation subsets. liblinear coordinate - # descent converges but is ~15x slower here. - return LogisticRegression( - penalty="l2", C=1.0, solver="lbfgs", max_iter=2000, - tol=1e-3, random_state=SEED - ) - if kind == "hgb": - return HistGradientBoostingClassifier( - max_iter=200, learning_rate=0.1, max_depth=None, - l2_regularization=1.0, random_state=SEED - ) - raise ValueError(kind) - - -def main(): - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--features", required=True, help="comp.parquet") - ap.add_argument("--registry", required=True, help="feature_registry.yaml") - ap.add_argument("--out", default=None, help="output directory") - ap.add_argument("--folds", type=int, default=3) - ap.add_argument("--fdp", type=float, default=0.01) - ap.add_argument("--model", choices=["logreg", "hgb", "both"], default="both") - ap.add_argument("--max-rows", type=int, default=0, help="0 = all rows") - ap.add_argument("--clip-sd", type=float, default=8.0, - help="winsorize standardized features to +/- this many SD " - "(0 disables); tames outlier-driven logreg miscalibration") - args = ap.parse_args() - - from sklearn.impute import SimpleImputer - from sklearn.preprocessing import StandardScaler - - out_dir = args.out or os.path.join(os.path.dirname(os.path.abspath(args.features)), - "feature_ablation") - os.makedirs(out_dir, exist_ok=True) - - registry = load_registry(args.registry) - - # ---- load comp.parquet (optionally capped) -------------------------------- - schema = pq.read_schema(args.features) - import pyarrow as pa - numeric_cols = [ - n for n in schema.names - if pa.types.is_floating(schema.field(n).type) - or pa.types.is_integer(schema.field(n).type) - ] - feature_cols = [c for c in numeric_cols if c not in META_COLUMNS] - read_cols = feature_cols + ["label", "peptidoform", "charge"] - - if args.max_rows and args.max_rows > 0: - pf = pq.ParquetFile(args.features) - batches = [] - got = 0 - for b in pf.iter_batches(batch_size=50000, columns=read_cols): - batches.append(b) - got += b.num_rows - if got >= args.max_rows: - break - table = pa.Table.from_batches(batches).slice(0, args.max_rows) - else: - table = pq.read_table(args.features, columns=read_cols) - - n = table.num_rows - label = np.array(table.column("label").to_pylist()) - is_target = (label == "target").astype(np.int64) - peptidoform = table.column("peptidoform").to_pylist() - charge = table.column("charge").to_numpy() - - # ---- leakage guard -------------------------------------------------------- - leaked = [c for c in feature_cols if c in META_COLUMNS] - assert not leaked, f"leakage: meta columns in feature matrix: {leaked}" - - # ---- feature matrix ------------------------------------------------------- - X = table.select(feature_cols).to_pandas().to_numpy(dtype=np.float64) - X[~np.isfinite(X)] = np.nan # inf -> nan for the imputer - - # ---- family map ----------------------------------------------------------- - fam_of = {c: registry.get(c, "unknown") for c in feature_cols} - families = {} - for j, c in enumerate(feature_cols): - families.setdefault(fam_of[c], []).append(j) - fam_breakdown = {f: len(idx) for f, idx in sorted(families.items())} - - # ---- minimal baseline ----------------------------------------------------- - minimal_names = [c for c in feature_cols if registry.get(c) == "minimal"] - if len(minimal_names) >= 5: - baseline_names = minimal_names - else: - baseline_names = feature_cols[:5] # fallback: first 5 available - baseline_idx = [feature_cols.index(c) for c in baseline_names] - - print(f"[data] rows={n} targets={int(is_target.sum())} " - f"decoys={int((1 - is_target).sum())}") - print(f"[features] {len(feature_cols)} numeric feature columns " - f"(of {len(schema.names)} total)") - print(f"[families] {len(families)}: " + - ", ".join(f"{f}={c}" for f, c in fam_breakdown.items())) - print(f"[baseline] {len(baseline_idx)} features " - f"({'minimal tier' if len(minimal_names) >= 5 else 'first-5 fallback'})") - - # ---- grouped folds -------------------------------------------------------- - folds_arr = np.array( - [stable_fold(f"{peptidoform[i]}|{int(charge[i])}", args.folds) - for i in range(n)], - dtype=np.int64, - ) - fold_sizes = [int((folds_arr == k).sum()) for k in range(args.folds)] - print(f"[cv] {args.folds} grouped folds, sizes={fold_sizes}") - - # ---- configs to evaluate -------------------------------------------------- - all_idx = list(range(len(feature_cols))) - configs = {"full": all_idx, "baseline": baseline_idx} - for fam, idx in families.items(): - fam_set = set(idx) - configs[f"minus::{fam}"] = [j for j in all_idx if j not in fam_set] - configs[f"base+::{fam}"] = sorted(set(baseline_idx) | fam_set) - - models = ["logreg", "hgb"] if args.model == "both" else [args.model] - - # ---- run: per model, per fold fit shared imputer/scaler once -------------- - results = {} # (model) -> {config -> ids} - for mkind in models: - print(f"\n[model] {mkind}") - # precompute per-fold transformed full matrices - fold_data = [] - for k in range(args.folds): - te = folds_arr == k - tr = ~te - imp = SimpleImputer(strategy="median", keep_empty_features=True) - scl = StandardScaler() - Xtr = scl.fit_transform(imp.fit_transform(X[tr])) - Xte = scl.transform(imp.transform(X[te])) - if args.clip_sd and args.clip_sd > 0: - Xtr = np.clip(Xtr, -args.clip_sd, args.clip_sd) - Xte = np.clip(Xte, -args.clip_sd, args.clip_sd) - fold_data.append((tr, te, Xtr, Xte)) - - config_scores = {c: np.full(n, -np.inf) for c in configs} - for ci, (cname, idxs) in enumerate(configs.items()): - idxs = np.array(idxs, dtype=int) - for (tr, te, Xtr, Xte) in fold_data: - model = build_model(mkind) - model.fit(Xtr[:, idxs], is_target[tr]) - proba = model.predict_proba(Xte[:, idxs])[:, 1] - config_scores[cname][te] = proba - ids = {c: target_ids_at_fdp(config_scores[c], is_target, args.fdp) - for c in configs} - results[mkind] = ids - print(f" full={ids['full']} baseline={ids['baseline']}") - - # ---- build table ---------------------------------------------------------- - rows = [] - for mkind in models: - ids = results[mkind] - ids_full = ids["full"] - ids_base = ids["baseline"] - tol_n = max(1, round(0.001 * ids_full)) - for fam in sorted(families): - new_ids = ids[f"base+::{fam}"] - minus_ids = ids[f"minus::{fam}"] - rel_gain = (new_ids - ids_base) / ids_base if ids_base else 0.0 - delta_full = minus_ids - ids_full # negative => removing hurts - if delta_full < -tol_n: - rec = "KEEP" - elif delta_full > tol_n: - rec = "HARMFUL" # removing it improves the model - elif rel_gain > 0.01: - rec = "REDUNDANT_BUT_INFORMATIVE" # helps alone, redundant in full - else: - rec = "REDUNDANT" - rows.append({ - "feature_family": fam, - "n_features": len(families[fam]), - "baseline_identifications": ids_base, - "new_identifications": new_ids, - "relative_gain": round(rel_gain, 5), - "full_identifications": ids_full, - "minus_identifications": minus_ids, - "delta_vs_full": delta_full, - "model": mkind, - "recommendation": rec, - }) - - # ---- write ---------------------------------------------------------------- - csv_path = os.path.join(out_dir, "feature_ablation.csv") - json_path = os.path.join(out_dir, "feature_ablation.json") - fieldnames = ["feature_family", "n_features", "baseline_identifications", - "new_identifications", "relative_gain", "full_identifications", - "minus_identifications", "delta_vs_full", "model", "recommendation"] - with open(csv_path, "w", newline="") as fh: - w = csv.DictWriter(fh, fieldnames=fieldnames) - w.writeheader() - for r in rows: - w.writerow(r) - summary = { - "params": {"folds": args.folds, "fdp": args.fdp, "model": args.model, - "max_rows": args.max_rows}, - "n_rows": n, - "n_targets": int(is_target.sum()), - "n_decoys": int((1 - is_target).sum()), - "n_features": len(feature_cols), - "family_breakdown": fam_breakdown, - "baseline_n_features": len(baseline_idx), - "full_identifications": {m: results[m]["full"] for m in models}, - "baseline_identifications": {m: results[m]["baseline"] for m in models}, - "table": rows, - } - with open(json_path, "w") as fh: - json.dump(summary, fh, indent=2) - - # ---- report --------------------------------------------------------------- - print(f"\n=== Feature-family ablation (fdp={args.fdp}) ===") - for mkind in models: - mrows = [r for r in rows if r["model"] == mkind] - by_delta = sorted(mrows, key=lambda r: r["delta_vs_full"]) # most negative first - print(f"\n[{mkind}] full={results[mkind]['full']} " - f"baseline={results[mkind]['baseline']}") - print(" most useful (largest target drop when removed):") - for r in by_delta[:3]: - print(f" {r['feature_family']:<28} delta_vs_full={r['delta_vs_full']:+d} " - f"rel_gain={r['relative_gain']:+.3f} {r['recommendation']}") - print(" least useful (removal helps or is neutral):") - for r in by_delta[-3:][::-1]: - print(f" {r['feature_family']:<28} delta_vs_full={r['delta_vs_full']:+d} " - f"rel_gain={r['relative_gain']:+.3f} {r['recommendation']}") - print(f"\n[written] {csv_path}") - print(f"[written] {json_path}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/feature_auc.py b/scripts/feature_auc.py deleted file mode 100644 index 1197fbb..0000000 --- a/scripts/feature_auc.py +++ /dev/null @@ -1,31 +0,0 @@ -import sys, re, numpy as np, pandas as pd - -def auc(y, x): - # rank-based AUC (Mann-Whitney), direction-agnostic - x = np.nan_to_num(x.astype(float)) - if x.std() == 0: return 0.5 - order = x.argsort() - ranks = np.empty(len(x)); ranks[order] = np.arange(1, len(x)+1) - # average ties - _, inv, cnt = np.unique(x, return_inverse=True, return_counts=True) - # simpler: use scipy-free tie-aware rank via pandas - ranks = pd.Series(x).rank().values - npos = y.sum(); nneg = len(y)-npos - if npos==0 or nneg==0: return 0.5 - a = (ranks[y==1].sum() - npos*(npos+1)/2) / (npos*nneg) - return max(a, 1-a) - -def strip(pf): return re.sub(r'\[[^\]]*\]','',str(pf)) -dn=pd.read_csv("out_diann/report.tsv",sep="\t",usecols=['Q.Value','Stripped.Sequence']) -diann=set(dn[dn['Q.Value']<=0.01]['Stripped.Sequence'].unique()) -c=pd.read_parquet(sys.argv[1]); c=c[c.label=='target'].copy() -c['human']=c.protein.str.contains("_HUMAN")&~c.protein.str.contains("_ECOLI") -c['strip']=c.peptidoform.map(strip) -c['tp']=(~c.human)&c.strip.isin(diann) -lab=c[c.tp|c.human].copy(); y=lab.tp.astype(int).values -skip={'candidate_id','label','base_peptide_id','peptidoform','protein','apex_rt','precursor_mz','prelim_score','human','strip','tp'} -feats=[x for x in c.columns if x not in skip] -print(f"pos(real ecoli in DIA-NN)={int(y.sum())} neg(human false)={int((y==0).sum())}",flush=True) -rows=[(f, auc(y, pd.to_numeric(lab[f],errors='coerce').values)) for f in feats] -rows.sort(key=lambda z:-z[1]) -for f,a in rows: print(f"AUC {f:24s} {a:.3f}",flush=True) diff --git a/scripts/feature_audit.py b/scripts/feature_audit.py deleted file mode 100644 index 4de2f85..0000000 --- a/scripts/feature_audit.py +++ /dev/null @@ -1,723 +0,0 @@ -#!/usr/bin/env python -"""Feature audit for MuMDIA scored/competed feature tables. - -Implements the sensitivity_plan backlog item P4.2 (spec -``sensitivity_plan/03_feature_evaluation.md`` section 5, evidence ladder -Level 1 to Level 3). The tool is non-invasive: it reads a competed features -Parquet plus the feature registry and writes an audit report. Nothing in the -engine or the input data is modified. - -Evidence ladder covered here: - -Level 1 (data quality), per feature: - missing percentage, NaN/inf percentage, unique-value count, quantiles - (5/25/50/75/95), constant flag, per-group distribution summaries for - targets, decoys and entrapments (mean/median), correlation with - prelim_score, correlation with peptide length and charge, and correlation - with log apex intensity. Flags: constant, asymmetric missingness between - targets and decoys, intensity-dominated, and target-vs-decoy separation - that is much larger than target-vs-entrapment separation (a decoy - construction artifact). - -Level 2 (univariate utility), per feature: - target-vs-decoy separation and target-vs-entrapment separation, both as a - Mann-Whitney U rank AUC and its rank-biserial magnitude. Features that - separate target from decoy strongly but target from entrapment weakly are - flagged LEAKAGE_RISK. - -Level 3 (redundancy): - Spearman correlation matrix over the non-constant features, single-linkage - clustering at ``|rho| >= threshold`` via union-find, with one representative - reported per cluster. - -Registry cross-check: - every audited feature must appear in the registry; features that do not are - reported as UNKNOWN, and per-family coverage is summarised. - -Entrapment framing: this is a target-decoy plus entrapment experiment. The -"real" target group is the target-labelled rows whose protein matches the -real-species substring (default ``_ECOLI``). The entrapment group is the -target-labelled rows whose protein matches the entrapment substring (default -``_HUMAN``); these are foreign-proteome spike-ins that behave as a null. The -decoy group is the decoy-labelled rows. A useful feature separates real -targets from both decoys and entrapments; a feature that separates real -targets from decoys but not from entrapments is likely exploiting the decoy -construction scheme. - -The tool is deterministic: sampling uses a fixed seed, and all iteration -orders are fixed. - -Interpreter: ``C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe``. - -Example: - - python feature_audit.py \ - --features C:/proteobench/out_ecoli/comp.parquet \ - --registry feature_registry.yaml \ - --max-rows 80000 --entrapment-substr _HUMAN -""" - -from __future__ import annotations - -import argparse -import json -import os -import re -import sys -from collections import defaultdict, OrderedDict - -import numpy as np -import pandas as pd -import pyarrow as pa -import pyarrow.parquet as pq -import yaml -from scipy.stats import rankdata - -# --- Meta columns that are never treated as scoring features. -------------- -META_COLUMNS = [ - "candidate_id", - "label", - "base_peptide_id", - "peptidoform", - "protein", - "apex_rt", - "precursor_mz", - "prelim_score", - "charge", - "q_value", - "peptide_q_value", - "pg_q_value", - "global_q_value", - "score", -] - -# --- Flag thresholds (documented, deterministic). -------------------------- -# A separation is "strong" when its rank-biserial magnitude reaches this. -SEP_STRONG = 0.15 -# Leakage: target-vs-entrapment separation below this fraction of the -# target-vs-decoy separation, while the latter is strong. -LEAKAGE_RATIO = 0.5 -# Intensity-dominated: absolute Spearman correlation with log apex intensity. -INTENSITY_CORR = 0.70 -# Asymmetric missingness: target vs decoy missing-rate ratio, with a floor on -# the larger rate so that trivially tiny rates are not flagged. -ASYM_RATIO = 5.0 -ASYM_MIN_RATE = 0.01 -# Redundancy clustering default correlation threshold. -DEFAULT_REDUNDANCY_THRESHOLD = 0.90 -# Default deterministic sampling seed. -DEFAULT_SEED = 1234 - -_BRACKET_RE = re.compile(r"\[[^\]]*\]") - - -# --------------------------------------------------------------------------- -# Registry parsing -# --------------------------------------------------------------------------- -def load_registry(path): - """Load the feature registry YAML. - - Returns a dict name -> {family, level, direction, source_file, ...}. Only - the active (non-collision) rows are usable as columns; rows carrying - ``dropped_collision: true`` (keyed like ``name@family``) are kept so that - coverage accounting is complete but are not expected as columns. - """ - with open(path, "r", encoding="utf-8") as handle: - doc = yaml.safe_load(handle) - features = doc.get("features", {}) or {} - reg = OrderedDict() - for name, meta in features.items(): - meta = meta or {} - reg[str(name)] = { - "family": str(meta.get("family", "?")), - "level": str(meta.get("level", "?")), - "direction": str(meta.get("direction", "?")), - "source_file": str(meta.get("source_file", "")), - "dropped_collision": bool(meta.get("dropped_collision", False)), - } - return reg - - -# --------------------------------------------------------------------------- -# Data loading (bounded memory via deterministic row sampling) -# --------------------------------------------------------------------------- -def load_features(path, max_rows, seed): - """Read the features Parquet, sampling rows deterministically if needed. - - Returns (dataframe, total_rows, sampled_flag). - """ - table = pq.read_table(path) - total_rows = table.num_rows - sampled = False - if max_rows and total_rows > max_rows: - rng = np.random.RandomState(seed) - idx = np.sort(rng.choice(total_rows, size=max_rows, replace=False)) - table = table.take(pa.array(idx)) - sampled = True - df = table.to_pandas() - del table - return df, total_rows, sampled - - -def peptide_length(pepform): - """Stripped peptide length from a ProForma-lite peptidoform. - - Drops a ``DECOY_`` prefix, removes bracketed modifications with the regex - ``\\[[^\\]]*\\]``, and counts uppercase residue letters. - """ - if not isinstance(pepform, str) or not pepform: - return np.nan - seq = pepform - if seq.startswith("DECOY_"): - seq = seq[len("DECOY_"):] - seq = _BRACKET_RE.sub("", seq) - return float(sum(1 for c in seq if "A" <= c <= "Z")) - - -# --------------------------------------------------------------------------- -# Statistics helpers -# --------------------------------------------------------------------------- -def safe_spearman(x, y): - """Spearman correlation over jointly finite pairs; NaN if undefined.""" - mask = np.isfinite(x) & np.isfinite(y) - if mask.sum() < 3: - return np.nan - xr = rankdata(x[mask]) - yr = rankdata(y[mask]) - if xr.std() == 0.0 or yr.std() == 0.0: - return np.nan - return float(np.corrcoef(xr, yr)[0, 1]) - - -def auc_mwu(pos, neg): - """Rank AUC = P(pos > neg) via the Mann-Whitney U statistic. - - Returns NaN when either group is empty. Ties contribute 0.5. - """ - pos = pos[np.isfinite(pos)] - neg = neg[np.isfinite(neg)] - n1 = len(pos) - n2 = len(neg) - if n1 == 0 or n2 == 0: - return np.nan - ranks = rankdata(np.concatenate([pos, neg])) - r1 = ranks[:n1].sum() - u1 = r1 - n1 * (n1 + 1) / 2.0 - return float(u1 / (n1 * n2)) - - -def separation(auc): - """Rank-biserial magnitude in [0, 1] from a rank AUC.""" - if auc is None or not np.isfinite(auc): - return np.nan - return abs(2.0 * auc - 1.0) - - -# --------------------------------------------------------------------------- -# Group masks -# --------------------------------------------------------------------------- -def substr_match(series, substrings): - """Boolean mask: True where any of the substrings occurs in the string.""" - result = np.zeros(len(series), dtype=bool) - for sub in substrings: - if sub: - result |= series.str.contains(re.escape(sub), regex=True).to_numpy() - return result - - -# --------------------------------------------------------------------------- -# Redundancy clustering -# --------------------------------------------------------------------------- -class UnionFind: - def __init__(self, items): - self.parent = {it: it for it in items} - - def find(self, a): - root = a - while self.parent[root] != root: - root = self.parent[root] - # Path compression. - while self.parent[a] != root: - self.parent[a], a = root, self.parent[a] - return root - - def union(self, a, b): - ra, rb = self.find(a), self.find(b) - if ra != rb: - # Deterministic: smaller name becomes the root. - if rb < ra: - ra, rb = rb, ra - self.parent[rb] = ra - - -def cluster_features(df, feature_cols, threshold): - """Cluster features by single-linkage Spearman ``|rho| >= threshold``. - - Returns (clusters, corr_dataframe) where clusters is a dict - root -> sorted member list, including singletons. - """ - if not feature_cols: - return {}, None - sub = df[feature_cols].replace([np.inf, -np.inf], np.nan) - corr = sub.corr(method="spearman") - cols = list(corr.columns) - values = corr.to_numpy() - uf = UnionFind(cols) - n = len(cols) - for i in range(n): - row = values[i] - for j in range(i + 1, n): - r = row[j] - if np.isfinite(r) and abs(r) >= threshold: - uf.union(cols[i], cols[j]) - groups = defaultdict(list) - for col in cols: - groups[uf.find(col)].append(col) - for root in groups: - groups[root].sort() - return dict(groups), corr - - -# --------------------------------------------------------------------------- -# Main audit -# --------------------------------------------------------------------------- -def run_audit(args): - registry = load_registry(args.registry) - df, total_rows, sampled = load_features(args.features, args.max_rows, args.seed) - n_used = len(df) - - if "label" not in df.columns: - raise SystemExit("features table has no 'label' column") - - # Reference vectors for Level 1 correlations. - label = df["label"].astype(str) - protein = df["protein"].astype(str) if "protein" in df.columns else pd.Series([""] * n_used) - prelim = df["prelim_score"].to_numpy(dtype=float) if "prelim_score" in df.columns else np.full(n_used, np.nan) - charge_ref = df["charge"].to_numpy(dtype=float) if "charge" in df.columns else np.full(n_used, np.nan) - pep_len = np.array([peptide_length(p) for p in df["peptidoform"]], dtype=float) if "peptidoform" in df.columns else np.full(n_used, np.nan) - has_intensity_col = "log_apex_intensity" in df.columns - log_int = df["log_apex_intensity"].to_numpy(dtype=float) if has_intensity_col else np.full(n_used, np.nan) - - real_subs = [s.strip() for s in args.real_substr.split(",") if s.strip()] - entrap_subs = [s.strip() for s in args.entrapment_substr.split(",") if s.strip()] - - is_target = (label == "target").to_numpy() - is_decoy = (label == "decoy").to_numpy() - real_mask_prot = substr_match(protein, real_subs) - entrap_mask_prot = substr_match(protein, entrap_subs) - - mask_target = is_target & real_mask_prot - mask_decoy = is_decoy - mask_entrap = is_target & entrap_mask_prot - - n_target = int(mask_target.sum()) - n_decoy = int(mask_decoy.sum()) - n_entrap = int(mask_entrap.sum()) - - # Feature columns: numeric, not meta. - meta_set = set(META_COLUMNS) - feature_cols = [] - for col in df.columns: - if col in meta_set: - continue - if pd.api.types.is_numeric_dtype(df[col]): - feature_cols.append(col) - feature_cols.sort() - - rows = [] - unknown_features = [] - for col in feature_cols: - arr = df[col].to_numpy(dtype=float) - finite = np.isfinite(arr) - n_nan = int(np.isnan(arr).sum()) - n_inf = int(np.isinf(arr).sum()) - finite_vals = arr[finite] - uniq = int(np.unique(finite_vals).size) if finite_vals.size else 0 - constant = uniq <= 1 - - if finite_vals.size: - q05, q25, q50, q75, q95 = ( - float(np.percentile(finite_vals, p)) for p in (5, 25, 50, 75, 95) - ) - else: - q05 = q25 = q50 = q75 = q95 = np.nan - - def grp_stats(mask): - vals = arr[mask & finite] - if vals.size == 0: - return np.nan, np.nan - return float(vals.mean()), float(np.median(vals)) - - mean_t, med_t = grp_stats(mask_target) - mean_d, med_d = grp_stats(mask_decoy) - mean_e, med_e = grp_stats(mask_entrap) - - corr_prelim = safe_spearman(arr, prelim) - corr_len = safe_spearman(arr, pep_len) - corr_charge = safe_spearman(arr, charge_ref) - corr_int = safe_spearman(arr, log_int) if has_intensity_col else np.nan - - # Missingness by label group for the asymmetry flag. - n_t_rows = int(is_target.sum()) - n_d_rows = int(is_decoy.sum()) - miss_rate_t = float(np.isnan(arr[is_target]).mean()) if n_t_rows else np.nan - miss_rate_d = float(np.isnan(arr[is_decoy]).mean()) if n_d_rows else np.nan - if np.isfinite(miss_rate_t) and np.isfinite(miss_rate_d): - hi = max(miss_rate_t, miss_rate_d) - lo = min(miss_rate_t, miss_rate_d) - asym_ratio = (hi + 1e-12) / (lo + 1e-12) - else: - hi = np.nan - asym_ratio = np.nan - - auc_td = auc_mwu(arr[mask_target], arr[mask_decoy]) - auc_te = auc_mwu(arr[mask_target], arr[mask_entrap]) - sep_td = separation(auc_td) - sep_te = separation(auc_te) - - # Flags. - flag_constant = constant - flag_asym = bool( - np.isfinite(asym_ratio) - and hi > ASYM_MIN_RATE - and asym_ratio > ASYM_RATIO - ) - flag_intensity = bool( - has_intensity_col - and col != "log_apex_intensity" - and np.isfinite(corr_int) - and abs(corr_int) >= INTENSITY_CORR - ) - flag_leakage = bool( - np.isfinite(sep_td) - and np.isfinite(sep_te) - and sep_td >= SEP_STRONG - and sep_te < LEAKAGE_RATIO * sep_td - ) - - reg = registry.get(col) - in_registry = reg is not None - if not in_registry: - unknown_features.append(col) - family = reg["family"] if in_registry else "UNKNOWN" - level = reg["level"] if in_registry else "?" - direction = reg["direction"] if in_registry else "?" - - rows.append( - { - "feature": col, - "family": family, - "level": level, - "direction": direction, - "in_registry": in_registry, - "n_rows_used": n_used, - "missing_pct": 100.0 * n_nan / n_used if n_used else np.nan, - "inf_pct": 100.0 * n_inf / n_used if n_used else np.nan, - "unique_count": uniq, - "constant": constant, - "q05": q05, - "q25": q25, - "q50": q50, - "q75": q75, - "q95": q95, - "n_target": n_target, - "mean_target": mean_t, - "median_target": med_t, - "n_decoy": n_decoy, - "mean_decoy": mean_d, - "median_decoy": med_d, - "n_entrap": n_entrap, - "mean_entrap": mean_e, - "median_entrap": med_e, - "corr_prelim_score": corr_prelim, - "corr_peptide_length": corr_len, - "corr_charge": corr_charge, - "corr_log_apex_intensity": corr_int, - "missing_rate_target": miss_rate_t, - "missing_rate_decoy": miss_rate_d, - "asym_missing_ratio": asym_ratio, - "auc_target_decoy": auc_td, - "sep_target_decoy": sep_td, - "auc_target_entrap": auc_te, - "sep_target_entrap": sep_te, - "sep_gap_decoy_minus_entrap": (sep_td - sep_te) if (np.isfinite(sep_td) and np.isfinite(sep_te)) else np.nan, - "flag_constant": flag_constant, - "flag_asymmetric_missing": flag_asym, - "flag_intensity_dominated": flag_intensity, - "flag_leakage": flag_leakage, - } - ) - - audit = pd.DataFrame(rows) - - # Level 3 redundancy over non-constant features. - nonconst = [r["feature"] for r in rows if not r["constant"]] - clusters, _corr = cluster_features(df, nonconst, args.redundancy_threshold) - - # Representative per cluster: highest target-vs-entrapment separation, - # then highest target-vs-decoy separation, then name. - sep_te_map = {r["feature"]: (r["sep_target_entrap"] if np.isfinite(r["sep_target_entrap"]) else -1.0) for r in rows} - sep_td_map = {r["feature"]: (r["sep_target_decoy"] if np.isfinite(r["sep_target_decoy"]) else -1.0) for r in rows} - - cluster_id_map = {} - representative_map = {} - multi_clusters = [] - cluster_counter = 0 - # Deterministic order: sort clusters by their sorted member list. - for root, members in sorted(clusters.items(), key=lambda kv: kv[1]): - if len(members) < 2: - continue - rep = sorted( - members, - key=lambda f: (-sep_te_map[f], -sep_td_map[f], f), - )[0] - cid = cluster_counter - cluster_counter += 1 - for m in members: - cluster_id_map[m] = cid - representative_map[m] = (m == rep) - multi_clusters.append( - { - "cluster_id": cid, - "size": len(members), - "representative": rep, - "representative_sep_target_entrap": (None if sep_te_map[rep] < 0 else round(sep_te_map[rep], 6)), - "members": members, - } - ) - - n_singletons = sum(1 for m in clusters.values() if len(m) == 1) - - audit["cluster_id"] = audit["feature"].map(lambda f: cluster_id_map.get(f, -1)) - audit["is_cluster_representative"] = audit["feature"].map(lambda f: representative_map.get(f, False)) - - return { - "audit": audit, - "rows": rows, - "clusters": multi_clusters, - "n_singletons": n_singletons, - "unknown_features": unknown_features, - "registry": registry, - "feature_cols": feature_cols, - "total_rows": total_rows, - "n_used": n_used, - "sampled": sampled, - "n_target": n_target, - "n_decoy": n_decoy, - "n_entrap": n_entrap, - "real_subs": real_subs, - "entrap_subs": entrap_subs, - "has_intensity_col": has_intensity_col, - } - - -# --------------------------------------------------------------------------- -# Output writers -# --------------------------------------------------------------------------- -def write_outputs(result, args, out_dir): - os.makedirs(out_dir, exist_ok=True) - audit = result["audit"] - rows = result["rows"] - registry = result["registry"] - - csv_path = os.path.join(out_dir, "feature_audit.csv") - audit.to_csv(csv_path, index=False, float_format="%.6g") - - clusters_path = os.path.join(out_dir, "redundancy_clusters.json") - with open(clusters_path, "w", encoding="utf-8") as handle: - json.dump( - { - "spearman_abs_threshold": args.redundancy_threshold, - "n_features_considered": int((~audit["constant"]).sum()), - "n_clusters": len(result["clusters"]), - "n_singletons": result["n_singletons"], - "clusters": result["clusters"], - }, - handle, - indent=2, - ) - - # Warnings file. - by_feature = {r["feature"]: r for r in rows} - constant_feats = [r["feature"] for r in rows if r["flag_constant"]] - asym_feats = [r["feature"] for r in rows if r["flag_asymmetric_missing"]] - intensity_feats = [r["feature"] for r in rows if r["flag_intensity_dominated"]] - leakage_feats = [r["feature"] for r in rows if r["flag_leakage"]] - unknown = result["unknown_features"] - - # Registry features absent from the table (active rows only). - table_names = set(result["feature_cols"]) - missing_from_table = [ - name - for name, meta in registry.items() - if not meta["dropped_collision"] and name not in table_names and name not in set(META_COLUMNS) - ] - - warnings_path = os.path.join(out_dir, "warnings.txt") - with open(warnings_path, "w", encoding="utf-8") as handle: - handle.write("MuMDIA feature audit warnings\n") - handle.write("=" * 60 + "\n\n") - - handle.write("CONSTANT features (%d)\n" % len(constant_feats)) - for f in constant_feats: - handle.write(" %s\n" % f) - handle.write("\n") - - handle.write("ASYMMETRIC MISSINGNESS target vs decoy > %gx (%d)\n" % (ASYM_RATIO, len(asym_feats))) - for f in asym_feats: - r = by_feature[f] - handle.write( - " %s target=%.4f decoy=%.4f ratio=%.1f\n" - % (f, r["missing_rate_target"], r["missing_rate_decoy"], r["asym_missing_ratio"]) - ) - handle.write("\n") - - handle.write("INTENSITY-DOMINATED |corr(log_apex_intensity)| >= %g (%d)\n" % (INTENSITY_CORR, len(intensity_feats))) - for f in intensity_feats: - handle.write(" %s corr=%.3f\n" % (f, by_feature[f]["corr_log_apex_intensity"])) - handle.write("\n") - - handle.write( - "LEAKAGE RISK sep(target,decoy) >= %g and sep(target,entrapment) < %g x sep(target,decoy) (%d)\n" - % (SEP_STRONG, LEAKAGE_RATIO, len(leakage_feats)) - ) - for f in leakage_feats: - r = by_feature[f] - handle.write( - " %s sep_td=%.3f sep_te=%.3f gap=%.3f\n" - % (f, r["sep_target_decoy"], r["sep_target_entrap"], r["sep_gap_decoy_minus_entrap"]) - ) - handle.write("\n") - - handle.write("UNKNOWN features not in registry (%d)\n" % len(unknown)) - for f in unknown: - handle.write(" %s\n" % f) - handle.write("\n") - - handle.write("REGISTRY features absent from the table (%d)\n" % len(missing_from_table)) - for f in missing_from_table: - handle.write(" %s\n" % f) - handle.write("\n") - - # Summary file. - summary_path = os.path.join(out_dir, "summary.txt") - n_feat = len(result["feature_cols"]) - with open(summary_path, "w", encoding="utf-8") as handle: - handle.write("MuMDIA feature audit summary\n") - handle.write("=" * 60 + "\n\n") - handle.write("Input features : %s\n" % os.path.abspath(args.features)) - handle.write("Registry : %s\n" % os.path.abspath(args.registry)) - handle.write("Output dir : %s\n" % os.path.abspath(out_dir)) - handle.write("\n") - if result["sampled"]: - handle.write( - "Sampling : %d of %d rows (seed=%d, deterministic)\n" - % (result["n_used"], result["total_rows"], args.seed) - ) - else: - handle.write("Sampling : none, all %d rows used\n" % result["total_rows"]) - handle.write("\n") - handle.write("Groups\n") - handle.write(" real targets (%s) : %d\n" % (",".join(result["real_subs"]), result["n_target"])) - handle.write(" decoys : %d\n" % result["n_decoy"]) - handle.write(" entrapments (%s) : %d\n" % (",".join(result["entrap_subs"]), result["n_entrap"])) - handle.write(" note: target-labelled rows matching neither the real nor the entrapment\n") - handle.write(" substring fall outside both the target and entrapment groups.\n") - if not result["has_intensity_col"]: - handle.write(" note: no log_apex_intensity column found; intensity checks skipped.\n") - handle.write(" note: per-run drift not computed (no run column in this input).\n") - handle.write("\n") - handle.write("Features audited : %d\n" % n_feat) - handle.write("Constant : %d\n" % len(constant_feats)) - handle.write("Asymmetric miss. : %d\n" % len(asym_feats)) - handle.write("Intensity-domin. : %d\n" % len(intensity_feats)) - handle.write("Leakage risk : %d\n" % len(leakage_feats)) - handle.write("Unknown (registry): %d\n" % len(unknown)) - handle.write("Redundancy clusters (size>=2): %d\n" % len(result["clusters"])) - handle.write("Redundancy singletons : %d\n" % result["n_singletons"]) - handle.write("\n") - - # Family coverage over audited features. - fam_counts = defaultdict(int) - for r in rows: - fam_counts[r["family"]] += 1 - handle.write("Family coverage (audited features)\n") - for fam in sorted(fam_counts): - handle.write(" %-32s %d\n" % (fam, fam_counts[fam])) - handle.write("\n") - - # Top separations. - def top_by(key, n=10): - valid = [r for r in rows if np.isfinite(r[key])] - return sorted(valid, key=lambda r: (-r[key], r["feature"]))[:n] - - handle.write("Top 10 target-vs-entrapment separation\n") - for r in top_by("sep_target_entrap"): - handle.write( - " %-40s sep_te=%.3f sep_td=%.3f family=%s\n" - % (r["feature"], r["sep_target_entrap"], r["sep_target_decoy"], r["family"]) - ) - handle.write("\n") - handle.write("Top 10 target-vs-decoy separation\n") - for r in top_by("sep_target_decoy"): - handle.write( - " %-40s sep_td=%.3f sep_te=%.3f family=%s\n" - % (r["feature"], r["sep_target_decoy"], r["sep_target_entrap"], r["family"]) - ) - handle.write("\n") - - return { - "csv_path": csv_path, - "clusters_path": clusters_path, - "warnings_path": warnings_path, - "summary_path": summary_path, - "constant_feats": constant_feats, - "leakage_feats": leakage_feats, - } - - -def parse_args(argv=None): - parser = argparse.ArgumentParser( - description="Audit a MuMDIA competed features Parquet against the feature registry." - ) - parser.add_argument("--features", required=True, help="competed features Parquet") - parser.add_argument("--registry", required=True, help="feature_registry.yaml") - parser.add_argument("--out", default=None, help="output directory (default /feature_audit/)") - parser.add_argument("--max-rows", type=int, default=None, help="deterministic row sample cap") - parser.add_argument("--entrapment-substr", default="_HUMAN", help="protein substring(s) for the entrapment null (comma-separated)") - parser.add_argument("--real-substr", default="_ECOLI", help="protein substring(s) for the real target species (comma-separated)") - parser.add_argument("--redundancy-threshold", type=float, default=DEFAULT_REDUNDANCY_THRESHOLD, help="abs Spearman rho for clustering") - parser.add_argument("--seed", type=int, default=DEFAULT_SEED, help="sampling seed") - return parser.parse_args(argv) - - -def main(argv=None): - args = parse_args(argv) - if args.out: - out_dir = args.out - else: - out_dir = os.path.join(os.path.dirname(os.path.abspath(args.features)), "feature_audit") - - result = run_audit(args) - paths = write_outputs(result, args, out_dir) - - rows = result["rows"] - top_te = sorted( - [r for r in rows if np.isfinite(r["sep_target_entrap"])], - key=lambda r: (-r["sep_target_entrap"], r["feature"]), - )[:3] - - print("feature_audit written to: %s" % os.path.abspath(out_dir)) - print("features audited : %d" % len(result["feature_cols"])) - print("constant : %d" % len(paths["constant_feats"])) - print("leakage-flagged : %d" % len(paths["leakage_feats"])) - print("redundancy clusters : %d (size>=2)" % len(result["clusters"])) - if result["sampled"]: - print("sampling : %d of %d rows (seed=%d)" % (result["n_used"], result["total_rows"], args.seed)) - print("top-3 target-vs-entrapment separation:") - for r in top_te: - print(" %-40s sep_te=%.3f (sep_td=%.3f)" % (r["feature"], r["sep_target_entrap"], r["sep_target_decoy"])) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/ft_calibrate_eval.py b/scripts/ft_calibrate_eval.py deleted file mode 100644 index 45702b3..0000000 --- a/scripts/ft_calibrate_eval.py +++ /dev/null @@ -1,124 +0,0 @@ -"""DeepLC CALIBRATION (no fine-tuning) vs fine-tune vs DIA-NN, on a common random -held-out split. Calibration fits a calibration curve + selects the best internal MT -model on the reference set WITHOUT changing weights, so it cannot overfit like -fine-tuning. Reports held-out MAD for: - (a) DeepLC native predict_and_calibrate (calibrated seconds, its own curve), - (b) DeepLC base predict + binned-median calibration (isolates base-model quality, - same calibrator used for DIA-NN), - (c) DIA-NN raw library iRT + binned-median calibration (baseline). - -Usage: python ft_calibrate_eval.py --q-train Q --held-frac F --seed S --raw-lib LIB -""" -import os -_T = os.environ.get("DEEPLC_FT_THREADS", "8") -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" -for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "NUMEXPR_NUM_THREADS"): - os.environ[k] = "1" - -import argparse -import re -import deeplc -import numpy as np -import pyarrow.parquet as pq -import torch -from psm_utils import PSM, PSMList - -STD = set("ACDEFGHIKLMNPQRSTVWY") -is_std = lambda pf: all(c in STD for c in re.sub(r"\[[^\]]*\]", "", pf)) - - -def calibrate_fit(x, y, nbins=80): - o = np.argsort(x); xs, ys = x[o], y[o] - e = np.linspace(0, len(xs), nbins + 1).astype(int) - cx, cy = [], [] - for b in range(nbins): - lo, hi = e[b], e[b + 1] - if hi > lo: - cx.append(np.median(xs[lo:hi])); cy.append(np.median(ys[lo:hi])) - cx, cy = np.array(cx), np.array(cy) - return lambda q: np.interp(q, cx, cy) - - -def mad(pred, obs): - r = np.asarray(pred, float) - np.asarray(obs, float) - return np.median(np.abs(r - np.median(r))) - - -def flat(a): - a = np.asarray(a, float) - return a.mean(axis=1) if a.ndim == 2 else a - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("seed_path") - ap.add_argument("--q-train", dest="q_train", type=float, default=0.01) - ap.add_argument("--held-frac", dest="held_frac", type=float, default=0.15) - ap.add_argument("--seed", type=int, default=0) - ap.add_argument("--threads", type=int, default=int(_T)) - ap.add_argument("--raw-lib", dest="raw_lib", default=None) - a = ap.parse_args() - torch.set_num_threads(max(1, a.threads)) - try: - torch.set_num_interop_threads(1) - except RuntimeError: - pass - - s = pq.read_table(a.seed_path).to_pydict() - have_irt = "predicted_irt" in s - best = {} - for i in range(len(s["peptidoform"])): - if s["label"][i] != "target": - continue - pf = s["peptidoform"][i] - if not is_std(pf): - continue - b = s["base_peptide_id"][i] - if b not in best or s["score"][i] > best[b][0]: - irt = float(s["predicted_irt"][i]) if have_irt else float("nan") - best[b] = (s["score"][i], pf, s["observed_rt"][i], s["spectrum_q"][i], irt) - conf = [(pf, rt, q) for (_, pf, rt, q, _) in best.values() if q < 0.01] - # raw iRT per peptidoform straight from the seed (the library iRT that was used), - # so the DIA-NN baseline needs no external lib file. - seed_irt = {pf: irt for (_, pf, rt, q, irt) in best.values()} - rng = np.random.default_rng(a.seed) - mask = rng.random(len(conf)) < a.held_frac - held = [(pf, rt) for i, (pf, rt, q) in enumerate(conf) if mask[i]] - train = [(pf, rt) for i, (pf, rt, q) in enumerate(conf) if not mask[i] and q < a.q_train] - print(f"q_train={a.q_train} train={len(train)} held={len(held)}", flush=True) - - ref = PSMList(psm_list=[PSM(peptidoform=pf, retention_time=rt, spectrum_id=str(k)) - for k, (pf, rt) in enumerate(train)]) - tr_pf = [pf for pf, _ in train]; tr_obs = np.array([rt for _, rt in train], float) - hd_pf = [pf for pf, _ in held]; hd_obs = np.array([rt for _, rt in held], float) - - # (a) DeepLC native predict_and_calibrate (calibrated seconds) - tr_cal = flat(deeplc.predict_and_calibrate(tr_pf, psm_list_reference=ref)) - hd_cal = flat(deeplc.predict_and_calibrate(hd_pf, psm_list_reference=ref)) - print(f"CAL q_train={a.q_train} train_MAD={mad(tr_cal, tr_obs):.3f} held_MAD={mad(hd_cal, hd_obs):.3f}", flush=True) - - # (b) DeepLC base predict + binned-median calibration - tr_base = flat(deeplc.predict(tr_pf)); hd_base = flat(deeplc.predict(hd_pf)) - cb = calibrate_fit(tr_base, tr_obs) - print(f"BASE q_train={a.q_train} train_MAD={mad(cb(tr_base), tr_obs):.3f} held_MAD={mad(cb(hd_base), hd_obs):.3f}", flush=True) - - # (c) DIA-NN raw. Prefer an external lib; else use the seed's own predicted_irt. - rm = None - if a.raw_lib: - rl = pq.read_table(a.raw_lib, columns=["peptidoform", "predicted_irt"]).to_pydict() - rm = {} - for pf, irt in zip(rl["peptidoform"], rl["predicted_irt"]): - rm.setdefault(pf, float(irt)) - elif have_irt: - rm = {pf: irt for pf, irt in seed_irt.items() if np.isfinite(irt)} - if rm: - rtr = np.array([rm[pf] for pf in tr_pf if pf in rm], float) - rtro = np.array([rt for pf, rt in train if pf in rm], float) - rhd = np.array([rm[pf] for pf in hd_pf if pf in rm], float) - rhdo = np.array([rt for pf, rt in held if pf in rm], float) - cr = calibrate_fit(rtr, rtro) - print(f"DIANN q_train={a.q_train} train_MAD={mad(cr(rtr), rtro):.3f} held_MAD={mad(cr(rhd), rhdo):.3f}", flush=True) - - -if __name__ == "__main__": - main() diff --git a/scripts/ft_epoch_eval.py b/scripts/ft_epoch_eval.py deleted file mode 100644 index e71b4eb..0000000 --- a/scripts/ft_epoch_eval.py +++ /dev/null @@ -1,139 +0,0 @@ -"""Fast DeepLC fine-tune epoch sweep for generalization. Fine-tune on the q detects overfitting. Skips the full -3.74M-peptide library prediction so each epoch setting costs only its training time. - -Usage: python ft_epoch_eval.py [--q-train Q] [--patience P] -""" -import os -_T = os.environ.get("DEEPLC_FT_THREADS", "8") -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" -for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "NUMEXPR_NUM_THREADS"): - os.environ[k] = "1" - -import argparse -import re -import deeplc -import numpy as np -import pyarrow.parquet as pq -import torch -from psm_utils import PSM, PSMList - -STD = set("ACDEFGHIKLMNPQRSTVWY") -strip_mods = lambda s: re.sub(r"\[[^\]]*\]", "", s) -is_std = lambda pf: all(c in STD for c in strip_mods(pf)) - - -def calibrate_fit(x, y, nbins=80): - """Robust monotone calibration curve (quantile-binned medians), numpy-only - stand-in for LOESS. Returns a predictor callable via np.interp.""" - o = np.argsort(x) - xs, ys = x[o], y[o] - edges = np.linspace(0, len(xs), nbins + 1).astype(int) - cx, cy = [], [] - for b in range(nbins): - lo, hi = edges[b], edges[b + 1] - if hi > lo: - cx.append(np.median(xs[lo:hi])) - cy.append(np.median(ys[lo:hi])) - cx, cy = np.array(cx), np.array(cy) - return lambda q: np.interp(q, cx, cy) - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("seed_path") - ap.add_argument("epochs", type=int) - ap.add_argument("--q-train", dest="q_train", type=float, default=0.001) - ap.add_argument("--patience", type=int, default=10) - ap.add_argument("--batch", type=int, default=512, - help="fine-tune batch size; small sets need a small batch so each " - "epoch has enough gradient steps to converge") - ap.add_argument("--threads", type=int, default=int(_T)) - ap.add_argument("--held-frac", dest="held_frac", type=float, default=0.0, - help="if >0, hold out this random fraction of the confident (q<0.01) " - "set (excluded from training regardless of q_train); the rest, " - "subject to q_train, is the train set. Enables fair comparison " - "of q_train choices on a common held-out set. 0 = legacy q-band held-out.") - ap.add_argument("--seed", type=int, default=0, help="RNG seed for the random held-out split") - ap.add_argument("--raw-lib", dest="raw_lib", default=None, - help="optional library parquet; also report the raw (un-fine-tuned) " - "iRT held-out MAD on the SAME split, as the baseline to beat") - a = ap.parse_args() - torch.set_num_threads(max(1, a.threads)) - try: - torch.set_num_interop_threads(1) - except RuntimeError: - pass - - s = pq.read_table(a.seed_path).to_pydict() - # best-scoring target PSM per base_peptide_id - best = {} - for i in range(len(s["peptidoform"])): - if s["label"][i] != "target": - continue - pf = s["peptidoform"][i] - if not is_std(pf): - continue - b = s["base_peptide_id"][i] - if b not in best or s["score"][i] > best[b][0]: - best[b] = (s["score"][i], pf, s["observed_rt"][i], s["spectrum_q"][i]) - rows = list(best.values()) - conf = [(pf, rt, q) for (_, pf, rt, q) in rows if q < 0.01] # confident universe - if a.held_frac > 0: - # Random split of the confident set: `held` is fixed across q_train choices so - # different train-set sizes are compared on the SAME held-out peptides. - rng = np.random.default_rng(a.seed) - mask = rng.random(len(conf)) < a.held_frac - held = [(pf, rt) for i, (pf, rt, q) in enumerate(conf) if mask[i]] - train = [(pf, rt) for i, (pf, rt, q) in enumerate(conf) if not mask[i] and q < a.q_train] - else: - train = [(pf, rt) for (pf, rt, q) in conf if q < a.q_train] - held = [(pf, rt) for (pf, rt, q) in conf if a.q_train <= q < 0.01] - print(f"epochs={a.epochs} q_train={a.q_train} held_frac={a.held_frac} " - f"train={len(train)} held={len(held)}", flush=True) - - ref = PSMList(psm_list=[PSM(peptidoform=pf, retention_time=rt, spectrum_id=str(k)) - for k, (pf, rt) in enumerate(train)]) - tk = {"num_workers": 0, "epochs": a.epochs, "batch_size": a.batch, - "patience": a.patience, "device": "cpu", "num_threads": max(1, a.threads)} - model = deeplc.finetune(ref, train_kwargs=tk) - - def predict(items): - pf = [p for p, _ in items] - pr = deeplc.predict(pf, model=model) - pr = np.asarray(pr, float) - if pr.ndim == 2: - pr = pr.mean(axis=1) - return pr, np.array([rt for _, rt in items], float) - - tr_irt, tr_obs = predict(train) - hd_irt, hd_obs = predict(held) - cal = calibrate_fit(tr_irt, tr_obs) - - def mad(pred, obs): - r = pred - obs - return np.median(np.abs(r - np.median(r))) - - tr_mad = mad(cal(tr_irt), tr_obs) - hd_mad = mad(cal(hd_irt), hd_obs) - print(f"RESULT q_train={a.q_train} epochs={a.epochs} train_MAD={tr_mad:.3f} held_MAD={hd_mad:.3f}", flush=True) - - # Raw (un-fine-tuned) baseline on the identical train/held split. - if a.raw_lib: - rl = pq.read_table(a.raw_lib, columns=["peptidoform", "predicted_irt"]).to_pydict() - rawmap = {} - for pf, irt in zip(rl["peptidoform"], rl["predicted_irt"]): - rawmap.setdefault(pf, float(irt)) - rtr = np.array([rawmap[pf] for pf, _ in train if pf in rawmap], float) - rtr_o = np.array([rt for pf, rt in train if pf in rawmap], float) - rhd = np.array([rawmap[pf] for pf, _ in held if pf in rawmap], float) - rhd_o = np.array([rt for pf, rt in held if pf in rawmap], float) - rcal = calibrate_fit(rtr, rtr_o) - print(f"RAW q_train={a.q_train} train_MAD={mad(rcal(rtr), rtr_o):.3f} " - f"held_MAD={mad(rcal(rhd), rhd_o):.3f}", flush=True) - - -if __name__ == "__main__": - main() diff --git a/scripts/import_diann_decoys.py b/scripts/import_diann_decoys.py deleted file mode 100644 index d789107..0000000 --- a/scripts/import_diann_decoys.py +++ /dev/null @@ -1,72 +0,0 @@ -"""Import a DIA-NN predicted decoy library (fragment-level parquet from ---gen-spec-lib --predictor) into the MuMDIA target-library schema. Emits clean -peptidoforms (no DECOY_ prefix yet) with DIA-NN-predicted fragments + iRT, ready for -per-run iRT fine-tuning; combine_mutdec.py then labels them decoy and prefixes. - -Usage: python import_diann_decoys.py -""" -import sys -import numpy as np -import pandas as pd - -BASE_OFF = 100_000_000 # keep decoy ids from colliding with target ids - - -def to_proforma(modseq): - return (str(modseq).replace("(UniMod:4)", "[Carbamidomethyl]") - .replace("(UniMod:35)", "[Oxidation]")) - - -def main(): - inp, outp, outf = sys.argv[1:4] - df = pd.read_parquet(inp) - # keep b/y no-loss fragments - lt = "Fragment.Loss.Type" - if lt in df.columns: - df = df[df[lt].astype(str).str.lower().isin(["noloss", "", "none", "nan"])] - df = df[df["Fragment.Type"].astype(str).str.lower().isin(["b", "y"])] - # drop precursors with unmapped mods - df = df[~df["Modified.Sequence"].astype(str).str.contains(r"\(UniMod:(?!4\)|35\))", regex=True)] - df["peptidoform"] = df["Modified.Sequence"].map(to_proforma) - df["key"] = df["peptidoform"] + "/" + df["Precursor.Charge"].astype(str) - - keys = df.drop_duplicates("key").reset_index(drop=True) - keys["candidate_id"] = np.arange(len(keys), dtype=np.uint32) - key2cand = dict(zip(keys["key"], keys["candidate_id"])) - # base peptide id per stripped sequence - keys["base_peptide_id"] = (pd.factorize(keys["Stripped.Sequence"])[0] + BASE_OFF).astype(np.uint32) - df["candidate_id"] = df["key"].map(key2cand).astype(np.uint32) - - nfrag = df.groupby("candidate_id").size() - prec = pd.DataFrame({ - "candidate_id": keys["candidate_id"], - "peptidoform_id": (keys["candidate_id"].astype(np.int64) + BASE_OFF).astype(np.uint32), - "base_peptide_id": keys["base_peptide_id"], - "peptidoform": keys["peptidoform"], - "charge": keys["Precursor.Charge"].astype(np.int32), - "precursor_mz": keys["Precursor.Mz"].astype(np.float64), - "predicted_irt": keys["RT"].astype(np.float32), - "label": "target", # placeholder; combine_mutdec relabels to decoy - "protein": "DECOY", - "n_fragments": keys["candidate_id"].map(nfrag).fillna(0).astype(np.int32), - }) - name = df["Fragment.Type"].astype(str) + df["Fragment.Series.Number"].astype(str) - fc = df["Fragment.Charge"].astype(np.int32) - name = np.where(fc > 1, name + "^" + fc.astype(str), name) - frag = pd.DataFrame({ - "candidate_id": df["candidate_id"], - "mz": df["Product.Mz"].astype(np.float64), - "predicted_intensity": df["Relative.Intensity"].astype(np.float32), - "name": name, - "ion_type": df["Fragment.Type"].astype(str).str.lower(), - "ordinal": df["Fragment.Series.Number"].astype(np.int32), - "frag_charge": fc, - }).sort_values("candidate_id").reset_index(drop=True) - - prec.to_parquet(outp, index=False) - frag.to_parquet(outf, index=False) - print(f"decoy precursors {len(prec)}, fragments {len(frag)} -> {outp}, {outf}") - - -if __name__ == "__main__": - main() diff --git a/scripts/localization.py b/scripts/localization.py deleted file mode 100644 index 81bf961..0000000 --- a/scripts/localization.py +++ /dev/null @@ -1,285 +0,0 @@ -"""Modification-localization competition features (first pass). - -Implements sensitivity_plan backlog item P6.2 as a non-invasive pass over -existing MuMDIA artifacts. Peptidoforms that share a stripped sequence, a -modification multiset, and a charge but place the modifications on different -sites are localization variants. When such variants co-elute they are a -localization-ambiguity group, and site-determining ions (fragments whose m/z is -unique to one variant) distinguish them. - -The engine is not modified. This is a first-pass ("fix later") implementation: -correctness, bounded memory, and a working smoke test are prioritised over -completeness. Computation is deterministic (stable sorts, sorted iteration; no -RNG). The large chromatogram list columns are read only for the small subset of -candidates that fall in an ambiguity group, which keeps memory bounded. - -Per candidate it writes to `localization.parquet`: - is_localization_ambiguous, n_localization_variants, - site_determining_ion_count, site_determining_ion_intensity, - localization_confidence. - -House style: literal prose, no em-dashes, no hardcoded secrets. -""" - -import argparse -import re -import sys -import time -from collections import defaultdict - -import numpy as np -import pyarrow as pa -import pyarrow.parquet as pq - -MOD_RE = re.compile(r"\[[^\]]*\]") -SITE_TOL_PPM = 20.0 # tolerance for calling a fragment m/z "shared" with a sibling - - -def log(msg): - print(msg, file=sys.stderr, flush=True) - - -def parse_peptidoform(pep): - """Parse a ProForma-lite peptidoform into (is_decoy, stripped_sequence, - modification_multiset, variant_signature). - - The multiset is the sorted tuple of modification tokens ignoring position. - The signature is the sorted tuple of (residue_index, token) pairs, which - distinguishes localization variants that share a multiset.""" - is_decoy = pep.startswith("DECOY_") - core = pep[6:] if is_decoy else pep - stripped = [] - mods = [] # (residue_index, token) - i = 0 - res_idx = -1 # index of the most recent residue; -1 == N-terminal - n = len(core) - while i < n: - ch = core[i] - if ch == "[": - j = core.find("]", i) - if j == -1: - # Malformed; treat the rest as sequence and stop mod parsing. - stripped.append(core[i:]) - break - token = core[i + 1:j] - mods.append((res_idx, token)) - i = j + 1 - else: - stripped.append(ch) - res_idx += 1 - i += 1 - stripped_seq = "".join(stripped) - multiset = tuple(sorted(t for _, t in mods)) - signature = tuple(sorted(mods)) - return is_decoy, stripped_seq, multiset, signature - - -def load_fragment_subset(path, wanted_ids, batch_size=1_000_000): - """Stream chrom and collect (frag_mz, observed_apex_intensity) per candidate - for candidates in `wanted_ids`. Observed apex intensity is the maximum of the - per-fragment intensity trace. Bounded memory: only the wanted subset is held.""" - store = defaultdict(list) # candidate_id -> list[(frag_mz, obs_intensity)] - if len(wanted_ids) == 0: - return store - wanted = np.asarray(sorted(wanted_ids), dtype=np.uint32) - pf = pq.ParquetFile(path) - cols = ["candidate_id", "frag_mz", "intensity"] - for batch in pf.iter_batches(columns=cols, batch_size=batch_size): - cid = batch.column("candidate_id").to_numpy(zero_copy_only=False) - keep = np.isin(cid, wanted) - if not keep.any(): - continue - idx = np.nonzero(keep)[0] - frag_mz_col = batch.column("frag_mz") - int_col = batch.column("intensity") - for k in idx.tolist(): - c = int(cid[k]) - mz = float(frag_mz_col[k].as_py()) - ints = int_col[k].as_py() - obs = float(max(ints)) if ints else 0.0 - store[c].append((mz, obs)) - return store - - -def unique_ion_evidence(self_mz, self_int, sibling_mz): - """Count and sum-intensity of self fragments whose m/z does not match any - sibling fragment m/z within SITE_TOL_PPM (site-determining ions).""" - if len(self_mz) == 0: - return 0, 0.0 - if len(sibling_mz) == 0: - # All fragments are unique when there are no sibling fragments. - return len(self_mz), float(np.sum(self_int)) - sib = np.sort(np.asarray(sibling_mz, dtype=np.float64)) - count = 0 - inten = 0.0 - for mz, it in zip(self_mz, self_int): - tol = mz * SITE_TOL_PPM * 1e-6 - pos = np.searchsorted(sib, mz) - matched = False - if pos < len(sib) and abs(sib[pos] - mz) <= tol: - matched = True - elif pos > 0 and abs(sib[pos - 1] - mz) <= tol: - matched = True - if not matched: - count += 1 - inten += it - return count, float(inten) - - -def main(): - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--psms", required=True) - ap.add_argument("--chrom", required=True) - ap.add_argument("--out", default="localization.parquet") - ap.add_argument("--rt-window-s", type=float, default=30.0) - ap.add_argument("--max-candidates", type=int, default=None, - help="cap the candidate set (for fast smoke tests)") - args = ap.parse_args() - - t0 = time.time() - log(f"[loc] loading psms {args.psms}") - t = pq.read_table(args.psms, - columns=["candidate_id", "apex_rt", "charge", "peptidoform"]) - cid = t.column("candidate_id").to_numpy() - order = np.argsort(cid, kind="stable") - cid = cid[order] - apex_rt = t.column("apex_rt").to_numpy()[order] - charge = t.column("charge").to_numpy()[order] - pep = np.asarray(t.column("peptidoform").to_pylist(), dtype=object)[order] - - if args.max_candidates is not None and args.max_candidates < len(cid): - cid = cid[:args.max_candidates] - apex_rt = apex_rt[:args.max_candidates] - charge = charge[:args.max_candidates] - pep = pep[:args.max_candidates] - - C = len(cid) - log(f"[loc] {C} candidates; grouping by (decoy, stripped_seq, multiset, charge)") - - # Group candidates by (is_decoy, stripped_seq, multiset, charge). - groups = defaultdict(list) # key -> list of local indices - signatures = [None] * C - for i in range(C): - is_decoy, stripped, multiset, sig = parse_peptidoform(str(pep[i])) - signatures[i] = sig - key = (is_decoy, stripped, multiset, int(charge[i])) - groups[key].append(i) - - # Per-candidate outputs (defaults for non-ambiguous candidates). - is_amb = np.zeros(C, dtype=bool) - n_variants = np.ones(C, dtype=np.int64) - # cluster[i] = list of local indices co-eluting with i (including i) - clusters = [None] * C - - rtw = args.rt_window_s - n_amb_groups = 0 - for key, members in groups.items(): - if len(members) < 2: - continue - # Distinct signatures present in the whole key-group. - if len({signatures[m] for m in members}) < 2: - continue # only one localization variant; not ambiguous - m_arr = np.array(members) - rts = apex_rt[m_arr] - sord = np.argsort(rts, kind="stable") - m_sorted = m_arr[sord] - rts_sorted = rts[sord] - group_has_amb = False - for a in range(len(m_sorted)): - i = int(m_sorted[a]) - rt_i = rts_sorted[a] - lo = np.searchsorted(rts_sorted, rt_i - rtw, side="left") - hi = np.searchsorted(rts_sorted, rt_i + rtw, side="right") - cluster_local = [int(m_sorted[b]) for b in range(lo, hi)] - distinct_sigs = {signatures[j] for j in cluster_local} - if len(distinct_sigs) > 1: - is_amb[i] = True - n_variants[i] = len(distinct_sigs) - clusters[i] = cluster_local - group_has_amb = True - if group_has_amb: - n_amb_groups += 1 - - amb_ids = {int(cid[i]) for i in range(C) if is_amb[i]} - log(f"[loc] {len(amb_ids)} candidates in ambiguity groups " - f"across {n_amb_groups} groups") - - sd_count = np.zeros(C, dtype=np.int64) - sd_inten = np.zeros(C, dtype=np.float64) - loc_conf = np.ones(C, dtype=np.float64) # non-ambiguous default: fully localized - - if amb_ids: - log(f"[loc] streaming chrom for {len(amb_ids)} candidates") - frag_store = load_fragment_subset(args.chrom, amb_ids) - cid_to_local = {int(cid[i]): i for i in range(C) if is_amb[i]} - - # Per-candidate self fragment arrays. - self_mz = {} - self_int = {} - for c, lst in frag_store.items(): - if lst: - arr = np.array(lst, dtype=np.float64) - self_mz[c] = arr[:, 0] - self_int[c] = arr[:, 1] - else: - self_mz[c] = np.zeros(0) - self_int[c] = np.zeros(0) - - # Site-determining evidence per ambiguous candidate. - for c, i in cid_to_local.items(): - cluster_local = clusters[i] or [i] - siblings = [j for j in cluster_local - if signatures[j] != signatures[i]] - sib_mz_parts = [] - for j in siblings: - jc = int(cid[j]) - if jc in self_mz and len(self_mz[jc]): - sib_mz_parts.append(self_mz[jc]) - sib_mz = np.concatenate(sib_mz_parts) if sib_mz_parts else np.zeros(0) - smz = self_mz.get(c, np.zeros(0)) - sint = self_int.get(c, np.zeros(0)) - cnt, inten = unique_ion_evidence(smz, sint, sib_mz) - sd_count[i] = cnt - sd_inten[i] = inten - - # Localization confidence = self site-determining intensity over the - # cluster sum. First pass: reuse each member's own site-determining - # intensity (computed against its own siblings). - for c, i in cid_to_local.items(): - cluster_local = clusters[i] or [i] - denom = 0.0 - for j in cluster_local: - denom += sd_inten[j] - if denom > 0: - loc_conf[i] = sd_inten[i] / denom - else: - loc_conf[i] = float("nan") - - out = pa.table({ - "candidate_id": cid.astype(np.uint32), - "is_localization_ambiguous": is_amb, - "n_localization_variants": n_variants, - "site_determining_ion_count": sd_count, - "site_determining_ion_intensity": sd_inten, - "localization_confidence": loc_conf, - }) - pq.write_table(out, args.out) - - n_amb = int(is_amb.sum()) - mean_variants = float(n_variants[is_amb].mean()) if n_amb else 0.0 - dt = time.time() - t0 - log("[loc] done") - print(f"localization.parquet written to {args.out}") - print(f"candidates: {C}") - print(f"ambiguity groups: {n_amb_groups}") - print(f"ambiguous candidates: {n_amb}") - print(f"mean variants per ambiguous candidate: {mean_variants:.4f}") - if args.max_candidates is not None: - print(f"NOTE: --max-candidates={args.max_candidates} limits the " - f"candidate set; ambiguity detection is over the subset only.") - print(f"runtime_s: {dt:.1f}") - - -if __name__ == "__main__": - main() diff --git a/scripts/make_mutation_decoys_fasta.py b/scripts/make_mutation_decoys_fasta.py deleted file mode 100644 index bf53cf1..0000000 --- a/scripts/make_mutation_decoys_fasta.py +++ /dev/null @@ -1,60 +0,0 @@ -"""Generate DIA-NN-style MUTATION decoys as a FASTA for DIA-NN to predict, so the -decoy fragments are predicted by the SAME model as the targets (a valid target-decoy -null that experiences the same chimeric interference). Reimplements the mutation-decoy -concept from DIA-NN (diann.cpp, CC BY 4.0, Demichev et al.); the substitution map is -independently chosen, not copied. - -Each interior residue is substituted via a fixed derangement whose images never -contain K or R, so interior tryptic sites are removed and each decoy peptide, written -as a one-peptide protein, digests back to exactly itself (C-term K/R kept). Terminal -residues are preserved. Decoys equal to a real target are re-mutated. - -Usage: python make_mutation_decoys_fasta.py -""" -import re -import sys -import pandas as pd - -# derangement over the 20 AAs; no image is K or R (keeps interior non-tryptic) -MUT = {'A': 'L', 'R': 'N', 'N': 'D', 'D': 'E', 'C': 'S', 'E': 'G', 'Q': 'A', - 'G': 'V', 'H': 'Y', 'I': 'M', 'L': 'F', 'K': 'Q', 'M': 'T', 'F': 'W', - 'P': 'S', 'S': 'T', 'T': 'V', 'W': 'Y', 'Y': 'H', 'V': 'I'} -STD = set("ACDEFGHIKLMNPQRSTVWY") -strip = lambda pf: re.sub(r"\[[^\]]*\]", "", str(pf).replace("DECOY_", "")) - - -def mutate(seq): - if len(seq) < 3: - return None - interior = [MUT.get(c, c) for c in seq[1:-1]] - return seq[0] + "".join(interior) + seq[-1] - - -def main(): - lib_in, out_fasta = sys.argv[1:3] - lib = pd.read_parquet(lib_in, columns=["peptidoform"]) - seqs = sorted({strip(p) for p in lib.peptidoform}) - seqs = [s for s in seqs if len(s) >= 5 and all(c in STD for c in s)] - target_set = set(seqs) - print(f"unique target sequences: {len(seqs)}", flush=True) - - out, made, collide = [], 0, 0 - for i, s in enumerate(seqs): - d = mutate(s) - if d is None: - continue - if d in target_set: - # second-pass: also mutate the first residue to break the collision - d = MUT.get(s[0], s[0]) + d[1:] - collide += 1 - if d in target_set: - continue - out.append(f">DECOY_{i}\n{d}") - made += 1 - with open(out_fasta, "w") as fh: - fh.write("\n".join(out) + "\n") - print(f"wrote {made} decoy proteins ({collide} needed collision fix) -> {out_fasta}") - - -if __name__ == "__main__": - main() diff --git a/scripts/nn_semisupervised_rescore.ipynb b/scripts/nn_semisupervised_rescore.ipynb deleted file mode 100644 index dd30f63..0000000 --- a/scripts/nn_semisupervised_rescore.ipynb +++ /dev/null @@ -1,567 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "3dc93822", - "metadata": {}, - "source": [ - "# Semi-supervised NN rescoring of a MuMDIA PIN\n", - "\n", - "Ingests a Percolator `.pin` (MuMDIA `features`/`compete` output) and rescores it with a\n", - "**PyTorch MLP** trained in the **Percolator/mokapot semi-supervised** scheme, i.e. the\n", - "DIA-NN-style nonlinear classifier over the same feature set.\n", - "\n", - "Algorithm (per cross-validation fold, so every PSM is scored by a model that never saw it):\n", - "1. Initialise a score from the single best feature+sign (most targets at `train_fdr`).\n", - "2. Repeat `n_iter` times: recompute target-decoy q-values on the training folds, take\n", - " **target PSMs with q <= train_fdr as positives** and **all decoys as negatives**, train\n", - " the MLP from scratch on that labelled set, rescore the training folds.\n", - "3. Score the held-out fold with the final model.\n", - "\n", - "Final target-decoy q-values are computed at PSM and peptide level; the peptide count at\n", - "1% FDR is the number to compare against mokapot (linear) on the same PIN.\n", - "\n", - "**Note.** The NN introduces run-to-run nondeterminism even with seeds set (CPU thread\n", - "scheduling, cuDNN off here). Treat the peptide count as approximate (+/- a few tens)." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "b42eead6", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:11:57.843746Z", - "iopub.status.busy": "2026-07-19T10:11:57.843746Z", - "iopub.status.idle": "2026-07-19T10:11:57.846628Z", - "shell.execute_reply": "2026-07-19T10:11:57.846628Z" - } - }, - "outputs": [], - "source": [ - "# ---- parameters ----\n", - "# PIN_PATH = r'C:/proteobench/out_ecoli_ox/run.pin' # apex-gated pool (107k): NN=10,195, mokapot=9,928\n", - "PIN_PATH = r'C:/proteobench/out_ox_gateoff/x.pin' # GATE-OFF pool (503k, no spectral gate)\n", - "N_FOLDS = 3 # cross-validation folds (mokapot default)\n", - "N_ITER = 5 # semi-supervised self-training iterations\n", - "TRAIN_FDR = 0.01 # positive-selection FDR during training\n", - "EPOCHS = 25 # NN epochs per iteration\n", - "HIDDEN = [128, 64]\n", - "DROPOUT = 0.3\n", - "LR = 1e-3\n", - "WEIGHT_DECAY = 1e-4\n", - "BATCH = 4096\n", - "SEED = 0\n", - "\n", - "# optional DIA-NN concordance (set to None to skip)\n", - "DIANN_REPORT = r'C:/Users/robbi/OneDrive - UGent/MuMDIA_NG/out_diann/report.tsv'\n", - "DIANN_TAG = 'ECOLI' # substring in Proteins marking the target proteome\n", - "MOKAPOT_BASELINE = 9928 # apex-gated mokapot @1% (the number to beat; gate-off has no mokapot ref)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "9dddbace", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:11:57.847633Z", - "iopub.status.busy": "2026-07-19T10:11:57.847633Z", - "iopub.status.idle": "2026-07-19T10:12:04.084669Z", - "shell.execute_reply": "2026-07-19T10:12:04.084669Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "device: cuda | torch 2.5.1+cu121\n" - ] - } - ], - "source": [ - "import re\n", - "import numpy as np\n", - "import pandas as pd\n", - "import torch\n", - "import torch.nn as nn\n", - "\n", - "np.random.seed(SEED)\n", - "torch.manual_seed(SEED)\n", - "torch.use_deterministic_algorithms(False)\n", - "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n", - "print('device:', DEVICE, '| torch', torch.__version__)" - ] - }, - { - "cell_type": "markdown", - "id": "110cac1f", - "metadata": {}, - "source": [ - "## Parse the PIN\n", - "\n", - "Columns are `SpecId, Label, ScanNr, ExpMass, CalcMass, , Peptide, Proteins`.\n", - "`ExpMass`/`CalcMass` are Percolator bookkeeping, not features. Folds are assigned by a hash\n", - "of the **stripped peptide** so all PSMs of a peptide (and its paired decoy) stay together,\n", - "avoiding train/test leakage." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "b96b880e", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:12:04.085673Z", - "iopub.status.busy": "2026-07-19T10:12:04.085673Z", - "iopub.status.idle": "2026-07-19T10:12:23.146046Z", - "shell.execute_reply": "2026-07-19T10:12:23.145039Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "503,251 PSMs | 379 features\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "C:\\Users\\robbi\\AppData\\Local\\Temp\\ipykernel_44372\\3106957684.py:14: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n", - " pin['strip'] = pin['Peptide'].map(strip_pep)\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "C:\\Users\\robbi\\AppData\\Local\\Temp\\ipykernel_44372\\3106957684.py:15: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n", - " pin['pep_mod'] = pin['Peptide'].map(lambda p: re.sub(r'^[A-Z-]\\.|\\.[A-Z-]$', '', str(p)))\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold sizes: [167533 167862 167856]\n" - ] - } - ], - "source": [ - "pin = pd.read_csv(PIN_PATH, sep='\\t')\n", - "NON_FEATURE = {'SpecId', 'Label', 'ScanNr', 'ExpMass', 'CalcMass', 'Peptide', 'Proteins'}\n", - "feat_cols = [c for c in pin.columns if c not in NON_FEATURE]\n", - "print(f'{len(pin):,} PSMs | {len(feat_cols)} features')\n", - "\n", - "y = (pin['Label'].to_numpy() == 1).astype(np.float32) # 1 target, 0 decoy\n", - "X = np.nan_to_num(pin[feat_cols].to_numpy(np.float32), nan=0.0, posinf=0.0, neginf=0.0)\n", - "\n", - "STRIP = lambda p: re.sub(r'\\[[^\\]]*\\]', '', str(p)).strip('-.').split('.')[-1] if '.' in str(p) else str(p)\n", - "def strip_pep(p):\n", - " s = re.sub(r'\\[[^\\]]*\\]', '', str(p)) # drop mods\n", - " s = re.sub(r'^[A-Z-]\\.', '', s); s = re.sub(r'\\.[A-Z-]$', '', s) # drop flanks\n", - " return s\n", - "pin['strip'] = pin['Peptide'].map(strip_pep)\n", - "pin['pep_mod'] = pin['Peptide'].map(lambda p: re.sub(r'^[A-Z-]\\.|\\.[A-Z-]$', '', str(p)))\n", - "\n", - "# deterministic fold by peptide hash (keeps a peptide's PSMs in one fold)\n", - "import hashlib\n", - "def h(s):\n", - " return int(hashlib.md5(s.encode()).hexdigest(), 16)\n", - "fold = pin['strip'].map(lambda s: h(s) % N_FOLDS).to_numpy()\n", - "print('fold sizes:', np.bincount(fold))\n", - "\n", - "# robust standardisation (median / IQR, clip) for stable NN training\n", - "med = np.median(X, axis=0)\n", - "iqr = np.subtract(*np.percentile(X, [75, 25], axis=0)); iqr[iqr == 0] = 1.0\n", - "Xs = np.clip((X - med) / iqr, -8, 8).astype(np.float32)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "6028bf21", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:12:23.149047Z", - "iopub.status.busy": "2026-07-19T10:12:23.148046Z", - "iopub.status.idle": "2026-07-19T10:12:23.153557Z", - "shell.execute_reply": "2026-07-19T10:12:23.153049Z" - } - }, - "outputs": [], - "source": [ - "def tda_q(scores, is_target):\n", - " \"\"\"Target-decoy q-values. scores desc; FDR=(decoys+1)/max(1,targets), q=running min.\"\"\"\n", - " order = np.argsort(-scores, kind='stable')\n", - " t = is_target[order].astype(float)\n", - " ct = np.cumsum(t); cd = np.cumsum(1 - t)\n", - " fdr = (cd + 1) / np.maximum(ct, 1)\n", - " q = np.minimum.accumulate(fdr[::-1])[::-1]\n", - " out = np.empty_like(q); out[order] = q\n", - " return out\n", - "\n", - "def n_targets_at(scores, is_target, fdr=0.01):\n", - " q = tda_q(scores, is_target)\n", - " return int(((q <= fdr) & (is_target == 1)).sum())\n", - "\n", - "def peptide_count(scores, is_target, pep, fdr=0.01):\n", - " \"\"\"Best PSM per peptide, then target-decoy q at peptide level.\"\"\"\n", - " df = pd.DataFrame({'s': scores, 't': is_target, 'p': pep})\n", - " best = df.sort_values('s', ascending=False).drop_duplicates('p')\n", - " q = tda_q(best['s'].to_numpy(), best['t'].to_numpy())\n", - " keep = (q <= fdr) & (best['t'].to_numpy() == 1)\n", - " return int(keep.sum()), best.loc[keep, 'p']" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "4ca8b2ff", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:12:23.154561Z", - "iopub.status.busy": "2026-07-19T10:12:23.154561Z", - "iopub.status.idle": "2026-07-19T10:12:52.551170Z", - "shell.execute_reply": "2026-07-19T10:12:52.550654Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "init feature: spectral_entropy_similarity_topk sign +1 -> 8571 targets @ 1%\n" - ] - } - ], - "source": [ - "# initial direction: best single feature+sign by targets at TRAIN_FDR (mokapot-style)\n", - "best_feat, best_sign, best_n = None, 1, -1\n", - "for j in range(Xs.shape[1]):\n", - " for sign in (1, -1):\n", - " n = n_targets_at(sign * Xs[:, j], y, TRAIN_FDR)\n", - " if n > best_n:\n", - " best_n, best_feat, best_sign = n, j, sign\n", - "print(f'init feature: {feat_cols[best_feat]} sign {best_sign:+d} -> {best_n} targets @ {TRAIN_FDR:.0%}')\n", - "init_score = best_sign * Xs[:, best_feat]" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "fe435008", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:12:52.552178Z", - "iopub.status.busy": "2026-07-19T10:12:52.552178Z", - "iopub.status.idle": "2026-07-19T10:12:52.556224Z", - "shell.execute_reply": "2026-07-19T10:12:52.556224Z" - } - }, - "outputs": [], - "source": [ - "class MLP(nn.Module):\n", - " def __init__(self, d_in, hidden, p):\n", - " super().__init__()\n", - " layers, d = [], d_in\n", - " for hdim in hidden:\n", - " layers += [nn.Linear(d, hdim), nn.BatchNorm1d(hdim), nn.ReLU(), nn.Dropout(p)]\n", - " d = hdim\n", - " layers += [nn.Linear(d, 1)]\n", - " self.net = nn.Sequential(*layers)\n", - " def forward(self, x):\n", - " return self.net(x).squeeze(-1)\n", - "\n", - "def train_model(Xtr, ytr, epochs, pos_weight):\n", - " torch.manual_seed(SEED)\n", - " m = MLP(Xtr.shape[1], HIDDEN, DROPOUT).to(DEVICE)\n", - " opt = torch.optim.Adam(m.parameters(), lr=LR, weight_decay=WEIGHT_DECAY)\n", - " lossf = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(pos_weight, device=DEVICE))\n", - " Xt = torch.from_numpy(Xtr).to(DEVICE); yt = torch.from_numpy(ytr).to(DEVICE)\n", - " n = len(Xt)\n", - " for _ in range(epochs):\n", - " m.train(); perm = torch.randperm(n, device=DEVICE)\n", - " for i in range(0, n, BATCH):\n", - " idx = perm[i:i + BATCH]\n", - " opt.zero_grad(); loss = lossf(m(Xt[idx]), yt[idx]); loss.backward(); opt.step()\n", - " return m\n", - "\n", - "@torch.no_grad()\n", - "def score_model(m, Xarr):\n", - " m.eval()\n", - " return m(torch.from_numpy(Xarr).to(DEVICE)).cpu().numpy()" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "054398f3", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:12:52.557230Z", - "iopub.status.busy": "2026-07-19T10:12:52.557230Z", - "iopub.status.idle": "2026-07-19T10:13:23.477226Z", - "shell.execute_reply": "2026-07-19T10:13:23.476219Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 0 iter 0: 5701 positives, 163106 decoys, train targets@1% = 7431\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 0 iter 1: 7431 positives, 163106 decoys, train targets@1% = 7732\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 0 iter 2: 7732 positives, 163106 decoys, train targets@1% = 8022\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 0 iter 3: 8022 positives, 163106 decoys, train targets@1% = 8138\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 0 iter 4: 8138 positives, 163106 decoys, train targets@1% = 8176\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 1 iter 0: 5667 positives, 162038 decoys, train targets@1% = 7529\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 1 iter 1: 7529 positives, 162038 decoys, train targets@1% = 7770\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 1 iter 2: 7770 positives, 162038 decoys, train targets@1% = 7905\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 1 iter 3: 7905 positives, 162038 decoys, train targets@1% = 7983\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 1 iter 4: 7983 positives, 162038 decoys, train targets@1% = 8056\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 2 iter 0: 5766 positives, 162700 decoys, train targets@1% = 7168\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 2 iter 1: 7168 positives, 162700 decoys, train targets@1% = 7664\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 2 iter 2: 7664 positives, 162700 decoys, train targets@1% = 8034\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 2 iter 3: 8034 positives, 162700 decoys, train targets@1% = 8125\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "fold 2 iter 4: 8125 positives, 162700 decoys, train targets@1% = 8143\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CV scoring done\n" - ] - } - ], - "source": [ - "# semi-supervised cross-validated training\n", - "final = np.zeros(len(Xs), np.float32)\n", - "for f in range(N_FOLDS):\n", - " tr = fold != f; te = fold == f\n", - " Xtr, ytr = Xs[tr], y[tr]\n", - " score_tr = init_score[tr].copy()\n", - " model = None\n", - " for it in range(N_ITER):\n", - " q = tda_q(score_tr, ytr)\n", - " pos = (q <= TRAIN_FDR) & (ytr == 1)\n", - " neg = ytr == 0\n", - " sel = pos | neg\n", - " pw = float(neg.sum()) / max(1.0, float(pos.sum())) # balance rare positives\n", - " model = train_model(Xtr[sel], ytr[sel], EPOCHS, pw)\n", - " score_tr = score_model(model, Xtr)\n", - " print(f'fold {f} iter {it}: {int(pos.sum())} positives, {int(neg.sum())} decoys, '\n", - " f'train targets@1% = {n_targets_at(score_tr, ytr, TRAIN_FDR)}')\n", - " final[te] = score_model(model, Xs[te])\n", - "print('CV scoring done')" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "c4d804cb", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:13:23.478225Z", - "iopub.status.busy": "2026-07-19T10:13:23.478225Z", - "iopub.status.idle": "2026-07-19T10:13:23.774638Z", - "shell.execute_reply": "2026-07-19T10:13:23.774638Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "PSMs @1% FDR: 11819\n", - "peptides @1% FDR (NN): 10447 | mokapot baseline: 9928 | delta +519\n" - ] - } - ], - "source": [ - "# final target-decoy q-values\n", - "psm_n = n_targets_at(final, y, 0.01)\n", - "pep_n, pep_ids = peptide_count(final, y, pin['pep_mod'].to_numpy(), 0.01)\n", - "print(f'PSMs @1% FDR: {psm_n}')\n", - "print(f'peptides @1% FDR (NN): {pep_n} | mokapot baseline: {MOKAPOT_BASELINE} | delta {pep_n - MOKAPOT_BASELINE:+d}')" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "e39d8742", - "metadata": { - "execution": { - "iopub.execute_input": "2026-07-19T10:13:23.776643Z", - "iopub.status.busy": "2026-07-19T10:13:23.776643Z", - "iopub.status.idle": "2026-07-19T10:13:24.064412Z", - "shell.execute_reply": "2026-07-19T10:13:24.064412Z" - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "DIA-NN E.coli @1%: 11642\n", - "NN E.coli peptides: 10327 | recovered 9707 = 83.4% of DIA-NN | concordance 94.0%\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "C:\\Users\\robbi\\AppData\\Local\\Temp\\ipykernel_44372\\2592420269.py:8: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n", - " df = pin.assign(q=q, s=final)\n", - "C:\\Users\\robbi\\AppData\\Local\\Temp\\ipykernel_44372\\2592420269.py:8: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n", - " df = pin.assign(q=q, s=final)\n" - ] - } - ], - "source": [ - "# optional: DIA-NN E.coli concordance\n", - "if DIANN_REPORT:\n", - " dn = pd.read_csv(DIANN_REPORT, sep='\\t', usecols=['Stripped.Sequence', 'Protein.Names', 'Q.Value'])\n", - " dn = dn[(dn['Q.Value'] <= 0.01) & (dn['Protein.Names'].astype(str).str.contains(DIANN_TAG, na=False))]\n", - " diann = set(dn['Stripped.Sequence'])\n", - " # NN target peptides that are E.coli (Proteins carries the tag)\n", - " q = tda_q(final, y)\n", - " df = pin.assign(q=q, s=final)\n", - " tgt = df[(df.Label == 1) & (df.q <= 0.01) & df.Proteins.astype(str).str.contains(DIANN_TAG, na=False)]\n", - " nn_strip = set(tgt['strip'])\n", - " rec = nn_strip & diann\n", - " print(f'DIA-NN E.coli @1%: {len(diann)}')\n", - " print(f'NN E.coli peptides: {len(nn_strip)} | recovered {len(rec)} = {100*len(rec)/len(diann):.1f}% of DIA-NN '\n", - " f'| concordance {100*len(rec)/max(1,len(nn_strip)):.1f}%')" - ] - }, - { - "cell_type": "markdown", - "id": "0bfa71cc", - "metadata": {}, - "source": [ - "## Notes\n", - "\n", - "- **Fair comparison:** run against the same PIN mokapot scored (`out_ecoli_ox/run.pin`, apex-gated pool,\n", - " mokapot = 9,928 peptides). Any lift is the nonlinear classifier extracting more from the 379 features.\n", - "- **q-value estimator:** `(decoys+1)/max(1,targets)` with running-min, standard TDA. mokapot's estimator\n", - " differs slightly, so treat the absolute count as comparable-not-identical; the delta is the signal.\n", - "- **Leakage:** folds are split by stripped peptide, so a target and its paired decoy never straddle\n", - " train/test. Standardisation is global (negligible leakage).\n", - "- **Levers to try:** deeper/wider `HIDDEN`, more `N_ITER`, ensembling several seeds and averaging scores\n", - " (reduces NN variance, closer to DIA-NN's NN ensemble), focal/ranking loss instead of BCE, or feeding a\n", - " gate-off PIN (larger pool) to test whether the NN tolerates the FDR-flood better than the linear model." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "py312_mumdia", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/scripts/normalize_output.py b/scripts/normalize_output.py deleted file mode 100644 index 49a5449..0000000 --- a/scripts/normalize_output.py +++ /dev/null @@ -1,295 +0,0 @@ -"""Normalize a MuMDIA scored table to the common benchmark schema. - -Sensitivity-plan backlog item P0.2 (normalized output converter). The script is -non-invasive: it reads existing MuMDIA artifacts (a scored PSM table, and -optionally the psms and peptide-quant tables) and writes a single Parquet in the -common schema of sensitivity_plan spec 02 section 4: - - run_id, precursor_id, stripped_sequence, modified_sequence, charge, - protein_ids, is_decoy, is_entrapment, precursor_mz, apex_rt, score, q_value, - pep, quantity, engine, engine_version - -Two export modes support the spec requirement to keep both "all top-scoring -candidates before FDR" and "final reported candidates": - --all-candidates keep every row (default) - --reported-only keep rows with q_value <= --q - -The output is deterministic: rows are sorted by (run_id, precursor_id), and -precursor keys are reproducible because precursor_id is the stable MuMDIA -candidate_id. - -Interpreter: C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe -(pyarrow, pandas, numpy, stdlib tomllib). - -Example -------- -python normalize_output.py \ - --scored C:/proteobench/out_ecoli/scored.parquet \ - --psms C:/proteobench/out_ecoli/psms.parquet \ - --out C:/proteobench/accept/ecoli_normalized.parquet \ - --run-id ecoli -""" - -from __future__ import annotations - -import argparse -import re -import sys -import tomllib -from pathlib import Path - -import numpy as np -import pandas as pd -import pyarrow as pa -import pyarrow.parquet as pq - -# Bracketed UniMod-style modification token and the decoy prefix. -_MOD_RE = re.compile(r"\[[^\]]*\]") -_DECOY_RE = re.compile(r"^DECOY_") - -# Preference order for the discriminant score. The final rescore "score" is -# preferred over the pre-rescore "prelim_score" so that the reported score and -# q_value come from the same model; override with --score-col. -_SCORE_PREF = ["score", "prelim_score"] -# Candidate column names for a posterior error probability, if present. -_PEP_PREF = ["pep", "posterior_error_prob", "PEP", "posterior_error_probability"] - -# Common schema, in the fixed spec-02 section-4 order. -_OUT_SCHEMA = pa.schema( - [ - ("run_id", pa.string()), - ("precursor_id", pa.uint32()), - ("stripped_sequence", pa.string()), - ("modified_sequence", pa.string()), - ("charge", pa.int32()), - ("protein_ids", pa.string()), - ("is_decoy", pa.bool_()), - ("is_entrapment", pa.bool_()), - ("precursor_mz", pa.float64()), - ("apex_rt", pa.float64()), - ("score", pa.float64()), - ("q_value", pa.float64()), - ("pep", pa.float64()), - ("quantity", pa.float64()), - ("engine", pa.string()), - ("engine_version", pa.string()), - ] -) - - -def resolve_engine_version(script_path: Path) -> str: - """Resolve the MuMDIA engine version from the Cargo manifests. - - mumdia-core declares ``version.workspace = true``, so the literal version - lives in the workspace root manifest. Falls back to "unknown". - """ - repo_root = script_path.resolve().parents[1] - core = repo_root / "rust" / "mumdia" / "crates" / "mumdia-core" / "Cargo.toml" - workspace = repo_root / "rust" / "mumdia" / "Cargo.toml" - try: - with core.open("rb") as fh: - core_cfg = tomllib.load(fh) - ver = core_cfg.get("package", {}).get("version") - if isinstance(ver, str): - return ver - with workspace.open("rb") as fh: - ws_cfg = tomllib.load(fh) - ver = ws_cfg.get("workspace", {}).get("package", {}).get("version") - if isinstance(ver, str): - return ver - except (OSError, tomllib.TOMLDecodeError): - pass - return "unknown" - - -def pick_column(columns: list[str], prefs: list[str]) -> str | None: - """Return the first preferred column that is present, else None.""" - present = set(columns) - for name in prefs: - if name in present: - return name - return None - - -def normalize( - scored: pd.DataFrame, - psms: pd.DataFrame | None, - pep_quant: pd.DataFrame | None, - run_id: str, - entrapment_substr: str, - score_col: str | None, - engine_version: str, -) -> pd.DataFrame: - """Map MuMDIA columns onto the common normalized schema.""" - n = len(scored) - out = pd.DataFrame(index=scored.index) - - out["run_id"] = run_id - out["precursor_id"] = scored["candidate_id"].astype("uint32") - - pforms = scored["peptidoform"].astype("string") - out["stripped_sequence"] = pforms.str.replace(_MOD_RE, "", regex=True).str.replace( - _DECOY_RE, "", regex=True - ) - out["modified_sequence"] = pforms - out["charge"] = scored["charge"].astype("int32") - out["protein_ids"] = scored["protein"].astype("string") - out["is_decoy"] = (scored["label"].astype("string") == "decoy").astype(bool) - out["is_entrapment"] = ( - scored["protein"].astype("string").str.contains(entrapment_substr, regex=False) - ).fillna(False).astype(bool) - - # precursor_mz and apex_rt live in the psms table (not in scored); take them - # from a join when available, otherwise from scored if it carries them, else - # leave null. - out["precursor_mz"] = _fill_from_join( - scored, psms, "precursor_mz", "candidate_id" - ) - out["apex_rt"] = _fill_from_join(scored, psms, "apex_rt", "candidate_id") - - if score_col is None: - score_col = pick_column(list(scored.columns), _SCORE_PREF) - if score_col is None: - out["score"] = np.nan - else: - if score_col not in scored.columns: - raise ValueError(f"score column {score_col!r} not in scored table") - out["score"] = pd.to_numeric(scored[score_col], errors="coerce") - - out["q_value"] = pd.to_numeric(scored["q_value"], errors="coerce") - - pep_col = pick_column(list(scored.columns), _PEP_PREF) - out["pep"] = ( - pd.to_numeric(scored[pep_col], errors="coerce") - if pep_col - else pd.Series(np.nan, index=scored.index) - ) - - out["quantity"] = _fill_from_join( - scored, pep_quant, "quantity", "candidate_id" - ) - - out["engine"] = "mumdia" - out["engine_version"] = engine_version - - assert len(out) == n - return out - - -def _fill_from_join( - scored: pd.DataFrame, - other: pd.DataFrame | None, - value_col: str, - key: str, -) -> pd.Series: - """Return value_col aligned to scored: from `other` via join, or from scored, - else all-null. The join collapses duplicate keys in `other` to the first.""" - if other is not None and value_col in other.columns and key in other.columns: - lut = other.drop_duplicates(subset=[key]).set_index(key)[value_col] - return pd.to_numeric(scored[key].map(lut), errors="coerce") - if value_col in scored.columns: - return pd.to_numeric(scored[value_col], errors="coerce") - return pd.Series(np.nan, index=scored.index) - - -def to_arrow(df: pd.DataFrame) -> pa.Table: - """Build an Arrow table with the fixed common schema (NaN -> null).""" - arrays = [] - for field in _OUT_SCHEMA: - arrays.append(pa.array(df[field.name], type=field.type, from_pandas=True)) - return pa.Table.from_arrays(arrays, schema=_OUT_SCHEMA) - - -def parse_args(argv: list[str] | None = None) -> argparse.Namespace: - p = argparse.ArgumentParser( - description="Normalize a MuMDIA scored table to the common schema (P0.2)." - ) - p.add_argument("--scored", required=True, type=Path, help="MuMDIA scored Parquet.") - p.add_argument( - "--psms", type=Path, help="MuMDIA psms Parquet (for apex_rt, precursor_mz)." - ) - p.add_argument( - "--peptide-quant", - type=Path, - help="MuMDIA peptide-quant Parquet (for quantity).", - ) - p.add_argument("--out", type=Path, help="Output normalized Parquet.") - p.add_argument("--run-id", default="run", help="Run identifier (default 'run').") - p.add_argument( - "--entrapment-substr", - default="_HUMAN", - help="Substring flagging a protein as entrapment (default '_HUMAN').", - ) - p.add_argument( - "--score-col", - default=None, - help="Score column to use (default: first of 'score', 'prelim_score').", - ) - mode = p.add_mutually_exclusive_group() - mode.add_argument( - "--all-candidates", - action="store_true", - help="Keep all candidates (default).", - ) - mode.add_argument( - "--reported-only", - action="store_true", - help="Keep only rows with q_value <= --q.", - ) - p.add_argument( - "--q", type=float, default=0.01, help="q-value cutoff for --reported-only." - ) - return p.parse_args(argv) - - -def main(argv: list[str] | None = None) -> int: - args = parse_args(argv) - if not args.scored.exists(): - print(f"error: scored table not found: {args.scored}", file=sys.stderr) - return 1 - - scored = pq.read_table(args.scored).to_pandas() - psms = pq.read_table(args.psms).to_pandas() if args.psms else None - pep_quant = ( - pq.read_table(args.peptide_quant).to_pandas() if args.peptide_quant else None - ) - - engine_version = resolve_engine_version(Path(__file__)) - out = normalize( - scored, - psms, - pep_quant, - run_id=args.run_id, - entrapment_substr=args.entrapment_substr, - score_col=args.score_col, - engine_version=engine_version, - ) - - n_total = len(out) - if args.reported_only: - out = out[out["q_value"] <= args.q].copy() - - # Deterministic ordering: precursor_id is the stable candidate_id. - out = out.sort_values(["run_id", "precursor_id"], kind="stable").reset_index( - drop=True - ) - - table = to_arrow(out) - if args.out: - args.out.parent.mkdir(parents=True, exist_ok=True) - pq.write_table(table, args.out) - print(f"wrote normalized table: {args.out}") - - n_dec = int(out["is_decoy"].sum()) - n_ent = int(out["is_entrapment"].sum()) - mode = "reported-only" if args.reported_only else "all-candidates" - print( - f"rows: {len(out)} (of {n_total} scored; mode={mode}); " - f"targets={len(out) - n_dec}, decoys={n_dec}, entrapment={n_ent}" - ) - print(f"columns: {', '.join(f.name for f in _OUT_SCHEMA)}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/scripts/patch_irt.py b/scripts/patch_irt.py deleted file mode 100644 index c1bd32d..0000000 --- a/scripts/patch_irt.py +++ /dev/null @@ -1,46 +0,0 @@ -"""One-off: recompute predicted_irt in a fragment_library_precursors.parquet -using the corrected DeepLC multitask aggregation, without rebuilding fragments. - -Usage: python patch_irt.py -""" -import sys -import numpy as np -import pyarrow as pa -import pyarrow.parquet as pq - - -def main(): - path = sys.argv[1] - import deeplc - - tbl = pq.read_table(path) - pforms = tbl.column("peptidoform").to_pylist() - - # unique peptidoforms (RT is charge-independent) - uniq = {} - order = [] - for pf in pforms: - if pf not in uniq: - uniq[pf] = None - order.append(pf) - - preds = {} - chunk = 200_000 - for start in range(0, len(order), chunk): - batch = order[start:start + chunk] - a = np.asarray(deeplc.predict(batch), dtype=np.float64) - if a.ndim == 2: - a = a.mean(axis=1) - for pf, v in zip(batch, a): - preds[pf] = float(v) - print(f" {min(start + chunk, len(order))}/{len(order)} unique peptidoforms") - - new_irt = np.array([preds[pf] for pf in pforms], dtype=np.float32) - idx = tbl.schema.get_field_index("predicted_irt") - tbl = tbl.set_column(idx, "predicted_irt", pa.array(new_irt, pa.float32())) - pq.write_table(tbl, path) - print(f"patched predicted_irt for {len(pforms)} rows in {path}") - - -if __name__ == "__main__": - main() diff --git a/scripts/peak_selection_model.py b/scripts/peak_selection_model.py deleted file mode 100644 index 496d97f..0000000 --- a/scripts/peak_selection_model.py +++ /dev/null @@ -1,193 +0,0 @@ -"""Peak-selection model (sensitivity_plan backlog P1.3, spec 03 §6 peak-selection). - -First-pass, grouped out-of-fold peak scoring over the top-K peak table emitted by -`extract` when `retain_top_peaks > 1` (`.peaks.parquet`). The question is: -given several retained chromatographic peaks for one precursor, which one is the -right one? This trains a simple ranker on per-peak descriptors and reports how -often it puts the correct peak at rank 1 / in the top 3, versus the raw -evidence-count and area rankings. - -Correctness label: - - with --diann (a DIA-NN report giving a reference apex RT per precursor): the - correct peak is the retained peak whose [start_rt, end_rt] contains, or whose - apex is nearest (within --rt-tol-s) to, the reference apex. This is the honest - label. - - without --diann: a WEAK self-label is used (the peak nearest the apex the - engine currently selected, from psms.apex_rt). This only measures whether the - ranker reproduces the current heuristic, not correctness, and is printed with - that caveat. - -Grouping: all peaks of one candidate stay in the same CV fold (group = candidate). -Leakage guards: fit scaling inside the training fold; the label is never a feature. - -Caveat (P1.2 dependency): the peak table carries apex/boundaries/evidence/area but -not a full per-peak feature vector (the engine computes features only for the -selected apex). A production peak-selection model needs per-peak features; this -first pass uses the peak-shape descriptors that are available. - -Usage: - python peak_selection_model.py --peaks .peaks.parquet --psms psms.parquet - [--diann report] [--folds 3] [--rt-tol-s 10] [--out metrics.json] -Requires an env with scikit-learn + pyarrow (py312_mumdia). Deterministic. -""" -import argparse -import hashlib -import json -import re -import sys - -import numpy as np -import pyarrow.parquet as pq - -strip = lambda p: re.sub(r"\[[^\]]*\]", "", str(p)) - - -def fold_of(cid, folds): - h = hashlib.sha1(str(int(cid)).encode()).hexdigest() - return int(h, 16) % folds - - -def load_diann(path): - """Return dict (stripped_seq, charge) -> apex RT in seconds. Defensive columns.""" - t = pq.read_table(path).to_pandas() if path.endswith(".parquet") else None - if t is None: - import pandas as pd - sep = "\t" if path.endswith((".tsv", ".txt")) else "," - t = pd.read_csv(path, sep=sep) - - def pick(*cands): - for c in cands: - if c in t.columns: - return c - return None - - seqc = pick("Modified.Sequence", "ModifiedPeptide", "Stripped.Sequence", "Peptide") - zc = pick("Precursor.Charge", "Charge", "PrecursorCharge") - rtc = pick("RT", "iRT", "Retention.Time", "RT.Start", "Apex.RT") - if not (seqc and zc and rtc): - raise SystemExit(f"peak_selection_model: DIA-NN columns not found in {path}") - rt = t[rtc].to_numpy(dtype=float) - # detect minutes vs seconds by magnitude (gradients are usually > 20 min in s) - if np.nanmax(rt) < 300: - rt = rt * 60.0 - out = {} - for s, z, r in zip(t[seqc].astype(str), t[zc].astype(int), rt): - out[(strip(s), int(z))] = float(r) - return out - - -def main(): - ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--peaks", required=True) - ap.add_argument("--psms", required=True) - ap.add_argument("--diann", default=None) - ap.add_argument("--folds", type=int, default=3) - ap.add_argument("--rt-tol-s", type=float, default=10.0) - ap.add_argument("--out", default=None) - args = ap.parse_args() - - from sklearn.linear_model import LogisticRegression - - pk = pq.read_table(args.peaks).to_pandas() - ps = pq.read_table(args.psms, columns=["candidate_id", "apex_rt", "peptidoform", "charge", "label"]).to_pandas() - sel_rt = dict(zip(ps.candidate_id.astype(int), ps.apex_rt.astype(float))) - key = {int(c): (strip(p), int(z)) for c, p, z in zip(ps.candidate_id, ps.peptidoform, ps.charge)} - - diann = load_diann(args.diann) if args.diann else None - label_kind = "diann-reference" if diann else "weak-self (current apex)" - - # Per-candidate peak groups. - pk = pk.sort_values(["candidate_id", "peak_rank"]).reset_index(drop=True) - feats, labels, groups = [], [], [] - per_cand = {} - n_labeled = 0 - for cid, g in pk.groupby("candidate_id"): - cid = int(cid) - ev = g.evidence_count.to_numpy(float) - ar = g.area.to_numpy(float) - ap = g.apex_rt.to_numpy(float) - width = (g.end_rt.to_numpy(float) - g.start_rt.to_numpy(float)) - # reference RT for the label - ref = None - if diann is not None: - ref = diann.get(key.get(cid)) - else: - ref = sel_rt.get(cid) - if ref is None: - continue - # correct peak = nearest apex within tol (or containing the ref) - d = np.abs(ap - ref) - if d.min() > args.rt_tol_s: - continue # no retained peak matches the reference -> unlabelable here - correct = int(np.argmin(d)) - n_labeled += 1 - mx_ev = ev.max() if ev.max() > 0 else 1.0 - mx_ar = ar.max() if ar.max() > 0 else 1.0 - for j in range(len(g)): - feats.append([ev[j], ar[j], float(g.peak_rank.iloc[j]), width[j], ev[j] / mx_ev, ar[j] / mx_ar]) - labels.append(1 if j == correct else 0) - groups.append(cid) - per_cand[cid] = (ev, ar, correct) - - if n_labeled < 20: - raise SystemExit(f"peak_selection_model: too few labelable candidates ({n_labeled})") - X = np.asarray(feats, float) - y = np.asarray(labels, int) - grp = np.asarray(groups, int) - - # Out-of-fold scores. - oof = np.zeros(len(y)) - folds = np.array([fold_of(c, args.folds) for c in grp]) - for f in range(args.folds): - tr, te = folds != f, folds == f - if tr.sum() == 0 or te.sum() == 0 or len(np.unique(y[tr])) < 2: - continue - mu, sd = X[tr].mean(0), X[tr].std(0) + 1e-9 - m = LogisticRegression(max_iter=2000, C=1.0) - m.fit((X[tr] - mu) / sd, y[tr]) - oof[te] = m.predict_proba((X[te] - mu) / sd)[:, 1] - - # Recall: does the top-scored peak per candidate match the correct one? - def recall(scorer): - top1 = top3 = 0 - i = 0 - by = {} - for c in grp: - by.setdefault(c, []).append(i) - i += 1 - for c, idx in by.items(): - idx = np.array(idx) - corr = np.array([y[k] for k in idx]).argmax() - order = np.argsort(-scorer[idx], kind="stable") - if order[0] == corr: - top1 += 1 - if corr in order[:3]: - top3 += 1 - n = len(by) - return top1 / n, top3 / n - - r_model = recall(oof) - r_ev = recall(X[:, 0]) # evidence_count - r_area = recall(X[:, 1]) # area - res = { - "label_kind": label_kind, - "n_labeled_candidates": n_labeled, - "model_top1": r_model[0], "model_top3": r_model[1], - "evidence_rank_top1": r_ev[0], "evidence_rank_top3": r_ev[1], - "area_rank_top1": r_area[0], "area_rank_top3": r_area[1], - } - print(f"=== peak-selection model ({label_kind}) ===") - print(f" labelable candidates: {n_labeled}") - print(f" learned model : top1={r_model[0]:.3f} top3={r_model[1]:.3f}") - print(f" evidence-rank : top1={r_ev[0]:.3f} top3={r_ev[1]:.3f}") - print(f" area-rank : top1={r_area[0]:.3f} top3={r_area[1]:.3f}") - if not diann: - print(" NOTE: weak self-label (current apex); measures agreement with the " - "existing heuristic, not correctness. Supply --diann for the honest label.") - if args.out: - json.dump(res, open(args.out, "w"), indent=1) - print(f" wrote {args.out}") - - -if __name__ == "__main__": - main() diff --git a/scripts/plot_ids.py b/scripts/plot_ids.py deleted file mode 100644 index d89a5cf..0000000 --- a/scripts/plot_ids.py +++ /dev/null @@ -1,235 +0,0 @@ -"""Per-peptide diagnostic plots for MuMDIA, drawn from the SEARCH ENGINE's OWN output -(not a Python re-extraction), so they can be used to evaluate the engine: - - XICs = the engine's emitted chromatograms (extract.rs; zero-filled window grid - when emit_window_grid is on, else matched scans). - - apex = the engine's apex_rt from psms_extracted (fragment-count + top-3). - - bounds = the engine's emitted elution_lo/elution_hi (features.rs), i.e. the window - over which the engine computed its trace features. Read from the features - parquet, NOT recomputed in Python. - Panel A: fragment XICs + summed + engine apex + engine feature bounds + predicted RT - and search window; DIA-NN [RT.Start,RT.Stop] shaded in 'diann' mode. - Panel B: apex MS2 mirror, observed (engine, at apex) vs predicted library. - -Modes (chrom source): - mumdia -> engine gate-on output (chrom_port / psms_port), all MuMDIA IDs. - diann -> engine gate-off grid output (chrom_portng_grid / psms_portng_grid), DIA-NN - IDs MuMDIA misses at the SEQUENCE level (a peptide identified under a - different charge/modform is NOT counted as missed). -Env: FRAC (bound fraction, default 1/3 = features default), CHUNK/NCHUNK, OUTSUB. -Usage: python plot_ids.py [limit] -""" -import os -import re -import sys -import json -import numpy as np -import pandas as pd -import matplotlib -matplotlib.use("Agg") -import matplotlib.pyplot as plt - -D = "c:/Users/robbi/OneDrive - UGent/MuMDIA_NG" -Q = 0.01 -FRAC = float(os.environ.get("FRAC", str(1.0 / 3.0))) -CHUNK = int(os.environ.get("CHUNK", "0")) -NCHUNK = int(os.environ.get("NCHUNK", "1")) -OUTSUB = os.environ.get("OUTSUB", "") - - -def to_pf(s): - return str(s).replace("(UniMod:4)", "[Carbamidomethyl]").replace("(UniMod:35)", "[Oxidation]") - - -def strip_seq(pf): - """Stripped amino-acid sequence (drop mods + DECOY_ prefix) for sequence-level ID matching.""" - s = re.sub(r"\[[^\]]*\]", "", str(pf)) - s = re.sub(r"\([^\)]*\)", "", s) - return s.replace("DECOY_", "") - - -def parse_frag(name): - m = re.match(r"([by])(\d+)(?:\^(\d+))?", str(name)) - return (m.group(1), int(m.group(2)), int(m.group(3) or 1)) if m else ("?", 0, 1) - - -def frag_color(ion, ordinal, maxord): - cmap = plt.cm.Blues if ion == "b" else plt.cm.Reds if ion == "y" else plt.cm.Greys - return cmap(0.45 + 0.5 * (ordinal / max(maxord, 1))) - - -def smooth(v): - if len(v) < 3: - return v.astype(float) - o = v.astype(float).copy() - o[1:-1] = 0.5 * v[1:-1] + 0.25 * v[:-2] + 0.25 * v[2:] - o[0] = 2 / 3 * v[0] + 1 / 3 * v[1] - o[-1] = 2 / 3 * v[-1] + 1 / 3 * v[-2] - return o - - -def peak_bounds(axis, prof, ai, frac): - """Match features.rs peak_bounds: descend from apex to prof < frac*peak.""" - if axis.size < 3: - return (axis[0], axis[-1]) if axis.size else (0.0, 0.0) - peak = prof[ai] if prof[ai] > 0 else prof.max() - if peak <= 0: - return axis[0], axis[-1] - thr = frac * peak - lo = ai - while lo > 0 and prof[lo - 1] >= thr: - lo -= 1 - hi = ai - while hi + 1 < axis.size and prof[hi + 1] >= thr: - hi += 1 - return axis[lo], axis[hi] - - -def plot_one(cid, pf, charge, qv, apex_rt, chrom_rows, lib_rows, outpath, - dn_win=None, pred_rt=None, half=None, elo=None, ehi=None): - # chrom_rows: list of (name, rt_array, int_array) = ENGINE chromatograms. - # Build a shared union axis for the mirror. - axis = np.unique(np.concatenate([r[1] for r in chrom_rows])) if chrom_rows else np.array([]) - names = [r[0] for r in chrom_rows] - predmap = {r[0]: r[1] for r in lib_rows} - mat = np.zeros((len(chrom_rows), axis.size)) - for i, (_, rt, it) in enumerate(chrom_rows): - idx = np.clip(np.searchsorted(axis, rt), 0, axis.size - 1) if axis.size else [] - mat[i, idx] = it - ai = int(np.argmin(np.abs(axis - apex_rt))) if axis.size else 0 - # elution bounds: use the ENGINE's emitted feature bounds (features.rs), not recomputed - lo_rt, hi_rt = (elo, ehi) if (elo is not None and ehi is not None and ehi > elo) else (apex_rt, apex_rt) - - fig, (axA, axB) = plt.subplots(1, 2, figsize=(13, 5)) - maxord = max([parse_frag(n)[1] for n in (names + list(predmap))] + [1]) - apex_col = mat[:, ai] if axis.size else np.zeros(len(names)) - athr = 0.05 * apex_col.max() if apex_col.max() > 0 else 0.0 - obs_apex = apex_col > athr - summ = mat.sum(axis=0) - for i, name in sorted(enumerate(names), key=lambda kv: parse_frag(kv[1])): - if not obs_apex[i]: - continue - ion, ordi, _ = parse_frag(name) - axA.plot(chrom_rows[i][1], chrom_rows[i][2], "-", lw=1.2, color=frag_color(ion, ordi, maxord), label=name) - if axis.size: - axA.plot(axis, summ, "-", lw=2.0, color="0.4", alpha=0.5, label="summed", zorder=1) - axA.axvspan(lo_rt, hi_rt, color="0.85", alpha=0.4, zorder=0, label="feature bounds (engine)") - axA.axvline(lo_rt, color="0.5", ls=":", lw=1); axA.axvline(hi_rt, color="0.5", ls=":", lw=1) - axA.axvline(apex_rt, color="k", ls="--", lw=1.3, label="engine apex") - if pred_rt is not None and np.isfinite(pred_rt): - axA.axvline(pred_rt, color="tab:purple", ls=":", lw=1.4, label="pred RT") - if half: - axA.axvline(pred_rt - half, color="tab:orange", ls=":", lw=1.0, alpha=0.7) - axA.axvline(pred_rt + half, color="tab:orange", ls=":", lw=1.0, alpha=0.7, label=f"search +/-{half:.0f}s") - if dn_win is not None and np.isfinite(dn_win[0]): - axA.axvspan(dn_win[0], dn_win[1], color="tab:green", alpha=0.12, zorder=0) - axA.axvline(dn_win[0], color="tab:green", lw=1, alpha=0.6) - axA.axvline(dn_win[1], color="tab:green", lw=1, alpha=0.6, label="DIA-NN RT") - all_names = list(predmap) - missing = [n for n in all_names if n not in set(np.array(names)[obs_apex]) if len(names)] - axA.set_xlabel("retention time (s)"); axA.set_ylabel("fragment intensity") - axA.set_title(f"XIC (engine) {int(obs_apex.sum())}/{len(all_names)} frags at apex" - + (f"\nmissing: {', '.join(missing[:8])}" if missing else ""), fontsize=9) - axA.legend(fontsize=6, ncol=2, loc="upper right") - - obs = {names[i]: (mat[i, ai] if axis.size else 0.0) for i in range(len(names))} - order = sorted(all_names, key=lambda n: (parse_frag(n)[0], parse_frag(n)[1])) - pmax = max(predmap.values()) if predmap else 1.0 - omax = max(obs.values()) if obs and max(obs.values()) > 0 else 1.0 - for i, name in enumerate(order): - ion, ordi, _ = parse_frag(name); c = frag_color(ion, ordi, maxord) - axB.bar(i, obs.get(name, 0) / omax, 0.8, color=c) - axB.bar(i, -(predmap.get(name, 0) / pmax), 0.8, color=c, alpha=0.55, - hatch="///" if obs.get(name, 0) <= athr else None) - axB.axhline(0, color="k", lw=0.8) - axB.set_xticks(range(len(order))); axB.set_xticklabels(order, rotation=90, fontsize=6) - axB.set_ylim(-1.15, 1.15); axB.set_ylabel("observed (up) / predicted (down)") - axB.set_title("apex MS2: observed vs predicted", fontsize=9) - fig.suptitle(f"{pf} (z={charge}) q={qv:.4f} apex={apex_rt:.0f}s cid={cid}", fontsize=10) - fig.tight_layout(rect=[0, 0, 1, 0.96]) - fig.savefig(outpath, dpi=90); plt.close(fig) - - -def main(): - mode = sys.argv[1] - limit = int(sys.argv[2]) if len(sys.argv) > 2 else 0 - lp = pd.read_parquet(f"{D}/out_dl/lib_precursors_ft.parquet", - columns=["candidate_id", "peptidoform", "charge", "label"]) - lf = pd.read_parquet(f"{D}/out_dl/lib_fragments_ft.parquet", - columns=["candidate_id", "name", "predicted_intensity"]) - scored = pd.read_parquet(f"{D}/out_dl/scored_port.parquet", - columns=["candidate_id", "peptidoform", "charge", "label", "q_value"]) - mm = scored[(scored.label == "target") & (scored.q_value <= Q)] - mm_ids = set(mm.candidate_id) - mm_seqs = set(mm.peptidoform.map(strip_seq)) # sequence-level IDs (any charge/modform) - _cal = json.load(open(f"{D}/out_dl/cal_ftgate.json")) - HALF = _cal["w_rt"] / _cal.get("multiplier", 1.0) * 1.10 - - if mode == "mumdia": - outdir = f"{D}/out_dl/plots_mumdia{OUTSUB}" - chrom_path = f"{D}/out_dl/chrom_port.parquet" - psms = pd.read_parquet(f"{D}/out_dl/psms_port.parquet", columns=["candidate_id", "apex_rt", "rt_pred_cal"]) - feat = pd.read_parquet(f"{D}/out_dl/feat_port.parquet", columns=["candidate_id", "elution_lo", "elution_hi"]) - t = mm[["candidate_id", "peptidoform", "charge", "q_value"]] - targets = t.set_index("candidate_id"); dn_bounds = {} - else: - outdir = f"{D}/out_dl/plots_diann_not_mumdia{OUTSUB}" - # Small pre-subset of the 2 GB gate-off grid to just the truly-missed candidates - # (scratchpad/subset_grid.py), so parallel workers don't each decode a ~4 GB row group. - chrom_path = f"{D}/out_dl/chrom_missed_grid.parquet" - psms = pd.read_parquet(f"{D}/out_dl/psms_portng_grid.parquet", columns=["candidate_id", "apex_rt", "rt_pred_cal"]) - feat = pd.read_parquet(f"{D}/out_dl/feat_portng_grid.parquet", columns=["candidate_id", "elution_lo", "elution_hi"]) - dn = pd.read_csv(f"{D}/out_diann/report.tsv", sep="\t") - dn = dn[dn["Q.Value"] <= Q].copy().rename(columns={"RT.Start": "rs", "RT.Stop": "re", - "Precursor.Charge": "pz", "Q.Value": "qv"}) - dn["pf"] = dn["Modified.Sequence"].map(to_pf); dn["key"] = list(zip(dn.pf, dn.pz.astype(int))) - lpt = lp[lp.label == "target"].copy(); lpt["key"] = list(zip(lpt.peptidoform, lpt.charge.astype(int))) - k2c = dict(zip(lpt.key, lpt.candidate_id)) - dn = dn.drop_duplicates("key"); dn["cid"] = dn.key.map(k2c); dn = dn[dn.cid.notna()] - dn["cid"] = dn.cid.astype(int) - # Exclude at SEQUENCE level: a peptide MuMDIA identified under any charge/modform is - # NOT a miss (compete keeps one charge per base peptide). Excluding only the exact - # candidate_id would leave sibling charges (e.g. z3 when z2 was identified) as false misses. - dn["seq"] = dn.pf.map(strip_seq) - dn = dn[~dn.cid.isin(mm_ids) & ~dn.seq.isin(mm_seqs)] - dn_bounds = {int(r.cid): (r.rs * 60, r.re * 60) for r in dn.itertuples()} - targets = dn.set_index("cid")[["pf", "pz", "qv"]]; targets.columns = ["peptidoform", "charge", "q_value"] - - os.makedirs(outdir, exist_ok=True) - apex = dict(zip(psms.candidate_id, psms.apex_rt)) - rtcal = dict(zip(psms.candidate_id, psms.rt_pred_cal)) - ebounds = {r.candidate_id: (r.elution_lo, r.elution_hi) for r in feat.itertuples()} - cids = [c for i, c in enumerate(targets.index) if i % NCHUNK == CHUNK] - if limit: - cids = cids[:limit] - cidset = set(int(c) for c in cids) - # Load only this chunk's candidates (predicate pushdown) so a 2 GB grid parquet does - # not blow up memory with 8 parallel workers each reading the whole file. - chrom = pd.read_parquet(chrom_path, filters=[("candidate_id", "in", cidset)]) - lf = lf[lf.candidate_id.isin(cidset)].sort_values(["candidate_id", "predicted_intensity"], ascending=[True, False]) - chrom_by = {c: [(r.frag_name, np.asarray(r.rt, float), np.asarray(r.intensity, float)) - for r in g.itertuples()] for c, g in chrom.groupby("candidate_id")} - lib_by = {c: list(zip(g.name.head(12), g.predicted_intensity.head(12))) for c, g in lf.groupby("candidate_id")} - print(f"mode={mode} chunk={CHUNK}/{NCHUNK} n={len(cids)} -> {outdir}", flush=True) - done = 0 - for cid in cids: - cr = chrom_by.get(cid); lr = lib_by.get(cid) - if not cr or not lr or cid not in apex: - continue - row = targets.loc[cid] - pf = str(row["peptidoform"]); z = int(row["charge"]); qv = float(row["q_value"]) - fn = re.sub(r"[^A-Za-z0-9]", "_", pf)[:40] + f"_z{z}_{cid}.png" - outpath = os.path.join(outdir, fn) - if os.path.exists(outpath): - done += 1; continue - eb = ebounds.get(cid, (None, None)) - plot_one(cid, pf, z, qv, apex[cid], cr, lr, outpath, - dn_win=dn_bounds.get(cid), pred_rt=rtcal.get(cid), half=HALF, - elo=eb[0], ehi=eb[1]) - done += 1 - if done % 500 == 0: - print(f" {done}/{len(cids)}", flush=True) - print(f"wrote/kept {done} -> {outdir}") - - -if __name__ == "__main__": - main() diff --git a/scripts/reference_apex_topk.py b/scripts/reference_apex_topk.py deleted file mode 100644 index d4f8d12..0000000 --- a/scripts/reference_apex_topk.py +++ /dev/null @@ -1,470 +0,0 @@ -#!/usr/bin/env python -"""Top-K chromatographic-peak oracle for MuMDIA (spec 02 Section 5, backlog P0.4). - -The central sensitivity hypothesis is that MuMDIA commits to one chromatographic -apex too early, discarding the correct peak before the scorer sees it. This -diagnostic quantifies that opportunity non-invasively: it rebuilds each -candidate's consensus elution profile from the extracted fragment chromatograms -(chrom.parquet), enumerates its chromatographic peaks with the same semantics as -the Rust `enumerate_peaks` (rust/mumdia/crates/mumdia/src/peaks.rs), and asks - - SELF : where in the area-ranked peak list does the apex MuMDIA actually chose - (psms.apex_rt) fall? If it is rank 1 almost always, top-K rescue offers - little; if the true apex is often rank 2+ or in no peak, the top-K peak - model has headroom. - - REFERENCE (optional, --diann): for precursors also identified by DIA-NN, is the - DIA-NN apex RT within tolerance of one of the top-K MuMDIA peaks? This is - a peak-recall upper bound against an external reference. - -The script reads existing Parquet artifacts only; it changes no engine state and -is deterministic (fixed numeric order, no RNG). -""" - -from __future__ import annotations - -import argparse -import json -import os -import re -import sys -from collections import defaultdict - -import numpy as np -import pyarrow.parquet as pq - -# Deterministic: no stochastic component, but seed numpy for defensiveness. -np.random.seed(0) - -MOD_BRACKET = re.compile(r"\[[^\]]*\]") # ProForma-lite [UniMod:xx] / [+15.99] -MOD_PAREN = re.compile(r"\([^)]*\)") # DIA-NN (UniMod:xx) - - -# --------------------------------------------------------------------------- # -# Peak enumeration: faithful port of rust/.../peaks.rs::enumerate_peaks -# --------------------------------------------------------------------------- # -def enumerate_peaks(profile, k, bound_fraction, min_prominence_frac): - """Return peak groups as dicts, area-ranked (rank 0 = strongest). - - Each group: apex_idx, start_idx, end_idx, apex_intensity, area, rank. - Semantics match the Rust reference exactly (local maxima with strict left / - non-strict right, prominence floor, fractional-height boundary walk that also - stops at a valley, dedup of maxima inside a stronger envelope, area sort with - apex-intensity then earliest-apex tie breaks). - """ - n = len(profile) - if k == 0 or n == 0: - return [] - global_max = float(profile.max()) if n else 0.0 - if global_max <= 0.0: - return [] - prom_floor = max(min_prominence_frac, 0.0) * global_max - - maxima = [] - for i in range(n): - v = profile[i] - if v <= 0.0 or v < prom_floor: - continue - left_ok = i == 0 or v > profile[i - 1] - right_ok = i + 1 == n or v >= profile[i + 1] - if left_ok and right_ok: - maxima.append(i) - if not maxima: - return [] - - peaks = [] - bf = max(bound_fraction, 0.0) - for apex in maxima: - apex_v = profile[apex] - thr = bf * apex_v - start = apex - while start > 0: - prev = profile[start - 1] - if prev < thr or prev > profile[start]: - break - start -= 1 - end = apex - while end + 1 < n: - nxt = profile[end + 1] - if nxt < thr or nxt > profile[end]: - break - end += 1 - area = float(profile[start:end + 1].sum()) - peaks.append( - { - "apex_idx": apex, - "start_idx": start, - "end_idx": end, - "apex_intensity": float(apex_v), - "area": area, - "rank": 0, - } - ) - - # Strongest first by area, then apex intensity, then earliest apex. - peaks.sort(key=lambda p: (-p["area"], -p["apex_intensity"], p["apex_idx"])) - kept = [] - for p in peaks: - overlaps = any( - p["apex_idx"] >= q["start_idx"] and p["apex_idx"] <= q["end_idx"] - for q in kept - ) - if not overlaps: - kept.append(p) - if len(kept) == k: - break - for r, p in enumerate(kept): - p["rank"] = r - return kept - - -# --------------------------------------------------------------------------- # -# Consensus elution profile -# --------------------------------------------------------------------------- # -def build_consensus(rows, top_frags, rt_round): - """Sum the top-`top_frags` fragment traces (by predicted_intensity) onto the - union RT axis. `rows` is a list of (predicted_intensity, rt_list, int_list). - - Returns (rt_axis[np.float64], profile[np.float32]); empty arrays if no data. - """ - if not rows: - return np.empty(0), np.empty(0, dtype=np.float32) - # Select strongest predicted fragments; empty traces simply contribute nothing. - rows_sorted = sorted(rows, key=lambda r: -r[0])[:top_frags] - acc = defaultdict(float) - for _predi, rt_list, int_list in rows_sorted: - if not rt_list: - continue - for t, inten in zip(rt_list, int_list): - if inten is None: - continue - acc[round(float(t), rt_round)] += float(inten) - if not acc: - return np.empty(0), np.empty(0, dtype=np.float32) - keys = np.array(sorted(acc.keys()), dtype=np.float64) - prof = np.array([acc[k] for k in keys], dtype=np.float32) - return keys, prof - - -def locate_apex_peak(apex_rt, rt_axis, peaks, rt_tol): - """Return the rank of the peak the given apex_rt belongs to, or None. - - First, peaks whose [start_rt, end_rt] envelope contains apex_rt (nearest by - apex distance wins); otherwise the nearest peak apex within rt_tol. - """ - if not peaks: - return None - containing = [] - for p in peaks: - s_rt = rt_axis[p["start_idx"]] - e_rt = rt_axis[p["end_idx"]] - if s_rt <= apex_rt <= e_rt: - containing.append(p) - if containing: - best = min(containing, key=lambda p: abs(rt_axis[p["apex_idx"]] - apex_rt)) - return best["rank"] - best = min(peaks, key=lambda p: abs(rt_axis[p["apex_idx"]] - apex_rt)) - if abs(rt_axis[best["apex_idx"]] - apex_rt) <= rt_tol: - return best["rank"] - return None - - -# --------------------------------------------------------------------------- # -# DIA-NN reference loading -# --------------------------------------------------------------------------- # -def strip_mods(seq): - if seq is None: - return "" - s = MOD_BRACKET.sub("", str(seq)) - s = MOD_PAREN.sub("", s) - return s.strip().upper() - - -def load_diann(path): - """Load a DIA-NN report (.tsv/.parquet); return dict (stripped_seq, charge)->rt_seconds. - - Column names are matched defensively; RT minutes are detected and converted by - the caller against the MuMDIA RT range. - """ - import pandas as pd - - if path.lower().endswith(".parquet"): - df = pd.read_parquet(path) - else: - df = pd.read_csv(path, sep="\t") - cols = {c.lower(): c for c in df.columns} - - def pick(*cands): - for c in cands: - if c.lower() in cols: - return cols[c.lower()] - return None - - seq_col = pick("Stripped.Sequence", "Modified.Sequence", "Precursor.Id", "Sequence") - chg_col = pick("Precursor.Charge", "Charge") - rt_col = pick("RT", "RT.Start", "Retention.Time", "iRT") - if seq_col is None or chg_col is None or rt_col is None: - raise ValueError( - f"DIA-NN report missing required columns; found {list(df.columns)[:20]}" - ) - out = {} - rts = [] - for seq, chg, rt in zip(df[seq_col], df[chg_col], df[rt_col]): - try: - c = int(chg) - except (ValueError, TypeError): - continue - key = (strip_mods(seq), c) - try: - rtf = float(rt) - except (ValueError, TypeError): - continue - out[key] = rtf - rts.append(rtf) - return out, (min(rts) if rts else 0.0), (max(rts) if rts else 0.0) - - -# --------------------------------------------------------------------------- # -# Main -# --------------------------------------------------------------------------- # -def main(): - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--psms", required=True) - ap.add_argument("--chrom", required=True) - ap.add_argument("--diann", default=None, help="optional DIA-NN report (.tsv/.parquet)") - ap.add_argument("--out", default=None, help="metrics JSON output path") - ap.add_argument("--max-candidates", type=int, default=0, - help="limit to the first N candidates (0 = all)") - ap.add_argument("--top-frags", type=int, default=6, - help="number of strongest predicted fragments summed into the consensus") - ap.add_argument("--bound-fraction", type=float, default=1.0 / 3.0) - ap.add_argument("--min-prominence", type=float, default=0.05) - ap.add_argument("--rt-tol-s", type=float, default=10.0) - ap.add_argument("--rank-by", choices=["area", "count"], default="area", - help="peak ranking: 'area' = integrated intensity (chimeric in DIA); " - "'count' = number of distinct predicted fragments co-eluting in the " - "peak window (breadth of evidence, interference-resistant)") - ap.add_argument("--rt-round", type=int, default=3, help="RT rounding decimals for axis union") - ap.add_argument("--batch-size", type=int, default=100000) - args = ap.parse_args() - - # ---- psms: candidate -> apex_rt, and (stripped_seq,charge) for reference --- - pcols = ["candidate_id", "apex_rt", "label", "peptidoform", "charge"] - ptab = pq.read_table(args.psms, columns=pcols) - cand_ids = ptab.column("candidate_id").to_numpy() - apex_rt = ptab.column("apex_rt").to_numpy() - labels = ptab.column("label").to_pylist() - peptidoforms = ptab.column("peptidoform").to_pylist() - charges = ptab.column("charge").to_numpy() - - order = np.argsort(cand_ids, kind="mergesort") - cand_ids = cand_ids[order] - apex_rt = apex_rt[order] - labels = [labels[i] for i in order] - peptidoforms = [peptidoforms[i] for i in order] - charges = charges[order] - - n_total = len(cand_ids) - n_sel = n_total if args.max_candidates <= 0 else min(args.max_candidates, n_total) - sel_slice = slice(0, n_sel) - sel_ids = cand_ids[sel_slice] - sel_set = set(int(x) for x in sel_ids) - max_sel = int(sel_ids[-1]) if n_sel else -1 - apex_by_cand = {int(c): float(r) for c, r in zip(sel_ids, apex_rt[sel_slice])} - seqkey_by_cand = { - int(c): (strip_mods(pf), int(ch)) - for c, pf, ch in zip(sel_ids, peptidoforms[:n_sel], charges[sel_slice]) - } - mumdia_rt_min = float(np.nanmin(apex_rt[sel_slice])) if n_sel else 0.0 - mumdia_rt_max = float(np.nanmax(apex_rt[sel_slice])) if n_sel else 0.0 - - # ---- DIA-NN reference (optional) ------------------------------------------ - diann = None - if args.diann: - diann_map, drt_min, drt_max = load_diann(args.diann) - # Minutes detection: if the DIA-NN RT span is far below the MuMDIA span - # (seconds), assume minutes and scale by 60. - scale = 1.0 - if drt_max > 0 and mumdia_rt_max > 0 and drt_max <= mumdia_rt_max / 5.0: - scale = 60.0 - diann = {k: v * scale for k, v in diann_map.items()} - print(f"[diann] loaded {len(diann)} precursors, rt span " - f"{drt_min:.1f}-{drt_max:.1f} (scale x{scale:g} -> seconds)") - - # ---- stream chrom, grouped by candidate_id (sorted) ----------------------- - pf = pq.ParquetFile(args.chrom) - ccols = ["candidate_id", "predicted_intensity", "rt", "intensity"] - - # accumulators - ranks = [] # peak rank the self apex fell into (int) - n_no_peak = 0 # self apex matched no enumerated peak - n_no_chrom = 0 # selected candidate absent from chrom / empty profile - peaks_per_cand = [] # peak count per processed candidate - processed = set() - ref_hits = {1: 0, 3: 0, 5: 0, 10: 0} - ref_matched = 0 # candidates matched to a DIA-NN precursor with a profile - - def process_candidate(cid, rows): - nonlocal n_no_peak, ref_matched - rt_axis, prof = build_consensus(rows, args.top_frags, args.rt_round) - peaks = enumerate_peaks(prof, len(prof) if len(prof) else 0, - args.bound_fraction, args.min_prominence) - # Evidence-count ranking (interference-resistant): rank each peak by the - # number of DISTINCT predicted fragments with any nonzero intensity inside - # its RT window, not by integrated intensity. In DIA the tallest peak is - # often a co-isolated interferent; the true peak is the one where the most - # of the peptide's own predicted transitions co-elute. Uses ALL fragment - # rows (breadth), not just the top-N summed into the detection profile. - if args.rank_by == "count" and peaks: - for p in peaks: - lo, hi = rt_axis[p["start_idx"]], rt_axis[p["end_idx"]] - ev = 0 - for _predi, rt_list, int_list in rows: - present = any( - (lo <= float(rt) <= hi) and (float(it) > 0.0) - for rt, it in zip(rt_list, int_list) - ) - if present: - ev += 1 - p["evidence"] = ev - peaks.sort(key=lambda p: (-p.get("evidence", 0), -p["area"], p["apex_idx"])) - for r, p in enumerate(peaks): - p["rank"] = r - peaks_per_cand.append(len(peaks)) - processed.add(cid) - # SELF - a_rt = apex_by_cand.get(cid) - if a_rt is not None: - rank = locate_apex_peak(a_rt, rt_axis, peaks, args.rt_tol_s) - if rank is None: - n_no_peak += 1 - else: - ranks.append(rank) - # REFERENCE - if diann is not None and peaks: - key = seqkey_by_cand.get(cid) - d_rt = diann.get(key) if key else None - if d_rt is not None: - ref_matched += 1 - apex_rts = [rt_axis[p["apex_idx"]] for p in peaks] # rank-ordered - for K in (1, 3, 5, 10): - topk = apex_rts[:K] - if any(abs(d_rt - ar) <= args.rt_tol_s for ar in topk): - ref_hits[K] += 1 - - cur_cid = None - cur_rows = [] - stop = False - for batch in pf.iter_batches(batch_size=args.batch_size, columns=ccols): - cids = batch.column("candidate_id").to_numpy() - if len(cids) == 0: - continue - if int(cids[0]) > max_sel: - break # sorted; nothing selected remains - predi = batch.column("predicted_intensity").to_numpy() - rt_lists = batch.column("rt").to_pylist() - int_lists = batch.column("intensity").to_pylist() - for j in range(len(cids)): - cid = int(cids[j]) - if cid != cur_cid: - if cur_cid is not None and cur_cid in sel_set: - process_candidate(cur_cid, cur_rows) - if cur_cid is not None and cur_cid > max_sel: - stop = True - break - cur_cid = cid - cur_rows = [] - if cid in sel_set: - cur_rows.append((float(predi[j]), rt_lists[j], int_lists[j])) - if stop: - break - # flush last - if not stop and cur_cid is not None and cur_cid in sel_set: - process_candidate(cur_cid, cur_rows) - - # selected candidates never seen in chrom - n_no_chrom = n_sel - len(processed) - - # ---- metrics -------------------------------------------------------------- - ranks_arr = np.array(ranks, dtype=int) - # denominator: all selected candidates that had a psms apex_rt (all of them) - denom = n_sel - n_rank1 = int((ranks_arr == 0).sum()) - n_top3 = int((ranks_arr < 3).sum()) - n_top5 = int((ranks_arr < 5).sum()) - n_top10 = int((ranks_arr < 10).sum()) - # "no enumerated peak" = matched to no peak OR no chrom/empty profile at all - n_no_peak_total = n_no_peak + n_no_chrom - ppc = np.array(peaks_per_cand, dtype=float) - - metrics = { - "inputs": { - "psms": os.path.abspath(args.psms), - "chrom": os.path.abspath(args.chrom), - "diann": os.path.abspath(args.diann) if args.diann else None, - }, - "params": { - "max_candidates": args.max_candidates, - "top_frags": args.top_frags, - "bound_fraction": args.bound_fraction, - "min_prominence": args.min_prominence, - "rt_tol_s": args.rt_tol_s, - }, - "n_candidates_total": int(n_total), - "n_candidates_selected": int(n_sel), - "n_candidates_processed": int(len(processed)), - "n_no_chrom_or_empty": int(n_no_chrom), - "self": { - "denominator": int(denom), - "n_apex_matched_to_peak": int(len(ranks_arr)), - "frac_rank1": n_rank1 / denom if denom else 0.0, - "frac_top3": n_top3 / denom if denom else 0.0, - "frac_top5": n_top5 / denom if denom else 0.0, - "frac_top10": n_top10 / denom if denom else 0.0, - "frac_no_peak": n_no_peak_total / denom if denom else 0.0, - "mean_peaks_per_candidate": float(ppc.mean()) if ppc.size else 0.0, - "median_peaks_per_candidate": float(np.median(ppc)) if ppc.size else 0.0, - "frac_ge2_peaks": float((ppc >= 2).mean()) if ppc.size else 0.0, - }, - } - if diann is not None: - metrics["reference"] = { - "n_reference_matched": int(ref_matched), - "reference_apex_in_top_1": ref_hits[1] / ref_matched if ref_matched else 0.0, - "reference_apex_in_top_3": ref_hits[3] / ref_matched if ref_matched else 0.0, - "reference_apex_in_top_5": ref_hits[5] / ref_matched if ref_matched else 0.0, - "reference_apex_in_top_10": ref_hits[10] / ref_matched if ref_matched else 0.0, - } - - # ---- report --------------------------------------------------------------- - s = metrics["self"] - print("\n=== Top-K peak oracle (SELF) ===") - print(f"selected candidates : {n_sel} (processed {len(processed)}, " - f"no chrom/empty {n_no_chrom})") - print(f"mean peaks / candidate : {s['mean_peaks_per_candidate']:.2f} " - f"(median {s['median_peaks_per_candidate']:.0f})") - print(f"candidates with >=2 peaks: {s['frac_ge2_peaks']*100:.1f}% " - f"(the top-K opportunity size)") - print(f"chosen apex is peak rank1: {s['frac_rank1']*100:.1f}%") - print(f" within top-3 : {s['frac_top3']*100:.1f}%") - print(f" within top-5 : {s['frac_top5']*100:.1f}%") - print(f" within top-10 : {s['frac_top10']*100:.1f}%") - print(f" in NO peak : {s['frac_no_peak']*100:.1f}%") - if diann is not None: - r = metrics["reference"] - print("\n=== Reference-apex recall (DIA-NN) ===") - print(f"matched precursors : {r['n_reference_matched']}") - for K in (1, 3, 5, 10): - print(f"DIA-NN apex in top-{K:<2} : " - f"{r[f'reference_apex_in_top_{K}']*100:.1f}%") - - out = args.out or (os.path.splitext(args.psms)[0] + ".topk_oracle.json") - with open(out, "w") as fh: - json.dump(metrics, fh, indent=2) - print(f"\n[written] {os.path.abspath(out)}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/rt_anchor.py b/scripts/rt_anchor.py deleted file mode 100644 index 60cec67..0000000 --- a/scripts/rt_anchor.py +++ /dev/null @@ -1,122 +0,0 @@ -"""Build clean RT training anchors for DeepLC fine-tuning. - -The seed search's observed_rt can sit on a bright interfering peak (the wrong-apex -problem that dominates GATE_LOSS). This pollutes the fine-tune training set. Here we -recompute, for each confident seed target peptide, a shape-based apex: take the peptide's -3 most intense predicted library fragments, sum their observed intensity across all MS2 -scans whose isolation window covers the precursor, and take the RT of the maximum of that -summed extracted-ion chromatogram. That apex is a far cleaner observed RT for training. - -Output: a seed-shaped parquet (peptidoform,label,spectrum_q,observed_rt=refined apex, +QC) -consumable by deeplc_finetune.py unchanged. - -Usage: python rt_anchor.py -""" -import json -import sys -import numpy as np -import pandas as pd - -TOP_N = 3 - - -def main(): - seed_p, frag_p, ms2_p, masscal_p, out_p = sys.argv[1:6] - with open(masscal_p) as fh: - mc = json.load(fh) - off_ppm = float(mc.get("frag_ppm_offset", 0.0)) - tol_ppm = float(mc.get("frag_tol_ppm", 20.0)) - - seed = pd.read_parquet(seed_p, columns=["candidate_id", "peptidoform", "charge", - "precursor_mz", "label", "spectrum_q", "observed_rt"]) - conf = seed[(seed.label == "target") & (seed.spectrum_q <= 0.01)].copy() - conf = conf.drop_duplicates("candidate_id").reset_index(drop=True) - cand_ids = set(conf.candidate_id) - print(f"confident seed targets: {len(conf)}", flush=True) - - # top-N predicted fragments per confident candidate (m/z, offset-corrected) - frag = pd.read_parquet(frag_p, columns=["candidate_id", "mz", "predicted_intensity"]) - frag = frag[frag.candidate_id.isin(cand_ids)] - frag = frag.sort_values(["candidate_id", "predicted_intensity"], ascending=[True, False]) - top = frag.groupby("candidate_id").head(TOP_N) - cand2frags = {c: g.mz.to_numpy() * (1.0 + off_ppm / 1e6) for c, g in top.groupby("candidate_id")} - - # Flat per-candidate fragment array padded to TOP_N (NaN = absent), in conf row order - cid_arr = conf.candidate_id.to_numpy() - F = np.full((len(conf), TOP_N), np.nan) - for i, cid in enumerate(cid_arr): - fm = cand2frags.get(cid) - if fm is not None: - F[i, :fm.size] = fm[:TOP_N] - prec = conf.precursor_mz.to_numpy() - - ms2 = pd.read_parquet(ms2_p, columns=["rt_seconds", "window_lower", "window_upper", "mz", "intensity"]) - wl = ms2.window_lower.to_numpy(); wu = ms2.window_upper.to_numpy() - rt = ms2.rt_seconds.to_numpy() - mz_lists = ms2.mz.to_list(); int_lists = ms2.intensity.to_list() - n_scan = len(ms2) - - apex_rt = np.full(len(conf), np.nan) - apex_int = np.full(len(conf), -1.0) - n_present = np.zeros(len(conf), dtype=int) - - # unique isolation windows -> scan indices - uniq = {} - for i in range(n_scan): - uniq.setdefault((wl[i], wu[i]), []).append(i) - print(f"{len(uniq)} unique isolation windows, {n_scan} MS2 scans", flush=True) - - for wk, (win, scans) in enumerate(uniq.items()): - lo, hi = win - cmask = np.where((prec >= lo) & (prec <= hi))[0] - if cmask.size == 0: - continue - Fw = F[cmask] # (nc, TOP_N) - nc = cmask.size - qmz = Fw.reshape(-1) # (nc*TOP_N,) - valid = ~np.isnan(qmz) - qv = np.where(valid, qmz, 0.0) - tol = qv * tol_ppm / 1e6 - qlo = qv - tol; qhi = qv + tol - best_sum = np.full(nc, -1.0) - best_rt = np.full(nc, np.nan) - best_np = np.zeros(nc, dtype=int) - for si in scans: - m = np.asarray(mz_lists[si], dtype=np.float64) - if m.size == 0: - continue - it = np.asarray(int_lists[si], dtype=np.float64) - if not np.all(np.diff(m) >= 0): - o = np.argsort(m); m = m[o]; it = it[o] - a = np.searchsorted(m, qlo); b = np.searchsorted(m, qhi) - present = (b > a) & valid - idx = np.clip(a, 0, m.size - 1) - val = np.where(present, it[idx], 0.0) - val = val.reshape(nc, TOP_N) - ssum = val.sum(axis=1) - spres = present.reshape(nc, TOP_N).sum(axis=1) - upd = ssum > best_sum - best_sum = np.where(upd, ssum, best_sum) - best_rt = np.where(upd, rt[si], best_rt) - best_np = np.where(upd, spres, best_np) - # merge into globals (candidate may appear in overlapping windows -> keep max) - for j, ci in enumerate(cmask): - if best_sum[j] > apex_int[ci]: - apex_int[ci] = best_sum[j]; apex_rt[ci] = best_rt[j]; n_present[ci] = best_np[j] - if (wk + 1) % 20 == 0: - print(f" window {wk+1}/{len(uniq)}", flush=True) - - conf["observed_rt_seed"] = conf.observed_rt - conf["observed_rt"] = apex_rt - conf["apex_top3_intensity"] = np.maximum(apex_int, 0.0) - conf["n_top3_present"] = n_present - good = conf[conf.observed_rt.notna() & (conf.n_top3_present >= 2)].copy() - print(f"anchors with >=2 of top-3 present: {len(good)} / {len(conf)}", flush=True) - good.to_parquet(out_p, index=False) - d = (good.observed_rt - good.observed_rt_seed).abs() - print(f"|refined - seed| RT: median {d.median():.1f}s p90 {d.quantile(.9):.1f}s >60s: {100*(d>60).mean():.1f}%") - print("wrote", out_p) - - -if __name__ == "__main__": - main() diff --git a/scripts/rt_anchor_extract.py b/scripts/rt_anchor_extract.py deleted file mode 100644 index 4f48cb9..0000000 --- a/scripts/rt_anchor_extract.py +++ /dev/null @@ -1,136 +0,0 @@ -"""Build clean RT training anchors for DeepLC fine-tuning, using the extract.rs apex rule. - -Unlike rt_anchor.py (which takes the RT of the single scan with max summed top-3 fragment -intensity), this mirrors the Rust extractor's apex (extract.rs:441-464): over ALL predicted -fragments, group by scan, and among scans whose DISTINCT-matched-fragment count is within -`apex_count_tol` of the per-peptide maximum, take the scan maximizing the summed intensity of -its 3 most intense fragments. The count gate makes the apex robust to a lone bright interfering -fragment that would win a pure max-intensity apex on chimeric DIA. - -Output: seed-shaped parquet (peptidoform,label,spectrum_q,observed_rt=apex, +QC), drop-in for -deeplc_finetune.py. - -Usage: python rt_anchor_extract.py - [--apex-count-tol 1] [--min-matched 2] -""" -import argparse -import json -import numpy as np -import pandas as pd - -TOP_N = 3 # top fragments summed for the intensity tiebreak (matches extract.rs top-3) - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("seed_p"); ap.add_argument("frag_p"); ap.add_argument("ms2_p") - ap.add_argument("masscal_p"); ap.add_argument("out_p") - ap.add_argument("--apex-count-tol", type=int, default=1) # extract.rs default - ap.add_argument("--min-matched", type=int, default=2) # keep anchor if apex has >= this many frags - args = ap.parse_args() - - with open(args.masscal_p) as fh: - mc = json.load(fh) - off_ppm = float(mc.get("frag_ppm_offset", 0.0)) - tol_ppm = float(mc.get("frag_tol_ppm", 20.0)) - - seed = pd.read_parquet(args.seed_p, columns=["candidate_id", "peptidoform", "charge", - "precursor_mz", "label", "spectrum_q", "observed_rt"]) - conf = seed[(seed.label == "target") & (seed.spectrum_q <= 0.01)].copy() - conf = conf.drop_duplicates("candidate_id").reset_index(drop=True) - cand_ids = set(conf.candidate_id) - print(f"confident seed targets: {len(conf)}", flush=True) - - # ALL predicted fragments per confident candidate (offset-corrected m/z) - frag = pd.read_parquet(args.frag_p, columns=["candidate_id", "mz", "predicted_intensity"]) - frag = frag[frag.candidate_id.isin(cand_ids)] - frag = frag.sort_values(["candidate_id", "predicted_intensity"], ascending=[True, False]) - cand2mz = {c: g.mz.to_numpy() * (1.0 + off_ppm / 1e6) for c, g in frag.groupby("candidate_id")} - MAXF = min(max((v.size for v in cand2mz.values()), default=0), 40) - print(f"max predicted fragments/candidate used: {MAXF}", flush=True) - - cid_arr = conf.candidate_id.to_numpy() - F = np.full((len(conf), MAXF), np.nan) - for i, cid in enumerate(cid_arr): - fm = cand2mz.get(cid) - if fm is not None: - F[i, :min(fm.size, MAXF)] = fm[:MAXF] - prec = conf.precursor_mz.to_numpy() - - ms2 = pd.read_parquet(args.ms2_p, columns=["rt_seconds", "window_lower", "window_upper", "mz", "intensity"]) - wl = ms2.window_lower.to_numpy(); wu = ms2.window_upper.to_numpy(); rt = ms2.rt_seconds.to_numpy() - mz_lists = ms2.mz.to_list(); int_lists = ms2.intensity.to_list() - n_scan = len(ms2) - - apex_rt = np.full(len(conf), np.nan) - apex_int = np.full(len(conf), -1.0) # summed top-3 at apex (cross-window selection key) - apex_np = np.zeros(len(conf), dtype=int) # distinct frags present at apex - - uniq = {} - for i in range(n_scan): - uniq.setdefault((wl[i], wu[i]), []).append(i) - print(f"{len(uniq)} unique isolation windows, {n_scan} MS2 scans", flush=True) - - for wk, (win, scans) in enumerate(uniq.items()): - lo, hi = win - cmask = np.where((prec >= lo) & (prec <= hi))[0] - if cmask.size == 0: - continue - Fw = F[cmask]; nc = cmask.size - qmz = Fw.reshape(-1); valid = ~np.isnan(qmz) - qv = np.where(valid, qmz, 0.0); tol = qv * tol_ppm / 1e6 - qlo = qv - tol; qhi = qv + tol - S = len(scans) - Count = np.zeros((nc, S), dtype=np.int16) - Top3 = np.zeros((nc, S), dtype=np.float64) - rtw = np.empty(S) - for k, si in enumerate(scans): - rtw[k] = rt[si] - m = np.asarray(mz_lists[si], dtype=np.float64) - if m.size == 0: - continue - it = np.asarray(int_lists[si], dtype=np.float64) - if not np.all(np.diff(m) >= 0): - o = np.argsort(m); m = m[o]; it = it[o] - a = np.searchsorted(m, qlo); b = np.searchsorted(m, qhi) - present = (b > a) & valid - idx = np.clip(a, 0, m.size - 1) - val = np.where(present, it[idx], 0.0).reshape(nc, MAXF) - Count[:, k] = present.reshape(nc, MAXF).sum(axis=1) - # sum of 3 largest per row - if MAXF > TOP_N: - part = np.partition(val, MAXF - TOP_N, axis=1)[:, -TOP_N:] - Top3[:, k] = part.sum(axis=1) - else: - Top3[:, k] = val.sum(axis=1) - # per-candidate apex via extract.rs count gate, then top-3 intensity - for j in range(nc): - cnt = Count[j]; maxc = int(cnt.max()) - if maxc < 1: - continue - thresh = max(1, maxc - args.apex_count_tol) - elig = cnt >= thresh - if not elig.any(): - continue - t3 = np.where(elig, Top3[j], -1.0) - kbest = int(np.argmax(t3)) - ci = cmask[j] - if t3[kbest] > apex_int[ci]: # cross-window: keep the stronger apex - apex_int[ci] = t3[kbest]; apex_rt[ci] = rtw[kbest]; apex_np[ci] = int(cnt[kbest]) - if (wk + 1) % 20 == 0: - print(f" window {wk+1}/{len(uniq)}", flush=True) - - conf["observed_rt_seed"] = conf.observed_rt - conf["observed_rt"] = apex_rt - conf["apex_top3_intensity"] = np.maximum(apex_int, 0.0) - conf["n_matched_at_apex"] = apex_np - good = conf[conf.observed_rt.notna() & (conf.n_matched_at_apex >= args.min_matched)].copy() - print(f"anchors with >={args.min_matched} matched frags at apex: {len(good)} / {len(conf)}", flush=True) - good.to_parquet(args.out_p, index=False) - d = (good.observed_rt - good.observed_rt_seed).abs() - print(f"|apex - seed| RT: median {d.median():.1f}s p90 {d.quantile(.9):.1f}s >60s: {100*(d>60).mean():.1f}%") - print("wrote", args.out_p) - - -if __name__ == "__main__": - main() diff --git a/scripts/search_space_manifest.py b/scripts/search_space_manifest.py deleted file mode 100644 index d85ee32..0000000 --- a/scripts/search_space_manifest.py +++ /dev/null @@ -1,564 +0,0 @@ -"""Derive an effective search-space manifest from a MuMDIA library. - -Sensitivity-plan backlog item P0.1 (search-space parity). The script is -non-invasive: it reads an existing MuMDIA library-precursors Parquet, derives a -machine-readable manifest in the shape of sensitivity_plan spec 02 section 3, -and optionally compares that manifest to a declared manifest (--compare) or a -DIA-NN report (--diann). With --fail-on-mismatch it exits nonzero when the -effective search spaces differ in a way that would invalidate a benchmark -comparison. - -Only fields that a precursor library actually encodes are derived. Fields that a -library cannot encode (enzyme, missed cleavages, fragment m/z window, FDR -thresholds, fixed-vs-variable mod classification) are written as the literal -"unknown_from_library" with an explanatory note, so a reader never mistakes an -absent value for a real setting. - -The output is deterministic: no wall-clock timestamps, sorted keys where order is -not semantically meaningful, and a SHA-256 content hash of the input library for -provenance. - -Interpreter: C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe -(pyarrow, pandas, numpy, pyyaml, stdlib tomllib/hashlib). - -Example -------- -python search_space_manifest.py \ - --library-precursors C:/proteobench/lib/lib_precursors_ft.parquet \ - --out C:/proteobench/accept/ecoli_search_space.yaml -""" - -from __future__ import annotations - -import argparse -import hashlib -import re -import sys -import tomllib -from pathlib import Path -from typing import Any - -import numpy as np -import pyarrow as pa -import pyarrow.compute as pc -import pyarrow.parquet as pq -import yaml - -UNKNOWN = "unknown_from_library" - -# Bracketed UniMod-style modification token, e.g. "[Carbamidomethyl]". -_MOD_RE = re.compile(r"\[[^\]]*\]") -_MOD_TOKEN_RE = re.compile(r"\[([^\]]*)\]") -_DECOY_RE = re.compile(r"^DECOY_") - - -# --------------------------------------------------------------------------- # -# Small shared helpers -# --------------------------------------------------------------------------- # -def sha256_file(path: Path, chunk: int = 1 << 20) -> str: - """Return the SHA-256 hex digest of a file, read in fixed-size chunks.""" - h = hashlib.sha256() - with path.open("rb") as fh: - for block in iter(lambda: fh.read(chunk), b""): - h.update(block) - return h.hexdigest() - - -def resolve_engine_version(script_path: Path) -> str: - """Resolve the MuMDIA engine version from the Cargo manifests. - - mumdia-core declares ``version.workspace = true``, so the literal version - lives in the workspace root manifest. Falls back to "unknown". - """ - repo_root = script_path.resolve().parents[1] - core = repo_root / "rust" / "mumdia" / "crates" / "mumdia-core" / "Cargo.toml" - workspace = repo_root / "rust" / "mumdia" / "Cargo.toml" - try: - with core.open("rb") as fh: - core_cfg = tomllib.load(fh) - ver = core_cfg.get("package", {}).get("version") - if isinstance(ver, str): - return ver - with workspace.open("rb") as fh: - ws_cfg = tomllib.load(fh) - ver = ws_cfg.get("workspace", {}).get("package", {}).get("version") - if isinstance(ver, str): - return ver - except (OSError, tomllib.TOMLDecodeError): - pass - return "unknown" - - -def _norm_col(name: str) -> str: - """Normalize a column name to lowercase alphanumerics for matching.""" - return re.sub(r"[^a-z0-9]", "", name.lower()) - - -def _find_col(columns: list[str], candidates: list[str]) -> str | None: - """Return the first column whose normalized name matches a candidate.""" - norm = {_norm_col(c): c for c in columns} - for cand in candidates: - if cand in norm: - return norm[cand] - return None - - -# --------------------------------------------------------------------------- # -# Manifest derivation -# --------------------------------------------------------------------------- # -def derive_manifest(lib_path: Path, engine_version: str) -> dict[str, Any]: - """Derive an effective search-space manifest from a library-precursors table.""" - needed = ["peptidoform", "charge", "precursor_mz", "label", "protein"] - schema_names = pq.read_schema(lib_path).names - read_cols = [c for c in needed if c in schema_names] - tbl = pq.read_table(lib_path, columns=read_cols) - n_rows = tbl.num_rows - - # Charge and m/z ranges are the same for targets and decoys, so compute - # them over all precursors for a faithful picture of the extraction space. - charge_arr = tbl.column("charge") - mz_arr = tbl.column("precursor_mz") - charge_mm = pc.min_max(charge_arr) - mz_mm = pc.min_max(mz_arr) - min_charge = charge_mm["min"].as_py() - max_charge = charge_mm["max"].as_py() - min_mz = mz_mm["min"].as_py() - max_mz = mz_mm["max"].as_py() - - # Charge histogram over all precursors, sorted by charge. - vc = pc.value_counts(charge_arr) - charge_hist = { - int(struct["values"].as_py()): int(struct["counts"].as_py()) for struct in vc - } - charge_hist = dict(sorted(charge_hist.items())) - - # Target / decoy split. - label_arr = tbl.column("label") - is_target = pc.equal(label_arr, "target") - n_target = int(pc.sum(pc.cast(is_target, pa.int64())).as_py()) - n_decoy = n_rows - n_target - - # String-level statistics are computed over the target subset (the real - # search space); decoys are artificial reverse/scramble sequences. - targets = tbl.filter(is_target).select( - [c for c in ["peptidoform", "protein"] if c in read_cols] - ) - tdf = targets.to_pandas() - pforms = tdf["peptidoform"].astype("string") - - stripped = pforms.str.replace(_MOD_RE, "", regex=True).str.replace( - _DECOY_RE, "", regex=True - ) - lengths = stripped.str.len().to_numpy(dtype="int64") - n_distinct_stripped = int(stripped.nunique()) - - has_mod = pforms.str.contains("[", regex=False) - n_with_mod = int(has_mod.sum()) - observed_mods: set[str] = set() - for pf in pforms[has_mod]: - observed_mods.update(_MOD_TOKEN_RE.findall(pf)) - observed_mods_sorted = sorted(observed_mods) - - # Distinct proteins over targets, splitting group strings on ';'. - prot_series = tdf["protein"].astype("string") if "protein" in tdf else None - if prot_series is not None: - prot_set: set[str] = set() - for grp in prot_series.dropna().unique(): - for pid in str(grp).split(";"): - pid = pid.strip() - if pid: - prot_set.add(pid) - n_distinct_proteins: int | str = len(prot_set) - else: - n_distinct_proteins = UNKNOWN - - length_note = ( - "peptide lengths derived from stripped target sequences; enzyme, " - "specificity, and missed cleavages are search-time settings not stored " - "in a precursor library" - ) - - manifest: dict[str, Any] = { - "schema_version": 1, - "provenance": { - "source_library": str(lib_path), - "source_sha256": sha256_file(lib_path), - "source_bytes": lib_path.stat().st_size, - "n_rows": n_rows, - "engine": "mumdia", - "engine_version": engine_version, - "generator": "search_space_manifest.py", - "manifest_schema_version": 1, - }, - "digestion": { - "enzyme": UNKNOWN, - "specificity": UNKNOWN, - "missed_cleavages": UNKNOWN, - "min_length": int(lengths.min()), - "max_length": int(lengths.max()), - "note": length_note, - }, - "precursors": { - "min_charge": int(min_charge), - "max_charge": int(max_charge), - "min_mz": round(float(min_mz), 4), - "max_mz": round(float(max_mz), 4), - }, - "fragments": { - "ion_series": UNKNOWN, - "min_mz": UNKNOWN, - "max_mz": UNKNOWN, - "note": "fragment ion series and m/z window require the fragment " - "table (lib_fragments), not provided here", - }, - "modifications": { - "fixed": UNKNOWN, - "variable": UNKNOWN, - "max_variable_modifications": UNKNOWN, - "observed_mod_tokens": observed_mods_sorted, - "n_peptidoforms_with_mod": n_with_mod, - "note": "fixed vs variable classification cannot be inferred from a " - "library; observed bracketed tokens are listed as-is", - }, - "fdr": { - "precursor": UNKNOWN, - "peptide": UNKNOWN, - "protein": UNKNOWN, - "note": "FDR thresholds are a search-time setting, not stored in the " - "library", - }, - "derived": { - "n_precursors": n_rows, - "n_target_precursors": n_target, - "n_decoy_precursors": n_decoy, - "n_distinct_stripped_sequences": n_distinct_stripped, - "n_distinct_proteins": n_distinct_proteins, - "charge_histogram": charge_hist, - "peptide_length": { - "min": int(lengths.min()), - "max": int(lengths.max()), - "median": float(np.median(lengths)), - }, - }, - } - return manifest - - -# --------------------------------------------------------------------------- # -# MuMDIA target precursor keys (for --diann overlap) -# --------------------------------------------------------------------------- # -def mumdia_target_keys(lib_path: Path) -> tuple[set[tuple[str, int]], dict[str, Any]]: - """Return {(stripped_seq_upper, charge)} for target precursors plus ranges.""" - tbl = pq.read_table( - lib_path, columns=["peptidoform", "charge", "precursor_mz", "label"] - ) - is_target = pc.equal(tbl.column("label"), "target") - tdf = tbl.filter(is_target).to_pandas() - stripped = ( - tdf["peptidoform"] - .astype("string") - .str.replace(_MOD_RE, "", regex=True) - .str.replace(_DECOY_RE, "", regex=True) - .str.upper() - ) - charges = tdf["charge"].astype("int64") - keys = set(zip(stripped.tolist(), charges.tolist())) - ranges = { - "charge_range": [int(charges.min()), int(charges.max())], - "mz_range": [ - round(float(tdf["precursor_mz"].min()), 4), - round(float(tdf["precursor_mz"].max()), 4), - ], - } - return keys, ranges - - -# --------------------------------------------------------------------------- # -# Reference (DIA-NN) loading -# --------------------------------------------------------------------------- # -def load_reference_keys(path: Path) -> tuple[set[tuple[str, int]], dict[str, Any]]: - """Load (stripped_seq_upper, charge) keys and ranges from a DIA-NN report. - - Accepts Parquet or a tab-separated report. Column names are matched - case-insensitively against the usual DIA-NN header set. - """ - suffix = path.suffix.lower() - if suffix in {".parquet", ".pq"}: - cols = pq.read_schema(path).names - else: - import pandas as pd - - cols = list(pd.read_csv(path, sep="\t", nrows=0).columns) - - stripped_col = _find_col( - cols, ["strippedsequence", "peptide", "sequence", "pepseq"] - ) - charge_col = _find_col(cols, ["precursorcharge", "charge", "z"]) - mz_col = _find_col(cols, ["precursormz", "mz"]) - if stripped_col is None or charge_col is None: - raise ValueError( - f"could not locate stripped-sequence and charge columns in {path}; " - f"available columns: {cols}" - ) - - read_cols = [stripped_col, charge_col] + ([mz_col] if mz_col else []) - if suffix in {".parquet", ".pq"}: - df = pq.read_table(path, columns=read_cols).to_pandas() - else: - import pandas as pd - - df = pd.read_csv(path, sep="\t", usecols=read_cols) - - df = df.dropna(subset=[stripped_col, charge_col]) - stripped = df[stripped_col].astype("string").str.upper() - charges = df[charge_col].astype("int64") - keys = set(zip(stripped.tolist(), charges.tolist())) - ranges: dict[str, Any] = { - "charge_range": [int(charges.min()), int(charges.max())], - } - if mz_col: - ranges["mz_range"] = [ - round(float(df[mz_col].min()), 4), - round(float(df[mz_col].max()), 4), - ] - return keys, ranges - - -# --------------------------------------------------------------------------- # -# Comparison logic -# --------------------------------------------------------------------------- # -def compare_overlap( - mumdia_keys: set[tuple[str, int]], - mumdia_ranges: dict[str, Any], - ref_keys: set[tuple[str, int]], - ref_ranges: dict[str, Any], - ref_label: str, - mz_tol: float, -) -> dict[str, Any]: - """Compute (stripped_seq, charge) precursor-key overlap and range mismatches.""" - shared = mumdia_keys & ref_keys - only_mum = mumdia_keys - ref_keys - only_ref = ref_keys - mumdia_keys - n_ref = len(ref_keys) - n_mum = len(mumdia_keys) - - charge_mismatch = mumdia_ranges["charge_range"] != ref_ranges.get("charge_range") - mz_mismatch = False - if "mz_range" in ref_ranges and "mz_range" in mumdia_ranges: - m = mumdia_ranges["mz_range"] - r = ref_ranges["mz_range"] - mz_mismatch = abs(m[0] - r[0]) > mz_tol or abs(m[1] - r[1]) > mz_tol - - return { - "reference": ref_label, - "n_mumdia_target_keys": n_mum, - "n_reference_keys": n_ref, - "n_shared": len(shared), - "n_only_in_mumdia": len(only_mum), - "n_only_in_reference": len(only_ref), - "shared_fraction_of_reference": (len(shared) / n_ref) if n_ref else 0.0, - "shared_fraction_of_mumdia": (len(shared) / n_mum) if n_mum else 0.0, - "charge_range_mumdia": mumdia_ranges["charge_range"], - "charge_range_reference": ref_ranges.get("charge_range"), - "mz_range_mumdia": mumdia_ranges.get("mz_range"), - "mz_range_reference": ref_ranges.get("mz_range"), - "charge_range_mismatch": charge_mismatch, - "mz_range_mismatch": mz_mismatch, - } - - -def compare_declared( - derived: dict[str, Any], other: dict[str, Any], mz_tol: float -) -> dict[str, Any]: - """Compare a derived manifest against a declared manifest, field by field.""" - diffs: list[dict[str, Any]] = [] - - def cmp(section: str, field: str, material_if_diff: bool, tol: float = 0.0) -> None: - a = derived.get(section, {}).get(field) - b = other.get(section, {}).get(field) - if a is None or b is None: - return - if a == UNKNOWN or b == UNKNOWN: - diffs.append( - { - "field": f"{section}.{field}", - "derived": a, - "declared": b, - "status": "not_comparable", - "material": False, - } - ) - return - if isinstance(a, (int, float)) and isinstance(b, (int, float)) and tol > 0.0: - differs = abs(float(a) - float(b)) > tol - else: - differs = a != b - diffs.append( - { - "field": f"{section}.{field}", - "derived": a, - "declared": b, - "status": "mismatch" if differs else "match", - "material": bool(differs and material_if_diff), - } - ) - - cmp("digestion", "enzyme", True) - cmp("digestion", "specificity", True) - cmp("digestion", "missed_cleavages", True) - cmp("digestion", "min_length", True) - cmp("digestion", "max_length", True) - cmp("precursors", "min_charge", True) - cmp("precursors", "max_charge", True) - cmp("precursors", "min_mz", True, tol=mz_tol) - cmp("precursors", "max_mz", True, tol=mz_tol) - cmp("modifications", "max_variable_modifications", True) - cmp("fdr", "precursor", True) - cmp("fdr", "peptide", True) - cmp("fdr", "protein", True) - - return { - "reference": "declared_manifest", - "fields": diffs, - "n_material_mismatches": sum(1 for d in diffs if d["material"]), - } - - -# --------------------------------------------------------------------------- # -# Reporting -# --------------------------------------------------------------------------- # -def print_overlap(cmp: dict[str, Any], min_shared_frac: float) -> bool: - """Print an overlap diff. Return True if it constitutes a material mismatch.""" - print(f"\n=== search-space overlap vs {cmp['reference']} ===") - print(f" MuMDIA target keys : {cmp['n_mumdia_target_keys']}") - print(f" reference keys : {cmp['n_reference_keys']}") - print(f" shared : {cmp['n_shared']}") - print(f" only in MuMDIA : {cmp['n_only_in_mumdia']}") - print(f" only in reference : {cmp['n_only_in_reference']}") - print( - f" shared / reference : {cmp['shared_fraction_of_reference']:.4f} " - f"(threshold {min_shared_frac:.4f})" - ) - print(f" shared / MuMDIA : {cmp['shared_fraction_of_mumdia']:.4f}") - print( - f" charge range : MuMDIA {cmp['charge_range_mumdia']} vs " - f"reference {cmp['charge_range_reference']}" - + (" MISMATCH" if cmp["charge_range_mismatch"] else "") - ) - print( - f" m/z range : MuMDIA {cmp['mz_range_mumdia']} vs " - f"reference {cmp['mz_range_reference']}" - + (" MISMATCH" if cmp["mz_range_mismatch"] else "") - ) - low_overlap = cmp["shared_fraction_of_reference"] < min_shared_frac - if low_overlap: - print(" -> shared fraction below threshold") - return low_overlap or cmp["charge_range_mismatch"] or cmp["mz_range_mismatch"] - - -def print_declared(cmp: dict[str, Any]) -> bool: - """Print a declared-manifest diff. Return True if any material mismatch.""" - print("\n=== declared-manifest comparison ===") - for d in cmp["fields"]: - flag = { - "match": "ok", - "mismatch": "MISMATCH", - "not_comparable": "n/a", - }[d["status"]] - material = " (material)" if d["material"] else "" - print( - f" {d['field']:<38} derived={d['derived']!r:<24} " - f"declared={d['declared']!r:<24} {flag}{material}" - ) - print(f" material mismatches: {cmp['n_material_mismatches']}") - return cmp["n_material_mismatches"] > 0 - - -# --------------------------------------------------------------------------- # -# CLI -# --------------------------------------------------------------------------- # -def parse_args(argv: list[str] | None = None) -> argparse.Namespace: - p = argparse.ArgumentParser( - description="Derive and validate a MuMDIA search-space manifest (P0.1)." - ) - p.add_argument( - "--library-precursors", - required=True, - type=Path, - help="MuMDIA library-precursors Parquet.", - ) - p.add_argument("--out", type=Path, help="Output manifest YAML (default: stdout).") - p.add_argument( - "--compare", type=Path, help="Declared manifest YAML to compare against." - ) - p.add_argument( - "--diann", type=Path, help="DIA-NN report (Parquet or TSV) to compare against." - ) - p.add_argument( - "--fail-on-mismatch", - action="store_true", - help="Exit nonzero on a material search-space mismatch.", - ) - p.add_argument( - "--min-shared-frac", - type=float, - default=0.9, - help="Minimum shared fraction of reference precursors (default 0.9).", - ) - p.add_argument( - "--mz-range-tol", - type=float, - default=5.0, - help="Tolerance (Th) for treating m/z-range endpoints as equal (default 5.0).", - ) - return p.parse_args(argv) - - -def main(argv: list[str] | None = None) -> int: - args = parse_args(argv) - if not args.library_precursors.exists(): - print(f"error: library not found: {args.library_precursors}", file=sys.stderr) - return 1 - - engine_version = resolve_engine_version(Path(__file__)) - manifest = derive_manifest(args.library_precursors, engine_version) - - yaml_text = yaml.safe_dump(manifest, sort_keys=False, allow_unicode=True) - if args.out: - args.out.parent.mkdir(parents=True, exist_ok=True) - args.out.write_text(yaml_text, encoding="utf-8") - print(f"wrote manifest: {args.out}") - else: - print(yaml_text) - - material_mismatch = False - - if args.compare or args.diann: - mum_keys, mum_ranges = mumdia_target_keys(args.library_precursors) - - if args.diann: - ref_keys, ref_ranges = load_reference_keys(args.diann) - ocmp = compare_overlap( - mum_keys, - mum_ranges, - ref_keys, - ref_ranges, - str(args.diann), - args.mz_range_tol, - ) - material_mismatch |= print_overlap(ocmp, args.min_shared_frac) - - if args.compare: - with args.compare.open("rb") as fh: - other = yaml.safe_load(fh) - dcmp = compare_declared(manifest, other, args.mz_range_tol) - material_mismatch |= print_declared(dcmp) - - if material_mismatch and args.fail_on_mismatch: - print("\nFAIL: material search-space mismatch detected.", file=sys.stderr) - return 2 - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/sensitivity_plan/01_workflow_and_gap_analysis.md b/sensitivity_plan/01_workflow_and_gap_analysis.md deleted file mode 100644 index 25ccbbe..0000000 --- a/sensitivity_plan/01_workflow_and_gap_analysis.md +++ /dev/null @@ -1,224 +0,0 @@ -# MuMDIA Workflow and Gap Analysis - -## 1. Purpose - -This document provides a working decomposition of the MuMDIA workflow so each stage can be evaluated independently. - -The intended pipeline model is: - -```text -FASTA and modification configuration - ↓ -peptide and peptidoform generation - ↓ -precursor charge-state generation - ↓ -fragment, retention-time, and optional mobility prediction - ↓ -candidate indexing and seed search - ↓ -run-specific calibration - ↓ -chromatogram extraction - ↓ -candidate peak-group generation - ↓ -peak-group feature calculation - ↓ -peak selection and precursor competition - ↓ -semi-supervised rescoring - ↓ -target-decoy competition and q-value estimation - ↓ -peptide and protein aggregation -``` - -The implementation agent must confirm which of these stages exist in the active MuMDIA branch and update this document when behavior differs. - -## 2. Primary comparison unit - -Use the following precursor identity as the primary comparison key: - -```text -normalized modified sequence + precursor charge -``` - -Also retain: - -- Run -- Protein assignment -- Precursor m/z -- Apex retention time -- Modification localization -- Decoy status -- Entrapment status - -Do not begin with protein counts. Protein inference can conceal precursor-level losses. - -## 3. Major differences to investigate relative to DIA-NN - -### 3.1 Timing of chromatographic peak selection - -Potential MuMDIA behavior: - -```text -extract traces - ↓ -choose one apex using heuristic evidence - ↓ -calculate final feature vector - ↓ -rescore -``` - -Recommended behavior: - -```text -extract traces - ↓ -generate top-K candidate peak groups - ↓ -calculate features for every peak group - ↓ -initial out-of-fold peak scoring - ↓ -select best peak per precursor - ↓ -final confidence scoring -``` - -A wrong early apex is unrecoverable if the correct peak is discarded before rescoring. - -### 3.2 Treatment of shared fragment evidence - -In wide-window DIA, a single observed fragment trace can match fragments from many candidate peptides. - -Potential failure: - -```text -one observed trace - ↓ -copied as full evidence to several candidates - ↓ -false candidates inherit real chromatographic signal - ↓ -target-decoy separation decreases -``` - -The preferred first step is not destructive peak ownership. Instead, calculate: - -- Number of candidates claiming each fragment -- Fraction of candidate intensity that is contested -- Fraction of evidence that is unique -- Best competing candidate score -- Shared-trace correlation -- Residual spectral similarity after stronger candidates are explained - -### 3.3 Candidate peak competition - -Distinguish: - -1. Multiple chromatographic peaks for the same precursor -2. Multiple peptides explaining the same local DIA evidence -3. Multiple peptidoforms explaining the same signal -4. Multiple charge states of the same peptide -5. Target-decoy competition -6. Duplicate reporting removal - -These must be separate stages with separate logs. - -### 3.4 Scoring architecture - -Potential differences to investigate: - -- Linear versus nonlinear feature interactions -- Separate peak-selection and confidence models -- Run normalization -- Prediction calibration -- Interference-aware scoring -- Unique-fragment evidence -- Candidate ambiguity -- Cross-charge corroboration -- Modification-aware residuals - -The question is not whether MuMDIA uses fewer or more features. The question is whether its features provide independent evidence in the low-FDR operating region. - -### 3.5 Calibration - -Evaluate whether MuMDIA uses: - -- Global or local mass calibration -- Nonlinear RT calibration -- Prediction uncertainty -- Run-specific peak-width distributions -- Ion-mobility calibration -- Modified-peptide-specific residual models - -A fixed extraction window may be simultaneously too wide for clean regions and too narrow for poorly calibrated regions. - -### 3.6 Decoy behavior and FDR - -More permissive extraction can increase both true and false candidates. If false candidates borrow real fragment traces, the high-scoring decoy tail can force a stricter score threshold. - -Therefore, sensitivity must be reported at: - -```text -matched empirical false discovery proportion -``` - -not only at each tool's nominal 1% q-value. - -## 4. Required workflow instrumentation - -For every candidate, store stage flags: - -```text -in_search_space -candidate_generated -traces_extracted -peak_group_generated -selected_as_peak_winner -selected_as_variant_winner -passed_initial_score -passed_target_decoy_competition -passed_precursor_fdr -passed_peptide_fdr -reported -``` - -Store one `rejection_reason` corresponding to the earliest failed stage. - -Suggested reasons: - -```text -PEPTIDE_NOT_GENERATED -MODIFICATION_NOT_ALLOWED -CHARGE_OUT_OF_RANGE -PRECURSOR_MZ_OUT_OF_RANGE -NO_VALID_FRAGMENTS -WRONG_ISOLATION_WINDOW -RT_PRUNED -CANDIDATE_CAP_REACHED -NO_FRAGMENT_TRACES -NO_PEAK_GROUP -PEAK_NOT_SELECTED -OUTCOMPETED_BY_TARGET -OUTCOMPETED_BY_DECOY -FAILED_PRECURSOR_FDR -FAILED_PEPTIDE_FDR -REMOVED_DURING_REPORTING -``` - -## 5. Questions the implementation must answer - -1. How many DIA-NN-only precursors are absent from the MuMDIA search space? -2. How many are generated in exhaustive candidate mode? -3. For how many is the DIA-NN apex within MuMDIA's top 1, 3, 5, and 10 candidate peaks? -4. How often does a wrong MuMDIA peak win over a peak near the DIA-NN apex? -5. How often is evidence dominated by shared fragments? -6. How many valid charge states or peptidoforms are removed by early competition? -7. How many targets rank well but fail q-value filtering? -8. Does competition improve empirical FDR enough to recover additional weak targets? -9. Which losses are enriched for low-intensity or modified peptides? -10. Which differences remain after matching the search space and prediction library? diff --git a/sensitivity_plan/03_feature_evaluation.md b/sensitivity_plan/03_feature_evaluation.md deleted file mode 100644 index 555a9ec..0000000 --- a/sensitivity_plan/03_feature_evaluation.md +++ /dev/null @@ -1,439 +0,0 @@ -# Feature Evaluation and Feature Expansion - -## 1. Objective - -Determine whether each feature: - -1. Contains useful signal -2. Adds information beyond existing features -3. Generalizes across datasets -4. Improves identifications at controlled empirical FDR -5. Does not introduce leakage or unstable subgroup behavior - -Feature importance alone is not sufficient. - -## 2. Feature registry - -Maintain a registry: - -```yaml -features: - rt_residual_normalized: - family: retention_time - level: peak_group - direction: lower_is_better - requires_calibration: true - uses_cross_run_information: false - missing_value_policy: median_plus_indicator - computational_cost: low - description: absolute calibrated RT residual divided by local uncertainty -``` - -Required fields: - -- Name -- Definition -- Family -- Level -- Units -- Expected direction -- Valid range -- Missing-value policy -- Calibration dependency -- Cross-run dependency -- Potential leakage risk -- Computational cost - -## 3. Leakage checks - -Never use as normal scoring features: - -- Final q-value -- Final posterior error probability -- Final target-decoy winner -- Candidate rank generated by the same model -- DIA-NN score or identification status -- Protein evidence calculated after precursor scoring -- Cross-run evidence calculated using the held-out run -- Calibration values fit using the held-out candidate -- Features computed globally before cross-validation - -Fit within each training fold: - -- Imputation -- Scaling -- Calibration -- Feature transformation -- Scoring model - -## 4. Validation design - -### Outer validation - -Prefer: - -```text -leave-one-dataset-out -leave-one-instrument-out -leave-one-acquisition-method-out -``` - -### Inner validation - -Group by: - -```text -modified_sequence + charge -``` - -Keep all peak groups for one precursor in the same fold. - -Entrapment labels are evaluation labels, not training labels. - -## 5. Feature evidence ladder - -### Level 1: Data quality - -Report per feature: - -- Missing percentage -- Invalid values -- Unique-value count -- Quantiles -- Distribution per run -- Distribution per instrument -- Distribution for targets, decoys, and entrapments -- Correlation with intensity -- Correlation with candidate count -- Correlation with peptide length, charge, and modification count - -Flag: - -- Constant features -- Asymmetric missingness -- Run-specific artifacts -- Features dominated by intensity -- Features separating target and decoy only because of decoy construction - -### Level 2: Univariate utility - -Evaluate: - -- Precision-recall area -- Partial ROC area in the low-error region -- Targets at matched empirical FDP -- Target-entrapment separation -- Monotonicity with correctness -- Utility by intensity decile - -Full ROC AUC is not the main metric. - -### Level 3: Redundancy - -Calculate: - -- Spearman correlation -- Mutual information -- Correlation within targets -- Correlation within decoys -- Correlation after controlling for intensity - -Cluster redundant features and test: - -```text -all features -one representative per cluster -leave-one-feature-out -leave-one-cluster-out -``` - -### Level 4: Feature-family ablation - -Recommended families: - -- Retention time -- Ion mobility -- Precursor mass accuracy -- Fragment mass accuracy -- Spectral agreement -- Chromatographic shape -- Fragment coelution -- MS1 and isotope evidence -- Interference evidence -- Candidate competition -- Peak morphology -- Search-space priors -- Modification-aware evidence -- Run-context evidence - -For every family compare: - -```text -full model -full model minus family -minimal baseline plus family -``` - -### Level 5: Conditional permutation - -Perform permutation inside held-out folds. - -Evaluate: - -- Standard permutation importance -- Grouped family permutation -- Conditional permutation for correlated features - -Primary metric: - -```text -change in identifications at 1% empirical FDP -``` - -### Level 6: Explanation analysis - -SHAP or similar methods may be used to diagnose: - -- Direction of effect -- Dataset stability -- Subgroup behavior -- Nonlinear thresholds -- Interactions - -Do not use SHAP rank as the final retention criterion. - -## 6. Separate scoring tasks - -Evaluate features separately for: - -### Peak-selection model - -Question: - -```text -Which chromatographic peak belongs to this precursor? -``` - -Metrics: - -- Correct peak at rank 1 -- Correct peak in top 3 -- Apex RT error -- Peak-boundary agreement - -### Peptide-ranking model - -Question: - -```text -Which peptide best explains this local peak group? -``` - -Metrics: - -- Correct target rank -- Margin over second-best target -- Margin over best decoy -- Recall at matched FDP - -### Final confidence model - -Question: - -```text -Is the winning precursor sufficiently reliable to report? -``` - -Metrics: - -- Identifications at matched empirical FDP -- q-value calibration -- posterior calibration -- entrapment FDP - -A feature can be useful for one task and unhelpful for another. - -## 7. Model comparisons - -Always compare at least: - -- Logistic regression or linear discriminant model -- Gradient-boosted tree model - -A large nonlinear-model advantage suggests useful interactions or thresholds. A small advantage suggests feature quality matters more than model capacity. - -## 8. Candidate feature families to add - -### 8.1 Uncertainty-normalized residuals - -```text -abs(RT residual) / local RT uncertainty -abs(mobility residual) / local mobility uncertainty -abs(mass error) / local mass uncertainty -``` - -Also retain signed residuals. - -### 8.2 Fragment evidence distributions - -Add: - -- Median fragment coelution -- Minimum fragment coelution -- Coelution interquartile range -- Fraction of fragments above a coelution threshold -- Median fragment mass error -- Fragment mass-error dispersion -- Fraction of top predicted fragments observed -- Explained observed intensity -- Explained predicted intensity -- Effective fragment count -- Evidence concentration in the strongest fragment - -### 8.3 Apex dispersion - -Add: - -- Fragment apex RT standard deviation -- Fragment apex RT median absolute deviation -- Maximum apex deviation -- Precursor-to-fragment apex deviation - -### 8.4 Peak-shape evidence - -Add: - -- Consensus chromatographic profile -- Fraction of each fragment explained by consensus -- Peak symmetry -- Tailing -- Shoulder score -- Number of local maxima -- Boundary agreement -- Peak truncation indicator - -### 8.5 Interference and contested evidence - -Add: - -- Number of candidate claimants per fragment -- Contested intensity fraction -- Unique-fragment count -- Unique-fragment intensity fraction -- Shared-fragment count -- Shared-trace correlation -- Local candidate density -- Isolation-window precursor density -- Correlation with competing candidate profiles -- Residual similarity after stronger candidates are explained - -### 8.6 Candidate ambiguity - -Add: - -- Margin from best alternative peak -- Margin from best alternative peptide -- Margin from best decoy -- Number of candidates within a score threshold -- Candidate score entropy -- Number of near-isobaric candidates -- Number of alternative modification localizations - -Do not feed a margin produced by the final model back into that same model. Use an earlier-stage score or a second-stage model. - -### 8.7 MS1 and isotope evidence - -Add: - -- Monoisotopic trace evidence -- Isotope-pattern correlation -- Isotope-spacing accuracy -- Isotope coelution -- Theoretical versus observed isotope ratio -- Precursor-fragment apex agreement -- Incorrect monoisotope indicator - -### 8.8 Modification-aware evidence - -Add: - -- Modification count -- Modification class -- Prediction-training coverage indicator -- Modification-specific RT residual percentile -- Modification-specific spectral residual percentile -- Localization ambiguity -- Site-determining ion count -- Site-determining ion intensity -- Number of alternative peptidoforms - -Validate FDR independently per modification group. - -### 8.9 Run context - -Add: - -- Isolation-window width -- Cycle time -- Number of scans across peak -- Local signal density -- Local candidate density -- Run-level calibration quality -- Run-level prediction residual statistics - -Normalize context features by run. - -## 9. Feature acceptance criteria - -Retain a feature or family when it produces one or more of: - -- At least 1% additional precursor identifications at 1% empirical FDP on at least two held-out datasets -- At least 5% gain in a scientifically important weak subgroup -- At least 2% improvement in correct-peak recall -- Better q-value calibration -- Lower runtime without sensitivity loss - -Also require: - -- No material FDP inflation -- Stable direction across datasets -- No leakage -- Similar availability for targets and decoys -- Acceptable computational cost - -Reject or revise when: - -- Gain appears in one dataset only -- Gain disappears in conditional ablation -- Feature mainly separates targets from artificial decoys, not entrapments -- Modified-peptide FDP worsens -- Feature effect changes direction across datasets -- Gain is smaller than seed-to-seed variability - -## 10. Required output table - -```text -feature_family -baseline_identifications -new_identifications -relative_gain -empirical_fdp -correct_peak_gain -low_intensity_gain -modified_peptide_gain -runtime_change -stability_score -recommendation -``` - -Allowed recommendations: - -```text -KEEP -KEEP_FOR_SUBGROUP -REVISE -REDUNDANT -UNSTABLE -LEAKAGE_RISK -TOO_EXPENSIVE -``` diff --git a/sensitivity_plan/04_peak_and_peptide_competition.md b/sensitivity_plan/04_peak_and_peptide_competition.md deleted file mode 100644 index bcb6e35..0000000 --- a/sensitivity_plan/04_peak_and_peptide_competition.md +++ /dev/null @@ -1,305 +0,0 @@ -# Peak and Peptide Competition Design - -## 1. Purpose - -Competition must prevent multiple false precursor hypotheses from borrowing the same physical DIA signal without suppressing genuinely coeluting peptides. - -Do not represent all competition as a single `compete` stage. - -## 2. Competition taxonomy - -Implement separate stages for: - -1. Chromatographic peak competition -2. Duplicate precursor competition -3. Peptide interference competition -4. Peptidoform competition -5. Modification-localization competition -6. Target-decoy competition -7. Reporting deduplication - -Each stage must record: - -- Input candidates -- Group identifier -- Winner -- Removed candidates -- Score used -- Removal reason - -## 3. Chromatographic peak competition - -### Problem - -One precursor may have several plausible chromatographic peaks. - -If one apex is selected before full scoring: - -```text -wrong peak selected - ↓ -correct peak discarded - ↓ -final scorer cannot recover precursor -``` - -### Recommended design - -1. Generate local maxima from consensus fragment evidence. -2. Retain top `K` peaks per precursor. -3. Calculate complete features for every peak group. -4. Score peaks out of fold. -5. Select the best peak for each modified-sequence and charge precursor. -6. Optionally retain a second peak when evidence supports repeated elution or unresolved ambiguity. - -Suggested initial values: - -```text -K ∈ {3, 5, 10} -``` - -### Evaluation - -Report reference-apex recall at top 1, 3, 5, and 10. - -## 4. Shared fragment evidence - -### Problem - -An observed MS2 trace may match fragments from many candidates: - -```text -observed m/z trace - ├── candidate A fragment - ├── candidate B fragment - ├── candidate C fragment - └── decoy D fragment -``` - -Giving full intensity to all candidates can create false high scores. Hard assignment can cause a high-abundance peptide to steal evidence from a real low-abundance peptide. - -### Initial non-destructive solution - -Do not reassign raw evidence in the first implementation. - -Calculate: - -- Claimant count per fragment -- Contested fragment count -- Contested intensity fraction -- Unique fragment count -- Unique intensity fraction -- Shared-trace correlation -- Strongest competitor score -- Score margin -- Residual spectral similarity -- Local conflict-group size - -Use these features during rescoring. - -## 5. Conflict graph - -Represent each candidate peak group as a node. - -Node identity: - -```text -run -modified sequence -charge -candidate apex -peak boundaries -``` - -Add an edge when candidates: - -1. Occur in compatible or overlapping DIA isolation windows -2. Have overlapping peak boundaries or close apexes -3. Share fragment m/z values within tolerance -4. Use a material amount of the same chromatographic evidence - -Edge features: - -```text -shared_fragment_count -shared_intensity_fraction_A -shared_intensity_fraction_B -apex_rt_difference -boundary_overlap -shared_trace_correlation -unique_fragment_count_A -unique_fragment_count_B -unique_intensity_fraction_A -unique_intensity_fraction_B -score_difference -``` - -## 6. Competition strategies to benchmark - -### Strategy A: No hard competition - -- Preserve all candidates -- Add conflict features -- Let final FDR handle ambiguity - -This is the baseline. - -### Strategy B: Hard winner-take-all - -Within sufficiently strong conflict groups: - -```text -retain highest initial discriminant score -``` - -This is simple but may suppress real coeluting peptides. - -### Strategy C: Unique-evidence-aware competition - -Allow multiple candidates to survive when each has enough independent evidence. - -Example rule: - -```text -unique_fragment_count >= 2 -and unique_intensity_fraction >= threshold -and unique_fragments_coelute -``` - -Otherwise, apply winner-take-all. - -### Strategy D: Margin-gated competition - -Remove a competitor only when: - -```text -winner_score - loser_score >= margin -``` - -and shared evidence exceeds a threshold. - -### Strategy E: Soft score penalty - -Do not remove candidates. Apply a penalty derived from: - -- Contested intensity -- Lack of unique evidence -- Stronger competing candidates -- Conflict-group size - -### Strategy F: Residual-evidence pass - -1. Score all candidates. -2. Select strongly supported candidates. -3. Model the fragment signal they explain. -4. Subtract or downweight explained signal. -5. Rescore weaker candidates using residual evidence. - -This is the most complex strategy and should be implemented only after A-E are benchmarked. - -## 7. Peptidoform and charge handling - -### Charge states - -Different charge states can both be real. - -Recommended rule: - -```text -do not directly compete distinct charge states -``` - -Instead, add cross-charge corroboration features. - -### Distinct modification states - -Unmodified and modified forms can co-exist. - -Do not compete all forms sharing the same base sequence. - -Compete only candidates that are mutually exclusive explanations of the same local signal. - -### Localization variants - -Candidates with the same composition but different modification sites require localization scoring. - -Retain them until site-determining ion evidence is evaluated. - -## 8. Target-decoy competition - -Target-decoy competition is not the same as interference competition. - -Recommended sequence: - -```text -peak scoring - ↓ -peak selection - ↓ -local peptide-interference handling - ↓ -peptidoform/localization handling - ↓ -final rescoring - ↓ -target-decoy competition - ↓ -q-value estimation -``` - -Document the competition unit explicitly: - -- Spectrum -- Peak group -- Precursor -- Peptide -- Peptidoform -- Protein group - -## 9. Competition feature audit - -For every competition feature, verify: - -- Symmetric calculation for targets and decoys -- No use of final labels -- Calculation within cross-validation folds where needed -- Stability across runs -- No dependence on candidate enumeration artifacts -- No direct reuse of final model score as an input to itself - -## 10. Primary experiment - -Run: - -| Mode | Peak candidates | Shared evidence | Variant competition | -|---|---:|---|---| -| A | 1 | full evidence | before rescoring | -| B | top 5 | full evidence | before rescoring | -| C | top 5 | conflict features | after initial rescoring | -| D | top 5 | unique-evidence-aware | after initial rescoring | -| E | top 5 | margin-gated hard competition | after initial rescoring | -| F | top 5 | residual-evidence pass | after initial rescoring | - -Measure: - -- Targets at matched empirical FDP -- Decoys and entrapments removed -- Correct targets removed -- Correct-peak recall -- Low-intensity identifications -- Modified-peptide identifications -- Conflict groups containing multiple independently supported targets -- Runtime and memory - -## 11. Recommended initial implementation - -Start with: - -```text -top-K peaks -+ non-destructive conflict graph -+ unique/contested evidence features -+ competition after initial rescoring -``` - -Do not start with raw centroid ownership or global winner-take-all assignment. diff --git a/sensitivity_plan/05_experiment_matrix.md b/sensitivity_plan/05_experiment_matrix.md deleted file mode 100644 index 46cb297..0000000 --- a/sensitivity_plan/05_experiment_matrix.md +++ /dev/null @@ -1,199 +0,0 @@ -# Experiment Matrix and Acceptance Criteria - -## 1. Experimental principles - -Each experiment must: - -- Change one component at a time -- Use fixed search spaces -- Use fixed random seeds -- Preserve raw candidate outputs -- Run on development and held-out datasets -- Report empirical FDP -- Report runtime and memory -- Report subgroup behavior - -## 2. Core experiment matrix - -| ID | Change | Main question | -|---|---|---| -| E0 | Current baseline | What is the present sensitivity gap? | -| E1 | Exhaustive candidate generation | Is pruning losing valid candidates? | -| E2 | Top-3 peak retention | Is early peak selection limiting sensitivity? | -| E3 | Top-5 peak retention | How much additional peak recall is gained? | -| E4 | RT oracle | Is RT calibration or pruning limiting sensitivity? | -| E5 | Peak oracle | Is the correct peak generated but ranked poorly? | -| E6 | Empirical fragment oracle | Are predicted transitions limiting sensitivity? | -| E7 | Wide extraction windows | Are tolerance settings too strict? | -| E8 | Adaptive calibrated windows | Can calibration improve signal-to-background? | -| E9 | No interference competition | Baseline effect of shared evidence | -| E10 | Conflict features only | Can non-destructive competition improve ranking? | -| E11 | Unique-evidence-aware competition | Can false candidates be suppressed safely? | -| E12 | Margin-gated hard competition | Does conservative removal improve FDR? | -| E13 | Residual-evidence rescoring | Can weaker coeluting peptides be recovered? | -| E14 | Move variant competition after rescoring | Is early competition removing real candidates? | -| E15 | Add MS1/isotope features | Does precursor evidence increase discrimination? | -| E16 | Add peak-shape features | Does chromatographic modeling improve peak ranking? | -| E17 | Add uncertainty-normalized residuals | Are raw prediction errors poorly calibrated? | -| E18 | Add candidate-ambiguity features | Does local competition improve confidence? | -| E19 | Linear versus LightGBM | Are nonlinear interactions important? | -| E20 | Alternative decoy design | Is FDR estimation limiting sensitivity? | -| E21 | Staged modification search | Is broad peptidoform competition too severe? | -| E22 | Run-specific prediction calibration | Does domain adaptation improve modified peptides? | -| E23 | Cross-run re-extraction | What is the experiment-level recovery ceiling? | - -## 3. Primary metrics - -### Sensitivity - -- Precursors at reported 1% q-value -- Precursors at 1% empirical FDP -- Peptides at 1% empirical FDP -- Modified peptides at 1% empirical FDP -- Protein groups at validated FDR - -### Pipeline recall - -- Search-space recall -- Candidate-generation recall -- Trace-extraction recall -- Correct peak in top 1 -- Correct peak in top 3 -- Correct peak in top 5 -- Correct peptide rank 1 -- Target-decoy win rate - -### Calibration - -- Reported q-value versus empirical FDP -- Posterior calibration -- Entrapment rate by score decile -- FDP by subgroup - -### Cost - -- Runtime -- Peak memory -- Candidate count -- Disk usage -- Feature-calculation time -- Model-training time - -## 4. Subgroup reporting - -Report all primary metrics by: - -- Intensity decile -- Precursor charge -- Peptide length -- Modification class -- Modification count -- Number of theoretical fragments -- Number of observed fragments -- Candidate density -- Isolation-window width -- Peak width -- RT region -- Instrument -- Acquisition method - -## 5. Decision rules - -### Candidate recall below 95% - -Prioritize: - -- Search-space generation -- Isolation-window assignment -- Modification enumeration -- RT pruning -- Candidate caps -- Isotope handling - -Do not prioritize final rescoring. - -### Candidate recall high, peak recall low - -Prioritize: - -- Top-K peak generation -- RT calibration -- Peak detection -- Peak boundaries -- Fragment consensus -- Adaptive peak widths - -### Correct peak available, peptide rank poor - -Prioritize: - -- Unique-fragment evidence -- Spectral agreement -- Fragment coelution -- Interference modeling -- Candidate ambiguity -- Prediction adaptation - -### Ranking good, q-value yield poor - -Prioritize: - -- Decoy construction -- Target-decoy competition -- Cross-validation -- Score calibration -- Group-specific FDR -- Removal of false candidates borrowing real signal - -### Single-run performance good, experiment coverage poor - -Prioritize: - -- Run alignment -- Empirical chromatogram libraries -- Controlled re-extraction -- Transfer-specific confidence estimation - -## 6. Acceptance criteria for code changes - -A change can be accepted when: - -- Empirical FDP remains controlled -- Gain reproduces on at least two held-out datasets -- No major subgroup degradation occurs -- Runtime and memory are acceptable -- Output is deterministic under fixed seeds -- All new rejection decisions are logged -- Feature and model versions are recorded - -Suggested quantitative thresholds: - -- At least 1% overall precursor gain at 1% empirical FDP, or -- At least 5% gain in a prespecified weak subgroup, or -- At least 2% gain in correct-peak recall, or -- Meaningful q-value calibration improvement - -## 7. Required statistical summaries - -For each experiment report: - -- Absolute identification difference -- Relative identification difference -- Bootstrap confidence interval by run or dataset -- Seed-to-seed variability -- Paired results by dataset -- Empirical FDP difference -- Runtime difference - -Do not select a feature based on one favorable run. - -## 8. Stop conditions - -Stop expanding model complexity when: - -- Candidate or peak recall, not scoring, remains the bottleneck -- Gains disappear on held-out datasets -- Empirical FDP degrades -- Runtime cost is disproportionate -- Additional features are redundant -- Feature effects are unstable across acquisition methods diff --git a/sensitivity_plan/06_agent_implementation_backlog.md b/sensitivity_plan/06_agent_implementation_backlog.md deleted file mode 100644 index 80e7848..0000000 --- a/sensitivity_plan/06_agent_implementation_backlog.md +++ /dev/null @@ -1,372 +0,0 @@ -# Agent Implementation Backlog - -## 1. Working conventions - -The agent should: - -- Implement small reviewable changes -- Add tests with every change -- Preserve backwards-compatible defaults where possible -- Record all configuration in output metadata -- Avoid irreversible candidate filtering in early stages -- Prefer diagnostic flags before production behavior changes -- Produce Parquet tables for large candidate-level outputs - -## 2. Priority 0: Observability and benchmark harness - -### P0.1 — Search-space manifest - -Implement a machine-readable search-space manifest and validation. - -Acceptance criteria: - -- Effective configuration is exported. -- MuMDIA and DIA-NN settings can be compared. -- The benchmark fails on important mismatches. -- Input hashes and software versions are stored. - -### P0.2 — Normalized output converter - -Implement a common output schema. - -Acceptance criteria: - -- All top candidates are retained. -- Final reported candidates are retained. -- Modified sequences are normalized consistently. -- Precursor keys are reproducible. - -### P0.3 — Candidate audit table - -Create `candidate_audit.parquet`. - -Required columns: - -```text -run_id -precursor_id -modified_sequence -charge -target_decoy_label -entrapment_label -candidate_generated -traces_extracted -peak_generated -peak_selected -variant_selected -target_decoy_winner -passed_precursor_fdr -passed_peptide_fdr -reported -rejection_reason -``` - -### P0.4 — Stage-level metrics - -Generate: - -- Search-space recall -- Candidate recall -- Trace recall -- Top-K peak recall -- Peptide-ranking recall -- Competition losses -- FDR losses - -### P0.5 — Entrapment runner - -Acceptance criteria: - -- Entrapment databases are generated reproducibly. -- Empirical FDP is reported at precursor, peptide, and protein levels. -- Results are stratified by modification and charge. - -## 3. Priority 1: Top-K chromatographic peaks - -### P1.1 — Candidate peak generator - -Implement configurable local peak generation. - -CLI: - -```text ---retain-top-peaks K -``` - -Acceptance criteria: - -- Supports `K = 1, 3, 5, 10`. -- Stores apex, boundaries, and initial score for every peak. -- Correct-peak top-K recall can be computed. -- Default behavior remains reproducible. - -### P1.2 — Peak-level feature calculation - -Calculate complete feature vectors independently for each candidate peak. - -Acceptance criteria: - -- No feature uses information from another candidate's final label. -- Peak features are cached. -- Computation is deterministic. - -### P1.3 — Peak-selection model - -Implement grouped out-of-fold peak scoring. - -Acceptance criteria: - -- Peak groups from one precursor remain in the same fold. -- Out-of-fold scores are stored. -- Top-1, top-3, and top-5 recall are reported. - -## 4. Priority 2: Competition graph and contested evidence - -### P2.1 — Fragment claimant index - -For every observed fragment trace, record candidate claimants. - -Acceptance criteria: - -- Claimant counts are available. -- Candidate-fragment mappings are queryable. -- Memory behavior is benchmarked. - -### P2.2 — Peak-group conflict graph - -Create graph edges for candidates with overlapping RT and fragment evidence. - -Acceptance criteria: - -- Edge thresholds are configurable. -- Edge features are stored. -- Graph construction is deterministic. -- Large connected components are diagnosed. - -### P2.3 — Conflict features - -Add: - -```text -contested_fragment_count -contested_intensity_fraction -unique_fragment_count -unique_intensity_fraction -strongest_competitor_score -score_margin -conflict_group_size -shared_trace_correlation -``` - -Acceptance criteria: - -- Target and decoy calculations are symmetric. -- Missing values are documented. -- Runtime is reported. - -### P2.4 — Competition modes - -CLI: - -```text ---competition-mode none ---competition-mode features-only ---competition-mode unique-evidence ---competition-mode margin-gated -``` - -Acceptance criteria: - -- Every removed candidate records its winner and reason. -- Competition happens after initial peak scoring. -- Empirical FDP is reported for every mode. - -## 5. Priority 3: Calibration and extraction - -### P3.1 — Two-pass mass calibration - -Acceptance criteria: - -- Robust precursor and fragment calibration. -- Local uncertainty estimate. -- Before-and-after residual plots. -- Fallback for insufficient calibrants. - -### P3.2 — Nonlinear RT calibration - -Acceptance criteria: - -- Monotonic mapping. -- Cross-validated residuals. -- Local RT uncertainty. -- Separate modified-peptide diagnostics. - -### P3.3 — Adaptive extraction windows - -Acceptance criteria: - -- Window width is based on calibrated uncertainty. -- Minimum and maximum bounds are configurable. -- Candidate recall and interference are compared with fixed windows. - -## 6. Priority 4: Feature evaluation framework - -### P4.1 — Feature registry - -Create `feature_registry.yaml`. - -Acceptance criteria: - -- Every feature has family, level, direction, missing policy, and leakage note. -- Documentation is generated automatically. -- Unknown features fail validation. - -### P4.2 — Feature audit command - -Suggested CLI: - -```bash -mumdia-feature-audit \ - --features candidate_features.parquet \ - --metadata candidate_metadata.parquet \ - --registry feature_registry.yaml \ - --output reports/feature_audit -``` - -Required outputs: - -- Missingness -- Distributions -- Correlation clusters -- Target/decoy/entrapment comparisons -- Run drift -- Leakage warnings - -### P4.3 — Ablation runner - -Suggested CLI: - -```bash -mumdia-feature-ablation \ - --config feature_experiments.yaml \ - --outer-group dataset_id \ - --inner-group precursor_id \ - --metric targets_at_empirical_fdp \ - --fdp 0.01 -``` - -Acceptance criteria: - -- Supports family removal and addition. -- Supports linear and tree models. -- Stores fold assignments. -- Produces paired dataset-level results. - -## 7. Priority 5: New features - -### P5.1 — Uncertainty-normalized residuals - -Implement RT, mass, and mobility normalized residuals. - -### P5.2 — Fragment evidence distributions - -Implement median, dispersion, threshold fractions, and effective fragment count. - -### P5.3 — Peak-shape and apex-dispersion features - -Implement consensus profile, boundary agreement, apex dispersion, symmetry, and shoulder score. - -### P5.4 — MS1 and isotope features - -Implement monoisotope, isotope correlation, spacing, coelution, and precursor-fragment agreement. - -### P5.5 — Candidate ambiguity features - -Implement alternative-target, alternative-peak, and decoy margins using an earlier-stage score. - -## 8. Priority 6: Peptidoforms and localization - -### P6.1 — Delay variant competition - -Acceptance criteria: - -- Different charge states are not prematurely collapsed. -- Modified and unmodified forms can coexist. -- Candidate counts before and after competition are reported. - -### P6.2 — Localization competition - -Acceptance criteria: - -- Localization variants remain until site-determining evidence is calculated. -- Site-determining ion count and intensity are available. -- Localization confidence is separated from precursor confidence. - -### P6.3 — Staged modification search - -Acceptance criteria: - -- Calibration stage and extended-modification stage are configurable. -- Combined confidence estimation is validated with entrapment. -- Modification-specific performance is reported. - -## 9. Priority 7: Reporting - -### P7.1 — HTML benchmark report - -Include: - -1. Search-space parity -2. Reported q-value versus empirical FDP -3. Identification counts at matched FDP -4. MuMDIA/DIA-NN overlap -5. Identification-loss waterfall -6. Candidate recall -7. Correct-peak recall -8. Peptide-ranking recall -9. Competition losses -10. Feature-family ablations -11. Performance by intensity -12. Performance by modification -13. Runtime and memory -14. Representative candidate chromatograms - -### P7.2 — Candidate diagnostic bundle - -For selected candidate IDs, export: - -- Fragment chromatograms -- MS1 traces -- Peak boundaries -- Predicted and observed spectra -- Feature values -- Conflict neighbors -- Removal reason - -## 10. Suggested first sprint - -Implement in this exact order: - -1. Search-space manifest -2. Candidate audit table -3. Rejection reason codes -4. Top-K peak retention -5. Reference-apex top-K analysis -6. Conflict graph -7. Unique and contested evidence features -8. Competition after initial rescoring -9. Entrapment validation -10. Feature-family ablation runner - -## 11. Completion checklist - -- [ ] Search-space parity is verified. -- [ ] Candidate-level audit output exists. -- [ ] At least 95% of missing precursors have a loss category. -- [ ] Top-K peak recall is measured. -- [ ] Competition stages are separated. -- [ ] Entrapment FDP is measured. -- [ ] Feature families have ablation results. -- [ ] Gains reproduce on held-out datasets. -- [ ] Modified peptides are evaluated separately. -- [ ] Runtime and memory are reported. diff --git a/sensitivity_plan/ARCHITECTURE_MAP.md b/sensitivity_plan/ARCHITECTURE_MAP.md deleted file mode 100644 index 91b979b..0000000 --- a/sensitivity_plan/ARCHITECTURE_MAP.md +++ /dev/null @@ -1,634 +0,0 @@ -# MuMDIA Architecture Map - -Stage-by-stage map of the real MuMDIA pipeline for the sensitivity-improvement -effort. It records where each stage lives, its key symbols with line numbers, its -input/output artifacts, the config knobs that govern it, its tests, and, most -importantly, every point at which a candidate precursor can be dropped. All paths -are repository-relative; all line numbers are 1-indexed against the working tree -on branch `feat/sensitivity-improvements`. - -Sources: the architecture-mapping workflow journal (7 stage maps: extract, -features, compete, rescore/FDR, configuration, IO+candidate-identity+calibration, -scripts) and direct reads of the source. The companion feature registry is -`FEATURE_REGISTRY.md` / `feature_registry.yaml`. - -## 0. Pipeline overview - -Each stage is an independent `mumdia ` subcommand reading path-addressable -inputs and writing Parquet plus an `.report.json` sidecar. The single-run -chain (orchestrated by `run`, `stages/run.rs`) is: - -``` -convert -> digest -> peptidoforms -> predict-frag -> search-seed -> rt-im-train - -> extract -> features -> compete -> rescore -> report (+ quant) -``` - -`candidate_id` is the dense identity key of the library and of every artifact from -extract onward. It is minted in `predict_frag.rs:163` as the `enumerate()` index -after sorting by precursor m/z. Pre-predict-frag stages (digest, peptidoforms) -have no `candidate_id`; they key on `base_peptide_id` / peptidoform id. -`Library::load` (`index.rs:78`) hard-asserts `candidate_id == 0..ncand` -contiguous and (`index.rs:135`) precursor-m/z ascending, so `candidate_id` is a -safe dense array index downstream. - -## 1. Modules added by the sensitivity lead agent - -Committed on `feat/sensitivity-improvements` across 17 commits (`2f46d6d` -rejection reasons + top-K enumerator + config scaffolding, `eb9da89` candidate -`audit` stage + subcommand, `de5ae2b` competition modes, `048cccd` apex-dispersion -+ mass-uncertainty feature families, `4ebf765` evidence-count apex selection, -`a9e3df6` adaptive RT window, `b183fec` top-K peak retention wired into `extract`, -`d61e3fe` two-pass mass calibration, `f6e6a6f` conflict/localization/peak-selection -sidecars). All additions are default-off / K=1-compatible; production defaults are -unchanged. - -Status of each addition: -- `mumdia audit` subcommand: WIRED and verified on real data (P0.3/P0.4). -- `CompetitionMode` in `compete`: WIRED (`de5ae2b`); default `WinnerTakeAll` - reproduces the legacy behaviour bit-for-bit; `none`/`features_only`/ - `unique_evidence`/`margin_gated` are selectable and unit tested. -- `enumerate_peaks` (`mumdia::peaks`): WIRED into `extract` (`b183fec`). When - `extract.retain_top_peaks > 1` the extractor enumerates top-K chromatographic - peak groups per candidate and writes `.peaks.parquet`; the default - `retain_top_peaks = 1` bypasses the enumerator and keeps the single-apex path. - Validated: K=5 on E. coli emitted 1,699,995 peak rows (mean ~4.97/candidate) - with the main `psms`/`chrom` flow and target-decoy FDR unchanged. -- `ExtractConfig.retain_top_peaks` / `emit_candidate_audit`: both CONSUMED. - `retain_top_peaks > 1` triggers the top-K peak table above; `emit_candidate_audit` - is read by `run` to write `candidate_audit.parquet` after rescore. -- `ExtractConfig.apex_evidence_rank`: IMPLEMENTED (default off). Selects the apex - by co-eluting fragment breadth (evidence count) rather than signature-ion - intensity. The peak-selection analysis found evidence-rank beats area-rank. -- `RtImTrainConfig.adaptive_rt_window` (+ `adaptive_rt_bins`, `rt_window_min_s`): - IMPLEMENTED (default off). Per-RT-region local residual window, clamped. -- `SearchSeedConfig.two_pass_mass_cal`: IMPLEMENTED (default off). Robust two-pass - precursor + fragment mass calibration. -- Two new feature families `apex_dispersion` (13) and `mass_uncertainty` (10) are - registered in `FAMILIES` (`048cccd`); the registry is now 383 features. -- None of the default-off knobs (top-K, `apex_evidence_rank`, `adaptive_rt_window`, - `two_pass_mass_cal`, competition modes) has passed the entrapment-holdout gate on - >=2 datasets yet, so all remain off by default; that validation is the next - action (see `NEXT_STEPS.md`). - -### `mumdia_core::rejection` (`rust/mumdia/crates/mumdia-core/src/rejection.rs`) - -- `RejectionReason` enum (`rejection.rs:19`): 16 loss categories plus a `Reported` - sentinel, `#[serde(rename_all = "SCREAMING_SNAKE_CASE")]`. The spellings match - spec 01 §4 exactly (`NO_PEAK_GROUP`, `OUTCOMPETED_BY_DECOY`, ...). -- `code()` (`rejection.rs:50`) stable string for Parquet/JSON; - `stage_order()` (`rejection.rs:76`) the identification-loss ladder (0 = earliest - stage, `Reported` = 255); `earliest()` (`rejection.rs:106`) keeps the earlier of - two losses; `is_rejection()` (`rejection.rs:100`). -- This is the type behind the audit table's `rejection_reason`; a candidate's row - records the earliest stage at which it was lost. - -### `mumdia::peaks` (`rust/mumdia/crates/mumdia/src/peaks.rs`) - -- `PeakGroup` struct (`peaks.rs:22`): `apex_idx`, `start_idx`, `end_idx`, - `apex_intensity`, `area`, `rank`. -- `enumerate_peaks(profile, k, bound_fraction, min_prominence_frac)` - (`peaks.rs:52`): pure, side-effect-free top-K local-maximum detector over a - consensus elution profile. Walks fractional-height boundaries (matching - `features.bound_peak_fraction`), drops maxima below a prominence floor, - deduplicates maxima inside a stronger peak's envelope, returns peaks strongest- - first by integrated `area`, deterministic (ties break by earlier `apex_idx`). - `k = 1` reproduces the single-strongest-apex behaviour, so callers can adopt it - incrementally. Fully unit-tested (`peaks.rs:154-268`), including the core - "interference-dominant but true peak retained with top-K" case - (`peaks.rs:190`). - -### `stages::audit` (`rust/mumdia/crates/mumdia/src/stages/audit.rs`, `mumdia audit`) - -- Wired as a subcommand (`main.rs:230`, `stages/mod.rs:5`). Non-destructive, - post-hoc: reconstructs per-candidate stage flags and the earliest - `RejectionReason` by tracking which `candidate_id`s survive across the artifact - chain library -> psms(extract) -> competed(compete) -> scored(rescore), then - writes `candidate_audit.parquet` and prints the identification-loss waterfall - (`audit.rs:64`, reason assignment `audit.rs:133-155`, waterfall `audit.rs:199`). -- `load_extract_reasons` (`audit.rs:51`) reads an optional - `.audit.parquet` sidecar to refine the coarse extract-stage bucket once an - in-extract audit is emitted. - -### New config fields (`rust/mumdia/crates/mumdia-core/src/config.rs`) - -- `ExtractConfig.retain_top_peaks: usize` (`config.rs:552`, default 1): K - chromatographic peak groups per candidate; 1 = legacy single apex. Validated - `>= 1` (`config.rs:951`). CONSUMED in `extract.rs` (`>1` calls `enumerate_peaks` - at `extract.rs:929` and writes `.peaks.parquet` at `extract.rs:1068`). -- `ExtractConfig.emit_candidate_audit: bool` (`config.rs:557`, default false): - when true, `run` writes `candidate_audit.parquet` after rescore (`run.rs:265`). - Near-zero cost when false. (The finer in-extract `.audit.parquet` sidecar - that `stages::audit::load_extract_reasons` can read is still future work; see - `NEXT_STEPS.md`.) -- `ExtractConfig.apex_evidence_rank: bool` (`config.rs:566`, default false): - when true, the apex loop scores by co-eluting fragment count rather than - signature-ion intensity (`extract.rs:735`). -- `SearchSeedConfig.two_pass_mass_cal: bool` (`config.rs:387`, default false): - robust two-pass precursor + fragment mass calibration in `search_seed.rs:188`. -- `RtImTrainConfig.adaptive_rt_window: bool` (`config.rs:433`, default false) with - `adaptive_rt_bins: usize` (`config.rs:435`, default 12) and `rt_window_min_s: f64` - (`config.rs:437`, default 1.0): per-RT-region local residual window in - `rt_im_train.rs:121`, clamped to `[rt_window_min_s, fallback_rt_window_s]`. -- `CompeteConfig.mode: CompetitionMode` (`config.rs:647`, default `WinnerTakeAll`), - plus `margin`, `unique_evidence_min_fragments`, and `emit_competition_audit`. -- `CompetitionMode` enum (`config.rs:680`): `WinnerTakeAll` (legacy) / `None` / - `FeaturesOnly` / `UniqueEvidence` / `MarginGated`, with `from_token` - (`config.rs:699`). Maps to spec 04 §6 strategies A/B/C/D. CONSUMED in - `compete.rs` via the pure `resolve_competition()` (`de5ae2b`); `WinnerTakeAll` - is bit-identical to the previous behaviour. - -## 2. Stage-by-stage map - -For each stage: main source, key symbols (line), input -> output artifacts, the -config knobs that govern it, and existing tests. - -### convert (Stage 0) - -- Source: `stages/convert.rs`. Entry `run` (`convert.rs:103`); `centroid` - (`convert.rs:19`), `ConvertParams` (`convert.rs:86`), `ConvertOutputs` - (`convert.rs:96`). -- IO: mzML -> `spectra_ms1.parquet`, `spectra_ms2.parquet`, - `isolation_windows.parquet`, `ms2_to_ms1.parquet` (`convert.rs:172-175`). - Assigns a monotonic `scan_index`, converts scan time to seconds, centroids - profile spectra (local maxima), caps peaks top-N, synthesizes a full-range - window for zero-bounded AIF scans. -- Knobs: `max_spectra` (`convert.rs:89`/`124`) caps spectra for fast iteration - (run-level, not a candidate drop). No per-candidate drops. -- Tests: none at stage level (CLAUDE.md test gap). - -### digest (Stage A) - -- Source: `stages/digest.rs`. Entry `run` (`digest.rs:147`); `make_decoy` - (`digest.rs` ~95-117). -- IO: FASTA -> `peptides.parquet` (keyed on `id`/`target_id`). Fully-tryptic - in-silico digest, mints paired reverse/scramble decoys immediately, dedups - targets by stripped sequence (insertion order preserved for determinism). -- Knobs: `digest.enzyme` (`config.rs:261`), `missed_cleavages` (271), `min_len` - / `max_len` (272), `decoy.strategy` (244, `Reverse` default; `DiannShift`/`None` - produce no decoy and `DiannShift` is rejected by validate). -- Tests: `trypsin_p_cleaves_after_kr`, `reverse_decoy_keeps_cterm`, - `scramble_is_deterministic`. - -### peptidoforms (Stage A2) - -- Source: `stages/peptidoforms.rs`. Entry `run` (`peptidoforms.rs:68`); - `proforma` (`peptidoforms.rs:22`). -- IO: `peptides.parquet` -> `peptidoforms.parquet` (keyed on peptidoform `id` + - `base_peptide_id`). Expands stripped peptides into peptidoforms with fixed + - variable mods and charges, emits ProForma-lite. Known limitation: a second mod - at the same position is dropped; no terminal mods. -- Knobs: `peptidoforms.fixed_mods` (`config.rs:283`), `variable_mods` (284), - `max_variable_mods` (285), `charge_min` / `charge_max` (286-287), - `unknown_modification` (289, `Error` default). -- Tests: `proforma_places_mods`, `combos_bounded`. - -### predict-frag (Stage C) - -- Source: `stages/predict_frag.rs`. Entry `run` (`predict_frag.rs:50`); - candidate_id mint after precursor-m/z sort (`predict_frag.rs:137,163`). -- IO: `peptidoforms.parquet` -> `fragment_library_precursors.parquet` + - `fragment_library_fragments.parquet` (the library). Computes precursor and b/y - fragment m/z (shared mass model), intensities (native or MS2PIP), iRT (native - or DeepLC), top-N, and builds the contiguous `candidate_id` the inverted index - needs. This is the birth of candidate identity. -- Knobs: `predict_frag.predictor` (`config.rs:334`), `rt_predictor` (335), - `top_n_fragments` (341), `charge2_from_precursor_charge` (340), - `ms2pip_model`/`ms2pip_python`/`deeplc_python` (342-344). -- Tests: none at stage level. - -### search-seed (Stage S) - -- Source: `stages/search_seed.rs`. Entry `run` (`search_seed.rs:45`); masscal.json - writer (`search_seed.rs:186`). -- IO: library + `spectra_ms2.parquet` -> `seed_psms.parquet` (best-per-candidate) - + `.masscal.json` `{frag_ppm_offset, frag_tol_ppm, n_dev}`. Native - Sage-lite hyperscore for calibration only (not a library filter). Drives per-run - mass recalibration; fallback `{0.0, cfg.fragment_tol_ppm}` when `n_dev < 20`. -- New opt-in knob (default off): `two_pass_mass_cal` runs a robust two-pass - precursor + fragment mass calibration (`search_seed.rs:188`). -- Knobs: `search_seed.fdr_seed` (`config.rs:368`), `fragment_tol_ppm` (370), - `min_matched_peaks` (373/374), `report_psms` (371), `matcher` (381). - `precursor_tol_ppm` (369) is a dead knob (warns). -- Tests: none at stage level. - -### rt-im-train (Stage B) - -- Source: `stages/rt_im_train.rs`. Entry `run` (`rt_im_train.rs:28`); run_windows - writer (`rt_im_train.rs:135`), cal.json writer (`rt_im_train.rs:149`). -- IO: `seed_psms.parquet` + library -> `run_windows.parquet` - (`candidate_id, rt_pred_cal, rt_lo, rt_hi, im_*` (IM null)) + `.cal.json` - `{method, slope, intercept, w_rt, p_rt, multiplier, n_train, calibration_status}`. - Fits predicted_irt -> observed RT (linear always, LOESS when configured), sets a - single global RT window half-width `w_rt` applied uniformly to every candidate - (no per-candidate uncertainty). Optional DeepLC multitask fine-tune first. -- New opt-in knob (default off): `adaptive_rt_window` (+ `adaptive_rt_bins`, - `rt_window_min_s`) replaces the single global `w_rt` with a per-RT-region local - residual window, clamped to `[rt_window_min_s, fallback_rt_window_s]` - (`rt_im_train.rs:121-134`). -- Knobs: `rt_im_train.calibration_method` (`config.rs:401`, `None` rejected), - `q_train` (402), `p_rt` / `rt_window_multiplier` (404-405), - `min_seed_for_calibration` (406), `loess_span` (408), `fallback_rt_window_s` - (410), `finetune_deeplc` (416). `tolerance_regime` (400) dead (warns). -- Tests: none at stage level. - -### extract (Stage D) - the core stage - -- Source: `stages/extract.rs`, `index.rs`, `matchers/fragindex.rs`. Entry `run` - (`extract.rs:222`); `Hit` (`extract.rs:80`), accumulator `acc` - (`extract.rs:302`), `extract_accumulate_windows` (`extract.rs:128`), per-candidate - parallel map `cand_hits`/`into_par_iter` (`extract.rs:566`), `CandOut` - (`extract.rs:569`), apex selection loop (`extract.rs:718`), smoothed rolling - count (`extract.rs:681`), signature ions (`extract.rs:710`), co-elution run - (`extract.rs:736`), acquisition-scan grid (`extract.rs:632`), chrom emission - (`extract.rs:851`). Index: `FragIndex` (`fragindex.rs:24`), `probe_peak` - (`fragindex.rs:152`), `Library::candidate_range` (`index.rs:233`), `cand_frags` - (`index.rs:208`), `page_search` (`index.rs:242`). -- IO: library + `run_windows.parquet` + spectra + masscal -> `psms_extracted.parquet` - (20 cols, one apex PSM per surviving candidate, write `extract.rs:960`) + - `chromatograms.parquet` (7 cols, `LargeListF32` traces, keyed by candidate_id, - write `extract.rs:986`). Peak-major over the SoA inverted index; a cheap-to- - expensive cascade (distinct-fragment presence -> co-elution run -> matched - fraction -> Pearson gate) accepts candidates. The main `psms_extracted` table - still emits exactly one apex per candidate; the peak dimension now exists only in - the opt-in `.peaks.parquet` sidecar (7 cols: `candidate_id, peak_rank, - apex_rt, start_rt, end_rt, evidence_count, area`) written when - `retain_top_peaks > 1` (`extract.rs:1068`). -- New opt-in knobs (default off): `retain_top_peaks > 1` enumerates top-K peak - groups (`extract.rs:927-929`, via `peaks::enumerate_peaks`); `apex_evidence_rank` - scores the apex by fragment breadth instead of intensity (`extract.rs:735`); - `emit_candidate_audit` makes `run` write `candidate_audit.parquet`. -- Knobs (all `ExtractConfig`, `config.rs:436-533`): `frag_tol_ppm` (440), - `prec_tol_ppm` (441), `presence_min_matched` (443), `presence_min_fragments` - (445), `presence_min_coelution` (447), `min_frag_corr` (452/453), - `min_matched_fraction` (458), `min_coelution_run` (515), `fixed_scan_window` - (439), `apex_top_fragments` (465), `apex_rt_prior_s` (469), `apex_count_tol` - (474), `apex_count_window` (484), `emit_window_grid` (489), `peak_claim` (498), - `emit_contested_features` (503), `peak_claim_margin` (507), `matcher` (509), - `ms1_rescue` (521), plus the new `retain_top_peaks` (552), - `emit_candidate_audit` (557) and `apex_evidence_rank` (566). Dead: `k_select` (491), - `max_fragment_charge` - (495), `scan_scale` (438), `ScanWindowMode::PeakWidthDerived`. -- Tests: none at stage level (exercised only via full `run`). - -### features (Stage E) - -- Source: `stages/features.rs` + `stages/features/*.rs`. Entry `run` - (`features.rs:491`); `FAMILIES` (49), `active_features` (201), - `feature_schema_id` (220), `FeatureSchema` (226), `Evidence` (269), - `build_evidence` (336), `fragment_features` (915), `peak_bounds` (853), - `prelim_score` (722), per-PSM parallel block (608), cross-charge block (573), - `write_pin` (1216). See `FEATURE_REGISTRY.md` for the 17-family breakdown. -- IO: `psms_extracted.parquet` + `chromatograms.parquet` (+ `seed_psms.parquet`) - -> `features.parquet` + `.schema.json` + `run.pin`. Drop-free: one - output row per input PSM; missing chromatograms yield default/zero vectors - (see drop table). Emits `prelim_score` (bookkeeping, not a feature column). -- Knobs (`FeaturesConfig`, `config.rs:560`): `set` (561, `FeatureSet`), - `coelution_corr_threshold` (562), `bound_features` (567), `bound_peak_fraction` - (571). `prec_tol_ppm` (563) is declared but unused here. Several family - thresholds are hardcoded consts, not config (`FRAG_TOL_PPM` - `mass_accuracy.rs:44`, `MAXLAG` `coelution.rs:62`, `MIN_FRAGS/MIN_SCANS` - `order_consistency.rs:38`). -- Tests: `feature_sets_sized` (`features.rs:1263`, asserts tier sizes), - `peptide_length_ignores_mods`, `xcorr_aligned_traces`, and per-family tests in - `chromatographic.rs` and `order_consistency.rs`. Most family modules - (similarity, entropy, coelution, interference, mass_accuracy, ion_series, ms1, - rt, novel, nonzero, peak_scans) have no unit tests. - -### compete (Stage F, part 1) - -- Source: `stages/compete.rs`. Entry `run` (`compete.rs:28`); `CompeteParams` - (`compete.rs:20`), `pform_id` (51), winner map (72), `label_code` (74), key - match (79), winner selection (92-95), `keep` (101). -- IO: `features.parquet` -> `psms_competed.parquet` (survivors + carried feature - schema). Selects one winner per competition group by `prelim_score`. The label - is part of the group key (`compete.rs:74-78`), so a target never competes - against its own decoy (the decoy null is preserved). Runs BEFORE rescore - (`run.rs:243` between features `run.rs:231` and rescore `run.rs:252`); the - winner is chosen on `prelim_score`, not on any rescorer output. -- Knobs (`CompeteConfig`, `config.rs:601`): `group_by` (607, `CompeteGroupBy` - Precursor/Apex/PeptidoformCharge), `apex_rt_tolerance_s` (608), plus the new - scaffolded `mode`/`margin`/`unique_evidence_min_fragments`/`emit_competition_audit` - (613-623). -- Tests: `features_compete_rescore_run_on_crafted_input` - (`tests/pipeline.rs:166`, default Precursor grouping only). - -### rescore (Stage F, part 2) + target-decoy FDR - -- Source: `stages/rescore.rs`, `rescoring.rs`, `fdr.rs`. Entry `run` - (`rescore.rs:41`); `percolator_lite` (`rescoring.rs:98`), `logreg_fit` - (`rescoring.rs:48`), `fit_standardizer` (`rescoring.rs:14`), `RescoreInput.fold_key` - (`rescoring.rs:90`), `grouped_q` (`rescore.rs:367`), `classify_entrapment` - (`rescore.rs:335`), `QMode` (`rescore.rs:22`), `run_mokapot` (`rescore.rs:485`), - `run_entrapment_gbm` (`rescore.rs:427`). FDR: `target_decoy_q` (`fdr.rs:7`, - numerator `n_decoys + 1`, tie-block collapsed, monotonized), `entrapment_q` - (`fdr.rs:63`). -- IO: `psms_competed.parquet` (concatenated across inputs) -> `psms_scored.parquet` - with ALL rows incl decoys (`rescore.rs:251`, no drop). Attaches PSM / peptide - (`base_peptide_id`) / protein-group / global q-values. Native `percolator_lite` - folds by `base_peptide_id % folds` (no peptide leaks across folds); - standardizer fit on train rows only. -- Knobs (`RescoreConfig`, `config.rs:723`): `classifier` (724, `RescorerKind` - NativeTda/Mokapot/Percolator/Entrapment), `folds` (725), `train_fdr` (726), - `num_iter` (728), `python` (729), `percolator_bin` (730), - `entrapment_marker`/`entrapment_exclude`/`entrapment_contaminant_markers`/ - `entrapment_ratio` (734-748), `strict` (754). -- Tests: `perfect_separation_q_is_conservative_plus_one`, `tied_scores_share_one_q`, - `entrapment_q_ranks_real_above_spike_in`, `separates_targets_from_decoys`. Gap: - no test covers `rescore::run` itself, the sidecar branches, or `QMode::Entrapment` - end to end. - -### report - -- Source: `stages/report.rs`. Entry `run` (`report.rs:49`); peptide gate - (`report.rs:92`), peptide dedup (97), protein gate (118), protein dedup (121). -- IO: `psms_scored.parquet` -> `peptides.tsv` + `proteins.tsv`. This is where FDR - gating actually removes rows from output (targets only, q <= `q_threshold`). -- Knobs: `q_threshold` (`ReportParams`, `report.rs:19`; `run` uses 0.01). -- Tests: `strip_mods_and_decoy` (`report.rs:143`). - -### quant (Stage G, beyond-MVP) and quant-lfq - -- Source: `stages/quant.rs`. Entry `run` (`quant.rs:147`), `run_lfq_combine` - (`quant.rs:413`). Trapezoid XIC integration + top-N sum + protein-group rollup + - per-fragment export; MaxLFQ/directLFQ cross-run via `quant-lfq`. -- Knobs (`QuantConfig`, `config.rs:682`): `q_threshold` (682), `top_n_fragments` - (684), `top_n_peptides` (686), `rollup` (688), `bound_peak` (690), - `peak_fraction` (694), `peak_grace` (698), `peak_window_mode` (700), - `reliable_q` (703). -- `align` (Stage D2, `align.rs:52`) and MBR (Stage D3) are beyond-MVP / stub and - not in the `run` chain. - -## 3. Where candidates are dropped - -Every removal or decision point that can lose a candidate precursor, grouped by -stage, collected from all stage maps. `Reason` is the matching -`RejectionReason::code`. Stages before predict-frag have no `candidate_id`, so -their drops are audited at `base_peptide_id` / peptidoform-id level. - -### digest (Stage A) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `digest.rs:72` | `missed = j-i-1 > cfg.missed_cleavages` -> break | peptide spanning too many missed cleavages never enumerated | `PEPTIDE_NOT_GENERATED` | -| `digest.rs:77` | `len < min_len` or `len > max_len` -> continue | peptide outside length bounds dropped | `PEPTIDE_NOT_GENERATED` | -| `digest.rs:81` | non-standard residue (B/J/O/U/X/Z) -> continue | ambiguous-residue peptide dropped | `PEPTIDE_NOT_GENERATED` | -| `digest.rs:95` | `make_decoy: n < 3` -> None | no decoy minted for a <3-residue target | `PEPTIDE_NOT_GENERATED` | -| `digest.rs:117` | `DecoyStrategy::DiannShift`/`None` -> None | no sequence-rewrite decoy (DiannShift unrealized; rejected by validate) | `PEPTIDE_NOT_GENERATED` | -| `digest.rs:162` | target already seen (dedup by stripped seq) | duplicate peptide merged; collapses protein multiplicity | `PEPTIDE_NOT_GENERATED` | - -### peptidoforms (Stage A2) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `peptidoforms.rs:80` | `unimod_mass(mod).is_none()` -> `bail!` | unknown fixed/variable mod aborts the run (hard error) | `MODIFICATION_NOT_ALLOWED` | -| `peptidoforms.rs:22` | second mod at the same residue position | silently keeps only the first mod (documented limitation) | `MODIFICATION_NOT_ALLOWED` | -| `peptidoforms.rs:121` | `z` outside `[charge_min, charge_max]` | charge states outside the range never enumerated | `CHARGE_OUT_OF_RANGE` | - -### predict-frag (Stage C) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `predict_frag.rs:73` | `parse_peptidoform(pform)` is Err | ProForma parse failure dropped before candidate_id | `MODIFICATION_NOT_ALLOWED` | -| `predict_frag.rs:83` | `frags.is_empty()` | peptidoform with no b/y fragments dropped | `NO_VALID_FRAGMENTS` | -| `predict_frag.rs:134` | `retain(!frags.is_empty())` after top-N | candidate left with zero fragments after top-N truncation removed | `NO_VALID_FRAGMENTS` | - -### search-seed (Stage S) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `search_seed.rs:81` | seed matched count < `min_matched_peaks` | seed PSM not emitted (calibration only); does NOT remove the candidate, it just lacks a `seed_score`/`seed_identified` feature | `NO_FRAGMENT_TRACES` (soft) | - -### extract (Stage D) - -The three accumulation paths (parallel window, serial non-two-pass, two-pass) -repeat the same three peak-level gates. `hi <= lo` (empty candidate range), -the RT gate, and empty claimants are per-peak skips that only cause a candidate to -be dropped if all of its collisions are skipped (the implicit non-materialization -at `extract.rs:302/554`). The `return None` gates are the explicit per-candidate -drops. - -| file:line | condition | effect | reason | -|---|---|---|---| -| `extract.rs:302` / `:554` | candidate never inserted into `acc` (no in-window, in-RT fragment collision anywhere) | silent, irreversible non-materialization; never appears in `psms_extracted`. The single largest invisible drop class | `NO_FRAGMENT_TRACES` | -| `extract.rs:154` / `:327` / `:390` / `:433` | `hi <= lo`: isolation-window group maps to an empty candidate range | window contributes no hits to any candidate | `WRONG_ISOLATION_WINDOW` | -| `extract.rs:169` / `:341` / `:402` / `:446` | `rt < rt_lo[c]` or `rt > rt_hi[c]` | peak collision discarded for the candidate (outside calibrated RT window) | `RT_PRUNED` | -| `extract.rs:174` / `:348` / `:453` | `claimants.is_empty()` | peak matched no in-range/in-RT candidate; peak-level skip | `NO_FRAGMENT_TRACES` | -| `extract.rs:600` | `distinct.len() < presence_min_matched.max(1)` | first explicit per-candidate drop (tier-b presence) | `NO_FRAGMENT_TRACES` / `NO_VALID_FRAGMENTS` | -| `extract.rs:750` | `distinct.len() < presence_min_fragments` (acceptance sub-cond 1) | too few distinct matched fragments | `NO_VALID_FRAGMENTS` | -| `extract.rs:751` | `best_run < scan_window` (= `fixed_scan_window.max(1)`) | co-elution run shorter than the consecutive-scan floor | `NO_PEAK_GROUP` | -| `extract.rs:752` | `best_run < min_coelution_run` (default 0 = off) | transient (likely interferent) co-elution | `NO_PEAK_GROUP` | -| `extract.rs:753` | `matched_fraction < min_matched_fraction` (default 0 = off) | too small a fraction of predicted fragments observed | `NO_VALID_FRAGMENTS` | -| `extract.rs:808` | `pearson(obs_apex, pred) < min_frag_corr` and not MS1-rescued | apex fragment pattern disagrees with the predicted spectrum | `PEAK_NOT_SELECTED` / `NO_VALID_FRAGMENTS` | -| `extract.rs:718` | single-argmax apex; only the best scan-group emitted | secondary co-eluting peaks of the same candidate are never emitted (top-1 only). No trigger today; becomes reachable with top-K | `PEAK_NOT_SELECTED` (latent) | - -### features (Stage E) - not drops, but silent zeroing - -The features stage drops nothing (one row per input PSM), but several branches -turn a candidate into a maximally decoy-like all-zero vector with no signal that -the zero is undefined rather than measured. These are the "undefined-zero" -ambiguities the sensitivity work targets. - -| file:line | condition | effect | reason (undefined-zero) | -|---|---|---|---| -| `features.rs:611` | `chrom.get(cid)` None/empty | `FragFeatures::default()`: all legacy fragment/coelution/mass/interference features 0.0 | `NO_FRAGMENT_TRACES` | -| `features.rs:644` | Extended active and no chromatogram | entire extended battery zeroed for the candidate | `NO_FRAGMENT_TRACES` | -| `features.rs:748` | active column has no `fmap` entry | column filled with 0.0 (masks a missing/misnamed feature) | `REMOVED_DURING_REPORTING` | -| `features.rs:80` | extended name collides with reserved / repeats | FEATURE-column drop (not a candidate drop): `spectral_angle`, `rt_error_abs`, `novel::peptide_length`, `novel::seed_identified` removed from the schema | `REMOVED_DURING_REPORTING` | -| `features.rs:862` | `peak_bounds` apex at zero height | apex relocated to global max; changes peak-bounded feature values | `PEAK_NOT_SELECTED` | -| family early returns | `entropy.rs:124` (k==0), `chromatographic.rs:72` (empty axis), `mass_accuracy.rs:139` (no ppm), `order_consistency.rs:122` (<3 frags/scans), `ms1.rs:242` (`ms1_xic.len() < 3`) | family returns all-zero vector; the last is dead until extract persists `ms1_xic` | `NO_PEAK_GROUP` / `NO_VALID_FRAGMENTS` | - -### compete (Stage F, part 1) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `compete.rs:95` | same-key group, target loser: `prelim[i] <= prelim[w]` (ties keep first-seen) | lower-prelim charge/mod/localization sibling of a target removed before FDR, no audit row | `OUTCOMPETED_BY_TARGET` | -| `compete.rs:95` | same-key group, decoy loser (label in key) | lower-prelim decoy sibling removed; thins the decoy null symmetrically (keeps FDR trustworthy) | `OUTCOMPETED_BY_DECOY` | -| `compete.rs:82` | Apex mode: two peaks of same base+label round to the same `apex_rt` bucket | a genuinely distinct RT peak of the same peptide is dropped if it shares the bucket | `PEAK_NOT_SELECTED` | -| `compete.rs:46` | `PeptidoformCharge` mode and `charge` column absent | whole stage `bail`s (config/precondition error, not a per-candidate drop) | run failure | - -### rescore (Stage F, part 2) - -| file:line | condition | effect | reason | -|---|---|---|---| -| `rescore.rs:251` | none | writes every PSM incl decoys; drops nothing (all real drops deferred to report) | `REMOVED_DURING_REPORTING` (deferred) | -| `rescore.rs:415` | `grouped_q`: PSM is not the max-score row of its peptide / protein-group key | losing sibling's group q hard-set to 1.0 (silent demotion; will fail the report gate) | `OUTCOMPETED_BY_TARGET` / `FAILED_PEPTIDE_FDR` | -| `rescoring.rs:117` | degenerate fold (empty train_idx/test_idx) | those PSMs silently keep `init_score` (prelim) instead of a rescored value; scoring degradation, not a drop | (none) | - -### report - -| file:line | condition | effect | reason | -|---|---|---|---| -| `report.rs:92` | `label != "target"` or `peptide_q_value > q_threshold` | row excluded from `peptides.tsv` (decoys; targets failing peptide FDR) | `FAILED_PEPTIDE_FDR` | -| `report.rs:97` | `(peptidoform, charge)` already emitted | duplicate precursor dropped (best-q kept via sort) | `REMOVED_DURING_REPORTING` | -| `report.rs:118` | `label != "target"` or PG empty or `pg_q_value > q_threshold` | row excluded from `proteins.tsv` (no dedicated protein-FDR reason code exists) | `FAILED_PEPTIDE_FDR` / `REMOVED_DURING_REPORTING` | -| `report.rs:121` | `protein_group` already emitted | duplicate protein group dropped (best-q kept) | `REMOVED_DURING_REPORTING` | - -### configuration (run-level gates, not per-candidate) - -`Config::validate` (`config.rs:825`) blocks the whole run rather than dropping a -candidate: `DiannShift` decoy (`config.rs:827`) and `CalibrationMethod::None` -(`config.rs:836`) are hard errors; an unknown `--profile` (`config.rs:888`) aborts. -Seven dead knobs only warn (`search_seed.precursor_tol_ppm`, -`rt_im_train.tolerance_regime`, `extract.k_select`, `extract.max_fragment_charge`, -`extract.scan_scale`, `digest.decoy.source`, `digest.decoy.ratio`) and have no -runtime effect. `extract.k_select` (default 50, unimplemented) is the natural home -for a candidate-count cap (`CANDIDATE_CAP_REACHED`), which is not realized today. - -## 4. Hooks for the sensitivity work - -Exact file:line sites for the six work items, from the stage maps. - -### 4.1 Candidate audit (rejection reasons + per-candidate ledger) - -- Already available: `mumdia audit` post-hoc reconstruction (`audit.rs:64`), the - `RejectionReason` type (`rejection.rs:19`), the `emit_candidate_audit` flag - (`config.rs:533`), and the sidecar reader `load_extract_reasons` (`audit.rs:51`) - that consumes a future `.audit.parquet`. -- In-extract emission (to write that sidecar): the never-materialized cohort is - `union(candidate_range over all windows)` minus `acc.keys()` (`extract.rs:302`/ - `:554`); separate `RT_PRUNED` from `NO_FRAGMENT_TRACES` by tallying the RT gates - (`extract.rs:169`/`:341`/`:402`/`:446`); change the per-candidate parallel map - (`extract.rs:593-921`) to return `Accepted|Rejected{reason, evidence}` instead of - `Option`, carrying the failing gate (`extract.rs:600`, `:750-756`, - `:808`) and its numeric evidence. -- Pre-candidate-id drops (no candidate_id yet): audit at peptidoform-id level in - `predict_frag.rs:103` (RowOut match) and `:134` (retain). -- Compete losers: complement of `keep` at `compete.rs:101` (winner - `and_modify`/`or_insert` at `:92-98`); `emit_competition_audit` - (`config.rs:623`) is the flag. -- Rescore/report flags: per-candidate q outcomes at `rescore.rs:220-238`; the - terminal reported/rejected state at `report.rs:91-107`. -- IO: `mumdia-io/table.rs` needs a null-aware `opt_i32` reader for a nullable - reason column (`Col::OptI32` exists at `table.rs:33`; only `opt_f64`/`opt_*` - readers check `is_null` today), or use `Col::Str`/`Col::Bool`. - -### 4.2 Top-K peaks - -- DONE (`b183fec`): when `retain_top_peaks > 1`, `extract` enumerates top-K peak - groups with `peaks::enumerate_peaks` (`extract.rs:927-929`) and writes them to - `.peaks.parquet` (`extract.rs:1068`). This is a diagnostic sidecar; the - scored `psms_extracted` path still reports the single heuristic apex. -- Remaining hook (close the loop): the single-argmax apex selection still governs - the reported PSM. Feed a top-K peak back into the scored path via a re-selection - pass (the peak-selection model `scripts/peak_selection_model.py` is the offline - prototype); `peaks::enumerate_peaks` (`peaks.rs:52`) is the ready-made pure - function. -- Emission: `CandOut` (`extract.rs:569`), single-row append (`extract.rs:926-958`), - `psms_extracted` writer (`extract.rs:960`), chrom writer keyed by candidate_id - (`extract.rs:986`, key `:989`). Add a `peak_index`/`peak_rank` column; candidate_id - ceases to be unique. -- Downstream: `features.rs:524-543` chrom grouping must key by `(candidate_id, - peak)`; compete group key (`compete.rs:72`) and rescore grouping (`rescore.rs:194` - peptide, `:208` protein) must include `peak_rank` or explicitly collapse peaks. -- Config: `extract.retain_top_peaks` (`config.rs:528`). - -### 4.3 Fragment claimant / conflict graph - -- Per-peak conflict set: `claimants` buffer (`extract.rs:304`; per-path at - `:337-347`/`:398-408`/`:443-452`). Two-pass arbitration loop `extract.rs:456-511` - already picks a winner and computes shares; record edges `(scan_rt, peak_mz, - frag, winner_cid, {loser_cids}, shares)` there. -- Contested scalar already flows: `contested` map (`extract.rs:308`, `:479-485`, - `:814-817`) -> `contested_frac` column -> `features.rs:509` (read) / `:711` - (`peak_contested_frac` push). Extend with `n_claimants`, `unique_fragment_count`, - competitor score/margin the same way. -- Two-pass is active only when `peak_claim` is a `Coelution*` variant or - `emit_contested_features = true` (`extract.rs:311`). Config surface: - `PeakClaim` (`config.rs:187`), `emit_contested_features` (`config.rs:503`), - `peak_claim_margin` (`config.rs:507`). -- Feature side: add `Evidence` fields (`features.rs:269`) + a new family module - following the `NAMES` + `values(&Evidence)` contract and append to `FAMILIES` - (`features.rs:49`); dedup/schema/PIN flow automatically. - -### 4.4 Competition modes - -- Config scaffolding present: `CompeteConfig.mode/margin/unique_evidence_min_fragments/ - emit_competition_audit` (`config.rs:613-623`), `CompetitionMode` enum - (`config.rs:646`, `from_token` `:667`), `CompeteGroupBy` (`config.rs:681`). -- Consume in the winner loop `compete.rs:73-101`: `None`/`FeaturesOnly` set - `keep = 0..nrows` (no removal); `MarginGated` replaces the strict `prelim[i] > - prelim[*w]` (`compete.rs:95`) with a margin test; `UniqueEvidence` keeps a loser - that carries enough independent fragment evidence (needs the claimant graph and - a `unique_fragment_count` feature). `group_by` stays orthogonal (equivalence - class); `mode` is the removal policy. WIRED (`de5ae2b`) via the pure - `resolve_competition()`; `UniqueEvidence` currently approximates unique evidence - from the existing features pending the claimant graph (§4.3). - -### 4.5 Feature registry - -- `FAMILIES` (`features.rs:49`), `active_features` (`:201`), `extended_names` - (`:93`), `reserved_names` (`:67`), `feature_schema_id` (`:220`, blake3), - `FeatureSchema` (`:226`) persisted as `.schema.json` (`:755`) and read - back (`FeatureSchema::read`, `:233`). -- The schema is carried compete (`compete.rs:38`/`:120`) -> rescore - (`rescore.rs:64`) so the classifier never runs under a mismatched set. New - columns require no rescore change: rescore consumes every schema column - uniformly (`rescore.rs:75`, PIN at `:507`). - -### 4.6 Calibration - -- Mass: `masscal.json` writer (`search_seed.rs:186`), consumed at - `extract.rs:280-293` (offset applied to every probe m/z; learned tol sets the - index build tolerance). Uncertainty is only the tolerance width + `n_dev`. -- RT: `cal.json` writer (`rt_im_train.rs:149`), `run_windows` writer - (`rt_im_train.rs:135`). `w_rt` is a single global scalar applied uniformly - (`rt_im_train.rs:124-133`); there is no per-candidate RT uncertainty. -- Hook for per-candidate uncertainty (spec 03 §8.1 / 01 §3.5): capture the full - residual distribution into `cal.json` (`rt_im_train.rs:104-114`) and - `masscal.json` (`search_seed.rs:176-185`); add a `pred_sigma` field to - `Evidence` (`features.rs:279`) fed by a new library/chromatogram column. -- Config: `CalibrationMethod` (`config.rs:97`), `RtImTrainConfig` (`config.rs:399`), - `finetune_deeplc` (`config.rs:416`). - -## 5. Differences between the code and the sensitivity_plan docs - -- Competition is one stage, not the spec's taxonomy. Spec 04 §2 asks for seven - separate competition stages with separate logs (chromatographic peak, duplicate - precursor, peptide interference, peptidoform, localization, target-decoy, - reporting dedup). Reality: `compete.rs` does ONLY within-label winner-take-all - variant collapse on `prelim_score`, controlled by `CompeteGroupBy`. Peptide/ - protein grouping and duplicate removal are folded into `grouped_q` - (`rescore.rs:367`) and the `report` gates, not separate stages. -- Target-decoy "competition" is not a competition here. Spec 01/04 list - target-decoy competition as a distinct stage. In the code the label is part of - the compete key (`compete.rs:74-78`) precisely so targets and decoys never - compete head-to-head; the target-decoy relationship is realized as q-values in - `fdr::target_decoy_q` (`fdr.rs:7`), and the target-decoy q-value is the trusted - FDR estimate. The spec's `passed_target_decoy_competition` stage flag therefore - has no producing stage; `audit.rs` maps it to `q_value <= threshold` instead - (`audit.rs:146-155`). -- Competition happens before rescoring, not after. Spec 04 §11 and §10 recommend - competition after initial rescoring. In the code `compete` runs before `rescore` - (`run.rs:243` before `:252`) and picks the winner on `prelim_score`, a heuristic - (`features.rs:722`). A candidate a trained classifier would rank higher can be - eliminated before the classifier ever sees it. This is the single largest - in-stage sensitivity risk and the reason the `CompetitionMode::None`/`FeaturesOnly` - modes now exist (`de5ae2b`): selecting them preserves every candidate so the - rescorer, not `prelim_score`, arbitrates. Moving competition to AFTER an initial - rescoring pass (spec 04 §11) remains future work. -- One apex per candidate in the SCORED path. Spec 01 §3.1 hypothesizes that a - single apex is chosen too early; confirmed for the default path: the - `psms_extracted` table still emits one apex PSM per candidate. Top-K peak - retention is now wired (`retain_top_peaks > 1` writes `.peaks.parquet` via - `peaks::enumerate_peaks`), but the retained peaks are a diagnostic sidecar; the - loop is not yet closed, i.e. the engine still reports the heuristic apex and does - not re-select a peak from the top-K set before features/compete/rescore. Closing - it needs full per-peak features and a re-selection pass (see `NEXT_STEPS.md`). -- Registry metadata is thinner in code than in spec. Spec 03 §2 wants - `requires_calibration`, `uses_cross_run_information`, `missing_value_policy`, - and `computational_cost` per feature; the code registry (`FAMILIES`) stores only - ordered names + a `values` fn. Direction and level are documented in - `FEATURE_REGISTRY.md`/`feature_registry.yaml`, not in the code. `missing_value_policy` - is de facto "fill with 0.0" everywhere (see the features drop table), which the - spec's leakage/undefined-zero concern flags. -- Calibration is a global pre-CV fit. Spec 03 §3 requires calibration to be fit - within each training fold. The code fits standardization within-fold - (`rescoring.rs:120`) but mass recalibration, RT calibration, and the DeepLC - iRT fine-tune are one-shot whole-run fits whose derived features enter every - fold (see `FEATURE_REGISTRY.md` §4). This is a documented leakage path, not yet - addressed. -- "Candidate" scope. In the spec a candidate spans the whole search space; in the - code `candidate_id` exists only from `predict_frag` onward (minted at - `predict_frag.rs:163`). Digest/peptidoforms losses (`PEPTIDE_NOT_GENERATED`, - `MODIFICATION_NOT_ALLOWED`, `CHARGE_OUT_OF_RANGE`) precede candidate identity and - must be audited at `base_peptide_id`/peptidoform-id level, as `audit.rs` notes. diff --git a/sensitivity_plan/BENCHMARK_GUIDE.md b/sensitivity_plan/BENCHMARK_GUIDE.md deleted file mode 100644 index a69947b..0000000 --- a/sensitivity_plan/BENCHMARK_GUIDE.md +++ /dev/null @@ -1,291 +0,0 @@ -# Sensitivity Benchmark Guide - -How to run the diagnostics added by the sensitivity program and how to read them. -All tools are non-destructive: they read existing artifacts and change no engine -output. Python tools use the `py312_mumdia` conda interpreter (pyarrow, pandas, -numpy, scikit-learn). - -Paths below use the E. coli / HYE example in `C:/proteobench/out_ecoli/`; substitute -your own run directory. - -## 1. Candidate audit (identification-loss waterfall) - -Reconstructs, per candidate, the pipeline stage flags and the EARLIEST rejection -reason from the artifact chain (library -> psms -> competed -> scored). - -``` -mumdia audit \ - --library-precursors lib/lib_precursors_ft.parquet \ - --psms out_ecoli/psms.parquet \ - --competed out_ecoli/comp.parquet \ - --scored out_ecoli/scored.parquet \ - --out out_ecoli/candidate_audit.parquet \ - --q 0.01 --run-id ecoli --entrapment-substr _HUMAN -``` - -Or automatically as part of a full run: set `extract.emit_candidate_audit = true` -in the config; `mumdia run` then writes `candidate_audit.parquet` after rescore. - -Outputs: -- `candidate_audit.parquet`: one row per candidate with `run_id, precursor_id, - modified_sequence, charge, target_decoy_label, entrapment_label, - candidate_generated, traces_extracted, peak_generated, peak_selected, - variant_selected, target_decoy_winner, passed_precursor_fdr, passed_peptide_fdr, - reported, rejection_reason`. -- `.metrics.json`: the waterfall counts + stage recalls. - -Example waterfall (E. coli file vs the 8.3M-candidate HYE library): - -``` -search_space = 8,334,126 extracted = 341,754 reported = 8,568 -NO_PEAK_GROUP = 7,992,372 FAILED_PRECURSOR_FDR = 332,120 -FAILED_PEPTIDE_FDR = 1,066 REPORTED = 8,568 -``` - -Reading it (spec 05 §5 decision rules): -- Large `NO_PEAK_GROUP` with a cross-species library is expected (most library - peptides are absent from a single-species sample). To make it actionable, restrict - the audit input library to candidates a reference tool (DIA-NN) identifies, or - stratify by `entrapment_label`. -- Large `FAILED_PRECURSOR_FDR` among candidates that DID extract points at scoring / - decoy / FDR (prioritize rescoring, decoy design). Large `OUTCOMPETED_*` points at - competition (try `compete.mode = none`). Large extraction loss among candidates - that should be present points at peak selection (top-K) or calibration. - -Limitation: at artifact resolution, extraction losses collapse to `NO_PEAK_GROUP`. -The in-extract audit sidecar (NEXT_STEPS #2) refines them to `NO_FRAGMENT_TRACES` / -`NO_VALID_FRAGMENTS` / `PEAK_NOT_SELECTED` / `RT_PRUNED`; `mumdia audit` already -reads `.audit.parquet` when present. - -## 2. Reference-apex top-K peak recall - -Quantifies the top-K peak opportunity: how often the selected apex is the strongest -peak, and (with a DIA-NN report) whether the reference apex is within the top-K -MuMDIA peaks. - -``` -python scripts/reference_apex_topk.py \ - --psms out_ecoli/psms.parquet \ - --chrom out_ecoli/chrom.parquet \ - [--diann diann_report.tsv] \ - [--out out_ecoli/topk_metrics.json] \ - [--max-candidates 20000] [--bound-fraction 0.333] [--rt-tol-s 10] \ - [--rank-by area|count] -``` - -`--rank-by count` (default `area`) ranks the enumerated peaks by co-eluting -fragment breadth (evidence count) instead of integrated area. This mirrors the -engine's `extract.apex_evidence_rank` option and is the ranking the peak-selection -analysis found stronger than area. - -Self analysis (no DIA-NN needed) on 20,000 E. coli candidates: - -| metric | value | -|---|---| -| mean peaks / candidate | 10.35 (median 9) | -| candidates with >= 2 peaks | 95.2% | -| selected apex is peak rank-1 | 52.5% | -| selected apex in top-3 | 79.2% | -| selected apex in top-5 | 88.3% | -| selected apex in top-10 | 94.8% | -| selected apex in no enumerated peak | 3.5% | - -Reading it: the selected apex is the strongest peak only about half the time, and a -correct alternative peak exists within the top 5 for ~88% of candidates. This is the -quantitative case for `extract.retain_top_peaks = 5` (NEXT_STEPS #1): retain the -alternative peaks and let a peak-selection model choose, instead of committing to one -apex. Run this before and after the top-K wiring to measure recovered peak recall. - -With `--diann`: reports `reference_apex_in_top_{1,3,5,10}` by matching stripped -sequence + charge and comparing the DIA-NN apex RT (auto-converted minutes->seconds) -to the MuMDIA peak apexes. This is the spec's peak-oracle metric (02 §6). - -## 3. Feature-family ablation - -Grouped cross-validated ablation with leakage guards (in-fold standardization, -group by peptidoform+charge, targets at empirical FDP). - -``` -python scripts/feature_ablation.py \ - --features out_ecoli/comp.parquet \ - --registry feature_registry.yaml \ - --out out_ecoli/ablation \ - [--folds 3] [--fdp 0.01] [--model both] [--max-rows 60000] [--clip-sd 8.0] -``` - -Outputs a CSV + JSON per model with columns `feature_family, baseline_identifications, -new_identifications, relative_gain, delta_vs_full, model, recommendation`. - -Smoke (60,000 rows, logreg, 3 folds): 355 features / 17 families; full model = 509, -minimal baseline = 414 targets @1% FDP. Most useful families (removal hurts): -`similarity` (-393), `rt` (-382), `entropy` (-6). On this subset removing -`interference` / `rich` / `coelution` raised the count (+267 / +204 / +119), -indicating redundancy or subset overfit. - -Caveats (spec 03 §9, 05 §7): this is ONE dataset subset and (in the smoke) ONE model. -Do not retain or drop a family on a single favourable subset. Rerun with `--model -both`, all rows, and a second dataset before acting; a family that only helps on one -subset or whose gain flips sign across models/datasets is not a keep. - -## 4. Empirical entrapment FDP (the acceptance gate) - -Already present. Trains on E. coli targets vs a human-TRAIN half and evaluates on the -unseen human-TEST half, giving a leakage-free identification count at a genuinely -controlled FDP. - -``` -python scripts/entrapment_holdout.py out_ecoli/comp.parquet --q 0.01 -``` - -Use this as the accept/reject gate for every change (spec 05 §6): a change ships only -if held-out entrapment identifications rise without FDP inflation, reproduced on a -second dataset. The on-disk SWATH TTOF file is NOT a valid second dataset here: it -produced 0 IDs against the Ox HYE library (the sample does not match the library). A -genuine second labelled run (a matching TTOF library, or a ProteoBench HYE run) is -still required for the held-out reproduction criterion. - -## 5. End-to-end recipe for one experiment (spec 05) - -1. `mumdia run ... --config ` (set `extract.emit_candidate_audit=true`). -2. `mumdia audit ...` (or read the run's `candidate_audit.parquet`) -> waterfall. -3. `python scripts/reference_apex_topk.py ...` -> peak recall. -4. `python scripts/feature_ablation.py ...` -> family contributions. -5. `python scripts/entrapment_holdout.py ...` -> honest FDP + identification count. -6. Change ONE component (e.g. `compete.mode`, `retain_top_peaks`, a feature family), - rerun 1-5, and compare at matched empirical FDP. Keep raw candidate outputs. - -## 6. Diagnostic and analysis tool reference - -Every diagnostic script in `scripts/` added by the sensitivity program, with its -one-line purpose and CLI. All are non-invasive (read artifacts, write new files, -never modify the engine or its inputs) and use the `py312_mumdia` interpreter. -The first three are covered in detail above; the rest are documented here. - -### `reference_apex_topk.py` (top-K peak recall) - -Detailed in §2. How often the selected apex is the strongest peak; with `--diann`, -whether the reference apex is within the top-K MuMDIA peaks. - -``` -python scripts/reference_apex_topk.py --psms psms.parquet --chrom chrom.parquet \ - [--diann report.tsv] [--out topk.json] [--rank-by area|count] [--max-candidates N] -``` - -### `feature_ablation.py` (feature-family ablation) - -Detailed in §3. Grouped cross-validated per-family ablation at empirical FDP. - -``` -python scripts/feature_ablation.py --features comp.parquet --registry feature_registry.yaml \ - --out ablation [--model both] [--folds 3] [--fdp 0.01] [--max-rows 60000] -``` - -### `feature_audit.py` (per-feature data-quality audit) - -Per-feature missingness / quantiles / constant detection, target-decoy-entrapment -separation, redundancy clusters, and leakage / intensity warnings (spec 03 §5). - -``` -python scripts/feature_audit.py --features comp.parquet --registry feature_registry.yaml \ - [--out DIR] [--max-rows N] [--entrapment-substr _HUMAN] [--real-substr _ECOLI] \ - [--redundancy-threshold 0.95] -``` - -Finding on E. coli: 355 features audited, 3 constant, 0 leakage-flagged -(decoy-vs-entrapment gap <= 0.043), 54 redundancy clusters. - -### `benchmark_report.py` (self-contained HTML report) - -Assembles the audit waterfall, top-K recall, ablation, and entrapment FDP into one -portable HTML file (spec 02 §8). All inputs optional; each present section renders. - -``` -python scripts/benchmark_report.py --out report.html \ - [--audit-metrics audit.metrics.json] [--audit candidate_audit.parquet] \ - [--topk topk.json] [--ablation ablation] [--entrapment holdout.txt] [--title "..."] -``` - -### `candidate_diagnostics.py` (per-candidate bundle) - -For selected `candidate_id`s, exports overlaid fragment chromatograms -(`fragments.png`), predicted-vs-observed intensities, and a `candidate.json` with -metadata + all features (spec 02 §9). - -``` -python scripts/candidate_diagnostics.py --chrom chrom.parquet --psms psms.parquet \ - [--comp comp.parquet] [--scored scored.parquet] \ - [--candidates 12,34,56 | --candidates-file ids.txt] [--out candidate_diag] -``` - -### `search_space_manifest.py` (P0.1 search-space parity) - -Derives an effective search-space manifest from a MuMDIA library and optionally -compares it to a declared manifest or a DIA-NN report; `--fail-on-mismatch` exits -nonzero on a benchmark-invalidating difference. - -``` -python scripts/search_space_manifest.py --library-precursors lib_precursors.parquet \ - [--out manifest.yaml] [--compare declared.yaml] [--diann report.parquet] \ - [--fail-on-mismatch] -``` - -### `normalize_output.py` (P0.2 common-schema converter) - -Converts a MuMDIA scored table to the spec 02 §4 common schema (`run_id, -precursor_id, stripped_sequence, modified_sequence, charge, ..., q_value, quantity`). - -``` -python scripts/normalize_output.py --scored scored.parquet [--out normalized.parquet] \ - [--run-id ecoli] [--reported-only | --all-candidates] [--q 0.01] -``` - -`--reported-only` on E. coli keeps 9,540 rows (q <= 0.01); default keeps all -candidates. - -### `conflict_features.py` (cross-candidate conflict features) - -Fragment-claimant index + peak-group conflict graph + contested / unique / ambiguity -features (spec 04 §5, P2.1-2.3, P5.5), one row per candidate to `conflict.parquet`, -joinable into rescoring by `candidate_id`. - -``` -python scripts/conflict_features.py --psms psms.parquet --chrom chrom.parquet \ - [--comp comp.parquet] [--out conflict.parquet] [--frag-tol-ppm 20] [--rt-window-s 30] \ - [--max-candidates N] -``` - -### `localization.py` (modification-localization competition) - -Groups localization variants (same stripped sequence + mod multiset + charge, -different sites) and scores site-determining ions (spec 06 P6.2) to -`localization.parquet`. - -``` -python scripts/localization.py --psms psms.parquet --chrom chrom.parquet \ - [--out localization.parquet] [--rt-window-s 30] [--max-candidates N] -``` - -Finding on E. coli: 0 ambiguity groups (as expected for an unmodified search). - -### `peak_selection_model.py` (top-K peak ranker) - -Grouped out-of-fold ranker over the `.peaks.parquet` table emitted by -`extract` when `retain_top_peaks > 1` (spec 03 §6): given several retained peaks for -one precursor, which is correct? Reports top-1 / top-3 accuracy vs raw -evidence-count and area rankings. Honest labels need `--diann`. - -``` -python scripts/peak_selection_model.py --peaks psms.parquet.peaks.parquet \ - --psms psms.parquet [--diann report.tsv] [--folds 3] [--rt-tol-s 10] [--out sel.json] -``` - -Finding on E. coli (weak self-label, 237,328 candidates): learned top1 0.426 / -top3 0.802; evidence-rank 0.418 beats area-rank 0.390, consistent with the argument -that peak intensity is chimeric. - -## Determinism and cost - -- All Rust stages are deterministic under a fixed seed; `mumdia audit` is a pure join. -- The Python tools seed RNG and use stable (SHA1) fold hashing. -- `reference_apex_topk.py` and `feature_ablation.py` support `--max-candidates` / - `--max-rows` for fast passes; full runs over ~1.3M rows are heavier (minutes). diff --git a/sensitivity_plan/FEATURE_REGISTRY.md b/sensitivity_plan/FEATURE_REGISTRY.md deleted file mode 100644 index a0cda4c..0000000 --- a/sensitivity_plan/FEATURE_REGISTRY.md +++ /dev/null @@ -1,648 +0,0 @@ -# MuMDIA Feature Registry - -Companion to `ARCHITECTURE_MAP.md`. This registry documents every scoring feature -MuMDIA can compute for a PSM, its family, its level, its expected monotone -direction, and its source file. It exists to satisfy the sensitivity program's -feature-registry requirement (spec `03_feature_evaluation.md` §2): before any -feature is added, removed, or ablated, the current set must be enumerated with -enough metadata to run the leakage checks (§3), the evidence ladder (§5), and the -family ablations (§4). - -A machine-readable copy of this table is `feature_registry.yaml` at the repository -root (one entry per feature: `family`, `level`, `direction`, `source_file`). - -> Update: the Extended battery now has **383** features (was 356 at first -> writing; the registry documents 383 including the Minimal/Rich tiers). Two -> families were added this session (append-only, so the Extended-count test is -> dynamic and stays green): -> - **apex_dispersion** (13, `stages/features/apex_dispersion.rs`, P5.3): per- -> fragment apex-RT scatter (std/mad/max/mean deviation, agreement fraction), -> precursor-fragment apex delta, and consensus peak shape (symmetry, tailing, -> local maxima, shoulder, FWHM, truncation, apex position in window). -> - **mass_uncertainty** (10, `stages/features/mass_uncertainty.rs`, P5.1/P5.2): -> fragment mass-error distribution (median/abs-median/std/IQR/max/range), -> effective fragment count, evidence concentration, and fraction of top-3/top-5 -> predicted ions observed (breadth of the strong ions). -> -> These are computed from the per-PSM `Evidence` (peak-bounded traces + mass -> errors) and are interference-resistant (shape and breadth, not absolute -> intensity). They are registered in `feature_registry.yaml`; the per-family -> tables below predate them. - -## 1. Feature-set tiers - -The active feature set is selected by `features.set` (`FeatureSet` enum, -`rust/mumdia/crates/mumdia-core/src/config.rs:110`). Tiers are cumulative: - -| Tier | `FeatureSet` | Count | Definition | -|---|---|---:|---| -| Minimal | `Minimal` (default) | 14 | RT error, matched fragments, co-elution, apex intensity, library agreement, metadata (`MINIMAL_FEATURES`, `features.rs:149`). | -| Rich | `Rich` / `Custom` | 44 | Minimal + 30 (`RICH_EXTRA`, `features.rs:167`). `Custom` is aliased to `Rich`. | -| Extended | `Extended` | 356 | Rich + the 12-family Evidence battery (deduplicated) + 4 psms-derived / cross-candidate extras. Enabled by `--profile dia`. | - -The tier sizes are asserted by the `feature_sets_sized` test -(`features.rs:1263`): `Minimal = 14`, `Rich = 44`, and -`Extended = 14 + 30 + extended_names().len() + 4`. - -This registry documents 360 feature definitions across 17 families. Four of them -(`spectral_angle`, `rt_error_abs`, `peptide_length`, `seed_identified` in their -extended-family form) collide with reserved Minimal/Rich names and are dropped -from the scored schema (see §3 of the mechanism below), so the Extended tier -scores 356 distinct columns. - -## 2. Registry mechanism (how the schema is built and frozen) - -All feature logic lives in `rust/mumdia/crates/mumdia/src/stages/features.rs` and -the per-family modules under `stages/features/`. - -- `FAMILIES` (`features.rs:49`) is the ordered, append-only const array of the 12 - extended-battery families: `[(NAMES, values_fn)]`. Family order defines the - frozen extended-schema column order. New families are appended here. -- Each family module exposes the fixed contract `pub const NAMES: &[&str]` and - `pub fn values(&Evidence) -> Vec` of matching length. `Evidence` - (`features.rs:269`) is the per-PSM input struct handed to every family. -- `extended_names()` (`features.rs:93`) deduplicates the family names: it drops - names colliding with the reserved Minimal/Rich set (`reserved_names`, - `features.rs:67`) and cross-family repeats (keep-first). The four dropped - collisions above are removed here; their computed values are discarded. -- `active_features(set)` (`features.rs:201`) assembles the ordered active column - list for the configured tier. -- `feature_schema_id(cols)` (`features.rs:220`) is the hashed feature-schema - mechanism: blake3 of the comma-joined ordered active column names. Any change - to the set (add/drop/reorder) changes the id. -- `FeatureSchema` (`features.rs:226`) is the companion record - `{feature_columns, schema_id}`, written to `.schema.json` - (`features.rs:755`) and read back in compete and rescore - (`FeatureSchema::read`, `features.rs:233`) so the classifier never runs under a - mismatched feature set. - -`level` values: `fragment` (per-fragment spectral/co-elution kernels), -`peak` (peak-bounded chromatographic quantities), `precursor` (MS1 / charge), -`candidate` (identity/metadata), `run` (run-context). `direction` is the expected -monotone sign toward a correct target: `higher_better`, `lower_better`, -`neutral`, or `?` when undetermined. - -## 3. Feature tables by family - -Grouped by family in schema (first-appearance) order. Notes are abbreviated from -the inventory; source file is the basename under `stages/features/` (or -`features.rs` for the Minimal/Rich/extra rows). Rows marked "DROPPED from schema" -are computed but removed by the dedup at `features.rs:93` because they collide -with a reserved name. - -### minimal (14) - -Minimal tier base features (RT error, matched fragments, co-elution, apex intensity, library agreement, metadata). Always active. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `rt_error_abs` | candidate | lower_better | `features.rs` | \|apex_rt - rt_pred_cal\|; pushed at line 663 | -| `rt_error_rel` | candidate | lower_better | `features.rs` | rt_err/gradient; line 664 | -| `n_matched_fragments` | candidate | higher_better | `features.rs` | line 665 | -| `coelution_run` | peak | higher_better | `features.rs` | from psms; line 666 | -| `log_apex_intensity` | peak | higher_better | `features.rs` | line 667 | -| `frag_corr` | fragment | higher_better | `features.rs` | pearson(obs_apex,pred); CONTESTED/interference-relevant; line 668 | -| `frag_cosine` | fragment | higher_better | `features.rs` | line 669 | -| `spectral_angle` | fragment | higher_better | `features.rs` | normalized similarity in [0,1]; line 670; RESERVED name (shadows similarity::spectral_a... | -| `coelution_mean` | fragment | higher_better | `features.rs` | mean pairwise trace pearson; line 671 | -| `coelution_best` | fragment | higher_better | `features.rs` | line 672 | -| `n_coelution_above` | fragment | higher_better | `features.rs` | count pairwise corr>=coelution_corr_threshold; line 673 | -| `charge` | precursor | neutral | `features.rs` | line 674 | -| `peptide_length` | candidate | neutral | `features.rs` | line 675; RESERVED (shadows novel::peptide_length) | -| `n_proteins` | candidate | lower_better | `features.rs` | protein-group multiplicity; line 676 | - -### rich (30) - -Rich tier additions (library agreement, xcorr, ion-series sums, mass error, MS1 isotope, S/N, seed corroboration, DIA-NN profile/interference proxies). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `library_norm_manhattan` | fragment | lower_better | `features.rs` | line 677 | -| `library_rmsd` | fragment | lower_better | `features.rs` | line 678 | -| `xcorr_coelution` | fragment | lower_better | `features.rs` | mean abs xcorr lag; line 679 | -| `xcorr_shape` | fragment | higher_better | `features.rs` | line 680 | -| `sum_b_intensity` | fragment | higher_better | `features.rs` | line 681 | -| `sum_y_intensity` | fragment | higher_better | `features.rs` | line 682 | -| `diff_by_intensity` | fragment | neutral | `features.rs` | sum_b-sum_y; line 683 | -| `n_b_ions` | fragment | higher_better | `features.rs` | line 684 | -| `n_y_ions` | fragment | higher_better | `features.rs` | line 685 | -| `weighted_mass_error` | fragment | lower_better | `features.rs` | intensity-weighted mean \|ppm\|; line 686 | -| `mean_mass_error` | fragment | lower_better | `features.rs` | line 687 | -| `isotope_corr` | precursor | higher_better | `features.rs` | MS1 averagine corr; line 688 | -| `ms1_isom1_ratio` | precursor | lower_better | `features.rs` | iso -1/mono contamination; line 689 | -| `log_mono_ms1` | precursor | higher_better | `features.rs` | line 690 | -| `has_ms1` | precursor | higher_better | `features.rs` | line 691 | -| `log_sn` | peak | higher_better | `features.rs` | apex vs median trace; line 692 | -| `n_observations` | peak | neutral | `features.rs` | scans in peak; line 693 | -| `base_width_rt` | peak | neutral | `features.rs` | line 694 | -| `seed_score` | candidate | higher_better | `features.rs` | from seed_psms map; line 695 | -| `seed_identified` | candidate | higher_better | `features.rs` | seed spectrum_q<=0.01 flag; line 696; RESERVED (shadows novel::seed_identified) | -| `matched_fraction` | fragment | higher_better | `features.rs` | n_matched/n_pred; line 700 | -| `profile_cos` | fragment | higher_better | `features.rs` | DIA-NN pCos; line 702 | -| `ref_corr` | fragment | higher_better | `features.rs` | DIA-NN pTimeCorr; line 703 | -| `best_ref_corr` | fragment | higher_better | `features.rs` | line 704 | -| `low_frag_coel` | fragment | higher_better | `features.rs` | pResCorr proxy; line 705 | -| `evidence` | fragment | higher_better | `features.rs` | summed frag-vs-ref corr (DIA-NN Evidence); CONTESTED-relevant; line 706 | -| `contrast_min` | fragment | higher_better | `features.rs` | INTERFERENCE: min frag-vs-others corr (low=interfered); line 707 | -| `resid_corr` | fragment | lower_better | `features.rs` | INTERFERENCE: mean residual pairwise corr (high=shared interferent); line 708 | -| `coel_clean` | fragment | higher_better | `features.rs` | INTERFERENCE: co-elution after 1.5*r*ref capping; line 709 | -| `shadow_frac` | fragment | lower_better | `features.rs` | INTERFERENCE: fraction of intensity above cap; line 710 | - -### extended-extra (psms-derived) (1) - -Extended extra carried from the extract psms table (contested fraction), not an Evidence family. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `peak_contested_frac` | candidate | lower_better | `features.rs` | EXISTING CONTESTED FEATURE: reads extract `contested_frac` column (line 509), pushed li... | - -### extended-extra (cross-candidate) (3) - -Extended extra aggregated across candidates of one peptidoform (cross-charge corroboration); the only cross-candidate path in the feature stage. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `n_charge_states` | precursor | higher_better | `features.rs` | cross-candidate aggregate over peptidoform; line 579/729 | -| `charge_multi_flag` | precursor | higher_better | `features.rs` | >=2 charge states; line 580/730 | -| `cross_charge_intensity_log` | precursor | higher_better | `features.rs` | ln(1+summed apex of other charge states); unbounded; line 583/731 | - -### similarity (64) - -FAMILIES[0] observed-vs-library spectral agreement kernels (cosine/pearson/spearman/kendall/distances/presence/area/unbounded evidence). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `spectrum_cosine_matched` | fragment | higher_better | `similarity.rs` | | -| `spectrum_cosine_sqrt` | fragment | higher_better | `similarity.rs` | | -| `spectrum_cosine_log` | fragment | higher_better | `similarity.rs` | | -| `spectral_angle` | fragment | higher_better | `similarity.rs` | DROPPED from schema: collides with reserved minimal::spectral_angle | -| `spectral_angle_sqrt` | fragment | higher_better | `similarity.rs` | | -| `spectral_angle_matched` | fragment | higher_better | `similarity.rs` | | -| `pearson_intensity_matched` | fragment | higher_better | `similarity.rs` | | -| `pearson_intensity_log` | fragment | higher_better | `similarity.rs` | | -| `spearman_intensity` | fragment | higher_better | `similarity.rs` | | -| `spearman_intensity_matched` | fragment | higher_better | `similarity.rs` | | -| `kendall_tau_intensity` | fragment | higher_better | `similarity.rs` | | -| `dot_product_raw` | fragment | higher_better | `similarity.rs` | unbounded | -| `dot_product_norm` | fragment | higher_better | `similarity.rs` | | -| `library_recall_intensity` | fragment | higher_better | `similarity.rs` | predicted intensity fraction observed | -| `manhattan_sim` | fragment | higher_better | `similarity.rs` | 1-0.5*L1 | -| `manhattan_sqrt` | fragment | lower_better | `similarity.rs` | raw L1 of sqrt-renormalized | -| `rmsd_norm` | fragment | lower_better | `similarity.rs` | | -| `mae_norm` | fragment | lower_better | `similarity.rs` | | -| `mse_log` | fragment | lower_better | `similarity.rs` | | -| `mae_weighted_pred` | fragment | lower_better | `similarity.rs` | | -| `abs_diff_q3` | fragment | lower_better | `similarity.rs` | | -| `max_positive_residual` | fragment | lower_better | `similarity.rs` | | -| `chebyshev_dist` | fragment | lower_better | `similarity.rs` | | -| `minkowski_p3` | fragment | lower_better | `similarity.rs` | | -| `bray_curtis` | fragment | higher_better | `similarity.rs` | Ruzicka min/max | -| `bray_curtis_sqrt` | fragment | higher_better | `similarity.rs` | | -| `canberra` | fragment | lower_better | `similarity.rs` | | -| `canberra_matched` | fragment | lower_better | `similarity.rs` | | -| `wave_hedges` | fragment | lower_better | `similarity.rs` | | -| `chi_square_pearson` | fragment | lower_better | `similarity.rs` | | -| `chi_square_symmetric` | fragment | lower_better | `similarity.rs` | | -| `divergence_distance` | fragment | lower_better | `similarity.rs` | | -| `bhattacharyya_coef` | fragment | higher_better | `similarity.rs` | | -| `hellinger` | fragment | lower_better | `similarity.rs` | | -| `squared_chord` | fragment | lower_better | `similarity.rs` | | -| `harmonic_mean_sim` | fragment | higher_better | `similarity.rs` | | -| `jaccard_presence` | fragment | higher_better | `similarity.rs` | predicted vs observed>1%max presence | -| `dice_presence` | fragment | higher_better | `similarity.rs` | | -| `intensity_weighted_pearson` | fragment | higher_better | `similarity.rs` | | -| `regression_slope` | fragment | neutral | `similarity.rs` | cov(l,o)/var(l), ~1 ideal | -| `gini_diff` | fragment | neutral | `similarity.rs` | Gini(obs_matched)-Gini(lib) | -| `wasserstein_mz` | fragment | lower_better | `similarity.rs` | EMD over m/z-ordered CDFs | -| `footrule_norm` | fragment | higher_better | `similarity.rs` | | -| `rank_overlap_top3` | fragment | higher_better | `similarity.rs` | | -| `top1_frag_match` | fragment | higher_better | `similarity.rs` | argmax(l)==argmax(o) flag | -| `top1_predicted_observed` | fragment | higher_better | `similarity.rs` | | -| `frac_top3_predicted_observed` | fragment | higher_better | `similarity.rs` | | -| `count_strong_predicted_absent` | fragment | lower_better | `similarity.rs` | | -| `frac_predicted_absent` | fragment | lower_better | `similarity.rs` | (n_pred-n_matched)/n_pred | -| `cosine_area` | peak | higher_better | `similarity.rs` | peak-XIC trapezoid area vs lib | -| `pearson_area` | peak | higher_better | `similarity.rs` | | -| `spectral_angle_area` | peak | higher_better | `similarity.rs` | | -| `cosine_fullwindow` | peak | higher_better | `similarity.rs` | full-window area vs lib | -| `stein_scott_weighted_dot` | fragment | higher_better | `similarity.rs` | mz^3 * intensity^0.6 weighted cosine | -| `log_dot_product` | fragment | higher_better | `similarity.rs` | UNBOUNDED evidence (anti-saturation) | -| `spectral_log_evidence` | fragment | higher_better | `similarity.rs` | UNBOUNDED DIA-NN Evidence analog | -| `scribe_score` | fragment | higher_better | `similarity.rs` | EncyclopeDIA Scribe, UNBOUNDED | -| `log_dot_product_area` | peak | higher_better | `similarity.rs` | area twin | -| `spectral_log_evidence_area` | peak | higher_better | `similarity.rs` | | -| `scribe_score_area` | peak | higher_better | `similarity.rs` | | -| `cosine_high_ordinal` | fragment | higher_better | `similarity.rs` | ordinal>=3 only (drop co-isolation-prone b1/b2/y1/y2) | -| `cosine_robust_trim1` | fragment | higher_better | `similarity.rs` | cosine after dropping worst-residual fragment; interference trajectory | -| `cosine_robust_trim2` | fragment | higher_better | `similarity.rs` | | -| `cosine_robust_trim3` | fragment | higher_better | `similarity.rs` | | - -### entropy (18) - -FAMILIES[1] spectral-entropy / divergence features (Li entropy sim, JSD/KL/cross-entropy, obs/pred entropy). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `spectral_entropy_similarity` | fragment | higher_better | `entropy.rs` | Li entropy sim | -| `weighted_spectral_entropy_similarity` | fragment | higher_better | `entropy.rs` | | -| `spectral_entropy_similarity_sqrt` | fragment | higher_better | `entropy.rs` | | -| `spectral_entropy_similarity_topk` | fragment | higher_better | `entropy.rs` | top-6 by predicted | -| `spectral_entropy_similarity_area` | peak | higher_better | `entropy.rs` | | -| `jensen_shannon_divergence` | fragment | lower_better | `entropy.rs` | | -| `jeffreys_divergence` | fragment | lower_better | `entropy.rs` | symmetric KL | -| `kl_obs_pred` | fragment | lower_better | `entropy.rs` | | -| `kl_pred_obs` | fragment | lower_better | `entropy.rs` | | -| `cross_entropy_obs_pred` | fragment | lower_better | `entropy.rs` | | -| `obs_spectrum_entropy` | fragment | neutral | `entropy.rs` | | -| `pred_spectrum_entropy` | fragment | neutral | `entropy.rs` | | -| `entropy_diff` | fragment | neutral | `entropy.rs` | | -| `entropy_ratio` | fragment | neutral | `entropy.rs` | | -| `obs_normalized_entropy` | fragment | neutral | `entropy.rs` | Pielou evenness | -| `normalized_entropy_diff` | fragment | neutral | `entropy.rs` | | -| `residual_spectrum_entropy` | fragment | lower_better | `entropy.rs` | | -| `entropy_weight_obs` | fragment | neutral | `entropy.rs` | Li exponent | - -### coelution (38) - -FAMILIES[2] fragment-vs-reference and pairwise co-elution + cross-correlation-lag stats + b/y and charge cross co-elution. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `frag_ref_corr_mean` | fragment | higher_better | `coelution.rs` | | -| `frag_ref_corr_obsweighted` | fragment | higher_better | `coelution.rs` | | -| `frag_ref_corr_min` | fragment | higher_better | `coelution.rs` | | -| `frag_ref_corr_std` | fragment | lower_better | `coelution.rs` | | -| `frag_ref_corr_sq_mean` | fragment | higher_better | `coelution.rs` | | -| `frag_ref_corr_topk_weighted` | fragment | higher_better | `coelution.rs` | | -| `n_frag_ref_corr_above_0_9` | fragment | higher_better | `coelution.rs` | | -| `frac_frag_ref_corr_above_0_8` | fragment | higher_better | `coelution.rs` | | -| `frag_ref_corr_mean_full` | fragment | higher_better | `coelution.rs` | full window | -| `full_vs_peak_corr_gain` | fragment | neutral | `coelution.rs` | | -| `pairwise_coelution_weighted` | fragment | higher_better | `coelution.rs` | == coelution_weighted_mean alias (emitted once) | -| `pairwise_coelution_min` | fragment | higher_better | `coelution.rs` | | -| `pairwise_coelution_median` | fragment | higher_better | `coelution.rs` | | -| `pairwise_coelution_std` | fragment | lower_better | `coelution.rs` | | -| `pairwise_coelution_frac_negative` | fragment | lower_better | `coelution.rs` | INTERFERENCE indicator | -| `pairwise_coelution_hi` | fragment | higher_better | `coelution.rs` | top-half-pred fragments | -| `pairwise_coelution_lo` | fragment | higher_better | `coelution.rs` | | -| `coelution_hi_lo_contrast` | fragment | neutral | `coelution.rs` | | -| `coelution_corr_entropy` | fragment | lower_better | `coelution.rs` | | -| `xcorr_shape_mean` | fragment | higher_better | `coelution.rs` | | -| `xcorr_shape_min` | fragment | higher_better | `coelution.rs` | | -| `xcorr_shape_std` | fragment | lower_better | `coelution.rs` | | -| `xcorr_lag_mean_abs` | fragment | lower_better | `coelution.rs` | | -| `xcorr_lag_std` | fragment | lower_better | `coelution.rs` | | -| `xcorr_lag_iqr` | fragment | lower_better | `coelution.rs` | | -| `xcorr_lag_frac_zero` | fragment | higher_better | `coelution.rs` | | -| `xcorr_lag_max_abs` | fragment | lower_better | `coelution.rs` | | -| `xcorr_lag_entropy` | fragment | lower_better | `coelution.rs` | | -| `ref_xcorr_lag_mean` | fragment | lower_better | `coelution.rs` | each frag vs reference | -| `ref_xcorr_shape_mean` | fragment | higher_better | `coelution.rs` | | -| `observed_sum_vs_template_corr` | fragment | higher_better | `coelution.rs` | | -| `frag_loo_ref_corr_mean` | fragment | higher_better | `coelution.rs` | leave-one-out reference; INTERFERENCE-relevant | -| `frag_loo_ref_corr_min` | fragment | higher_better | `coelution.rs` | | -| `frac_frags_apex_aligned` | peak | higher_better | `coelution.rs` | apex-dispersion-relevant | -| `top3_frag_ref_corr` | fragment | higher_better | `coelution.rs` | | -| `by_cross_coelution` | fragment | higher_better | `coelution.rs` | b-vs-y cross pairs | -| `by_cross_lag_mean` | fragment | lower_better | `coelution.rs` | | -| `charge_cross_coelution` | fragment | higher_better | `coelution.rs` | charge-1 vs charge>=2 pairs | - -### interference (26) - -FAMILIES[3] contested/interference: explained variance, residual fraction, iterative prune, area ratios, apex purity, PCA rank, competing-peak counts. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `explained_variance_ref` | fragment | higher_better | `interference.rs` | CONTESTED family | -| `profile_residual_fraction` | fragment | lower_better | `interference.rs` | | -| `n_interfered_fragments` | fragment | lower_better | `interference.rs` | | -| `corrected_vs_raw_cos` | fragment | neutral | `interference.rs` | | -| `corrected_vs_raw_ratio` | fragment | higher_better | `interference.rs` | | -| `ifs_removed_count` | fragment | lower_better | `interference.rs` | remove_ifs iterative prune | -| `ifs_removed_intensity_frac` | fragment | lower_better | `interference.rs` | | -| `ifs_corr_gain` | fragment | neutral | `interference.rs` | large gain = was interfered | -| `ifs_retained_frac` | fragment | higher_better | `interference.rs` | | -| `matched_frac_after_ifs` | fragment | higher_better | `interference.rs` | | -| `peak_to_full_area_ratio_profile` | peak | higher_better | `interference.rs` | | -| `peak_to_full_area_ratio_frag_mean` | peak | higher_better | `interference.rs` | | -| `peak_to_full_area_ratio_weighted` | peak | higher_better | `interference.rs` | | -| `out_of_peak_intensity_frac` | peak | lower_better | `interference.rs` | | -| `profile_corr_full_vs_peak_delta` | fragment | neutral | `interference.rs` | | -| `frac_frag_ref_corr_below_0_5` | fragment | lower_better | `interference.rs` | | -| `explained_apex_intensity_frac` | peak | higher_better | `interference.rs` | | -| `apex_purity` | peak | higher_better | `interference.rs` | CONTESTED: fraction of apex intensity from coherent fragments | -| `interference_apex_residual_fraction` | peak | lower_better | `interference.rs` | | -| `dominant_frag_ref_corr` | fragment | higher_better | `interference.rs` | | -| `explained_variance_ratio` | peak | higher_better | `interference.rs` | top eigenvalue / trace of Gram (power_top) | -| `second_component_fraction` | peak | lower_better | `interference.rs` | CONTESTED: second eigenvalue fraction = 2-component (chimera) evidence | -| `profile_second_peak_ratio` | peak | lower_better | `interference.rs` | | -| `n_competing_peaks_in_window` | peak | lower_better | `interference.rs` | CONTESTED: competing chromatographic peaks in window | -| `matched_pred_intensity_fraction` | fragment | higher_better | `interference.rs` | | -| `top_pred_frag_matched` | fragment | higher_better | `interference.rs` | | - -### chromatographic (43) - -FAMILIES[4] peak shape: Gaussian/EMG fits, FWHM/asymmetry/tailing, roughness/zigzag, RT moments, apex dispersion, per-fragment gaussianity. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `gaussian_fit_r2` | peak | higher_better | `chromatographic.rs` | | -| `gaussian_cosine` | peak | higher_better | `chromatographic.rs` | | -| `emg_fit_improvement` | peak | neutral | `chromatographic.rs` | tailing indicator | -| `apex_prominence` | peak | higher_better | `chromatographic.rs` | | -| `profile_peak_snr` | peak | higher_better | `chromatographic.rs` | | -| `fwhm_seconds` | peak | neutral | `chromatographic.rs` | | -| `fwhm_to_window_ratio` | peak | neutral | `chromatographic.rs` | | -| `width_at_10pct` | peak | neutral | `chromatographic.rs` | | -| `width_ratio_10_50` | peak | neutral | `chromatographic.rs` | | -| `hwhm_asymmetry` | peak | lower_better | `chromatographic.rs` | abs toward 0 | -| `tailing_factor_usp` | peak | neutral | `chromatographic.rs` | ~1 ideal | -| `asymmetry_factor_10pct` | peak | neutral | `chromatographic.rs` | ~1 ideal | -| `apex_sharpness` | peak | higher_better | `chromatographic.rs` | | -| `apex_curvature` | peak | higher_better | `chromatographic.rs` | | -| `apex_to_boundary_ratio` | peak | higher_better | `chromatographic.rs` | | -| `apex_dominance` | peak | higher_better | `chromatographic.rs` | | -| `zigzag_index` | peak | lower_better | `chromatographic.rs` | | -| `jaggedness` | peak | lower_better | `chromatographic.rs` | | -| `roughness_2nd_deriv` | peak | lower_better | `chromatographic.rs` | | -| `n_local_maxima` | peak | lower_better | `chromatographic.rs` | | -| `modality` | peak | lower_better | `chromatographic.rs` | multimodality valley depth | -| `rt_skewness` | peak | neutral | `chromatographic.rs` | | -| `rt_excess_kurtosis` | peak | neutral | `chromatographic.rs` | | -| `rt_std_seconds` | peak | neutral | `chromatographic.rs` | | -| `mean_mode_offset` | peak | lower_better | `chromatographic.rs` | | -| `fraction_area_within_fwhm` | peak | higher_better | `chromatographic.rs` | | -| `triangle_area_similarity` | peak | lower_better | `chromatographic.rs` | | -| `baseline_fraction` | peak | neutral | `chromatographic.rs` | | -| `peak_completeness` | peak | higher_better | `chromatographic.rs` | window-edge/degeneracy indicator | -| `apex_centering_offset` | peak | lower_better | `chromatographic.rs` | apex distance from window center | -| `intensity_score` | peak | higher_better | `chromatographic.rs` | | -| `total_xic_log` | peak | higher_better | `chromatographic.rs` | | -| `frag_fwhm_cv` | fragment | lower_better | `chromatographic.rs` | | -| `frag_fwhm_mean` | fragment | neutral | `chromatographic.rs` | | -| `frag_apex_rt_dispersion` | peak | lower_better | `chromatographic.rs` | APEX DISPERSION (existing); std of per-fragment apex RTs | -| `frag_apex_rt_dispersion_weighted` | peak | lower_better | `chromatographic.rs` | APEX DISPERSION (existing), pred-weighted | -| `frag_apex_offset_from_profile_mean` | peak | lower_better | `chromatographic.rs` | APEX DISPERSION (existing) | -| `frag_gaussianity_mean` | fragment | higher_better | `chromatographic.rs` | | -| `frag_gaussianity_weighted` | fragment | higher_better | `chromatographic.rs` | | -| `frag_zigzag_mean` | fragment | lower_better | `chromatographic.rs` | | -| `sumtrace_unweighted_gaussian_r2` | peak | higher_better | `chromatographic.rs` | | -| `reference_profile_rt_entropy_peak` | peak | lower_better | `chromatographic.rs` | | -| `reference_profile_rt_entropy_ratio` | peak | neutral | `chromatographic.rs` | | - -### mass_accuracy (17) - -FAMILIES[5] fragment ppm-error distribution + positive mass-evidence. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `median_abs_frag_ppm` | fragment | lower_better | `mass_accuracy.rs` | | -| `signed_mean_frag_ppm` | fragment | neutral | `mass_accuracy.rs` | ~0 ideal | -| `ppm_std` | fragment | lower_better | `mass_accuracy.rs` | | -| `ppm_iqr` | fragment | lower_better | `mass_accuracy.rs` | | -| `ppm_range` | fragment | lower_better | `mass_accuracy.rs` | | -| `max_abs_frag_ppm` | fragment | lower_better | `mass_accuracy.rs` | | -| `intensity_weighted_abs_ppm` | fragment | lower_better | `mass_accuracy.rs` | | -| `intensity_weighted_signed_ppm` | fragment | neutral | `mass_accuracy.rs` | | -| `intensity_weighted_ppm_std` | fragment | lower_better | `mass_accuracy.rs` | | -| `lib_weighted_abs_ppm` | fragment | lower_better | `mass_accuracy.rs` | | -| `frac_frag_within_half_tol` | fragment | higher_better | `mass_accuracy.rs` | uses hardcoded HALF_TOL_PPM=10 | -| `high_ppm_intensity_frac` | fragment | lower_better | `mass_accuracy.rs` | | -| `ppm_intensity_anticorr` | fragment | lower_better | `mass_accuracy.rs` | pearson(\|ppm\|,intensity); real trends negative | -| `mass_error_mz_trend` | fragment | lower_better | `mass_accuracy.rs` | | -| `mean_abs_mz_error_da` | fragment | lower_better | `mass_accuracy.rs` | | -| `mass_evidence_gauss` | fragment | higher_better | `mass_accuracy.rs` | positive evidence; SIGMA_PPM=10 hardcoded | -| `mass_log_evidence` | fragment | higher_better | `mass_accuracy.rs` | UNBOUNDED DIA-NN Mass.Evidence analog | - -### ion_series (34) - -FAMILIES[6] b/y series coverage, ladder runs, complementarity, per-series similarity/co-elution, charge-resolved cosine. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `n_matched_b` | fragment | higher_better | `ion_series.rs` | | -| `n_matched_y` | fragment | higher_better | `ion_series.rs` | | -| `frac_matched_b` | fragment | higher_better | `ion_series.rs` | | -| `frac_matched_y` | fragment | higher_better | `ion_series.rs` | | -| `by_count_balance` | fragment | higher_better | `ion_series.rs` | | -| `by_intensity_ratio` | fragment | neutral | `ion_series.rs` | | -| `by_ratio_agreement` | fragment | lower_better | `ion_series.rs` | tanh log-odds discrepancy | -| `by_ratio_consistency` | fragment | lower_better | `ion_series.rs` | | -| `longest_b_run` | fragment | higher_better | `ion_series.rs` | | -| `longest_y_run` | fragment | higher_better | `ion_series.rs` | | -| `longest_run_max` | fragment | higher_better | `ion_series.rs` | | -| `longest_run_frac_length` | fragment | higher_better | `ion_series.rs` | | -| `series_coverage_b` | fragment | higher_better | `ion_series.rs` | | -| `series_coverage_y` | fragment | higher_better | `ion_series.rs` | | -| `sequence_coverage` | candidate | higher_better | `ion_series.rs` | | -| `series_gap_fraction` | fragment | lower_better | `ion_series.rs` | | -| `by_complement_count` | fragment | higher_better | `ion_series.rs` | | -| `by_complement_mz_consistency` | fragment | lower_better | `ion_series.rs` | ppm dev of b+y from M+2H | -| `by_complement_coelution` | fragment | higher_better | `ion_series.rs` | | -| `ordinal_intensity_concordance_y` | fragment | higher_better | `ion_series.rs` | | -| `ordinal_intensity_concordance_b` | fragment | higher_better | `ion_series.rs` | | -| `series_coelution_y` | fragment | higher_better | `ion_series.rs` | | -| `series_coelution_b` | fragment | higher_better | `ion_series.rs` | | -| `spectral_angle_b` | fragment | higher_better | `ion_series.rs` | | -| `spectral_angle_y` | fragment | higher_better | `ion_series.rs` | | -| `pearson_b` | fragment | higher_better | `ion_series.rs` | | -| `pearson_y` | fragment | higher_better | `ion_series.rs` | | -| `cosine_charge1` | fragment | higher_better | `ion_series.rs` | | -| `cosine_charge2` | fragment | higher_better | `ion_series.rs` | | -| `charge_corr_balance` | fragment | higher_better | `ion_series.rs` | | -| `mean_matched_ordinal_norm` | fragment | neutral | `ion_series.rs` | | -| `by_ion_contiguous_intensity` | fragment | higher_better | `ion_series.rs` | | -| `by_ion_contiguous_lib_frac` | fragment | higher_better | `ion_series.rs` | | -| `both_series_present` | fragment | higher_better | `ion_series.rs` | | - -### ms1 (25) - -FAMILIES[7] MS1 isotope-envelope agreement + MS1/MS2 XIC co-elution/shape (10 XIC features are 0.0 until extract persists ms1_xic). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `ms1_isotope_cosine_apex` | precursor | higher_better | `ms1.rs` | | -| `ms1_isotope_spectral_angle_apex` | precursor | higher_better | `ms1.rs` | | -| `ms1_isotope_chi2_apex` | precursor | lower_better | `ms1.rs` | | -| `ms1_isotope_manhattan_apex` | precursor | higher_better | `ms1.rs` | | -| `iso_ratio_1_0` | precursor | neutral | `ms1.rs` | | -| `iso_ratio_2_0` | precursor | neutral | `ms1.rs` | | -| `iso_plus1_ratio_dev` | precursor | lower_better | `ms1.rs` | | -| `iso_plus2_ratio_dev` | precursor | lower_better | `ms1.rs` | | -| `iso_minus_one_fraction` | precursor | lower_better | `ms1.rs` | co-isolation contamination | -| `iso_overlap_flag` | precursor | lower_better | `ms1.rs` | | -| `log_ms1_mono` | precursor | higher_better | `ms1.rs` | | -| `ms1_total_isotope_log` | precursor | higher_better | `ms1.rs` | | -| `has_ms1_signal` | precursor | higher_better | `ms1.rs` | | -| `ms1_isotope_apex_entropy_3` | precursor | neutral | `ms1.rs` | | -| `ms1_m1_entropy_contribution` | precursor | neutral | `ms1.rs` | | -| `ms1_ms2_time_corr` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted by extract | -| `ms1_ms2_envelope_time_corr` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_iso_coelution` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_ms2_apex_rt_delta` | precursor | lower_better | `ms1.rs` | 0.0 until ms1_xic persisted; was the mis-scaled/unbounded feature, now bounded d/(d+width) | -| `ms1_iso_ratio_stability` | precursor | lower_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_mono_gaussianity` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_ms2_fwhm_ratio` | precursor | neutral | `ms1.rs` | ~1 ideal; 0.0 until ms1_xic persisted | -| `ms1_isotope_corr_xic` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_envelope_over_time_corr` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | -| `ms1_isotope_xic_shape_consistency` | precursor | higher_better | `ms1.rs` | 0.0 until ms1_xic persisted | - -### rt (13) - -FAMILIES[8] RT-agreement (signed/abs/squared, gradient- and peak-width-normalized, profile-apex delta). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `rt_error_signed` | candidate | neutral | `rt.rs` | ~0 ideal | -| `rt_error_abs` | candidate | lower_better | `rt.rs` | DROPPED from schema: collides with reserved minimal::rt_error_abs | -| `rt_error_squared` | candidate | lower_better | `rt.rs` | | -| `rt_error_signed_norm_gradient` | run | neutral | `rt.rs` | | -| `rt_error_abs_norm_gradient` | run | lower_better | `rt.rs` | | -| `observed_rt_raw` | candidate | neutral | `rt.rs` | | -| `predicted_rt_raw` | candidate | neutral | `rt.rs` | | -| `observed_rt_fraction` | run | neutral | `rt.rs` | | -| `predicted_rt_fraction` | run | neutral | `rt.rs` | | -| `rt_error_over_peak_width` | peak | lower_better | `rt.rs` | | -| `rt_error_over_fwhm` | peak | lower_better | `rt.rs` | | -| `rt_diff_profile_apex` | peak | lower_better | `rt.rs` | | -| `predicted_rt_in_gradient` | run | higher_better | `rt.rs` | flag | - -### novel (12) - -FAMILIES[9] seed corroboration + precursor/charge metadata + count/intensity summaries. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `log_seed_hyperscore` | candidate | higher_better | `novel.rs` | | -| `seed_hyperscore_per_matched` | candidate | higher_better | `novel.rs` | | -| `seed_identified` | candidate | higher_better | `novel.rs` | DROPPED from schema: collides with reserved rich::seed_identified | -| `peptide_length` | candidate | neutral | `novel.rs` | DROPPED from schema: collides with reserved minimal::peptide_length | -| `precursor_charge` | precursor | neutral | `novel.rs` | | -| `charge_is_2` | precursor | neutral | `novel.rs` | flag | -| `charge_is_3` | precursor | neutral | `novel.rs` | flag | -| `charge_is_4plus` | precursor | neutral | `novel.rs` | flag | -| `precursor_mass` | precursor | neutral | `novel.rs` | | -| `log_total_matched_intensity` | fragment | higher_better | `novel.rs` | | -| `n_matched_frags` | fragment | higher_better | `novel.rs` | | -| `n_predicted_frags` | fragment | neutral | `novel.rs` | | - -### nonzero (12) - -FAMILIES[10] zero-ignoring variants (per-fragment peak-max spectral agreement, both-positive co-elution) to fix apex-scan-alignment zeros. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `frag_corr_peakmax` | fragment | higher_better | `nonzero.rs` | peak-max obs (fixes apex-scan-alignment zeros) | -| `frag_cosine_peakmax` | fragment | higher_better | `nonzero.rs` | | -| `spectral_angle_peakmax` | fragment | higher_better | `nonzero.rs` | | -| `frag_corr_matched_nz` | fragment | higher_better | `nonzero.rs` | | -| `frag_cosine_matched_nz` | fragment | higher_better | `nonzero.rs` | | -| `peakmax_apex_gain` | fragment | lower_better | `nonzero.rs` | how much apex scan understates peak; high=off-apex/chimera | -| `n_frag_present_inpeak` | fragment | higher_better | `nonzero.rs` | | -| `frac_frag_present_inpeak` | fragment | higher_better | `nonzero.rs` | | -| `coelution_mean_bothpos` | fragment | higher_better | `nonzero.rs` | | -| `coelution_mean_summpos` | fragment | higher_better | `nonzero.rs` | | -| `ref_corr_nz` | fragment | higher_better | `nonzero.rs` | | -| `profile_cos_nz` | fragment | higher_better | `nonzero.rs` | | - -### order_consistency (8) - -FAMILIES[11] prediction-free MS2-XIC rank-stability (per-scan Spearman/Kendall vs apex, top1/2 persistence, argmax entropy). - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `rank_corr_vs_apex_mean` | peak | higher_better | `order_consistency.rs` | prediction-free MS2-XIC rank stability | -| `rank_corr_vs_apex_std` | peak | lower_better | `order_consistency.rs` | | -| `rank_corr_adjacent_mean` | peak | higher_better | `order_consistency.rs` | | -| `kendall_vs_apex_mean` | peak | higher_better | `order_consistency.rs` | | -| `top1_frag_persistence` | peak | higher_better | `order_consistency.rs` | | -| `top2_order_persistence` | peak | higher_better | `order_consistency.rs` | | -| `argmax_frag_entropy` | peak | lower_better | `order_consistency.rs` | 0=one fragment dominates (clean), 1=noisy | -| `self_cosine_vs_apex_mean` | peak | higher_better | `order_consistency.rs` | | - -### peak_scans (2) - -FAMILIES[12] label-blind window-degeneracy indicators (n_peak_scans, peak_window_degenerate); the observability-model reference family. - -| name | level | direction | source file | note | -|---|---|---|---|---| -| `n_peak_scans` | peak | higher_better | `peak_scans.rs` | OBSERVABILITY: label-blind measured-scan count; disambiguates undefined vs genuine zero | -| `peak_window_degenerate` | peak | lower_better | `peak_scans.rs` | OBSERVABILITY: flag when <3 non-empty scans (window-based families collapse to 0) | -## 4. Leakage audit (spec 03 §3) - -Spec 03 §3 lists inputs that must never be used as normal scoring features, and -requires imputation/scaling/calibration/scoring to be fit within each training -fold. The rescore stage -(`rust/mumdia/crates/mumdia/src/stages/rescore.rs`) consumes every column in -`FeatureSchema.feature_columns` uniformly (`rescore.rs:64,75`) with no -label-leakage screen, so leakage prevention currently depends entirely on which -columns the features stage emits, not on a guard in the classifier. - -| Forbidden input (spec 03 §3) | Used as a feature today? | Assessment | -|---|---|---| -| Final q-value | No | `q_value` / `peptide_q_value` / `pg_q_value` are computed after scoring in rescore and written to `psms_scored`; they are not in the feature list. | -| Final posterior error probability | No | Not computed as a feature. | -| Final target-decoy winner | No | Competition winner is not fed back; `label` is used only for training targets/decoys, not as a feature column. | -| Candidate rank generated by the same model | No (with caveat) | `prelim_score` seeds the native semi-supervised loop as `init_score` (`rescore.rs:321`) but is bookkeeping, not a feature column. In the native path it is used only within-fold via `target_decoy_q` on train rows, so it is not a direct leak. It is not an output-model rank fed back to itself. | -| DIA-NN score or identification status | No | The imported DIA-NN library contributes predicted fragment intensities and iRT only; DIA-NN scores/IDs are not ingested as features. | -| Protein evidence computed after precursor scoring | No | `n_proteins` is protein-group multiplicity from the library/digest, available before scoring; `pg_q` is computed after and is not a feature. | -| Cross-run evidence using the held-out run | No | Single-run pipeline. `cross_charge_intensity_log` / `n_charge_states` aggregate charge siblings within the same run, not across runs. No cross-run feature exists. | -| Calibration values fit using the held-out candidate | RISK | RT features (`rt` family, `rt_error_*`) use per-run LOESS/linear RT calibration fit on all confident seed PSMs before cross-validation; mass-accuracy features use the per-run `masscal.json` offset/tolerance; the optional DeepLC iRT fine-tune is likewise global. The native rescorer folds by `base_peptide_id`, so a held-out peptide's own confident seed PSM can be inside the calibration anchor set. This is a genuine, documented pre-CV global-fit leakage path (fdr-rescore map, leakage risk #4). | -| Features computed globally before cross-validation | RISK | Same root cause: search-seed mass recalibration, rt-im-train RT calibration, and DeepLC fine-tune are one-shot whole-run fits, and the derived RT/mass/iRT features carry that global fit into every fold. Standardization itself is fit within-fold in the native path (`fit_standardizer` on train rows only, `rescoring.rs:120`), so the leak is in the upstream calibration, not the scaler. | - -Two further leakage risks recorded in the rescore/fdr stage map, not tied to a -single feature: - -1. Mokapot fold split is not peptide-grouped. `run_mokapot` writes the PIN keyed - on a flat row index (`rescore.rs:515-522`) and mokapot's internal brew CV can - place charge/mod variants of one peptide in different folds. The native - `percolator_lite` path avoids this by folding on `base_peptide_id` - (`rescoring.rs:107`); the mokapot sidecar carries a peptide-level leakage risk - the native path does not. -2. No label-leakage screen exists on the feature columns - (`rescore.rs:64,75`). Any future feature that encodes decoy/reverse status - (for example a sequence-derived quantity that differs systematically for the - documented reverse/scramble decoy scheme) would leak the label directly. This - is also the caution behind spec 04 §9 (competition features must be computed - symmetrically for targets and decoys). New contested/claimant features must be - audited for target/decoy symmetry before they are added. - -## 5. Gaps vs spec 03 §8 (candidate feature families to add) - -Status of each requested family against the current registry. "Exists" means a -populated, scored column; "present-but-dead" means the column is emitted but -returns a constant because an upstream input is not persisted; "partial" means -some members exist and others do not; "missing" means no analog is scored. - -| Spec 03 §8 family | Status | Evidence in the current registry / gap | -|---|---|---| -| 8.1 Uncertainty-normalized residuals | Partial | Signed/abs/squared and gradient- and peak-width-normalized RT residuals exist (`rt` family). They are NOT divided by a per-candidate RT uncertainty: `w_rt` is a single global scalar in `cal.json`, and `Evidence` carries only `e.pred` (a point prediction), no per-fragment `pred_sigma`. Mass errors (`mass_accuracy`) are not normalized by a predicted per-fragment mass uncertainty. Mobility residuals are missing (IM null, 3D MVP). | -| 8.2 Fragment evidence distributions | Mostly exists | `coelution` (38) covers median/min/IQR co-elution and `n_coelution_above` (fraction above threshold); `mass_accuracy` covers median ppm + dispersion; `similarity` covers `frac_top3_predicted_observed`, `library_recall_intensity` (explained predicted intensity), `gini_diff` (evidence concentration); `interference` covers explained observed intensity. Effective-fragment-count as a named feature is the main missing member. | -| 8.3 Apex dispersion | Exists | `chromatographic` has `frag_apex_rt_dispersion`, `frag_apex_rt_dispersion_weighted`, `frag_apex_offset_from_profile_mean`; `coelution` has `frac_frags_apex_aligned`. Precursor-to-fragment apex deviation exists in `ms1` (and the mis-scaled `ms1_ms2_apex_rt_delta` in `novel`, flagged for a fix in CLAUDE.md). | -| 8.4 Peak-shape evidence | Mostly exists | `chromatographic` (43) covers Gaussian/EMG fits, FWHM, asymmetry, tailing, roughness/zigzag, and bimodal detection (a local-maxima proxy). Missing: an explicit peak-truncation indicator, boundary-agreement, and a distinct shoulder score. Consensus profile and fraction-explained-by-consensus are computed internally (`ref_profile` / `interference`) but not all exported as named peak-shape features. | -| 8.5 Interference and contested evidence | Partial | Per-candidate proxies exist: `interference` (26) with contrast/residual/apex-purity/competing-peak counts/PCA rank, plus `peak_contested_frac` (contested intensity fraction from extract two-pass arbitration) and `resid_corr` (residual similarity). Missing the cross-candidate claimant-graph members: number of claimants per fragment, unique-fragment count, unique-fragment intensity fraction, shared-fragment count, shared-trace correlation, local/isolation-window candidate density, and correlation with competing candidate profiles. These require the fragment claimant/conflict graph (spec 04 §5), which extract computes internally (the `contested` map) but does not persist as edges. The scaffolded `CompetitionMode::UniqueEvidence` needs a `unique_fragment_count` feature that does not exist yet. This is the largest feature gap. | -| 8.6 Candidate ambiguity | Missing | No margin/rank-gap features are scored today. `compete` drops losers without recording margins; the new `emit_competition_audit` scaffolding would expose group size, winner/loser scores, and margins, but no ambiguity features (margin-from-best-alternative-peak/peptide/decoy, candidates-within-threshold, score entropy, near-isobaric count, alternative-localization count) are emitted. `n_charge_states` / `charge_multi_flag` are cross-charge corroboration, not ambiguity margins. Note spec 03 §8.6 forbids feeding the final-model margin back into the same model, so these must come from an earlier-stage score. | -| 8.7 MS1 and isotope evidence | Partial | Apex-level isotope agreement exists: `isotope_corr` (averagine correlation), `ms1_isom1_ratio` (incorrect-monoisotope / iso-1 contamination), `log_mono_ms1`, `has_ms1`, and the `ms1` family's isotope-envelope kernels. Present-but-dead: the 10 MS1/MS2 XIC co-elution/shape features (`ms1.rs`, `xic_features`) return 0.0 for every PSM until extract persists `ms1_xic`, so isotope co-elution and precursor-fragment apex agreement over the XIC are not yet live. | -| 8.8 Modification-aware evidence | Missing | No modification-count/class, prediction-training-coverage indicator, modification-specific RT/spectral residual percentile, localization ambiguity, site-determining-ion count/intensity, or alternative-peptidoform count. `novel` carries only `peptide_length` and charge metadata. The peptidoforms stage also drops a second mod at the same position and does no localization scoring, so the upstream evidence is not produced. | -| 8.9 Run context | Partial | Scans-across-peak exists (`n_observations`, `peak_scans::n_peak_scans`, `base_width_rt`). Missing as features: isolation-window width, cycle time, local signal density, local candidate density, run-level calibration quality (`w_rt` / `masscal` live in `cal.json` / `masscal.json` but are not exposed as feature columns), and run-level prediction-residual statistics. Run-context features must be normalized by run when added. | - -### Summary - -Well covered: fragment evidence distributions (8.2), apex dispersion (8.3), -peak shape (8.4), and apex-level isotope evidence (part of 8.7). - -Largest gaps, in priority order matching spec 04 and the sensitivity backlog: - -1. Cross-candidate contested/claimant-graph features (8.5) - blocked on - persisting the extract `contested` conflict edges; unlocks - `CompetitionMode::UniqueEvidence` and margin/ambiguity work. -2. Candidate-ambiguity margins (8.6) - blocked on the top-K peak dimension and - the competition-audit emission. -3. Uncertainty-normalized residuals (8.1) - blocked on persisting a per-candidate - RT/mass/iRT uncertainty (`pred_sigma`) rather than the global `w_rt` scalar. -4. Modification-aware evidence (8.8) - blocked on localization scoring in the - peptidoforms stage. -5. Live MS1 XIC isotope co-elution (8.7) - blocked on extract persisting - `ms1_xic`. -6. Run-context features (8.9) - straightforward to add from `cal.json` / - `masscal.json` and the acquisition grid. diff --git a/sensitivity_plan/IMPLEMENTATION_STATUS.md b/sensitivity_plan/IMPLEMENTATION_STATUS.md deleted file mode 100644 index 8fb82bf..0000000 --- a/sensitivity_plan/IMPLEMENTATION_STATUS.md +++ /dev/null @@ -1,188 +0,0 @@ -# MuMDIA Sensitivity Work — Implementation Status - -Living log for the autonomous sensitivity-improvement session. Updated throughout. - -## Phase 0 — Baseline (recorded) - -- **Repo:** `c:\Users\robbi\OneDrive - UGent\MuMDIA_NG` (git). -- **Branch created:** `feat/sensitivity-improvements`, forked from `fix/audit-correctness-evidence` @ `c6f268f`. - - The base branch carries the audit-correctness fixes + the 64-bit `LargeListF32` chrom fix (commit `c6f268f`). It is **not** on `main` and is **not pushed**. -- **Build target dir:** `C:/Users/robbi/mumdia_build/{debug,release}` (redirected off OneDrive by machine-local `rust/mumdia/.cargo/config.toml`; do not "fix"). -- **Build cmd:** `cd rust/mumdia && cargo build --release`. **Test cmd:** `cargo test`. -- **Baseline tests:** `cargo test` = **64 passed, 0 failed** (50 lib + 2 + 11 + 1 across binaries) — verified this session before any Phase-1 change. No pre-existing failures. -- **Baseline release build:** succeeds (`c6f268f`). -- **Empirical baseline (E. coli file, HYE library, audit-fixed binary), stripped E. coli seqs @1%:** - - mokapot logreg: 7,907 · native_tda: 6,914 · held-out entrapment (honest, unseen null): 7,909. - - DIA-NN reference on same file ≈ 10,072; pre-audit MuMDIA base ≈ 9,042. - - Gate-off probe (Pearson off, presence≥3, native): honest held-out 8,078 (+2.1%) — the Pearson gate discards ~170 real IDs recoverable only by a strong rescorer. - -## Assumptions (conservative; recorded per non-interactive policy) - -- A1. The primary comparison unit is `normalized modified sequence + charge` (spec 01 §2). MuMDIA's `peptidoform` string + `charge` is that key; `base_peptide_id` is the stripped-sequence key. -- A2. "Candidate" in the MuMDIA code == one row keyed by `candidate_id` (a library precursor peptidoform-charge). Extraction emits **one apex-level PSM per candidate** today (spec 01 §3.1 "one apex" hypothesis holds). -- A3. All new behavior is **default-off / K=1-compatible**; production defaults are unchanged unless an ablation proves a change and tests cover it. -- A4. Large candidate-level output uses **Parquet** via the existing `mumdia-io` typed `Col`/`Table` layer (dependency stack already supports it; spec P0 prefers Parquet). -- A5. Diagnostic instrumentation must be near-zero-cost when disabled (guarded by a config flag). - -## Plan (priority order, adapted to the real codebase) - -| # | Work | Spec | Status | Commit | -|---|---|---|---|---| -| 0 | Baseline + branch + status log | P0 | DONE | - | -| M | Architecture map (7-agent parallel workflow) | 01 | DONE | ARCHITECTURE_MAP.md | -| 1 | Rejection-reason enum | P0.3, 01 §4 | DONE | 2f46d6d | -| 2 | Candidate audit table + `mumdia audit` (+ metrics/waterfall) | P0.3/P0.4 | DONE | eb9da89 | -| 3 | Top-K peak enumerator + config `retain_top_peaks` (K=1 compat) + tests | P1 | PARTIAL (enumerator+config+tests done; extract wiring = NEXT_STEPS #1) | 2f46d6d | -| 4 | Competition-mode enum wired in compete (none/features-only/unique-evidence/margin-gated) | P2.4 | DONE | de5ae2b | -| 5 | `ARCHITECTURE_MAP.md`, `FEATURE_REGISTRY.md`, `feature_registry.yaml` | P4.1 | DONE | docs | -| 6 | Reference-apex top-K analysis script (Python) | P0.4, 02 §5 D | DONE | ed44e2a | -| 7 | Feature-ablation runner (Python) | P4.3 | DONE | ed44e2a | -| 8 | `BENCHMARK_GUIDE.md`, `NEXT_STEPS.md` | P7 | DONE | docs | -| 9 | Fragment claimant / conflict features | P2.1-2.3 | DEFERRED -> NEXT_STEPS #4 (nucleus exists: contested_frac) | - | -| 10 | In-extract precise reason emitter | P0.3 | DEFERRED -> NEXT_STEPS #2 (audit reads sidecar already) | - | - -## Completed tasks - -- Phase 0 baseline recorded; dedicated branch `feat/sensitivity-improvements`. -- 7-agent architecture-map workflow (679k tokens); ARCHITECTURE_MAP.md written. -- `RejectionReason` enum (16 codes + Reported sentinel), tested. -- `mumdia::peaks::enumerate_peaks` top-K enumerator, 9 synthetic-chromatogram tests. -- Config: `retain_top_peaks` (+validation), `emit_candidate_audit`, `CompetitionMode` - + `CompeteConfig.mode/margin/unique_evidence_min_fragments/emit_competition_audit`. -- `mumdia audit` stage + subcommand: candidate_audit.parquet + metrics/waterfall, - verified on real E. coli/HYE data (2 tests). -- Competition modes wired into `compete` via pure `resolve_competition()` (7 tests). -- FEATURE_REGISTRY.md (360 features, 17 families) + feature_registry.yaml. - -## Partially completed - -- Top-K peaks: enumerator + config + tests done; NOT wired into `extract` (the - destructive-stage change). Exact hook site documented in NEXT_STEPS.md #1. - -## Tests run - -- `cargo test` baseline: 64 passed / 0 failed. -- `cargo test` after all Rust changes: 87 passed / 0 failed (68 mumdia lib incl - peaks(9)/audit(2)/compete(7), 16 mumdia-core incl rejection(5), 2 integration, 1). -- `mumdia audit` real-data smoke on out_ecoli: waterfall reproduced (below). - -## Key diagnostic result (candidate audit, E. coli file vs HYE library) - -``` -search_space = 8,334,126 extracted = 341,754 (trace_recall 4.1%) reported = 8,568 -waterfall: NO_PEAK_GROUP = 7,992,372 FAILED_PRECURSOR_FDR = 332,120 - FAILED_PEPTIDE_FDR = 1,066 REPORTED = 8,568 -``` - -Interpretation: with an 8.3M-candidate combined-species library searched against a -single-species (E. coli) sample, the vast majority of candidates correctly never -form a peak. The audit now makes every loss category countable and stratifiable, -which is the P0 prerequisite for targeting the recoverable losses (peak selection -and FDR), per the spec's decision rules (05 §5). - -## Key diagnostic result (reference-apex top-K, 20,000 E. coli candidates) - -| metric | value | -|---|---| -| mean peaks / candidate | 10.35 | -| candidates with >= 2 peaks | 95.2% | -| selected apex is peak rank-1 | **52.5%** | -| selected apex in top-3 / top-5 / top-10 | 79.2% / 88.3% / 94.8% | -| selected apex in no enumerated peak | 3.5% | - -Interpretation: the current single-apex selection lands on the STRONGEST peak only -about half the time; a better-ranked alternative peak exists within the top 5 for -~88% of candidates. This is direct quantitative support for the spec's central -hypothesis (01 §3.1) and for `extract.retain_top_peaks = 5` (NEXT_STEPS #1): the -enumerator + config are in place; wiring them into `extract` is the highest-value -remaining change. (Whether recovering those peaks raises IDENTIFICATIONS at matched -empirical FDP must still be confirmed with the entrapment gate.) - -## Feature-family ablation (60,000 rows, logreg, 3-fold grouped CV) - -355 features / 17 families; full model 509 vs minimal-baseline 414 targets @1% FDP. -Most useful (removal hurts): similarity (-393), rt (-382), entropy (-6). On this -subset, removing interference / rich / coelution raised the count (+267 / +204 / -+119), indicating redundancy or subset overfit. One-subset, one-model result: not an -action, a lead. Rerun with `--model both`, all rows, and a second dataset (spec 05 §7) -before dropping any family. - -## Known limitations / risks (running) - -- Empirical FDP validation here uses the single E. coli file + HYE entrapment null (one dataset); the spec's held-out multi-dataset reproduction cannot be completed in this environment without additional data. -- Real DIA-NN reference-apex tables are not loaded into the repo; the reference-apex top-K analysis is implemented as a runnable module pending a supplied DIA-NN report. - -## Recommended next steps - -- See `NEXT_STEPS.md` (written at end of session). - ---- - -## Session 2 (implement-more push) — full backlog status - -Branch `feat/sensitivity-improvements`. All Rust changes keep the suite green -(95 tests) and are default-off / behaviour-preserving unless noted. - -### Done this push -- P0.1 `scripts/search_space_manifest.py` (effective manifest + DIA-NN/declared parity, `--fail-on-mismatch`). -- P0.2 `scripts/normalize_output.py` (MuMDIA scored -> spec 02 §4 common schema; `--reported-only` = 9,540 on E. coli). -- P4.2 `scripts/feature_audit.py` (missingness/quantiles/constant, target-decoy-entrapment separation, redundancy clusters, leakage/intensity warnings). Finding on E. coli: 355 audited, 3 constant, 0 leakage-flagged (decoy-vs-entrapment gap <=0.043), 54 clusters. -- P4.3 already done (`feature_ablation.py`); full both-model out_ecoli run produced in `accept/`. -- P5.1 `mass_uncertainty` family (10) + P5.3 `apex_dispersion` family (13). Registry now 383 features. -- P5.2 partially covered by `mass_uncertainty` (fragment-evidence distributions). -- Apex evidence-count option `extract.apex_evidence_rank` (the interference insight: pick the apex by co-eluting fragment breadth, not intensity). Default off. -- P3.2/P3.3 `rt_im_train.adaptive_rt_window` (per-RT-region local residual window, clamped). Default off. -- P7.1 `scripts/benchmark_report.py` (self-contained HTML; smoke -> `accept/benchmark_report.html`). -- P7.2 `scripts/candidate_diagnostics.py` (per-candidate chromatogram + feature bundle). - -### Still TODO (large or non-additive; documented in NEXT_STEPS.md) -- P1.1/P1.2/P1.3 top-K peak RETENTION wired into scoring (enumerator + config + opportunity analysis done; the downstream rewire that carries multiple peaks per candidate through features/compete/rescore + a peak-selection model is the large remaining piece). The reference_apex_topk analysis already measures the opportunity (rank-1 47.9% full). -- P2.2 peak-group conflict graph + P2.3 full conflict-feature list (only `contested_frac` + the new evidence-breadth features exist; a cross-candidate claimant graph is not built). -- P3.1 two-pass mass calibration (RT adaptive window done; mass side not). -- P5.4 remaining MS1/isotope features; P5.5 candidate-ambiguity margins (needs cross-candidate neighborhood). -- P6.2 localization competition + P6.3 staged modification search (need site-determining-ion scoring). -- P0.1 parity check not yet run against a real DIA-NN report (needs the report). - -### Acceptance validation status -- Full out_ecoli ablation (logreg + hgb) produced in `accept/`. HYE + TTOF runs - were launched (`run_accept.sh`); TTOF sample identity is unconfirmed (see the - chat log). None of the new default-off knobs (apex_evidence_rank, - adaptive_rt_window, competition modes) has an engine-gain measurement yet: each - must pass the entrapment-holdout gate on >=2 datasets before being enabled by - default (spec 05 §6). That validation loop is the next action, not more code. - ---- - -## Session 3 (large items) — status - -All default-off / sidecar; suite green (95 tests), release builds. Validated on the -real E. coli artifacts. - -### Done -- P1.1/P1.2 top-K peak retention: `extract.retain_top_peaks>1` writes - `.peaks.parquet` (evidence-breadth-ranked peak groups). Validated: K=5 on - E. coli emitted 1,699,995 peaks (mean 4.97/candidate); main psms/chrom flow and - FDR unchanged (accepted 341,754 as at K=1). -- P1.3 peak-selection model `scripts/peak_selection_model.py`: grouped-OOF ranker - over the peaks table. E. coli (weak self-label, 237,328 candidates): learned - top1 0.426 / top3 0.802; evidence-rank 0.418; area-rank 0.390. Evidence beats - area (consistent with the intensity-is-chimeric argument). Honest label needs - `--diann`. -- P3.1 two-pass mass calibration (`search_seed.two_pass_mass_cal`). -- P2.1/P2.2/P2.3/P5.5 `scripts/conflict_features.py`: fragment-claimant index + - peak-group conflict graph + contested/unique/ambiguity features -> conflict.parquet. -- P6.2 `scripts/localization.py`: localization-variant grouping + site-determining - ions -> localization.parquet (0 ambiguity groups on E. coli, as expected). - -### Genuinely remaining (fix later, per user) -- P6.3 staged modification search (calibration stage then extended-mod stage): not - started. -- In-core wiring of the sidecar features (conflict_features / localization) into the - rescorer feature schema: they currently land as joinable Parquet sidecars. Wiring - needs the cross-candidate pass inside `features.rs` (precedent: the cross_charge - extended-extras) + schema-count test updates. -- Top-K peak SELECTION fed back into the reported PSM (the peaks are retained + a - selection model exists, but the engine still reports the heuristic apex; closing - the loop needs per-peak features, P1.2-full, then re-selection before rescore). -- All new default-off knobs still require the entrapment-gate validation on >=2 - datasets before being enabled by default (spec 05 §6). diff --git a/sensitivity_plan/NEXT_STEPS.md b/sensitivity_plan/NEXT_STEPS.md deleted file mode 100644 index b551d31..0000000 --- a/sensitivity_plan/NEXT_STEPS.md +++ /dev/null @@ -1,132 +0,0 @@ -# Sensitivity Program - Next Steps - -Prioritized, with exact hook sites (from `ARCHITECTURE_MAP.md`). Items are ordered -by expected sensitivity value per unit risk. Everything below builds on the -`feat/sensitivity-improvements` branch. The large scaffolding is now in place: -top-K peak retention, competition modes, two feature families, two-pass mass -calibration, and the adaptive RT window are all implemented and default-off. What -remains is closing loops and validating each knob, not building new plumbing. - -## 1. Close the top-K peak loop (highest value, highest risk) - -Peak RETENTION is done: `extract.retain_top_peaks > 1` enumerates top-K peak groups -with `peaks::enumerate_peaks` and writes `.peaks.parquet` -(`extract.rs:927-929`, `:1068`; 7 cols `candidate_id, peak_rank, apex_rt, start_rt, -end_rt, evidence_count, area`). Validated K=5 on E. coli = 1,699,995 peaks -(mean ~4.97/candidate) with the scored path unchanged. The offline peak-selection -prototype `scripts/peak_selection_model.py` learns to rank the retained peaks -(E. coli weak-self-label: learned top1 0.426 / top3 0.802; evidence-rank 0.418 beats -area-rank 0.390). What is NOT done: the engine still reports the single heuristic -apex; the retained peaks are a diagnostic sidecar, not fed back into scoring. - -- **Remaining hook (close the loop):** the retained peaks carry only coarse - descriptors today (apex/start/end RT, evidence count, area). To re-select a peak - before rescore, each retained `PeakGroup` needs the FULL per-peak feature vector. - Emit one `psms_extracted` + `chromatograms` row per `(candidate_id, peak_rank)` - (peak-restricted apex/chrom emission from `CandOut`, `extract.rs:569`), stamp a - `peak_rank` column, and let `features.rs` compute per-peak features (chrom grouping - must key by `(candidate_id, peak_rank)`). -- **Re-selection:** run the peak-selection ranker (port `peak_selection_model.py`) - or a first native pass, keep the winning peak, then collapse to one row before - `compete` (`compete.rs:72` key) and `rescore` (peptide/protein grouping) so - multiple peaks of one candidate do not all reach the report. -- **Compatibility:** `K == 1` must keep the current byte-identical path (regression - test on a fixed input). -- **Validation:** `scripts/reference_apex_topk.py` before/after; confirm the - entrapment FDP is not inflated (item 7). - -## 2. In-core wiring of the conflict / localization sidecar features - -The conflict graph and localization competition exist as non-invasive Python -sidecars: `scripts/conflict_features.py` -> `conflict.parquet` (fragment-claimant -index + peak-group conflict graph + contested/unique/ambiguity features, joinable by -`candidate_id`) and `scripts/localization.py` -> `localization.parquet` -(localization-variant grouping + site-determining ions; 0 ambiguity groups on -E. coli, as expected). They are not yet in the rescorer feature schema. - -- **Hook:** add the cross-candidate pass inside `features.rs` (precedent: the - cross_charge extended-extras block), add the new fields to `Evidence` - (`features.rs:269`), and register a new family module following the - `NAMES` + `values(&Evidence)` contract appended to `FAMILIES` (`features.rs:49`); - dedup / schema-id / PIN flow then handle it automatically. -- **Schema:** bump the family count and update the `feature_sets_sized` test - (`features.rs:1263`) and `feature_registry.yaml`. -- These directly feed `compete.mode = unique_evidence`, which currently approximates - unique evidence from the existing features. - -## 3. P6.3 staged modification search - -Not started. A calibration stage (unmodified / common-mod search) followed by an -extended-mod stage that opens the search space only where the calibration stage -found evidence. - -- **Prerequisite:** site-determining-ion scoring; `scripts/localization.py` is the - offline prototype for the localization half. -- **Hook:** a second `peptidoforms` -> `predict-frag` -> `extract` pass gated on the - first pass's confident precursors; needs a new orchestration branch in `run.rs` - and a config flag. This is genuinely new plumbing (unlike the items above) and is - the largest remaining feature. - -## 4. Entrapment-gate validation of every default-off knob on >=2 datasets - -Every knob shipped this program is OFF by default because none has passed the -acceptance gate. This is the next ACTION, not more code. - -- **Knobs to validate:** `extract.retain_top_peaks` (once item 1 closes the loop), - `extract.apex_evidence_rank`, `rt_im_train.adaptive_rt_window`, - `search_seed.two_pass_mass_cal`, and `compete.mode` - (`none`/`features_only`/`unique_evidence`/`margin_gated`). -- **Gate:** `scripts/entrapment_holdout.py` gives a leakage-free held-out entrapment - count on an unseen human null. A knob ships as a default only if held-out - entrapment identifications rise WITHOUT FDP inflation, reproduced on a SECOND - dataset (spec 05 §6). -- **Blocker:** only one valid dataset is loaded here (the E. coli / HYE file). The - on-disk SWATH TTOF file is NOT a valid second dataset: it produced 0 IDs against - the Ox HYE library (the sample does not match the library). A genuine second - labelled run (a matching TTOF library, or a ProteoBench HYE run) is required. - -## 5. In-extract candidate-audit emitter (precise extract-stage reasons) - -`emit_candidate_audit` makes `run` write `candidate_audit.parquet` after rescore, but -extraction losses still collapse to `NO_PEAK_GROUP` at artifact resolution. - -- **Hook:** the per-candidate cascade at `extract.rs:566` returns `Option`; - every `return None` (the mapped reasons in `ARCHITECTURE_MAP.md` §3) plus the - never-materialized cohort (library candidate range minus accumulator keys, - `extract.rs:302`) should write a `.audit.parquet` (candidate_id, - rejection_reason). `stages::audit::load_extract_reasons` already reads that sidecar - and refines the waterfall, so no audit-stage change is needed. -- **Cost guard:** only allocate the audit vector when the flag is set. - -## 6. Competition after an initial rescoring pass (spec 04 §11) - -`compete` runs before `rescore` on the heuristic `prelim_score` (`run.rs` order), so -a candidate a trained model would keep can be removed early. The modes are wired; -the reordering is not. - -- **Interim (available now):** run with `compete.mode = none` (or `features_only`) so - competition removes nothing and the rescorer arbitrates. Benchmark against - `winner_take_all` at matched empirical FDP (validate per item 4). -- **Full:** add a first-pass native `percolator_lite` rescore that writes an - out-of-fold score, then run `compete` on that score instead of `prelim_score`. Keep - the fold grouping by peptidoform+charge to avoid leakage. - -## 7. Local calibration uncertainty (finish spec 03 §5 / P3) - -Two-pass mass calibration (`search_seed.two_pass_mass_cal`) and the adaptive RT -window (`rt_im_train.adaptive_rt_window`) are done. The remaining piece is exporting a -LOCAL per-region uncertainty estimate so features can normalize residuals -(`abs(rt_residual)/local_rt_sigma`, `abs(mass_error)/local_mz_sigma`). - -- **Hook:** capture the residual distribution into `cal.json` (`rt_im_train.rs`) and - `masscal.json` (`search_seed.rs`); add a `pred_sigma` field to `Evidence` - (`features.rs:269`) fed by a new library/chromatogram column. The `mass_uncertainty` - and `apex_dispersion` families are the consumers already in place. - -## Cannot be completed in this environment - -- Held-out reproduction on a second dataset (needs a valid second labelled run; the - on-disk TTOF file is the wrong sample for the Ox HYE library). -- Reference-apex recall vs DIA-NN (needs a DIA-NN report; `scripts/reference_apex_topk.py` - and `scripts/peak_selection_model.py` compute it once `--diann` is supplied). -- Ion-mobility / diaPASEF families (no 4D data). diff --git a/sensitivity_plan/README.md b/sensitivity_plan/README.md deleted file mode 100644 index 99b789d..0000000 --- a/sensitivity_plan/README.md +++ /dev/null @@ -1,91 +0,0 @@ -# MuMDIA Sensitivity Evaluation and Improvement - -This documentation set describes a structured program for diagnosing and improving the sensitivity of MuMDIA relative to DIA-NN. - -The central principle is: - -> Do not optimize the final classifier until the workflow identifies where candidate precursors are lost. - -A DIA-NN-only identification can disappear because it was absent from the search space, never generated as a candidate, extracted incorrectly, assigned to the wrong chromatographic peak, outcompeted by another peptide interpretation, ranked poorly, or removed by false-discovery-rate filtering. - -## Implementation status - -The spec files in this directory (01-06) are the requirements and stay fixed. The -derived docs track what is built: `IMPLEMENTATION_STATUS.md` is the source of truth -for progress (candidate audit + `mumdia audit`, top-K peak retention, competition -modes, two-pass mass calibration, adaptive RT window, two new feature families, and -ten diagnostic scripts are all implemented, default-off). `ARCHITECTURE_MAP.md` maps -the code, `BENCHMARK_GUIDE.md` documents the diagnostics, and `NEXT_STEPS.md` lists -the remaining work with exact hook sites. - -## Documentation map - -1. [`01_workflow_and_gap_analysis.md`](01_workflow_and_gap_analysis.md) - Working model of the MuMDIA workflow and the major conceptual differences to investigate relative to DIA-NN. - -2. [`02_sensitivity_diagnostic_plan.md`](02_sensitivity_diagnostic_plan.md) - End-to-end identification-loss ladder, benchmark design, oracle experiments, and reporting requirements. - -3. [`03_feature_evaluation.md`](03_feature_evaluation.md) - Framework for auditing current features, testing new features, preventing leakage, and deciding which features to retain. - -4. [`04_peak_and_peptide_competition.md`](04_peak_and_peptide_competition.md) - Recommended design for chromatographic peak competition, shared-fragment competition, peptidoform competition, and target-decoy competition. - -5. [`05_experiment_matrix.md`](05_experiment_matrix.md) - Controlled experiment matrix, evaluation metrics, decision rules, and acceptance criteria. - -6. [`06_agent_implementation_backlog.md`](06_agent_implementation_backlog.md) - Agent-ready implementation tickets ordered by priority. - -## Main hypotheses - -The largest potential sensitivity losses are expected to arise from one or more of the following: - -1. A single chromatographic apex is selected before the final scoring model can compare alternative peak groups. -2. Candidate peptides can reuse the same observed fragment signal without sufficiently modeling contested evidence. -3. Charge states, modification states, or related precursor candidates may be competed too early. -4. Retention-time, mass-error, and spectral predictions are insufficiently calibrated to the current run. -5. The scorer lacks features describing unique evidence, contested evidence, candidate ambiguity, peak shape, or interference. -6. The decoy or competition design produces a high-scoring false-candidate tail, causing conservative q-value thresholds. -7. Feature-selection results are evaluated using target-decoy separation rather than empirical false discovery. - -## Recommended first implementation sequence - -1. Add full candidate observability and rejection reason codes. -2. Retain the top `K` chromatographic peak groups per precursor. -3. Build a peak-group conflict graph without removing candidates. -4. Add unique-evidence and contested-evidence features. -5. Move variant competition until after initial rescoring. -6. Introduce empirical entrapment validation. -7. Add two-pass RT and mass calibration. -8. Run grouped feature-family ablations. -9. Add conservative post-scoring interference competition. -10. Evaluate cross-run evidence only after single-run behavior is validated. - -## Evidence boundary - -The exact current internals of DIA-NN are not fully public and may differ by version. Treat DIA-NN-inspired competition strategies in these documents as hypotheses to benchmark, not as claims of exact implementation equivalence. - -Every comparison must record: - -- MuMDIA commit -- DIA-NN version -- Complete command lines -- Input hashes -- Search-space manifest -- Prediction-library version -- FDR columns and filtering rules -- Random seeds -- Runtime and memory - -## Definition of success - -The program is successful when: - -- At least 95% of DIA-NN-only precursors receive an explicit earliest-loss category. -- Candidate recall, correct-peak recall, target-ranking recall, and FDR losses are quantified separately. -- Improvements reproduce on held-out datasets. -- Identification gains are measured at matched empirical false discovery proportion. -- Modified peptides and low-intensity precursors are evaluated independently. -- Every retained feature or competition mechanism has an ablation result. From ff6427d54c5ea01fca299a0b13371fecf87258a6 Mon Sep 17 00:00:00 2001 From: MuMDIA Date: Thu, 23 Jul 2026 13:59:58 +0200 Subject: [PATCH 2/7] docs: add extensive developer guide (docs/, ~5.8k lines) Deep code + workflow reference for handoff: 14 subsystem docs + an index, grounded in the current post-cleanup code with file:line citations. Covers the end-to-end dataflow, config + data model, IO layer, every pipeline stage (convert -> ... -> rescore), quant/align/mbr/report/audit, the 10 Python sidecars, and build/test/deploy/gotchas. Sits between CLAUDE.md (orientation) and plan.md (spec + findings). Co-Authored-By: Claude Opus 4.8 --- docs/01_overview_and_dataflow.md | 440 ++++++++++++++ docs/02_config_and_data_model.md | 542 +++++++++++++++++ docs/03_io_layer.md | 377 ++++++++++++ docs/04_convert.md | 302 ++++++++++ docs/05_digest_peptidoforms.md | 326 +++++++++++ docs/06_predict_frag_index_matchers.md | 376 ++++++++++++ docs/07_search_seed.md | 315 ++++++++++ docs/08_rt_im_train.md | 350 +++++++++++ docs/09_extract.md | 450 +++++++++++++++ docs/10_features.md | 607 ++++++++++++++++++++ docs/11_compete_rescore_fdr.md | 458 +++++++++++++++ docs/12_quant_lfq_align_mbr_report_audit.md | 433 ++++++++++++++ docs/13_sidecars.md | 432 ++++++++++++++ docs/14_build_test_deploy_gotchas.md | 321 +++++++++++ docs/README.md | 46 ++ 15 files changed, 5775 insertions(+) create mode 100644 docs/01_overview_and_dataflow.md create mode 100644 docs/02_config_and_data_model.md create mode 100644 docs/03_io_layer.md create mode 100644 docs/04_convert.md create mode 100644 docs/05_digest_peptidoforms.md create mode 100644 docs/06_predict_frag_index_matchers.md create mode 100644 docs/07_search_seed.md create mode 100644 docs/08_rt_im_train.md create mode 100644 docs/09_extract.md create mode 100644 docs/10_features.md create mode 100644 docs/11_compete_rescore_fdr.md create mode 100644 docs/12_quant_lfq_align_mbr_report_audit.md create mode 100644 docs/13_sidecars.md create mode 100644 docs/14_build_test_deploy_gotchas.md create mode 100644 docs/README.md diff --git a/docs/01_overview_and_dataflow.md b/docs/01_overview_and_dataflow.md new file mode 100644 index 0000000..0f593bc --- /dev/null +++ b/docs/01_overview_and_dataflow.md @@ -0,0 +1,440 @@ +# Overview and end-to-end dataflow + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +MuMDIA is a clean-room Rust reimplementation of a data-independent-acquisition +(DIA) proteomics search engine. It takes an mzML run plus a spectral library +source and produces peptide and protein-group identifications at target-decoy +FDR, with optional quantification. This document describes the pipeline as a +graph: every stage, the artifacts each stage reads and writes, the two library +sources (FASTA digest versus imported DIA-NN library) and how the orchestrator +branches between them, the single-run `run` orchestration and its +`manifest.json`, and the current best-performing workflow. + +The design principle is that every stage is an independent command over +path-addressable inputs and outputs on disk (plan.md Section 3.5). No stage +shares in-memory state with another; the orchestrator only threads file paths. +This makes any stage runnable standalone on prior outputs, on a different +configuration, or on hand-crafted minimal files. `run` is a convenience that +chains the stages in order and records provenance; it computes nothing itself. + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI: one `clap` subcommand per stage (`Cmd` enum, `main.rs:16`); each arm loads config, hashes it, and calls the stage `run`. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | `run` orchestrator: preflight, library-source branch, per-run chain, manifest assembly (`run.rs:83`). | +| `rust/mumdia/crates/mumdia/src/stages/mod.rs` | Re-exports every stage module. | +| `rust/mumdia/crates/mumdia/src/stages/*.rs` | The stage implementations (`convert`, `digest`, `peptidoforms`, `predict_frag`, `search_seed`, `rt_im_train`, `extract`, `features`, `compete`, `rescore`, `quant`, `report`, `align`, `audit`). | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | Frozen `(logical name, schema version)` for every artifact (`artifact` module, `schema.rs:6`). | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `Manifest` and `ArtifactRecord` (`manifest.rs:10`, `manifest.rs:22`). | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | The single typed serde `Config` (`config.rs:1010`) with per-stage sections and `apply_profile` (`config.rs:1098`). | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | The `Col`/`Table` typed Parquet layer every stage writes through (`table.rs:23`). | +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | `record_artifact`, `inspect`, `init_logging`, blake3 hashing. | + +## Inputs and outputs + +The pipeline consumes an mzML file plus one of two library sources, and produces +a fixed set of Parquet artifacts plus JSON sidecars. Every stage also writes a +small `.report.json` next to each Parquet output (row counts, key +distributions, parameters, timing; plan.md Section 3.5), which is not listed +below per artifact but always exists. + +External inputs: +- `--mzml ` (mzML; `.raw`/`.d` must be converted upstream by msconvert). +- FASTA mode: `--fasta ` (digested into the library). +- Library-input mode: `--lib-precursors ` + `--lib-fragments ` + (a prebuilt library, for example an imported DIA-NN speclib). + +Parquet artifacts and their column schemas as written in the code: + +`spectra_ms1.parquet` (`convert.rs:180`): `scan_index` u32, `rt_seconds` f64, +`mz` list, `intensity` list. + +`spectra_ms2.parquet` (`convert.rs:207`): `scan_index` u32, `id` str, +`rt_seconds` f64, `window_id` u32, `window_target` f64, `window_lower` f64, +`window_upper` f64, `precursor_mz` opt, `precursor_charge` opt, +`mz` list, `intensity` list. + +`isolation_windows.parquet` (`convert.rs:224`): `window_id` u32, `target` f64, +`lower` f64, `upper` f64. + +`ms2_to_ms1.parquet` (`convert.rs:234`): `ms2_scan_index` u32, +`ms1_scan_index` i32 (the preceding MS1 scan; -1 if none). + +`peptides.parquet` (digest; FASTA mode only): fully-tryptic stripped peptides +plus their paired decoys. + +`peptidoforms.parquet` (peptidoforms; FASTA mode only): concrete peptidoforms +with fixed/variable mods and charges as ProForma-lite strings. + +`fragment_library_precursors.parquet` (`predict_frag.rs:189`): `candidate_id` +u32, `peptidoform_id` u32, `base_peptide_id` u32, `peptidoform` str, `charge` +i32, `precursor_mz` f64, `predicted_irt` f32, `label` str +(`target`/`decoy`), `protein` str, `n_fragments` i32. `candidate_id` is a +contiguous 0-based index sorted by precursor m/z (the fragment-index build +precondition). + +`fragment_library_fragments.parquet` (`predict_frag.rs:204`): `candidate_id` +u32, `mz` f64, `predicted_intensity` f32, `name` str, `ion_type` str, `ordinal` +i32, `frag_charge` i32. + +`seed_psms.parquet` (`search_seed.rs:220`): `candidate_id` u32, `peptidoform` +str, `charge` i32, `precursor_mz` f64, `base_peptide_id` u32, `protein` str, +`label` str, `score` f64 (hyperscore), `spectrum_q` f64 (per-spectrum +target-decoy q), `observed_rt` f64, `predicted_irt` f32, `matched_peaks` i32, +`scan_index` u32. Sidecar `seed_psms.parquet.masscal.json` carries the per-run +mass recalibration consumed by extract. + +`run_windows.parquet` (`rt_im_train.rs:185`): `candidate_id` u32, `rt_pred_cal` +f64, `rt_lo` f64, `rt_hi` f64, `im_pred_cal`/`im_lo`/`im_hi` opt (IM columns +null in 3D). Sidecar `cal.json` records the fitted RT calibration. + +`psms_extracted.parquet` (`extract.rs:1359`): `candidate_id` u32, `apex_rt` f64, +`apex_im` opt, `apex_intensity` f32, `n_matched_fragments` i32, +`n_predicted_fragments` i32, `coelution_run` i32, `rt_pred_cal` f64, +`precursor_mz` f64, `charge` i32, `label` str, `base_peptide_id` u32, +`peptidoform` str, `protein` str, `predicted_irt` f32, `contested_frac` f64, +`ms1_isom1`/`ms1_mono`/`ms1_iso1`/`ms1_iso2` opt. Optional columns: +`contested_count_frac`, `apportioned_frac` when `emit_contested_features` +(`extract.rs:1383`); `gate_apex`, `gate_peak_spectral`, `gate_coelution`, +`gate_spectral_entropy` when `emit_gate_diagnostics` (`extract.rs:1389`). + +`chromatograms.parquet` (`extract.rs:1399`): `candidate_id` u32, `frag_name` +str, `frag_mz` f64, `frag_obs_mz` f64, `predicted_intensity` f32, `rt` +largelist, `intensity` largelist. `LargeList` (64-bit offsets) is +required because total list-value count can exceed the 32-bit `ListArray` limit +when extraction accepts a very large candidate set. + +`.peaks.parquet` (`extract.rs:1419`, written only when +`extract.retain_top_peaks > 1`): `candidate_id` u32, `peak_rank` i32, `apex_rt` +f64, `start_rt` f64, `end_rt` f64, `evidence_count` f64, `area` f64. This is a +diagnostic sidecar not yet scored downstream (see finding 8, Gotchas). + +`features.parquet` (`features.rs:828`): bookkeeping columns `candidate_id`, +`label`, `base_peptide_id`, `peptidoform`, `protein`, `apex_rt`, `elution_lo`, +`elution_hi`, `precursor_mz`, `prelim_score`, followed by one f64 column per +active feature name (`features.rs:839`). Sidecar `features.parquet.schema.json` +records the ordered feature-column list and its hashed `schema_id`; `run.pin` is +the Percolator input written deterministically alongside. + +`psms_competed.parquet` (`compete.rs:118`): the same bookkeeping columns plus +every feature column carried through unchanged (`compete.rs:128`), for surviving +rows only. Sidecar `psms_competed.parquet.schema.json` carries the schema +forward for rescore. + +`psms_scored.parquet` (schema version 2, `rescore.rs:323`): `candidate_id` u32, +`peptidoform` str, `charge` i32, `label` str, `protein` str, `base_peptide_id` +u32, `score` f64, `q_value` f64, `peptide_q_value` f64, `protein_group` str, +`pg_q_value` f64, `global_q_value` f64, `prelim_score` f64, `source` u32, +`run_psm_q` f64, `experiment_psm_q` f64, `precursor_q` f64. The q-value columns +are independent per level, not a rollup (see Gotchas). + +`peptide_quant.parquet` (`quant.rs:317`): `candidate_id`, `peptidoform`, +`charge`, `protein_group`, `quantity` f64, `n_fragments_used` i32. +`protein_group_quant.parquet` (`quant.rs:344`): `protein_group`, `quantity`, +`n_peptides` i32. Optional `fragment_quant.parquet` (`quant.rs:372`) and +`peak_bounds` (`quant.rs:278`) diagnostics. `quant-lfq` emits a +protein-by-run matrix (`quant.rs:479`): `protein_group`, `run` i32, `quantity`, +`n_features` i32. + +`candidate_audit.parquet` (`audit.rs:180`, written when +`extract.emit_candidate_audit` or via `mumdia audit`): `run_id` str, +`precursor_id` u32, `modified_sequence` str, `charge` i32, +`target_decoy_label` str, `entrapment_label` bool, then the boolean stage-flags +`candidate_generated`, `traces_extracted`, `peak_generated`, `peak_selected`, +`variant_selected`, `target_decoy_winner`, `passed_precursor_fdr`, +`passed_peptide_fdr`, `reported`, and `rejection_reason` str (earliest loss +reason). + +Human-readable outputs: `peptides.tsv`, `proteins.tsv` (report), and +`manifest.json` (orchestrator). + +## How it works + +### The pipeline as a graph + +``` + FASTA mode library-input mode + (--fasta present) (--lib-precursors + --lib-fragments) + | | + digest peptides.parquet | + | | + peptidoforms peptidoforms.parquet | + | | + predict-frag fragment_library_precursors.parquet (supplied directly, digest/ + fragment_library_fragments.parquet peptidoforms/predict-frag skipped) + \________________________________________/ + | + lib_p (precursors) + lib_f (fragments) + | + --mzml ---> convert ---> spectra_ms1.parquet, spectra_ms2.parquet, + isolation_windows.parquet, ms2_to_ms1.parquet + | + spectra_ms2 + lib_p + lib_f --> search-seed --> seed_psms.parquet + seed_psms.parquet.masscal.json + | + [optional] lib_p + seed --> deeplc_finetune --> fragment_library_precursors_ft.parquet + (rewrites predicted_irt; lib_p := *_ft) + | + seed + lib_p --> rt-im-train --> run_windows.parquet, cal.json + | + spectra_ms2 + lib_p + lib_f + run_windows + spectra_ms1 + masscal + --> extract --> psms_extracted.parquet, chromatograms.parquet + [.peaks.parquet if retain_top_peaks>1] + | + psms_extracted + chromatograms + seed --> features --> features.parquet, + features.parquet.schema.json, run.pin + | + features --> compete --> psms_competed.parquet (+ .schema.json) + | + psms_competed --> rescore --> psms_scored.parquet + | + [optional] lib_p + psms_extracted + competed + scored --> audit --> candidate_audit.parquet + | + psms_scored + chromatograms --> quant --> peptide_quant.parquet, + protein_group_quant.parquet, fragment_quant.parquet + | + psms_scored + quant --> report --> peptides.tsv, proteins.tsv + | + all artifacts recorded --> manifest.json +``` + +### The two library sources and how run.rs branches + +The spectral library is the experiment-level, run-independent artifact pair +`(lib_p, lib_f)`. The orchestrator produces it in one of two ways, decided by a +`match` on `(p.lib_precursors, p.lib_fragments)` at `run.rs:96`: + +- Library-input mode (`(Some(lp), Some(lf))`, `run.rs:97`): the supplied library + is consumed directly. `digest`, `peptidoforms`, and `predict-frag` are skipped + and the FASTA is never read. The orchestrator only reads the two Parquet files + to record their row counts in the manifest, then uses their paths downstream + (`run.rs:105-109`). This is the highest-sensitivity path, used with an imported + DIA-NN speclib that already carries fragment intensities and iRT. + +- FASTA-digest mode (the `_ =>` arm, `run.rs:111`): the library is built from the + FASTA. `digest::run` writes `peptides.parquet` (`run.rs:116`), + `peptidoforms::run` writes `peptidoforms.parquet` (`run.rs:126`), and + `predict_frag::run` writes both library Parquet files (`run.rs:136`). The + returned `(lib_p, lib_f)` paths feed the rest of the chain. This path has zero + external runtime dependencies (native predictors). + +`preflight` (`run.rs:37`) validates the mode before any compute: exactly one of +the two source configurations must be present, and every required input path must +exist (`run.rs:42-62`). It also checks sidecar prerequisites: `finetune_deeplc` +requires `predict_frag.deeplc_python` (`run.rs:63`); `rescore.classifier=mokapot` +requires `rescore.python`; `rescore.classifier=entrapment` requires +`rescore.entrapment_marker` (`run.rs:69-79`). + +### The per-run chain + +After the library is resolved, the orchestrator runs the per-run stages in order, +recording each output in the manifest with `record_artifact` (`run.rs:83-343`): + +1. `convert::run` (`run.rs:152`) reads the mzML through mzdata, centroids profile + spectra, caps peaks, synthesizes a full-range window for AIF/all-ion scans, + and writes the four spectra artifacts. It returns a `ConvertOutputs` struct of + paths (`convert.rs:96`). + +2. `search_seed::run` (`run.rs:171`) runs a native Sage-lite broad DIA search over + `spectra_ms2` against the fragment index for calibration (not final ID). It + writes `seed_psms.parquet` and the mass recalibration `masscal.json`. The seed + is iRT-independent, so it is computed once here on the base library and reused. + +3. Optional DeepLC multitask fine-tune (`run.rs:187`, gated on + `rt_im_train.finetune_deeplc`): `sidecar::run_deeplc_finetune` adapts the RT + model to this run's confident seed PSMs and rewrites `predicted_irt` into + `fragment_library_precursors_ft.parquet`. The `lib_p` binding is then rebound + to the fine-tuned file so rt-im-train and extract read it. `lib_f` is + unchanged (fine-tune touches iRT only). This step is nondeterministic (no fixed + torch/numpy seed). + +4. `rt_im_train::run` (`run.rs:212`) maps the run-independent iRT onto this run's + observed RT (LOESS or linear) and sets per-candidate RT windows from the + residual percentile, writing `run_windows.parquet` and `cal.json`. + +5. `extract::run` (`run.rs:224`) is the core stage: a peak-major cascade over the + inverted fragment index. It reads `spectra_ms2`, both library files, + `run_windows`, `spectra_ms1` (for MS1 isotope XICs), and the `masscal.json`; it + writes `psms_extracted.parquet` and `chromatograms.parquet`. `run` passes + `restrict_candidates: None` (no allowlist). + +6. `features::run` (`run.rs:242`) computes the config-selected feature vector plus + `prelim_score` per PSM and emits `features.parquet`, its schema sidecar, and + `run.pin`. It also reads `seed` for search-engine corroboration features. + +7. `compete::run` (`run.rs:254`) keeps the best candidate per competition group + before FDR counting, writing `psms_competed.parquet` and carrying the feature + schema forward. + +8. `rescore::run` (`run.rs:263`) does semi-supervised rescoring and native + target-decoy q-values at multiple contexts, writing `psms_scored.parquet`. In + `run` it is passed a single competed table (`&[competed.clone()]`), so this is + single-run rescoring; the standalone `rescore` command accepts several tables + for experiment-wide scoring. + +9. Optional `audit::run` (`run.rs:276`, gated on `extract.emit_candidate_audit`) + reconstructs the per-candidate identification-loss ladder into + `candidate_audit.parquet`. It is a cheap join over the artifact chain and runs + no extraction. + +10. `quant::run` (`run.rs:293`) integrates fragment chromatograms and rolls up to + protein groups, writing `peptide_quant`, `protein_group_quant`, and + `fragment_quant`. + +11. `report::run` (`run.rs:309`) writes `peptides.tsv` and `proteins.tsv` from the + scored table plus quant, thresholded at `quant.q_threshold`, and prints a + stdout summary naming the actual rescorer used. + +### The manifest + +`Manifest::new` (`manifest.rs:34`) is seeded with the fully-resolved canonical +config JSON and its blake3 hash (`run.rs:87,91`). Every stage output is recorded +by `record_artifact` (`mumdia-io`), which builds an `ArtifactRecord` +(`manifest.rs:10`) holding the logical name, path, format, schema name and +version, row count, content hash, producing stage, and config hash. Model +identities (RT predictor, fragment predictor, rescorer, and the feature schema +id) are recorded at `run.rs:323-333`. The manifest is written last to +`/manifest.json` (`run.rs:334`) with `mumdia_io::json::write_json`. The +manifest is provenance recorded, not required: because inputs are +path-addressable, no stage depends on it to run (plan.md Section 3.5). + +### Current best workflow + +The validated best-performing configuration (plan.md preamble "Best workflow", +finding 1) is library-input mode with: +- an imported DIA-NN library (fragment intensities + iRT), +- per-run DeepLC multitask fine-tune of iRT (`rt_im_train.finetune_deeplc = true` + + `predict_frag.deeplc_python`), which is essential in library mode because raw + DIA-NN iRT is locally noisy (~110 s MAD) and the fine-tune tightens it to + ~13-27 s (finding 5), +- the Extended feature set (`features.set = extended`, applied by `--profile dia`, + `config.rs:1101`), +- the `nn_torch` PyTorch MLP rescorer (`rescore.classifier = nn_torch` + + `rescore.python`), which is the dominant sensitivity lever and beats the native + linear rescorer by ~8.5% (finding 1), +- a loose extraction gate (`extract.min_frag_corr ~= 0.2`, now the default), since + a strong nonlinear rescorer prefers recall and absorbs the loose-gate flood + (finding 2), +- the MS2 peak cap kept at 300 on AIF (finding 4). + +On `LFQ_Orbitrap_AIF_Ecoli_01.mzML` this reaches ~10,300 peptides at 1% FDR. The +zero-dependency native FASTA-digest mode (~1,213 peptides) is the high-precision +fallback. This preset is invoked through `docker/config.diann-lib.json` plus +`--profile dia`. + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `Cmd` (enum) | `main.rs:16` | One clap subcommand per stage; `Run` at `main.rs:195`. | +| `main` | `main.rs:373` | Parses the CLI, loads config, dispatches to the stage `run`. | +| `load_config` | `main.rs:363` | Reads a config JSON or returns `Config::default()`. | +| `doctor` | `main.rs:311` | Probes configured sidecar interpreters for required packages. | +| `RunParams` | `run.rs:17` | The orchestrator's inputs (config, fasta, mzml, out_dir, lib paths, caps). | +| `preflight` | `run.rs:37` | Validates mode, input existence, and sidecar prerequisites before compute. | +| `stages::run::run` | `run.rs:83` | The orchestrator: library branch, per-run chain, manifest write. | +| `Manifest` / `ArtifactRecord` | `manifest.rs:22` / `manifest.rs:10` | Provenance record for a chained run. | +| `artifact` module | `schema.rs:6` | Frozen `(logical name, schema version)` constants. | +| `Config::apply_profile` | `config.rs:1098` | Applies the `dia` preset (Extended features, apex window 5, RT prior 120 s). | +| `Config::canonical_json` | `config.rs:1116` | Canonical JSON for hashing into the manifest. | +| `Col` (enum) | `table.rs:23` | Typed column for writing Parquet; `LargeListF32` for chromatograms. | +| `Table::read` + typed getters | `table.rs:204` | Reads a Parquet artifact and extracts typed columns. | + +## Configuration + +The orchestrator reads a single typed `Config` (`config.rs:1010`) with per-stage +sections. Every field has `#[serde(default)]` and the struct uses +`deny_unknown_fields`, so a partial JSON overrides only named fields and an +unknown key is a hard error. `--profile dia` is applied on top of the loaded +config (`main.rs:581`) before `run`. The fields this overview area touches: + +| Field | Default | Effect | +|---|---|---| +| `rng_seed` | `0` | Seeds decoy generation and rescorer folds (determinism). | +| `digest.decoy.strategy` | `reverse` | Decoy scheme; `diann_shift` and `none` are rejected/no-op. | +| `peptidoforms.charge_min` / `charge_max` | `2` / `3` | Precursor charge range enumerated in FASTA mode. | +| `predict_frag.rt_predictor` | `native` | `native` or `deeplc` iRT source (recorded in manifest). | +| `predict_frag.predictor` | `native` | `native` or `ms2pip` fragment intensities. | +| `predict_frag.deeplc_python` | `None` | Interpreter for DeepLC; required if `finetune_deeplc`. | +| `predict_frag.sidecar_script_dir` | `"scripts"` | Where sidecar workers are resolved from. | +| `search_seed.top_n_peaks` | `300` | Seed-only MS2 peak cap (keep 300 on AIF, finding 4). | +| `rt_im_train.calibration_method` | `loess` | RT calibration; `none` is rejected by validation. | +| `rt_im_train.finetune_deeplc` | `false` | Enables the per-run DeepLC fine-tune of iRT (best workflow). | +| `rt_im_train.finetune_batch` | `0` | `0` auto-scales the fine-tune batch to seed size. | +| `extract.min_frag_corr` | `0.2` | Spectral-agreement gate threshold; loose default suits `nn_torch`. | +| `extract.gate_mode` | `apex_pearson` | Which spectral score the gate thresholds (`GateMode`, `config.rs:591`). | +| `extract.retain_top_peaks` | `1` | `>1` writes the `.peaks.parquet` sidecar (not yet scored). | +| `extract.emit_candidate_audit` | `false` | Emits `candidate_audit.parquet` in `run`. | +| `extract.emit_gate_diagnostics` | `false` | Adds the four `gate_*` columns to `psms_extracted`. | +| `features.set` | `minimal` | `minimal` (14) / `rich` (44) / `extended` (~381); `dia` profile sets extended. | +| `compete.mode` | `winner_take_all` | Within-group competition resolution (`CompetitionMode`). | +| `rescore.classifier` | `native_tda` | Rescorer; `nn_torch` is the best lever, needs `rescore.python`. | +| `rescore.python` | `None` | Interpreter for mokapot / nn_torch / entrapment sidecars. | +| `quant.q_threshold` | (see `QuantConfig`) | q-value threshold for the human-readable report. | + +The config surface was recently pruned of dead or unwired fields (plan.md +"Config-surface note"): `threads`, `extract.{k_select, max_fragment_charge, +scan_scale, scan_window_mode}` + `ScanWindowMode`, `digest.decoy.{ratio, source}` ++ `DecoySource`, `search_seed.precursor_tol_ppm`, `rt_im_train.tolerance_regime` ++ `ToleranceRegime`, `FeatureSet::Custom`, `MatcherKind::Naive`, and +`CompetitionMode::from_token` were removed. Do not reintroduce them. Kept +deliberately as documented hooks: `DecoyStrategy::DiannShift` (deferred, +license-checked; rejected by validation), `CalibrationMethod::None` (rejected by +validation with a clear message), and the whole `mbr` section (planned feature). + +## Invariants, determinism, gotchas + +- Determinism is required (plan.md Section 7): seed the RNG, keep numeric + summation order fixed. A HashMap f32 sum shifting the apex once broke + reproducibility; use ordered maps or sorted iteration where floats are summed. + The DeepLC fine-tune is the one deliberate exception (nondeterministic; no fixed + torch seed), so a `run` with `finetune_deeplc = true` is not bit-reproducible. +- `candidate_id` must be a contiguous 0-based index sorted by precursor m/z; the + fragment index build in extract depends on it. An imported library must be + re-indexed by `make_reverse_decoys.py` to satisfy this. +- q-value columns in `psms_scored` are independent per level, not a rollup + (finding 6): `q_value` (== `experiment_psm_q`), `run_psm_q` (per source), + `precursor_q` (peptidoform+charge), `peptide_q_value` (stripped sequence), + `pg_q_value`. Coarser grouping pools evidence, so peptide counts can exceed + precursor counts. Report at the correct context: `precursor_q` for a precursor + matrix, `run_psm_q` for cross-run library-mode quant. Do not threshold PSM q + then deduplicate. +- Every default-off knob keeps the production chain byte-identical when unset; + `retain_top_peaks=1`, `emit_candidate_audit=false`, `emit_gate_diagnostics=false`, + and the other sensitivity-program toggles all default to the legacy path. +- The extraction gate is a training-pool and null-curation lever, not feature + work (finding 3). Loosening it floods the pool with decoy-enriched junk that + corrupts a linear rescorer; a nonlinear rescorer absorbs it. The gate optimum + therefore inverts by rescorer. +- The selected apex is the strongest peak only ~48-52% of the time (finding 8); + the `.peaks.parquet` top-K sidecar exists to expose alternative peaks but + is not yet promoted through features/rescore. + +## How to extend / modify + +- To add a stage: implement it as `stages/.rs` with a `run(Params)` that + reads path-addressable inputs and writes Parquet through the `Col`/`Table` + layer plus an `.report.json`; add a `(logical name, version)` to + `schema.rs`; add a `Cmd` arm in `main.rs`; and thread it into `run.rs` with a + `record_artifact` call. Keep the stage runnable standalone. +- To add an artifact column: add the `Col::` in the writing stage, update the + reading stage's typed getter, and bump the schema version in `schema.rs` if the + change is not backward-compatible (`PSMS_SCORED` is already at version 2). +- To add a tuning profile: extend the `match` in `Config::apply_profile` + (`config.rs:1098`); it should only set existing typed config fields, never add + plumbing. +- To add a library source: extend the `match` on `(lib_precursors, + lib_fragments)` in `run.rs:96` and the corresponding `preflight` branch + (`run.rs:42`). The rest of the chain consumes `(lib_p, lib_f)` unchanged. +- To wire a new sidecar: follow the positional-CLI file contract + (`sidecar::resolve_script` + a worker in `scripts/`), gate it behind a config + field, and check its prerequisites in `preflight` and `doctor`. +- Stubs and unwired areas to be aware of: `mbr` is a partial sidecar, not the full + experiment-wide flow; `align` is not in the `run` chain (needs >=2 runs); ion + mobility / 4D is a data-model stub only (IM columns are always null in 3D); + vendor readers other than mzML do not exist. Do not document these as complete. diff --git a/docs/02_config_and_data_model.md b/docs/02_config_and_data_model.md new file mode 100644 index 0000000..195fb7d --- /dev/null +++ b/docs/02_config_and_data_model.md @@ -0,0 +1,542 @@ +# Config, mass model, constants, schema, manifest +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This document covers the `mumdia-core` crate: the shared vocabulary that every +stage depends on but that contains no stage logic itself. Four concerns live +here. + +1. **Typed configuration** (`config.rs`): one serde structure with per-stage + sections and one strategy enum per algorithmic choice point. `Config` parses + from JSON, rejects unknown keys, and validates on load so a misconfiguration + fails at startup rather than producing a bogus run. +2. **Mass model** (`mass.rs` + `constants.rs`): the single source of residue + masses, physical constants, m/z conversion, ppm predicates, ProForma-lite + peptidoform parsing, and b/y fragment generation. No other crate defines a + mass constant or a ppm predicate. +3. **Frozen artifact schema ids** (`schema.rs`): `(logical name, version)` pairs + stamped into every Parquet artifact and its `report.json` so a stage can + detect a schema mismatch. +4. **Run manifest** (`manifest.rs`): per-artifact provenance (content hash, + producing stage, config hash) written once by the `run` orchestrator. + +The crate is declared in `lib.rs:6-13` (`config`, `constants`, `error`, +`manifest`, `mass`, `rejection`, `schema`, `types`) and exposes +`version()` from `CARGO_PKG_VERSION` (`lib.rs:15-17`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia-core/src/config.rs` | All config structs + every strategy enum; `Config::from_json`, `validate`, `apply_profile`, `canonical_json` (1160 lines) | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | Physical constants (`PROTON`, `WATER`, `AMMONIA`, `ISOTOPE_SPACING`), `residue_mass`, `mass_to_mz`, ppm predicates | +| `rust/mumdia/crates/mumdia-core/src/mass.rs` | `unimod_mass`, `IonType`, `Fragment`, `ParsedPeptidoform`, `parse_peptidoform`, b/y fragment generation | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | Frozen `(name, version)` ids for every artifact; `PSMS_SCORED` is v2, all others v1 | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `Manifest` + `ArtifactRecord` (provenance) | +| `rust/mumdia/crates/mumdia-core/src/types.rs` | `Peak`, `IsolationWindow`, `Label`, `Ms2Scan` | +| `rust/mumdia/crates/mumdia-core/src/rejection.rs` | `RejectionReason` ladder for the candidate-audit table | +| `rust/mumdia/crates/mumdia-core/src/error.rs` | `MassError`, `ConfigError` (thiserror) | +| `rust/mumdia/crates/mumdia-core/src/lib.rs` | Module wiring + `version()` | +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | `record_artifact` (builds `ArtifactRecord` by hashing the file), `inspect` | +| `rust/mumdia/crates/mumdia-io/src/hash.rs` | `blake3_file`, `blake3_str` (content + config hashing) | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport` written as `.report.json` | + +## Inputs and outputs + +`mumdia-core` produces no artifacts of its own. It defines the types the other +crates read and write. Two things it owns appear on disk: + +**`manifest.json`** (one per `run`, written at `run.rs:334-335`). Serialized +`Manifest`; fields (`manifest.rs:22-31`): + +| Field | Type | Meaning | +|---|---|---| +| `mumdia_version` | String | `CARGO_PKG_VERSION` at build time (`manifest.rs:37`) | +| `config_json` | String | Fully-resolved config, from `Config::canonical_json()` | +| `config_hash` | String | `blake3_str(canonical_json)` (`run.rs:87`) | +| `model_identities` | BTreeMap | `rt_predictor`, `fragment_predictor`, `rescorer`, `feature_schema_id` (`run.rs:323-332`) | +| `artifacts` | BTreeMap | one entry per produced artifact, keyed by logical name | + +**`ArtifactRecord`** (`manifest.rs:9-20`), one per artifact: +`logical_name`, `path`, `format` (always `"parquet"`), `schema_name`, +`schema_version`, `rows`, `content_hash` (blake3 of the file bytes), +`producing_stage`, `config_hash`. Built by `record_artifact` +(`mumdia-io/src/lib.rs:20-39`), which hashes the written Parquet file with +`blake3_file` (`hash.rs:8-19`, streamed in 64 KiB chunks). + +**`ArtifactReport`** / `.report.json` (`report.rs:11-24`) is written +alongside each artifact by its producing stage, not by core: `logical_name`, +`schema_name`, `schema_version`, `stage`, `rows`, `content_hash`, `params` +(resolved parameters), `stats` (key metrics), `model_identity`, `elapsed_ms`. +`write_for` appends `.report.json` to the artifact path (`report.rs:26-31`). + +### Artifact schema ids (`schema.rs:6-24`) + +Every id is a `(&str, u32)` constant. A stage stamps `.0` and `.1` into its +`ArtifactReport` and `ArtifactRecord`. + +| Constant | Logical name | Version | +|---|---|---| +| `SPECTRA_MS1` | `spectra_ms1` | 1 | +| `SPECTRA_MS2` | `spectra_ms2` | 1 | +| `ISOLATION_WINDOWS` | `isolation_windows` | 1 | +| `MS2_TO_MS1` | `ms2_to_ms1` | 1 | +| `PEPTIDES` | `peptides` | 1 | +| `PEPTIDOFORMS` | `peptidoforms` | 1 | +| `FRAGMENT_LIBRARY_PRECURSORS` | `fragment_library_precursors` | 1 | +| `FRAGMENT_LIBRARY_FRAGMENTS` | `fragment_library_fragments` | 1 | +| `SEED_PSMS` | `seed_psms` | 1 | +| `RUN_WINDOWS` | `run_windows` | 1 | +| `PSMS_EXTRACTED` | `psms_extracted` | 1 | +| `CHROMATOGRAMS` | `chromatograms` | 1 | +| `FEATURES` | `features` | 1 | +| `PSMS_COMPETED` | `psms_competed` | 1 | +| `PSMS_SCORED` | `psms_scored` | **2** | +| `PEPTIDE_QUANT` | `peptide_quant` | 1 | +| `PROTEIN_GROUP_QUANT` | `protein_group_quant` | 1 | + +`PSMS_SCORED` is the only artifact bumped past v1. Its v2 column layout is +written by the rescore stage (`rescore.rs:320-347`) and is the schema a +downstream reader must expect: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | library candidate id | +| `peptidoform` | Str | ProForma-lite string | +| `charge` | I32 | precursor charge | +| `label` | Str | `target` / `decoy` | +| `protein` | Str | protein accession | +| `base_peptide_id` | U32 | interned stripped-peptide id (peptide-level q grouping) | +| `score` | F64 | rescorer score | +| `q_value` | F64 | per-PSM q (pooled) | +| `peptide_q_value` | F64 | peptide-level q (global per base peptide) | +| `protein_group` | Str | protein-group key | +| `pg_q_value` | F64 | protein-group q | +| `global_q_value` | F64 | experiment-wide PSM q (== `q_value` for a single-run rescore) | +| `prelim_score` | F64 | pre-rescore feature-stage score | +| `source` | U32 | run index (0 for single-run; index into `--competed` for experiment-wide) | +| `run_psm_q` | F64 | per-run PSM FDR | +| `experiment_psm_q` | F64 | pooled PSM FDR | +| `precursor_q` | F64 | per (peptidoform+charge) FDR | + +The three trailing multi-context q columns (`run_psm_q`, `experiment_psm_q`, +`precursor_q`, `rescore.rs:340-345`) are the reason for the v2 bump; a v1 reader +that assumed the old column set would misread them. `quant.q_filter` +(see [QuantConfig](#quantconfig-quantrs)) selects which of these q columns the +quant stage filters on. + +## How it works + +### Config load path + +`load_config` (`main.rs:363-371`) reads the `--config` file to a string and +calls `Config::from_json` (`config.rs:1046-1051`), which does +`serde_json::from_str` and then `validate()`. With no `--config` it returns +`Config::default()` (`config.rs:1025-1042`), which is fully populated from the +per-field defaults. `--profile ` then calls `apply_profile` +(`main.rs:582`, `config.rs:1098-1112`) on top of the loaded config. + +`deny_unknown_fields` on every section (`config.rs:207`, `218`, `239`, ...) means +a typo like `{"digest":{"min_len":7,"bogus":1}}` is a parse error, verified by +`unknown_key_rejected` (`config.rs:1136-1139`). `#[serde(default)]` on every +section and field means a partial JSON overlays defaults: `{"digest":{"min_len":7}}` +keeps `max_len=50`, `charge_max=3`, `top_n_peaks=300` (`config.rs:1142-1149`). +The `t::()` helper (`config.rs:199-204`) is a terse `T::default()` used inside +the per-struct `Default` impls. + +### `validate()` hard-error checks (`config.rs:1056-1091`) + +Four combinations are rejected at load so they never silently produce a wrong +result. Defaults always pass. + +1. `digest.decoy.strategy == DiannShift` -> `Invalid`: the engine digest + produces zero decoys under it, giving an invalid target-decoy FDR + (`config.rs:1058-1066`). +2. `rt_im_train.calibration_method == None` -> `Invalid`: `None` would silently + fall through to the linear fit, so it is rejected and the user must pick + `linear` or `loess` (`config.rs:1067-1073`). +3. `extract.retain_top_peaks == 0` -> `Invalid`: must be `>= 1` (1 = legacy + single apex) (`config.rs:1074-1080`). +4. `extract.min_frag_corr` not finite or outside `[0, 1]` -> `Invalid` + (`config.rs:1081-1089`). 0 disables the gate. Verified by + `explicit_uncapped_seed_and_invalid_gate_are_distinguished` + (`config.rs:1152-1158`). + +`canonical_json` (`config.rs:1116-1118`) serializes the fully-resolved config; +`run` hashes it with `blake3_str` into `config_hash` (`run.rs:87`) and stores the +JSON verbatim in the manifest. There is no separate pretty vs canonical form; it +is a plain `serde_json::to_string`. + +### `apply_profile` (`config.rs:1098-1112`) + +Only `dia` is defined. It sets `features.set = Extended`, +`extract.apex_count_window = 5`, `extract.apex_rt_prior_s = 120.0`. Any other name +is an `Invalid` error. All other extraction defaults stay at their conservative +baselines; the profile is a convenience shortcut, not a full preset file. + +### Mass model math (`mass.rs`, `constants.rs`) + +**Neutral mass** (`mass.rs:66-73`): `WATER + n_term_mod + c_term_mod` plus, per +residue, `residue_mass(r) + mod_delta`. `residue_mass` (`constants.rs:26-51`) is +the 20-standard-amino-acid table; `B, J, O, U, X, Z` return `None` (ambiguous / +non-standard) and cause `MassError::AmbiguousResidue` at parse time. Leucine and +isoleucine share `113.084064015` (`constants.rs:34-35`). + +**m/z conversion** (`constants.rs:59-62`): +`mass_to_mz(m, z) = (m + z * PROTON) / z`. `precursor_mz` (`mass.rs:76-78`) is +this over the neutral mass. + +**Fragment generation** (`mass.rs:82-127`): a forward prefix scan builds b ions +(`b2..b(n-1)`, dropping `b1`) and a reverse suffix scan builds y ions +(`y3..y(n-1)`, dropping `y1` and `y2`) because b1/y1/y2 are low-information +(`mass.rs:93`, `mass.rs:113`). Each fragment m/z is +`(residue_sum_incl_terminus + z * PROTON) / z` (`mass.rs:97`, `mass.rs:116`). +`Fragment` (`mass.rs:45-53`) carries `ion_type`, `ordinal`, `charge`, `mz`, and a +stable `name` (`b3`, or `y5^2` for charge 2, `frag_name` at `mass.rs:130-136`). + +**ppm predicates.** Three exist and they differ by which mass normalizes the +difference; a maintainer must not treat them as interchangeable. + +- `ppm_diff(observed, theoretical)` (`constants.rs:66-68`): + `1e6 * (observed - theoretical) / theoretical`. **Signed**, normalized by the + **theoretical** mass. Used for reporting a mass error and for + `ppm_match(obs, theo, tol) = |ppm_diff| <= tol` (`constants.rs:72-74`). +- `ppm_bounds(mz, tol)` (`constants.rs:78-81`): returns `(mz - d, mz + d)` with + `d = mz * tol * 1e-6`. **Symmetric window centered on the query** `mz`, + normalized by the query itself. +- `within_ppm(a, b, tol)` (`constants.rs:92-96`): `lo = min(a,b)`, `hi = max(a,b)`, + true iff `hi - lo <= tol * 1e-6 * lo`. **Min-relative**, symmetric in argument + order. This is the canonical fragment-index predicate (fragindex_spec 2.1): + it is algebraically `hi/lo <= 1 + delta` and `ln(hi) - ln(lo) <= ln(1 + delta)`, + the last form being what makes log-space binning exact and is proven exact + against the log-bin +/-1 probe. `within_ppm_three_forms_agree` + (`constants.rs:102-122`) checks the three forms round identically away from the + edge; `within_ppm_edges` (`constants.rs:124-132`) checks edge inclusivity and + argument-order symmetry. + +The three normalize by theoretical (`ppm_diff`), by query center (`ppm_bounds`), +and by the smaller mass (`within_ppm`) respectively, so they disagree at the +tolerance edge. Index probing must use `within_ppm`; do not substitute a +`ppm_bounds` window there. + +**UniMod subset** (`mass.rs:13-26`): `Carbamidomethyl` 57.021463735, +`Oxidation` 15.994914620, `Acetyl` 42.010564684, `Phospho` 79.966331090, +`Deamidated` 0.984016106, `Methyl` 14.015650064, `Dimethyl` 28.031300128, +`Carbamyl` 43.005813726. An unknown name is `MassError::UnknownModification` +(`mass.rs:219`), never a silent zero. + +**ProForma-lite parse** (`parse_peptidoform`, `mass.rs:143-203`): optional +N-terminal `[Mod]-`, residues each optionally followed by `[Mod]`, optional +trailing `-[Mod]`. A `[Mod]` is a UniMod name or a signed float such as +`[+15.9949]` (`parse_bracket`, `mass.rs:207-223`, tries `unimod_mass` first then +`f64` parse). Non-alphabetic characters and ambiguous residues are errors. A +second mod at the same residue position accumulates (`+=`, `mass.rs:189`); the +data model has one `mods[i]` slot per residue plus separate terminal deltas +(`ParsedPeptidoform`, `mass.rs:56-62`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `Config` | config.rs:1008-1042 | Top-level config; 11 stage sections + `mbr` + `rng_seed` | +| `Config::from_json` | config.rs:1046-1051 | Parse (deny unknown) then `validate` | +| `Config::validate` | config.rs:1056-1091 | Four hard-error checks | +| `Config::apply_profile` | config.rs:1098-1112 | `dia` preset only | +| `Config::canonical_json` | config.rs:1116-1118 | Serialize for manifest hashing | +| `residue_mass` | constants.rs:26-51 | 20-AA monoisotopic table; `None` for ambiguous | +| `mass_to_mz` | constants.rs:59-62 | Neutral mass -> m/z | +| `ppm_diff` / `ppm_match` | constants.rs:66-74 | Theoretical-relative signed ppm + tolerance test | +| `ppm_bounds` | constants.rs:78-81 | Query-centered symmetric m/z window | +| `within_ppm` | constants.rs:92-96 | Min-relative canonical index predicate | +| `parse_peptidoform` | mass.rs:143-203 | ProForma-lite parser | +| `ParsedPeptidoform::fragments` | mass.rs:82-127 | b/y ions, drops b1/y1/y2 | +| `unimod_mass` | mass.rs:13-26 | 8-name UniMod subset | +| `Manifest` / `ArtifactRecord` | manifest.rs:9-51 | Run provenance | +| `record_artifact` | mumdia-io/src/lib.rs:20-39 | Build `ArtifactRecord`, hash file | +| `RejectionReason` | rejection.rs:19-113 | Ordered identification-loss ladder | +| `Label` | types.rs:33-51 | Target/Decoy; `.pin()` = +1/-1 | + +## Configuration + +Every section is `#[serde(default, deny_unknown_fields)]`. Below, each field is +listed with its default and effect. Fields marked **stub** or **default-off** are +called out explicitly. + +### Strategy enums (live variants) + +| Enum | file:line | Variants (default in bold) | Notes | +|---|---|---|---| +| `DecoyStrategy` | config.rs:17-27 | **`reverse`**, `scramble`, `diann_shift`, `none` | `diann_shift` rejected by validate; `none` produces no decoys (invalid FDR) | +| `MatcherKind` | config.rs:42-45 | `bucketed`, **`fragindex`** | fragment-matcher backend for search-seed + extract | +| `Enzyme` | config.rs:54-59 | **`trypsin_p`**, `trypsin` | cut after K/R, with/without before-P | +| `CalibrationMethod` | config.rs:68-72 | **`loess`**, `linear`, `none` | `none` rejected by validate (falls through to linear) | +| `FeatureSet` | config.rs:81-90 | **`minimal`** (14), `rich` (44), `extended` (~383) | superset battery in `stages/features/` | +| `RtPredictorKind` | config.rs:99-105 | **`native`**, `deeplc` | DeepLC is a Python sidecar | +| `FragPredictorKind` | config.rs:113-119 | **`native`**, `ms2pip` | MS2PIP is a Python sidecar | +| `RescorerKind` | config.rs:128-150 | **`native_tda`**, `mokapot`, `nn_torch`, `percolator`, `entrapment` | see RescoreConfig | +| `PeakClaim` | config.rs:164-188 | **`none`**, `winner_predicted_intensity`, `proportional`, `coelution_winner`, `coelution_proportional`, `coelution_winner_margin` | shared-peak apportionment | +| `GateMode` | config.rs:591-614 | **`apex_pearson`**, `peak_spectral`, `spectral_entropy`, `coelution`, `combined` | which spectral score `min_frag_corr` thresholds | +| `UnknownModPolicy` | config.rs:280-283 | **`error`**, `skip` | unknown-mod behavior | +| `CompetitionMode` | config.rs:713-730 | **`winner_take_all`**, `none`, `features_only`, `unique_evidence`, `margin_gated` | within-group resolution | +| `CompeteGroupBy` | config.rs:734-741 | **`precursor`**, `apex`, `peptidoform_charge` | competition grouping key | +| `RollupMethod` | config.rs:745-750 | **`top_n_sum`**, `sum` | protein rollup | +| `PeakWindowMode` | config.rs:760-771 | **`per_candidate`**, `consensus` | quant integration window | +| `NormalizeMethod` | config.rs:779-792 | `none`, **`median_ratio`**, `median` | cross-run LFQ normalization; `from_token` at 796-803 | +| `QuantQColumn` | config.rs:816-828 | **`peptide_q`**, `psm_q`, `run_psm_q` | which q column quant filters on | +| `MbrStrategy` | config.rs:883-893 | **`none`**, `empirical_library`, `rt_transfer`, `full` | **stub**: only `none` runs; the rest are config hooks (Stage D3 not wired) | +| `DecoyTransfer` | config.rs:900-907 | **`permuted_rt`**, `reverse_sequence`, `both` | MBR false-transfer null (**stub**, MBR-only) | + +### `DigestConfig` (config.rs:217-236) + +| Field | Default | Effect | +|---|---|---| +| `enzyme` | `trypsin_p` | cleavage rule | +| `missed_cleavages` | 2 | max missed cleavages | +| `min_len` | 5 | min peptide length | +| `max_len` | 50 | max peptide length | +| `decoy.strategy` | `reverse` | decoy scheme (`DecoyConfig`, config.rs:206-215) | + +### `PeptidoformsConfig` (config.rs:238-267) + +| Field | Default | Effect | +|---|---|---| +| `fixed_mods` | `[{C, Carbamidomethyl}]` | applied to every matching residue | +| `variable_mods` | `[{M, Oxidation}]` | optionally applied | +| `max_variable_mods` | 1 | max simultaneous variable mods | +| `charge_min` | 2 | lowest precursor charge | +| `charge_max` | 3 | highest precursor charge | +| `unknown_modification` | `error` | `error` or `skip` | + +`ResidueMod` (config.rs:269-276) is `{residue: char, name: String}` where `name` +is a UniMod name. `deny_unknown_fields` applies but it has no `#[serde(default)]`, +so both keys are required inside a `ResidueMod` entry. + +### `PredictFragConfig` (config.rs:290-322) + +| Field | Default | Effect | +|---|---|---| +| `predictor` | `native` | fragment-intensity source (native heuristic or MS2PIP) | +| `rt_predictor` | `native` | iRT source (native or DeepLC) | +| `charge2_from_precursor_charge` | 2 | add charge-2 fragments for precursor charge >= this | +| `top_n_fragments` | 6 | fragments kept per candidate | +| `ms2pip_model` | `"HCD"` | MS2PIP model name | +| `ms2pip_python` | `None` | interpreter for MS2PIP sidecar | +| `deeplc_python` | `None` | interpreter for DeepLC sidecar | +| `sidecar_script_dir` | `"scripts"` | directory holding worker scripts | + +### `SearchSeedConfig` (config.rs:324-360) + +| Field | Default | Effect | +|---|---|---| +| `fdr_seed` | 0.01 | seed FDR cutoff for calibration anchors | +| `fragment_tol_ppm` | 20.0 | fragment match tolerance | +| `report_psms` | 5 | max PSMs reported per spectrum | +| `min_matched_peaks` | 4 | min matched fragments per seed PSM | +| `top_n_peaks` | 300 | probe only the N most intense MS2 peaks (0 = all) | +| `matcher` | `fragindex` | matcher backend | +| `two_pass_mass_cal` | `false` | **default-off** robust two-pass mass calibration (P3.1) | + +### `RtImTrainConfig` (config.rs:362-427) + +| Field | Default | Effect | +|---|---|---| +| `calibration_method` | `loess` | RT calibration (`none` rejected by validate) | +| `q_train` | 0.01 | q cutoff for calibration anchors | +| `p_rt` | 0.95 | residual percentile for the RT window | +| `rt_window_multiplier` | 1.0 | scales the RT half-window | +| `min_seed_for_calibration` | 50 | min anchors before calibrating | +| `loess_span` | 0.3 | LOESS local-fit fraction | +| `fallback_rt_window_s` | 120.0 | fixed window when calibration cannot fit | +| `finetune_deeplc` | `false` | **default-off** DeepLC multitask fine-tune (nondeterministic; needs `deeplc_python`) | +| `finetune_epochs` | 25 | fine-tune epoch cap (early stopping usually halts earlier) | +| `finetune_patience` | 10 | early-stopping patience | +| `finetune_batch` | 0 | 0 = auto-scale batch to seed size | +| `adaptive_rt_window` | `false` | **default-off** per-region residual window (P3.2/P3.3) | +| `adaptive_rt_bins` | 12 | RT bins for the adaptive window | +| `rt_window_min_s` | 1.0 | lower clamp on any RT half-window | + +### `ExtractConfig` (config.rs:429-585) + +The largest section; the core extraction stage. Cascade thresholds and apex +selection dominate. + +| Field | Default | Effect | +|---|---|---| +| `fixed_scan_window` | 3 | scan-window floor around the apex | +| `frag_tol_ppm` | 20.0 | fragment tolerance | +| `prec_tol_ppm` | 20.0 | precursor tolerance | +| `presence_min_matched` | 3 | tier-(b) min matched fragment count | +| `presence_min_fragments` | 3 | min distinct fragments for acceptance | +| `presence_min_coelution` | 2 | min simultaneously-present fragments over the run | +| `min_frag_corr` | 0.2 | tier-(d) spectral-agreement gate (0 disables; must be in [0,1]) | +| `min_matched_fraction` | 0.0 | tier-(c) min fraction of predicted fragments observed | +| `apex_top_fragments` | 0 | signature-fragment apex (0 = all matched; superseded by `apex_count_tol`) | +| `apex_rt_prior_s` | 0.0 | Gaussian RT prior sigma on apex (0 = off) | +| `apex_count_tol` | 1 | fragment-count apex slack | +| `apex_count_window` | 1 | rolling distinct-fragment count width (1 = none; profile `dia` sets 5) | +| `emit_window_grid` | `true` | zero-filled window-grid chromatograms | +| `bucket_size` | 8192 | m/z bucket size (power of two) | +| `peak_claim` | `none` | shared-peak apportionment mode | +| `emit_contested_features` | `false` | **default-off** `contested_frac` feature (forces two-pass) | +| `peak_claim_margin` | 2.0 | dominance factor for `coelution_winner_margin` | +| `matcher` | `fragindex` | matcher backend | +| `min_coelution_run` | 0 | **default-off** min consecutive co-elution scans | +| `ms1_rescue` | `false` | **default-off** MS1-isotope rescue of gate failures | +| `retain_top_peaks` | 1 | K peak groups per candidate (1 = legacy; must be >= 1) | +| `emit_candidate_audit` | `false` | **default-off** writes `.audit.parquet` (P0.3) | +| `apex_evidence_rank` | `false` | **default-off** evidence-count apex | +| `emit_gate_diagnostics` | `false` | **default-off** four gate-score columns | +| `gate_mode` | `apex_pearson` | which spectral score `min_frag_corr` thresholds | +| `gate_coelution_min` | 0.5 | second threshold for `gate_mode = combined` | + +The `min_frag_corr` default was relaxed from a historical 0.5 to 0.2 +(config.rs:557-562). Every default-off knob is explicitly documented as +requiring entrapment/target-decoy FDR validation before use. + +### `FeaturesConfig` (config.rs:616-664) + +| Field | Default | Effect | +|---|---|---| +| `set` | `minimal` | feature set (minimal / rich / extended) | +| `coelution_corr_threshold` | 0.9 | co-elution correlation cutoff | +| `prec_tol_ppm` | 20.0 | precursor tolerance for MS1 features | +| `bound_features` | `true` | restrict trace features to the elution peak | +| `bound_peak_fraction` | 1/3 | peak-boundary descent fraction of apex height | +| `bound_peak_grace` | 0 | consecutive sub-threshold scans to bridge | +| `bound_from_confident` | `true` | learn one global peak width from confident seed PSMs | +| `bound_confident_pct` | 50.0 | percentile of confident half-widths as the shared width | + +### `CompeteConfig` (config.rs:666-703) + +| Field | Default | Effect | +|---|---|---| +| `group_by` | `precursor` | competition grouping key | +| `apex_rt_tolerance_s` | 5.0 | RT bucket for `apex` grouping | +| `mode` | `winner_take_all` | within-group resolution | +| `margin` | 0.0 | score margin for `margin_gated` | +| `unique_evidence_min_fragments` | 2 | min unique fragments for `unique_evidence` | +| `emit_competition_audit` | `false` | **default-off** writes `.compete_audit.parquet` | + +### `QuantConfig` (config.rs:830-874) + +| Field | Default | Effect | +|---|---|---| +| `q_threshold` | 0.01 | peptide-q cutoff for inclusion | +| `top_n_fragments` | 3 | fragments summed per peptidoform | +| `top_n_peptides` | 3 | peptides summed per protein group (TopNSum) | +| `rollup` | `top_n_sum` | protein rollup method | +| `bound_peak` | `true` | integrate only over the detected peak window | +| `peak_fraction` | 1/6 | descent threshold for the peak-window walk | +| `peak_grace` | 1 | zig-zag grace (bridge N sub-threshold scans) | +| `peak_window_mode` | `per_candidate` | per-candidate vs consensus window | +| `reliable_q` | 0.001 | confident-set q for the consensus width | +| `q_filter` | `peptide_q` | which q column to filter on (`QuantQColumn`) | + +### `RescoreConfig` (config.rs:951-1002) + +| Field | Default | Effect | +|---|---|---| +| `classifier` | `native_tda` | rescorer backend | +| `folds` | 3 | CV folds | +| `train_fdr` | 0.01 | positive-selection FDR | +| `num_iter` | 10 | semi-supervised iterations (native) | +| `python` | `None` | interpreter for a Python rescorer (mokapot/nn_torch/entrapment) | +| `percolator_bin` | `None` | percolator.exe path | +| `entrapment_marker` | `None` | protein substring marking spike-in negatives (required for `entrapment`) | +| `entrapment_exclude` | `None` | substring that de-marks (own-species) | +| `entrapment_contaminant_markers` | `[]` | substrings that keep a spike-in hit as a real target | +| `entrapment_ratio` | 1.0 | N_real_lib / N_entrap_lib scaling | +| `strict` | `false` | **default-off** turn any rescorer failure into a hard error (recommended for production) | + +### `MbrConfig` (config.rs:909-949) — stub + +Config hooks only; the MBR stage (D3) is not wired into the run chain. Fields: +`strategy` (`none` default), `q_anchor` 0.01, `min_anchor_runs` 2, +`q_transfer` 0.01, `rt_window_s` 20.0, `decoy_transfer` `permuted_rt`, +`consensus_corr_min` 0.0, `requant_all` `false`, `python` `None`. `mbr` is the +one top-level section with an explicit extra `#[serde(default)]` attribute +(config.rs:1022-1023). With `strategy = none` the chain is byte-identical to no +MBR. + +### Top-level `Config` (config.rs:1008-1024) + +`rng_seed` (default 0) plus the eleven stage sections and `mbr`. `rng_seed` +seeds every RNG (decoy scramble, CV fold assignment) for determinism. + +### Removed / not present + +The recent cleanup deleted several dead enums that older docs still mention +(`DecoySource`, `ToleranceRegime`, `ScanWindowMode::PeakWidthDerived`, +`FeatureSet::Custom`). They are **not** in the current `config.rs`. Do not +re-add or document them; if you see them referenced elsewhere, that reference is +stale. + +## Invariants, determinism, gotchas + +- **Single source of masses.** No crate outside `mumdia-core` defines a residue + mass, `PROTON`, `WATER`, `AMMONIA`, or `ISOTOPE_SPACING` + (`constants.rs:10-22`). `ISOTOPE_SPACING = 1.003354835` is the 13C-12C mass + difference used for MS1 envelope spacing. `PROTON = 1.007276466812` is the + physically correct proton mass, deliberately not DIA-NN's H-atom value + 1.007825035 (`constants.rs:8-10`). All constants are own-derived from + CODATA/AME; nothing is copied from another engine (clean-room boundary). +- **Three ppm predicates are not interchangeable.** `within_ppm` (min-relative) + is the only one the fragment index may use; `ppm_bounds` (query-centered) and + `ppm_diff`/`ppm_match` (theoretical-relative) disagree at the tolerance edge. + Substituting one for another shifts which fragments match at the boundary. +- **Validation is fail-loud, not fail-quiet.** `DiannShift` decoys and + `CalibrationMethod::None` are rejected rather than silently degraded + (`config.rs:1058-1073`). `DecoyStrategy::None` is *not* rejected but produces no + decoys and therefore an invalid target-decoy FDR; treat it as a diagnostic-only + setting. +- **`deny_unknown_fields` everywhere.** A misspelled key fails the whole load, so + a config file cannot silently ignore a setting the author intended. +- **Determinism.** `rng_seed` seeds all randomness. `canonical_json` is a stable + `serde_json::to_string` (field order fixed by struct declaration order), so the + same config hashes to the same `config_hash` across runs. `manifest.artifacts` + and `model_identities` are `BTreeMap`s (`manifest.rs:28-30`), so manifest key + order is deterministic. Note `finetune_deeplc` fine-tuning is itself + nondeterministic (no torch seed) and breaks byte-identical reproducibility when + enabled. +- **Fragment generation drops b1/y1/y2** unconditionally (`mass.rs:93`, + `mass.rs:113`); a peptide shorter than 2 residues yields no fragments + (`mass.rs:85-87`). +- **`ParsedPeptidoform::neutral_mass` uses `.expect`** on residue masses + (`mass.rs:70`), safe only because `parse_peptidoform` already rejected + ambiguous residues. Constructing a `ParsedPeptidoform` by hand with a + non-standard residue byte would panic. +- **Second mod at the same position accumulates** (`mass.rs:189`), which is a + behavior difference from engines that drop it; terminal mods are separate + fields, not part of `mods[i]`. +- **`PSMS_SCORED` is v2, all other artifacts v1.** A reader that hardcodes v1 for + the scored table will misread the three multi-context q columns. +- **`content_hash` is the file's blake3, not the logical content.** Any byte + change (compression, column order) changes the hash; it is a change detector, + not a canonical-content identity. + +## How to extend / modify + +- **Add a config field.** Add it to the section struct, give it a value in that + struct's `Default` impl (every field must have one; `#[serde(default)]` is at + the struct level), and document the effect inline. Do not hardcode a choice the + config could express (project convention). If the field can be set to a value + that would silently corrupt results, add a `validate()` check + (`config.rs:1056-1091`) that rejects it with `ConfigError::Invalid`. +- **Add a strategy variant.** Extend the enum, keep `#[serde(rename_all = + "snake_case")]`, and (if it is not implementable yet) reject it in `validate` + the way `DiannShift` and `CalibrationMethod::None` are, rather than letting it + fall through. State default-off status in the doc comment. +- **Add a UniMod modification.** Add a `name => mass` arm to `unimod_mass` + (`mass.rs:13-26`). Use the PSI-MS/UniMod monoisotopic delta so the Python + sidecar adapters map the name. No table is copied from another tool. +- **Add an artifact / bump a schema.** Add or bump the constant in + `schema.rs:6-24`. Bump the version whenever the column set changes + (as `PSMS_SCORED` went to 2). Update the producing stage's `ArtifactReport` + (`schema_name`, `schema_version`) and its `record_artifact` call so the manifest + and the sidecar report agree. +- **Add a manifest field.** Extend `Manifest` or `ArtifactRecord` + (`manifest.rs`); both are plain serde structs. Keep new maps as `BTreeMap` for + deterministic key order. +- **Never edit `plan.md`** (gitignored spec, project rule). Keep validated + numbers consistent across `README`, `COMPARISON.md`, and `CLAUDE.md`. diff --git a/docs/03_io_layer.md b/docs/03_io_layer.md new file mode 100644 index 0000000..7596ac3 --- /dev/null +++ b/docs/03_io_layer.md @@ -0,0 +1,377 @@ +# IO layer: Col/Table, Parquet, report.json, hashing, inspect +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +The `mumdia-io` crate is the on-disk contract layer for the whole engine. Every +stage reads its path-addressable inputs and writes its outputs through this +crate, so no stage hand-rolls Arrow `RecordBatch`es or touches the Parquet +reader/writer directly. The crate provides five things: + +1. A small typed column model (`Col`) and a read-back table (`Table`) over + Arrow + Parquet, so a stage declares its schema as a `Vec` and reads it + back by column name with typed getters (`table.rs`). +2. Parquet write (`write_table`) and read (`Table::read`), SNAPPY-compressed, + the open self-describing interstage format (plan.md Section 3.3). +3. The per-artifact `.report.json` sidecar (`ArtifactReport` in + `report.rs`) so a stage can be evaluated (row counts, resolved params, key + distributions, model identity, timing) without loading the full table. +4. blake3 content hashing of files and strings (`hash.rs`), which feeds the + `content_hash` on every artifact record and the `config_hash` used to + invalidate downstream artifacts. +5. The `mumdia inspect ` implementation (`inspect` in `lib.rs`): + schema + head sample + row count for any Parquet file. + +This crate reads no `Config` fields of its own. It is a mechanism layer; the +concrete per-artifact column schemas live in each stage's `write_table` call, +and the frozen schema identifiers live in `mumdia-core` (`schema.rs`). The only +"configuration" here is compile-time (SNAPPY compression, a 64 KiB hash buffer, +the schema-id tuples). + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | crate root: `init_logging`, `record_artifact`, `inspect`; re-exports the modules | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | `Col` enum (write side), `write_table`, `Table` (read side) and the typed getters | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport` struct + `write_for` (the `.report.json` sidecar) | +| `rust/mumdia/crates/mumdia-io/src/hash.rs` | `blake3_file`, `blake3_str` | +| `rust/mumdia/crates/mumdia-io/src/json.rs` | `write_json`, `read_json` (pretty JSON via serde) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | frozen `(logical name, schema version)` tuples for every artifact | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `ArtifactRecord` / `Manifest` (populated from `record_artifact`) | + +## Inputs and outputs + +The IO layer is schema-agnostic: it does not itself consume or produce a fixed +set of named artifacts. It is the read/write mechanism that every stage uses. +What is fixed here is the artifact **identity registry** and the shape of the +two JSON sidecars. + +### Artifact schema registry (`mumdia-core/src/schema.rs:7-23`) + +Each artifact carries a logical schema name and version so a stage can validate +its inputs and a model is never applied under a mismatched schema. The tuples +are `(name, version)`: + +| logical name | version | producing stage | +|---|---|---| +| `spectra_ms1` | 1 | convert | +| `spectra_ms2` | 1 | convert | +| `isolation_windows` | 1 | convert | +| `ms2_to_ms1` | 1 | convert | +| `peptides` | 1 | digest | +| `peptidoforms` | 1 | peptidoforms | +| `fragment_library_precursors` | 1 | predict-frag / library-input | +| `fragment_library_fragments` | 1 | predict-frag / library-input | +| `seed_psms` | 1 | search-seed | +| `run_windows` | 1 | rt-im-train | +| `psms_extracted` | 1 | extract | +| `chromatograms` | 1 | extract | +| `features` | 1 | features | +| `psms_competed` | 1 | compete | +| `psms_scored` | 2 | rescore | +| `peptide_quant` | 1 | quant | +| `protein_group_quant` | 1 | quant | + +Note only `psms_scored` is at version 2; everything else is version 1. The +version is not consumed for a hard gate in this crate; it is recorded in the +report and the manifest so a downstream reader can detect a mismatch. + +### Example concrete column schemas + +The IO crate stores no schema definitions; a stage's `write_table(path, vec![ +Col::… ])` is the schema. Two examples read from the actual code: + +`isolation_windows` (`stages/convert.rs:223-228`), one row per distinct window: +`window_index: u32`, `target: f64`, `lower: f64`, `upper: f64`. +`ms2_to_ms1` (`stages/convert.rs:231-237`): `ms2_scan_index: u32`, +`ms1_scan_index: i32`. + +`peptides` (`stages/digest.rs:216-225`): `id: u32`, `peptide: utf8`, +`protein: utf8`, `start: i32`, `end: i32`, `label: utf8`, `target_id: i32`, +`decoy_strategy: utf8`. + +To see any artifact's real schema, run `mumdia inspect ` (see +below) rather than trusting a doc; the schema is authoritative on disk. + +### JSON sidecars produced + +- `.report.json` (per artifact, written by every stage via + `ArtifactReport::write_for`, `report.rs:28`). Fields below. +- `manifest.json` (written once by the `run` orchestrator from the collected + `ArtifactRecord`s, `stages/run.rs`). This crate supplies the per-record + builder `record_artifact` (`lib.rs:20`); it does not write the manifest file. + +## How it works + +### Write side: `Col` -> Arrow -> Parquet + +`Col` (`table.rs:23-42`) is an enum with one variant per supported column type. +Each variant carries `(String name, Vec)`. The scalar variants are +`I64`, `I32`, `U32`, `F64`, `F32`, `Bool`, `Str`; the nullable variants are +`OptF64`, `OptF32`, `OptI32`, `OptStr` (each a `Vec>`); the list +variants are `ListF32`, `ListF64`, and `LargeListF32`. `LargeListF32` +(`table.rs:41`) is encoded as an Arrow `LargeList` with 64-bit offsets, needed +when the total list-value count across all rows can exceed the ~2.1 billion +limit of a 32-bit `ListArray` offset buffer (for example per-fragment +chromatograms when extraction accepts a very large candidate set). + +`Col` has three private helpers used by the writer: +- `name()` (`table.rs:45`): the column name, via a match over every variant. +- `len()` (`table.rs:64`): the row count of the inner `Vec`. +- `field()` (`table.rs:83`): the Arrow `Field`. Scalar variants are + `nullable = false`; the `Opt*` and all list variants are `nullable = true`. + List inner items are declared `Field::new("item", Float32/64, true)`. +- `into_array()` (`table.rs:107`): the consuming conversion into an `ArrayRef`. + It **moves** the `Vec` into the Arrow array instead of cloning, so the column + data is copied only once during a write. Scalar `Vec` and `Vec>` + go straight through `PrimitiveArray::from`; list variants are built with a + `ListBuilder`/`LargeListBuilder`, appending each row's slice and then + `append(true)` (a present, possibly empty, list). + +`write_table(path, cols)` (`table.rs:151`) is the single write entry point and +returns the row count as `u64`: +1. Reject an empty column set (`table.rs:152`). +2. Reject duplicate column names via a `HashSet` (`table.rs:157-162`). Arrow + allows duplicate names but readers resolve a name to the first match, which + would silently hide the second column, so this is a hard error. +3. Take `nrows` from column 0 and require every column to match + (`table.rs:163-173`); a mismatch is a hard error naming the offending column. +4. Build the `Schema` from `field()` over all columns (`table.rs:174-175`), + then consume the columns into `ArrayRef`s (`table.rs:178`). The `fields` + vector is captured before the consume so the schema still has everything. +5. `RecordBatch::try_new` (`table.rs:179`), create parent dirs + (`create_dir_all(...).ok()`, best-effort, `table.rs:182-184`), create the + file, build `WriterProperties` with `Compression::SNAPPY` (`table.rs:186`), + and write a single batch through `ArrowWriter`, then `close()` + (`table.rs:189-191`). One `write_table` call produces exactly one row group / + one logical batch. + +### Read side: Parquet -> `Table` -> typed `Vec` + +`Table` (`table.rs:197-201`) holds the `Arc`, the `Vec`, +and `nrows`. `Table::read(path)` (`table.rs:204`) opens the file, builds a +`ParquetRecordBatchReaderBuilder`, captures the schema, then iterates the reader +collecting every batch and summing `num_rows()`. The whole file is materialized +into memory; there is no streaming or predicate pushdown. + +Column access is by name. `idx(name)` (`table.rs:228`) resolves a name to a +column index via `schema.index_of`, returning a descriptive error listing all +column names if the name is absent. The typed getters each downcast every +batch's column to the concrete Arrow array type and concatenate across batches +into one `Vec`. They error with `"column '' is not "` if the +downcast fails, so the type is checked at read time. + +The getters and their exact null behaviour: + +- `f64` (`table.rs:234`), `f32` (`table.rs:254`): fast path when + `null_count() == 0` uses `extend_from_slice(a.values())`; otherwise iterate + and map a null to `f64::NAN` / `f32::NAN`. Nulls become NaN. +- `i64` (`table.rs:274`), `i32` (`table.rs:294`), `u32` (`table.rs:314`): fast + path on no nulls; otherwise iterate pushing `a.value(k)` **without checking + `is_null`**. A null therefore comes through as the underlying buffer value + (typically 0), not as a sentinel. See the gotcha below. +- `bool` (`table.rs:334`): always iterates `a.value(k)`; never checks null. +- `str` (`table.rs:350`): iterates; a null maps to an empty `String`. +- `opt_f64` (`table.rs:370`): the only null-preserving getter. Returns + `Vec>`, mapping a null to `None`. +- `list_f32` (`table.rs:389`): reads an f32 list column and accepts **both** + `List` (32-bit offsets) and `LargeList` (64-bit offsets) encodings, so a + chromatogram artifact written by `Col::ListF32` or `Col::LargeListF32` reads + back through the same call. An outer null list becomes an empty `Vec`; a + present list is materialized via `f.values().to_vec()`. + +`column_names()` (`table.rs:224`) returns the schema field names in order. + +### Hashing (`hash.rs`) + +`blake3_file(path)` (`hash.rs:8`) streams the file in 64 KiB chunks +(`[0u8; 1 << 16]`) through a `blake3::Hasher` and returns the hex digest. This +is the artifact `content_hash`. `blake3_str(s)` (`hash.rs:23`) is a one-shot +hex digest of a string, used for the `config_hash`. The engine derives the +config hash from `Config::canonical_json()` (`config.rs:1116`, a plain +`serde_json::to_string`), for example at `main.rs:404`. The `convert` command is +a deliberate exception: because `--max-spectra`, `--top-peaks-ms2`, and +`--top-peaks-ms1` are CLI caps that change the spectra output but are not part +of `Config`, they are folded into the hash with a unit-separator (`\u{1f}`) +alongside the canonical config JSON so two different caps do not collapse to the +same `config_hash` (`main.rs:389-392`). + +### Report sidecar (`report.rs`) + +`ArtifactReport` (`report.rs:11-24`) is the per-artifact JSON summary. A stage +constructs it and calls `write_for(artifact_path)` (`report.rs:28`), which +appends `.report.json` to the artifact path and writes it via +`json::write_json` (pretty-printed). Fields: + +| field | type | meaning | +|---|---|---| +| `logical_name` | String | logical artifact name (usually the schema name) | +| `schema_name` | String | schema name from the `schema.rs` tuple | +| `schema_version` | u32 | schema version from the tuple | +| `stage` | String | producing stage, e.g. `"digest"` | +| `rows` | u64 | row count returned by `write_table` | +| `content_hash` | String | `blake3_file` of the artifact just written | +| `params` | `serde_json::Value` | the resolved parameters the stage actually used | +| `stats` | `BTreeMap` | summary key distributions / metrics (ordered) | +| `model_identity` | `Option` | sidecar / predictor identity, else `None` | +| `elapsed_ms` | u128 | wall-clock ms for the stage | + +`stats` is a `BTreeMap` (not a `HashMap`) so the JSON key order is +deterministic. Concrete example: `digest` records `params` with enzyme, missed +cleavages, length bounds, decoy strategy, and rng seed, and `stats` with +`n_targets` / `n_decoys` (`stages/digest.rs:229-249`). `convert` has a +stage-local helper `write_reports` (`stages/convert.rs:272-294`) that writes one +report per artifact with a shared `params` and empty `stats`; note this +`write_reports` lives in `convert.rs`, it is not part of the `mumdia-io` public +API. Every other stage builds its `ArtifactReport` directly (see the +`write_for` call sites in `align`, `compete`, `extract`, `search_seed`, +`predict_frag`, `peptidoforms`, `rt_im_train`, `rescore`, `features`, `quant`). + +### `record_artifact` and the manifest + +`record_artifact(logical_name, schema, path, rows, stage, config_hash)` +(`lib.rs:20`) builds an `ArtifactRecord` (`manifest.rs:10-20`) for the run +manifest. It hard-codes `format = "parquet"`, hashes the file with +`blake3_file`, and copies the schema name/version and the config hash. The +`run` orchestrator calls it once per artifact and records each into the +`Manifest` (`stages/run.rs`, many call sites around `record_artifact(...)`), +which is then serialized to `manifest.json`. Standalone single-stage invocations +write only the `.report.json` sidecar, not a manifest. + +### `inspect` (`lib.rs:43`) + +`inspect(path)` reads the whole table via `Table::read`, then builds a string: +`artifact: `, `rows: `, a `schema:` block listing each field as +` : ` with a ` (nullable)` suffix when the field is nullable, +and a `head:` block. The head is the first up-to-10 rows sliced from the +**first batch only** (`first.slice(0, min(num_rows, 10))`, `lib.rs:57-59`), +formatted with `arrow::util::pretty::pretty_format_batches`. If pretty-print +fails, the head is silently omitted (`Err(_) => {}`, `lib.rs:66`). The CLI +command `Cmd::Inspect { artifact }` (`main.rs:257`) simply prints the returned +string (`main.rs:658-659`). + +### Logging + +`init_logging()` (`lib.rs:13`) initializes `tracing_subscriber` once, honouring +`RUST_LOG` and defaulting to `info`, with `with_target(false)`. It uses +`try_init` so a second call is a no-op rather than a panic. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `Col` (enum) | `table.rs:23` | typed write-side column: scalar, `Opt*`, and list variants | +| `Col::field` | `table.rs:83` | Arrow `Field`; scalars non-nullable, `Opt*`/lists nullable | +| `Col::into_array` | `table.rs:107` | consuming move of the `Vec` into an `ArrayRef` (copy once) | +| `write_table` | `table.rs:151` | validate + write one SNAPPY Parquet batch; returns row count | +| `Table` (struct) | `table.rs:197` | read-back table: schema, batches, nrows | +| `Table::read` | `table.rs:204` | read a Parquet file fully into memory | +| `Table::f64` / `f32` | `table.rs:234` / `254` | float getters; null -> NaN | +| `Table::i64`/`i32`/`u32` | `table.rs:274`/`294`/`314` | integer getters; null NOT checked (-> buffer value) | +| `Table::bool` | `table.rs:334` | bool getter; null NOT checked | +| `Table::str` | `table.rs:350` | string getter; null -> `""` | +| `Table::opt_f64` | `table.rs:370` | only null-preserving getter; -> `Vec>` | +| `Table::list_f32` | `table.rs:389` | f32 list getter; reads `List` and `LargeList`; null row -> empty `Vec` | +| `ArtifactReport` | `report.rs:11` | per-artifact JSON summary struct | +| `ArtifactReport::write_for` | `report.rs:28` | write `.report.json` | +| `blake3_file` | `hash.rs:8` | streamed blake3 hex digest of a file (`content_hash`) | +| `blake3_str` | `hash.rs:23` | one-shot blake3 hex digest of a string (`config_hash`) | +| `write_json` / `read_json` | `json.rs:7` / `16` | pretty serde JSON write / read, creating parent dirs on write | +| `record_artifact` | `lib.rs:20` | build an `ArtifactRecord` (format hard-coded `"parquet"`) | +| `inspect` | `lib.rs:43` | schema + head(<=10, first batch) + row count as a string | +| `init_logging` | `lib.rs:13` | `tracing` init honouring `RUST_LOG`, default `info` | + +## Configuration + +This subsystem reads no `Config` fields. It has no config surface of its own, so +the recent pruning of dead config fields in `mumdia-core::config` did not touch +it. Its behaviour is fixed at compile time: + +- Compression is `SNAPPY`, hard-coded in `write_table` (`table.rs:186-188`); it + is not configurable and there is no other codec path. +- The hash read buffer is 64 KiB (`hash.rs:11`). +- Schema identifiers are the constants in `mumdia-core/src/schema.rs:7-23`; a + new artifact requires a new tuple there, not a config change. +- Dependency features are pinned in the workspace `Cargo.toml`: `arrow` v59 with + `["prettyprint"]` (needed by `inspect`), `parquet` v59 with + `default-features = false, features = ["arrow", "snap"]` (pure-Rust snap, no + cmake/C toolchain), `blake3` v1. Do not re-enable Parquet default features; + they pull in a C compression backend that breaks the pure-Rust build + constraint (see CLAUDE.md environment gotchas). + +## Invariants, determinism, gotchas + +- **Determinism.** `write_table` writes columns in the order given and a single + batch, so byte layout is stable for stable input. `ArtifactReport.stats` is a + `BTreeMap`, so JSON key order is fixed. `config_hash` comes from + `Config::canonical_json` (serde field order) so it is reproducible. blake3 is + deterministic. All of this is required by plan.md Section 7. +- **Integer/bool null gotcha.** `i64`/`i32`/`u32`/`bool` getters do not check + `is_null`; in the presence of nulls they return the raw buffer value (usually + 0 / false), silently. This is safe today only because the MVP integer/bool + columns are written non-nullable (the `I*`/`Bool` variants set + `nullable = false`). If you write a nullable integer column via `OptI32`, + reading it back with `i32()` will erase the null distinction. Only `opt_f64` + is null-preserving on the read side. This is a known gap (CLAUDE.md + "Correctness": add null-aware getters). +- **Read-side coverage asymmetry.** The write side has `OptF32`, `OptI32`, + `OptStr`, `ListF64` variants with no matching null-aware / typed reader. + `OptF32` reads back via `f32()` (null -> NaN, acceptable), `OptStr` via + `str()` (null -> `""`), `OptI32` via `i32()` (null erased), and there is no + `list_f64` reader at all. Add the corresponding getter before relying on a + round-trip of those variants. +- **Inner-list nulls are not represented.** List item fields are declared + nullable, but the builders only ever `append(true)` present lists and never + append inner-element nulls; `list_f32` reads inner values with + `values().to_vec()` and cannot surface an inner null. Only the outer + list-level null (empty `Vec`) is modeled. +- **Duplicate column names are rejected** at write time (`table.rs:157-162`) + because Arrow readers resolve to the first match and would hide the rest. +- **Length equality is enforced**: all columns must have identical length or + `write_table` errors (`table.rs:163-173`). +- **Everything is loaded into memory.** `Table::read` collects all batches and + `inspect` reads the full table just to print 10 rows; there is no streaming + path. For very large artifacts this is a real memory cost. `inspect`'s head is + taken from the first batch only, so a file with tiny leading batches shows few + rows even when later batches are large. +- **`inspect` swallows pretty-print errors** (`lib.rs:66`): a formatting failure + omits the head silently rather than erroring, so absence of a head block is + not proof of an empty table. +- **`create_dir_all` on write is best-effort** (`.ok()`, `table.rs:183`); a real + permission failure surfaces later at `File::create`, not at the mkdir. +- **`record_artifact` hard-codes `format = "parquet"`** (`lib.rs:31`); it is not + suitable for a non-Parquet artifact without a change there. +- **LargeList transparency.** `list_f32` intentionally reads both `List` and + `LargeList`, so an artifact whose encoding differs between two builds (32- vs + 64-bit offsets) still reads identically; do not assume a fixed offset width + when consuming chromatograms. + +## How to extend / modify + +- **Add a new column type.** Add a variant to `Col` (`table.rs:23`) and extend + the four matches: `name`, `len`, `field`, `into_array`. Decide nullability in + `field`. Then add the matching read-side getter on `Table` following the + downcast + concatenate pattern of the existing getters, and be explicit about + null handling (prefer an `opt_*` return over a silent sentinel for anything + that can be null in practice). +- **Add a null-aware integer/string getter.** This is the standing correctness + item. Mirror `opt_f64` (`table.rs:370`): iterate `is_null(k)` and return + `Vec>`. Do not change the existing non-optional getters' signatures; + add new ones so current call sites are unaffected. +- **Register a new artifact.** Add a `(name, version)` tuple to + `mumdia-core/src/schema.rs` and pass it to `write_table` (schema = the + `Vec` you write) plus the `ArtifactReport`/`record_artifact` calls. Bump + the version only on a breaking column-schema change (as was done for + `psms_scored` -> 2), and document the change so readers can detect a mismatch. +- **Change compression.** It is a one-line change in `write_table` + (`table.rs:186`); keep it to a codec available under the pinned pure-Rust + Parquet features, and re-measure round-trip and file size. +- **Extend the report.** Add a field to `ArtifactReport` (`report.rs:11`). Keep + `stats` a `BTreeMap` for deterministic key order. Since it is + `Serialize`/`Deserialize`, adding a field is backward compatible only if it is + `Option` or has a serde default; otherwise old `.report.json` files fail to + deserialize. +- **Reuse the inspector.** Any tool that needs a schema/head view should call + `mumdia_io::inspect` rather than re-opening Parquet, so the output format stays + consistent with the `mumdia inspect` CLI command. diff --git a/docs/04_convert.md b/docs/04_convert.md new file mode 100644 index 0000000..2e72243 --- /dev/null +++ b/docs/04_convert.md @@ -0,0 +1,302 @@ +# convert (Stage 0): mzML to normalized spectra +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +`convert` is Stage 0 of the pipeline (PLAN.md Stage 0). It is the single point in +the engine that touches the vendor mass-spectrometry format. It reads one mzML +run through the `mzdata` crate and writes a normalized, self-describing spectra +artifact set (four Parquet files) that every downstream stage consumes instead of +the raw file. Everything after this stage (`search-seed`, `rt-im-train`, +`extract`, `features`, `quant`) reads spectra only through +`crates/mumdia/src/spectra.rs`, never through `mzdata`. Consequences: the +vendor-format dependency is isolated here, and any format quirk (profile vs +centroid, AIF/all-ion windows, missing precursor) must be resolved at this stage +because later stages assume the normalized shape. + +The MVP is mzML-only and 3D. Ion-mobility columns are therefore absent from the +artifacts; the in-memory `Peak`/`IsolationWindow` types carry `Option` IM fields +that convert always leaves `None` (`crates/mumdia-core/src/types.rs:11`, `:21`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/convert.rs` | The whole stage: mzML reading, centroiding, peak capping, window synthesis, artifact writing. | +| `rust/mumdia/crates/mumdia/src/main.rs` (`Cmd::Convert`, lines 18-37, 377-401) | CLI subcommand; builds the provenance `config_hash` and calls `convert::run`. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` (lines 152-158) | The `run` orchestrator's call into convert (top_peaks_ms1 hardcoded to 0). | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | The read-back side: `load_ms1` / `load_ms2` turn the artifacts back into in-memory scans for downstream stages. | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` (lines 7-10) | Frozen `(logical name, version)` identifiers for the four output artifacts. | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | `Col` / `write_table` typed Parquet writer used to emit the artifacts. | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport`, the `.report.json` sidecar written per output. | +| `rust/mumdia/crates/mumdia-core/src/types.rs` | `Peak`, `Ms2Scan`, `IsolationWindow` (used on the read-back side). | + +## Inputs and outputs + +Input: one mzML file path (`--mzml`). `mzdata` is pinned at version `0.65` with +`default-features = false, features = ["mzml", "miniz_oxide"]` +(`rust/mumdia/Cargo.toml:26`), so only mzML is compiled in; `miniz_oxide` gives +pure-Rust gzip for compressed mzML. No other input is read. The stage takes no +config file (see Configuration). + +Outputs: four Parquet artifacts written to `--out-dir`, each with a sibling +`.report.json`. All are written with SNAPPY compression via +`write_table` (`table.rs:151`). List columns (`mz`, `intensity`) are Arrow +`List` with the list field marked nullable, but convert never writes a +null list; an empty scan is a non-null empty list. + +### `spectra_ms1.parquet` (`SPECTRA_MS1`, schema v1; written at `convert.rs:177`) + +| Column | Type | Meaning | +|---|---|---| +| `scan_index` | `u32` | Global monotonic index over all spectra in the run. | +| `rt_seconds` | `f64` | Retention time in seconds (mzdata minutes x 60). | +| `mz` | `List` | Centroided, m/z-ascending peak m/z (widened to f64 on read). | +| `intensity` | `List` | Peak intensities, aligned to `mz`. | + +### `spectra_ms2.parquet` (`SPECTRA_MS2`, schema v1; written at `convert.rs:204`) + +| Column | Type | Meaning | +|---|---|---| +| `scan_index` | `u32` | Global monotonic index (shares the same counter as MS1). | +| `id` | `Utf8` | Native mzML spectrum id string (`spec.id()`), kept for traceability/USI. | +| `rt_seconds` | `f64` | Retention time in seconds. | +| `window_id` | `u32` | Index into `isolation_windows.parquet`, dedup by (lower, upper). | +| `window_target` | `f64` | Isolation window center m/z (0.0 for AIF/all-ion). | +| `window_lower` | `f64` | Isolation window lower bound m/z. | +| `window_upper` | `f64` | Isolation window upper bound m/z (1.0e6 for AIF/all-ion). | +| `precursor_mz` | `f64?` (nullable) | Selected precursor m/z, or null when absent. | +| `precursor_charge` | `i32?` (nullable) | Precursor charge, or null when absent. | +| `mz` | `List` | Centroided, m/z-ascending fragment m/z. | +| `intensity` | `List` | Fragment intensities, aligned to `mz`. | + +### `isolation_windows.parquet` (`ISOLATION_WINDOWS`, schema v1; written at `convert.rs:221`) + +| Column | Type | Meaning | +|---|---|---| +| `window_id` | `u32` | First-seen id (0-based) of a distinct (lower, upper) window. | +| `target` | `f64` | Window center m/z. | +| `lower` | `f64` | Window lower bound m/z. | +| `upper` | `f64` | Window upper bound m/z. | + +### `ms2_to_ms1.parquet` (`MS2_TO_MS1`, schema v1; written at `convert.rs:231`) + +| Column | Type | Meaning | +|---|---|---| +| `ms2_scan_index` | `u32` | MS2 scan_index. | +| `ms1_scan_index` | `i32` | scan_index of the most recent preceding MS1, or `-1` if none seen yet. | + +The `-1` sentinel (not null) is the "no preceding MS1" marker; it occurs for MS2 +scans acquired before the first MS1 in the run (`convert.rs:166`). + +## How it works + +Control flow is `run` (`convert.rs:103-270`), a single linear pass over the mzML +reader plus four table writes. + +1. **Open the run.** `mzdata::MZReader::open_path(p.mzml)` (`convert.rs:107`) + returns an iterator over spectra in acquisition order. The out directory is + created first (`convert.rs:105`). + +2. **Iterate spectra.** For each spectrum (`convert.rs:123`): if `max_spectra > 0` + and the handled count reached it, break (`convert.rs:124`). `scan_index = idx` + and `idx += 1` run for every spectrum regardless of MS level + (`convert.rs:128-129`), so `scan_index` is a run-global monotonic counter. + Retention time is `spec.start_time() * 60.0` because mzdata returns start time + in minutes and the artifact stores seconds (`convert.rs:130`). + +3. **Dispatch on MS level** (`convert.rs:131`): + - **MS1** (`convert.rs:132-139`): centroid+cap peaks with `top_peaks_ms1`, + push into the MS1 accumulators, and record `last_ms1_index = scan_index` so + subsequent MS2 scans can point back to it. + - **MS2** (`convert.rs:140-167`): centroid+cap peaks with `top_peaks_ms2`, then + resolve the isolation window and precursor (details below), and append the + `(ms2_scan_index, ms1_scan_index)` mapping row. + - **Any other level** (MS3, etc.): ignored by the `_ => {}` arm + (`convert.rs:168`), but it still consumed a `scan_index`, so the per-level + tables have globally unique but non-contiguous indices. + +4. **Centroiding** happens inside `peaks_of` (`convert.rs:56`). It first pulls the + raw arrays via `spec.raw_arrays()` -> `mzs()` (f64) and `intensities()` (f32) + (`convert.rs:57-64`). If `spec.signal_continuity() == SignalContinuity::Profile` + (`convert.rs:65`) it calls `centroid` (`convert.rs:19`); already-centroided + spectra pass through unchanged. `centroid` does simple local-maxima detection + with 3-point parabolic m/z refinement: + - If fewer than 3 samples, return the input as-is (`convert.rs:21-23`). + - Compute a relative noise floor `floor = max_intensity * 1e-4` + (`convert.rs:24-25`), i.e. 0.01% of the base peak. This threshold is + hardcoded, not a config field. + - For each interior sample `i` in `1..n-1` with neighbors `y0, y1, y2` + (`convert.rs:28-34`): keep it only if `y1 > floor` and it is a local maximum + under the asymmetric test `y1 >= y0 && y1 > y2` (left inclusive, right + strict, so a flat-topped pair keeps the left sample once). Otherwise skip. + - Parabolic apex refinement on m/z (`convert.rs:35-43`): with + `denom = y0 - 2*y1 + y2`, the sub-sample offset is + `delta = 0.5 * (y0 - y2) / denom` when `|denom| > 1e-12`, else 0. The local + m/z spacing is `spacing = (mz[i+1] - mz[i-1]) * 0.5`, and the refined center + is `cm = mz[i] + delta * spacing`. Only the m/z is refined; the emitted + intensity is the raw apex sample `y1`, not a parabola-interpolated height + (`convert.rs:44-45`). + - If no local maximum survived, `centroid` returns the original profile arrays + as a fallback (`convert.rs:47-51`). This is a safety net; a pathological + profile scan can therefore leak raw profile samples downstream. + +5. **Filter, cap, sort** (still in `peaks_of`, `convert.rs:70-83`): drop peaks + with intensity `<= 0`. If `top_n > 0` and there are more than `top_n` peaks, + sort descending by intensity and truncate to `top_n` (`convert.rs:76-79`). + Then always sort ascending by m/z (`convert.rs:80`). Finally, cast m/z to + `f32` for output (`*m as f32`, `convert.rs:81`) while intensity stays `f32`. + Storing observed m/z as f32 halves peak storage; the read side widens back to + f64 (`spectra.rs:69`, `spectra.rs:131`). At the ppm tolerances used in DIA + matching, f32 m/z (~7 significant digits) is adequate for observed peaks; + library/theoretical m/z stay f64. + +6. **Isolation window and precursor resolution** (`convert.rs:142-154`): read + `spec.precursor()` and clone its `isolation_window`. If a real window is + present (not both bounds zero), use `(target, lower_bound, upper_bound)` + (`convert.rs:145-147`). Otherwise, the AIF / all-ion path synthesizes a + full-range window `(target=0.0, lower=0.0, upper=1.0e6)` (`convert.rs:148-149`). + This `_` arm fires both when the quadrupole reported a zero-width window + (AIF/all-ion acquisition) and when there is no precursor at all, so any MS2 + with no usable window is treated as covering the entire m/z range. The + downstream `IsolationWindow::covers` then returns true for every fragment + (`types.rs:26`). The precursor m/z and charge come from the first precursor ion + or are `None` (`convert.rs:151-154`). + +7. **Distinct isolation windows** (`convert.rs:187-202`): a `HashMap` keyed by the + raw bit patterns of `(window_lower, window_upper)` via `f64::to_bits` + (`convert.rs:194`) assigns a first-seen `window_id`. The key uses bits, not + float equality, so identical window bounds always collapse to the same id and + the id order is the acquisition order of first appearance. Each MS2 row records + its `window_id` in `win_id_col`. + +8. **Write the four tables** (`convert.rs:177-237`) with `write_table`, then + `write_reports` (`convert.rs:272-294`) emits one `ArtifactReport` per file: + logical/schema name and version from `schema.rs`, `stage = "convert"`, row + count, a blake3 content hash of the written file, the resolved params + (`mzml`, `max_spectra`, `top_peaks_ms2`, `top_peaks_ms1`), and elapsed ms. The + function returns `ConvertOutputs` with the four paths for chaining + (`convert.rs:264-269`). + +Note the artifacts are written in acquisition order. RT-sorting is deferred to the +read side: `spectra::load_ms1` / `load_ms2` sort by `rt_seconds` after loading +(`spectra.rs:92`, `spectra.rs:152`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `centroid` | `convert.rs:19` | Local-maxima centroiding with parabolic m/z refinement and a relative noise floor. | +| `peaks_of` | `convert.rs:56` | Profile-detect + centroid, drop non-positive intensity, top-N cap, m/z sort, cast m/z to f32. | +| `ConvertParams` | `convert.rs:86` | Inputs: `mzml`, `out_dir`, `max_spectra`, `top_peaks_ms2`, `top_peaks_ms1`, `config_hash`. | +| `ConvertOutputs` | `convert.rs:96` | Returned paths: `ms1`, `ms2`, `isolation_windows`, `ms2_to_ms1`. | +| `run` | `convert.rs:103` | The stage entry point; single pass over mzML, then four table writes. | +| `write_reports` | `convert.rs:272` | Writes the per-artifact `report.json` sidecars. | +| `artifact::SPECTRA_MS1/_MS2/ISOLATION_WINDOWS/MS2_TO_MS1` | `schema.rs:7-10` | Frozen `(name, version)` schema identifiers, all v1. | +| `Col` / `write_table` | `table.rs:23` / `table.rs:151` | Typed columns and the SNAPPY Parquet writer; validates equal lengths and rejects duplicate names. | +| `ArtifactReport` | `report.rs:11` | The report struct written next to each artifact. | +| `load_ms2` / `load_ms1` | `spectra.rs:20` / `spectra.rs:97` | Read-back into `Ms2Scan` / `Ms1Scan`, RT-sorted, m/z widened to f64. | + +## Configuration + +convert reads no `Config` fields. It is driven entirely by CLI arguments +(`main.rs:18-37`). The stage function signature does not take a `Config`; the +`config_hash` it receives is only recorded for provenance. + +| CLI flag | Default | Effect | +|---|---|---| +| `--mzml` | (required) | Input mzML path. | +| `--out-dir` | (required) | Output directory for the four artifacts. | +| `--max-spectra` | `0` (all) | Read at most N spectra, for fast iteration. Counts all MS levels. | +| `--top-peaks-ms2` | `0` (uncapped) | Keep at most N most-intense MS2 peaks per scan. Irreversible conversion-time cap. | +| `--top-peaks-ms1` | `0` (uncapped) | Keep at most N most-intense MS1 peaks per scan. Irreversible. | + +Defaults are asserted by `conversion_caps_default_to_uncapped` (`main.rs:687`) and +the explicit-cap case by `explicit_conversion_cap_is_preserved` (`main.rs:727`). +The `run` orchestrator exposes `--max-spectra` and `--top-peaks-ms2` but not +`--top-peaks-ms1`; it hardcodes `top_peaks_ms1: 0` when calling convert +(`run.rs:157`). The MS2 cap is documented as "irreversible" because it discards +peaks before they reach extraction, features, and quantification; for a seed-only +peak limit use `search_seed.top_n_peaks` instead (`main.rs:27-31`, +`config.rs:337`). + +Provenance handling of the caps: because the caps change the spectra output but +are not part of the `Config`, `main.rs:389-392` folds them into the blake3 +`config_hash` with a unit-separator (`\u{1f}`) so two different caps do not +collapse to an identical hash. The same values are recorded in the convert report +`params` (`convert.rs:249-254`). + +The centroid noise floor (`1e-4` relative, `convert.rs:25`) and the full-range AIF +window value (`1.0e6`, `convert.rs:149`) are hardcoded constants, not config +fields. There is no `ScanWindowMode` or centroiding strategy knob for this stage; +the project config was recently pruned of dead fields, and convert exposes no +strategy enum. Any change to centroiding or window synthesis is a code change +here, not a config toggle. + +## Invariants, determinism, gotchas + +- **Determinism.** The output is a deterministic function of the input file and + the CLI caps. `scan_index` is assigned in reader order; `window_id` is assigned + in first-appearance order via a bit-keyed map, not float equality. The + intensity-descending truncation sort (`convert.rs:77`) uses Rust's stable + `sort_by`, so ties preserve the incoming (m/z-ascending) order; the final + m/z-ascending sort (`convert.rs:80`) fixes output order regardless. +- **`scan_index` is run-global, not per-level.** MS3 and other unhandled levels + still increment the counter (`convert.rs:128-129`), so the MS1 and MS2 tables + have unique but non-contiguous indices. Do not assume contiguity; use + `ms2_to_ms1.parquet` to relate MS2 to its parent MS1. +- **AIF and no-precursor collapse to the same full-range window.** The `_` arm at + `convert.rs:149` handles both a zero-width reported window (AIF/all-ion) and a + missing precursor. If a future non-AIF format reports a genuinely absent window, + it will be silently treated as full-range. `window_target = 0.0` is the marker + for a synthesized window. +- **Observed m/z is f32 on disk.** `convert.rs:81` casts to f32; `spectra.rs` + widens back to f64. This is intentional (storage) and fine at DIA ppm + tolerances, but do not round-trip observed m/z through convert expecting f64 + precision. +- **Intensity is the raw apex sample.** Parabolic refinement adjusts m/z only; the + reported intensity is `y1` (`convert.rs:45`), not an interpolated peak height. +- **Centroid fallback can leak profile samples.** If no local maximum clears the + floor, `centroid` returns the original profile arrays (`convert.rs:47-51`). + Rare, but a downstream stage could then see profile-shaped data for that scan. +- **List columns are nullable in the Arrow schema but never null in practice.** + An empty scan is written as a non-null empty list; the read side treats null and + empty identically (`spectra.rs:52`, `table.rs:404`). +- **`partial_cmp().unwrap()` on sorts** (`convert.rs:77`, `:80`, + `spectra.rs` sorts) would panic on NaN. Convert filters intensity `<= 0` before + sorting and does not sort on m/z NaN in practice, so this is safe for real mzML + but is a latent trap if malformed data ever reaches it. +- **Test coverage.** Only the two CLI-parsing tests above exercise this area. The + centroiding math, window synthesis, and artifact writing have no stage-level + unit test (see CLAUDE.md "test gaps"). MS1 extraction and mass-calibration paths + that depend on convert output are exercised only in full runs. + +## How to extend / modify + +- **Add a vendor format** (Thermo `.raw`, Bruker `.d`/TDF): this stage is the only + place to touch. Either extend `mzdata` features or add a reader that yields the + same per-spectrum interface, and keep the four output schemas byte-compatible so + no downstream stage changes. Convert must stay the sole vendor-format touch + point. +- **Ion mobility / 4D (diaPASEF).** The artifact schemas here are 3D. Adding IM + means new nullable columns on `spectra_ms2` (and the isolation-window IM bounds + already modeled as `Option` in `types.rs:21`), plus populating `Peak.ion_mobility` + on the read side. Bump the affected schema versions in `schema.rs` when columns + change, since the version guards downstream model/schema matching. +- **Change centroiding.** Edit `centroid` (`convert.rs:19`). If the choice should + be user-selectable, add a config field and strategy enum in `mumdia-core` + (per the project convention that every algorithmic choice is a typed config + field) rather than a second hardcoded branch, and thread it through + `ConvertParams`. Remember the noise floor and parabolic step are currently + hardcoded. +- **Change the AIF window sentinel.** The full-range bound `1.0e6` + (`convert.rs:149`) and `window_target = 0.0` marker are relied on by extraction's + `IsolationWindow::covers`. Changing either requires auditing the extract stage. +- **Add an artifact column.** Add the column to the relevant `write_table` call, + add a matching getter/reader in `spectra.rs`, and bump the schema version in + `schema.rs`. `write_table` rejects duplicate names and mismatched lengths, so + every new column vector must match the row count. +- **Preserve provenance semantics.** Any new conversion-time parameter that + changes the output but is not part of `Config` must be folded into the + `config_hash` key in `main.rs` (as the caps are), or two different settings will + produce artifacts with an identical hash. diff --git a/docs/05_digest_peptidoforms.md b/docs/05_digest_peptidoforms.md new file mode 100644 index 0000000..053e460 --- /dev/null +++ b/docs/05_digest_peptidoforms.md @@ -0,0 +1,326 @@ +# digest (Stage A) and peptidoforms (Stage A2) + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +These two stages build the run-independent, experiment-wide peptide search space +from a FASTA file, before any spectra are touched. They run once per experiment +and their outputs are reused across all runs. + +- **digest** (Stage A, PLAN.md) performs a fully-tryptic in-silico digest of the + input proteins, deduplicates the resulting peptides by stripped sequence, and + mints a paired target-decoy null (reverse or seeded scramble). Output is one + `peptides.parquet` table of stripped target and decoy sequences with a + `target`/`decoy` label. +- **peptidoforms** (Stage A2) expands each stripped peptide into concrete + peptidoforms by enumerating fixed and variable modifications and precursor + charge states, emitting each as a ProForma-lite string with UniMod modification + names. Output is one `peptidoforms.parquet` table. + +Neither stage reads spectra or config-run parameters (tolerances, RT, etc.); they +depend only on the FASTA and the `digest` / `peptidoforms` config sections. In the +library-input path (`--lib-precursors`/`--lib-fragments`) both stages are skipped +entirely; the imported library already carries sequences, modifications, charges, +and decoys (see `run.rs` library-input branch, and CLAUDE.md "DIA-NN library +recipe"). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/digest.rs` | Stage A: FASTA parse, tryptic digest, dedup, decoy generation, `peptides.parquet` writer | +| `rust/mumdia/crates/mumdia/src/stages/peptidoforms.rs` | Stage A2: fixed/variable mod + charge enumeration, ProForma-lite emission, `peptidoforms.parquet` writer | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `DigestConfig`, `DecoyConfig`, `PeptidoformsConfig`, `ResidueMod`, `Enzyme`, `DecoyStrategy`, `UnknownModPolicy`, and `Config::validate` | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `is_standard_residue` / `residue_mass` (the 20-residue allowlist that gates the digest) | +| `rust/mumdia/crates/mumdia-core/src/mass.rs` | `unimod_mass` (the UniMod name allowlist that validates modification names) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::PEPTIDES` and `artifact::PEPTIDOFORMS` logical names + schema versions | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI wiring: `Cmd::Digest` (main.rs:402), `Cmd::Peptidoforms` (main.rs:413) | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | Orchestrator wiring in the FASTA-build branch (run.rs:116, run.rs:126) | + +## Inputs and outputs + +**digest** +- Consumes: a FASTA file path (`DigestParams.fasta`, digest.rs:140). Parsed by + `read_fasta` (digest.rs:19) into `(accession, sequence)` pairs. FASTA parsing is + minimal: `>` starts a record, the accession is the first whitespace-delimited + token after `>`, and sequence lines are concatenated and trimmed. There is no + handling of `*` stop characters or lowercase residues beyond the residue + allowlist check applied later. +- Produces: `peptides.parquet` (logical name `peptides`, schema version 1, from + `artifact::PEPTIDES`, schema.rs:11) plus a sibling `.report.json` + (`ArtifactReport`, digest.rs:232). Report `stats` carries `n_targets` and + `n_decoys`; `params` records enzyme, missed cleavages, length bounds, decoy + strategy, and `rng_seed` (digest.rs:239). + +`peptides.parquet` column schema (written at digest.rs:214-226): + +| Column | Type | Meaning | +|---|---|---| +| `id` | u32 | Monotonic row id, assigned interleaved (target then its decoy) | +| `peptide` | str | Stripped sequence (target sequence, or the rewritten decoy sequence) | +| `protein` | str | `;`-joined accessions for targets; `DECOY_` + the same join for decoys | +| `start` | i32 | 0-based start offset of the peptide in the protein of first occurrence | +| `end` | i32 | 0-based end offset (exclusive) of that first occurrence | +| `label` | str | Literal `"target"` or `"decoy"` | +| `target_id` | i32 | `-1` for a target; for a decoy, the `id` of its paired target | +| `decoy_strategy` | str | Lowercased strategy name (e.g. `reverse`, `scramble`) | + +**peptidoforms** +- Consumes: `peptides.parquet` (`PeptidoformsParams.peptides`, peptidoforms.rs:62), + read via `Table::read` (peptidoforms.rs:70). It reads columns `id`, `peptide`, + `protein`, `label`, `target_id`. +- Produces: `peptidoforms.parquet` (logical name `peptidoforms`, schema version 1, + from `artifact::PEPTIDOFORMS`, schema.rs:12) plus `.report.json`. Report + `params` record fixed/variable mods, `max_variable_mods`, and the charge range + (peptidoforms.rs:157); `stats` is empty. + +`peptidoforms.parquet` column schema (written at peptidoforms.rs:135-147): + +| Column | Type | Meaning | +|---|---|---| +| `id` | u32 | Monotonic peptidoform row id (fresh counter across all output rows) | +| `peptide_id` | u32 | The digest `id` of the row this peptidoform came from | +| `base_peptide_id` | u32 | The digest `id` of the underlying **target** peptide (for decoys, the paired target id; for targets, the own id) | +| `peptide` | str | Stripped sequence, copied through unchanged | +| `peptidoform` | str | ProForma-lite string with UniMod names in brackets, e.g. `PEPC[Carbamidomethyl]M[Oxidation]K` | +| `charge` | i32 | Precursor charge (one row per charge in `[charge_min, charge_max]`) | +| `label` | str | `"target"` / `"decoy"`, copied from the digest row | +| `protein` | str | Copied from the digest row | + +## How it works + +### digest (digest.rs:147 `run`) + +1. **Parse FASTA** with `read_fasta` (digest.rs:19). +2. **Per-protein cleavage** in `cleavage_sites` (digest.rs:43). Sites is seeded + with `0`, then for each residue that is `K` or `R` a cut index `i+1` is pushed; + Trypsin/P (`Enzyme::TrypsinP`, the default) always cuts, classic `Enzyme::Trypsin` + suppresses the cut when the next residue is `P` (`before_p`, digest.rs:48). The + sequence length is appended as a final site if not already present + (digest.rs:58). Sites are strictly increasing, so peptides are contiguous + half-open `[start, end)` spans. +3. **Peptide enumeration** in `digest_protein` (digest.rs:66). For every ordered + pair of sites `(i, j)` with `i < j`, the number of missed cleavages is + `j - i - 1`; the inner loop `break`s once that exceeds `cfg.missed_cleavages` + (digest.rs:72), so at most `missed_cleavages + 1` fragments are joined. Length + bounds `min_len`/`max_len` filter on residue count (digest.rs:77). A peptide + is dropped entirely if any residue is not one of the 20 standard amino acids + (`is_standard_residue`, digest.rs:81 -> constants.rs:54), which is how `X`, `B`, + `Z`, `U`, `*`, and lowercase get excluded. +4. **Dedup by stripped sequence** (digest.rs:153-172). Three structures: a + `HashMap>` mapping peptide to accessions, a `HashMap` from + peptide to its first `(start, end)`, and an insertion-order `Vec` + named `order`. On a repeat occurrence only the accession list is extended (and + only if the accession is not already present, digest.rs:163); the first + occurrence records position and pushes the peptide into `order`. Iteration for + output is over `order`, so **insertion order is the emission order** and is + deterministic (a protein's peptides in sequence order, proteins in FASTA + order). The borrow-then-clone pattern (digest.rs:162) is a deliberate + allocation optimization, not a semantic detail. +5. **Row assembly and decoy minting** (digest.rs:178-210). A single monotonic + `next_id` counter assigns ids. For each target peptide in `order`: emit the + target row (`label = "target"`, `target_id = -1`, digest.rs:189-191), then, if + the strategy is neither `None` nor `DiannShift`, call `make_decoy` + (digest.rs:197) and, on `Some`, emit the decoy row immediately after with a + fresh id, `label = "decoy"`, `target_id = tid` (the target's id), and the + `DECOY_`-prefixed protein string. Ids therefore interleave target/decoy in + pairs. +6. **Write** the eight columns with `write_table` (digest.rs:214) and emit the + `ArtifactReport` (digest.rs:232). + +**Decoy generation** (`make_decoy`, digest.rs:92). Peptides shorter than 3 +residues return `None` (no decoy). Both realized strategies keep the C-terminal +residue fixed and rewrite the interior `b[..n-1]`: +- `DecoyStrategy::Reverse` (digest.rs:99): reverse the first `n-1` residues, then + re-append the original last residue. Reversing all-but-last keeps the enzyme's + C-terminal `K`/`R` in place while moving the N-terminus. Test + `reverse_decoy_keeps_cterm` (digest.rs:280) pins `PEPTIDER -> EDITPEPR`. +- `DecoyStrategy::Scramble` (digest.rs:105): a deterministic Fisher-Yates shuffle + of the first `n-1` residues. The PRNG state is seeded per peptide as + `rng_seed ^ fnv1a(pep)` (digest.rs:108) and advanced with `splitmix64` + (digest.rs:122). The loop runs `i` from `len-2` down to `1` and swaps index `i` + with a uniform `j` in `0..=i` (digest.rs:109-113), so every interior position + including index 0 (the N-terminal residue) can move while the last residue is + re-appended untouched. Test `scramble_is_deterministic` (digest.rs:287) pins + reproducibility and the fixed C-terminus. +- `DecoyStrategy::DiannShift` and `DecoyStrategy::None` return `None` + (digest.rs:117-118); `DiannShift` is unrealized here by design (the comment + notes it would be a predict-frag fragment-shift, PLAN.md Section 11) and is + additionally rejected by `Config::validate` (config.rs:1058) because it would + yield zero decoys and an invalid FDR. + +### peptidoforms (peptidoforms.rs:68 `run`) + +1. **Read** the digest table (peptidoforms.rs:70-75). +2. **Validate modification names up front** (peptidoforms.rs:78-82): every fixed + and variable mod name is looked up in `unimod_mass` (mass.rs:13); an unknown + name is a hard `anyhow::bail!`. This runs once before any row is processed, so + a typo fails fast. +3. **Per digest row** (peptidoforms.rs:89): compute `base_peptide_id` as + `target_id` when it is `>= 0` (the row is a decoy) else the row's own `id` + (peptidoforms.rs:91-95). +4. **Fixed mods** (peptidoforms.rs:98-105): for each residue index, if any + `fixed_mods` entry's `residue` char matches, record `(index, name)`. Fixed mods + apply to every matching residue and are always present. +5. **Variable-mod candidate positions** (peptidoforms.rs:107-114): the same + residue-char scan collects `(index, name)` candidates for `variable_mods`. +6. **Combination enumeration** (`variable_combos`, peptidoforms.rs:33): returns the + empty subset plus every size-`k` subset of candidate positions for + `k = 1..=max_variable_mods.min(n)`, using an in-place combinatorial index + advance (peptidoforms.rs:40-57). The count is `1 + sum_{k=1..min(max,n)} C(n,k)`; + test `combos_bounded` (peptidoforms.rs:184) pins `n=3, max=1 -> 4` and + `max=2 -> 7`. +7. **Emit** (peptidoforms.rs:116-132): for each combo, merge fixed mods and the + combo, sort by position (peptidoforms.rs:119), build the ProForma-lite string + with `proforma` (peptidoforms.rs:18), and emit one output row per charge in + `[charge_min, charge_max]` (peptidoforms.rs:121). Each row gets a fresh `id`, + the source `peptide_id`, the computed `base_peptide_id`, and the copied + `label`/`protein`. + +`proforma` (peptidoforms.rs:18) walks the stripped sequence and, at each residue +index, appends `[]` for the **first** mod found at that position +(`mods.iter().find`, peptidoforms.rs:22). This is where a second modification at +the same position is silently dropped (see gotchas). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `digest::run` | digest.rs:147 | Stage A entry point; parse, digest, dedup, mint decoys, write `peptides.parquet` | +| `read_fasta` | digest.rs:19 | Minimal FASTA parser to `(accession, sequence)` pairs | +| `cleavage_sites` | digest.rs:43 | Trypsin/P vs Trypsin cut-site indices for one sequence | +| `digest_protein` | digest.rs:66 | Enumerate length-bounded, missed-cleavage-bounded, standard-residue-only peptides | +| `make_decoy` | digest.rs:92 | Reverse or seeded-scramble decoy of the interior, C-term fixed; `None` if `len < 3` or strategy `None`/`DiannShift` | +| `splitmix64` | digest.rs:122 | Deterministic PRNG step for the scramble shuffle | +| `fnv1a` | digest.rs:130 | Per-peptide hash mixed into the scramble seed | +| `DigestParams` | digest.rs:139 | `fasta`, `out`, `cfg: &DigestConfig`, `rng_seed`, `config_hash` | +| `peptidoforms::run` | peptidoforms.rs:68 | Stage A2 entry point; validate mods, enumerate mods+charges, write `peptidoforms.parquet` | +| `proforma` | peptidoforms.rs:18 | Build ProForma-lite string from stripped peptide + `(position, name)` mods | +| `variable_combos` | peptidoforms.rs:33 | Enumerate variable-mod position subsets up to `max_var` | +| `PeptidoformsParams` | peptidoforms.rs:61 | `peptides`, `out`, `cfg: &PeptidoformsConfig`, `config_hash` | +| `unimod_mass` | mass.rs:13 | Name -> monoisotopic delta allowlist (the set of mods that validate) | +| `is_standard_residue` | constants.rs:54 | The 20-residue allowlist gating digest output | + +## Configuration + +`digest` reads `DigestConfig` (config.rs:219). `peptidoforms` reads +`PeptidoformsConfig` (config.rs:240). Both structs use `#[serde(default, +deny_unknown_fields)]`, so an unknown key is a load error and any omitted field +takes the default below. `rng_seed` (config.rs:1011, default `0`) is a top-level +`Config` field, passed to digest as `DigestParams.rng_seed` (main.rs:409, +run.rs:120). + +| Field | Section | Default | Effect | +|---|---|---|---| +| `enzyme` | digest | `TrypsinP` | `trypsin_p` cuts after K/R including before P; `trypsin` suppresses the cut before P | +| `missed_cleavages` | digest | `2` | Max missed cleavages; joins at most `missed_cleavages + 1` tryptic fragments | +| `min_len` | digest | `5` | Minimum peptide length (residues) | +| `max_len` | digest | `50` | Maximum peptide length (residues) | +| `decoy.strategy` | digest | `reverse` | `reverse` / `scramble` realized; `diann_shift` and `none` produce no decoy (`diann_shift` rejected by `validate`) | +| `fixed_mods` | peptidoforms | `[{C: Carbamidomethyl}]` | Applied to every matching residue | +| `variable_mods` | peptidoforms | `[{M: Oxidation}]` | Enumerated as optional subsets | +| `max_variable_mods` | peptidoforms | `1` | Max simultaneous variable mods per peptidoform | +| `charge_min` | peptidoforms | `2` | Lowest precursor charge emitted | +| `charge_max` | peptidoforms | `3` | Highest precursor charge emitted | +| `unknown_modification` | peptidoforms | `error` | Declared policy `error`/`skip`; see gotchas (only `error` behavior is realized) | + +`rng_seed` is `0` by default (config.rs:1028) and only affects the `scramble` +decoy strategy; `reverse` is independent of the seed. + +Config was recently pruned of dead fields in this area. `ResidueMod` +(config.rs:271) is only `{ residue: char, name: String }`; there is no +position/terminal specifier field. The doc comment on `residue` mentions `*` for +"any / terminal handled separately in MVP", but no such handling exists in the +code (see gotchas). There is no `DecoySource` field on `DigestConfig` (the +"dead/unwired config surface" list in CLAUDE.md refers to types elsewhere, not +here). `DecoyConfig` (config.rs:208) carries only `strategy`. + +## Invariants, determinism, gotchas + +- **Determinism.** Emission order is FASTA order then in-protein sequence order, + preserved by the `order` vector (digest.rs:169), not by HashMap iteration. Ids + are a single monotonic counter. Scramble is fully deterministic for a fixed + `rng_seed` because the PRNG is seeded per peptide from the sequence + (digest.rs:108). No floats are summed in either stage, so the numeric-order + determinism concerns from other stages do not apply. +- **Target/decoy pairing.** A decoy's `target_id` points at its target's `id` and + the pair is emitted adjacently; targets carry `target_id = -1`. Downstream FDR + keeps the label in competition keys (see CLAUDE.md FDR / compete), so this + pairing must stay intact. `peptidoforms` propagates the link via + `base_peptide_id` (peptidoforms.rs:91). +- **Residue allowlist.** Peptides containing any non-standard residue are dropped + wholesale at digest time (digest.rs:81); they never reach peptidoforms. This is + the only place ambiguity codes (`X`/`B`/`Z`/`U`) are handled. +- **No decoy-equals-target guard.** `make_decoy` does not check that the decoy + differs from its target, and there is no cross-target decoy dedup. Short or + palindromic sequences can yield a decoy identical to some target; this is a + known limitation (CLAUDE.md "Correctness"). Peptides with `len < 3` silently get + no decoy (digest.rs:95); with the default `min_len = 5` this only bites if a + caller lowers `min_len`. +- **No terminal modifications.** `ResidueMod` matches on an exact residue char + (peptidoforms.rs:100, peptidoforms.rs:109). There is no N-term/C-term or + wildcard `*` matching, so protein/peptide N-terminal acetylation and similar + terminal mods cannot be enumerated. The `mass.rs` parser does model terminal mod + deltas, but Stage A2 never produces terminal-mod ProForma strings. +- **Second mod at the same position is dropped.** `proforma` (peptidoforms.rs:22) + keeps only the first `(position, name)` at each index. If a residue matches both + a fixed mod and a variable mod (or two variable mod specs), the merged `mods` + list has two entries at that index and the second is silently lost from the + emitted string. This is the documented "second mod at same position dropped" + limit (CLAUDE.md, peptidoforms status "partial"). +- **`unknown_modification` policy is declared but not wired.** The config field + exists with values `error`/`skip` (config.rs:247, `UnknownModPolicy`, + config.rs:280), but `peptidoforms::run` unconditionally `bail!`s on an unknown + name (peptidoforms.rs:78-82) and never reads `cfg.unknown_modification`. `skip` + behaves identically to `error` today. Treat `skip` as unimplemented. +- **`config_hash` param is unused.** Both `DigestParams.config_hash` + (digest.rs:144) and `PeptidoformsParams.config_hash` (peptidoforms.rs:65) are + threaded from the CLI/orchestrator but never read inside either `run`. The + content hash written to the report comes from `blake3_file` over the output + Parquet (digest.rs:238, peptidoforms.rs:156), not from `config_hash`. +- **`decoy_strategy` string is derived by Debug-formatting the enum** + (`format!("{:?}", ...).to_lowercase()`, digest.rs:191). It is a display string, + not a parseable round-trip; changing enum variant names changes the column + value. +- **Charge range is inclusive and unvalidated.** `charge_min..=charge_max` + (peptidoforms.rs:121) is emitted as-is; a `charge_min > charge_max` config + yields zero peptidoform rows silently rather than an error. +- **Library-input mode bypasses both stages.** When a prebuilt library is passed, + `run` takes the library-input branch and never calls `digest`/`peptidoforms` + (run.rs, and the FASTA branch at run.rs:111); do not assume `peptides.parquet` / + `peptidoforms.parquet` exist for every run. + +## How to extend / modify + +- **New enzyme.** Add a variant to `Enzyme` (config.rs:54) and a match arm in + `cleavage_sites` (digest.rs:49). Keep the site list strictly increasing and + terminated at `seq.len()`; the rest of `digest_protein` is enzyme-agnostic. + Semi-tryptic search would require changing how spans are enumerated in + `digest_protein` (digest.rs:66), not just the cut list. +- **New decoy scheme.** Add a `DecoyStrategy` variant (config.rs:17) and a match + arm in `make_decoy` (digest.rs:98). Preserve the C-terminal residue and + determinism (seed the PRNG from the sequence as scramble does, digest.rs:108). + If the scheme can produce a decoy equal to its target, add the guard the current + code lacks. Realizing `DiannShift` means removing the `validate` rejection + (config.rs:1058) and implementing a fragment-m/z shift at predict-frag, not a + sequence rewrite here. +- **New modification.** Add the UniMod name and monoisotopic delta to + `unimod_mass` (mass.rs:13); it is the single allowlist both the Stage A2 + validation (peptidoforms.rs:79) and the mass model consult. Then reference it by + name in `fixed_mods`/`variable_mods`. +- **Support a second mod per position or terminal mods.** Change `proforma` + (peptidoforms.rs:18) to emit all mods at a position (e.g. concatenate multiple + `[...]` groups) instead of `find`-ing the first, and extend `ResidueMod` + (config.rs:271) plus the residue-matching loops (peptidoforms.rs:98-114) with a + position/terminal specifier. The mass model in `mass.rs` already carries + `n_term_mod` fields to build on. +- **Schema changes.** Adding or reordering `peptides.parquet` / + `peptidoforms.parquet` columns is a schema change; bump the version in + `artifact::PEPTIDES` / `artifact::PEPTIDOFORMS` (schema.rs:11-12) and update the + downstream readers (`Table::read` callers in predict-frag and later stages). +- **Testing.** Existing unit tests live at the bottom of each file + (digest.rs:256, peptidoforms.rs:173) and cover cleavage, decoy C-term/ + determinism, ProForma placement, and combo bounds. There is no stage-level test + that round-trips through Parquet or exercises `run` end to end (CLAUDE.md "test + gaps"); add one when changing the output schema. diff --git a/docs/06_predict_frag_index_matchers.md b/docs/06_predict_frag_index_matchers.md new file mode 100644 index 0000000..b79fedf --- /dev/null +++ b/docs/06_predict_frag_index_matchers.md @@ -0,0 +1,376 @@ +# predict-frag, the library, the inverted fragment index, matchers + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This subsystem produces the run-independent spectral library and the data +structures that make fragment matching cheap. It has three parts: + +1. **predict-frag** (Stage C, `stages/predict_frag.rs`): turn concrete + peptidoforms into a library of precursor and b/y fragment m/z with predicted + fragment intensities and predicted iRT, keep the top-N fragments per + candidate, and assign each candidate a `candidate_id` equal to its rank in + precursor-m/z order. The output is two Parquet artifacts that are constant for + the whole run (they do not depend on the mzML being searched). + +2. **the library loader** (`index.rs`, `Library::load`): read those two Parquet + artifacts back into a Structure-of-Arrays in-memory model, group fragments by + candidate, build the bucketed peak-major inverted fragment index, and enforce + the preconditions the matchers rely on. + +3. **the matchers** (`matchers/`): given an observed peak m/z, a ppm tolerance, + and an isolation-window candidate range, return the predicted fragments that + fall within tolerance. Two backends exist: the default log-bin CSR `fragindex` + (`matchers/fragindex.rs`, spec in `fragindex_spec.md`) and the fallback + bucketed `Library::page_search`. A naive band-join (`matchers/naive.rs`) is the + correctness oracle, not a production path. + +The predictor traits (`predict.rs`) and the sidecar file contract (`sidecar.rs`) +sit under predict-frag: they are the boundary between the native (zero external +dependency) intensity/RT models and the optional Python predictors MS2PIP and +DeepLC. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/predict_frag.rs` | Stage C: build the library (parse, fragments, intensity, iRT, top-N, sort, write Parquet) | +| `rust/mumdia/crates/mumdia/src/predict.rs` | `RtPredictor`/`FragmentPredictor` traits + `NativeRt`/`NativeFrag` fallbacks | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | Python sidecar clients (MS2PIP, DeepLC, DeepLC fine-tune, MBR) over the file contract | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library` (SoA model + bucketed inverted index), `load()` preconditions, `page_search`, `candidate_range`, `deconvolve` | +| `rust/mumdia/crates/mumdia/src/matchers/mod.rs` | matcher module tree; `MatcherKind` selects the backend | +| `rust/mumdia/crates/mumdia/src/matchers/binning.rs` | `LogBins`: log-space bin geometry (fragindex_spec Section 2.2) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` CSR index, `SeedScratch` epoch-stamped accumulator, `probe_peak`, equivalence-gate scorer | +| `rust/mumdia/crates/mumdia/src/matchers/naive.rs` | band-join reference for the equivalence gate | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `within_ppm` (min-relative predicate), `ppm_bounds` (query-relative), `PROTON` | +| `fragindex_spec.md` | language-agnostic algorithm spec the fragindex matcher implements | + +## Inputs and outputs + +### Consumed + +**peptidoforms** (Stage A2 output), read at `predict_frag.rs:53-58`: + +| column | type | note | +|---|---|---| +| `id` | u32 | peptidoform id | +| `base_peptide_id` | u32 | stripped-peptide id | +| `peptidoform` | str | ProForma-lite string with UniMod names | +| `charge` | i32 | precursor charge | +| `label` | str | `"target"` or `"decoy"` | +| `protein` | str | protein accession | + +### Produced (two Parquet artifacts + a `report.json` each) + +**fragment_library_precursors** (schema `("fragment_library_precursors", 1)`, +`schema.rs:13`), written at `predict_frag.rs:186-200`: + +| column | type | +|---|---| +| `candidate_id` | u32 (dense 0..N in precursor-m/z order) | +| `peptidoform_id` | u32 | +| `base_peptide_id` | u32 | +| `peptidoform` | str | +| `charge` | i32 | +| `precursor_mz` | f64 | +| `predicted_irt` | f32 | +| `label` | str | +| `protein` | str | +| `n_fragments` | i32 (kept fragment count after top-N) | + +**fragment_library_fragments** (schema `("fragment_library_fragments", 1)`, +`schema.rs:14`), written at `predict_frag.rs:201-212`: + +| column | type | +|---|---| +| `candidate_id` | u32 (foreign key into precursors) | +| `mz` | f64 (fragment m/z at its own `frag_charge`) | +| `predicted_intensity` | f32 | +| `name` | str (e.g. `b2`, `y3`) | +| `ion_type` | str (`b` or `y`) | +| `ordinal` | i32 (residue-count ordinal of the fragment) | +| `frag_charge` | i32 (1, or 2 when enabled) | + +Each artifact also gets an `ArtifactReport` (`predict_frag.rs:219-238`) recording +row count, blake3 content hash, params (`top_n`, `ms2pip_model`, `rt_predictor`, +`fragment_predictor`), and `model_identity` (the concatenated RT + fragment model +ids, `predict_frag.rs:114`). + +The library-input path (`--lib-precursors`/`--lib-fragments`) skips Stage C and +feeds externally built Parquet with these exact schemas directly into +`Library::load`. + +## How it works + +### predict-frag (`stages/predict_frag.rs:50`) + +**Phase A: parse and enumerate fragments** (`predict_frag.rs:65-110`). Each +peptidoform row is parsed with `parse_peptidoform` and fragmented independently, +so rows are mapped in parallel with rayon (`into_par_iter`, line 70). The closure +returns `RowOut::Raw`, `RowOut::ParseErr`, or `RowOut::Empty`. Fragment charges +are chosen per row: charge 1 always, charge 2 added when the precursor charge is +at least `charge2_from_precursor_charge` (`predict_frag.rs:78-81`). `collect` +preserves row order, and the sequential fold at `predict_frag.rs:103-109` +reproduces the exact serial `raws` order and `n_parse_err` count. The parsed form +is stored on the `Raw` struct so intensity/iRT assignment reuses it instead of +re-parsing (`predict_frag.rs:45-47`). + +**iRT assignment** (`assign_rt`, `predict_frag.rs:245`). Native path calls +`NativeRt::predict_irt` per candidate. DeepLC path deduplicates by peptidoform +string (RT is charge-independent, `predict_frag.rs:262-271`), runs the sidecar +once over the unique set, then maps results back. Peptidoforms DeepLC returns no +prediction for are anchored at `irt = 0.0` and counted; if any are missing a +`tracing::warn!` fires (`predict_frag.rs:284-289`). This is the DeepLC-miss iRT +warning: it makes the silent "unmatched peptidoform gets iRT 0.0" failure visible, +because an iRT-0 anchor collapses the RT window onto the gradient origin and +misplaces the candidate at extraction. + +**intensity assignment** (`assign_intensities`, `predict_frag.rs:296`). Native +path calls `NativeFrag::predict_intensities`. MS2PIP path runs the sidecar over +all candidates; an empty whole-map result is a hard error (`bail`, +`predict_frag.rs:316-318`). Per candidate, MS2PIP supplies charge-1 b/y +intensities keyed by `(ion_byte, ordinal)`; charge-2 fragments (which MS2PIP does +not emit) fall back to the native model (`predict_frag.rs:329-337`). MS2PIP +values (TIC-fraction scale, roughly 0.02-0.3) and the native charge-2 fallback +(max-normalized, roughly 0.19-0.5) live on different scales, so each charge group +is max-normalized to its own peak before they compete for top-N slots +(`predict_frag.rs:344-359`); otherwise the larger-scale group would always win the +truncation. A candidate MS2PIP returns nothing for falls back wholesale to native. + +**top-N and candidate_id** (`predict_frag.rs:119-137`). For each candidate, +fragments are ranked by predicted intensity descending, truncated to +`top_n_fragments`, then re-sorted ascending to restore stored order; an in-place +forward-swap gather compacts the kept fragments without reallocating. Candidates +left with zero fragments are dropped (`retain`, line 134). Then `raws` is sorted +by `precursor_mz` with a stable `sort_by` (`predict_frag.rs:137`), and +`candidate_id` is assigned by `enumerate` over that order +(`predict_frag.rs:163-164`). This is the single most load-bearing invariant of +the whole subsystem: `candidate_id` is the dense precursor-m/z rank, which is what +lets the index recover the isolation-window candidate slice by binary search and +lets `candidate_id` directly index the dense accumulator. + +### Library load and the bucketed inverted index (`index.rs:55`) + +`Library::load` reads both Parquet artifacts and rebuilds the per-candidate +fragment arrays grouped and contiguous (`index.rs:100-128`), so `cand_frags(cid)` +is a slice (`index.rs:208-213`). It then builds the bucketed inverted index +(`index.rs:158-187`): + +1. Emit one `(frag_mz as f32, candidate_id, frag_int)` entry per fragment. +2. Globally sort entries by fragment m/z with `par_sort_by` (parallel stable + sort, identical result to the serial stable sort, `index.rs:169`). +3. Chunk the sorted entries into fixed buckets of `bucket_size`; record each + bucket's first (== minimum) m/z in `bucket_min`; within each bucket sort by + `candidate_id` (`index.rs:172-178`). +4. Split into the three parallel arrays `idx_mz`/`idx_cid`/`idx_int`. + +`page_search` (`index.rs:242`) probes this index for an observed neutral m/z `q`: +`ppm_bounds(q, tol_ppm)` gives the query window `[lo, hi]` (cast to f32); a +`partition_point` over `bucket_min` selects the buckets overlapping the window +(`index.rs:258-260`); within each bucket, because `idx_cid` is ascending, two +`partition_point` calls narrow to the `[cand_lo, cand_hi)` slice +(`index.rs:266-268`); a linear tail applies the exact f32 m/z bound +(`index.rs:269-275`). `candidate_range` (`index.rs:233`) turns an isolation window +into `[lo, hi)` over `prec_mz` by two `partition_point`s. + +### The fragindex CSR matcher (`matchers/fragindex.rs`, fragindex_spec Section 2-3) + +`FragIndex::build` (`fragindex.rs:46`) is a two-pass counting sort into a CSR +layout keyed by log-space bin. `LogBins::new` (`binning.rs:27`) precomputes the +geometry: `delta = tol_ppm * 1e-6`, bin width `w = ln(1 + delta)`, `inv_w`, +`ln_min`, and `n_bins = floor(span * inv_w) + 2` (the `+2` pads the top so +`bin+1` never overflows). `LogBins::bin` (`binning.rs:43`) maps m/z to +`floor((ln(mz) - ln_min) * inv_w)`, clamped to `[0, n_bins-1]`. Pass 1 counts +per-bin occupancy with the `+1` counting-sort offset (`fragindex.rs:83-86`); a +prefix sum turns counts into CSR start offsets (`fragindex.rs:88-90`); pass 2 +scatters postings in candidate-id order (`fragindex.rs:98-110`) so `post_cand` is +ascending within every bin. Postings are Structure-of-Arrays +(`post_cand`/`post_mz`/`post_int`/`post_frag`) so the verify hot loop streams only +`post_mz`. + +`probe_peak` (`fragindex.rs:152`) probes bins `bin(peak)-1 ..= bin(peak)+1` +(clamped, `fragindex.rs:162-165`), narrows each bin to `[cand_lo, cand_hi)` by +binary search over the ascending `post_cand` (`fragindex.rs:172-174`), and +verifies each posting with the exact `within_ppm` predicate in f64 +(`fragindex.rs:176-179`). `SeedScratch` (`fragindex.rs:191`) is the epoch-stamped +dense accumulator: `stamp[cc]` records the last epoch a candidate was touched; +`epoch` is incremented before each scan (`fragindex.rs:221`) and starts at 0 so 0 +is never a live epoch; on first touch the score is zeroed and the candidate pushed +to `touched` (`fragindex.rs:227-232`); after a scan only touched candidates are +read. The seed accumulates a fused `(count, obs_sum)` semiring where `obs_sum` +sums the observed peak intensity per matched posting (predicted intensity +deliberately discarded, `fragindex.rs:234`). + +**The +/-1 probe exactness.** The correctness of probing only three bins rests on: +two m/z values within tolerance differ by at most one bin. Proof sketch: within +tolerance means `|ln(a) - ln(b)| <= w`, one bin width, so the two points span at +most two adjacent bins (`binning.rs` test `within_tol_pairs_are_at_most_one_bin_apart`, +`fragindex_spec.md` Section 2.2). One subtlety makes this exact in the +implementation: posting m/z is stored f32, so build bins each posting by the +**same f32-rounded value** the verify uses (`fragindex.rs:85` and +`fragindex.rs:102`), not the raw f64. Binning by raw f64 while verifying the f32 +value could place a posting two bins from the peak and silently drop a +boundary-straddling within-tolerance pair. The test +`probe_finds_within_tol_across_bin_boundaries` (`fragindex.rs:426`) sweeps the m/z +range to guard this. + +**The bucketed fallback vs fragindex predicate.** The two backends do not use the +same tolerance predicate. `page_search` uses `ppm_bounds` (a window symmetric +around the query `q`, `constants.rs:78`). `fragindex` uses `within_ppm` (a +min-relative predicate, `hi - lo <= tol_ppm*1e-6*lo`, `constants.rs:92`). These +differ at the tolerance edge, so a handful of edge pairs are accepted by one and +not the other. On AIF full-range-window data the predicate difference shifts +identifications enough to matter, which is why the bucketed path is retained for +A/B (`config.rs:37-39`). The fragindex `probe_peak` and its `naive` band-join both +verify under `within_ppm` on f32-rounded m/z (`naive.rs:30`), so the equivalence +gate is not contaminated by storage precision. + +### Wiring into stages + +Both stages build the fragindex backend only when `MatcherKind::Fragindex` is +selected and otherwise fall through to the bucketed `Library` path. +search-seed: `FragIndex::build` at `search_seed.rs:52-53`; fragindex path +`seed_fragindex_windows` parallelizes across isolation-window groups with a +per-thread `SeedScratch` and a deterministic total-order merge +(`search_seed.rs:297-382`); bucketed path uses `page_search` +(`search_seed.rs:73`). extract selects via the `probe_matched` helper +(`extract.rs:42-59`): fragindex carries the true generating fragment ordinal in +`post_frag`, whereas the bucketed path recovers the ordinal by nearest stored m/z +(`Library::local_frag_index`, `index.rs:217`). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `predict_frag::run` | `predict_frag.rs:50` | Stage C entry: parse, fragment, assign intensity/iRT, top-N, sort, write | +| `assign_rt` | `predict_frag.rs:245` | native or DeepLC iRT; emits the DeepLC-miss warning | +| `assign_intensities` | `predict_frag.rs:296` | native or MS2PIP intensity with per-charge-group normalization + native charge-2 fallback | +| `RtPredictor` / `FragmentPredictor` | `predict.rs:13` / `predict.rs:19` | predictor traits (predict + `identity`) | +| `NativeRt` | `predict.rs:25` | additive retention-coefficient model, `identity` `native-rt-v1` | +| `NativeFrag` | `predict.rs:73` | heuristic b/y intensity model, max-normalized, `identity` `native-frag-v1` | +| `resolve_script` | `sidecar.rs:18` | locate a worker script (CWD, exe dir, exe dir/scripts) | +| `run_ms2pip` | `sidecar.rs:37` | MS2PIP client; returns `cid -> (ion_byte, ordinal) -> intensity` | +| `run_deeplc` | `sidecar.rs:77` | DeepLC client; returns `id -> predicted_rt` | +| `Library` | `index.rs:38` | SoA candidate + fragment model plus the bucketed inverted index | +| `Library::load` | `index.rs:55` | read Parquet, group fragments, build index, enforce preconditions | +| `Library::page_search` | `index.rs:242` | bucketed probe: bucket select -> candidate slice -> f32 ppm verify | +| `Library::candidate_range` | `index.rs:233` | isolation window -> `[lo, hi)` over `prec_mz` | +| `Library::local_frag_index` | `index.rs:217` | nearest-stored-m/z fragment ordinal (bucketed path only) | +| `deconvolve` | `index.rs:283` | z-charged peak m/z -> neutral m/z, in f64 | +| `LogBins` / `LogBins::bin` | `binning.rs:11` / `binning.rs:43` | log-space bin geometry and mapping | +| `FragIndex::build` | `fragindex.rs:46` | two-pass counting-sort CSR build at a fixed tolerance | +| `FragIndex::probe_peak` | `fragindex.rs:152` | +/-1 bin probe + `within_ppm` verify, candidate-window narrowed | +| `SeedScratch` | `fragindex.rs:191` | epoch-stamped dense `(count, obs_sum)` accumulator | +| `score_scan_count_dot` (fragindex / naive) | `fragindex.rs:260` / `naive.rs:16` | equivalence-gate scorers under an identical predicate | +| `within_ppm` / `ppm_bounds` | `constants.rs:92` / `constants.rs:78` | min-relative vs query-relative tolerance predicates | + +## Configuration + +`PredictFragConfig` (`config.rs:292-322`, `#[serde(default, deny_unknown_fields)]`, +so the struct was pruned to exactly these fields and unknown keys are rejected): + +| field | default | effect | +|---|---|---| +| `predictor` | `Native` (`FragPredictorKind`) | native heuristic vs `Ms2pip` sidecar for fragment intensities | +| `rt_predictor` | `Native` (`RtPredictorKind`) | native additive model vs `Deeplc` sidecar for iRT | +| `charge2_from_precursor_charge` | `2` | precursor charge at/above which charge-2 fragments are added (was 3; lowered to keep the ~16% of charge-2 precursors' doubly-charged transitions) | +| `top_n_fragments` | `6` | fragments kept per candidate after intensity ranking (top-6 is standard DIA) | +| `ms2pip_model` | `"HCD"` | MS2PIP model name passed as argv | +| `ms2pip_python` | `None` | interpreter for the MS2PIP sidecar; required when `predictor=ms2pip`, else the stage errors | +| `deeplc_python` | `None` | interpreter for the DeepLC sidecar; required when `rt_predictor=deeplc`, else the stage errors | +| `sidecar_script_dir` | `"scripts"` | directory searched by `resolve_script` for the worker scripts | + +Matcher selection (both stages default to `Fragindex`): + +| field | default | effect | +|---|---|---| +| `MatcherKind` | `Fragindex` (`config.rs:46-49`) | `Fragindex` (log-bin CSR) or `Bucketed` (`Library::page_search`) | +| `search_seed.matcher` | `Fragindex` (`config.rs:340`) | backend for the seed search | +| `search_seed.fragment_tol_ppm` | `20.0` (`config.rs:352`) | tolerance the seed index is built at | +| `extract.matcher` | `Fragindex` (`config.rs:499,574`) | backend for extraction | +| `extract.bucket_size` | `8192` (`config.rs:485,570`) | fixed bucket size of the bucketed `Library` index (fragindex ignores it) | + +## Invariants, determinism, gotchas + +- **candidate_id contiguity.** `candidate_id` must be the dense range `0..N` in + precursor-m/z-ascending row order. predict-frag guarantees it + (`predict_frag.rs:137,163`); `Library::load` re-checks it and bails with a clear + message (`index.rs:78-87`); `FragIndex::build` asserts it (`fragindex.rs:50-56`). + An external library fed in unsorted or unindexed (for example + `import_diann_lib.py` output not passed through `make_reverse_decoys.py`) fails + here rather than silently misgrouping fragments. +- **precursors ascending by m/z.** `Library::load` verifies `prec_mz` is + non-decreasing and bails otherwise (`index.rs:135-146`); the `candidate_range` + binary search assumes it. +- **fragment foreign key.** A fragment row referencing `candidate_id >= N` is a + hard error (`index.rs:92-96`). +- **decoy presence is a warning, not an error.** A library with zero decoys makes + target-decoy q-values invalid, but bailing would break intentional decoy-free + diagnostics, so `load` warns loudly instead (`index.rs:150-156`). +- **total_frags must fit u32.** `FragIndex::build` asserts this + (`fragindex.rs:58`) because posting indices are u32. +- **f32 posting m/z, f64 math.** Index/posting m/z is stored f32; the ppm verify + widens to f64 (`fragindex.rs:176-177`, `index.rs` module docs). f32 ULP is + ~0.12 ppm, 200-400x below the 20-50 ppm regime, so storage precision is not a + gate disagreement source; build and verify use the same f32-rounded value. +- **per-posting accumulation.** A candidate with two fragments within tolerance of + one peak counts twice; a peak within tolerance of two candidate fragments counts + twice (fragindex_spec Section 1.4). Do not deduplicate. Tests + `two_frags_one_peak_counts_both` (`fragindex.rs:368`) and the count assertion in + `equivalence_gate_vs_naive` (`fragindex.rs:363`) guard this. +- **epoch, not value, for first touch.** First touch is `stamp[cc] != epoch`, never + `acc[cc] == 0`; a legitimate zero score would otherwise be misclassified + (`fragindex.rs:227`, spec Section 3.5). +- **determinism.** predict-frag maps rows in parallel but `collect` + a sequential + fold reproduce the serial order; the precursor-m/z sort is stable and top-N is a + deterministic ranking; `Library::load` uses a parallel stable sort equal to the + serial one; `FragIndex::build` is fully serial (candidate-order scatter, no + hashing); `seed_fragindex_windows` merges partial results in a total order + independent of thread/group order (`search_seed.rs:364-380`). `SeedScratch` + sums `obs_sum` in the caller's fixed peak order and `touched()` is in + first-touch order, so callers sort before any float reduction + (`fragindex.rs:239-243`). +- **DeepLC nondeterminism and the iRT-0 anchor.** The DeepLC sidecar and fine-tune + are not seeded, so iRT values vary run to run. Any peptidoform DeepLC does not + return lands at iRT 0.0; the warning at `predict_frag.rs:284-289` reports the + count so a large miss is visible rather than silent. +- **predicate mismatch between backends.** `page_search` (query-relative + `ppm_bounds`) and `fragindex` (`within_ppm`) accept slightly different edge + pairs; switching `MatcherKind` can move a small number of identifications. This + is expected, not a bug (`config.rs:37-39`). +- **`naive.rs` is not production.** It is O(C x frags x peaks), used only as the + equivalence oracle. + +## How to extend / modify + +- **Add a fragment or RT predictor.** Implement `FragmentPredictor` / + `RtPredictor` (`predict.rs:13,19`) for a native model, or add a variant to + `FragPredictorKind` / `RtPredictorKind` (`config.rs:114,99`) plus a branch in + `assign_intensities` / `assign_rt`. Match the existing `identity()` convention; + the id flows into `model_identity` and the artifact report. Keep the native + fallback path intact so the engine still runs with no Python. +- **Add a sidecar.** Follow the positional-CLI file contract in `sidecar.rs`: + write an input Parquet, invoke `run_worker(python, script, &[argv...], utf8)`, + read an output Parquet keyed by id. Use `resolve_script` so a deployed binary + finds the worker regardless of CWD. Set `utf8 = true` for Keras/torch workers + (they crash on the Windows cp1252 console). Never hardcode the interpreter path; + it comes from config (`ms2pip_python`, `deeplc_python`, or the rescore/finetune + equivalents). +- **Change the matcher.** New backends go under `matchers/` behind a `MatcherKind` + variant. Any new backend must pass the equivalence gate against `naive.rs` at + `K = C` under the same `within_ppm` predicate (test pattern in + `fragindex.rs:331`) before its speed is trusted. Before attempting a fragindex + optimization, read `fragindex_spec.md` Section 5: cache-blocking, accumulator + prefetch, radix partitioning, bin-major inversion, and distinct-m/z dedup were + all measured null or negative in the realistic DIA regime. The real levers are + top-N reduction (already applied, `top_n_fragments`) and per-window parallelism + (already applied in `seed_fragindex_windows`). +- **Change fragment charges or top-N.** `charge2_from_precursor_charge` and + `top_n_fragments` are the two knobs; both are sensitivity/speed tradeoffs. + Lowering top-N removes collisions roughly proportionally (spec Section 5.5) and + shrinks the library; it does not change matcher correctness. +- **Feeding an external library.** It must satisfy the `load()` preconditions + (dense `candidate_id` in precursor-m/z order, ascending `prec_mz`, decoys + present). The decoy-builder scripts (`make_reverse_decoys.py`) sort and reindex + to satisfy them; do not bypass that step. diff --git a/docs/07_search_seed.md b/docs/07_search_seed.md new file mode 100644 index 0000000..7110979 --- /dev/null +++ b/docs/07_search_seed.md @@ -0,0 +1,315 @@ +# search-seed (Stage S): broad search + mass recalibration + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage S is a native broad, DIA-aware search whose product is **calibration +anchors, not final identifications**. It scores library candidates against every +MS2 scan with a Sage-lite hyperscore, keeps the single best-scoring spectrum per +candidate, assigns target-decoy q-values, and from the confident subset derives a +per-run fragment mass recalibration (systematic ppm offset plus a learned +tolerance). Downstream stages consume its two outputs: + +- `rt-im-train` reads `seed_psms.parquet` (the confident target PSMs give the + observed-RT-vs-predicted-iRT anchors for LOESS/linear calibration and the RT + window widths). +- `extract` reads `.masscal.json` (the offset shifts predicted fragment + m/z, the learned tolerance replaces `extract.frag_tol_ppm`). + +The stage sits behind a file contract (`search_seed.rs:2-4`), so a real Sage / +sage-core adapter can replace the native scorer later without touching the +consumers, as long as it writes the same `seed_psms` schema and `masscal.json`. +Library-level decoys are the single source of truth: there is no engine-side decoy +generation here, so target-decoy counting is never mixed-method +(`search_seed.rs:6-7`). Because the seed is broad and best-per-candidate rather +than best-per-spectrum, it is deliberately high-recall and low-precision at this +point; precision is recovered downstream by RT/mass calibration, feature +rescoring, and FDR control. + +The stage is iRT-independent: `predicted_irt` is only copied through to the +output, never used in scoring. That is why the `run` orchestrator computes the +seed once on the base library and reuses it both before and after the optional +DeepLC fine-tune (`run.rs:182-208`). + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/search_seed.rs` | the stage: scoring loop, best-per-candidate merge, mass recalibration, output writers | +| `rust/mumdia/crates/mumdia/src/fdr.rs` | `target_decoy_q`, `count_targets_at_q`, `ln_factorial` (shared with `rescore`) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` (CSR inverted fragment index) + `SeedScratch` epoch accumulator | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library::load`, `candidate_range`, `page_search` (bucketed backend), `cand_frags` | +| `rust/mumdia/crates/mumdia/src/calibrate.rs` | `percentile` used by the tolerance fit | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | `load_ms2` (reads the converted MS2 Parquet into `Ms2Scan`) | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `ppm_bounds`, `ppm_diff`, `within_ppm` (the shared ppm math) | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `SearchSeedConfig` (`:326`), `MatcherKind` (`:42`) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::SEED_PSMS = ("seed_psms", 1)` (`:15`) | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI `Cmd::SearchSeed` (`:71`, dispatch `:441`) | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | orchestrator wiring (`:170`), masscal handoff to extract (`:230`) | + +## Inputs and outputs + +### Consumed + +- **MS2 spectra** (`--ms2`, a `convert` output Parquet), loaded by `load_ms2` + into `Vec`. Each `Ms2Scan` carries `scan_index`, `id`, `rt_seconds`, + an `IsolationWindow { target_mz, lower_mz, upper_mz, im_lower, im_upper }`, and + `peaks: Vec`. +- **Library** (`--library-precursors` + `--library-fragments`), loaded by + `Library::load` (`index.rs:55`). Provides `cands: Vec` + (`candidate_id`, `peptidoform`, `charge`, `precursor_mz`, `base_peptide_id`, + `protein`, `is_decoy`, `predicted_irt: f32`, `frag_start`, `n_frag`), the flat + fragment arrays (`frag_mz`, `frag_int`, `frag_name`), and the bucketed inverted + index for `page_search`. +- **Config**: `cfg.search_seed` plus `cfg.extract.bucket_size` (the bucketed + Library index bucket width; `run.rs:177`, `main.rs:456`). + +### Produced + +**`seed_psms.parquet`** (schema id `seed_psms` v1, `schema.rs:15`), one row per +candidate that matched at least `min_matched_peaks` fragments in at least one +scan, sorted by `candidate_id`. Written at `search_seed.rs:217-234`: + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | dense library candidate id (== library row index) | +| `peptidoform` | str | ProForma-lite peptidoform string | +| `charge` | i32 | precursor charge | +| `precursor_mz` | f64 | library precursor m/z | +| `base_peptide_id` | u32 | stripped-peptide id (for peptide-level rollup) | +| `protein` | str | protein accession | +| `label` | str | `"target"` or `"decoy"` (from `Candidate::is_decoy`) | +| `score` | f64 | best hyperscore over all scans | +| `spectrum_q` | f64 | best-per-candidate target-decoy q-value | +| `observed_rt` | f64 | RT (seconds) of the best-scoring scan | +| `predicted_irt` | f32 | library iRT, copied through (unused in scoring) | +| `matched_peaks` | i32 | matched-fragment count at the best scan | +| `scan_index` | u32 | `scan_index` of the best-scoring scan | + +**`.masscal.json`** (written at `search_seed.rs:204-214`): + +| key | type | meaning | +|---|---|---| +| `frag_ppm_offset` | f64 | median matched-fragment ppm deviation (systematic offset) | +| `frag_tol_ppm` | f64 | learned tolerance: `max(5.0, 1.5 * P95(|dev - offset|))` | +| `frag_ppm_sigma` | f64 | duplicate of `frag_tol_ppm` (the learned tolerance is the local mass-uncertainty estimate) | +| `n_dev` | usize | number of matched-fragment deviations collected | +| `cal_passes` | int | 0 (fallback, too few devs), 1 (single pass), or 2 (robust second pass) | + +**`.report.json`** (`ArtifactReport`, `search_seed.rs:240-258`): rows, +content hash, `params` (`fragment_tol_ppm`, `report_psms`, `min_matched_peaks`, +`top_n_peaks`, `fdr_seed`), a `stats` map (`psms`, `targets_at_q`), +`model_identity = "native-seed-hyperscore-v1"`, and `elapsed_ms`. + +## How it works + +Entry point: `search_seed::run(SearchSeedParams)` (`search_seed.rs:45-267`). + +**1. Load** the library (`Library::load`, `index.rs:55`) and MS2 scans +(`load_ms2`), then log candidate and scan counts (`search_seed.rs:47-49`). + +**2. Build the matcher.** When `cfg.matcher == Fragindex` (the default), a +`FragIndex` is built once over the whole library at `cfg.fragment_tol_ppm` +(`search_seed.rs:52-53`; `FragIndex::build`, `fragindex.rs:46`). This is a +log-space-binned CSR inverted fragment index: postings are scattered in +candidate-id order so `post_cand` is ascending within every bin, which lets +`probe_peak` narrow to the precursor-window candidate sub-range by binary search +(`fragindex.rs:151-182`). The bucketed backend uses the Library's own inverted +index via `page_search` and needs no separate build. + +**3. Best-per-candidate accumulation.** Two paths produce the same +`HashMap` where `Best { score, rt, matched, scan_index }` +(`search_seed.rs:37-43`): + +- *Fragindex path* (`seed_fragindex_windows`, `search_seed.rs:297-382`): scans + are grouped by isolation window, keyed on + `(lower_mz.to_bits(), upper_mz.to_bits())` in a `BTreeMap` for deterministic + group order (`:305-311`). Each scan belongs to exactly one window, so groups + are independent parallel units; `rayon` `par_iter().map_init(SeedScratch::new)` + processes them (`:315-362`). Per group, `candidate_range` is computed once + (`:324`). Per scan, `select_peaks` picks the probe set, `SeedScratch::accumulate` + probes each peak and fuses `(count, obs_sum)` per touched candidate + (`fragindex.rs:214-237`), candidates with `count >= min_matched_peaks` are + scored by `hyperscore`, sorted by score desc then candidate-id asc, truncated to + `report_psms`, and folded into a group-local best with a strictly-greater update + (`:347-357`). +- *Bucketed path* (serial, `search_seed.rs:62-98`): per scan, `candidate_range` on + the Library, `select_peaks`, then `page_search` for each probed peak accumulates + `(count, obs_sum)` into a `HashMap`. Same `min_matched_peaks` filter, hyperscore, + sort, `report_psms` truncation, and best-per-candidate update. + +Both accumulate `obs_sum` as the summed **observed** peak intensity of matched +postings; the predicted fragment intensity is deliberately discarded in the seed +(`fragindex.rs:185-190`, and the `_pi`/`_mz` discards at `search_seed.rs:73`). + +**Hyperscore** (`search_seed.rs:385-387`): + +``` +hyperscore = ln(matched!) + ln(1 + sum_obs) +``` + +`ln(matched!)` is `ln_factorial(matched)` = the summed logs of `2..=matched` +(`fdr.rs:132-138`), rewarding fragment breadth; `ln(1 + sum_obs)` adds a bounded +intensity term. This is the Sage-style form and is intentionally simple because +the seed is a calibration pass, not the scored identification. + +**4. Cross-group merge** (fragindex path, `search_seed.rs:365-381`): a total-order +merge over the per-group partials keeps, per candidate, `max score`, ties broken by +`earliest rt`, then `min scan_index`. This is order-independent, so it is +bit-identical to the serial global best regardless of thread or group scheduling. + +**5. q-values.** Rows are collected and sorted by `candidate_id` +(`search_seed.rs:102-103`), a `(score, is_decoy)` vector is built, and +`target_decoy_q` (`fdr.rs:7-52`) computes per-candidate q. That routine sorts by +score descending, walks tied-score blocks together so every PSM in a block gets +the same q, uses the conservative numerator `q = (n_decoys + 1) / max(1, n_targets)`, +and monotonizes from worst to best score so q is non-increasing. The output column +is `spectrum_q`. `count_targets_at_q(q, is_decoy, fdr_seed)` (`fdr.rs:110`) reports +the confident target count into `stats` (`search_seed.rs:139`). + +**6. Fragment mass recalibration** (`search_seed.rs:146-215`). A +`scan_index -> &Ms2Scan` map is built (`:146-149`). For every **confident target** +PSM (`!is_decoy && q <= fdr_seed`, `:152`), each library fragment m/z of that +candidate (`lib.cand_frags(cid)`, `index.rs:208`) is searched against the best +scan's peaks inside a **hardcoded 50 ppm window** (`ppm_bounds(fmz, 50.0)`, +`:158`; `partition_point` finds the low edge, then a linear scan to the high edge). +The nearest peak by absolute m/z distance contributes one signed ppm deviation +(`ppm_diff(peak_mz, fmz)`, `constants.rs:66`) to `devs` (`:156-173`). The 50 ppm +net is deliberately wider than `fragment_tol_ppm` so a systematic offset larger +than the tolerance can still be measured. + +The fit closure (`:178-185`) takes deviations, sorts them, takes the median as the +offset, then sets `tol = max(5.0, 1.5 * P95(|dev - offset|))` using +`calibrate::percentile(.., 0.95)`. Control flow (`:186-203`): + +- `devs.len() < 20`: fallback, `(0.0, fragment_tol_ppm, cal_passes = 0)` (no + offset, tolerance unchanged). +- otherwise single pass: `(o1, t1, 1)`. +- if `two_pass_mass_cal`: keep only deviations inside `|dev - o1| <= t1` and, when + `>= 20` survive, re-fit to `(o2, t2, 2)`; otherwise keep the single-pass result. + The second pass rejects random-match outliers so they cannot bias the median. + +The result is written to `.masscal.json` (`:204-214`). + +**7. Write** `seed_psms.parquet` (`write_table`, `:217`) and the `ArtifactReport` +(`:240-258`), then log `psms`, `confident`, `elapsed_ms`. + +### How the outputs are consumed + +`run` writes the seed to `seed_psms.parquet` (`run.rs:170-180`) and passes +`mass_cal: Some(".masscal.json")` to `extract` (`run.rs:230`). In +`extract` (`extract.rs:620-634`), the JSON is read: `frag_ppm_offset` becomes a +multiplicative shift `offset_factor = 1.0 + frag_ppm_offset * 1e-6` applied to +predicted fragment m/z, and `frag_tol_ppm` replaces `extract.frag_tol_ppm` as the +matching tolerance (falling back to the config value if the file is absent). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `run` | `search_seed.rs:45` | stage entry point; orchestrates load, score, q, masscal, write | +| `SearchSeedParams` | `search_seed.rs:27` | input struct (ms2/library paths, cfg, bucket_size, config_hash) | +| `Best` | `search_seed.rs:37` | per-candidate best `{ score, rt, matched, scan_index }` | +| `seed_fragindex_windows` | `search_seed.rs:297` | parallel per-window fragindex scoring + deterministic merge | +| `select_peaks` | `search_seed.rs:272` | top-N-by-intensity peak selection, re-sorted to index order | +| `hyperscore` | `search_seed.rs:385` | `ln(matched!) + ln(1 + sum_obs)` | +| `FragIndex::build` | `fragindex.rs:46` | build CSR inverted index at a fixed tolerance | +| `FragIndex::probe_peak` | `fragindex.rs:152` | matched postings for one peak in a candidate range | +| `SeedScratch::accumulate` | `fragindex.rs:214` | epoch-stamped fused `(count, obs_sum)` accumulation | +| `Library::candidate_range` | `index.rs:233` | half-open candidate id range for an isolation window | +| `Library::page_search` | `index.rs:242` | bucketed inverted-index probe (bucketed backend) | +| `Library::cand_frags` | `index.rs:208` | `(m/z, intensity, name)` slices for a candidate | +| `target_decoy_q` | `fdr.rs:7` | tied-block, monotonized `(n_decoys+1)/max(1,n_targets)` q | +| `count_targets_at_q` | `fdr.rs:110` | target count at or below a q threshold | +| `ln_factorial` | `fdr.rs:132` | `ln(n!)` via summed logs | +| `ppm_diff` / `ppm_bounds` | `constants.rs:66` / `:78` | signed ppm and ppm window bounds | +| `percentile` | `calibrate.rs:154` | nearest-rank percentile (used for the tolerance) | + +## Configuration + +`SearchSeedConfig` (`config.rs:326-360`, `#[serde(default, deny_unknown_fields)]`). +The struct was pruned to the fields actually read here; unknown keys are rejected +on load. + +| field | default | effect | +|---|---|---| +| `fdr_seed` | `0.01` | q threshold defining "confident" for the calibration subset and the reported `targets_at_q` stat | +| `fragment_tol_ppm` | `20.0` | matching tolerance for scoring; also the masscal fallback tolerance when too few deviations | +| `report_psms` | `5` | max candidates kept per spectrum before the best-per-candidate fold (wide-window DIA) | +| `min_matched_peaks` | `4` | minimum matched fragments for a candidate to score in a scan | +| `top_n_peaks` | `300` | probe only the N most intense peaks per scan (`0` = all); seed-only, does not shrink the extract artifact | +| `matcher` | `Fragindex` | backend; `MatcherKind::Bucketed` (`config.rs:42`) uses the serial `page_search` path | +| `two_pass_mass_cal` | `false` | robust second-pass mass fit on the in-window inliers (sensitivity_plan P3.1) | + +`bucket_size` for the bucketed Library index is taken from `cfg.extract.bucket_size`, +not from `SearchSeedConfig` (`run.rs:177`). The 50 ppm mass-calibration search +window, the `min 5.0 ppm` tolerance floor, the `1.5 * P95` scale, and the `>= 20` +deviation threshold are hardcoded in `search_seed.rs` (`:158`, `:183`, `:186`), +not config fields. + +## Invariants, determinism, gotchas + +- **Determinism** (PLAN.md Section 7): the fragindex parallel path is bit-identical + to the serial best-per-candidate. Groups run in `BTreeMap` key order, within a + group scans run in file (RT-ascending) order with a strictly-greater update, and + the cross-group merge is a total order (`max score`, tie earliest RT, tie min + scan_index; `search_seed.rs:365-381`). `select_peaks` re-sorts the top-N back to + index-ascending (`:283`) so the `obs_sum` float reduction is summed in a fixed + order. The `HashMap` is only ever reduced through this total order, never + iterated for a float sum. +- **Best-per-candidate, not best-per-spectrum.** One output row per candidate that + ever cleared `min_matched_peaks`. Ties in the update use strictly-greater + (`score > entry.score`, `:93` and `:354`), so the earliest-RT scan wins a tie. +- **Library decoys only.** `label` comes straight from `Candidate::is_decoy` + (`:129`); the stage never mints decoys. The target-decoy null therefore depends on + the library carrying paired decoys (see the DIA-NN library recipe / `digest`). +- **obs_sum is observed intensity.** The seed drops predicted fragment intensity on + purpose (`fragindex.rs:185-190`); do not "fix" the discarded `_pi`/`_mz`. +- **Mass-cal fallback is an identity.** With `< 20` deviations the offset is `0.0` + and the tolerance is left at `fragment_tol_ppm` (`cal_passes = 0`); extract then + applies no shift. `frag_ppm_sigma` is an exact copy of `frag_tol_ppm`. +- **The 50 ppm mass-cal window is fixed** and independent of `fragment_tol_ppm`; it + is intentionally wide so a systematic error larger than the scoring tolerance is + still observable. Do not tie it to `fragment_tol_ppm`. +- **`config_hash` is carried but unused** inside `run` (part of `SearchSeedParams` + for call-site uniformity, `search_seed.rs:34`); the report hash is the content + hash of the output file, not this value. +- **`top_n_peaks` is seed-only.** It caps the probe set to reduce index probing on + the dominant cost, but abundant peptides supply the calibration anchors anyway; + the downstream `extract` stage still sees all converted peaks. +- **`MatcherKind::Bucketed` stays serial** (`search_seed.rs:59-98`); only the + fragindex path is parallelized. +- **iRT is inert here.** `predicted_irt` is only copied to the output, so the seed + can be computed once and reused across the DeepLC fine-tune boundary + (`run.rs:182-208`). +- **Test coverage.** Only `select_peaks` has a unit test here + (`search_seed.rs:389-422`); `target_decoy_q`/`ln_factorial` are tested in + `fdr.rs` and the index in `fragindex.rs`. There is no stage-level test for + `search_seed::run`, the masscal output, or the bucketed vs fragindex equivalence + at the stage level (see CLAUDE.md "test gaps"). + +## How to extend / modify + +- **Swap in a real Sage / sage-core scorer.** Replace the accumulation in `run` + behind the existing file contract. Keep the `seed_psms` schema and + `masscal.json` keys unchanged so `rt-im-train` and `extract` need no edits. Bump + `model_identity` (`search_seed.rs:255`) so provenance in `report.json` is honest. +- **Add a `seed_psms` column.** Bump the schema version at `schema.rs:15` + (`("seed_psms", 1)` -> `2`), push the new `Col` in the `write_table` call + (`search_seed.rs:217-234`), and update any consumer that reads by column name. +- **Make the mass-cal window / thresholds configurable.** The 50 ppm search + window, the `>= 20` minimum, the `1.5x` scale, and the `5.0 ppm` floor are + literals in `run`; promote them to `SearchSeedConfig` fields (with conservative + defaults) if you need to tune them, per the "every algorithmic choice is a typed + config field" convention. +- **Add a matcher backend.** Add a `MatcherKind` variant (`config.rs:42`) and a + branch in the `if let Some(idx) = fidx` dispatch (`search_seed.rs:59-99`); mirror + the deterministic merge if the new path is parallel. +- **Charge-2 / m/z-binned tolerance.** The current fit produces one global offset + and tolerance. To make them charge- or m/z-dependent, partition `devs` before the + `fit` closure and emit per-bin entries in `masscal.json`, then teach `extract` to + pick the matching bin. +- **Change the score.** `hyperscore` (`search_seed.rs:385`) is a free function; keep + it monotone in matched-fragment count and observed intensity so the target-decoy q + ordering stays meaningful, and keep the summation order fixed for determinism. diff --git a/docs/08_rt_im_train.md b/docs/08_rt_im_train.md new file mode 100644 index 0000000..be89eb2 --- /dev/null +++ b/docs/08_rt_im_train.md @@ -0,0 +1,350 @@ +# rt-im-train (Stage B): RT calibration + DeepLC fine-tune + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage B (`mumdia rt-im-train`, PLAN.md Stage B) turns the run-independent +predicted iRT carried on each library candidate into a per-run predicted +retention time in seconds, and derives a per-candidate RT acceptance window that +the extractor uses to bound its scan search. It is the bridge between the +library (built once, run-independent) and this run's chromatography. + +The stage does two things: + +1. Fit a calibration map `predicted_irt -> observed_rt` from the confident seed + PSMs of this run (linear least squares always; a LOESS local-linear smoother + when configured and enough anchors are present). +2. Set an RT half-window from the residual distribution of that fit + (residual percentile times a multiplier), and emit `rt_lo`/`rt_hi` around the + calibrated RT of every library candidate. + +An optional pre-step, wired into the `run` orchestrator rather than into this +stage, fine-tunes the DeepLC multitask model on the same confident seed PSMs and +rewrites the library's `predicted_irt` before Stage B reads it. The fine-tune is +a Python sidecar and is nondeterministic. It is default-off. + +Ion mobility (IM) is stubbed. The MVP is 3D, so the IM columns are always +written null; there is no IM calibration or IM window. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs` | The stage. Joins iRT to seed PSMs, fits calibration, computes windows, writes `run_windows.parquet` + `cal.json`. | +| `rust/mumdia/crates/mumdia/src/calibrate.rs` | Calibration math: `linear_fit`, the `Loess` local-linear smoother, and `percentile`. Shared, no external deps. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | Orchestrator. Runs the optional DeepLC fine-tune between `search-seed` and this stage, then calls `rt_im_train::run` (see `run.rs:187-219`). | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | `run_deeplc_finetune` (sidecar.rs:106-129): the file-contract client that invokes the fine-tune worker. | +| `scripts/deeplc_finetune.py` | The fine-tune worker: transfer-learns DeepLC 4.0 on the seed and rewrites `predicted_irt` in the library parquet. | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `RtImTrainConfig` (config.rs:362-427), `CalibrationMethod` enum (config.rs:66-77), and the load-time validation that rejects `calibration_method=none` (config.rs:1067-1073). | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::RUN_WINDOWS = ("run_windows", 1)` (schema.rs:16). | + +## Inputs and outputs + +### Consumed + +**`fragment_library_precursors.parquet`** (the run-independent library; in +library-input mode this is the imported speclib, and when `finetune_deeplc` is +set it is the `_ft` rewrite). Columns read (rt_im_train.rs:34-35): + +| column | type | use | +|---|---|---| +| `candidate_id` | u32 | join key; also the output row identity | +| `predicted_irt` | f32 | the value calibrated to observed RT | + +The library is the single source of truth for iRT (rt_im_train.rs:31-32): both the +training anchors and the applied calibration read `predicted_irt` from this same +table, so a patched or fine-tuned library iRT is used consistently for fit and +apply. + +**`seed_psms.parquet`** (from `search-seed`). Columns read (rt_im_train.rs:44-49): + +| column | type | use | +|---|---|---| +| `candidate_id` | u32 | join to library `predicted_irt` | +| `base_peptide_id` | u32 | best-per-peptide grouping key | +| `spectrum_q` | f64 | confidence gate (`< q_train`) | +| `score` | f64 | picks the best PSM within a `base_peptide_id` | +| `observed_rt` | f64 | the calibration target (seconds) | +| `label` | str | only rows equal to `"target"` anchor the fit | + +### Produced + +**`run_windows.parquet`** (`artifact::RUN_WINDOWS`, version 1). One row per +library candidate. Columns (rt_im_train.rs:182-193): + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | library candidate identity | +| `rt_pred_cal` | f64 | calibrated predicted RT (seconds) | +| `rt_lo` | f64 | `rt_pred_cal - width` (window lower bound, seconds) | +| `rt_hi` | f64 | `rt_pred_cal + width` (window upper bound, seconds) | +| `im_pred_cal` | f64, nullable | always `None` (3D MVP) | +| `im_lo` | f64, nullable | always `None` | +| `im_hi` | f64, nullable | always `None` | + +**`cal.json`** (side artifact, written with `mumdia_io::json::write_json`, +rt_im_train.rs:196-208). Fields: + +| field | meaning | +|---|---| +| `method` | `"loess"` if the LOESS path was used, else `"linear"` | +| `slope`, `intercept` | the linear fit coefficients (always computed, even under LOESS) | +| `w_rt` | the global RT half-window (seconds) | +| `p_rt` | the residual percentile used | +| `multiplier` | `rt_window_multiplier` | +| `n_train` | number of calibration anchors | +| `calibration_status` | `"loess"`, `"linear"`, or `"fallback_fixed"` | + +**`.report.json`** (`ArtifactReport`, rt_im_train.rs:215-227) records +rows, content hash, params (`q_train`, `p_rt`, `method`), and stats (`n_train`, +`w_rt`, `calibration_status`). + +## How it works + +The stage entry point is `run(p: RtImTrainParams)` (rt_im_train.rs:28). + +### 1. Join iRT to candidates + +Read the library precursors and build `irt_by_cid: HashMap` mapping +`candidate_id -> predicted_irt` (rt_im_train.rs:33-39). This is the only source of +iRT; both training and application use it. + +### 2. Select calibration anchors + +Read the seed PSMs and build `best_per_pep: HashMap`, +keyed by `base_peptide_id` (rt_im_train.rs:51-69). A seed row enters the pool only +if: + +- `spectrum_q < q_train` (rt_im_train.rs:53), and +- `label == "target"` (rt_im_train.rs:58); a decoy anchor would inject a random + iRT/RT pair into the fit, and +- its `candidate_id` resolves to a library `predicted_irt` (rt_im_train.rs:61). + +Within a `base_peptide_id`, the highest-`score` PSM wins (rt_im_train.rs:66-67), so +each confident peptide contributes exactly one `(predicted_irt, observed_rt)` +anchor. The anchors are then split into parallel vectors `train_irt`/`train_rt` +with `n_train = train_irt.len()` (rt_im_train.rs:70-72). + +### 3. Fit the calibration map + +A linear fit is always computed: `let (slope, intercept) = linear_fit(&train_irt, +&train_rt)` (rt_im_train.rs:76). LOESS is used only when the method is +`CalibrationMethod::Loess` and there are at least `min_seed_for_calibration` +anchors (rt_im_train.rs:77-78); it is fit with `Loess::fit(&train_irt, &train_rt, +loess_span, 200)` (200 grid points, rt_im_train.rs:80). + +`predict` is a closure (rt_im_train.rs:85-90): `loess.predict(irt)` when a LOESS +model exists, else `slope * irt + intercept`. The linear coefficients are +therefore both the primary map (Linear method) and the extrapolation fallback +(LOESS outside its training range; see below). + +The math: + +- **`linear_fit`** (calibrate.rs:6-28) is ordinary least squares. With + `n = xs.len()`: `slope = (n*Sxy - Sx*Sy) / (n*Sxx - Sx*Sx)` and + `intercept = (Sy - slope*Sx) / n`. Degenerate guards: fewer than 2 points + returns `(0, mean(ys))` (a constant map, calibrate.rs:8-16); a near-zero + denominator (all `x` equal) returns `(0, Sy/n)` (calibrate.rs:22-24). +- **`Loess::fit`** (calibrate.rs:42-83) first computes the global linear fit as + the extrapolation fallback (calibrate.rs:43), sorts the points by `x` + (calibrate.rs:45-48), and if fewer than 4 points are present just fills the grid + from the linear line (calibrate.rs:50-66). Otherwise the local window size is + `k = clamp(ceil(span * n), 3, n)` (calibrate.rs:67) and it evaluates a + local-linear regression on a uniform grid of `grid_n` points spanning + `[min(x), max(x)]` (calibrate.rs:69-76). +- **`local_linear`** (calibrate.rs:107-151) selects the `k` nearest anchors by + walking outward from the insertion point (calibrate.rs:110-127), assigns each a + tricubic weight `w = (1 - d^3)^3` with `d = |x_i - x0| / dmax` (calibrate.rs: + 131-137), and solves a weighted least-squares line, returning the fitted value + at `x0`. If the weighted system is degenerate it falls back to the global line + (calibrate.rs:145-147). +- **`Loess::predict`** (calibrate.rs:87-102) uses the linear fallback for `x` at or + outside the grid ends (calibrate.rs:89-94) and otherwise linearly interpolates + between the two bracketing grid nodes found by `partition_point` + (calibrate.rs:95-101). Interpolating a precomputed grid makes bulk application + over the whole library cheap. + +### 4. Derive the RT window + +`min_anchors = min_seed_for_calibration.max(2)` (rt_im_train.rs:97). When +`n_train >= min_anchors`, the absolute residuals `|observed_rt - predict(irt)|` +are formed (rt_im_train.rs:99-103), and the global half-window is `w_rt = +percentile(resid, p_rt) * rt_window_multiplier`, floored at 1.0 second +(rt_im_train.rs:104-105). The status is `"loess"` or `"linear"`. When +`n_train < min_anchors` the stage warns and uses the fixed fallback +`fallback_rt_window_s` with status `"fallback_fixed"` (rt_im_train.rs:108-114). + +The anchor-count gate exists for a concrete failure mode documented in the code +(rt_im_train.rs:92-96): with only a handful of anchors a linear fit passes almost +exactly through them, so residuals are near zero, the percentile window collapses +to the 1-second floor, and that floor then discards nearly every true co-elution +downstream. The fixed fallback avoids that collapse. + +### 5. Optional adaptive per-region window (default off) + +When `adaptive_rt_window` is set (and `n_train >= min_anchors`), the global +`w_rt` is replaced by a per-bin half-width (rt_im_train.rs:120-153). Anchors are +binned into `adaptive_rt_bins` equal-width bins over the calibrated-RT range; each +bin's half-width is its local residual percentile times the multiplier, clamped to +`[rt_window_min_s, fallback_rt_window_s]` (rt_im_train.rs:134-146). Empty bins fall +back to the global `w_rt` (rt_im_train.rs:139-140). Each candidate then takes the +width of the bin its calibrated RT lands in (rt_im_train.rs:165-171). The rationale +(config.rs:393-402): a single fixed window is simultaneously too wide for +well-calibrated regions and too narrow for poorly-calibrated ones. This knob is +part of the sensitivity program and has not passed the entrapment gate, so it stays +default-off. + +### 6. Apply to every candidate and write + +For each library candidate (rt_im_train.rs:163-180): `cal = predict(irt)`, the width +is either the adaptive per-bin value or the global `w_rt`, and the row is +`(candidate_id, cal, cal - width, cal + width)` with the three IM columns pushed as +`None`. The table is written (rt_im_train.rs:182-193), `cal.json` is written +(rt_im_train.rs:196-208), and the artifact report is emitted (rt_im_train.rs:215-227). + +### DeepLC multitask fine-tune (orchestrator pre-step, default off) + +This is not part of `rt_im_train::run`; it runs in the `run` orchestrator between +`search-seed` and Stage B, guarded by `cfg.rt_im_train.finetune_deeplc` +(run.rs:187-208). Preflight (`run.rs:63-68`) rejects `finetune_deeplc` unless +`predict_frag.deeplc_python` names a Python interpreter with DeepLC 4.0 multitask. + +When enabled, the orchestrator resolves `deeplc_finetune.py` +(`sidecar::resolve_script`), writes a fine-tuned library +`fragment_library_precursors_ft.parquet`, and rebinds `lib_p` to that file so both +Stage B and `extract` read the fine-tuned iRT (run.rs:187-208). The fine-tune is +driven through `run_deeplc_finetune` (sidecar.rs:106-129), whose positional CLI +contract is `deeplc_finetune.py --epochs E --patience P +--q-train Q --batch B`. The worker runs with `PYTHONUTF8=1` because DeepLC/Keras +crash on the Windows cp1252 console (sidecar.rs:173-182). + +Inside the worker (`scripts/deeplc_finetune.py`): the reference set is the +confident target seed PSMs (`label == "target"`, `spectrum_q <= q_train`, standard +residues only), one observed RT per peptidoform (deeplc_finetune.py:97-109). The +batch size auto-scales when `--batch 0`: `min(512, max(16, n_ref // 30))`, so each +epoch runs at least about 30 gradient steps; a fixed large batch underfits small +references (deeplc_finetune.py:111-117). `deeplc.finetune` transfer-learns the +model (deeplc_finetune.py:128), and predictions are made on the DECOY_-stripped +underlying sequence so decoys land on the same iRT scale as targets +(deeplc_finetune.py:44, 135-154). The rewritten column overwrites `predicted_irt` +in place in the output parquet (deeplc_finetune.py:156-159); peptidoforms with no +prediction keep their original iRT (deeplc_finetune.py:156). + +The fine-tune sets no torch/numpy seed, so it is nondeterministic across runs +(CLAUDE.md notes this). Its main use is library-input mode, where the base iRT is +the imported DIA-NN library value rather than a native/DeepLC prediction, and a +per-run fine-tune materially tightens the RT window. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `RtImTrainParams` | rt_im_train.rs:19-26 | Input struct: seed PSMs path, library precursors path, output windows + cal paths, config, config hash. | +| `rt_im_train::run` | rt_im_train.rs:28-231 | The stage: join iRT, select anchors, fit, window, apply, write. | +| `linear_fit` | calibrate.rs:6-28 | OLS `y = slope*x + intercept` with degenerate-case guards. | +| `Loess` | calibrate.rs:31-37 | Grid-based local-linear smoother; carries a linear fallback for extrapolation. | +| `Loess::fit` | calibrate.rs:42-83 | Sorts anchors, builds a `grid_n`-point local-linear grid, `k = clamp(ceil(span*n),3,n)`. | +| `Loess::predict` | calibrate.rs:87-102 | Grid interpolation inside range, linear extrapolation outside. | +| `local_linear` | calibrate.rs:107-151 | Tricubic-weighted local least squares at one point. | +| `percentile` | calibrate.rs:154-162 | Nearest-rank percentile: sorts a copy, `rank = round(p*(len-1))`. Not interpolated. | +| `CalibrationMethod` | config.rs:66-77 | Enum `{ Loess, Linear, None }`; default `Loess`. `None` is rejected at load. | +| `RtImTrainConfig` | config.rs:362-427 | All Stage B config fields (below). | +| `run_deeplc_finetune` | sidecar.rs:106-129 | Sidecar client: `deeplc_finetune.py [flags]`. | + +## Configuration + +All fields live under `rt_im_train` in the config (`RtImTrainConfig`, +config.rs:362-427). The struct is `#[serde(default, deny_unknown_fields)]`, so any +unknown key is a hard load error and every field has the default below. The config +surface was pruned: there are no IM calibration fields (IM is a stub), and +`CalibrationMethod::None` is a foot-gun rejected at load (config.rs:1067-1073) even +though the enum variant still exists. + +| field | default | effect | +|---|---|---| +| `calibration_method` | `loess` | `loess`, `linear`, or `none`. `loess` uses the smoother when anchors suffice, else falls back to linear. `none` is rejected by `Config::validate` (config.rs:1067-1073) because the code path silently degrades to linear. | +| `q_train` | `0.01` | Max `spectrum_q` for a seed PSM to become a calibration anchor (rt_im_train.rs:53) and to enter the DeepLC fine-tune reference (`--q-train`). | +| `p_rt` | `0.95` | Residual percentile for the RT half-window (rt_im_train.rs:104). 0.95 keeps ~95% of anchor residuals inside the window. | +| `rt_window_multiplier` | `1.0` | Scales the residual-percentile window (rt_im_train.rs:104). Larger widens the window: higher recall, more interference. | +| `min_seed_for_calibration` | `50` | Minimum anchors to (a) use LOESS instead of linear (rt_im_train.rs:78) and (b) trust the residual window instead of the fixed fallback (via `min_anchors = max(this, 2)`, rt_im_train.rs:97). | +| `loess_span` | `0.3` | LOESS local-fit fraction; passed as `span` to `Loess::fit` (rt_im_train.rs:80). Fraction of anchors in each local window. | +| `fallback_rt_window_s` | `120.0` | Fixed half-window (seconds) when anchors are too few (rt_im_train.rs:113); also the upper clamp for adaptive bin widths (rt_im_train.rs:135). | +| `finetune_deeplc` | `false` | Enable the DeepLC multitask fine-tune pre-step in `run` (run.rs:187). Requires `predict_frag.deeplc_python` (preflight, run.rs:63-68). | +| `finetune_epochs` | `25` | Fine-tune epoch cap (`--epochs`); early stopping usually halts sooner. Used only when `finetune_deeplc`. | +| `finetune_patience` | `10` | Early-stopping patience (`--patience`): epochs without val-loss improvement before stopping. | +| `finetune_batch` | `0` | Fine-tune batch size (`--batch`). `0` auto-scales to the seed size so each epoch has ~30+ steps (config.rs:388-392); a fixed large batch underfits small seeds. | +| `adaptive_rt_window` | `false` | Per-region window instead of one global width (rt_im_train.rs:120-153). Sensitivity-program knob, not yet default-on. | +| `adaptive_rt_bins` | `12` | Number of equal-width calibrated-RT bins for the adaptive window (rt_im_train.rs:126). | +| `rt_window_min_s` | `1.0` | Lower clamp (seconds) for any adaptive half-window (rt_im_train.rs:134); mirrors the 1s floor on the global window. | + +## Invariants, determinism, gotchas + +- **IM is a stub.** `im_pred_cal`, `im_lo`, `im_hi` are always `None` + (rt_im_train.rs:177-179). There is no IM calibration, no IM window, and no IM + config field. Any 4D/diaPASEF work must add both the model and the columns. +- **Only targets anchor the fit.** Decoy seed PSMs are excluded + (rt_im_train.rs:58); admitting them would inject random iRT/RT pairs. Keep this + filter if you refactor anchor selection. +- **Library is the single iRT source.** Training and application both read + `predicted_irt` from the same library table (rt_im_train.rs:31-39, 156-157). When + the fine-tune rewrites the library, the orchestrator rebinds `lib_p` to the `_ft` + file (run.rs:205) so both this stage and `extract` see the updated iRT. Do not + reintroduce a second iRT source. +- **The 1-second floor.** The global window is floored at 1.0s (rt_im_train.rs:105). + With too few anchors the fit is near-exact, residuals collapse, and this floor + would discard true co-elutions, which is exactly why the `min_anchors` fixed + fallback exists (rt_im_train.rs:92-114). Do not remove the fallback branch. +- **`percentile` is nearest-rank, not interpolated** (calibrate.rs:154-162). It + sorts a copy each call, so it is order-independent, but repeated calls on the + same data re-sort. This is fine at Stage B sizes. +- **Determinism caveat: HashMap iteration order.** `best_per_pep` is a `HashMap` + and `train_irt`/`train_rt` are built from `.values()` (rt_im_train.rs:70-71), + whose order is randomized per process. `linear_fit` sums over that order + (calibrate.rs:17-20), and floating-point addition is not associative, so `slope`, + `intercept`, `w_rt`, and the calibrated RTs can differ in the last few ULPs + between runs. The effect is tiny but real. `Loess::fit` re-sorts by `x` + internally (calibrate.rs:45-48), so its grid is order-stable, yet it too reuses the + order-dependent linear fallback. If byte-identical calibration is required, sort + the anchors by `candidate_id` (or `base_peptide_id`) before summing. CLAUDE.md's + determinism rule (ordered iteration where floats are summed) applies here. +- **The fine-tune is explicitly nondeterministic.** `deeplc_finetune.py` sets no + torch/numpy seed (CLAUDE.md), so `finetune_deeplc = true` makes `predicted_irt`, + and therefore the whole run, non-reproducible. Treat it as an accuracy lever, not + a deterministic default. +- **`slope`/`intercept` are always emitted in `cal.json` even under LOESS** + (rt_im_train.rs:76, 197-208). They are the LOESS extrapolation fallback, not dead + values; do not assume they were unused when `method == "loess"`. +- **`CalibrationMethod::None` still exists but is rejected** at config load + (config.rs:1067-1073). The stage would otherwise fall through to the linear path + because `use_loess` matches only `Loess` (rt_im_train.rs:77). This fallthrough is + a known correctness wart (see CLAUDE.md "Correctness"). +- **Report schema version is 1** (`artifact::RUN_WINDOWS`, schema.rs:16). Bump it if + the column set changes. + +## How to extend / modify + +- **Add IM (4D).** Populate `im_pred_cal`/`im_lo`/`im_hi` (currently `None` at + rt_im_train.rs:177-179) from an IM calibration analogous to the RT path (an + IM2Deep-style model), and add IM config fields to `RtImTrainConfig`. The output + columns already exist and are nullable, so downstream reads survive the + transition. `extract` and the IM feature families must then consume them. +- **Change the calibration model.** Extend `CalibrationMethod` (config.rs:66-77) and + branch in the `predict` closure setup (rt_im_train.rs:77-90). Keep `linear_fit` as + the universal fallback so a degenerate anchor set never panics. New models belong + in `calibrate.rs` next to `Loess`, with unit tests like `loess_tracks_nonlinear` + (calibrate.rs:176-183). +- **Tune the window policy.** The residual-percentile logic is localized at + rt_im_train.rs:97-153. New window strategies (for example a symmetric-vs-asymmetric + window, or a learned per-charge width) should be config-gated and default-off, then + validated against the entrapment holdout before becoming a default, per the + sensitivity program (`sensitivity_plan/NEXT_STEPS.md`). +- **Make calibration byte-deterministic.** Sort the anchors deterministically (by + `candidate_id`) before building `train_irt`/`train_rt` at rt_im_train.rs:70-71, so + `linear_fit`'s summation order is fixed. +- **Swap the fine-tune worker.** The contract is purely the CLI and the parquet + columns (`sidecar.rs:106-129`, `scripts/deeplc_finetune.py`). A replacement worker + need only accept ` ` plus the four flags and rewrite the + `predicted_irt` column. Preserve the DECOY_-stripping behavior + (deeplc_finetune.py:44, 135-156) so decoys stay on the target iRT scale, and set a + seed if you want the fine-tune to be reproducible. diff --git a/docs/09_extract.md b/docs/09_extract.md new file mode 100644 index 0000000..8ca8fe7 --- /dev/null +++ b/docs/09_extract.md @@ -0,0 +1,450 @@ +# extract (Stage D): the core targeted extraction +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +`extract` is the central stage of the pipeline. It takes the run-independent +spectral library (precursor and b/y fragment m/z, predicted intensities, iRT), +the per-candidate RT windows learned in `rt-im-train`, and the converted MS2 +(optionally MS1) scans, and produces one apex-level PSM per surviving candidate +plus per-fragment chromatograms. Exact intensity-based scores are not computed +here; they are computed downstream in `features` from the chromatograms this +stage emits. `extract` is therefore an evidence-gathering and coarse-acceptance +stage, not a scoring stage. + +The design is **data-driven and peak-major** (see the module doc comment, +`extract.rs:1`). Observed peaks probe an inverted fragment index, and a candidate +hypothesis is materialized only where fragment evidence actually collides with an +observed peak. Work scales with peak-candidate collisions, not with library size. +In wide-window DIA roughly 98% of fragment m/z collide within tolerance +(`extract.rs:678`), so one observed peak typically matches many co-isolated +candidates; the peak-claim strategies decide how that shared intensity is +apportioned. + +RT is applied as a per-candidate window post-filter (the documented Stage D part +2 fallback), and the MVP is 3D so the ion-mobility (IM) dimension is absent +(`extract.rs:6`, `apex_im` is always written `None` at `extract.rs:1319`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/extract.rs` | The stage: accumulation, cascade, apex, chromatogram/MS1 emission, schema writing | +| `rust/mumdia/crates/mumdia/src/peaks.rs` | Pure top-K chromatographic peak enumerator (`enumerate_peaks`) for the `retain_top_peaks` path | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library` (SoA library + bucketed `page_search`, `candidate_range`, `cand_frags`) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` backend (`build`, `probe_peak`, `candidate_range`), the default matcher | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | `load_ms2` / `load_ms1` / `Ms1Scan` scan loaders | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `ExtractConfig`, `GateMode`, `PeakClaim`, `MatcherKind` | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `ISOTOPE_SPACING`, `ppm_bounds` | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::PSMS_EXTRACTED`, `artifact::CHROMATOGRAMS` schema ids | + +Entry point: `stages::extract::run(ExtractParams)` (`extract.rs:550`), wired from +the CLI in `main.rs:492` (`Cmd::Extract`) and from the orchestrator in +`stages/run.rs:224`. + +## Inputs and outputs + +### Inputs (`ExtractParams`, `extract.rs:61`) + +- `ms2` (Parquet): converted MS2 scans, loaded via `load_ms2` (`extract.rs:586`). + Each `Ms2Scan` carries `rt_seconds`, an isolation `window` (`lower_mz`, + `upper_mz`), and centroided `peaks` (`mz`, `intensity`). +- `library_precursors` + `library_fragments` (Parquet): loaded once into + `Library::load` (`extract.rs:552`) at `cfg.bucket_size`. +- `run_windows` (Parquet): per-candidate RT windows, columns `candidate_id`, + `rt_pred_cal`, `rt_lo`, `rt_hi` (read at `extract.rs:568`, scattered into the + dense `rt_lo`/`rt_hi`/`rt_cal` arrays indexed by `candidate_id`, + `extract.rs:574`). Candidates with no window row keep `[-inf, +inf]`. +- `ms1` (optional Parquet): MS1 scans via `load_ms1` (`extract.rs:587`). When + absent, all MS1 columns are null. +- `mass_cal` (optional JSON, the seed's `.masscal.json`): reads + `frag_ppm_offset` and `frag_tol_ppm` (`extract.rs:621`). Missing file = + offset 0 and `cfg.frag_tol_ppm`. +- `restrict_candidates` (optional prior `psms.parquet`): a `candidate_id` + allowlist (`extract.rs:557`) for the "gate first, then compete" workflow. + +### Output: `psms_extracted` (`out_psms`, schema `psms_extracted` v1) + +Column order and types from `extract.rs:1358`: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | Library candidate id (sorted ascending in the file) | +| `apex_rt` | F64 | Selected apex RT (seconds) | +| `apex_im` | OptF64 | Always null (3D MVP) | +| `apex_intensity` | F32 | Full summed intensity of the apex scan group | +| `n_matched_fragments` | I32 | Distinct matched predicted fragments | +| `n_predicted_fragments` | I32 | Predicted fragment count for the candidate | +| `coelution_run` | I32 | Longest consecutive-scan co-elution run | +| `rt_pred_cal` | F64 | Calibrated predicted RT for this candidate | +| `precursor_mz` | F64 | Candidate precursor m/z | +| `charge` | I32 | Precursor charge | +| `label` | Str | `"target"` or `"decoy"` | +| `base_peptide_id` | U32 | Stripped-peptide id (for competition grouping) | +| `peptidoform` | Str | ProForma-lite string | +| `protein` | Str | Protein accession | +| `predicted_irt` | F32 | Library iRT | +| `contested_frac` | F64 | Intensity fraction lost to a better co-eluter (0 when two-pass off) | +| `ms1_isom1` | OptF64 | MS1 intensity at (mono - spacing/z) | +| `ms1_mono` | OptF64 | MS1 monoisotopic intensity | +| `ms1_iso1` | OptF64 | MS1 +1 isotope intensity | +| `ms1_iso2` | OptF64 | MS1 +2 isotope intensity | + +Conditional columns (default-off, added only when the knob is set so the +production schema stays byte-identical): +- `emit_contested_features` -> `contested_count_frac` F64, `apportioned_frac` F64 (`extract.rs:1382`). +- `emit_gate_diagnostics` -> `gate_apex`, `gate_peak_spectral`, `gate_coelution`, `gate_spectral_entropy` (all F32, `extract.rs:1388`). + +### Output: `chromatograms` (`out_chrom`, schema `chromatograms` v1) + +Column order from `extract.rs:1396`: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | Candidate id | +| `frag_name` | Str | Fragment name (`b3`, `y7`, or `ms1_mono`/`ms1_iso1`/`ms1_iso2`) | +| `frag_mz` | F64 | Theoretical fragment m/z (or precursor isotope m/z for MS1 rows) | +| `frag_obs_mz` | F64 | Intensity-weighted observed m/z (falls back to theoretical) | +| `predicted_intensity` | F32 | Library predicted intensity (0.0 for MS1 rows) | +| `rt` | LargeListF32 | Per-scan RT axis of the trace | +| `intensity` | LargeListF32 | Per-scan intensity, 0-filled on the window grid | + +One row is emitted for **every** predicted transition (`extract.rs:1191`), so +`features` sees the full predicted set and can penalize a missing strong ion. A +never-observed fragment carries an **empty** trace (not a grid-length zero +vector) to avoid bloating the list column (`extract.rs:1213`). `rt`/`intensity` +are `LargeList` (64-bit offsets) because the total list-value count can exceed +the ~2.1B 32-bit `ListArray` offset ceiling when gates are opened wide +(`extract.rs:1404`). + +### Output: top-K peaks sidecar (`.peaks.parquet`) + +Written only when `retain_top_peaks > 1` (`extract.rs:1414`), one row per +(candidate, peak). Columns: `candidate_id` U32, `peak_rank` I32, `apex_rt` F64, +`start_rt` F64, `end_rt` F64, `evidence_count` F64, `area` F64. **These peaks are +not scored** and do not affect FDR; they are candidate peaks for an offline +peak-selection model (see below). + +Each artifact also gets an `.report.json` (`extract.rs:1439`) recording +row count, content hash, the effective/nominal fragment tolerance, and the gate +parameters. + +## How it works + +The control flow of `run` (`extract.rs:550`) is: load library and windows, load +scans, build the matcher, accumulate peak-candidate hits (peak-major), then per +candidate run the acceptance cascade and apex selection, emit chromatograms and +MS1 XICs, and write the tables. + +### 1. Matcher and mass recalibration + +The fragment tolerance and a systematic ppm offset come from `mass_cal` +(`extract.rs:621`). The offset is applied as a divisor `offset_factor = 1 + +offset*1e-6` (`extract.rs:634`); every observed peak m/z is corrected by +`q_mz = peak.mz / offset_factor` before probing (`extract.rs:676`). + +Two matcher backends dispatch through `probe_matched` (`extract.rs:43`): +- `MatcherKind::Fragindex` (default): builds a `FragIndex` once at the learned + tolerance (`extract.rs:639`). `FragIndex::probe_peak` (`fragindex.rs:152`) + probes bins `bin-1 ..= bin+1`, verifies each posting with the exact f64 ppm + predicate, and carries the **true generating fragment ordinal** in `post_frag`. +- Bucketed fallback: `Library::page_search` (`index.rs:242`) resolves the + fragment ordinal by nearest stored m/z via `Library::local_frag_index` + (`index.rs:217`). This is a semantic difference for fragments at + sub-f32-identical m/z (`extract.rs:38`). + +`Library::candidate_range` / `FragIndex::candidate_range` (`index.rs:233`, +`fragindex.rs:137`) give the half-open candidate-id range `[lo, hi)` whose +precursor m/z falls in an isolation window, exploiting that the library is sorted +by precursor m/z. This is what makes a per-window probe cheap. + +### 2. Peak-major accumulation + +The accumulator `acc: HashMap>` (`extract.rs:643`) maps each +candidate to the observed hits it collected. A `Hit` (`extract.rs:86`) is +`{rt, frag, inten, obs_mz}`. Entries are created lazily on the first collision. + +There are three accumulation paths: + +- **Parallel per-window, single-pass** (`extract_accumulate_windows`, + `extract.rs:257`): used when `fidx` is present and there is no `restrict` list + and the path is not two-pass. Scans are grouped by isolation window (each scan + belongs to exactly one window), and the ~150 windows are processed in parallel + with rayon. It is bit-identical to the serial loop because the per-candidate + cascade rt-sorts hits before summing, and same-rt hits for a candidate all come + from one window (`extract.rs:249`). +- **Serial single-pass** (`extract.rs:668`): the fallback when there is a + `restrict` allowlist or no fragindex. It honors every non-co-elution + `peak_claim` strategy. +- **Two-pass co-elution** (`extract_twopass_windows`, `extract.rs:360`): used + when `peak_claim` is one of the `Coelution*` variants **or** + `emit_contested_features` is set (`extract.rs:651`). Pass 1 builds each + candidate's per-scan elution profile (summed matched intensity). Pass 2 + arbitrates each shared peak to the claimant most eluting at that scan (highest + profile height at `rt`, `extract.rs:463`), tracks won/lost/apportioned + contested intensity in `Contested` (`extract.rs:101`), and reassigns intensity + per the co-elution claim variant. + +Per-peak claim strategies (`PeakClaim`, applied in the single-pass loop at +`extract.rs:700` and mirrored in the parallel path): +- `None`: every matching candidate gets the full peak intensity (legacy default). +- `WinnerPredictedIntensity`: only the claimant with the highest predicted + intensity keeps the peak; ties break by lowest `candidate_id` for determinism + (`extract.rs:706`). +- `Proportional`: split the peak by predicted-intensity share (`extract.rs:713`). +- `CoelutionWinner` / `CoelutionProportional` / `CoelutionWinnerMargin`: two-pass + variants keyed on elution-profile height, arbitrated in `extract.rs:501`. + `CoelutionWinnerMargin` only strips a peak from the runner-up when the top + eluter dominates by `peak_claim_margin` (`extract.rs:483`); otherwise the peak + stays shared, avoiding stripping real peptides at ambiguous peaks. + +### 3. Per-candidate cascade (parallel over candidates) + +`acc` is drained into `(cid, hits)` in sorted `candidate_id` order +(`extract.rs:800`, `cand_hits` at `extract.rs:812`) for determinism, then each +candidate is processed in parallel via `into_par_iter().map(...)` returning +`Option` (`extract.rs:857`). Each candidate's work depends only on its +own hits plus read-only library/window/MS1 data, so `collect()` in the sorted +order reproduces the serial push order byte-for-byte (`extract.rs:805`). + +The cheap-to-expensive acceptance cascade, in order: + +1. **Distinct-fragment presence (tier b)**: distinct matched fragments must be at + least `presence_min_matched` (`extract.rs:864`), else the candidate is dropped + before any grouping work. +2. **Scan grouping**: hits are rt-sorted and grouped into scan groups + `Vec<(rt, BTreeMap)>`, deduping the same fragment within one + scan by max (`extract.rs:869`). The `BTreeMap` fixes per-scan fragment order so + the f32 apex sum is deterministic. +3. **Acquisition-scan grid projection** (`extract.rs:896`): when + `emit_window_grid` is on, the sparse groups are projected onto the full set of + covering-window scans inside the RT window, so missing acquisition scans count + as 0 and break a co-elution run rather than being invisible. +4. **Apex selection** (see below). +5. **Co-elution run**: the longest run of consecutive scan groups with at least + `presence_min_coelution` fragments present (`extract.rs:1013`). +6. **Acceptance (tier c)** (`extract.rs:1028`): reject unless distinct fragments + >= `presence_min_fragments`, `best_run >= scan_window` (the + `fixed_scan_window` floor), `best_run >= min_coelution_run`, and + `matched_fraction >= min_matched_fraction`. `matched_fraction` is + `distinct / n_predicted` (`extract.rs:1027`); it is the primary symmetric + discriminator (real peptides match a large fraction, chimeric and decoy matches + a small fraction alike, keeping the target-decoy null valid). +7. **MS1 isotope evidence** (`extract.rs:1041`) computed *before* the Pearson gate + so it can rescue a candidate. `ms1_support` requires a present mono and a +1/mono + ratio in `[0.1, 1.5]` (`extract.rs:1058`). +8. **Pearson gate (tier d, optional)** (`extract.rs:1102`): only when + `min_frag_corr > 0.0`. It thresholds the score of the **active** `GateMode` + and only the active score is computed (the closures at `extract.rs:1087` are + lazy). If `ms1_rescue` is set, a gate failure is overridden when the candidate + has MS1 support and enough matched fragments (`extract.rs:1118`). + +### 4. Apex selection (`extract.rs:942`) + +The apex is chosen among scan groups whose smoothed distinct-fragment count +qualifies, then by a per-scan score: + +- **Rolling-window count** (`apex_count_window`): a centered rolling **sum** of + the per-scan distinct-fragment count (`extract.rs:945`). It is deliberately a + sum, not a mean: edge truncation makes interior positions accumulate more, + center-weighting the apex toward the RT-window centre (a mild RT prior). Window + 1 reproduces exact per-scan counts. Only scans with smoothed count `>= maxc - + apex_count_tol` qualify (`extract.rs:957`). +- **Signature ions** (`apex_top_fragments`, default 3 via `k_sig`, + `extract.rs:969`): the top-K predicted fragments, so a bright interferent on a + non-signature ion cannot define the apex. +- **RT prior** (`apex_rt_prior_s`): when > 0 and `rt_cal > 0`, each qualifying + scan's score is multiplied by `exp(-0.5*((rt - rt_cal)/sigma)^2)` + (`extract.rs:987`). +- **Scoring mode** (`apex_evidence_rank`, `extract.rs:992`): when true, the score + is `n_frag + sig_sum/(sig_sum+1)` times the prior, so the number of distinct + co-eluting predicted fragments dominates and observed signature intensity only + breaks sub-integer ties. This is interference-resistant in wide-window DIA + because intensity is chimeric. When false (default), the legacy + `sig_sum * prior` is used, bit-identical to the pre-feature behaviour. + +`apex_intensity` reported is the full summed intensity of the winning scan group +(`extract.rs:1009`), not the signature-only score. + +### 5. Gate scoring internals + +Spectral-agreement scores share a helper `peak_window` (`extract.rs:156`) that +finds the contiguous elution-peak scan range `[lo, hi]` around the signature-ion +apex (scans above 10% of the reference apex height), returning `None` when there +are fewer than 3 scans or no reference signal. This restriction matters: over the +full wide extraction window the traces are mostly zeros and any correlation is +noise, so the co-elution and spectral gates are only meaningful across the +elution peak itself (`extract.rs:151`). `GateMode` scores: +- `ApexPearson`: Pearson of observed-vs-predicted intensities at the single apex + scan (`extract.rs:1087`); one chimeric scan can dominate it. +- `PeakSpectral`: `peak_spectral_score` (`extract.rs:192`) correlates the + peak-summed observed spectrum (each fragment integrated over the peak scans) + with predicted intensities, averaging out a single interfered scan. +- `SpectralEntropy`: Li spectral-entropy similarity of the sqrt-transformed apex + spectrum via the shared `features::entropy` kernel (`extract.rs:1093`). +- `Coelution`: `coelution_gate_score` (`extract.rs:218`), the + predicted-intensity-weighted mean Pearson of each matched fragment's XIC to the + signature-ion reference profile, restricted to the elution peak. Orthogonal to + intensity agreement (temporal, not shape). +- `Combined`: requires `peak_spectral >= min_frag_corr` **and** `coelution >= + gate_coelution_min` (`extract.rs:1113`). + +All four diagnostic scores are computed for every accepted candidate only when +`emit_gate_diagnostics` is set (`extract.rs:1132`); otherwise they are zeroed and +not written, so the default chain pays no extra cost. + +### 6. Chromatograms and MS1 XICs + +Per-fragment observed m/z is an intensity-weighted mean per fragment +(`extract.rs:1159`) for mass-accuracy features. Every predicted transition emits a +chromatogram row (`extract.rs:1191`); observed fragments carry the grid-sampled +(or rt-sorted) trace, absent ones an empty trace. When MS1 is present and grid +mode is on, three MS1 isotope XICs (`ms1_mono`, `ms1_iso1`, `ms1_iso2`) are +sampled on the same scan grid via nearest-MS1-scan lookup (`extract.rs:1222`), +using `ISOTOPE_SPACING / charge` (`constants.rs:22`, value `1.003354835`) for the +isotope offsets and `sum_near` (`extract.rs:128`) to integrate within +`prec_tol_ppm`. The apex MS1 isotope intensities (`ms1_isom1`/mono/`iso1`/`iso2`) +are the per-PSM columns, taken from the nearest MS1 scan to the apex RT +(`extract.rs:1041`). + +### 7. Top-K peak enumeration (`retain_top_peaks`) + +When `retain_top_peaks > 1` (`extract.rs:1245`), the per-scan distinct-fragment +count profile is passed to `crate::peaks::enumerate_peaks` (`peaks.rs:52`) with +`bound_fraction = 1/3` and `min_prominence_frac = 0.1`. `enumerate_peaks` finds +local maxima above a prominence floor, walks fractional-height boundaries +(stopping below `bound_fraction * apex` or at a valley, `peaks.rs:88`), +deduplicates maxima falling inside a stronger peak's envelope (`peaks.rs:136`), +and returns up to K groups ranked strongest-first by integrated `area` with ties +broken by earliest `apex_idx`. `enumerate_peaks(.., k=1, ..)` returns the single +global-argmax peak, so callers can adopt it incrementally. The retained peaks +rank by **co-eluting fragment breadth (count), not intensity**, per the finding +that intensity is chimeric in DIA. The main PSM still reports the single selected +apex, so FDR is unaffected; the sidecar peaks are unscored candidate peaks for an +offline peak-selection model (`extract.rs:1239`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `run` | `extract.rs:550` | Stage entry point; orchestrates load, accumulate, cascade, write | +| `ExtractParams` | `extract.rs:61` | Input path bundle + config + config hash | +| `Hit` | `extract.rs:86` | One observed hit: `rt`, `frag`, `inten`, `obs_mz` | +| `Contested` | `extract.rs:101` | Per-candidate won/lost/apportioned contested intensity (two-pass) | +| `CandOut` | `extract.rs:815` | Per-candidate parallel-map result (PSM row + chrom rows + peaks) | +| `probe_matched` | `extract.rs:43` | Dispatch a peak probe to fragindex or bucketed backend | +| `extract_accumulate_windows` | `extract.rs:257` | Parallel single-pass per-window accumulation | +| `extract_twopass_windows` | `extract.rs:360` | Parallel two-pass co-elution arbitration + contested stats | +| `peak_window` | `extract.rs:156` | Elution-peak scan range around the signature apex (10% height) | +| `peak_spectral_score` | `extract.rs:192` | Peak-integrated observed-vs-predicted Pearson | +| `coelution_gate_score` | `extract.rs:218` | Weighted mean XIC-to-reference co-elution Pearson | +| `nearest_index` | `extract.rs:111` | Binary search for nearest RT in a sorted array | +| `sum_near` | `extract.rs:128` | Sum intensities within a ppm window (m/z-sorted) | +| `enumerate_peaks` | `peaks.rs:52` | Pure top-K peak-group enumerator | +| `PeakGroup` | `peaks.rs:22` | One enumerated peak (apex/start/end idx, apex intensity, area, rank) | +| `Library::candidate_range` | `index.rs:233` | Candidate-id range for an isolation window | +| `Library::cand_frags` | `index.rs:208` | Fragment m/z, predicted intensity, name slices | +| `FragIndex::probe_peak` | `fragindex.rs:152` | Verified postings for one observed peak in a candidate range | + +## Configuration + +All fields live in `ExtractConfig` (`config.rs:431`); defaults in +`config.rs:548`. The config was recently pruned of dead fields in the +"clean-slate declutter" commit: `extract.scan_window_mode` (and the +`ScanWindowMode` enum), `extract.scan_scale`, `extract.k_select`, +`extract.max_fragment_charge`, and `extract.tolerance_regime` were removed. Do +not reintroduce them. + +| Field | Default | Effect | +|---|---|---| +| `fixed_scan_window` | 3 | Minimum co-elution run length (`scan_window` floor); `.max(1)` at `extract.rs:763` | +| `frag_tol_ppm` | 20.0 | Fragment match tolerance (overridden by `mass_cal`) | +| `prec_tol_ppm` | 20.0 | MS1 isotope integration tolerance | +| `presence_min_matched` | 3 | Tier-b: minimum distinct matched fragments (`extract.rs:864`) | +| `presence_min_fragments` | 3 | Acceptance: minimum distinct fragments (`extract.rs:1028`) | +| `presence_min_coelution` | 2 | Min simultaneously-present fragments to extend a run (`extract.rs:1017`) | +| `min_frag_corr` | **0.2** | Pearson/gate threshold; 0 disables the gate. Relaxed from a historical 0.5 to recover low-abundance candidates (`config.rs:559`) | +| `min_matched_fraction` | 0.0 | Acceptance: min matched/predicted fraction (default off) | +| `apex_top_fragments` | 0 | Signature-ion count for apex; 0 -> default 3 (`extract.rs:969`) | +| `apex_rt_prior_s` | 0.0 | Gaussian RT-prior sigma on apex tiebreak; 0 = off | +| `apex_count_tol` | 1 | Count slack for qualifying apex scans | +| `apex_count_window` | 1 | Rolling-sum width for the count profile; 1 = no smoothing. Window 5 cut AIF apex misassignment (median \|dRT\| 131s -> 9s) | +| `apex_evidence_rank` | false | Breadth-of-evidence apex vs legacy signature-intensity apex | +| `emit_window_grid` | true | Zero-filled full-window-grid chromatograms | +| `bucket_size` | 8192 | m/z bucket size (power of two) | +| `peak_claim` | `None` | Shared-peak apportionment strategy (`PeakClaim`) | +| `peak_claim_margin` | 2.0 | Dominance factor for `CoelutionWinnerMargin` | +| `emit_contested_features` | false | Adds `contested_count_frac`/`apportioned_frac`; forces the two-pass path | +| `matcher` | `Fragindex` | Fragment-matcher backend | +| `min_coelution_run` | 0 | Extra co-elution-run floor (0 = off; `scan_window` still applies) | +| `ms1_rescue` | false | Rescue Pearson-gate failures with MS1 isotope support | +| `retain_top_peaks` | 1 | K>1 writes the `.peaks.parquet` sidecar (unscored) | +| `emit_candidate_audit` | false | Candidate-audit sidecar (diagnostic) | +| `emit_gate_diagnostics` | false | Adds the four `gate_*` diagnostic columns | +| `gate_mode` | `ApexPearson` | Which score `min_frag_corr` thresholds (`GateMode`) | +| `gate_coelution_min` | 0.5 | Second threshold for `GateMode::Combined` | + +Note: `emit_candidate_audit` is a declared knob but the candidate-audit write is +not present in this `extract.rs`; the audit ladder is produced by the separate +`mumdia audit` command. Treat the in-stage audit as unwired here. + +## Invariants, determinism, gotchas + +- **Determinism**: output is emitted in ascending `candidate_id` order + (`extract.rs:800`); the parallel per-candidate map preserves that order via + `collect()`. Per-scan fragment maps are `BTreeMap` so f32 apex sums have a fixed + addition order (`extract.rs:872`). The parallel window accumulation is documented + as bit-identical to the serial loop (`extract.rs:249`, `extract.rs:659`). A + HashMap f32 sum shifting the apex once broke reproducibility; keep ordered maps + and sorted iteration wherever floats are summed. +- **Default-off contract**: `retain_top_peaks=1`, `apex_evidence_rank=false`, + `emit_contested_features=false`, `emit_gate_diagnostics=false`, `peak_claim=None` + make the schema and per-candidate compute byte-identical to the production chain. + Every sensitivity knob added here must keep that property. +- **The gate optimum depends on the rescorer**, not the gate in isolation. The + full-feature search found `spectral_entropy_similarity_sqrt` the single best + target/decoy discriminator (AUC 0.826), yet gating on it *regressed* + end-to-end identifications versus the apex gate, because gating on the + rescorer's own best feature enriches hard decoys. "Best discriminator" is the + wrong criterion for a gate; the lever is the rescorer. This is why every gate + change must pass an entrapment-holdout gate before being enabled by default. +- **Mass recal is a divisor, not a subtraction**: observed m/z is corrected by + `q_mz = peak.mz / (1 + offset*1e-6)` (`extract.rs:676`); a missing/absent + `masscal.json` yields offset 0 and the config tolerance. +- **Empty vs zero traces**: a never-observed predicted fragment carries an empty + trace, not a grid-length zero vector (`extract.rs:1213`). Downstream code must + treat an empty trace as `obs_apex = 0`. +- **`apex_im` is always null** (3D MVP); do not assume an IM value. +- **`restrict_candidates` routes to the serial path** (`extract.rs:660`), which is + slower but honors the allowlist filter and every `peak_claim` strategy. +- **`peaks.parquet` is not scored** and never enters FDR; it is a research sidecar. +- Chromatogram list columns are `LargeListF32` on purpose (`extract.rs:1404`); do + not downgrade to 32-bit `ListF32` or wide-open gates overflow the offset buffer. + +## How to extend / modify + +- **A new gate metric**: add a variant to `GateMode` (`config.rs:591`), add a lazy + score closure next to `apex_pearson`/`peak_spec`/`coel` (`extract.rs:1087`), and + a match arm in the acceptance gate (`extract.rs:1108`). If it is worth + diagnosing, also wire it into the `emit_gate_diagnostics` tuple + (`extract.rs:1132`) and the conditional column block (`extract.rs:1388`). Default + the gate off and validate against entrapment before enabling. +- **A new peak-claim strategy**: add a `PeakClaim` variant (`config.rs:164`); if it + needs elution profiles, extend the two-pass trigger (`extract.rs:651`) and the + reassignment match in `extract_twopass_windows` (`extract.rs:501`); otherwise add + it to the single-pass match (`extract.rs:700`) and the parallel accumulation + (`extract.rs:306`). Keep tie-breaks deterministic (lowest `candidate_id`). +- **Scoring the top-K peaks**: the sidecar peaks are currently unscored. To close + the loop, per-peak feature computation must be added (each retained peak needs + the full per-peak feature vector `features` computes for the selected apex), + then an out-of-fold peak-selection model chooses. This is the open + `retain_top_peaks` work item in the sensitivity plan. +- **New per-PSM columns**: append to the `CandOut` struct (`extract.rs:815`), set + it in the `Some(CandOut { .. })` block (`extract.rs:1265`), push it in the serial + append loop (`extract.rs:1305`), and add the `Col` in the `psms_cols` vector + (`extract.rs:1358`). Gate any non-production column behind an `emit_*` flag to + preserve the byte-identical default schema, and bump the schema version in + `schema.rs` if the default schema changes. +- **IM / 4D**: `apex_im` and the IM data-model hooks exist but are unfilled; a + diaPASEF extension adds an IM window post-filter alongside the RT window and IM + apex/feature families. It cannot be validated without diaPASEF data. diff --git a/docs/10_features.md b/docs/10_features.md new file mode 100644 index 0000000..5129dde --- /dev/null +++ b/docs/10_features.md @@ -0,0 +1,607 @@ +# features (Stage E): the feature battery + PIN + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage E (`mumdia features`) turns one extracted PSM (apex-level identification +plus its per-fragment chromatograms) into a fixed, named, versioned feature +vector for the semi-supervised rescorer. It computes a config-selected feature +set, a scalar `prelim_score` used by `compete`, a Percolator PIN, and a +`blake3` schema id that pins the ordered column list so the classifier is never +trained or applied under a mismatched feature layout. + +The stage is pure per-PSM: each row's features are a function of that row's own +inputs only, with two cross-row exceptions computed up front (charge-state +corroboration grouped by peptidoform, and a global elution half-width learned +from the confident seed set). This makes the heavy per-PSM work embarrassingly +parallel (`rayon`) while keeping byte-identical output to a serial run. + +Everything is driven by `FeaturesConfig` (`mumdia-core/src/config.rs:618`). The +three feature sets are `Minimal` (14), `Rich` (44), and `Extended` (381). The +default is `Minimal`; the tuned `--profile dia` preset and the DIA-NN-library +recipe use `Extended`. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/features.rs` | stage entry point, `run`, `Evidence`, `build_evidence`, `fragment_features`, boundary detection, `prelim_score`, PIN, schema hash, the Minimal/Rich column lists, and the Extended family registry | +| `rust/mumdia/crates/mumdia/src/stages/features/similarity.rs` | Extended family: observed-vs-library intensity agreement kernels (64 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/entropy.rs` | Extended family: spectral-entropy / information-divergence (18 names); also exports the gate kernel `spectral_entropy_similarity_sqrt` | +| `rust/mumdia/crates/mumdia/src/stages/features/coelution.rs` | Extended family: fragment-vs-reference and pairwise co-elution + cross-correlation (38 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/interference.rs` | Extended family: co-isolation / chimera detection, interference removal, rank decomposition (26 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs` | Extended family: peak-shape descriptors of the reference profile and fragments (43 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs` | Extended family: fragment ppm distribution + positive mass evidence (17 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs` | Extended family: b/y series coverage, runs, complementarity, per-series similarity (34 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/ms1.rs` | Extended family: precursor isotope-envelope agreement + MS1/MS2 XIC co-elution (25 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/rt.rs` | Extended family: RT-agreement variants (13 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/novel.rs` | Extended family: seed corroboration + precursor/charge metadata (12 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs` | Extended family: zero-ignoring apex/co-elution variants (12 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs` | Extended family: prediction-free MS2-XIC rank stability (8 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/peak_scans.rs` | Extended family: peak-scan count / window-degeneracy indicator (2 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs` | Extended family: fragment apex dispersion + consensus peak shape (13 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs` | Extended family: fragment mass-error dispersion + evidence breadth (10 names) | +| `rust/mumdia/crates/mumdia/src/stats.rs` | shared `pearson`/`cosine`/`spectral_angle` kernels used by every family | +| `rust/mumdia/crates/mumdia-core/src/config.rs:618` | `FeaturesConfig` | + +## Inputs and outputs + +`FeaturesParams` (`features.rs:503`) names the paths: `psms` (extracted PSMs), +`chromatograms`, optional `seed` (seed PSMs), `out` (features Parquet), +`out_pin` (PIN), `cfg`, `config_hash`. + +### Consumed: psms_extracted (`features.rs:516`-`540`) + +Columns read (getter -> column): `candidate_id` (u32), `apex_rt` (f64), +`apex_intensity` (f32), `n_matched_fragments` (i32), `n_predicted_fragments` +(i32, optional; defaults to 6 when absent), `coelution_run` (i32), +`rt_pred_cal` (f64), `charge` (i32), `label` (str), `base_peptide_id` (u32), +`peptidoform` (str), `protein` (str), `precursor_mz` (f64). Optional soft- +competition columns default to 0.0 when absent: `contested_frac`, +`contested_count_frac`, `apportioned_frac`. Optional MS1 apex isotope columns +(`opt_f64`, default `None`): `ms1_isom1`, `ms1_mono`, `ms1_iso1`, `ms1_iso2`. + +### Consumed: chromatograms (`features.rs:543`-`570`) + +`candidate_id` (u32), `frag_name` (str), `frag_mz` (f64), `frag_obs_mz` (f64, +optional; falls back to `frag_mz`), `predicted_intensity` (f32), `rt` +(list), `intensity` (list). Rows whose `frag_name` starts with +`ms1_` are routed to a separate `ms1x` map and fed to the MS1 XIC evidence; +all others are the fragment chromatograms. + +### Consumed: seed PSMs (optional, `features.rs:576`-`600`) + +`candidate_id` (u32), `score` (f64), `spectrum_q` (f64), `label` (str). Builds +the `candidate_id -> seed score` and `-> identified flag` maps, and the +confident-target set (`spectrum_q <= 0.01` and `label == "target"`) used for +`bound_from_confident`. + +### Produced: features Parquet (`features.rs:827`-`843`) + +Bookkeeping columns, in order: `candidate_id` (u32), `label` (str), +`base_peptide_id` (u32), `peptidoform` (str), `protein` (str), `apex_rt` (f64), +`elution_lo` (f64), `elution_hi` (f64), `precursor_mz` (f64), `prelim_score` +(f64). Then one `F64` column per name in `active_features(cfg.set)`, in order. +`elution_lo`/`elution_hi` are the RT bounds the stage actually used (emitted so +downstream and plotting read them rather than re-derive). + +### Produced: companion + PIN + report + +- `.schema.json` (`features.rs:846`): `FeatureSchema { feature_columns, + schema_id }`, read back by `FeatureSchema::read` (`features.rs:241`). +- ``: Percolator PIN, header + `SpecId Label ScanNr ExpMass CalcMass Peptide Proteins` + (`write_pin`, `features.rs:1388`). +- `.report.json` (`ArtifactReport`, `features.rs:866`): logical name + `features`, schema version 1, `stats` carrying `feature_schema_id`, + `n_features`, and `set`. + +## How it works + +Control flow of `run` (`features.rs:514`): + +1. Read the PSM table and pull the scalar columns (`features.rs:516`-`540`). +2. Read chromatograms and group `ChromRow`s by `candidate_id` into `chrom`, + splitting off `ms1_*` rows into `ms1x` (`features.rs:543`-`570`). A + `ChromRow` (`features.rs:264`) holds `frag_name`, `frag_mz`, `frag_obs_mz`, + `pred_int`, and the `rt`/`inten` vectors. +3. Build the seed maps and confident-target set from the optional seed table + (`features.rs:576`-`600`). +4. If `bound_from_confident` is set, learn a global elution half-width + (see "bound_from_confident" below), producing `Option<(L, R)>` in seconds + (`features.rs:606`-`650`). +5. Compute the gradient as `max(apex_rt).max(1.0)` (`features.rs:652`); this is + the run length used to normalize RT errors. +6. Compute the three cross-charge corroboration columns by grouping rows by + `peptidoform` (`features.rs:662`-`674`). +7. In parallel over rows (`features.rs:697`-`742`), compute the two expensive + per-PSM pieces: `fragment_features` (the Minimal/Rich fragment battery) and, + when Extended, `build_evidence` + `extended_values`. Results collect into a + `Vec` indexed by row, preserving order. +8. Serially assemble `fmap` (name -> per-row value vector), `prelim`, and the + elution bounds (`features.rs:744`-`819`). The serial loop reads `per[i]` and + pushes each named value with the `push` closure (`features.rs:678`). +9. Insert the three cross-charge columns (`features.rs:822`-`824`). +10. Build the output columns (bookkeeping + `active_features`), write Parquet, + write the schema JSON, build the feature matrix, write the PIN, and emit the + report (`features.rs:826`-`878`). + +### Fragment features (`fragment_features`, `features.rs:1078`) + +For each fragment it takes the observed intensity at the scan nearest the apex +RT (`obs`), the predicted intensity (`pred`), and `|ppm|`. It then computes: +`frag_corr`=`pearson(obs,pred)`, `frag_cosine`=`cosine(obs,pred)`, +`spectral_angle`; sum-normalized L1 (`norm_manhattan`) and `rmsd`; intensity- +weighted and unweighted mean `|ppm|`; b/y intensity sums and counts. + +It then aligns all fragment traces on the union RT axis (`axis_full`), restricts +to the elution peak (`lo_i..=hi_i`) so co-elution/profile features are not +diluted over the whole `+/- w_rt` window, and computes pairwise Pearson +statistics (`coelution_mean`, `coelution_best`, `n_coelution_above`), lag- +optimized cross-correlation (`xcorr_coelution` mean absolute lag, +`xcorr_shape`), the DIA-NN-style profile block (`profile_cos` = elution^2- +weighted spectral cosine, `ref_corr` = mean fragment-vs-reference Pearson, +`best_ref_corr`, `low_frag_coel`), and the interference-correction block +(`evidence` = summed fragment-vs-reference correlations, `contrast_min`, +`resid_corr`, `coel_clean`, `shadow_frac` from the `1.5*r*ref` cap). Finally +`log_sn` from apex vs median trace point. `elution_lo`/`elution_hi`/ +`base_width_rt`/`n_observations` come from the peak-bounded axis. + +### Elution-boundary detection + +`peak_bounds` (`features.rs:946`) descends from the apex-nearest scan while the +smoothed profile stays `>= frac * apex_height`, bridging up to `grace` +consecutive sub-threshold scans. If the supplied apex sits at zero height it is +relocated to the global maximum first (`features.rs:955`), so a zero-height apex +does not collapse the window. The reference profile for boundary finding is the +`smooth3` (`features.rs:925`) of the summed top-3-predicted-intensity fragment +XICs. + +### prelim_score (`features.rs:815`) + +``` +prelim = n_matched * (0.5 + max(0, frag_corr)) + + max(0, coelution_mean) + + 0.1 * ln(1 + apex_intensity) + - rt_err / gradient +``` + +A cheap heuristic (not a trained score) that rewards matched-fragment count +scaled by spectral correlation, co-elution, and log intensity, penalized by +gradient-normalized RT error. `compete` uses it as the within-group ranking key. + +### The Evidence struct (`features.rs:278`) + +`build_evidence` (`features.rs:345`) constructs the per-PSM `Evidence` handed to +every Extended family. It mirrors the alignment and peak-bounding of +`fragment_features` so families see the same elution peak. Fields: + +- Time series: `axis` (RT seconds over the detected elution peak), `traces` + (per-fragment intensity over `axis`, zero-filled, fragment order), `axis_full` + / `traces_full` (whole extracted window), `apex_idx` (index of apex in + `axis`), `ref_profile` (predicted-intensity-weighted sum of the peak traces). +- Fragment-indexed arrays (shared order): `pred` (library intensity), + `obs_apex` (intensity at the apex scan; `> 0` defines "matched"), `is_b`, + `ordinal`, `frag_charge`, `frag_mz` (theoretical), `frag_obs_mz` (intensity- + weighted observed), `mass_err_ppm` (signed ppm). +- Scalars filled by the caller after build (`features.rs:718`-`732`): + `apex_rt`, `rt_pred_cal`, `rt_err`, `gradient`, `precursor_mz`, `charge`, + `seq_len`, `n_matched`, `n_predicted`, `seed_score`, `seed_identified`, + `apex_intensity`. +- MS1: `ms1_mono`/`ms1_iso1`/`ms1_iso2`/`ms1_isom1` (apex isotope intensities, + `None` when no MS1) and `ms1_xic` (the `[mono,+1,+2]` XICs resampled onto + `axis`; empty unless the extract stage persisted `ms1_*` chromatogram rows). + +`parse_ion` (`features.rs:331`) parses `b3`, `y7`, `b3^2` into +`(is_b, ordinal, charge)`. + +### The Extended battery + +`FAMILIES` (`features.rs:52`) is an ordered array of `(NAMES, values)` pairs; +the order is part of the frozen schema and is append-only. Each family exposes +`NAMES: &[&str]` and `values(&Evidence) -> Vec` of identical length and +matching order. `extended_values` (`features.rs:134`) calls each family and +applies the precomputed dedup plan (`extended_value_plan`, `features.rs:108`), +keeping names and values in lockstep, and coerces any non-finite value to 0.0. + +Deduplication (`extended_name_refs`, `features.rs:80`): a name that already +appears in `MINIMAL_FEATURES` or `RICH_EXTRA` (`reserved_names`, +`features.rs:72`), or that repeats across families, is kept only on first +appearance. In the current tree four Extended names are dropped as reserved +collisions: `spectral_angle` (similarity vs Minimal), `rt_error_abs` (rt vs +Minimal), `peptide_length` and `seed_identified` (novel vs Minimal/Rich). So the +335 raw family names reduce to 331 unique Extended names. + +## The three feature sets + +`active_features(set)` (`features.rs:206`) returns the ordered active column +list: + +- `Minimal` (14, `MINIMAL_FEATURES` `features.rs:154`): `rt_error_abs`, + `rt_error_rel`, `n_matched_fragments`, `coelution_run`, `log_apex_intensity`, + `frag_corr`, `frag_cosine`, `spectral_angle`, `coelution_mean`, + `coelution_best`, `n_coelution_above`, `charge`, `peptide_length`, + `n_proteins`. +- `Rich` (44 = Minimal + 30, `RICH_EXTRA` `features.rs:172`): adds + `library_norm_manhattan`, `library_rmsd`, `xcorr_coelution`, `xcorr_shape`, + `sum_b_intensity`, `sum_y_intensity`, `diff_by_intensity`, `n_b_ions`, + `n_y_ions`, `weighted_mass_error`, `mean_mass_error`, `isotope_corr`, + `ms1_isom1_ratio`, `log_mono_ms1`, `has_ms1`, `log_sn`, `n_observations`, + `base_width_rt`, `seed_score`, `seed_identified`, `matched_fraction`, + `profile_cos`, `ref_corr`, `best_ref_corr`, `low_frag_coel`, `evidence`, + `contrast_min`, `resid_corr`, `coel_clean`, `shadow_frac`. +- `Extended` (381 = Minimal + Rich + 331 family names + 6 psms-derived): appends + `extended_names()` then six columns computed outside the family registry. + Three co-elution peak-contest metrics (`peak_contested_frac`, + `peak_contested_count_frac`, `peak_apportioned_frac`) come from the PSM + columns (default 0.0 when absent) and separate peak-borrowing decoys from + genuine IDs. Three charge-corroboration columns (`n_charge_states`, + `charge_multi_flag`, `cross_charge_intensity_log`) aggregate across the charge + states of one peptidoform, an axis invisible to the per-PSM Evidence families. + +The size invariant is asserted in `feature_sets_sized` (`features.rs:1434`): +`Extended.len() == 14 + 30 + extended_names().len() + 6`, and all Extended names +are unique. `FeatureSet` (`config.rs:81`) has exactly `Minimal`, `Rich`, +`Extended`; the earlier `Custom` variant is gone (do not document it). + +## Extended family reference + +Each family below lists its name count and what it measures. All values are +finite (NaN/Inf coerced to 0.0), and every family returns a stable-length vector +even on degenerate evidence. + +### similarity (64, `similarity.rs`) + +Observed-vs-library fragment-intensity agreement under many kernels. `o` is the +per-fragment apex intensity, `l` the predicted intensity; `_matched` restricts +to `o_i > 0`, `_area` replaces `o` with the per-fragment peak-XIC trapezoid +area, `on`/`ln` are sum-normalized. Names: `spectrum_cosine_matched`, +`spectrum_cosine_sqrt`, `spectrum_cosine_log`, `spectral_angle` (dropped as +reserved), `spectral_angle_sqrt`, `spectral_angle_matched`, +`pearson_intensity_matched`, `pearson_intensity_log`, `spearman_intensity`, +`spearman_intensity_matched`, `kendall_tau_intensity`, `dot_product_raw`, +`dot_product_norm`, `library_recall_intensity`, `manhattan_sim`, +`manhattan_sqrt`, `rmsd_norm`, `mae_norm`, `mse_log`, `mae_weighted_pred`, +`abs_diff_q3`, `max_positive_residual`, `chebyshev_dist`, `minkowski_p3`, +`bray_curtis`, `bray_curtis_sqrt`, `canberra`, `canberra_matched`, +`wave_hedges`, `chi_square_pearson`, `chi_square_symmetric`, +`divergence_distance`, `bhattacharyya_coef`, `hellinger`, `squared_chord`, +`harmonic_mean_sim`, `jaccard_presence`, `dice_presence`, +`intensity_weighted_pearson`, `regression_slope`, `gini_diff`, `wasserstein_mz`, +`footrule_norm`, `rank_overlap_top3`, `top1_frag_match`, +`top1_predicted_observed`, `frac_top3_predicted_observed`, +`count_strong_predicted_absent`, `frac_predicted_absent`, `cosine_area`, +`pearson_area`, `spectral_angle_area`, `cosine_fullwindow`, +`stein_scott_weighted_dot`, and the unbounded/granular scores that address +cosine saturation (`log_dot_product`, `spectral_log_evidence`, `scribe_score`, +plus their `_area` twins), `cosine_high_ordinal` (drop b1/b2/y1/y2), and the +robust trimmed-cosine trajectory (`cosine_robust_trim1/2/3`). Local helpers +include tie-corrected `ranks`, `kendall_tau_b`, interpolated `quantile`, `gini`, +`weighted_pearson`, and `trapz`. + +### entropy (18, `entropy.rs`) + +Li spectral-entropy similarity and information divergences between sum- +normalized `o` and `l`. `entropy_sim` (`entropy.rs:91`) is +`1 - (2 H(m) - H(o) - H(l)) / ln 4` with `m = (o+l)/2`, clamped to [0,1]. +Names: `spectral_entropy_similarity`, `weighted_spectral_entropy_similarity` +(Li per-spectrum weighting), `spectral_entropy_similarity_sqrt`, +`spectral_entropy_similarity_topk` (top-6 by predicted intensity), +`spectral_entropy_similarity_area`, `jensen_shannon_divergence`, +`jeffreys_divergence`, `kl_obs_pred`, `kl_pred_obs`, `cross_entropy_obs_pred`, +`obs_spectrum_entropy`, `pred_spectrum_entropy`, `entropy_diff`, +`entropy_ratio`, `obs_normalized_entropy` (Pielou evenness), +`normalized_entropy_diff`, `residual_spectrum_entropy`, `entropy_weight_obs`. +The public `spectral_entropy_similarity_sqrt(obs, pred)` (`entropy.rs:82`) is +reused by the extraction gate `GateMode::SpectralEntropy` so the kernel is not +duplicated. + +### coelution (38, `coelution.rs`) + +Fragment co-elution against the predicted-weighted reference profile R and +pairwise between fragments. Three groups: per-fragment vs R (peak/full windows, +leave-one-out), pairwise Pearson statistics and lag cross-correlation +(`best_xcorr`, `MAXLAG=5`), and structured cross-correlations (b vs y ions, +charge-1 vs multiply-charged). Names include `frag_ref_corr_mean`, +`frag_ref_corr_obsweighted`, `frag_ref_corr_min`, `frag_ref_corr_std`, +`frag_ref_corr_sq_mean`, `frag_ref_corr_topk_weighted`, +`n_frag_ref_corr_above_0_9`, `frac_frag_ref_corr_above_0_8`, +`frag_ref_corr_mean_full`, `full_vs_peak_corr_gain`, +`pairwise_coelution_weighted`, `pairwise_coelution_min/median/std/frac_negative`, +`pairwise_coelution_hi/lo`, `coelution_hi_lo_contrast`, +`coelution_corr_entropy`, the `xcorr_shape_*` and `xcorr_lag_*` statistics +(`_mean/_min/_std/_mean_abs/_iqr/_frac_zero/_max_abs/_entropy`), +`ref_xcorr_lag_mean`, `ref_xcorr_shape_mean`, `observed_sum_vs_template_corr`, +`frag_loo_ref_corr_mean/_min`, `frac_frags_apex_aligned`, `top3_frag_ref_corr`, +`by_cross_coelution`, `by_cross_lag_mean`, `charge_cross_coelution`. Note the +JSON `coelution_weighted_mean` is an exact alias of `pairwise_coelution_weighted` +and is emitted once under the latter. + +### interference (26, `interference.rs`) + +Co-isolation/chimera detection via the least-squares scale +`r_f = / ` projecting each fragment onto R. Names: +`explained_variance_ref`, `profile_residual_fraction`, `n_interfered_fragments`, +`corrected_vs_raw_cos`, `corrected_vs_raw_ratio`, the iterative interference- +removal block (`ifs_removed_count`, `ifs_removed_intensity_frac`, +`ifs_corr_gain`, `ifs_retained_frac`, `matched_frac_after_ifs`), the peak-vs- +full area ratios (`peak_to_full_area_ratio_profile/_frag_mean/_weighted`, +`out_of_peak_intensity_frac`), `profile_corr_full_vs_peak_delta`, +`frac_frag_ref_corr_below_0_5`, `explained_apex_intensity_frac`, `apex_purity`, +`interference_apex_residual_fraction`, `dominant_frag_ref_corr`, the rank +decomposition of the matched fragment x time Gram matrix by power iteration +(`explained_variance_ratio`, `second_component_fraction`), competing-peak +descriptors on the full-window profile (`profile_second_peak_ratio`, +`n_competing_peaks_in_window`), `matched_pred_intensity_fraction`, and +`top_pred_frag_matched`. + +### chromatographic (43, `chromatographic.rs`) + +Peak-shape quality of R (Gaussian moment-match fit, EMG grid-fit comparison via +`erfc`) and of individual fragments. Names: `gaussian_fit_r2`, +`gaussian_cosine`, `emg_fit_improvement`, `apex_prominence`, `profile_peak_snr` +(MAD-based over out-of-peak scans), width descriptors (`fwhm_seconds`, +`fwhm_to_window_ratio`, `width_at_10pct`, `width_ratio_10_50`), +asymmetry/tailing (`hwhm_asymmetry`, `tailing_factor_usp`, +`asymmetry_factor_10pct`), apex shape (`apex_sharpness`, `apex_curvature`, +`apex_to_boundary_ratio`, `apex_dominance`), roughness (`zigzag_index`, +`jaggedness`, `roughness_2nd_deriv`), multimodality (`n_local_maxima`, +`modality`), RT moments (`rt_skewness`, `rt_excess_kurtosis`, `rt_std_seconds`, +`mean_mode_offset`), area descriptors (`fraction_area_within_fwhm`, +`triangle_area_similarity`), `baseline_fraction`, `peak_completeness`, +`apex_centering_offset`, `intensity_score`, `total_xic_log`, per-fragment +descriptors (`frag_fwhm_cv/_mean`, `frag_apex_rt_dispersion/_weighted`, +`frag_apex_offset_from_profile_mean`, `frag_gaussianity_mean/_weighted`, +`frag_zigzag_mean`), `sumtrace_unweighted_gaussian_r2`, and +`reference_profile_rt_entropy_peak/_ratio`. Has unit tests +(`chromatographic.rs:876`). + +### mass_accuracy (17, `mass_accuracy.rs`) + +Fragment ppm-error distribution over matched fragments (uses a fixed +`FRAG_TOL_PPM = 20.0`; the config tolerance is not carried in Evidence). Names: +`median_abs_frag_ppm`, `signed_mean_frag_ppm`, `ppm_std`, `ppm_iqr`, +`ppm_range`, `max_abs_frag_ppm`, `intensity_weighted_abs_ppm`, +`intensity_weighted_signed_ppm`, `intensity_weighted_ppm_std`, +`lib_weighted_abs_ppm`, `frac_frag_within_half_tol`, `high_ppm_intensity_frac`, +`ppm_intensity_anticorr`, `mass_error_mz_trend`, `mean_abs_mz_error_da`, and the +positive DIA-NN-style evidence `mass_evidence_gauss` (predicted-weighted +Gaussian concentration, sigma 10 ppm) and `mass_log_evidence`. The plan's +`precursor_mass_error_ppm` is deliberately skipped (no theoretical precursor m/z +in Evidence). + +### ion_series (34, `ion_series.rs`) + +b/y series coverage, ladder contiguity, complementarity, per-series similarity. +Names: `n_matched_b/_y`, `frac_matched_b/_y`, `by_count_balance`, +`by_intensity_ratio`, `by_ratio_agreement` (tanh-squashed log-odds), +`by_ratio_consistency`, `longest_b_run/_y_run`, `longest_run_max`, +`longest_run_frac_length`, `series_coverage_b/_y`, `sequence_coverage` +(cleavage-site union), `series_gap_fraction`, `by_complement_count`, +`by_complement_mz_consistency` (b+y obs m/z vs M + 2 proton), +`by_complement_coelution`, `ordinal_intensity_concordance_y/_b`, +`series_coelution_y/_b`, `spectral_angle_b/_y`, `pearson_b/_y`, +`cosine_charge1/_charge2`, `charge_corr_balance`, `mean_matched_ordinal_norm`, +`by_ion_contiguous_intensity`, `by_ion_contiguous_lib_frac`, +`both_series_present`. Uses `PROTON` from `mumdia-core::constants`. + +### ms1 (25, `ms1.rs`) + +Precursor isotope-envelope agreement against a Poisson-averagine model +(`lambda = 0.000594 * M`) plus MS1/MS2 XIC co-elution. Apex names: +`ms1_isotope_cosine_apex`, `ms1_isotope_spectral_angle_apex`, +`ms1_isotope_chi2_apex`, `ms1_isotope_manhattan_apex`, `iso_ratio_1_0`, +`iso_ratio_2_0`, `iso_plus1_ratio_dev`, `iso_plus2_ratio_dev`, +`iso_minus_one_fraction`, `iso_overlap_flag`, `log_ms1_mono`, +`ms1_total_isotope_log`, `has_ms1_signal`, `ms1_isotope_apex_entropy_3`, +`ms1_m1_entropy_contribution`. The XIC block (`ms1_ms2_time_corr`, +`ms1_ms2_envelope_time_corr`, `ms1_iso_coelution`, `ms1_ms2_apex_rt_delta`, +`ms1_iso_ratio_stability`, `ms1_mono_gaussianity`, `ms1_ms2_fwhm_ratio`, +`ms1_isotope_corr_xic`, `ms1_envelope_over_time_corr`, +`ms1_isotope_xic_shape_consistency`) reads `Evidence.ms1_xic` and returns 0.0 +until the extract stage persists MS1 isotope XICs (evidence gap; currently +unpopulated in the default chain). + +### rt (13, `rt.rs`) + +RT agreement between the observed apex and the calibrated predicted RT. Names: +`rt_error_signed`, `rt_error_abs` (dropped as reserved), `rt_error_squared`, +`rt_error_signed_norm_gradient`, `rt_error_abs_norm_gradient`, `observed_rt_raw`, +`predicted_rt_raw`, `observed_rt_fraction`, `predicted_rt_fraction`, +`rt_error_over_peak_width` (base width at 10%), `rt_error_over_fwhm`, +`rt_diff_profile_apex` (vs full-window profile argmax RT), +`predicted_rt_in_gradient`. + +### novel (12, `novel.rs`) + +Seed corroboration and precursor/charge metadata. Names: +`log_seed_hyperscore`, `seed_hyperscore_per_matched`, `seed_identified` (dropped +as reserved), `peptide_length` (dropped as reserved), `precursor_charge`, +`charge_is_2/_is_3/_is_4plus`, `precursor_mass`, `log_total_matched_intensity`, +`n_matched_frags`, `n_predicted_frags`. Sequence-dependent features from the +plan (missed cleavages, C-terminal residue, modification count) are skipped +because Evidence carries only `seq_len`. + +### nonzero (12, `nonzero.rs`) + +Zero-ignoring variants of the apex spectral and co-elution features. The apex +scan samples one grid point, so a fragment peaking one scan off reads 0 there +(~8% of fragments); these recompute over the per-fragment peak-max and over +present-only scans. Names: `frag_corr_peakmax`, `frag_cosine_peakmax`, +`spectral_angle_peakmax`, `frag_corr_matched_nz`, `frag_cosine_matched_nz`, +`peakmax_apex_gain`, `n_frag_present_inpeak`, `frac_frag_present_inpeak`, +`coelution_mean_bothpos`, `coelution_mean_summpos`, `ref_corr_nz`, +`profile_cos_nz`. + +### order_consistency (8, `order_consistency.rs`) + +Prediction-free MS2-XIC rank stability: at each scan the fragments are ranked by +observed intensity, and the features measure whether that ranking persists +across the peak (orthogonal to library-agreement families). Names: +`rank_corr_vs_apex_mean`, `rank_corr_vs_apex_std`, `rank_corr_adjacent_mean`, +`kendall_vs_apex_mean`, `top1_frag_persistence`, `top2_order_persistence`, +`argmax_frag_entropy`, `self_cosine_vs_apex_mean`. Degenerate below 3 fragments +or 3 non-empty scans. Has unit tests (`order_consistency.rs:271`). + +### peak_scans (2, `peak_scans.rs`) + +Label-blind window-degeneracy indicator, emitted for every PSM so the rescorer +can tell an undefined zero from a measured zero when the window-based families +collapse. Names: `n_peak_scans`, `peak_window_degenerate` (1 when fewer than 3 +non-empty scans, mirroring `order_consistency::MIN_SCANS`). + +### apex_dispersion (13, `apex_dispersion.rs`) + +Fragment apex dispersion and consensus peak shape, intensity-independent +(breadth of co-elution rather than height). Names: `frag_apex_rt_std`, +`frag_apex_rt_mad`, `frag_apex_max_dev`, `frag_apex_mean_dev`, +`frag_apex_agree_frac`, `precursor_frag_apex_delta`, `peak_symmetry`, +`peak_tailing`, `peak_n_local_maxima`, `peak_shoulder_score`, `peak_fwhm_scans`, +`peak_truncation`, `apex_frac_of_window`. Has unit tests +(`apex_dispersion.rs:226`). + +### mass_uncertainty (10, `mass_uncertainty.rs`) + +Fragment mass-error distribution over matched fragments plus evidence-breadth. +Names: `frag_mass_err_median`, `frag_mass_err_abs_median`, `frag_mass_err_std`, +`frag_mass_err_iqr`, `frag_mass_err_max_abs`, `frag_mass_err_range`, +`effective_frag_count` (inverse participation ratio), `evidence_concentration` +(fraction in the strongest fragment), `frac_top3_pred_observed`, +`frac_top5_pred_observed`. Has unit tests (`mass_uncertainty.rs:133`). + +## The Percolator PIN (`write_pin`, `features.rs:1388`) + +Streamed row-by-row through a `BufWriter` (not materialized as one String). +Header: `SpecId\tLabel\tScanNr\tExpMass\tCalcMass\t\t +Peptide\tProteins`. Per row: `SpecId = cand_`, +`Label = -1` when `label == "decoy"` else `1`, `ScanNr = candidate_id`, +`ExpMass = CalcMass = precursor_mz` (`{:.5}`), each feature at `{:.6}`, then +`Peptide = -..-` and `Proteins = `. The feature matrix +passed to the PIN is built column-parallel from `fmap` in `active_features` +order (`features.rs:856`), so the PIN and the Parquet share the same ordered +feature list. + +## The feature-schema hash (`feature_schema_id`, `features.rs:229`) + +`blake3_str(cols.join(","))` of the ordered active column list. Written to +`.schema.json` and recorded in the report `stats`. It is a content hash of +the exact ordered names, so any addition, removal, or reordering changes the id +and a classifier trained under one schema is never silently applied under +another. Because the family registry order is frozen and append-only, appending +a new family or feature at the end changes the id predictably while leaving all +prior positions stable. + +## bound_from_confident (elution-boundary calibration, `features.rs:606`) + +When `bound_from_confident` is true (the default), the stage learns one pair of +elution half-widths `(L, R)` in seconds from the confident-target seed set +(`spectrum_q <= 0.01`, `label == "target"`; the same anchor set used for RT +calibration and DeepLC fine-tune). For each confident candidate it detects the +per-candidate peak with `elution_peak_rt_bounds` (`features.rs:1038`) and +records `apex - lo` and `hi - apex`. If at least 20 anchors resolve, it takes +the `bound_confident_pct` percentile of the left and right half-widths (median +by default) and returns `Some((L, R))`; every candidate is then bounded on +`[apex - L, apex + R]` via `global_bound_indices` (`features.rs:1011`). This +removes per-candidate boundary manipulation, so a chimeric decoy is scored over +a real-peptide-width window centred on its apex rather than one it can widen or +narrow. With fewer than 20 anchors the stage logs a warning and falls back to +per-candidate boundary detection for that run. When the flag is false, every +candidate detects its own peak boundary from its top-3-predicted-fragment +profile (the legacy path). The same `global_bounds` argument threads into both +`fragment_features` and `build_evidence`, so Minimal/Rich and Extended see the +identical window. + +## The shared stats kernel (`stats.rs`) + +One implementation of `pearson`, `cosine`, `spectral_angle`, used by +`fragment_features` and every family (do not reimplement). `pearson` +(`stats.rs:6`) is population Pearson with a zero-variance guard returning 0.0 +for `n < 2` or zero variance. `cosine` (`stats.rs:29`) returns 0.0 if either +vector is all-zero. `spectral_angle` (`stats.rs:44`) is +`1 - 2 * acos(clamp(cosine, -1, 1)) / pi`, in [0,1] with 1 = identical. Families +add local specializations that do not belong in the shared kernel (weighted +Pearson, Spearman via `pearson` on average ranks, Kendall tau, windowed cross- +correlation via `super::best_xcorr`), but the base Pearson/cosine call the +shared functions. + +## Configuration + +`FeaturesConfig` (`config.rs:618`); all fields have `#[serde(default)]` and the +config was pruned of dead fields (the `FeatureSet::Custom` variant no longer +exists). + +| field | default | effect | +|---|---|---| +| `set` | `Minimal` | which set `active_features` returns (Minimal 14 / Rich 44 / Extended 381) | +| `coelution_corr_threshold` | 0.9 | threshold for `n_coelution_above` (count of pairwise fragment correlations at or above it) | +| `prec_tol_ppm` | 20.0 | precursor tolerance carried for feature bookkeeping | +| `bound_features` | true | restrict trace-based features to the elution peak instead of the whole extracted window | +| `bound_peak_fraction` | 1/3 | peak-boundary threshold as a fraction of apex height (DIA-NN-style; matched DIA-NN RT bounds best) | +| `bound_peak_grace` | 0 | consecutive sub-threshold scans to bridge before stopping (0 = stop at first miss; 1 bridges a single-scan dip) | +| `bound_from_confident` | true | learn one global left/right half-width from the confident seed set and apply it to every candidate; false = per-candidate detection | +| `bound_confident_pct` | 50.0 | percentile of the confident-set half-widths taken as the global half-width (50 = median) | + +Note that the fragment tolerance used inside `mass_accuracy` is a hardcoded +`FRAG_TOL_PPM = 20.0` (`mass_accuracy.rs:44`), not `prec_tol_ppm`; Evidence does +not carry the configured fragment tolerance. + +## Invariants, determinism, gotchas + +- Every family must return exactly `NAMES.len()` values in `NAMES` order; a + `debug_assert_eq!` in `extended_values` (`features.rs:139`) and in most + families catches a mismatch in debug builds. Non-finite values are coerced to + 0.0 at family boundaries and again in `extended_values`. +- The `FAMILIES` registry order and each family's `NAMES` order are the frozen + schema; they are append-only. Reordering or renaming changes `schema_id` and + invalidates any trained classifier. +- Deduplication is stable and precomputed once (`extended_value_plan`, + `features.rs:108`), reproducing the same survivors and order as + `extended_name_refs`, so names and values stay in lockstep across runs. +- Determinism: the parallel per-PSM pass collects into a `Vec` indexed by row, + so the serial assembly is byte-identical to a serial run regardless of thread + count. RT-axis alignment maps intensities keyed by `f32::to_bits` + (`features.rs:389`), which is exact-equality safe because the same `rt` + values are reused, not recomputed. +- "Matched" throughout the families means `obs_apex[i] > 0.0` (observed at the + apex scan), which differs subtly from "present in the peak" used by the + `nonzero` family (per-fragment peak-max `> 0`). +- The apex scan samples a single grid point; a fragment peaking one scan off + reads 0.0 at the apex. The `nonzero` family exists specifically to give the + classifier zero-tolerant variants alongside the originals. +- `peptide_length` (`features.rs:246`) strips a leading `DECOY_` prefix before + counting residues, and ignores bracketed modifications, so the decoy marker is + not a length-based target/decoy label leak (tested at `features.rs:1425`). +- MS1 XIC features return 0.0 unless the extract stage wrote `ms1_*` + chromatogram rows; `ms1_xic` is otherwise empty. This is a known evidence gap, + not a bug. +- The peptidoform grouping for cross-charge features uses the ProForma string, + which is charge-independent and keeps `DECOY_` peptidoforms grouped among + themselves, so it is not a target/decoy label leak (`features.rs:662`). + +## How to extend / modify + +- To add a new Extended family: create a module under `stages/features/` + exposing `pub const NAMES: &[&str]` and `pub fn values(&Evidence) -> Vec` + of matching length and order, `mod`-declare it (`features.rs:32`), and append + `(name, values)` to `FAMILIES` (`features.rs:52`). Appending at the end keeps + all prior schema positions stable. Reuse `crate::stats` and the parent helpers + (`mean`, `normalize_sum`, `best_xcorr`, `smooth3`, `peak_bounds`) rather than + reimplementing kernels. Add an arity unit test (`values(&e).len() == + NAMES.len()`) and a degenerate-evidence finiteness test. +- To add a Minimal/Rich feature: append the name to `MINIMAL_FEATURES` or + `RICH_EXTRA` and push its value in the serial loop (`features.rs:754`); make + sure no Extended family already uses the name, or it will be dropped as a + reserved collision. +- New names must be globally unique across Minimal, Rich, and every family; a + collision is silently dropped by the dedup filter, so run + `feature_sets_sized` (`features.rs:1434`) after any change to confirm the + size and uniqueness invariants. +- Do not reach into vendor formats or duplicate the mass model / stats kernel; + Evidence is the sole per-PSM interface for families, and any new scalar a + family needs must be added to `Evidence` (`features.rs:278`) and filled by the + caller (`features.rs:718`). +- Prefer adding a config field (backed by a default) over hardcoding a + threshold, consistent with the project convention; the current hardcoded + fragment tolerance in `mass_accuracy` is a documented exception awaiting a + tolerance carried on `Evidence`. diff --git a/docs/11_compete_rescore_fdr.md b/docs/11_compete_rescore_fdr.md new file mode 100644 index 0000000..ff2988c --- /dev/null +++ b/docs/11_compete_rescore_fdr.md @@ -0,0 +1,458 @@ +# compete, rescore, and FDR + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This subsystem is the tail of the identification chain (PLAN.md Stage F). It +turns the per-PSM feature table produced by the `features` stage into a scored, +FDR-controlled result set. It has three parts: + +1. **compete** (`mumdia compete`): within each competition group, resolve + redundant candidates for the same elution peak before target-decoy counting, + so multiple plausible candidates for one peak cannot each be counted as a + discovery. Default behaviour keeps only the best-scoring candidate per group. +2. **rescore** (`mumdia rescore`): train a semi-supervised classifier over the + competed PSMs of the whole experiment, produce a single discriminant `score` + per PSM, and derive native target-decoy q-values at several aggregation + levels (PSM, per-run PSM, precursor, peptide, protein group). +3. **fdr** (`crate::fdr`): the shared, stateless q-value kernels. Both + `search-seed` and `rescore` call these; they are not a stage. + +The design principle behind the sensitivity work is "preserve candidate evidence +until the workflow can make a well-calibrated decision". That is why every +non-default competition mode and the label rule keep decoys and redundant +variants alive so the rescorer and FDR have a valid null to work against. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/compete.rs` | `compete` stage: grouping, within-group competition resolution, competed table + optional audit | +| `rust/mumdia/crates/mumdia/src/stages/rescore.rs` | `rescore` stage: input concat, classifier dispatch, multi-context q-values, scored table | +| `rust/mumdia/crates/mumdia/src/rescoring.rs` | `percolator_lite`, the native semi-supervised linear rescorer (the `native_tda` path) | +| `rust/mumdia/crates/mumdia/src/fdr.rs` | q-value kernels: `target_decoy_q`, `entrapment_q`, `count_targets_at_q`, `validate_labels`, `ln_factorial` | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `CompeteConfig`, `CompetitionMode`, `CompeteGroupBy`, `RescoreConfig`, `RescorerKind` | +| `scripts/mokapot_worker.py` | Mokapot PIN sidecar (`RescorerKind::Mokapot`) | +| `scripts/nn_rescore_worker.py` | PyTorch MLP PIN sidecar (`RescorerKind::NnTorch`) | +| `scripts/entrapment_worker.py` | entrapment GBM sidecar (`RescorerKind::Entrapment`) | + +## Inputs and outputs + +### compete + +Consumes the features artifact `psms_features` (path passed as `--features`), +plus its schema companion `.schema.json` (read at compete.rs:39 via +`FeatureSchema::read`). Reads these columns (compete.rs:31-45): `candidate_id` +(u32), `label` (str), `base_peptide_id` (u32), `peptidoform` (str), `protein` +(str), `apex_rt` (f64), `precursor_mz` (f64), `prelim_score` (f64), `charge` +(f64, only needed for `peptidoform_charge` grouping), and every feature column +named in the schema. + +Produces `psms_competed` (schema version 1, `artifact::PSMS_COMPETED`, +schema.rs:20) at `--out`. Column schema (compete.rs:118-130): `candidate_id` +(u32), `label` (str), `base_peptide_id` (u32), `peptidoform` (str), `protein` +(str), `apex_rt` (f64), `precursor_mz` (f64), `prelim_score` (f64), then every +feature column (f64) carried through unchanged. It also writes +`.schema.json` (compete.rs:133) so rescore recovers the exact feature list, +and `.report.json`. + +Optional sidecar `.compete_audit.parquet` (only when +`compete.emit_competition_audit = true`, compete.rs:138-180): one row per removed +candidate with `candidate_id` (u32), `label` (str), `peptidoform` (str), +`winner_candidate_id` (u32), `loser_prelim` (f64), `winner_prelim` (f64), +`rejection_reason` (str; `outcompeted_by_decoy` or `outcompeted_by_target` from +`RejectionReason::code()`). + +### rescore + +Consumes one or more `psms_competed` tables (`--competed`, a `Vec`, +main.rs:154). The feature list is read from the schema companion of the first +input only (rescore.rs:64); all inputs are assumed to share that schema. Reads +`candidate_id`, `label`, `base_peptide_id`, `peptidoform`, `protein`, `charge` +(f64, cast to i32), `prelim_score`, `precursor_mz`, and the feature columns +(rescore.rs:66-87). + +Produces `psms_scored` (schema version 2, `artifact::PSMS_SCORED`, schema.rs:21) +at `--out`. Column schema (rescore.rs:320-347): `candidate_id` (u32), +`peptidoform` (str), `charge` (i32), `label` (str), `protein` (str), +`base_peptide_id` (u32), `score` (f64, the classifier discriminant), +`q_value` (f64, pooled PSM q), `peptide_q_value` (f64), `protein_group` (str, +the protein-accession-set string, a duplicate of `protein`), `pg_q_value` (f64), +`global_q_value` (f64, byte-identical alias of `q_value`), `prelim_score` (f64), +`source` (u32, index into `--competed` identifying the run), `run_psm_q` (f64), +`experiment_psm_q` (f64, alias of `q_value`), `precursor_q` (f64). Plus +`.report.json` recording the classifier actually used and the 1% counts. + +Sidecar working files are written under `--work-dir`: `rescore.pin` and +`rescore_sidecar_out.parquet` for the PIN sidecars, `entrapment_in.parquet` / +`entrapment_out.parquet` for the entrapment GBM (rescore.rs:518-519, 581-582). + +## How it works + +### compete: grouping + +`compete::run` (compete.rs:28) builds a competition group key per PSM +(compete.rs:72-93). The key is a fixed-size tuple `(u32, u8, i64)` rather than a +freshly allocated String, chosen so grouping does not allocate per PSM: + +- element 0 is `base_peptide_id` for `Precursor`/`Apex`, or a dense + first-appearance `pform_id` for `PeptidoformCharge` (built at compete.rs:52-62); +- element 1 is the label code `0=target, 1=decoy, 2=other` (compete.rs:74-78); +- element 2 is a bucket: constant `0` for `Precursor`; the rounded apex-RT bucket + `(apex_rt / apex_rt_tolerance_s).round()` for `Apex` (compete.rs:82); the + rounded charge for `PeptidoformCharge` (compete.rs:88). + +**The label is part of the key on purpose.** A target is never competed against +its own decoy; competition only arbitrates redundant charge/mod variants within +the target population and, separately, within the decoy population. If a target +could evict its paired decoy, the decoy population would be depleted and the +target-decoy null would collapse, badly underestimating FDR. This is stated at +compete.rs:64-71 and re-stated on `CompetitionMode` in config.rs:705-710. + +`PeptidoformCharge` (config.rs:734-741) is the precursor-level grouping DIA-NN and +Spectronaut report at: sibling charges of one peptide are kept as separate groups +rather than collapsed. It requires the `charge` column and bails if absent +(compete.rs:46-48). + +### compete: resolution + +`resolve_competition` (compete.rs:247) is a pure, unit-tested function. It visits +group keys in **sorted order** (compete.rs:256-257) for determinism, and picks +the winner as the highest `prelim_score`, ties broken by smallest row index +(compete.rs:262-265). The per-mode behaviour (`CompetitionMode`, config.rs:711): + +- `WinnerTakeAll` (default): keep only `win`; everything else is a removal pair + `(loser, winner)` (compete.rs:270-273). +- `None`: keep all members, remove nothing (compete.rs:267-269). FDR handles the + ambiguity downstream. +- `FeaturesOnly`: identical retained set to `None` (same match arm, + compete.rs:267); the distinct name documents intent for the experiment matrix, + the idea being that conflict/contested features carry the interference signal + into rescoring instead of removal. +- `UniqueEvidence`: keep the winner; keep a loser only when its unique-fragment + evidence `>= unique_evidence_min_fragments` (compete.rs:274-287). Evidence comes + from `unique_evidence` (compete.rs:218-232): prefers an explicit + `unique_fragment_count` column, else approximates it as + `n_matched_fragments * (1 - contested_frac)` clamped to [0,1], else raw + `n_matched_fragments`, else `None`. If the mode is selected but no column is + available it warns and falls back to winner-take-all (compete.rs:100-105, + 281 `.unwrap_or(false)`). +- `MarginGated`: keep the winner; remove a loser only when + `prelim[win] - prelim[m] >= margin`, otherwise keep it (compete.rs:288-300). + Conservative removal for the low-FDR region. + +The returned `keep` indices are sorted and deduped (compete.rs:303-304). Kept +rows are projected into the output columns (compete.rs:117-131), and the schema +is forwarded. The report records `group_by`, `mode`, `input_rows`, `kept`, +`removed` (compete.rs:184-197). + +### rescore: input concat and classifier dispatch + +`rescore::run` (rescore.rs:41) concatenates all `--competed` tables into flat +vectors (rescore.rs:65-88). It records a per-PSM `source` = the index of the +input file the PSM came from (rescore.rs:61, 85); for a single-run rescore this +is all-zero. `source` is why the PIN and entrapment sidecars key on a unique flat +row index rather than `candidate_id`: `candidate_id` is the library index and +repeats across runs, so an experiment-wide table would collide on it +(rescore.rs:56-60, 522-525, 588-593). + +Immediately after loading, `crate::fdr::validate_labels` (rescore.rs:89) rejects +any label that is not exactly `"target"` or `"decoy"`. `is_decoy` is derived at +rescore.rs:90; entrapment status is derived separately from the protein string by +`classify_entrapment` (rescore.rs:91, 411). + +The classifier is dispatched on `RescoreConfig::classifier` (rescore.rs:104-192). +The stage tracks `classifier_used`, `model_identity`, and `qmode` so the report +reflects the path actually taken rather than the requested one (rescore.rs:97-99). + +**`RescorerKind::NativeTda`** (config.rs:131, the default): calls `native_scores` +-> `percolator_lite` (rescore.rs:142, 386-402). `percolator_lite` +(rescoring.rs:98) is a Percolator/Mokapot-style linear model: + +- Fold assignment is `fold_key % folds` where `fold_key = base_peptide_id` + (rescoring.rs:107), so every charge/mod variant of a peptide lands in the same + fold and no peptide leaks between train and test. "CV folds by candidate + hashing" is the modulo-of-the-id scheme; the id used is the base-peptide id. +- Folds are processed in parallel (`rayon`, rescoring.rs:112-113) but each fold is + independent and scores only its disjoint test set, so the result is + order-independent and deterministic. +- Each fold fits its **own** standardizer on its training rows only + (`fit_standardizer`, rescoring.rs:14-40, 120), which is the leak-free choice: + test-fold statistics never enter standardization. std < 1e-9 is clamped to 1.0. +- Semi-supervised loop (`num_iter` iterations, rescoring.rs:129-172): compute + target-decoy q on the current train scores, take confident targets + (`q <= train_fdr`) as positives and all decoys as negatives, fit an L2 logistic + regression (`logreg_fit`, rescoring.rs:48-75; full-batch gradient descent, + `l2=1e-3`, `epochs=200`, `lr=0.5`, weight[0]=bias), and re-score the train fold. + If fewer than 10 confident targets exist, it falls back to the top-scoring half + as positives (rescoring.rs:152-169). +- Final score for each test row is `score_row(w, std_row(...))` written back by + original index (rescoring.rs:173-186). Weights start at zero and there is no + RNG, so the whole path is deterministic. + +**`RescorerKind::Mokapot`** (config.rs:132): runs `mokapot_worker.py` through +`run_pin_sidecar` (rescore.rs:105-119, 563). On success it uses the returned +scores; on failure it either hard-errors (if `rescore.strict`) or warns and falls +back to `native_scores` (rescore.rs:112-118). Requires `rescore.python`. + +**`RescorerKind::NnTorch`** (config.rs:134-140): runs `nn_rescore_worker.py` +through the same `run_pin_sidecar` contract (rescore.rs:120-134). A nonlinear +PyTorch MLP with the same CV-fold + iterative positive-reselection scheme. The +worker receives the NN hyperparameters through environment variables +`MUMDIA_NN_FOLDS`, `MUMDIA_NN_ITERS`, `MUMDIA_NN_TRAIN_FDR`, set from +`cfg.folds/num_iter/train_fdr` (rescore.rs:610-616), so the report reflects the +values actually used. Same strict/fallback logic as Mokapot. + +> **NnTorch is nondeterministic.** The worker seeds torch/numpy +> (`nn_rescore_worker.py:205,258-259`) but PyTorch training on floats plus +> nondeterministic kernels means runs are not byte-identical. It also has a +> **scaler leak**: standardization statistics are computed over the **full** +> feature matrix, not per training fold. In-memory backend uses global +> median/IQR (`nn_rescore_worker.py:134-137`); streaming backend accumulates a +> global mean/std in one pass (`:159-170`). This differs from the leak-free +> native `percolator_lite`, which fits the scaler on the train fold only. Both +> facts are properties of the sidecar, not the Rust dispatch. + +**`RescorerKind::Percolator`** (config.rs:141-142): **not wired.** The arm +either hard-errors under `rescore.strict` or warns and uses `native_tda` +(rescore.rs:135-141). `percolator.exe` is installed on the dev machine but no +adapter exists; `rescore.percolator_bin` (config.rs:960) is a dead config field +until it is. + +**`RescorerKind::Entrapment`** (config.rs:143-149): a decoy-independent path for +spike-in entrapment experiments. It requires `entrapment_marker` and at least one +matching PSM, else it warns/errors and falls back to native (rescore.rs:144-160). +With `rescore.python` set it runs `run_entrapment_gbm` (rescore.rs:162, 503): a +gradient-boosted sidecar trained out-of-fold by base peptide, positives = real +targets, negatives = spike-in targets (rescore.rs:497-556). Without python it +uses the native linear rescorer but with `is_entrapment` (not `is_decoy`) as the +negative label (rescore.rs:178, 189). Any of these sets `qmode = Entrapment`, so +q-values are computed by `entrapment_q` instead of `target_decoy_q`. + +### rescore: the multi-context q columns + +After scoring, the stage computes q-values at several aggregation levels, each an +**independent** target-decoy (or entrapment) analysis run on the appropriate +best-per-group reduction. This is deliberate: q-values at different levels are not +derived from one another, they are separately calibrated nulls. + +| output column | grouping | how computed | file:line | +|---|---|---|---| +| `q_value` | none (pooled PSM) | `target_decoy_q` / `entrapment_q` over all PSMs | rescore.rs:196-204 | +| `experiment_psm_q` | none (pooled PSM) | clone of `q_value` | rescore.rs:231 | +| `global_q_value` | none (pooled PSM) | clone of `q_value`, backward-compat alias | rescore.rs:230 | +| `run_psm_q` | by `source` | independent TDA within each run, scattered back by row index | rescore.rs:235-259 | +| `precursor_q` | `(peptidoform, charge)` | best PSM per precursor, TDA over that set | rescore.rs:262-271 | +| `peptide_q_value` | `base_peptide_id` | best PSM per base peptide, TDA over that set | rescore.rs:210 | +| `pg_q_value` | protein-accession-set string | best PSM per protein group, TDA over that set | rescore.rs:215-224 | + +The per-level reduction is `grouped_q` (rescore.rs:443): for each key, keep the +best-scoring member `(score, is_decoy, is_entrapment, is_real)` +(rescore.rs:453-461), run the chosen q kernel over the one-row-per-group set +(rescore.rs:463-474), then assign the group q **only to the winning row** of each +group and give every losing sibling q = 1.0 (rescore.rs:481-494). A lower-scoring +charge/mod variant (possibly itself a false target) must not inherit the winner's +low q. Group counts dedup by key on the winner, so counts are unchanged, but +per-PSM peptide/pg/precursor q no longer propagate to losers. + +`run_psm_q` groups rows by `source` in a `BTreeMap` for deterministic iteration +(rescore.rs:236-239) and runs a full independent TDA within each run, so a per-run +report gets a genuine per-run FDR instead of the pooled value. For a single-run +rescore (`source` all-zero) it equals `q_value`. + +**Why peptide q >= precursor q at the same threshold.** Both are best-per-group +TDA, but the peptide grouping (`base_peptide_id`) is coarser than the precursor +grouping (`peptidoform+charge`): several precursors collapse into one peptide. A +coarser grouping has fewer, higher-scoring representatives and a different +target/decoy balance among the survivors, so the monotonized q at a given level +is generally not lower than at the finer level. Reporting both lets a consumer +pick the FDR granularity that matches its claim (a peptide-level ID list vs a +precursor-level one). + +Interned dense u32 ids are used for the protein and precursor groupings +(rescore.rs:215-223, 262-270) purely as a performance optimization: interning the +accession-set string and the `(peptidoform, charge)` tuple to first-seen integers +avoids hashing and cloning hundreds of thousands of strings inside `grouped_q`. +The mapping is bijective, so grouping and the resulting q-values are unchanged. + +`is_reported` (rescore.rs:275-278) selects which rows count toward the 1% +summaries: real targets in entrapment mode (spike-in excluded), all non-decoys +otherwise. The report records `target_psms_at_1pct`, `target_peptides_at_1pct`, +`target_protein_groups_at_1pct`, `target_precursors_at_1pct`, and in entrapment +mode the `entrapment_ratio` and the entrapment-leak count `entrapment_peptides_at_1pct` +(spike-in peptides passing the 1% gate, a running FDR-validity check, +rescore.rs:308-318, 350-360). + +### fdr: the kernels + +`target_decoy_q` (fdr.rs:7) is the no-pi0 estimator +`q = (n_decoys + 1) / max(1, n_targets)`, monotonized: + +1. Sort record indices by descending score (fdr.rs:12-18). +2. Walk in score order, processing **tied-score blocks together** so every PSM in + a block gets the same FDR regardless of its arbitrary within-tie order + (fdr.rs:27-43). This is a determinism requirement (PLAN.md Section 7): a + target/decoy interleave inside one tie block must not change the q. +3. FDR at rank = `(td + 1) / max(1, tt)` where `td`, `tt` are cumulative decoy and + target counts at that score (fdr.rs:38). The `+1` is the conservative + finite-sample pseudocount; the bare `n_decoys/n_targets` is optimistic in the + low-count regime. +4. Monotonize from worst-scoring to best so q is non-increasing with score + (fdr.rs:45-51): `q[i] = min(fdr at all ranks worse-or-equal)`. + +The best target with perfect separation gets `q = 1/n_targets`, not 0 +(test at fdr.rs:144-157). + +`entrapment_q` (fdr.rs:64) is the empirical-null analog: +`FDR(t) = (ratio * n_entrap(>=t) + 1) / max(1, n_real(>=t))`. `ratio` = +`N_real_lib / N_entrap_lib` corrects for unequal library sizes; the `+1` is the +same pseudocount. Rows that are neither entrapment nor real (decoys) are ranked +but enter no count (fdr.rs:87-92). Same tied-block walk and worst-to-best +monotonization as `target_decoy_q`. Unlike in-silico decoys, the entrapment +population experiences the same chimeric DIA interference as real targets, so the +estimate is not optimistic (fdr.rs:54-63). Uses a stable sort so ties keep input +order (fdr.rs:70-75). + +`count_targets_at_q` (fdr.rs:110): count of non-decoy records with `q <= threshold`. + +`validate_labels` (fdr.rs:122): hard-error on any label other than `"target"` or +`"decoy"`. An unknown or malformed label must not silently count as a target +because the target-decoy null depends on exact labeling. Entrapment status is +derived from the protein accession, not the label, which is why only two label +values are valid here. + +`ln_factorial` (fdr.rs:132): `ln(n!)` via summed logs, used where matched-fragment +counts feed a hyperscore-style term; `n` is small so the naive loop is fine. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `compete::run` | compete.rs:28 | full compete stage: group, resolve, write competed table + optional audit | +| `compete::resolve_competition` | compete.rs:247 | pure per-mode within-group resolution; returns kept indices + removal pairs | +| `compete::unique_evidence` | compete.rs:218 | derive per-candidate unique-fragment evidence for `UniqueEvidence` mode | +| `CompeteParams` | compete.rs:21 | `features`, `out`, `cfg`, `config_hash` | +| `rescore::run` | rescore.rs:41 | full rescore stage: concat, dispatch, multi-context q, scored table | +| `rescore::native_scores` | rescore.rs:386 | thin wrapper calling `percolator_lite` with the config knobs | +| `rescore::classify_entrapment` | rescore.rs:411 | per-PSM `(is_entrapment, is_real_target)` from the protein string | +| `rescore::grouped_q` | rescore.rs:443 | best-per-group reduction + level q, winner-only assignment | +| `rescore::run_pin_sidecar` | rescore.rs:563 | write PIN, run Mokapot/NnTorch worker, read scores by row index | +| `rescore::run_entrapment_gbm` | rescore.rs:503 | write parquet, run entrapment GBM worker, read scores by row_id | +| `QMode` | rescore.rs:23 | which null q is computed against: `Decoy` or `Entrapment` | +| `RescoreParams` | rescore.rs:30 | `competed`, `out`, `work_dir`, `script_dir`, `cfg`, `config_hash` | +| `percolator_lite` | rescoring.rs:98 | native semi-supervised L2 logreg rescorer with per-fold scaler | +| `RescoreInput` | rescoring.rs:85 | features, is_decoy, fold_key, init_score, folds, num_iter, train_fdr | +| `fit_standardizer` | rescoring.rs:14 | per-fold mean/std over training rows only (leak-free) | +| `logreg_fit` | rescoring.rs:48 | full-batch GD logistic regression with L2, weight[0]=bias | +| `target_decoy_q` | fdr.rs:7 | `(D+1)/T` monotonized tied-block q from `(score, is_decoy)` | +| `entrapment_q` | fdr.rs:64 | `(ratio*E+1)/R` monotonized tied-block entrapment q | +| `count_targets_at_q` | fdr.rs:110 | count non-decoy records at or below a q threshold | +| `validate_labels` | fdr.rs:122 | label whitelist; rejects anything but target/decoy | +| `ln_factorial` | fdr.rs:132 | `ln(n!)` via summed logs | + +## Configuration + +The config was recently pruned of dead fields; the fields below are the live ones +this subsystem reads. + +### `compete` (`CompeteConfig`, config.rs:668-703) + +| field | default | effect | +|---|---|---| +| `group_by` | `precursor` | competition grouping (`CompeteGroupBy`, config.rs:734): `precursor` groups a peptide's charge/mod variants and its decoy; `apex` also buckets by rounded apex RT; `peptidoform_charge` keeps each peptidoform+charge separate | +| `apex_rt_tolerance_s` | `5.0` | RT bucket width (s) for `group_by = apex` (compete.rs:82) | +| `mode` | `winner_take_all` | within-group resolution (`CompetitionMode`, config.rs:713): `winner_take_all`, `none`, `features_only`, `unique_evidence`, `margin_gated` | +| `margin` | `0.0` | score margin required to remove a loser under `margin_gated` (compete.rs:294) | +| `unique_evidence_min_fragments` | `2` | min unique-fragment count for a loser to survive under `unique_evidence` (compete.rs:276) | +| `emit_competition_audit` | `false` | write `.compete_audit.parquet` (compete.rs:138) | + +All non-default `mode` values are part of the sensitivity program and are +default-off; the production chain uses `winner_take_all` and is byte-identical +unless a knob is set. + +### `rescore` (`RescoreConfig`, config.rs:953-1002) + +| field | default | effect | +|---|---|---| +| `classifier` | `native_tda` | which rescorer (`RescorerKind`, config.rs:128): `native_tda`, `mokapot`, `nn_torch`, `percolator` (unwired), `entrapment` | +| `folds` | `3` | CV folds (native + PIN sidecars via env) | +| `train_fdr` | `0.01` | q threshold for the confident-positive set in the semi-supervised loop | +| `num_iter` | `10` | semi-supervised iterations for the native rescorer | +| `python` | `None` | interpreter for the Mokapot/NnTorch/entrapment sidecars; required for those paths | +| `percolator_bin` | `None` | dead field until the `percolator` path is wired | +| `entrapment_marker` | `None` | protein substring marking spike-in negatives; required for `entrapment` | +| `entrapment_exclude` | `None` | substring that, if also present, keeps a PSM as a real target (shared peptides) | +| `entrapment_contaminant_markers` | `[]` | substrings marking genuine contaminants inside the spike-in proteome; matching PSMs stay real targets | +| `entrapment_ratio` | `1.0` | `N_real_lib / N_entrap_lib`, scales the entrapment FDR estimate | +| `strict` | `false` | when true, any sidecar failure / misconfiguration is a hard error instead of a native fallback | + +## Invariants, determinism, gotchas + +- **Label stays in the competition key** (compete.rs:64-93, config.rs:705-710). A + target never competes against its own decoy. Removing this depletes decoys and + underestimates FDR. +- **compete is deterministic.** Groups are visited in sorted key order and the + winner tie-breaks to the smallest row index (compete.rs:256-265). Kept indices + are sorted+deduped (compete.rs:303-304). No floats are summed across an unordered + map. +- **Tied-score blocks share one q** in both `target_decoy_q` and `entrapment_q` + (fdr.rs:27-43, 82-99). Within-tie order is arbitrary and must not change the q. +- **`native_tda` is fully deterministic**: zero-initialized weights, no RNG, + per-fold work is order-independent even under rayon (rescoring.rs:112-186). +- **`nn_torch` is nondeterministic** and has a **scaler leak** (standardization + fit over the full matrix, not per fold; `nn_rescore_worker.py:134-137,159-170`). + Do not treat its scores as reproducible; do not use it where byte-identity is + required. +- **`percolator` is unwired** (rescore.rs:135-141). It silently degrades to + `native_tda` unless `strict = true`. `percolator_bin` is a dead field. +- **Sidecars key on the flat row index, not `candidate_id`** (rescore.rs:56-60, + 522-525, 588-593). `candidate_id` is the library index and repeats across runs; + keying on it collides in an experiment-wide (multi-file) rescore. A missing row + in the sidecar output gets the worst score (rescore.rs:553-555, 629-630). +- **`global_q_value` and `experiment_psm_q` are exact clones of `q_value`** + (rescore.rs:230-231). `global_q_value` is kept only for backward-compat. +- **Losing siblings get q = 1.0** at the peptide/precursor/pg levels + (rescore.rs:481-494); do not read a loser's level q as its FDR. +- **Multi-context q-values are independent per level**, each a separate TDA on its + own best-per-group reduction. They are not derived from `q_value` by + aggregation. +- **`validate_labels` runs before any counting** (rescore.rs:89). Any label other + than target/decoy aborts the stage. +- **Schema companion is mandatory.** compete writes `.schema.json` + (compete.rs:133); rescore reads only the first input's schema (rescore.rs:64) and + assumes all inputs share it. Concatenating tables with divergent feature schemas + will misread columns. + +## How to extend / modify + +- **Add a competition mode**: extend `CompetitionMode` (config.rs:713), add a match + arm in `resolve_competition` (compete.rs:266-301), and add a unit test alongside + the existing ones (compete.rs:308-390). Keep the label in the key and keep group + visitation in sorted order so determinism holds. +- **Add a grouping**: extend `CompeteGroupBy` (config.rs:734) and add a key arm in + compete.rs:79-91. If it needs a new column, guard for its presence as + `PeptidoformCharge` does (compete.rs:46-48). +- **Wire the `percolator` path**: replace the warn/bail at rescore.rs:135-141 with a + PIN round-trip. The PIN writer already exists (`run_pin_sidecar`, + rescore.rs:563); percolator consumes the same PIN, so the work is invoking + `percolator_bin` (config.rs:960) and parsing its output back into a per-row score + vector aligned to input order. Honor `strict`. +- **Add a rescorer sidecar** that follows the PIN contract: reuse `run_pin_sidecar` + (rescore.rs:563), which writes SpecId `psm_i` / ScanNr `i`, ExpMass=CalcMass=mz, + the feature columns in schema order, and `-..-` / `` + (rescore.rs:585-601). The worker must echo the SpecId tail as `candidate_id` and + emit a `score` column; scores are mapped back by that row index (rescore.rs:625-630). + NN hyperparameters are passed as `MUMDIA_NN_*` env vars (rescore.rs:614-616). +- **Add a q-value context**: add a grouping vector and a `grouped_q` call + (mirror `precursor_q` at rescore.rs:262-271), then add the output column at + rescore.rs:320-347 and a 1% count if desired. Reuse the interning pattern for + string keys to avoid hashing large columns. +- **Change the FDR estimator**: `target_decoy_q` (fdr.rs:7) and `entrapment_q` + (fdr.rs:64) are the only two kernels; both `search-seed` and `rescore` call them, + so a change here is global. Preserve the tied-block walk and the worst-to-best + monotonization or determinism breaks. The `+1` pseudocount is intentional; do not + drop it to chase counts. +- **Fix the NnTorch scaler leak**: fit standardization per training fold in + `nn_rescore_worker.py` (mirror `fit_standardizer` in rescoring.rs:14-40) instead + of over the full matrix at `:134-137` / `:159-170`. diff --git a/docs/12_quant_lfq_align_mbr_report_audit.md b/docs/12_quant_lfq_align_mbr_report_audit.md new file mode 100644 index 0000000..e270870 --- /dev/null +++ b/docs/12_quant_lfq_align_mbr_report_audit.md @@ -0,0 +1,433 @@ +# quant, quant-lfq, align, mbr, report, audit + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This document covers the six "tail" subcommands of the pipeline: the ones that run +after `rescore` has produced `psms_scored.parquet`, plus the two experiment-level +stages that operate across multiple runs. + +- **quant** (Stage G, `mumdia quant`): integrate per-fragment chromatograms over the + elution peak, sum the top-N fragments into a per-peptidoform quantity, and roll up + to protein groups. Single-run. +- **quant-lfq** (Stage G, `mumdia quant-lfq`): combine several per-run quant tables + into a protein-by-run abundance matrix by MaxLFQ (peptide-level) or directLFQ + (ion/fragment-level), with optional cross-run median-ratio normalization. +- **align** (Stage D2, `mumdia align`): put >=2 runs on a common RT coordinate by + fitting a reference LOESS RT map per run and recording the residual spread. +- **mbr** (Stage D3, `mumdia mbr`): partially wired match-between-runs identification + transfer. A Rust CLI + config gate that shells out to `scripts/mbr_worker.py`. +- **report** (`mumdia report`): emit human-readable `peptides.tsv` + `proteins.tsv` + from the scored PSM table, joined to quant. +- **audit** (`mumdia audit`): reconstruct, per candidate, the pipeline stage flags and + the earliest rejection reason across the artifact chain, without re-running compute. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/quant.rs` | `quant` stage + `run_lfq_combine`/`size_factors` for `quant-lfq` | +| `rust/mumdia/crates/mumdia/src/quant_lfq.rs` | MaxLFQ/directLFQ ratio-alignment core (`lfq_profile`, `maxlfq`, `directlfq`) | +| `rust/mumdia/crates/mumdia/src/stages/align.rs` | Stage D2 cross-run RT alignment | +| `rust/mumdia/crates/mumdia/src/stages/report.rs` | `peptides.tsv` + `proteins.tsv` writer | +| `rust/mumdia/crates/mumdia/src/stages/audit.rs` | candidate identification-loss ladder | +| `rust/mumdia/crates/mumdia-core/src/rejection.rs` | `RejectionReason` enum + ladder ordering | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `QuantConfig`, `MbrConfig`, `RtImTrainConfig`, and the strategy enums | +| `scripts/mbr_worker.py` | MBR transfer sidecar (rescuable + re-extraction tiers) | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI definitions + handlers for all six subcommands | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | `run_mbr` (builds argv for `mbr_worker.py`) | + +## Inputs and outputs + +### quant +Consumes `psms_scored.parquet` (from `rescore`) and `chromatograms.parquet` (from +`extract`). Produces two mandatory artifacts and two optional ones. + +`peptide_quant.parquet` (schema `PEPTIDE_QUANT` v1, `quant.rs:314`): + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | library candidate id | +| `peptidoform` | str | ProForma peptidoform | +| `charge` | i32 | precursor charge | +| `protein_group` | str | protein-group key | +| `quantity` | f64 | sum of top-N fragment areas | +| `n_fragments_used` | i32 | number of fragments actually summed (`min(len, top_n_fragments)`) | + +`protein_group_quant.parquet` (schema `PROTEIN_GROUP_QUANT` v1, `quant.rs:341`): +`protein_group` (str), `quantity` (f64), `n_peptides` (i32). + +`fragment_quant.parquet` (optional, `--out-fragment`; `quant.rs:369`): one row per +fragment area, `candidate_id`, `peptidoform`, `charge`, `protein_group`, +`fragment_name` (str), `quantity` (f64). This is the input for ion-level directLFQ. + +`.parquet` (optional diagnostic, `--out-peak-bounds`, only when +`bound_peak` is on; `quant.rs:273`): `candidate_id`, `lo_rt`, `hi_rt`, `width_s` +(all f64). Not part of the quant contract; it is a view of the integration windows. + +### quant-lfq +Consumes N per-run tables (peptide_quant for maxlfq, fragment_quant for directlfq). +Produces a long-form matrix (`quant.rs:476`): `protein_group` (str), `run` (i32, +0-based input index), `quantity` (f64), `n_features` (i32). + +### align +Consumes one `seed_psms.parquet` per run (`--seeds`, first is the reference). +Produces `alignment.parquet` (`align.rs:113`): `run_id` (u32), `source_rt` (f64), +`reference_rt` (f64), `residual_spread` (f64). The mapping is emitted on a grid of +`grid_n` points per run (`grid_n = 100`, hardcoded in `main.rs:633`). + +### mbr +Consumes an experiment-wide `scored_combined.parquet` (must carry a `source` column) +and one `psms.parquet` per run in `source` order. Produces `.parquet`, one row +per accepted transfer (`mbr_worker.py:254`): `candidate_id`, `source`, `peptidoform`, +`charge`, `protein_group`, `label`, `expected_rt`, `observed_rt`, `rt_delta`, +`transfer_q`. Optionally writes an augmented scored table (`--out-scored`) that lowers +each accepted transfer's `q_value` to its `transfer_q` and adds an `is_transferred` +flag (`mbr_worker.py:272`). + +### report +Consumes `psms_scored.parquet` and optionally the two quant tables. Writes two TSVs. +`peptides.tsv` header (`report.rs:89`): `precursor`, `stripped_sequence`, `charge`, +`protein`, `q_value`, `score`, `quantity`. `proteins.tsv` header (`report.rs:115`): +`protein_group`, `q_value`, `quantity`. + +### audit +Consumes `library_precursors.parquet` (the full search space) plus `psms` +(extract), `competed` (compete), `scored` (rescore). Optionally reads +`.audit.parquet` (in-extract sidecar) for reason refinement. Writes +`candidate_audit.parquet` (16 columns, `audit.rs:177`) and `.metrics.json` +(`audit.rs:213`). + +## How it works + +### quant (`quant.rs:147`, `run`) + +1. Load `psms_scored`. The q-value column to filter on is selected by + `cfg.q_filter` (`quant.rs:160`): `PeptideQ` -> `peptide_q_value`, `PsmQ` -> + `q_value`, `RunPsmQ` -> `run_psm_q`. This choice matters for cross-run quant off + an experiment-wide rescore, where the peptide q is global and would give disjoint + per-run sets (see `QuantQColumn` doc, `config.rs:806`). +2. Group chromatogram rows by `candidate_id` into `cand_rows` (`quant.rs:176`). Rows + whose `frag_name` starts with `ms1_` are MS1 isotope XIC pseudo-traces, not + fragment ions; they are excluded from both peak detection and the top-N sum. +3. **Phase 1** (`quant.rs:194`, only when `cfg.bound_peak`): compute a per-candidate + elution window `(lo_rt, hi_rt, apex_rt)` via `peak_window` (`quant.rs:75`). The + summed XIC across all of a candidate's fragments is built in a `BTreeMap` keyed by + the f32 RT bit pattern, so both the union RT axis and the f64 summation order are + fixed (determinism, plan.md Section 7; non-negative RTs make bit order equal value + order). The apex is chosen by a co-elution rule (`quant.rs:108`): among scans whose + co-eluting nonzero-fragment count is within 1 of the maximum (`thresh = + max_cnt-1`), take the highest summed intensity; fall back to plain summed argmax + only if no scan has a fragment. This rejects a lone tall interferent that a plain + intensity argmax would pick (test `peak_window_apex_prefers_coelution_over_lone_interferent`, + `quant.rs:686`). `peak_bounds` (`features.rs:946`) then walks out from the apex with + `peak_fraction` and `peak_grace`. A collapsed `lo==hi` window is widened to the + adjacent grid scans so `trapezoid_window` never returns a raw height (units bug, + `quant.rs:136`). +4. **Consensus mode** (`quant.rs:210`, `peak_window_mode == Consensus`): peak width is + treated as a near-constant instrument/gradient property. Over confident target + peptides (`pep_q <= reliable_q`) it takes the median left half-width `apex - lo` and + right half-width `hi - apex`, and applies `(apex - ml, apex + mr)` around each + candidate's apex. Requires `>= 20` anchors (`quant.rs:232`), else falls back to the + per-candidate windows. Being global, the same width is used in every run, so fold + changes are preserved. +5. **Phase 2** (`quant.rs:246`): integrate each fragment trace. With `bound_peak` off, + integrate the whole trace with `trapezoid` (`quant.rs:36`). With it on, restrict to + the chosen window via `trapezoid_window` (`quant.rs:51`). Areas are accumulated per + candidate in `areas` and `frag_areas`. +6. **Top-N sum** (`quant.rs:287`): for each PSM that is a target with `pep_q <= + q_threshold`, sort the candidate's area vector descending in place and sum the top + `top_n_fragments` (`quant.rs:297`). The per-candidate vector is sorted by mutable + reference (no clone); a repeated `candidate_id` re-sorts an already-sorted vector + (idempotent). Accumulate per protein group in `per_group`. +7. **Protein rollup** (`quant.rs:327`): per group, sort peptide quantities descending + and apply `RollupMethod` (`TopNSum` = sum of top `top_n_peptides`, `Sum` = sum of + all). Groups are iterated in sorted key order. +8. Optional fragment export (`quant.rs:351`) and peak-bounds diagnostic + (`quant.rs:273`). `ArtifactReport` records params + stats for each table + (`quant.rs:386`). + +Trapezoid math: `trapezoid` sums `dt*(y_i+y_{i+1})/2` in f64 over consecutive RT +samples; a single sample returns its raw intensity (`quant.rs:37`). `trapezoid_window` +first filters to samples with `lo <= rt <= hi`, then calls `trapezoid` on the subset +so the single-sample rule is identical; an empty window integrates to 0. + +**Known limits.** quant re-detects the apex from the summed XIC rather than reusing +the apex `extract`/`features` already chose, so the quant apex and the scoring apex can +differ. There is no quantifiability gate: a confident PSM with only one noisy fragment +still produces a `quantity` (possibly a single-sample height widened to two grid +scans); nothing downstream flags low-evidence quantities beyond `n_fragments_used`. + +### quant-lfq (`quant.rs:421`, `run_lfq_combine`) + +Reads each input table and builds `data: protein_group -> feature_key -> Vec>` +of length N runs (`quant.rs:430`). The feature key is `peptidoform|charge` for MaxLFQ +and `peptidoform|charge|fragment_name` for directLFQ (`quant.rs:439`). Missing entries +stay `None`. `size_factors` (`quant.rs:509`) computes one global size factor per run; +when `normalize != None` every present value is divided by its run factor before +rollup (`quant.rs:453`). For each protein group the feature-by-run matrix is passed to +`lfq_profile` (`quant_lfq.rs:83`) and the per-run abundances are written long-form. + +`lfq_profile` is the MaxLFQ least-squares reconstruction: +- Column sums/counts per sample give the fallback and the anchoring total + (`quant_lfq.rs:88`). Single sample returns the column sum (`quant_lfq.rs:100`). +- For each sample pair `(a,b)`, the median over shared features of `ln(va)-ln(vb)` is + an edge weight (`quant_lfq.rs:105`). Median-of-log-ratios is robust to a minority of + genuinely changing features. +- Connected components of the sample graph are found by union-find (`quant_lfq.rs:125`). + A singleton component falls back to its column sum. +- Each multi-sample component solves a Laplacian normal system `L x = c` with the first + variable fixed at 0 (`solve_fixed`, `quant_lfq.rs:28`, dense Gaussian elimination + with partial pivoting), giving log-abundances. The exp-profile is then scaled so the + component preserves its measured total intensity (`quant_lfq.rs:175`). + +`size_factors` methods (`quant.rs:509`): +- `MedianRatio` (default, DESeq-style): over complete-case features (positive in all + runs) take each run's log2 deviation from the per-feature mean; the run factor is + `2^median` of those deviations. Robust so a spike-in design's real fold changes are + not flattened (test `median_ratio_recovers_global_scale_not_real_changes`, + `quant.rs:648`). +- `Median`: align each run's median log2 intensity to the median of the per-run + medians. +- `None`: all factors 1.0. + +Determinism: medians sort in place, the matrix is iterated in `BTreeMap` key order. +With a single input, `run_lfq_combine` reduces to the per-run sum. + +### align (`align.rs:53`, `run`) + +`confident_rts` (`align.rs:32`) reads a seed table's `base_peptide_id`, `spectrum_q`, +`observed_rt`, `score`, `label`, validates labels, and returns the best-scoring +observed RT per base peptide among confident targets (`q <= q_train`, target only). +The reference is `seeds[0]`. For each run, shared base peptides with the reference give +paired `(this_rt, ref_rt)`. A LOESS map is fit only when there are `>= 4` shared +peptides (`align.rs:85`, `Loess::fit(xs, ys, span=0.4, grid_n)`); the span is hardcoded +at 0.4 and is independent of `rt_im_train.loess_span`. The residual spread is the p95 of +`|ref - loess(this)|` on the shared set (`align.rs:91`); it is what sets how tight an +MBR window can be. The mapping is emitted on `grid_n.max(2)` grid points; the reference +run and any run with too few shared peptides emit the identity map with residual 0 +(the insufficient-anchor guard, `align.rs:85`/`align.rs:101`). This is an experiment- +level stage: with one run it degenerates to identity, and it is not part of the `run` +chain. Real multi-run validation needs a multi-file experiment; only crafted two-run +unit input exercises it. + +### mbr (partially wired Stage D3) + +The Rust side is a gate, not the algorithm. `Cmd::Mbr` (`main.rs:637`): loads config; +bails if `cfg.mbr.strategy == None` (`main.rs:639`); bails if fewer than 2 psms paths +(`main.rs:644`); requires `cfg.mbr.python` (`main.rs:647`); resolves `mbr_worker.py` +relative to the binary; calls `sidecar::run_mbr`. `run_mbr` (`sidecar.rs:136`) joins +the psms paths into a comma-separated `psms_csv` and forwards a fixed set of flags. + +**Which `MbrConfig` knobs are forwarded, and which are NOT.** Forwarded to the worker: +`q_anchor` (`--q-anchor`), `min_anchor_runs` (`--min-anchor-runs`), `q_transfer` +(`--q-transfer`), `consensus_corr_min` (`--consensus-corr-min`, only together with +`--frag-csv` and only when `frag` is non-empty and `consensus_corr_min > 0`, +`sidecar.rs:166`), plus `cfg.rng_seed` as `--seed` and the optional `--out-scored`. +NOT forwarded (present in `MbrConfig`, `config.rs:911`, but dead in the wired path): +- `strategy` beyond the None/not-None gate: `EmpiricalLibrary`, `RtTransfer`, `Full` + are indistinguishable to the worker, which always runs the rescuable transfer tier. +- `rt_window_s` (`config.rs:922`): the worker's `--rt-window` (default 20 s) is only + read by the `--emit-transfer-targets` re-extraction tier, which `run_mbr` never + invokes, so this knob has no effect on the wired path. +- `decoy_transfer` (`config.rs:924`): the worker hardcodes the permuted-RT null; + `ReverseSequence`/`Both` are not implemented in the worker. +- `requant_all` (`config.rs:930`): `Full`-only requantification, unused. + +The worker (`scripts/mbr_worker.py`) implements two tiers: +- **Rescuable transfer** (default path, `mbr_worker.py:156`): for each precursor + confident (`q <= q_anchor`, target) in `>= min_anchor_runs` OTHER runs, sub-threshold + in a target run where it WAS extracted, predict its RT in that run from the median of + the other runs' binned-median-aligned apex RTs (`expected_rt`, `mbr_worker.py:116`; + calibration via `binned_map`, `mbr_worker.py:31`). The false-transfer FDR is a + permuted-RT decoy-transfer null: each candidate is given a shuffled candidate's + predicted RT (`mbr_worker.py:177`). The transfer q-value is standard target/decoy + competition on `rt_delta = |observed - predicted|` (smaller is better), computed as a + running min from the tail (`mbr_worker.py:192`). Accept when `q <= q_transfer`. +- **Fragment-consensus guard** (`mbr_worker.py:209`, M4 enhancement, active only with + `--frag-csv` and `--consensus-corr-min > 0`): reject an accepted transfer whose + observed fragment pattern in the target run has cosine `< corr_min` with the + empirical consensus over its confident runs. Removes RT-concordant interference. +- **Re-extraction tier** (`--emit-transfer-targets`, `mbr_worker.py:126`): for the + ABSENT set (confident elsewhere, not extracted here) emit per-run `run_windows`-format + tables at the tight predicted-RT window plus a permuted-RT decoy target file, to feed + `extract --restrict-candidates --run-windows`. This tier is fully implemented in the + worker but has no Rust CLI plumbing, so it is unreachable from `mumdia mbr`. + +The M5 augmented scored output (`--out-scored`, `mbr_worker.py:272`) requires the +scored table to have a `source` column and matches transfers on `(candidate_id, +source)`. + +### report (`report.rs:49`, `run`) + +Reads the scored table columns `peptidoform`, `charge`, `protein`, `label`, +`peptide_q_value`, `protein_group`, `pg_q_value`, `score`. Builds two quant lookup +maps: peptide quant keyed `(peptidoform, charge)` (`report.rs:61`), protein quant keyed +`protein_group` (`report.rs:71`). Peptides: sort rows by `peptide_q_value` ascending, +keep the first (best-q) row per unique `(peptidoform, charge)`, targets only with +`peptide_q_value <= q_threshold` (`report.rs:82`). The row unit is the precursor +(peptidoform + charge), not the stripped sequence; `strip` (`report.rs:24`) removes a +`DECOY_` prefix and bracketed/parenthesized mod blocks for the `stripped_sequence` +column only, and a separate stripped-sequence count is logged. `q_value` is printed at +6 decimals, `score` at 4, `quantity` via `qcell` (1 decimal, empty on NaN, +`report.rs:39`). Proteins: sort by `pg_q_value`, unique non-empty protein groups, +targets with `pg_q_value <= q_threshold` (`report.rs:117`). Returns `(n_precursors, +n_protein_groups)`. + +### audit (`audit.rs:64`, `run`) + +The search space is every library precursor (`candidate_id`, `peptidoform`, `charge`, +`label`, `protein`). Survivor sets are built as `HashSet` of `candidate_id` from +`psms` (extracted), `competed`, and `scored`; scored also yields `q_by_cid` and +optional `pepq_by_cid` (`audit.rs:88`). `load_extract_reasons` (`audit.rs:51`) reads +the optional `.audit.parquet` sidecar to refine the extract-stage bucket. For +each candidate the earliest rejection reason is assigned along the ladder +(`audit.rs:133`): +- not in `extracted` -> refined from the sidecar + (`NO_FRAGMENT_TRACES`/`NO_VALID_FRAGMENTS`/`PEAK_NOT_SELECTED`/`RT_PRUNED`/ + `WRONG_ISOLATION_WINDOW`) or the generic `NO_PEAK_GROUP` when no sidecar; +- extracted but not in `competed` -> `OUTCOMPETED_BY_DECOY` (decoy) or + `OUTCOMPETED_BY_TARGET` (target); +- competed but `q > q_threshold` -> `FAILED_PRECURSOR_FDR`; +- passes precursor but fails peptide q -> `FAILED_PEPTIDE_FDR` (peptide q falls back to + the precursor gate when absent, `audit.rs:127`); +- else `REPORTED`. + +The waterfall counts per reason and is logged; `.metrics.json` records +`search_space`, `extracted`, `competed`, `reported`, `trace_recall`, and the waterfall +map (`audit.rs:203`). This stage never re-runs compute and never mutates a pipeline +output, so it is safe to run after any search. + +`RejectionReason` (`rejection.rs:19`) is a 17-variant enum with a stable +SCREAMING_SNAKE_CASE `code()` (`rejection.rs:50`), a `stage_order()` ladder position +(`rejection.rs:76`), and `earliest()` to keep the smaller stage (`rejection.rs:106`). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `QuantParams` | quant.rs:19 | inputs/outputs + `&QuantConfig` for the quant stage | +| `trapezoid` | quant.rs:36 | f64 trapezoidal integral of one trace; single point = raw intensity | +| `trapezoid_window` | quant.rs:51 | trapezoid restricted to `[lo,hi]`; empty window = 0 | +| `peak_window` | quant.rs:75 | summed-XIC apex (co-elution rule) + descent-walk window | +| `quant::run` | quant.rs:147 | full quant stage | +| `run_lfq_combine` | quant.rs:421 | build the protein-by-run matrix for `quant-lfq` | +| `size_factors` | quant.rs:509 | per-run normalization factors (MedianRatio/Median/None) | +| `median_sorted` | quant.rs:564 | in-place median helper | +| `lfq_profile` | quant_lfq.rs:83 | MaxLFQ least-squares per-sample profile | +| `solve_fixed` | quant_lfq.rs:28 | dense Laplacian solve, first var fixed at 0 | +| `maxlfq` / `directlfq` | quant_lfq.rs:186 / 193 | granularity-specific wrappers over `lfq_profile` | +| `AlignParams` | align.rs:22 | seeds + q_train + grid_n | +| `confident_rts` | align.rs:32 | best observed RT per confident target base peptide | +| `align::run` | align.rs:53 | reference LOESS RT map per run | +| `run_mbr` | sidecar.rs:136 | build argv and spawn `mbr_worker.py` | +| `binned_map` | mbr_worker.py:31 | monotone binned-median RT calibration | +| `expected_rt` | mbr_worker.py:116 | cross-run predicted RT for a candidate in a run | +| `ReportParams` / `report::run` | report.rs:13 / 49 | TSV writer | +| `strip` | report.rs:24 | stripped sequence from a peptidoform | +| `AuditParams` / `audit::run` | audit.rs:28 / 64 | candidate loss ladder | +| `load_extract_reasons` | audit.rs:51 | optional `.audit.parquet` refinement | +| `RejectionReason` | rejection.rs:19 | earliest-loss category enum | + +## Configuration + +**`QuantConfig`** (`config.rs:832`, defaults at `config.rs:859`), read by `quant`: +- `q_threshold` (0.01): peptide q cutoff for inclusion in the quant tables. +- `top_n_fragments` (3): fragments summed per peptidoform. +- `top_n_peptides` (3): peptides summed per protein group under `TopNSum`. +- `rollup` (`TopNSum`): `TopNSum` or `Sum` (`config.rs:745`). +- `bound_peak` (true): integrate over the detected window vs the whole trace. +- `peak_fraction` (1/6): descent threshold, fraction of apex height. +- `peak_grace` (1): consecutive sub-threshold scans bridged during the walk. +- `peak_window_mode` (`PerCandidate`): `PerCandidate` or `Consensus` (`config.rs:760`). +- `reliable_q` (0.001): confident-set cutoff calibrating the consensus half-widths. +- `q_filter` (`PeptideQ`): which q column to filter on; `PeptideQ`/`PsmQ`/`RunPsmQ` + (`config.rs:816`). Use `PsmQ`/`RunPsmQ` for cross-run quant off an experiment-wide + rescore. + +**`quant-lfq`** takes no config struct; `--method` (maxlfq/directlfq) and `--normalize` +(median_ratio/median/none, parsed by `NormalizeMethod::from_token`, `config.rs:796`) +are CLI args. Default normalize is `median_ratio`. + +**align** reads `rt_im_train.q_train` (0.01, `config.rs:366`) as the anchor cutoff; +`grid_n` is hardcoded to 100 in `main.rs:633` and the LOESS span to 0.4 in `align.rs:85`. + +**`MbrConfig`** (`config.rs:911`, defaults at `config.rs:935`): `strategy` (`None`; +`MbrStrategy` at `config.rs:883`), `q_anchor` (0.01), `min_anchor_runs` (2), +`q_transfer` (0.01), `rt_window_s` (20.0, unwired), `decoy_transfer` (`PermutedRt`, +`DecoyTransfer` at `config.rs:902`, unwired), `consensus_corr_min` (0.0), `requant_all` +(false, unwired), `python` (None, required when `strategy != None`). See the mbr +section for exactly which of these reach the worker. + +**report** takes `q_threshold` (`--q`, default 0.01) and optional quant table paths as +CLI args; no config struct. + +**audit** takes `q` (0.01), `run_id` ("run"), `entrapment_substr` ("") as CLI args. + +The config was recently pruned of dead fields. The `MbrConfig` knobs noted as +"unwired" above are intentionally documented as such: they still exist in the struct +but do not affect the wired `mumdia mbr` path. + +## Invariants, determinism, gotchas + +- **Determinism.** quant uses `BTreeMap` for candidate iteration, the RT union axis, + and f64 summation order; quant-lfq iterates `BTreeMap` keys and sorts medians in + place; align iterates via `HashMap` but writes a fixed grid. The `mbr_worker.py` + permutation is seeded from `cfg.rng_seed` (`--seed`). The DeepLC fine-tune elsewhere + is nondeterministic, but none of these stages depend on it. +- **quant units bug guard.** A collapsed `lo==hi` window would make `trapezoid_window` + return a raw height (intensity, not intensity*seconds), mixing units against + broad-peak peptides; `peak_window` widens to two grid scans to prevent this + (`quant.rs:136`, test `peak_window_never_collapses_to_single_sample`). +- **quant apex is re-detected**, not inherited from `extract`/`features`; the quant apex + and the scoring apex can differ. There is no quantifiability gate. +- **MS1 traces excluded.** `ms1_*` chromatogram rows are precursor channels and never + enter peak detection or the top-N sum (`quant.rs:178`). +- **report row unit.** `peptides.tsv` rows are precursors (peptidoform + charge), not + stripped sequences; the returned count and the header reflect that (`report.rs:87`). + A separate stripped-sequence count is only logged. +- **audit artifact resolution.** `peak_generated`, `peak_selected`, and + `traces_extracted` are all set from the same `traces` flag (`audit.rs:167`) because + the artifacts only record presence in `psms`; the in-extract sidecar is the only way + to split "no traces" from "traces but no accepted peak". +- **audit `reported` vs `REPORTED`.** The `reported` bool column is set from + `passed_prec` alone (`audit.rs:173`), while the `REPORTED` rejection reason additionally + requires the peptide gate. A candidate can therefore have `reported=true` yet + `rejection_reason=FAILED_PEPTIDE_FDR`. Treat `rejection_reason` as authoritative. +- **audit reason coverage.** Only the extract/compete/FDR ladder reasons are produced + by `audit.rs`. Earlier codes (`PEPTIDE_NOT_GENERATED`, `MODIFICATION_NOT_ALLOWED`, + `CHARGE_OUT_OF_RANGE`, `PRECURSOR_MZ_OUT_OF_RANGE`, `CANDIDATE_CAP_REACHED`, + `REMOVED_DURING_REPORTING`) exist in the enum but are only reachable via the sidecar + or not at all in the current chain. +- **align / mbr need >=2 runs.** align degenerates to identity on one run; `mumdia mbr` + hard-bails on `<2` psms paths (`main.rs:644`). Neither is in the single-run `run` + chain; `run` invokes only `quant` (with fragment export, no peak-bounds) and `report` + (`run.rs:293`/`309`). +- **quant-lfq single input** reduces to the per-run sum; MaxLFQ/directLFQ have only + synthetic-matrix unit tests, no real multi-run validation. + +## How to extend / modify + +- **New rollup or normalization strategy:** add a variant to `RollupMethod` + (`config.rs:745`) or `NormalizeMethod` (`config.rs:779`) and a match arm in + `quant::run` (`quant.rs:333`) or `size_factors` (`quant.rs:514`). Keep defaults + conservative and iterate `BTreeMap` in key order for determinism. +- **Quantifiability gate:** filter the top-N sum loop (`quant.rs:290`) on + `n_fragments_used` or a minimum window width; emit the reason so `report` can surface + low-evidence quantities. +- **Wire the MBR strategy tiers:** thread `cfg.mbr.strategy` past the None gate in + `main.rs:639` and forward `rt_window_s`/`decoy_transfer`/`requant_all` in + `run_mbr` (`sidecar.rs:159`). The re-extraction tier already exists in the worker + (`--emit-transfer-targets`); it needs a Rust subcommand that then calls + `extract --restrict-candidates --run-windows` and re-scores. Implement the + `ReverseSequence`/`Both` nulls in the worker before honoring `decoy_transfer`. +- **Consume `alignment.parquet` in MBR** instead of the worker's internal + `binned_map`, so the RT map is computed once and shared; align already records the + p95 residual that should set `rt_window_s`. +- **Add an audit reason:** extend `RejectionReason` (`rejection.rs:19`) with a + `code()`, `stage_order()`, and a branch in the ladder (`audit.rs:133`); prefer + emitting it from the in-extract sidecar so `load_extract_reasons` can pass it through. +- **New quant artifact column:** add the `Col` in the relevant `write_table` + (`quant.rs:314`/`341`/`369`), bump the schema version in + `mumdia-core/src/schema.rs`, and update the consumer (`report.rs`, `run_lfq_combine`). diff --git a/docs/13_sidecars.md b/docs/13_sidecars.md new file mode 100644 index 0000000..6505bb1 --- /dev/null +++ b/docs/13_sidecars.md @@ -0,0 +1,432 @@ +# Python sidecars (the 10 scripts) + conda envs +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +MuMDIA is a pure-Rust engine with deterministic native fallbacks for every ML +step, so it runs with zero Python. The sidecars are the opt-in path to real +predictors and rescorers that raise identification counts over the native +defaults. Each sidecar is a standalone Python worker invoked as a subprocess over +a **positional-CLI file contract** (plan.md Section 3.2): the Rust caller writes +an input file (Parquet or PIN), runs `python `, and reads an +output Parquet keyed by `id` or `candidate_id`. There is no JSON request file, no +long-lived server, and no stdin/stdout data channel; the files on disk are the +entire interface. This means any sidecar can be swapped for a native Rust +implementation (or a different tool) without touching the callers, and a broken +sidecar can fall back to the native path (`rescore.strict = false`). + +The 10 scripts split into four groups: **predictors** (`ms2pip_worker`, +`deeplc_worker`, `deeplc_finetune`) feed the run-independent library; +**rescorers** (`mokapot_worker`, `nn_rescore_worker`, `entrapment_worker`) score +the competed PSMs in Stage F; **MBR** (`mbr_worker`) transfers identifications +across runs (Stage D3); and the **DIA-NN recipe** (`import_diann_lib`, +`make_reverse_decoys`, `make_shift_decoys`) is an offline, one-time toolchain to +convert a user-produced DIA-NN library into MuMDIA's target+decoy library schema. + +## Files + +| path | role | +|---|---| +| `scripts/ms2pip_worker.py` | Predictor: MS2PIP b/y fragment intensities per peptidoform+charge | +| `scripts/deeplc_worker.py` | Predictor: DeepLC iRT per peptidoform (uncalibrated) | +| `scripts/deeplc_finetune.py` | Predictor: transfer-learn DeepLC on this run's seed, rewrite library iRT | +| `scripts/mokapot_worker.py` | Rescorer: mokapot brew over a PIN (model env-switchable: nn/logreg/xgb/percolator) | +| `scripts/nn_rescore_worker.py` | Rescorer: PyTorch semi-supervised MLP over a PIN, in-memory or streaming memmap | +| `scripts/entrapment_worker.py` | Rescorer: GBM/NN on real-target-vs-spike-in negatives, out-of-fold by base peptide | +| `scripts/mbr_worker.py` | MBR (Stage D3): cross-run RT transfer + permuted-RT decoy-transfer FDR | +| `scripts/import_diann_lib.py` | Recipe: DIA-NN fragment-level parquet -> MuMDIA target `lib_precursors`+`lib_fragments` | +| `scripts/make_reverse_decoys.py` | Recipe: reverse-sequence decoys with no-target-overlap invariant | +| `scripts/make_shift_decoys.py` | Recipe: fragment-shift (CH2) decoys, DIA-NN-style terminal shift | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | Rust clients: `resolve_script`, `run_ms2pip`, `run_deeplc`, `run_deeplc_finetune`, `run_mbr`, `run_worker` | +| `rust/mumdia/crates/mumdia/src/stages/predict_frag.rs` | Call sites for MS2PIP + DeepLC (Stage C) | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | Call site for DeepLC fine-tune (between search-seed and rt-im-train) | +| `rust/mumdia/crates/mumdia/src/stages/rescore.rs` | Call sites for mokapot/nn_torch (PIN) + entrapment (Parquet) sidecars | +| `rust/mumdia/crates/mumdia/src/main.rs` | Call site for MBR (`Cmd::Mbr`) + `doctor` env probe | +| `env/docker-rescore.yml` | Docker env `rescore`: mokapot 0.10.0 + ms2pip 4.0.0.dev9 (py3.11) | +| `env/docker-deeplc.yml` | Docker env `deeplc`: DeepLC 4.0 multitask (pinned commit) + CPU torch (py3.11) | +| `env/mumdia-rescore.yml` | Minimal portable env for the mokapot logreg rescore path (py3.12) | + +## Inputs and outputs + +Each sidecar's on-disk contract, with the exact column schema read/written by the +code. + +**ms2pip_worker** (`sidecar.rs:37` `run_ms2pip`) +- IN `ms2pip_in.parquet`: `id` u32, `peptidoform` str (ProForma), `charge` i32. +- OUT `ms2pip_out.parquet`: `id` u32, `ion_type` str (`"b"`/`"y"`), `ordinal` i32 + (1-based), `intensity` f32 (linear). Rust folds this into + `HashMap>` (`sidecar.rs:66-73`). + +**deeplc_worker** (`sidecar.rs:77` `run_deeplc`) +- IN `deeplc_in.parquet`: `id` u32, `peptidoform` str. +- OUT `deeplc_out.parquet`: `id` u32, `predicted_rt` f32. Rust returns + `HashMap` (`sidecar.rs:100`). + +**deeplc_finetune** (`sidecar.rs:106` `run_deeplc_finetune`) +- IN `` = `fragment_library_precursors.parquet` (needs `peptidoform`, + `predicted_irt`); `` = seed PSMs (`peptidoform`, `label`, `spectrum_q`, + `observed_rt`). +- OUT `` = `fragment_library_precursors_ft.parquet`: the input table with + the `predicted_irt` column replaced (`deeplc_finetune.py:156-159`). Same schema, + values rewritten. + +**mokapot_worker** / **nn_rescore_worker** (`rescore.rs:563` `run_pin_sidecar`) +- IN `rescore.pin`: Percolator tab-separated. Fixed columns + `SpecId Label ScanNr ExpMass CalcMass Peptide Proteins` + (`rescore.rs:585-601`). `SpecId = psm_`, `ScanNr = ` where `i` is the + unique flat row index (NOT `candidate_id`, which repeats across runs); + `ExpMass=CalcMass=precursor_mz`; `Label` +1 target / -1 decoy. +- OUT `rescore_sidecar_out.parquet`: `candidate_id` u32 (echoes the SpecId tail = + row index), `score` f64, `q_value` f64 (written as zeros; Rust computes q). Rust + maps `score` back by row index (`rescore.rs:625-630`). + +**entrapment_worker** (`rescore.rs:503` `run_entrapment_gbm`) +- IN `entrapment_in.parquet`: `row_id` u32 (unique flat index), `candidate_id` + u32, `base_peptide_id` u32, `is_entrapment` i32 (0/1), `is_decoy` i32 (0/1), + then one f64 column per feature (`rescore.rs:521-534`). +- OUT `entrapment_out.parquet`: `row_id` u32, `candidate_id` u32, `score` f64. + Rust maps back by `row_id` (`rescore.rs:549-555`). + +**mbr_worker** (`sidecar.rs:136` `run_mbr`) +- IN ``: experiment-wide scored table, columns read = + `candidate_id, source, label, q_value, peptidoform, charge, protein_group` + (`mbr_worker.py:81-82`). ``: comma-joined per-run psms.parquet paths + in `source` order, each read for `candidate_id, apex_rt` (`mbr_worker.py:96`). + Optional `--frag-csv`: per-run fragment_quant paths + (`candidate_id, fragment_name, quantity`). +- OUT `.parquet`: one row per accepted transfer with + `candidate_id, source, peptidoform, charge, protein_group, label, expected_rt, + observed_rt, rt_delta, transfer_q` (`mbr_worker.py:254-265`). Optional + `--out-scored` writes the scored table with accepted transfers' `q_value` + lowered to `transfer_q` and an `is_transferred` flag added. Optional + `--emit-transfer-targets` writes per-run `run_windows`-format tables + (`candidate_id, rt_pred_cal, rt_lo, rt_hi, im_*`) plus a permuted-RT decoy file + for the re-extraction tier (`mbr_worker.py:142-151`). + +**import_diann_lib** (offline; no Rust caller) +- IN ``: DIA-NN fragment-level speclib (columns + `Decoy, Fragment.Loss.Type, Fragment.Type, Modified.Sequence, Precursor.Charge, + Precursor.Mz, RT, Stripped.Sequence, Product.Mz, Relative.Intensity, + Fragment.Series.Number, Fragment.Charge, Protein.Names`/`Protein.Ids`). +- OUT ``: `candidate_id, peptidoform_id, base_peptide_id, + peptidoform, charge, precursor_mz, predicted_irt, label(="target"), protein, + n_fragments`. ``: `candidate_id, mz, predicted_intensity, name, + ion_type, ordinal, frag_charge` (`import_diann_lib.py:59-83`). + +**make_reverse_decoys** / **make_shift_decoys** (offline; no Rust caller) +- IN/OUT the same precursor+fragment schema as `import_diann_lib`, reading a + target-only (or target-half) library and emitting a target+decoy library + re-sorted by `precursor_mz` with contiguous re-indexed `candidate_id`. + +## How it works + +### Predictors (Stage C, and one Stage-B step) + +**MS2PIP** (`predict_frag.rs:305-315`, worker `ms2pip_worker.py`). Selected by +`predict_frag.predictor = "ms2pip"`. `assign_intensities` collects one `id` per +`Raw` candidate, its peptidoform and charge, calls `run_ms2pip`, then for each +fragment looks up `(ion_byte, ordinal)` for **charge-1** fragments only; charge-2 +fragments fall back to the native heuristic (`predict_frag.rs:320-338`). The +worker builds `psm_utils.PSMList` in 100k-row chunks, calls +`ms2pip.predict_batch(model, processes=min(8, cpu_count))`, and converts MS2PIP's +log2 intensities to linear via `2**x - 0.001` clipped at 0 +(`ms2pip_worker.py:52-53`). Ordinals are emitted 1-based +(`ms2pip_worker.py:57`). The `__main__` guard makes the Windows `spawn` start +method safe for multiprocessing. + +**DeepLC predict** (`predict_frag.rs:254-291`, worker `deeplc_worker.py`). +Selected by `predict_frag.rt_predictor = "deeplc"`. `assign_rt` deduplicates by +peptidoform (RT is charge-independent, `predict_frag.rs:262-271`), calls +`run_deeplc`, and writes `r.irt`. Peptidoforms with no returned iRT are anchored +at `0.0` with a warning (`predict_frag.rs:278-289`; this is the "unmatched +peptidoforms silently get iRT 0.0" foot-gun noted in CLAUDE.md). The worker calls +`deeplc.predict` in 200k chunks and, when the multitask model returns an +ensemble matrix `(N, n_models)`, averages across models (`deeplc_worker.py:34-35`). +Predictions are uncalibrated; rt-im-train's per-run LOESS/linear maps them onto +observed RT. + +**DeepLC fine-tune** (`run.rs:187-208`, worker `deeplc_finetune.py`). Wired into +`run` only when `rt_im_train.finetune_deeplc = true`; runs **between** +search-seed and rt-im-train, rewriting `predicted_irt` in a copy of the library +(`fragment_library_precursors_ft.parquet`) that rt-im-train and extract then read. +Algorithm: (1) build the reference from confident **target** seed PSMs with +`spectrum_q <= q_train` and a standard-AA sequence (`deeplc_finetune.py:99-104`); +(2) auto-scale batch size so each epoch runs >= ~30 gradient steps, clamped to +[16, 512] (`deeplc_finetune.py:114-117`) because a fixed 512 underfits a small +(~4k) E.coli seed; (3) `deeplc.finetune(ref_psms, train_kwargs)` transfer-learns +the weights (`deeplc_finetune.py:128`); (4) predict every unique standard +peptidoform on its **`DECOY_`-stripped** underlying sequence so decoys land on the +same iRT scale as targets (`deeplc_finetune.py:44-45, 135-156`). The long +docstring and the thread-cap block at the top exist to prevent an intermittent +machine crash: numpy's OpenBLAS (GNU OpenMP) and torch's Intel OpenMP coexist +under `KMP_DUPLICATE_LIB_OK=TRUE`, and without pinning `OMP/MKL/OPENBLAS` to 1 +thread and bounding torch's pool, the two full thread pools oversubscribe the CPU +during the backward pass (`deeplc_finetune.py:6-28, 82-91`). `--device cuda` +sidesteps this entirely by moving compute off the CPU pools. + +### Rescorers (Stage F) + +**mokapot** and **nn_torch** share the exact PIN contract via `run_pin_sidecar` +(`rescore.rs:563-631`). Rust concatenates the competed feature tables, writes the +PIN, spawns the worker, and passes `MUMDIA_NN_FOLDS/ITERS/TRAIN_FDR` from +`rescore.{folds,num_iter,train_fdr}` as env vars (`rescore.rs:614-616`) so the +report's recorded params match what ran; mokapot ignores those three. On success +the classifier label and `model_identity` are recorded; on failure the path falls +back to `native_scores` unless `rescore.strict = true` (`rescore.rs:105-134`). + +- `mokapot_worker.py` reads the PIN with `mokapot.read_pin`, builds a model + chosen by `MUMDIA_RESCORE_MODEL` (`make_model`, `mokapot_worker.py:34-97`: + `nn` -> sklearn `MLPClassifier`; `logreg` -> `LogisticRegression`; `xgb` -> + `XGBClassifier`; `percolator`/`linear`/`svm` -> mokapot's default, `model=None`), + runs `mokapot.brew(..., rng=0, max_workers=MUMDIA_MOKAPOT_WORKERS)`, and prefers + mokapot's **out-of-fold** confidence scores (each PSM scored by the fold that + did not train on it, `mokapot_worker.py:135-156`). It falls back to averaging + all fold models over all rows only if the confidence API differs; that fallback + is not truly OOF and inflates apparent sensitivity, so it is a last resort. + Every PSM (targets and decoys) is scored, `SpecId` tail parsed to + `candidate_id` (`mokapot_worker.py:167`). +- `nn_rescore_worker.py` implements the semi-supervised scheme itself in PyTorch. + Fold assignment is `md5(stripped_peptide) % FOLDS` (`nn_rescore_worker.py:123`), + so peptides never leak across folds and the split is deterministic. Per fold it + initialises from the single best feature+sign (by targets at `TRAIN_FDR` on a + sample, `nn_rescore_worker.py:175-188`), then iterates {recompute target-decoy q + on the training folds -> targets at `q<=TRAIN_FDR` positive, all decoys negative + -> train MLP from scratch -> rescore} for `ITERS` rounds, then scores the + held-out fold (`one_pass`, `nn_rescore_worker.py:233-253`). Two feature + backends behind one accessor `get`: in-memory (median/IQR standardisation, + `:127-141`) or a disk-backed float32 **memmap** streamed in `MUMDIA_NN_CHUNK` + chunks with mean/std accumulated in one text pass (`:143-173`), auto-selected + when the PIN exceeds `MUMDIA_NN_STREAM_GB` (4 GB). This is what makes an + experiment-wide multi-run rescore tractable: the full PIN never lives in RAM. + `tda_q` (`:77-87`) is the shared q formula `(decoys+1)/max(1,targets)`, running + min from the tail. Seeds are ensembled by averaging rank-normalised OOF scores + (`:255-262`). + +**entrapment** (`rescore.rs:503-556`, worker `entrapment_worker.py`). Selected by +`rescore.classifier = "entrapment"` with `rescore.entrapment_marker` set and +`rescore.python` present; otherwise it falls back to a native linear entrapment +rescorer or `native_tda` (`rescore.rs:143-191`). `classify_entrapment` +(`rescore.rs:411-438`) marks a target as entrapment when its protein contains the +marker, does not contain `entrapment_exclude`, and matches none of +`entrapment_contaminant_markers`. The worker trains real-target (positive) vs +spike-in (negative), decoys excluded from training (`entrapment_worker.py:80-83`), +out-of-fold with `GroupKFold` grouped by `base_peptide_id` +(`entrapment_worker.py:93-101`); a final model fit on all non-decoy PSMs scores +decoys and any single-class-fold gaps (`:103-108`). Model is `gbm` +(`HistGradientBoostingClassifier`, `early_stopping=False` so `random_state=0` is +reproducible) or `nn` (StandardScaler + MLP pipeline) via +`MUMDIA_ENTRAPMENT_MODEL` (`entrapment_worker.py:28-60`). The rationale: spike-in +negatives experience the same chimeric DIA interference as real targets, so a +flexible model helps (AUC ~0.97 vs ~0.62 on in-silico decoys), unlike the linear +default which is all decoys can support. + +### MBR (Stage D3) + +`mbr_worker.py` (`main.rs:637-657` `Cmd::Mbr`, `sidecar.rs:136` `run_mbr`). +Requires `mbr.strategy != none`, `>= 2` runs, and `mbr.python` +(`main.rs:639-649`). Reads the experiment-wide scored table and per-run apex RTs. +It builds per-run to-reference / from-reference RT maps by monotone binned-median +calibration against run 0 (`binned_map`, `mbr_worker.py:31-43`, needs >= 200 +shared anchors else identity). For a precursor confident (target, `q<=q_anchor`) +in `>= min_anchor_runs` OTHER runs but sub-threshold in a target run where it was +still extracted (the **rescuable** tier), it predicts RT as the from-ref map of +the median of the other runs' to-ref-mapped apex RTs (`expected_rt`, +`mbr_worker.py:116-121`). The false-transfer FDR uses a **permuted-RT +decoy-transfer null**: each candidate is assigned a shuffled candidate's predicted +RT, and transfer q is standard target/decoy competition on `|observed - predicted|` +(`mbr_worker.py:190-199`). An optional fragment-consensus cosine guard +(`--frag-csv`/`--consensus-corr-min`) rejects RT-concordant interference +(`mbr_worker.py:209-241`). `--emit-transfer-targets` instead emits per-run +`run_windows` for the **absent** set (confident elsewhere, not extracted here) so +`extract --restrict-candidates --run-windows` can re-extract them (the +re-extraction tier). Note the MBR sidecar is validated as a prototype but Stage +D3 is a stub in the engine (config hooks only; not in the `run` chain). + +### DIA-NN recipe (offline, license-clean) + +Run once by the user, who must hold their own DIA-NN license (MuMDIA ships no +DIA-NN). `import_diann_lib.py` filters to targets, b/y no-loss fragments, and +peptides carrying only Carbamidomethyl/Oxidation (the only mapped mods, +`import_diann_lib.py:28-37`), rewrites `(UniMod:4/35)` to ProForma bracket names, +sorts precursors by m/z with a stable mergesort and assigns contiguous +`candidate_id` (`:52-56`), and preserves species-flagged protein names for the +ProteoBench metric. Then either decoy builder adds the null population: +`make_reverse_decoys.py` reverses each target keeping the C-terminal residue, +recomputes the real b/y m/z of the reversed sequence from a residue-mass table +(validated against the library's own target m/z to < 5 ppm at the 99th +percentile, `make_reverse_decoys.py:88-98`), and enforces a hard no-overlap +invariant: any reversed stripped sequence colliding with a real target (palindrome +or reverse-equals-another-target) is re-scrambled by a per-peptide-seeded +Fisher-Yates, dropped after `MAX_TRIES=30`, and a final assertion requires +`decoy_stripped ∩ target_stripped == {}` (`:100-145`). `make_shift_decoys.py` is +the alternative: copy intensities+iRT, keep precursor m/z, shift b-ion m/z by +`-DELTA` and y-ion by `+DELTA` where `DELTA = 14.01565 Da` (one CH2), net +precursor shift zero (`make_shift_decoys.py:17, 38-44`). Both concatenate +target+decoy, re-sort by `precursor_mz`, reassign contiguous `candidate_id`, and +re-sort fragments by `candidate_id` so the fragment index's contiguous-range +precondition holds (`make_reverse_decoys.py:132-140`, `make_shift_decoys.py:47-55`). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `resolve_script` | `sidecar.rs:18` | Resolve a worker path: CWD-relative dir, then `/`, then `/scripts`, else CWD-relative fallback | +| `run_worker` | `sidecar.rs:174` | Spawn `python