Add cross-dataset algorithm ranking visualization scripts - #84
Open
BioGeek wants to merge 4 commits into
Open
Conversation
|
Warning Review limit reached
More reviews will be available in 36 minutes and 42 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (6)
📒 Files selected for processing (9)
📝 WalkthroughWalkthroughThis PR adds a comprehensive benchmarking analysis toolkit with one data-loading utility module, one report generator, and 16 standalone plotting scripts that visualize algorithm performance across datasets using various metrics and visualization techniques. ChangesBenchmark Analysis and Visualization Suite
🎯 3 (Moderate) | ⏱️ ~25 minutes
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 14
🧹 Nitpick comments (1)
scripts/plot_confidence_calibration.py (1)
34-35: ⚡ Quick winHarden curve parsing for non-finite tokens (but downgrade severity)
plot_confidence_calibration.py(and the same pattern inplot_db_agreement_at_5pct_fdr.py/plot_precision_at_coverage.py) usesast.literal_evalonrow["coverage"]androw["metric"], which would be brittle if future CSVs containednan/inftokens. In the currentresults/*/peptide_precision_plot_data.csvinputs,nan/inf/-infdo not appear, so plotting shouldn’t fail today.Optional: align these scripts’ parsing with the existing approach in
scripts/plot_rt_sa_analysis.pyandscripts/plot_radar_chart.py(regex-sanitize +json.loads) for consistency and future-proofing.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/plot_confidence_calibration.py` around lines 34 - 35, The parsing using ast.literal_eval on row["coverage"] and row["metric"] is brittle to non-finite tokens; change the logic in plot_confidence_calibration.py so that the cov/prec parsing first tries ast.literal_eval(row["coverage"]) / ast.literal_eval(row["metric"]) and on failure (ValueError/SyntaxError) falls back to a sanitized JSON parse similar to scripts plot_rt_sa_analysis.py and plot_radar_chart.py: sanitize the string with a regex to replace/normalize "nan"/"inf"/"-inf" to null or a safe numeric sentinel, then json.loads the sanitized string; ensure this fallback is used for both cov and prec and that exceptions are caught and logged at a lower severity rather than crashing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/generate_benchmark_report.py`:
- Around line 378-380: The code writes the report to out_path (set to
os.path.expanduser("~/Downloads/benchmark_report_updated.md")) without ensuring
the parent directory exists; update the block that opens and writes md to first
resolve out_path, compute its parent directory (e.g.,
os.path.dirname(out_path)), create that directory if missing using
os.makedirs(..., exist_ok=True) after expanduser, and then proceed to open
out_path and write md (keep variable names out_path and md unchanged).
- Around line 351-352: The summary currently multiplies by 2 and assumes both
peptide metrics exist for every dataset; instead compute the denominator by
summing the actual present metric files per dataset. Replace the expression 2 *
sum(1 for ds in all_datasets if load_auc_for_dataset(ds,
'peptide_precision_plot_data.csv')) with something like sum((1 if
load_auc_for_dataset(ds,'peptide_precision_plot_data.csv') else 0) + (1 if
load_auc_for_dataset(ds,'peptide_aa_precision_plot_data.csv') else 0) for ds in
all_datasets), using the existing all_datasets and load_auc_for_dataset symbols
and keep the rest of the out.append string unchanged.
- Around line 218-225: The loop that finds v122_rank stops at the first
"instanovo" entry unconditionally, which hides later v1.2.2 occurrences; update
the loop over ranked so it only breaks when ver == v122_label (i.e., when you've
found v1.2.2) and otherwise continue searching through subsequent entries; set
v122_rank = i when ver == v122_label and break, but do not break for other
"instanovo" versions so v122_rank will be correctly detected if present later.
In `@scripts/load_ranking_data.py`:
- Around line 31-34: If rows can be empty, guard the block that builds and
groups the DataFrame: after creating df = pd.DataFrame(rows) check if df is
empty (or if the required columns "dataset", "algorithm", "auc" are missing) and
if so return an empty DataFrame with the expected schema (columns
["dataset","algorithm","auc"]) and reset_index(drop=True); otherwise proceed
with df = df.loc[df.groupby(["dataset","algorithm"])["auc"].idxmax()] and return
the selected columns. Ensure you reference the existing variables/expressions
(rows, pd.DataFrame(rows), df.groupby([...])["auc"].idxmax()) when adding the
guard so behavior is unchanged for non-empty input.
- Around line 24-29: The code currently builds rows with direct dict access
("algorithm", "version", "auc") which will raise KeyError on malformed CSV rows;
update the logic around the rows.append (the block that constructs each row for
the rows list) to use row.get(...) for "algorithm", "version", and "auc",
validate that none are None/empty and that "auc" can be converted to float, and
skip/log any bad rows instead of appending them (retain "dataset" from the
existing variable). Ensure the validation is done just before appending so
malformed rows are omitted and the loader does not crash.
In `@scripts/plot_aa_peptide_gap.py`:
- Around line 140-142: The save block can raise FileNotFoundError if the
"plots/" directory doesn't exist; before calling fig.savefig(out, ...), ensure
the parent directory of out exists (e.g., use os.makedirs(os.path.dirname(out),
exist_ok=True) or Path(out).parent.mkdir(parents=True, exist_ok=True)) so
fig.savefig and the print(f"Saved {out}") succeed; locate the save logic where
out is defined and fig.savefig is called and add the directory-creation step
immediately before saving.
In `@scripts/plot_algorithm_agreement.py`:
- Around line 29-39: The corr_matrix may contain NaNs which will break
clustering; after computing corr_matrix in the function that builds and returns
the DataFrame, ensure the matrix is symmetric, replace non-finite values and
NaNs with finite defaults, and set diagonals to 1 before returning. Concretely:
symmetrize corr_matrix (e.g., average with its transpose), replace
np.nan/np.isfinite issues with a finite value (0 is a safe neutral correlation),
and force the diagonal entries to 1 so the returned pd.DataFrame(index=algos,
columns=algos) contains only finite values usable by clustermap; update the
return to use this cleaned corr_matrix.
In `@scripts/plot_category_performance.py`:
- Around line 70-99: Guard against an empty keep_cats before computing n_rows
and creating subplots: if keep_cats is empty (no category with >= 2 datasets)
then avoid proceeding — log/print a message and return early (or exit) so you
don't compute n_rows==0 or run the plotting loop; specifically, check keep_cats
right before n_cats = len(keep_cats) and bail out if empty. This also prevents
using an undefined loop index i and calling axes[j] when axes may be empty;
alternatively ensure i is initialized (e.g., i = -1) only if you prefer to
continue, but the clean fix is the early return when keep_cats is empty. Ensure
references to keep_cats, n_rows, axes, i and df_filtered are handled
accordingly.
In `@scripts/plot_dataset_clustering.py`:
- Around line 44-46: The script currently saves to
"plots/dataset_clustering.png" without ensuring the "plots" directory exists,
causing failures on clean checkouts; before calling g.savefig(out, ...) ensure
the directory for the path in variable out exists (e.g., use os.makedirs or
pathlib.Path(out).parent.mkdir with exist_ok=True) so the directory is created
if missing, then call g.savefig(out, dpi=150, bbox_inches="tight") and print the
same message; update the block around the out variable and g.savefig call
accordingly.
In `@scripts/plot_instanovo_version_improvement.py`:
- Line 50: Replace the Unicode minus (U+2212) used in axis label strings with
the ASCII hyphen-minus '-' to avoid RUF001; update the calls to ax.set_xlabel
that contain "AUC improvement (v1.2.2 − v1.1.2)" (and the other similar label
later in the file) to use "AUC improvement (v1.2.2 - v1.1.2)" instead, ensuring
both occurrences use the ASCII '-' character.
- Around line 100-102: The save step uses out and fig.savefig but doesn't ensure
the target directory exists; before calling fig.savefig(out, ...) create the
parent directory for out (e.g., using Path(out).parent.mkdir(parents=True,
exist_ok=True) or os.makedirs) so saving won't fail when "plots/" is missing;
update the code around the out/fig.savefig lines to create the directory then
call fig.savefig and print as before.
In `@scripts/plot_radar_chart.py`:
- Around line 100-121: Compute per-metric min/max from the collected lists
(all_pep, all_aa, all_rt_inv, all_sa), normalize each metric to 0–1 (for RT
first compute rt_inv = 1 - rt for all_algos, then normalize rt_inv), and use
those normalized values when building the per-algo values list (the list
currently created in the loop that uses pep[algo], aa[algo], 1 - rt[algo],
sa[algo]); keep the values += values[:1] step and plotting calls (ax.plot,
ax.fill) but replace raw medians with the normalized equivalents so plotted
points are within 0–1 and comparable across metrics (use the existing names
all_pep/all_aa/all_rt_inv/all_sa, angles, metrics, axes, and all_algos to locate
where to change).
In `@scripts/plot_ranking_boxplot.py`:
- Around line 42-43: The save can fail if the target directory doesn't exist;
before calling fig.savefig(out, dpi=150) ensure the directory for out is created
(use the dirname of the out string and call os.makedirs(..., exist_ok=True) or
pathlib.Path(...).mkdir(parents=True, exist_ok=True)); add the
directory-creation logic just before the fig.savefig call so plots/ is created
on fresh checkouts/CI.
In `@scripts/plot_ranking_cd.py`:
- Around line 26-27: The studentized-range quantile q from
stats.studentized_range.ppf is missing the required division by √2 for the
Nemenyi critical difference; update the calculation so q =
stats.studentized_range.ppf(1 - alpha, k, np.inf) is divided by np.sqrt(2) (or
math.sqrt(2)) before returning q * np.sqrt(k * (k + 1) / (6 * n)), keeping the
same alpha, k, n variables.
---
Nitpick comments:
In `@scripts/plot_confidence_calibration.py`:
- Around line 34-35: The parsing using ast.literal_eval on row["coverage"] and
row["metric"] is brittle to non-finite tokens; change the logic in
plot_confidence_calibration.py so that the cov/prec parsing first tries
ast.literal_eval(row["coverage"]) / ast.literal_eval(row["metric"]) and on
failure (ValueError/SyntaxError) falls back to a sanitized JSON parse similar to
scripts plot_rt_sa_analysis.py and plot_radar_chart.py: sanitize the string with
a regex to replace/normalize "nan"/"inf"/"-inf" to null or a safe numeric
sentinel, then json.loads the sanitized string; ensure this fallback is used for
both cov and prec and that exceptions are caught and logged at a lower severity
rather than crashing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 93541d0a-9cb4-4c4f-b096-3ce1d2b94e54
⛔ Files ignored due to path filters (17)
plots/aa_peptide_gap.pngis excluded by!**/*.pngplots/algorithm_agreement.pngis excluded by!**/*.pngplots/algorithm_agreement_auc.pngis excluded by!**/*.pngplots/algorithm_ranking_boxplot.pngis excluded by!**/*.pngplots/algorithm_ranking_bump.pngis excluded by!**/*.pngplots/algorithm_ranking_cd.pngis excluded by!**/*.pngplots/algorithm_ranking_heatmap.pngis excluded by!**/*.pngplots/category_performance.pngis excluded by!**/*.pngplots/confidence_calibration.pngis excluded by!**/*.pngplots/dataset_clustering.pngis excluded by!**/*.pngplots/dataset_difficulty.pngis excluded by!**/*.pngplots/db_agreement_at_5pct_fdr.pngis excluded by!**/*.pngplots/instanovo_version_improvement.pngis excluded by!**/*.pngplots/precision_at_coverage.pngis excluded by!**/*.pngplots/radar_chart.pngis excluded by!**/*.pngplots/rank_stability.pngis excluded by!**/*.pngplots/rt_sa_analysis.pngis excluded by!**/*.png
📒 Files selected for processing (20)
plots/algorithm_ranking_heatmap.htmlscripts/generate_benchmark_report.pyscripts/load_ranking_data.pyscripts/plot_aa_peptide_gap.pyscripts/plot_algorithm_agreement.pyscripts/plot_category_performance.pyscripts/plot_confidence_calibration.pyscripts/plot_dataset_clustering.pyscripts/plot_dataset_difficulty.pyscripts/plot_db_agreement_at_5pct_fdr.pyscripts/plot_instanovo_version_improvement.pyscripts/plot_precision_at_coverage.pyscripts/plot_radar_chart.pyscripts/plot_rank_stability.pyscripts/plot_ranking_boxplot.pyscripts/plot_ranking_bump.pyscripts/plot_ranking_cd.pyscripts/plot_ranking_heatmap.pyscripts/plot_ranking_heatmap_interactive.pyscripts/plot_rt_sa_analysis.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds visualization scripts for analyzing and comparing de novo sequencing algorithm performance across all 83 public benchmark datasets. All scripts are in
scripts/and share a common data loading module (load_ranking_data.py). They read theresults/*/peptide_precision_plot_data.csv(and AA/RT/SA) files and produce plots inplots/.They assume the following packages are availble in the environment:
Ranking visualizations
plot_ranking_boxplot.py— Rank distribution box + strip plotShows the distribution of ranks for each algorithm across all datasets, ordered by median rank. Each dot is one dataset. Quickly answers: "which algorithms are consistently top-ranked?"
plot_ranking_bump.py— Bump chartRank lines across all datasets. Each algorithm is a colored line, x-axis = dataset, y-axis = rank. Shows how rankings shift across different experimental conditions.
plot_ranking_heatmap.py— Static rank heatmapAlgorithms (rows) vs datasets (columns), colored by rank. Gives a dense overview of where each algorithm excels or struggles.
plot_ranking_heatmap_interactive.py— Interactive Plotly heatmapSame as above but as an interactive HTML file. Hover shows algorithm, dataset, rank, and AUC for each cell.
plot_ranking_cd.py— Critical difference diagram (Demsar 2006)Shows average rank per algorithm with Nemenyi post-hoc significance test. Algorithms connected by a bar are not statistically different. Answers: "are the ranking differences significant?"
Algorithm-focused analysis
plot_rank_stability.py— Rank stability / consistency analysisTwo-panel figure: (left) rank range per algorithm showing IQR and full range, (right) median rank vs IQR scatter with four labeled quadrants (good & consistent, good but unpredictable, predictably poor, poor & inconsistent).
plot_instanovo_version_improvement.py— InstaNovo v1.1.2 vs v1.2.2Paired comparison across all datasets for both peptide and AA precision AUC. v1.2.2 improves peptide AUC on 73/83 datasets and AA AUC on 82/83 datasets.
plot_aa_peptide_gap.py— AA vs peptide precision gapThree-panel figure showing the discrepancy between amino acid-level and peptide-level precision.
plot_rt_sa_analysis.py— RT difference and spectral angle analysisFour-panel figure: boxplots of RT error and spectral angle AUC per algorithm, plus heatmaps across datasets.
plot_radar_chart.py— Multi-metric radar chartsPer-algorithm spider plots showing peptide AUC, AA AUC, RT quality (1-error), and spectral angle.
plot_algorithm_agreement.py— Algorithm agreement matrixSpearman rank correlation between all algorithm pairs across datasets.
plot_confidence_calibration.py— Confidence calibration analysisThree panels analyzing how well confidence scores predict actual correctness:
Dataset-focused analysis
plot_dataset_difficulty.py— Dataset difficulty rankingRanks all 83 datasets by mean peptide precision AUC across algorithms. PT_orbitrap TMT datasets are the hardest (mean AUC ~0.2); standard tryptic human datasets are the easiest (~0.95).
plot_dataset_clustering.py— Dataset clusteringHierarchical clustering of datasets by their algorithm ranking profile (Ward's method on rank vectors). Clear clusters emerge: TMT/labelled datasets group together (universally hard), 9_species and standard human datasets cluster separately (easier). Dataset type matters more than species.
plot_category_performance.py— Category-wise performanceFaceted boxplots grouping datasets by experimental type (9 Species, ProteomeTools, Immunopeptidomics, Phospho, mAb, etc.). Reveals category-specific ranking differences.
plot_precision_at_coverage.py— Precision at fixed coverage thresholdsCompares precision at 25%, 50%, and 75% coverage. Rankings are stable across thresholds. The gap between top and bottom algorithms widens at higher coverage, meaning weaker algorithms degrade faster when including less confident predictions.
plot_db_agreement_at_5pct_fdr.py— DB search agreement rate at 5% FDRComputes the fraction of DB-identified peptides that each de novo algorithm also finds correctly, at a minimum of 95% precision (5% FDR). This is calculated as
0.95 * coverage_at_95%_precisioni.e., the number of correct de novo predictions at the 95% precision cutoff, divided by the total number of DB-identified spectra. Note: this measures agreement with database search, not true recall, since de novo algorithms may find real peptides that DB search missed (which would not be counted as correct here).Let me know if you want to see more vizualizations or if you want me to integrate some of the plots in the streamlit site.
Summary by CodeRabbit