DDA identification - de novo

[bittremieux]

aim of the new module

This module intends to evaluate the performance of de novo peptide sequencing tools. De novo tools annotate MS/MS spectra with their originating peptides without any external information, such as a protein database. Due to the lack of dependence on a set of predefined proteins, de novo sequencing has several important use cases, including:

• Immunopeptidomics, as non-specific digestion results in a very large search space.
• Metaproteomics, as the organisms to be considered might not be known.
• Discovery of unexpected peptide and protein variants not included in the database.
• ...

full description of the new module

This module will be restricted to the identification of MS/MS spectra measured using DDA. Thus, the input and output can be simplied as:

• Input MS/MS spectra (in MGF format).
• Output peptide sequences assigned to the MS/MS spectra (e.g. in mzTab or CSV format).
• Ground truth peptide sequencs for each of the MS/MS spectra (obtained using sequence database searching) to calculate evaluation metrics (e.g. in mzTab or CSV format).

Data

As the evaluation data, the nine-species benchmark dataset introduced by DeepNovo can be used. This dataset is commonly used to evaluate (and train) de novo tools. This dataset contains MS/MS data from nine different species as follows:

• Vigna mungo: 932,848 spectra from 24 runs.
• Mus musculus: 306,786 spectra from 13 runs.
• Methanosarcina mazei: 3,728,183 spectra from 72 runs.
• Bacillus subtilis: 4,336,428 spectra from 106 runs.
• Candidatus endoloripes: 2,272,023 spectra from 11 runs.
• Solanum lycopersicum: 603,506 spectra from 60 runs.
• Saccharomyces-cerevisiae: 1,477,397 spectra from 27 runs.
• Apis mellifera: 823,169 spectra from 17 runs.
• Homo sapiens: 684,821 spectra from 26 runs.

Recently, a more balanced subset of this data has been proposed that restricts each species to approximately 100,000 high-quality PSMs, amounting to 779,796 spectra and 180,238 unique peptides in total (Noble, under review at Scientific Data). I propose that we use this last version to use a consistent and manageable dataset. The DOI for this dataset is pending.

Q: What would be the best format for this data? Nine different MGF files, one for each species? Or one (relatively big) MGF file with all spectra?

A: Consensus is to combine all of the data in a single MGF file for simplicity.

The following search settings were used to obtain the PSMs:

• Static modification: Cys carbamidomethylation
• Variable modifications: Met oxidation, Asn deamidation, Gln deamidation, N-term acetylation, N-term loss of ammonia, N-term carbamylation, the combination of N-term loss of ammonia and N-term carbamylation
• Protease: trypsin
• Precursor mass tolerance: between 10 ppm and 30 ppm, depending on the species
• Fragment mass tolerance: 0.02 Da or 0.05 Da, depending on the species
• Search engine: Tide (using XCorr scoring and Tailor calibration) + Percolator

Metric calculation

The performance will be evaluated at the amino acid and peptide level. As introduced by DeepNovo, a correct amino acid prediction is defined as any predicted amino acid whose mass differs by less than 0.1 Da from the corresponding ground truth amino acid. Additionally, this predicted amino acid must have either a prefix or suffix that differs by no more than 0.5 Da in mass from the corresponding amino acid sequence in the ground truth peptide. Correct peptides are defined as those sequences where all amino acid predictions meet these criteria, ensuring that only fully accurate predictions are considered correct at the peptide level.

This information will be condensed into two values:

• The amino acid precision at coverage=1.
• The peptide precision at coverage=1.

With precision measuring the proportion of correct predictions among all predictions, and defined as in standard classification. Coverage is analogous to recall, but adapted to the fact that some de novo tools don't report results for all spectra. Thus, coverage represents the proportion of predictions made. We will use the precision at full coverage, i.e. considering predictions for all input amino acids or peptides, respectively. Spectra for which a tool can't make a prediction will be considered as fully incorrect.

This can be implemented as two swarmplots side by side, for both metrics, to visualize the performance of all tools.

To calculate the metrics, the output peptide sequences need to be compared with the ground truth peptide sequences. Thus, the users will need to upload a custom CSV with the necessary information (recommended in ProForma notation). As no standardized output format is currently being used by the many newly introduced de novo tools, I propose to not try to directly support all these different formats, but instead request a simple CSV format to be provided, irrespective of the tool used (columns: file name, spectrum index, peptide sequence).

potential reviewers

No response

Will you be able to work on the implementation (coding) yourself, with additional help from the ProteoBench maintainers?

• yes
• no

any other information

Tagging @PominovaMS.

[bittremieux]

This is the updated nine-species benchmark dataset that was suggested as evaluation data and since then has been fully published: https://www.nature.com/articles/s41597-024-04068-4

[SamvPy]

The de novo module is implemented with an initial set of datapoint submissions.

All checkpoint links below were verified (GitHub Releases API, Zenodo/FigShare/HuggingFace pages, or curl -I) before being added. Google Drive links are taken as-is from the developers' own READMEs.

Submitted datapoints

• AdaNovo — checkpoint: ckpt_1.ckpt — repo: Westlake-OmicsAI/adanovo_v1 — trained on the 9-species benchmark dataset.
• Casanovo 4.0.0 — checkpoint: casanovo_massivekb.ckpt — repo: Noble-Lab/casanovo — trained on MassIVE-KB v1 (~30M PSMs, human HCD).
• ContraNovo — checkpoint: ContraNovo.ckpt — repo: BEAM-Labs/ContraNovo.
• DeepNovo — checkpoint: translate.ckpt-134100 — repo used to run it: denisbeslic/denovopipeline.
• pi-HelixNovo — checkpoint: MSV000081142-epoch-5-step-800000.ckpt — repo: PHOENIXcenter/pi-HelixNovo (pi-HelixNovo branch).
• pi-PrimeNovo — checkpoint: model_massive.ckpt — repo: PHOENIXcenter/pi-PrimeNovo.
• PepNet — checkpoint: model.h5 — repo: Ruixiang-Bio/PepNet (see Zenodo record for the associated code archive).

Many more datapoints can still be added. Below is a comprehensive, tool-by-tool overview of additional checkpoints and tools, verified against the developers' GitHub releases, Zenodo/FigShare records, and HuggingFace Hub. Tick off items as they get submitted.

1. Casanovo

Repo: Noble-Lab/casanovo.

• Casanovo 4.0.1 — non-enzymatic — casanovo_nontryptic.ckpt — non-tryptic variant, sampled from MassIVE-KB + PROSPECT with a uniform C-terminal amino-acid distribution.
• Casanovo 4.2.0 — casanovo_v4_2_0.ckpt — combined tryptic/non-tryptic, ~2M PSMs from MassIVE-KB v1 + v2.0.15.
• Casanovo 5.0.0 — casanovo_v5_0_0.ckpt — same combined tryptic/non-tryptic MassIVE-KB training set as 4.2.0.
• Casanovo 5.2.0-Orbitrap — casanovo_orbitrap_v5-2-0.ckpt — default model selector --model orbitrap from v5.2.0 onward.
• Casanovo 5.2.0-timsTOF — casanovo_timstof_v5-2-0.ckpt — first timsTOF-specific checkpoint (--model timstof).
• Casanovo 3.0.0 — non-enzymatic — casanovo_non-enzy.checkpt — note the non-standard .checkpt extension in the release asset itself.

2. InstaNovo

Repo: instadeepai/InstaNovo.

• instanovo-v1.2.0 (current default transformer model) — instanovo-v1.2.0.ckpt.
• instanovo-v1.1.0 — instanovo-v1.1.0.ckpt; also on HuggingFace.
• instanovo-v1.0.0 ("extended" residue vocabulary) — instanovo_extended.ckpt; also on HuggingFace.
• instanovo-phospho-v1.0.0 (InstaNovo-P, phosphoproteomics) — instanovo-phospho-v1.0.0.ckpt; trained on ~2.76M phosphorylated PSMs from 29 PRIDE projects. Also on HuggingFace. See also project repo instadeepai/InstaNovo-P.
• instanovoplus-v1.1.0 (current default diffusion / InstaNovo+ refinement model) — instanovoplus-v1.1.0.ckpt; also on HuggingFace.
• instanovoplus-v1.1.0-alpha — instanovoplus-v1.1.0-alpha.ckpt — earlier preliminary diffusion checkpoint.
• instanovoplus_yeast — instanovoplus_yeast.zip — diffusion model pretrained on HC-PT, fine-tuned on nine-species-excluding-yeast.
• instanovo (v0.1.4, original transformer) — instanovo.pt and its yeast-excluded fine-tune instanovo_yeast.pt.

3. pi-HelixNovo

Repo: PHOENIXcenter/pi-HelixNovo — main is a landing page pointing to two code branches, pi-HelixNovo (v1) and pi-HelixNovoV2.

• pi-helixnovo_massivekb.ckpt (recommended v1 checkpoint) — download — trained on MassIVE-KB; README's recommended checkpoint for large-scale de novo sequencing. Run with the pi-HelixNovo branch.
• pi-helixnovo_massivekb-old.ckpt — download — superseded MassIVE-KB checkpoint, same release.
• pi-HelixNovoV2 — MassIVE-KB weights — Model-weights.zip on Zenodo (file Evaluation_on_MassIVE-KB_dataset.zip, contains ModelWeights/pi-helixnovo2_massivekb.ckpt) — run with the pi-HelixNovoV2 branch. See also the earlier weight/ablation bundle at Zenodo 12527491 (Powerful-pi-HelixNovoV2.zip, pi-HelixNovoV2-Ablation.zip).
• pi-HelixNovo v1 full model weights (Zenodo) — 10.5281/zenodo.10405582, file Model-weights.zip (11.2 GB) — codes, model weights, de novo peptide lists and constructed datasets for the original π-HelixNovo project.

4. pi-PrimeNovo

Repo: PHOENIXcenter/pi-PrimeNovo (mirrored at BEAM-Labs/pi-PrimeNovo). Checkpoints are distributed via Google Drive links in the README.

• PTM_Phosphorylation.ckpt — Google Drive — fine-tuned from model_massive.ckpt on the 2020-Cell-LUAD dataset, adds a phosphorylation (+79.97 Da) PTM token. Run with the pi-PrimeNovo-PTM subdirectory of the same repo.

5. AdaNovo / ContraNovo

No additional checkpoints found: both Westlake-OmicsAI/adanovo_v1 and BEAM-Labs/ContraNovo currently ship exactly the one checkpoint already submitted above, and no "v2" successor repo exists for either as of 2026-08.

6. New tools not yet in the benchmark

Verified as having both an open repo and an actually downloadable pretrained checkpoint.

• DeepNovo-DIA — DDA+DIA extension of the original DeepNovo. Repo: nh2tran/DeepNovo-DIA. Checkpoint + knapsack matrix on MassIVE FTP (ftp://massive.ucsd.edu/MSV000082368/other/), mirrored on Google Drive.
• SMSNet — sequence-mask-search hybrid CNN/LSTM de novo sequencer, reconciled against a database search. Repo: cmb-chula/SMSNet. Checkpoint: FigShare.
• PowerNovo — ensemble of transformer + BERT models with sequence-quality denoising. Repo: protdb/PowerNovo. Checkpoint (auto-downloaded on first run): FigShare, DOI 10.6084/m9.figshare.25329586.
• PowerNovo2 — non-autoregressive, flow-based successor to PowerNovo, ~4–5x faster. Repo: protdb/PowerNovo2. Checkpoint (auto-downloaded on launch): FigShare DOI 10.6084/m9.figshare.25329586 / DOI 10.6084/m9.figshare.28517777.v1.
• PepGo — transformer + tree-search de novo model using N-/C-terminal transformers. Repo: alifare/PepGo. Checkpoint: Zenodo, DOI 10.5281/zenodo.15055488. Companion synthetic training data: Zenodo, DOI 10.5281/zenodo.15032178.
• DiffNovo — transformer + diffusion model targeted at DIA spectra. Repo: Biocomputing-Research-Group/DiffNovo. Checkpoint: Zenodo, DOI 10.5281/zenodo.14611534 (pretrained-models.zip, 456 MB); trained on Plasma, Ovarian Cancer and Urinary Tract Infection DIA datasets.
• DIANovo — deep learning de novo sequencing for multiplexed DIA spectra, reports 25–89% recall improvement over DeepNovo-DIA. Repo: hearthewind/dianovo. Checkpoint: Dropbox folder referenced in the README (folder contents not visually re-verified, JS-rendered UI). Paper: arXiv:2411.15684.
• Modanovo — PTM-aware de novo sequencing built on Casanovo v4.0.0, covers 19 amino acid–PTM combinations. Repo: gagneurlab/Modanovo. Checkpoint: HuggingFace. Development dataset: HuggingFace datasets.
• NovoB — bidirectional de novo transformer predicting from both N- and C-termini simultaneously. Repo: ProteomeTeam/NovoB. Checkpoint: Zenodo, DOI 10.5281/zenodo.10258874 (see TrainedModel/usingCasanovoDataSet/<organism>/); trained on the MassIVE-KB dataset.
• RankNovo — list-wise reranker over multiple base de novo models (Casanovo, ContraNovo, ByNovo and reversed variants) via axial attention over multi-sequence alignments. Repo: BEAM-Labs/denovo. Checkpoint: Google Drive. Paper: arXiv:2505.17552 (ICML 2025).
• RefineNovo — non-autoregressive (CTC-based) de novo model stabilized via curriculum learning. Repo: BEAM-Labs/denovo (RefineNovo subfolder). Checkpoint: "RefineNovo-30M" via Google Drive. Paper: "Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing" (ICML 2025).
• CrossNovo — hybrid AR/NAR framework augmenting autoregressive generation with bidirectional context from non-autoregressive models. Repo: BEAM-Labs/denovo. Checkpoint: Google Drive link in the repo's model table (⚠ same file ID listed for RankNovo in that table — double-check). Paper: arXiv:2510.08169 (NeurIPS 2025).

Note: BEAM-Labs/denovo also lists a "PrimeNovo" checkpoint that links to the same model_massive.ckpt already covered under pi-PrimeNovo above.

Checked but not eligible — no downloadable pretrained checkpoint found

Kept here so these don't need to be re-investigated later; revisit if the developers publish weights.

• PointNovo (irleader/PointNovo) — training data/code only. Community fork PGPointNovo claims weights will be shared via a SharePoint link that currently fails to resolve.
• GraphNovo (AmadeusloveIris/Graphnovo) — only a Google Drive dataset link, no checkpoint.
• DeepNovoV2 (volpato30/DeepNovoV2) — training/data instructions only.
• DPST (Yan98/DPST) — code only, no checkpoint in README.
• Transformer-DIA (Biocomputing-Research-Group/Transformer-DIA) — README's "pretrained models" link is an empty placeholder.
• SearchNovo (junxia97/SearchNovo, ICLR 2025) — training/inference code only.
• LIPNovo (usr922/LIPNovo, ICML 2025) — README: "pre-trained models will be uploaded later."
• DePS — no GitHub repo found (arXiv 2022 paper only).
• ReNovo (ICLR 2025) — paper/OpenReview exists, no GitHub repo located.
• OmniNovo (arXiv:2512.12272, Dec 2025) — PTM-aware de novo tool, no GitHub repo found yet.
• Novor / Novor.Cloud — closed-source web service, no open weights.
• pNovo3 / pNovo+ — distributed as a full software package from pFind Studio, requires registration/contacting the authors; not a deep-learning checkpoint in the usual sense, and not classical (deep learning) de novo in the same framing as the tools above.

[bittremieux]

What does the scatter plot show? It seems almost (but not entirely) the precision at the tool's max coverage.

This is a flawed metric because of several reasons. You can artificially get high precision by only making predictions for very easy spectra. In this case your coverage will be low, but in the scatterplot your performance will look amazing.

Instead, precision should be calculated at full coverage, so also considering spectra for which a tool doesn't make a prediction.

Even better would be the area under the precision-coverage curve, rather than the precision@coverage=1, because that captures more nuance. A tool that drops to 0.5 precision immediately and then stays there until full coverage is obviously worse than a tool that stays at high precision for most of the spectra and only drops down to lower precision at the end. This is again not captured by just reporting the precision, but the area under the curve does capture it.

[SamvPy]

Hi Wout,

Thanks for this.

Yes, indeed, the scatterplot shows the precision at the tool's max coverage. And I am aware that this will look artificially good when only making predictions for the easy cases. Given this, and to accommodate your next (and previous) comments, a toggle was added to switch metrics in the main plot towards the AUC (area under precision coverage curve). There's also a new tab on the main Public Results view to switch from the scatterplot to the full precision-coverage curves, on both amino acid and peptide levels. However, when checking this, I noticed a discrepancy here which I reported here, which I'll fix in a PR. Given the concerns you've raised, I can (i) change the precision calculation as you suggested to precision@coverage=1 and thus essentially consider unpredicted spectra as incorrect, or (ii) set AUC as default metric instead and keep the precision as it is now. Happy to hear what you'd prefer, but I would opt for the second option?

A few other things, slightly broader than your specific comment: several other concerns can be raised about the evaluation as a whole, e.g. (i) some tools support specific modifications while others don't, (ii) the ground truth is itself the output of a database search and can be wrong, and (iii) there's limited insight into why a given prediction is wrong. To address these aspects, we have the Compare Results tab which breaks down the precision further by (i) modification, (ii) by whether the false prediction still matches with some candidate in the search space of the database search (FASTA-evaluation) [still a small server-side bug there as described here, and (iii) by plotting precision against peptide/spectra characteristics such as peptide length, missing fragmentation sites, %explained intensity.

Longer term, I'd like to add retention time and fragment-intensity feature distributions to further contextualize as well (and also comparing these with the features from non-matching predictions). These features exist already in the ground-truth dataset but are currently unused. But I'll scope this out as a separate future PR.

[bittremieux]

Given the concerns you've raised, I can (i) change the precision calculation as you suggested to precision@coverage=1 and thus essentially consider unpredicted spectra as incorrect, or (ii) set AUC as default metric instead and keep the precision as it is now. Happy to hear what you'd prefer, but I would opt for the second option?

Yes, the second option (area under the precision–coverage curve as default) would be good. Although I'd still argue for also using precision@coverage=1, because some tools fail to make predictions for a considerable fraction (>10%) of spectra.

Side note: what's going on with PepNet in the current public curves? Its AA coverage starts at 0.028 rather than at 0. Similarly, several tools have peptide coverage starting at >0.

To address these aspects, we have the Compare Results tab which...

Yes, makes sense. 👍

Longer term, I'd like to add retention time and fragment-intensity feature distributions...

Sounds good. We're also doing that in our de novo benchmark.

[SamvPy]

I'll change precision to precision@coverage=1 in a PR alongside the other changes. Regarding the curves and some starting at > 0 coverage, I'm aware and this is related to sampling x datapoints along the coverage range. When lets say the 100 best predictions have the same score, this will not draw a point at coverage = 0. So its easy to fix and on my todo anyway.

Thanks a lot for your advice on this module!

[bittremieux]

Nice work!