Feature Request: Quantification vs Identification Curves

[jcharkow]

Is your feature request related to a problem? Please describe.
Different DIA software tools calculate FDR differently meaning that it is difficult to compare them directly in terms of just identification rates. As presented at HUPO-PSI there seems to be a tradeoff between quantitative accuracy and peptide identifications, where software that reports more identifications at 1% FDR also have lower quantitative accuracy.

Gao et al., 2025 (@huhehaotecrystal) introduces a strategy to compare the tradeoff between quantification and identification. The idea is similar to an AUC curve where the best workflows maximize the area under the curve.

Describe the solution you'd like
I spoke briefly with @RalfG about implementing this feature in protebench.
To construct these curves, peptide precursors are ranked based on their FDR and a rolling median/mean (or just picking specific FDR cutoffs and connecting them to form a curve)

• This would require that FDR is reported in proteobench (when I downloaded the csv I did not see this)
• These curves work best when results are not filtered by FDR.

Additional context

Here is a screenshot from the publication with an idea of how these curves can look like

https://www.biorxiv.org/content/10.1101/2024.12.19.629475v1.full

[mlocardpaulet]

Hi,
This is a really nice suggestion. Thanks. We actually see this trend (high sensitivity -> low quantification accuracy) in our current DIA modules (which are still work in progress).
In our current modules, users set a FDR threshold and send us results after validation. So we can not directly add this visualisation now. We would need to make a new module for which input data would be all the PSMs before validation. It could be done (new module proposals can be submitted here).
One limitation I see to this visualisation is that only a limited number of output runs can be plotted together, right? How many do you think you can fit max. without making the figure unreadable? Currently, we have 35 runs compared in our DDA module (the oldest), and we expect to have more since we allow different versions and parameter sets for the same workflow tool.

[mlocardpaulet]

Hi, there is now a FDR evaluation module that is live here: https://proteobench.cubimed.rub.de/Entrapment_DIA_ion_Astral
It is not what you suggested, but could it somehow address your comment?
Would you like to discuss further other representations/metrics based on mixed-species samples?