DDA and DIA identification using entrapment

[mlocardpaulet]

Since we decided to give up the entrapment strategy, we discussed a different type of DDA identification module:
It would be nice to plot overlap of identification from pairs of searches. It is too complicated to have it as a main figure: we would need to select pairs. So we think of a main figure to select from

• Number of ID vs rank, we could have a slider that select the peptides matched to proteins identified with 1, 2, etc.. peptides; or 2 plots (proteins with >1 proteotypic peptide or all)
• Number of ID vs AUC (based on number of ID per FDR). Apparently we only need the q-value per psm in the output.
  What do we count? Ions? Peptidoforms? PSMs would be easier, but what do we do if we have multiple PSMs per spectrum: We keep the best one.
• Number of peptidoform sequence vs number or validated spectra
  What file to use? Human sample?

[julianu]

Dataset for testing:
https://ftp.pride.ebi.ac.uk/pride/data/archive/2022/02/PXD028735/LFQ_Orbitrap_DDA_Human_01.raw

[julianu]

We should record the way of post-processing

• "simple" target decoy, percolator, other re-scoring

[veitveit]

Shouldn't be the level (PSM, peptidoform, protein) of the validation be on what was uploaded?

For example, if you want to check for protein IDs, then the user should upload a table with protein accession names + their scores.

Otherwise, ProteoBench carries out part of the data analysis, and thus would influence the outcome by whatever choices have been made.

[julianu]

I would not really care about proteins for this module.
We definitely need the user to upload PSMs, I would suggest already with calculated q-values. Alternatively, we could also simply accept the PSMs of targets, decoys and the search engine score and calculate the scores ourself.

The benefit with already calculated q-values would be, we can also accept re-scored or percolated data.
But having a little "sanity check" would be good. Otherwise, one could just upload a totally synthetic, overtuned dataset.

[julianu]

A first impression of possible metrics (Dataset not run with the discussed FASTA, but using cRAP for contaminants)

[julianu]

For now, the upset plot uses all datapoint. But these should probably be selected and allow only up to 5 datasets or something. Otherwise, this will get extremely messy.

[mlocardpaulet]

I would be more interested in the overlaps between all and all but one workflow. I personally would focus on these (if we have to choose).

[julianu]

Updates of the plots:

Plots with AUC from more "real" ROC (normalized by number of IDs per searchengine)

And a plot of the number of IDs against q-value (not normalized)

... and normalized

[mlocardpaulet]

I put here the QQ-plot paper that was suggested in another discussion: https://pubs.acs.org/doi/10.1021/acs.jproteome.2c00423 (discussion https://github.com/orgs/Proteobench/discussions/205 now closed to be discussed here)

[mlocardpaulet]

We had a very productive discussion at the Lorentz Workshop on Trustworthiness in Proteomics on a ProteoBench module that would estimate random discovery proportion using entrapment.

The idea is to use entrapment search a la Wen et al. https://www.nature.com/articles/s41592-025-02719-x.pdf

General idea:

• The module will assess global FDR estimates provided by the tools for sequence identifications (or peptidoform? -> to explore based on what is possible with tools outputs).
  Module design:

Input data:

• Single data set with the choice between species (for example, for the DDA bulk data, we plan to have one raw file ecoli, one raw file yeast and one human, and to provide the corresponding fastas; users can upload the outputs file by file -- indicating the species of origin):
  • bulk human and single cell.
  • Initially Thermo Astral (from https://www.biorxiv.org/content/10.64898/2026.01.29.702266v1.full.pdf).
  • in Raw (vendor) format files.
• Protein FASTA (human + contaminants) and entrapment FASTA is provided to participants.

Metric

1. Use FDRBench (https://github.com/Noble-Lab/FDRBench) to create entrapment DB that all participants use.

• Potential problem is that it creates a peptide database, not protein-level
  -> Reassemble? Not trivial maybe with missed cleavages. After quick testing, we seem to be able to derive the peptide pairing from the protein-level fasta output. Ask Bo Wen for help if this does not work. Use mimic (https://github.com/percolator/mimic) as fallback.

2. Reimplement entrapment FDR calculations from FDRBench (Java) in Python

Participants must provide:

• Identification results in CSV/TSV/mzTab file thresholded at X% FDR (X≤1)
• and optionally: full list of unthresholded peptides (for more in-depth plots); Sequence/peptidoform q-values as reported by the tool
• Full parameter set

To define:

• what parameters we want to parse and show in plots/tables
• Find a name for this module

[mlocardpaulet]

TODO: One thing that remains to be discussed/test:
do we count sequences/peptidoforms?

1. figure out what tools provide output at peptide / peptidoform level?
2. Related to previous: should we keep only one row per peptidoform, keeping the best score?

[Cajac102]

Based on the discussion we had at the proteobench online meeting last week:

• Test data: 3 DIA single species (human) raw files from Barts Benchmarking paper
• Fasta file: Human proteome from UniProtKB + matched entrapment (1 per protein, scrambled, but I think the cleavage sites are kept)
• Focus on DIA precursor ion level for now -> we might need new parsers
• Main plot: Showing #IDs against eFDR at 1% reported precursor level FDR
• In depth plot: Showing eFDR against reported precursor level FDR

I will start setting up the module based on these decisions. If you object to any of them, let me know asap :)

[mlocardpaulet]

For this you could use colors or shapes. Use the paired eFDR in the horizontal axis, with 3 different colors if it is inconclusive, valid or else.

[Cajac102]

Hmmm maybe we can use the shapes for the categories and keep the colors for the search engines. Hard to make that plot colorblindfriendly then though.

[julianu]

I like the idea to give a "shift from paired/upper/lower" bound and the number of IDs. Interestingly, for the DDA Test of this data, the plots rather looked like this:

Showing the FDR against FDP for lower and upper bound as well as paired. Shown is MaxQuant and Oktoberfest, but this looked similar for all, just slightly shifted. So without taking care of the pairs, it would be inconclusive in the example. But with using the info, it actually is pretty valid at 1% (and basically all other values).

[Cajac102]

That's the decision we have to do in terms of interpreting the outcome of an entrapment experiment. I agree with you Julian that the FDR calculation shown looks quite solid - it is between a computed upper and lower bound. However in the paper, they see this scenario still as a potential 'strike' against the tool, as the 'actual FDP could be larger than the controlled FDR threshold (i.e., above the diagonal, invalid) or smaller than the controlled FDR threshold (i.e., below the diagonal, valid)' (see Noble-Lab/FDRBench#14 (comment)).
Maybe you discussed this during the meeting today though?

[julianu]

Actually, that's also what I saw in another dataset. Maybe just need to test a bit around - and also test our calculations on the dataset from the FDRBench manuscript.

[mlocardpaulet]

Update following a discussion in an online meeting (some new stuff, some that were already discussed here before):

• Quick concept of this ProteoBench module:

  1. We provide input data that everyone can download from the module documentation (here, it would be DIA raw files of a Human sample described here and a fasta file that contains human sequences, contaminants, and a set of entrapped protein sequences generated with FDRBench).
  2. Anyone can run a workflow for precursors/peptides identification
  3. Users can then upload to ProteoBench their analysis result and the module will calculate the empirical FDR. The module displays the results in what we call a “main figure”, where each workflow output (from a given tool, with a given set of parameters) is plotted. Additional tabs in the web application allow users to explore in more depths the results of each workflow.

• What to plot in the main figure? We concluded that the figure could be the flollowing:

  1. Vertical axis = empirical FDR (calculated with the paired entrapped sequences generated with FDRBench) divided by the estimated FDR (FDR validation threshold of the workflow). (or shall we call them “FDP”?)
  2. Horizontal axis = workflows, maybe grouped by software tool or a given set of parameters? This will need a bit of testing. Caro had the nice idea to split them between inconclusive, valid and invalid, maybe we could try it? Or have a selection option to reorder based on parameters/criteria? Or can we find another metric that we would like to plot in this axis?
  3. Regarding the level of the analysis (peptide or precursor?): we’ll need to have both. Different tools perform validation at different levels. So we need to have one plot for precursor-level validation, and another for peptide level. Question: do we want to have them in the same module or do we need to develop 2 modules?

• Ideally, users would send us the entire set of spectrum-to-spectrum matches (SSMs), with the scores (no FDR-based filter). We all agree on that, but we believe that this would restrict too much the panel of tools that we could compare (since it is not always possible to get the full export). As a workaround, we would like to set up 2 options to upload workflow runs to the module:

  1. The filtered set of peptides/precursors, so that we can calculate metrics in the main figure (with no other plots possible)
  2. The full list of SSMs with the scores. This would allow to calculate more metrics and provide more figures.

• We shortly discussed keeping protein-N-terminal methionine in the entrapped sequences because a lot of software tools include N-ter methionine excision as default during the search.

• Shall we consider PTMs? Tricky. Bill suggested that we have a look at this paper, and that we discuss this with Uri. I think that the best idea we came up with for now was: require that users perform the search without any variable PTM. We all agreed that this was not ideal, but did not come up with a better workable plan…

[mlocardpaulet]

Follow up on this discussion:

Idea for a summary metric proposed by Ville:

We could consider using the area under the curve? For example, in the image from the FDRBench github page:

The green line initially hugs the diagonal, then deviates slightly. The area between the green line and the diagonal is > 0, and it can be calculated or estimated using standard numerical integration methods. If the green line were entirely below the diagonal, probably you want to set the area under the curve to zero. Then zero, or close to zero, indicates a valid method.
If you only have, say, 10 data points, then an equivalent metric could be the sum of the excess. For example, if estimated FDP (green) is 6.3% at 6% FDR threshold, then the excess at that point is 6.3% - 6% = 0.003. Sum the excess over the 10 points and you get a number. If it's >0, the method is inconclusive or not valid. Makes sense?

This would only be possible for the "full" submission where users provide all PSMs/SSMs with scores.

[Cajac102]

-- MOCK DATA! --

Here are some mockups for potential main plots:

1. Vanilla: # IDs vs. [lower bound OR paired OR combined]

Negative:

• Confusing: Should upper bound be lower or higher - I would not know how to interpret this plot
• only works with 1% (or any other fixed threshold) across all points, as there is no comparison to the reported FDR
• does not spark any joy in me

Positive:

• shows # IDs
• scalable with lots of points
• similar to our usual main plots

2. Pretty Forests: FDP Interval Plot

Negative:

• Does not scale well with lots of points (will get longer and longer)
• Does not show # IDs (could be implemented by e.g. scaling the size of the marker. edit: now in clear text next to label)
• Hard to make colorblind friendly as color is used to categorise valid invalid inconclusive

Positive:

• Clearly shows lower bound, upper bound, and reported, therefore easy to interpret
• sparks joy

3. Get ratio'd: FDP / FDR

Negative:

• Confusing: why is one point orange and one is red, when they are both above the line (due to only one bound shown)
• Turning FDR or FDP into ratio makes it a bit more abstract
• Does not show # IDs (could be implemented, e.g. by scaling point size)

Positive:

• can also use other thresholds than 1%
• Scalable with lots of points

4. Get ratio'd x Vanilla:

Negative:

• Confusing: why is one point invalid and one is inconclusive, when they are both above the line (due to only one bound shown)
• Turning FDR or FDP into ratio makes it a bit more abstract
• Telling apart colors and shapes takes a couple of mental CPU cycles

Positive:

• can also use other thresholds than 1%
• Scalable with lots of points
• Shows # IDs

5. Guinea Flag:

Negative

• unclear how conservative the FDR calculation is, as only the three categories are shown

Positive:

• very interpretable

[Cajac102]

Current fan favorite:
Show two main plots:

1. Get ratio'd x Vanilla
2. Pretty Forests underneath

We could maybe under certain circumstances potentially... connect the two. You click on one scatterpoint, you get the bar highlighted in the forest plot.

[mlocardpaulet]

We had a few meetings with interested contributors to refine the design of the module. Here are the meetings notes:

• Quick concept of this ProteoBench module:

1. We provide input data that everyone can download from the module documentation (here, it would be DIA raw files of a Human sample described here and a fasta file that contains human sequences, contaminants, and a set of entrapped protein sequences generated with FDRBench).
2. Anyone can run a workflow for precursors/peptides identification
3. Users can then upload to ProteoBench their analysis result and the module will calculate the empirical FDR. The module displays the results in what we call a “main figure”, where each workflow output (from a given tool, with a given set of parameters) is plotted. Additional tabs in the web application allow users to explore in more depths the results of each workflow.
   An alpha version of the module with some mock data can be found here: https://proteobench.cubimed.rub.de/Entrapment_DIA_ion_Astral.

• For the moment, the module calculates metrics at precursor ions level. The plan is to have another one at peptide level, and we will stop here for now, waiting to have a better strategy for protein group FDR benchmarking. For both modules (precursor and peptide), we use peptide-level shuffling for the database generation.

• We had a discussion on the FDR threshold at which the input data should be filtered. Matthew would like to have all the precursors (so results of a search with 100% FDR) and filter at a given FDR percentage ourselves. We think that this makes sense, but software could have some other post-processing magic that lead to our filtering at 1% being different from the software output filtered at 1%. So the conclusion was to keep it flexible: users can provide results filtered at the threshold they want. We can totally encourage them to provide all precursors.

• Order of the main figures? We think that to cover all aspects of FDR evaluation, we need to provide several figures. Our two favorite (that we discussed more) were the “forest plot” that is the first one on the current page, and is accompanied with a horizontal barplot that shows the number of identified features; and the scatter plot below showing the number of identified features (vertical axis) and the upper or lower bound (horizontal axis: its default value should be upper bound, it can be changed with a selection field). The point shape can correspond to valid, invalid, inconclusive. Please have a look and tell us which one you prefer. We’ll keep both but need to decide which one to put first. I think that we all agreed that the forest plot conveys the information we think important very nicely, but risk to become overcrowded. For this reason, and for clarity, we should always have a button that allows to plot the metrics at a given FDR.

• In the figures, we propose to have a filter to set the FDR threshold that we want to apply to the results. This means that for all the search outputs with a FDR threshold above the one that is selected for the plot, we will report the estimated FDP for the same FDR target as all other workflows.

• How do we handle modifications? For this point, I copy/paste here Uri’s comments from his email:

  • Regarding Robbin's comment about the fact that disabling PTMs might increase false discoveries because a PTM-generated spectrum might match to a non-modified wrong peptide (I think Matthew was making an argument along the same lines). I agree that the risk is there, and it’d certainly diminish the tool’s power. However, I think it’s also valid to demand that a tool controls the FDR when searching without PTMs even if there are PTMs in the sample, so I don’t think this can invalidate the entrapment protocol.
  • Robbe raised the point that it might create an issue when people might forget to switch off PTMs and that’s also a good point, but I think that we can address this by offering two modes: with PTMs and w/o, each with its own page and set of figures, and by simply looking at the list of reported peptides we can determine whether PTMs were enabled or not, and route the results to the right page.
  • When PTMs are disabled things are fairly straightforward and similar to peptide level analysis.
  • When PTMs are enabled, the lower bound is still valid, albeit it’s even more conservative than it normally is. The upper bound needs an asterisk which we can display as a clear message, preferably somewhere on the top of the page, along the lines of “Note that the current entrapment construction does not account for localization errors. For example, the tool might report one localization instead of another, or might report two separate localizations whereas the sample only contains one. None of these errors is currently properly accounted for."
  • What do you think?

• One more question that we did not have time to discuss. If you have a strong opinion I am interested: how shall we call the entrapment-based FDR estimation metric that we calculate?

  • FDP
  • eFDR
  • RDP

• We shortly discussed keeping protein-N-terminal methionine in the entrapped sequences because a lot of software tools include N-ter methionine excision as default during the search.

[mlocardpaulet]

Follow up on our weekly ProteoBench meeting:

• we can call the metric “estimated FDP”
• for modifications, we will go with the strategy proposed by Uri: parameters will be set up by users, and if they add variable modifications to the search we will add a note/warning stating that localisation errors are not properly accounted for.

[mlocardpaulet]

Hi @Cajac102, really great work!
In the "main" scatter plot, I wonder if we should keep the choice to visualise the "Maximum reported" Q-value threshold. I think that it can be too different between points, and it is not super easy to find in the plot what these parameters are set to. Maybe this could be missleading?

[Cajac102]

we could remove it, the reason it exists is for cases where people upload FDRs that are not in the list and want to see the point anyways at that exact FDR. But that will probably never happen anyways. Ill remove it.