Skip to content

README_PREDICTION_NEW_DATA.md

Latest commit

 

History

History
162 lines (114 loc) · 4.45 KB

README_PREDICTION_NEW_DATA.md

File metadata and controls

162 lines (114 loc) · 4.45 KB
Raw

New Data Prediction Guide

This guide explains how to run predictions on the large 3M PubMed dataset, filter to autoregulatory papers, enrich with PubTator, and merge with the existing enriched app dataset.

1) Download Required Files (Google Drive)

Download required files (Google Drive): https://drive.google.com/drive/folders/1cHp6lodUptxHGtIgj3Cnjd7nNBYWHItM?usp=sharing

Place the files here:

  • data/pred/abstracts-authors-date.tsv
  • models/stage1_best.pt
  • models/stage2_best.pt

2) Create Environment

conda env create -f environment.yml
conda activate autoregulatory
pip install torch --index-url https://download.pytorch.org/whl/cpu

3) Run Predictions

This runs the 3M prediction step:

python scripts/python/prediction/predict_new_data.py \
  --input data/pred/abstracts-authors-date.tsv \
  --output results/new_predictions.csv \
  --checkpoint-interval 10000

4) Filter to Autoregulatory Only

After predictions, filter to keep only rows with autoregulatory mechanisms:

python scripts/python/data_processing/filter_non_autoregulatory.py \
  --input results/new_predictions.csv \
  --output results/new_predictions_autoregulatory_only.csv

5) PubTator Enrichment (CSV)

After filtering, enrich the CSV with protein and gene info using PubTator + UniProt.

python scripts/python/data_processing/enrich_pubtator_csv.py \
  --input results/new_predictions_autoregulatory_only.csv \
  --output results/new_predictions_autoregulatory_only_enriched.csv

Output columns added:

  • UniProtKB_accessions (comma-separated UniProtKB accession numbers, if found)
  • Protein_ID
  • Protein_Name
  • Gene_Name

Optional: if any PubMed fields are missing (e.g., Year/Month/PublicationDate), add --fill-pubmed to fetch them from PubMed (cached):

python scripts/python/data_processing/enrich_pubtator_csv.py \
  --input results/new_predictions_autoregulatory_only.csv \
  --output results/new_predictions_autoregulatory_only_enriched.csv \
  --fill-pubmed

6) Merge with Existing Enriched Dataset

Once the 3M predictions are enriched, merge them with the enriched metadata dataset from the unused unlabeled flow.

Script

scripts/python/data_processing/merge_enriched_predictions.py

Inputs

Base (unused unlabeled metadata, enriched):

  • results/unused_predictions_autoregulatory_only_metadata_enriched.csv

New (3M predictions enriched):

  • results/new_predictions_autoregulatory_only_enriched.csv

Output

  • shiny_app/data/predictions.csv

Recommended Command

python scripts/python/data_processing/merge_enriched_predictions.py \
  --base results/unused_predictions_autoregulatory_only_metadata_enriched.csv \
  --new results/new_predictions_autoregulatory_only_enriched.csv \
  --output shiny_app/data/predictions.csv

This merge step also generates a database-specific unique row identifier:

  • AC = SOORENA-<SourceCode>-<PMID>-<n> (keeps duplicate PMIDs as separate rows)
  • Source codes: U=UniProt, P=Predicted (Non-UniProt), O=OmniPath, S=SIGNOR, T=TRRUST

7) Integrate External Resources (OmniPath, SIGNOR, TRRUST)

Add curated self-loop data from external biological databases:

python scripts/python/data_processing/integrate_external_resources.py \
  --input shiny_app/data/predictions.csv \
  --output shiny_app/data/predictions.csv \
  --others-dir others/

What this does:

  • Reads external resource files from others/ directory:
    • OmniAll.xlsx (OmniPath protein-protein interactions)
    • Signor.xlsx (SIGNOR signaling/phosphorylation data)
    • TRUST.xlsx (TRRUST transcription factor autoregulation)
  • Extracts self-loop entries (where source = target)
  • Maps mechanism types to SOORENA categories
  • Adds entries with Source = "OmniPath", "SIGNOR", or "TRRUST"
  • Sets Mechanism_Probability = 1.0 (curated, not predicted)

Requirements:

  • Place external resource Excel files in others/ directory
  • Files are optional; missing files are skipped with a warning

More detail: docs/README_EXTERNAL_RESOURCES.md

8) Build SQLite DB (from merged enriched CSV)

Create the Shiny app database directly from the merged enriched CSV:

python scripts/python/data_processing/create_sqlite_db.py \
  --input shiny_app/data/predictions.csv \
  --output shiny_app/data/predictions.db

Notes

  • This step can take days on CPU.
  • Checkpointing lets you resume from results/new_predictions_checkpoint.csv.