Skip to content

README.md

Latest commit

 

History

History
191 lines (143 loc) · 6.22 KB

README.md

File metadata and controls

191 lines (143 loc) · 6.22 KB
Raw

Data/Processed Directory

This directory contains all processed datasets used for training, validation, testing, and prediction.

File Overview

File Size Papers Description Generated By
Core Dataset
modeling_dataset.csv 331M 254,212 MASTER dataset - All papers (labeled + unlabeled) after cleaning and filtering prepare_data.py
Labeled Data (Ground Truth)
train.csv 1.3M 932 Labeled papers with mechanisms (70% split) prepare_data.py
val.csv 285K 200 Labeled papers with mechanisms (15% split) prepare_data.py
test.csv 283K 200 Labeled papers with mechanisms (15% split) prepare_data.py
Stage 1 Training Data (Binary Classification)
stage1_unlabeled_negatives.csv 3.5M 2,664 Unlabeled papers sampled as negative examples during Stage 1 training (2:1 ratio) train_stage1.py
stage1_unlabeled_unused.csv 326M 250,216 Unlabeled papers NOT used in training - for prediction only train_stage1.py
Evaluation Results
stage1_test_eval.csv 820K 600 Stage 1 binary classifier test results (200 positive + 400 negative) evaluate.py
stage2_test_eval.csv 284K 200 Stage 2 multiclass classifier test results evaluate.py

Data Pipeline Flow

┌──────────────────────────┐
│ prepare_data.py          │
│ ┌────────────────────────┤
│ │ Raw data cleaning      │
│ │ Term normalization     │
│ │ Filter rare terms      │
│ └───────────┬────────────┘
│             │
│             ▼
├─────> modeling_dataset.csv (254,212 papers)
│             │
│             ├─> 1,332 labeled (has mechanism)
│             │   ├─> train.csv (932)
│             │   ├─> val.csv (200)
│             │   └─> test.csv (200)
│             │
│             └─> 252,880 unlabeled (no mechanism)
│
└──────────────────────────┘

┌──────────────────────────┐
│ train_stage1.py          │
│ ┌────────────────────────┤
│ │ Sample 2:1 ratio       │
│ │ (2 negative : 1 pos)   │
│ └───────────┬────────────┘
│             │
│             ├─> stage1_unlabeled_negatives.csv (2,664)
│             │   Used as negative samples during training
│             │
│             └─> stage1_unlabeled_unused.csv (250,216)
│                 Never seen by model - for prediction
│
└──────────────────────────┘

Important Notes

For Reproducibility

All sampling uses RANDOM_SEED=42. This ensures:

  • The exact same papers are selected every time
  • Training is reproducible
  • Results can be verified by others

Stage 1 Sampling Strategy

Stage 1 trains a binary classifier (has mechanism vs. no mechanism):

  • Positive examples: 1,332 labeled papers (known mechanisms)
  • Negative examples: 2,664 unlabeled papers (sampled at 2:1 ratio)
  • Total training data: 3,996 papers (1,332 + 2,664)

The 2:1 ratio balances the classes while maintaining enough positive examples.

Which Files to Use?

For Training:

  • Use train.csv, val.csv, test.csv
  • Use stage1_unlabeled_negatives.csv (generated during train_stage1.py)

For Prediction:

  • Use stage1_unlabeled_unused.csv - These papers were NEVER seen during training
  • Do not predict on stage1_unlabeled_negatives.csv - Already used in training (data leakage)

For Analysis:

  • Use modeling_dataset.csv - Complete overview of all available data
  • Use evaluation CSVs - Check model performance

File Management

Essential Files:

  • modeling_dataset.csv - Master source
  • train.csv, val.csv, test.csv - Ground truth
  • stage1_unlabeled_negatives.csv - Documents training negatives
  • stage1_unlabeled_unused.csv - For predictions

Can be Regenerated:

  • stage1_test_eval.csv - Run evaluate.py to recreate
  • stage2_test_eval.csv - Run evaluate.py to recreate

File Relationships

modeling_dataset.csv (254,212 total)
│
├─ Labeled (1,332)
│  ├─ train.csv (932)
│  ├─ val.csv (200)
│  └─ test.csv (200)
│
└─ Unlabeled (252,880)
   ├─ stage1_unlabeled_negatives.csv (2,664) ← Used in training
   └─ stage1_unlabeled_unused.csv (250,216) ← For prediction

Column Descriptions

Core Columns (all files)

  • PMID: PubMed ID (unique identifier)
  • text: Combined title + abstract (cleaned)
  • Terms: Autoregulatory mechanism types (comma-separated)
  • has_mechanism: Boolean - paper has autoregulatory mechanism

Additional Columns

In labeled files (train/val/test):

  • label: Primary mechanism type (for stratified splitting)

In Stage 1 files:

  • split: Which split the paper belongs to (train/val/test)

In evaluation files:

  • predicted: Model prediction
  • actual: Ground truth label
  • confidence: Prediction confidence score

Quick Reference

Regenerate These Files

# Regenerate master dataset and splits
python scripts/python/data_processing/prepare_data.py

# Regenerate Stage 1 sampled datasets (during training)
python scripts/python/training/train_stage1.py

# Regenerate evaluation results
python scripts/python/training/evaluate.py

Check File Sizes

ls -lh data/processed/
wc -l data/processed/*.csv

Frequently Asked Questions

Why so many unlabeled papers?

We have 252K unlabeled papers but only used 2,664 for training. The rest (~250K) are for testing model predictions on truly unseen data.

Can I retrain with different ratios?

Yes. Change the sampling in train_stage1.py (line 32-36), but remember to update RANDOM_SEED for reproducibility.

Where are the 3M new predictions?

Those are in data/pred/ (input) and results/ (output), not in this directory.