Skip to content

README.md

Latest commit

 

History

History
88 lines (63 loc) · 4.01 KB

README.md

File metadata and controls

88 lines (63 loc) · 4.01 KB
Raw

data_samples/ — Provenance and Regeneration

This directory holds the exact train/test splits used in the paper. Each subdirectory contains a train.csv and a test.csv with prompt / y_true columns.

Raw source datasets are not redistributed here. Each entry below lists the upstream source and the exact transformation applied. The regeneration scripts in scripts/ are seeded (random.seed(42)) and reproduce the corresponding shipped splits bit-for-bit when given the same raw inputs at the expected paths under a top-level datasets/ directory.

Fact-oriented datasets

pubmed/ — PubMed

  • Source: qiaojin/PubMedQA on Hugging Face.
  • Transformation: extracted the question and final_answer columns from the pqa_labeled split; randomly sampled 500 questions and split 50/50 into train.csv / test.csv. Each prompt was wrapped with the boilerplate Respond only in JSON in the format {"choice": "yes"}. The only two options are "yes" and "no".

nfcorpus_short_questions/ — NFCorpus

  • Source: NFCorpus dev split via ir-datasets.
  • Transformation: NFCorpus associates each article with multiple papers at varying degrees of relevance. We converted this into a pairwise binary-choice task by, for each article, picking one paper at the closest level of separation and one at the furthest, yielding a (correct, distractor) pair. From all such pairs we kept the 500 shortest article+paper combinations (to avoid context-length issues) and split 50/50 into train/test.

multi_condition_ranking_multichoice/ — Multi-Condition Ranking

  • Source: Megagon Labs Multi-Condition Ranking (3-conditions_5-items.jsonl).
  • Expected raw path: datasets/3-conditions_5-items.jsonl.
  • Transformation: shuffled and sampled 500 entries; for each entry the correct 5-item ordering is paired with three distinct random permutations as distractors to form a 4-option multiple-choice question; 50/50 train/test.
  • Regeneration script: scripts/create_multichoice_ranking_samples.py.

Preference datasets

Each preference dataset ships as ten independent <domain>_sample_{1..10} directories, one per randomly selected user. Per user: 50 items from their history are paired with 50 non-history items of equivalent popularity, yielding 50 binary-choice questions split 50/50 into train/test (25 / 25).

anime_sample_{1..10}/ — Anime Pref

book_sample_{1..10}/ — Book Pref

  • Source: Book Recommendation Dataset on Kaggle.
  • Expected raw paths: datasets/book_pref/Ratings.csv, datasets/book_pref/Books.csv (both semicolon-separated).

movie_sample_{1..10}/ — Movie Pref

steam_sample_{1..10}/ — Steam Pref

Regeneration

After placing the raw files at the paths above:

python scripts/create_preference_data_samples.py

This (re)writes all forty preference-sample directories. The script assumes the specific column layouts of the four Kaggle sources cited above; changes to those datasets upstream may require small edits to the corresponding create_* function in the script.