Skip to content

fix: persistent_workers crash at num_workers=0 + warn on silent token truncation - #1329

Open
rohitsynth1208 wants to merge 1 commit into
fishaudio:mainfrom
SynthioLabsLtd:fix/dataloader-persistent-workers-and-truncation-warning
Open

fix: persistent_workers crash at num_workers=0 + warn on silent token truncation#1329
rohitsynth1208 wants to merge 1 commit into
fishaudio:mainfrom
SynthioLabsLtd:fix/dataloader-persistent-workers-and-truncation-warning

Conversation

@rohitsynth1208

@rohitsynth1208 rohitsynth1208 commented Aug 27, 2026

Copy link
Copy Markdown

Is this PR adding new feature or fix a BUG?

Fix BUG (two small, independent fixes in fish_speech/datasets/semantic.py).

Is this pull request related to any issue? If yes, please link the issue.

Not filed as a separate issue, but related in spirit to #1163 (pretrained weights being silently destroyed) - this PR is about a different pair of silent failure modes in the dataset/dataloader path that we hit while LoRA fine-tuning text2semantic on a small (~1,000 clip) custom dataset.

1. persistent_workers=True hardcoded, crashes at num_workers=0

SemanticDataModule.train_dataloader()/val_dataloader() hardcode persistent_workers=True regardless of num_workers. PyTorch's DataLoader raises immediately if persistent_workers=True is passed with num_workers=0:

ValueError: persistent_workers option needs num_workers > 0

num_workers=0 is a legitimate, common choice for small datasets (few shards) or when debugging - the finetune docs' own recipe doesn't forbid it. Made persistent_workers conditional on num_workers > 0, matching the flag's actual precondition.

2. Silent truncation with no warning in TextDataCollator.batchify()

_tokens = example[tokens_key][:, :max_tokens_length]

This slices any example longer than max_length with zero log/warning. Since a packed example's token sequence includes the full audio-codec token stream for its sentence(s) - not just text - and AutoTextSemanticInstructionIterableDataset.sample_data()'s num_samples = self.max_length // 20 heuristic only estimates length from text tokens (# estimate that each sample is at least 20 tokens), it's easy to end up with examples that exceed max_length once audio tokens are counted, and get silently truncated mid-audio - corrupting the training target with no indication anything went wrong.

Added a log.warning(...) when this truncation actually fires, so users get a clear signal to raise max_length (or reduce num_samples) instead of unknowingly training on truncated audio. Purely additive/observability - no change to actual training behavior.

Testing

Both fixes verified against a real fine-tuning run (LoRA, text2semantic_finetune, ~1,000 clip custom dataset, num_workers=0, H100): the first fix is required just to get training running at num_workers=0 at all; the second warning was verified to correctly fire on examples we confirmed (by manual inspection) were being truncated mid-audio under the default heuristic.

— Submitted by SynthioLabs


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

… truncation

- SemanticDataModule's train/val dataloaders hardcoded
  persistent_workers=True, which PyTorch's DataLoader rejects outright when
  num_workers=0 ("persistent_workers option needs num_workers > 0"). This is
  a legitimate config for small datasets/debugging, so make it conditional.

- TextDataCollator.batchify() silently slices any example longer than
  max_length with no signal that truncation happened. Since each packed
  example's length includes its full audio-codec token sequence (not just
  text), and sample_data()'s num_samples heuristic only estimates length
  from text tokens, examples can end up truncated mid-audio with zero
  indication - silently corrupting the training target. Log a warning so
  users can notice and raise max_length instead of training on truncated
  audio without knowing it.

Found while fine-tuning text2semantic on a small (~1k clip) custom dataset,
where num_workers=0 is necessary and truncation was happening unnoticed.
@rohitsynth1208
rohitsynth1208 force-pushed the fix/dataloader-persistent-workers-and-truncation-warning branch from 1a97f8a to 33eec14 Compare August 27, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant