fix: persistent_workers crash at num_workers=0 + warn on silent token truncation - #1329
Open
rohitsynth1208 wants to merge 1 commit into
Conversation
… truncation
- SemanticDataModule's train/val dataloaders hardcoded
persistent_workers=True, which PyTorch's DataLoader rejects outright when
num_workers=0 ("persistent_workers option needs num_workers > 0"). This is
a legitimate config for small datasets/debugging, so make it conditional.
- TextDataCollator.batchify() silently slices any example longer than
max_length with no signal that truncation happened. Since each packed
example's length includes its full audio-codec token sequence (not just
text), and sample_data()'s num_samples heuristic only estimates length
from text tokens, examples can end up truncated mid-audio with zero
indication - silently corrupting the training target. Log a warning so
users can notice and raise max_length instead of training on truncated
audio without knowing it.
Found while fine-tuning text2semantic on a small (~1k clip) custom dataset,
where num_workers=0 is necessary and truncation was happening unnoticed.
rohitsynth1208
force-pushed
the
fix/dataloader-persistent-workers-and-truncation-warning
branch
from
August 27, 2026 11:09
1a97f8a to
33eec14
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is this PR adding new feature or fix a BUG?
Fix BUG (two small, independent fixes in
fish_speech/datasets/semantic.py).Is this pull request related to any issue? If yes, please link the issue.
Not filed as a separate issue, but related in spirit to #1163 (pretrained weights being silently destroyed) - this PR is about a different pair of silent failure modes in the dataset/dataloader path that we hit while LoRA fine-tuning
text2semanticon a small (~1,000 clip) custom dataset.1.
persistent_workers=Truehardcoded, crashes atnum_workers=0SemanticDataModule.train_dataloader()/val_dataloader()hardcodepersistent_workers=Trueregardless ofnum_workers. PyTorch'sDataLoaderraises immediately ifpersistent_workers=Trueis passed withnum_workers=0:num_workers=0is a legitimate, common choice for small datasets (few shards) or when debugging - the finetune docs' own recipe doesn't forbid it. Madepersistent_workersconditional onnum_workers > 0, matching the flag's actual precondition.2. Silent truncation with no warning in
TextDataCollator.batchify()This slices any example longer than
max_lengthwith zero log/warning. Since a packed example's token sequence includes the full audio-codec token stream for its sentence(s) - not just text - andAutoTextSemanticInstructionIterableDataset.sample_data()'snum_samples = self.max_length // 20heuristic only estimates length from text tokens (# estimate that each sample is at least 20 tokens), it's easy to end up with examples that exceedmax_lengthonce audio tokens are counted, and get silently truncated mid-audio - corrupting the training target with no indication anything went wrong.Added a
log.warning(...)when this truncation actually fires, so users get a clear signal to raisemax_length(or reducenum_samples) instead of unknowingly training on truncated audio. Purely additive/observability - no change to actual training behavior.Testing
Both fixes verified against a real fine-tuning run (LoRA,
text2semantic_finetune, ~1,000 clip custom dataset,num_workers=0, H100): the first fix is required just to get training running atnum_workers=0at all; the second warning was verified to correctly fire on examples we confirmed (by manual inspection) were being truncated mid-audio under the default heuristic.— Submitted by SynthioLabs
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.