Support for collections of HF datasets, --top-k, update to input masking training, etc.#4
Open
Support for collections of HF datasets, --top-k, update to input masking training, etc.#4
Conversation
Added --mask-input/--no-mask-input. Integrated handling for input masking and learning rate schedules. Added new CLI argument `--mask-input`.
…_sweep) This commit adds a new script entry for 'mlx_tuning_fork_wandb_sweep' in the pyproject.toml file. This allows running the 'wandb_sweep' module directly as a script, simplifying the workflow.
Replaced an outdated dataset import with config imports for more accurate settings. Also, incorporated default validation parameters to enhance configuration consistency.
Renamed CONFIG_DEFAULTS to TF_CONFIG_DEFAULTS to clarify its origin from the tuning fork module. This change ensures better readability and maintainability of the configuration parameters being used in the project.
Updated length calculations to use a fixed number of iterations rather than the dataset length and updates to training steps and validation intervals
Added `--mask-inputs` option to enable input masking during training. Integrates new batch iterator and loss function when input masking is activated, and replaces learning schedule building logic with mlx_lm's.
Updated `train_set` to `train_dataset` and `valid_set` to `val_dataset` for consistency with the rest of the codebase.
Eliminated the colorize option from CLI arguments and its associated print statements to clean up the code and simplify the `generate.py` logic. The `wandb_sweep.py` file had a minor formatting update as well.
Removed unused 'generate' import from training.py and 'save_config' and 'Path' imports from wandb_sweep.py to clean up the code. Moved the YAML loader setup inside the main function in wandb_sweep.py for better encapsulation.
Support Fix config defaults
…ing training Input masking training added in lieu of ml-explore/mlx-examples#1103
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.