Skip to content

README.md

Latest commit

 

History

History
168 lines (109 loc) · 6.82 KB

README.md

File metadata and controls

168 lines (109 loc) · 6.82 KB
Raw

Adding User Feedback To Enhance CB-Whisper; Massive Open-Vocabulary Keyword-Spotting

This repository contains code that allows to reproduce all experiments performed in the papers "Adding User Feedback To Enhance CB-Whisper" and "Massive Open-Vocabulary Keyword-Spotting".

Setup

Create a conda environment and activate it

  conda create -n biasing-whisper python=3.10
  conda activate biasing-whisper

Install ffmpeg and the necessary requirements

  conda install 'ffmpeg<5'
  pip install -r requirements.txt

Build Datasets

The following bash scripts will create compatible folder structures for the scripts present in this repository, as well as do all the pre-processing necessary to train and evaluate the KWS classifier.

Aishell-KWS and Aishell hotwords subsets

To build the Aishell-KWS dataset, download the data_aishel.tgz file from here and place it on the directory where the dataset will be built.

In the project directory, do as follows

cd datasets/aishell/

Activate the conda environment

conda activate kws

And run the bash script

bash build.sh

You will be asked to provide the path to the tgz file. It will take several hours, so make sure you open some tmux session and let it run uninterruptedly.

MLS-KWS

To build the MLS-KWS dataset, download the zip files from here for the English, German, French, Spanish, Portuguese and Polish languages. Everything else is equivalent to what was done with the Aishell-KWS dataset.

ACL6060

To build the ACL6060 dataset, download the zip file from here. Everything else is equivalent to what was done with the Aishell-KWS dataset.

The KWS Classifier

The CNN classifier for KWS was inspired in the one originally proposed in CB-Whisper. This repository contains additional features that allow to reproduce the experiments on KWS performed in the afforementioned paper:

  • Training using features derived from either TTS-generated or natural-speech audios for the keywords, or a mixture of both;
  • DANN and DANNCE implementation;
  • Validation of checkpoints using more than one dataset (for domain generalization analysis).

Adding User Feedback To Enhance CB-Whisper

In the project directory, do as follows

cd src/

And run the following command

python3 run_CLI.py fit --config configs/train.yaml

In the train.yaml file, you will be able to set different hyperparameters, the paths to the dataset folders, which datasets to validate the model every epoch, the logger, and other training details. Important settings that must be introduced are capitalized and between square brackets.

Evaluation

The following can be used to evaluate the precision, recall and F1 scores of the KWS classifier on the test sets of the different datasets.

In the project directory, do as follows

cd src/

And run the following command

python3 kws.py test --config configs/kws-***.yaml

For ease of use, there is one config yaml file per dataset. Do not forget to set the paths to the dataset folders and the given checkpoint to evaluate. Important settings that must be introduced are capitalized and between square brackets.

Evaluate CB-Whisper with PBAWhisper

The following can be used to evaluate the entity recall of the CB-Whisper model on the test sets of the different datasets, using the KWS classifiers developed with these scripts. These results were not reported in the paper "Adding User Feedback To Enhance CB-Whisper". This version of CB-Whisper uses a wrapped version of Huggingface's WhisperForConditionalGeneration, also known as PBAWhisper, that can perform longform transcription jointly with keyword spotting on the go.

In the project directory, do as follows

cd src/

And run the following command

python3 cb-whisper.py test --config configs/cb-whisper-***.yaml

For ease of use, there is one config yaml file per dataset. Do not forget to set the paths to the dataset folders and the given checkpoint to evaluate. Important settings that must be introduced are capitalized and between square brackets.

Massive Open-Vocabulary Keyword-Spotting

The src/efficient_kws/ package contains a self-contained, pruned variant of the KWS classifier used for recreating the experiments in "Massive Open-Vocabulary Keyword-Spotting". The keyword and utterance embeddings are first compressed and only then turned into cosine-similarity matrices that feed a ResNet classifier. Three projection variants are provided:

  • L — layer: the similarity matrices are computed on the raw Whisper embeddings with spase layer selection;
  • LE — layer + embeddings: as L, plus a per-layer MLP compresses the embedding dimension before computing the similarity matrices;
  • LEF — layer + embeddings + frames: as LE, plus a per-layer temporal Conv1d that also compresses the frames dimension.

This package coexists with the original CNN classifier above: it lives under its own import namespace (efficient_kws.*) and is driven by its own entry point, src/run_efficient_kws.py. The models are trained on MLS-KWS and evaluated on the test sets of ACL6060 and Aishell-KWS.

Training

cd src/
python3 run_efficient_kws.py fit --config efficient_kws/configs/train-L.yaml
python3 run_efficient_kws.py fit --config efficient_kws/configs/train-LE.yaml
python3 run_efficient_kws.py fit --config efficient_kws/configs/train-LEF.yaml

Evaluation

cd src/
python3 run_efficient_kws.py test --config efficient_kws/configs/eval-L-comp-acl.yaml

There is one evaluation config per projection variant and dataset, i.e. eval-{L,LE,LEF}-comp-{acl,aishell}.yaml. As in the configs above, settings that must be provided are capitalized and between square brackets: the dataset roots ([MLS_ROOT], [ACL_ROOT], [AISHELL_ROOT]), the checkpoint to evaluate ([CKPT]), the operating point found on the dev set ([THRESHOLD]), the checkpoints directory ([DEFAULT_ROOT_DIR]) and the MLflow tracking URI ([URL]).

License

See the LICENSE.md file for details.

Citation

If you use any of the resources in this repository, please cite the following papers:

@inproceedings{cbwhisper_userfeedback_2024,
  title     = {Adding User Feedback to Enhance CB-Whisper},
  author    = {Raul Monteiro},
  booktitle = {Proc. Interspeech 2024},
  year      = {2024},
}

@inproceedings{massive_openvocab_kws_2026,
  title     = {Massive Open-Vocabulary Keyword-Spotting},
  author    = {Leonor Barreiros, Raul Monteiro, Afonso Mendes, Gonçalo M. Correia},
  booktitle = {Proc. Interspeech 2026},
  year      = {2026},
  note      = {Accepted; to appear},
}