TeMuDance: Zero-Shot Textual Control for Music-Driven Dance Generation

Achieving high-fidelity 3D dance generation that simultaneously aligns with musical rhythms and adheres to textual instructions presents a significant challenge in multimodal synthesis. To address this gap, we introduce a joint music–text conditioned framework that produces sequences that are both rhythmically aligned and semantically controllable. More extensive visualizations and quantitative analyses are available on our project page.

---

Overview

This repository provides the official implementation of TeMuDance, a framework for music-conditioned 3D dance generation with text guidance.
It supports a training pipeline with:

• Motion-centered contrastive alignment,
• Music-conditioned backbone training,
• Text-guided control branch fine-tuning.

---

Contents

• Installation
• Data Preparation
• Training
  • 1) Contrastive Alignment
  • 2) Train Music-conditioned Backbone
  • 3) Fine-tune Text-guided Control Branch

---

Installation

Requirements

• Python (recommended: 3.8+)
• PyTorch (CUDA recommended)
• accelerate

Setup

git clone https://github.com/XinranLiu7715/TeMuDance.git
cd TeMuDance

If you plan to use multi-GPU training, configure Accelerate once:

accelerate config

---

Data Preparation

This project employs the FineDance dataset for both training and evaluation, and uses HumanML3D as the text–motion source.

1) Download and Placement

• FineDance Dataset
  Download and extract to ./data:
  ⬇️ Download

• HumanML3D Dataset
  Download and extract to ./data:
  ⬇️ Download

• SMPL / SMPL-X Models
  Download required models from the official SMPL-X website and place them in ./assets:
  https://smpl-x.is.tue.mpg.de/

2) Expected Directory Layout

After preparation, your files should be organized as:

TeMuDance/
  data/
    FineDance/              # FineDance dataset root
    HumanML3D/              # HumanML3D dataset root
  assets/
    smplx/                  # SMPL-X models (e.g., SMPLX_NEUTRAL.npz, etc.)
  ...

3) HumanML3D Preprocessing (52-joint Conversion)

We reprocess HumanML3D motions to match the 52-joint setting used in this project.

• Status: Preprocessing code will be released soon
• Output: HumanML3D motions aligned to the 52-joint joint ordering used by TeMuDance

---

Training

All training is performed via train_seq.py. Below we provide the canonical three-stage pipeline.

1) Contrastive Alignment

Train the motion-centered contrastive alignment module:

python train_seq.py --finetune --contrastive

2) Train Music-conditioned Backbone

Train the backbone with Accelerate:

accelerate launch train_seq.py

3) Fine-tune Text-guided Control Branch

Fine-tune the text-guided control branch:

accelerate launch train_seq.py --finetune

Notes:

• If you encounter OOM, reduce batch size and/or enable gradient checkpointing (if supported by your config).
• Ensure your accelerate config matches your GPU environment before multi-GPU runs.

---

Citation

If you find this work useful for your research, please consider citing:

@article{liu2026temudance,
  title={TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation},
  author={Liu, Xinran and Kanojia, Diptesh and Wang, Wenwu and Feng, Zhenhua},
  journal={arXiv preprint arXiv:2604.17005},
  year={2026}
}