Enable FSDP strategy on CPU (accelerator="cpu") #21812

Motivation

Currently, attempting to use FSDPStrategy with CPU in PyTorch Lightning raises an artificial ValueError during trainer initialization:

ValueError: The strategy `fsdp` requires a GPU accelerator, but received `accelerator='cpu'`.

Allowing FSDP to run on CPU is highly valuable for several key workflows:

1. Storage I/O Macrobenchmarking: Enabling CPU-based FSDP allows us to run macrobenchmarks on checkpoint save/restore storage I/O without consuming expensive GPU hours. Isolating storage network and disk throughput from GPU compute provides clear signals for future performance improvements in distributed checkpointing.
2. Local Development & Debugging: Developers can build, debug, and verify distributed FSDP code locally on laptops without requiring GPU access.
3. CI & Unit Testing: Enables fast, inexpensive multi-process FSDP end-to-end testing in CPU-only CI environments (e.g., GitHub Actions).

Related Issues

• This feature proposal is blocked by / related to SIGSEGV (Segmentation Fault) when saving checkpoints with FSDP on CPU #21804, which tracks the crash/segmentation fault when saving full state dict checkpoints under CPU FSDP. Once it is resolved, CPU support can be fully enabled and tested for both full and sharded checkpoint formats.
• Precursor work in fix(FSDP): pass explicit CPU device to satisfy torch>=2.5 device_id contract #21774 resolved PyTorch >= 2.5 contract issues by passing explicit CPU device_id s to FSDP, unblocking CPU compatibility at the PyTorch wrapper level.

Pitch

I propose officially enabling accelerator="cpu" support for FSDPStrategy :

1. Remove Accelerator Restrictions: Update _AcceleratorConnector to accept "cpu" and CPUAccelerator instances when using FSDPStrategy .
2. Update Error Messaging: Revise validation error messages to state that FSDP accepts CUDA, GPU, or CPU accelerators.
3. Add Comprehensive E2E Testing: Introduce multi-rank CPU training tests (e.g., test_fsdp_cpu_trainable ) that verify:
   • Genuine FSDP layer wrapping ( FullyShardedDataParallel ) on CPU across multiple ranks.
   • Training loss convergence and optimizer stepping under the Gloo distributed backend.
   • Checkpoint saving and round-trip parameter verification for both sharded and full ( state_dict_type ) checkpoint formats once SIGSEGV (Segmentation Fault) when saving checkpoints with FSDP on CPU #21804 is resolved.

Alternatives

No response

Additional context

No response

cc @lantiga