https://github.com/NVIDIA/TransformerEngine/pull/3186/changes recently added support for TE to pass `cu_seqlens*` tensors directly to cuDNN. It'd be nice if cuDNN could also accept "actual seqlens" because TE-JAX has to launch a kernel for it.
https://github.com/NVIDIA/TransformerEngine/pull/3186/changes recently added support for TE to pass
cu_seqlens*tensors directly to cuDNN.It'd be nice if cuDNN could also accept "actual seqlens" because TE-JAX has to launch a kernel for it.