btl/self: add accelerator-aware memory copy for put/get - #13746
Conversation
|
@edgargabriel will have a deeper look before signoff, mostly correct |
bosilca
left a comment
bosilca
left a comment
There was a problem hiding this comment.
You need to set the BTL flags to reflect this new capability and avoid the PML staging the transfers through CPU memory. Add MCA_BTL_FLAGS_ACCELERATOR_RDMA.
I would have expected to have to define a btl_register_mem function or the RDMCA cuda will skip the optimized path.
You need to call the provided callback even in an error case and pass the appropriate error to it. We didn't do it before because memcpy could not fail (or really bad things were happening), but now there are more opportunities for badness.
The btl/self component was not accelerator aware. Send-to-self operations involving GPU memory would copy data through the host instead of performing a direct device memory copy. Add mca_btl_self_memcpy() which checks whether src/dst buffers reside in accelerator memory via opal_accelerator.check_addr() and dispatches to opal_accelerator.mem_copy() accordingly, falling back to memcpy() when both buffers are in host memory. Signed-off-by: Gonzalosilvalde <gonzalo.silvalde@gmail.com>
|
Hi, thanks for the review. |
|
@Gonzalosilvalde @bosilca thank you both for this PR, can we cherry-pick that to the 6.0.x branch? |
Sure, go ahead! No objections from my side. |
|
/backport v5.0.x |
|
/backport v5.0.x |
|
/backport v6.0.x |
|
Hm, please wait with merging this to 5.0.x and 6.0.x. I have tested the patch and I do not see the accelerator path ever be invoked, something is still missing. |
Hi, I've been looking at the commit again since I didn't understand why you were seeing it not work. I think I got mixed up at some point while undoing changes or something like that. I'm pretty sure the line |
|
Should be enough now, sorry for the trouble. |
|
@Gonzalosilvalde sorry, where is this patch? Did you file a new PR with the modification? |
4 participants
The btl/self component was not accelerator aware. Send-to-self operations involving GPU memory would copy data through the host instead of performing a direct device memory copy.
Add mca_btl_self_memcpy() which checks whether src/dst buffers reside in accelerator memory via opal_accelerator.check_addr() and dispatches to opal_accelerator.mem_copy() accordingly, falling back to memcpy() when both buffers are in host memory.
Fixes #12230