Skip to content

First Iteration of size configurable GPU buffer for faster checkpoint capture - #49

Open
Jaiaid wants to merge 4 commits into
ECP-VeloC:gpu_supportfrom
Jaiaid:gpu_support
Open

First Iteration of size configurable GPU buffer for faster checkpoint capture#49
Jaiaid wants to merge 4 commits into
ECP-VeloC:gpu_supportfrom
Jaiaid:gpu_support

Conversation

@Jaiaid

@Jaiaid Jaiaid commented Aug 15, 2026

Copy link
Copy Markdown

Modification:

  • A slot-based GPU buffer and host staging area where each slots are of 64 MiB size and the size of allocated memory is controllable by two optional config variables
    • gpu_buffer_slots: controls the number of slots in the GPU (default 8)
    • host_stage_slots: control number of slots in host, pinned if pinning is available (default 16)
  • transfer engine update with two-thread transfer worker to accelerate capture when checkpoint size is small enough or GPU/host buffer is large enough
  • GPU device is set to rank in round-robin manner

Correctness is checked by:

  • checking if checkpoint restore works
  • MPI run works
  • performance values match with the system

Jaiaid added 4 commits August 15, 2026 02:45
* buffer provision, slot acquire, release method's implementation for CUDA GPU
* round-robin GPU buffer placement based on process rank and device count
* transfer_engine instance method interface update to provide process rank, which helps determining in which GPU to allocate buffer
* A two thread architecture to decouple user data -> device -> host transfer and host -> file transfer. This is done to have accelerated checkpoint capture if GPU buffer is large enough or checkpoint is small enough
* two config variables to control host slot count and GPU buffer slot count
* Transfer chunk size is increased to 64 MiB to have better throughput in transfer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant