Repository navigation
nvidia: discover the driver under / and normalize container paths - #1083
Merged
maherthomsi merged 2 commits intoOct 16, 2026
Merged
Conversation
containerDriverRoot was the cross-compilation sysroot, so that the generated CDI specification reported library paths as /usr/lib. That sysroot holds only usr, so the nvidia-persistenced socket under /run and the GSP firmware under /lib/firmware were left out of the specification, and nvidia-smi in a container reported persistence mode as Disabled. Set containerDriverRoot to / and add patch 1004, which introduces a --container-path-root flag that replaces that root with / in the container paths. The host paths keep the sysroot. Signed-off-by: Maher Homsi <maherhom@amazon.com>
arnaldo2792
reviewed
Oct 16, 2026
| From ecbbcbfd1e0f7ff6e03ff4c5ad189e24cb2b3d05 Mon Sep 17 00:00:00 2001 | ||
| From: Maher Homsi <maherhom@amazon.com> | ||
| Date: Wed, 14 Oct 2026 21:06:21 +0000 | ||
| Subject: [PATCH] Normalize container paths in generated CDI specifications |
Contributor
There was a problem hiding this comment.
You should open an issue if not already in NVIDIA Device Plugin making the case to support this and we can upstream it.
generate-cdi-specs pinned --device-name-strategy uuid, so /etc/cdi/nvidia.json held UUID names only and declared CDI version 0.3.0, while the device plugin names its devices by index. Pass both naming strategies, which is the nvidia-ctk default, and run nvidia-ctk cdi transform root to replace the cross-compilation sysroot with / in the container paths. The host paths keep the sysroot. Signed-off-by: Maher Homsi <maherhom@amazon.com>
maherthomsi
force-pushed
the
nvidia-cdi-usr-lib-paths
branch
from
October 16, 2026 21:44
673f58d to
b107ce4
Compare
arnaldo2792
approved these changes
Oct 16, 2026
jpculp
approved these changes
Oct 16, 2026
Comment on lines
14
to
+15
| --device-name-strategy uuid \ | ||
| --device-name-strategy index \ |
Member
There was a problem hiding this comment.
I would have thought one would overwrite the other but per the --help:
If this is specified multiple times, the devices will be duplicated for each strategy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes:
The device plugin discovered the driver under the Bottlerocket cross-compilation sysroot, so that the generated CDI specification reported library paths as
/usr/lib/<library>. That sysroot holds onlyusr, so thenvidia-persistencedsocket under/runand the GSP firmware under/lib/firmwarenever resolved there, and the plugin left both out of the specification it writes to/var/run/cdi. A container then reported persistence mode asDisabledwhile the host reportedEnabled. Persistence mode stayed enabled on the GPU, and only the reported value was wrong.nvidia-k8s-device-plugin now discovers the driver under
/, where the sockets and the firmware live, and patch 1004 adds a--container-path-rootflag that replaces that root with/in the container paths of the generated specification. The host paths keep the sysroot, because the files are there.nvidia-container-toolkit now passes
--device-name-strategy uuid --device-name-strategy index, which is the nvidia-ctk default and which the unit had narrowed to UUID names, and runsnvidia-ctk cdi transform root --relative-to containerso that/etc/cdi/nvidia.jsonreports the same/usr/libpaths as the plugin specification.Related PRs:
Testing done:
aws-k8s-1.36-nvidiaandaws-k8s-1.37-nvidiaAMIsnvidia-smi --query-gpu=pci.bus_id,persistence_modereportsEnabledboth on the host and inside a container that requestsnvidia.com/gpu/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.jsoncontains the/run/nvidia-persistenced/socketmount, and that the plugin log reportsSelecting /run/nvidia-persistenced/socket/usr/libwith thelibcuda.so.1symlink chain, and thatldconfig -presolveslibcuda.so.1andlibnvidia-ml.so.1/usr/lib/nvidia/teslacompatibility symlinks still appear on a variant that renders the compat exec-start template/etc/cdi/nvidia.jsondeclares CDI version 0.5.0 and holds both index and UUID device names/dev and devmounts
== ls -l /dev (nvidia) crw-rw-rw-. 1 root root 195, 254 Oct 16 21:24 nvidia-modeset crw-rw-rw-. 1 root root 239, 0 Oct 16 21:24 nvidia-uvm crw-rw-rw-. 1 root root 239, 1 Oct 16 21:24 nvidia-uvm-tools crw-rw-rw-. 1 root root 195, 0 Oct 16 21:24 nvidia0 crw-rw-rw-. 1 root root 195, 255 Oct 16 21:24 nvidiactl == mountinfo entries under /dev or proc driver 899 897 0:95 / /dev rw,nosuid - tmpfs tmpfs rw,context="system_u:object_r:data_t:s0:c247,c908",size=65536k,mode=755 901 899 0:97 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,context="system_u:object_r:data_t:s0:c247,c908",gid=5,mode=620,ptmxmode=666 902 899 0:92 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw,seclabel 903 897 259:17 /var/lib/docker/containers/8e4245c3309dfc5f4d965f647dfc7dc98077b34df1fda92de5ba1b13aff55cc4/resolv.conf /etc/resolv.conf rw,nosuid,nodev,noatime - xfs /dev/nvme1n1p1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=8,swidth=8,noquota 904 897 259:17 /var/lib/docker/containers/8e4245c3309dfc5f4d965f647dfc7dc98077b34df1fda92de5ba1b13aff55cc4/hostname /etc/hostname rw,nosuid,nodev,noatime - xfs /dev/nvme1n1p1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=8,swidth=8,noquota 905 897 259:17 /var/lib/docker/containers/8e4245c3309dfc5f4d965f647dfc7dc98077b34df1fda92de5ba1b13aff55cc4/hosts /etc/hosts rw,nosuid,nodev,noatime - xfs /dev/nvme1n1p1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=8,swidth=8,noquota 906 899 0:72 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,context="system_u:object_r:data_t:s0:c247,c908",size=65536k 909 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/nvidia-cuda-mps-control /usr/bin/nvidia-cuda-mps-control ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 910 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/nvidia-cuda-mps-server /usr/bin/nvidia-cuda-mps-server ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 911 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/nvidia-debugdump /usr/bin/nvidia-debugdump ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 912 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/nvidia-persistenced /usr/bin/nvidia-persistenced ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 913 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/nvidia-smi /usr/bin/nvidia-smi ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 914 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libEGL.so.580.178.04 /usr/lib/libEGL.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 915 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libEGL_nvidia.so.580.178.04 /usr/lib/libEGL_nvidia.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 916 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libGLESv1_CM_nvidia.so.580.178.04 /usr/lib/libGLESv1_CM_nvidia.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 917 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libGLESv2_nvidia.so.580.178.04 /usr/lib/libGLESv2_nvidia.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 918 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libGLX_nvidia.so.580.178.04 /usr/lib/libGLX_nvidia.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 919 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libcuda.so.580.178.04 /usr/lib/libcuda.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 920 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libcudadebugger.so.580.178.04 /usr/lib/libcudadebugger.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 921 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvcuvid.so.580.178.04 /usr/lib/libnvcuvid.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 922 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-allocator.so.580.178.04 /usr/lib/libnvidia-allocator.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 923 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-cfg.so.580.178.04 /usr/lib/libnvidia-cfg.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 924 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-eglcore.so.580.178.04 /usr/lib/libnvidia-eglcore.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 925 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-encode.so.580.178.04 /usr/lib/libnvidia-encode.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 926 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-fbc.so.580.178.04 /usr/lib/libnvidia-fbc.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 927 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-glcore.so.580.178.04 /usr/lib/libnvidia-glcore.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 928 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-glsi.so.580.178.04 /usr/lib/libnvidia-glsi.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 929 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-glvkspirv.so.580.178.04 /usr/lib/libnvidia-glvkspirv.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 930 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-gpucomp.so.580.178.04 /usr/lib/libnvidia-gpucomp.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 931 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-ml.so.580.178.04 /usr/lib/libnvidia-ml.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 932 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-ngx.so.580.178.04 /usr/lib/libnvidia-ngx.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 933 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-nvvm.so.580.178.04 /usr/lib/libnvidia-nvvm.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 934 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-opencl.so.580.178.04 /usr/lib/libnvidia-opencl.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 935 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-opticalflow.so.580.178.04 /usr/lib/libnvidia-opticalflow.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 936 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-pkcs11-openssl3.so.580.178.04 /usr/lib/libnvidia-pkcs11-openssl3.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 937 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-pkcs11.so.580.178.04 /usr/lib/libnvidia-pkcs11.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 938 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-present.so.580.178.04 /usr/lib/libnvidia-present.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 939 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-ptxjitcompiler.so.580.178.04 /usr/lib/libnvidia-ptxjitcompiler.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 940 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-rtcore.so.580.178.04 /usr/lib/libnvidia-rtcore.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 941 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvidia-tls.so.580.178.04 /usr/lib/libnvidia-tls.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 942 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libnvoptix.so.580.178.04 /usr/lib/libnvoptix.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 943 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/libvdpau_nvidia.so.580.178.04 /usr/lib/libvdpau_nvidia.so.580.178.04 ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 944 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/kmod /usr/local/sbin/modprobe ro,relatime master:1 - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 945 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/vulkan/icd.d/nvidia_icd.json /etc/vulkan/icd.d/nvidia_icd.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 946 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/vulkan/icd.d/nvidia_layers.json /etc/vulkan/icd.d/nvidia_layers.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 947 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/vulkan/icd.d/nvidia_layers.json /etc/vulkan/implicit_layer.d/nvidia_layers.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 948 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/nvidia/nvoptix.bin /usr/share/nvidia/nvoptix.bin ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 949 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/firmware/nvidia/580.178.04/gsp_ga10x.bin /usr/lib/firmware/nvidia/580.178.04/gsp_ga10x.bin ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 950 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/firmware/nvidia/580.178.04/gsp_tu10x.bin /usr/lib/firmware/nvidia/580.178.04/gsp_tu10x.bin ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 951 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 952 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 953 897 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/share/glvnd/egl_vendor.d/10_nvidia.json /usr/share/glvnd/egl_vendor.d/10_nvidia.json ro,nosuid,nodev,relatime - erofs /dev/root ro,seclabel,user_xattr,acl,cache_strategy=readaround 967 898 0:99 /nvct-params /proc/driver/nvidia/params rw,relatime - tmpfs tmpfs rw,seclabel,size=4kTerms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.