EKS GPUs - Disable GSP from userData script #176
Conversation
| rmmod nvidia_drm | ||
| rmmod nvidia_modeset | ||
| rmmod nvidia_uvm | ||
| rmmod nvidia |
There was a problem hiding this comment.
This is on a best-effort basis which is why this section is at the end of the script. rmmod for a module fails if some processes are actively using the module or any of its dependents have not been unloaded before.
| echo "Writing NVreg_EnableGpuFirmware=0 to /etc/modprobe.d/nvidia-gsp.conf" | ||
| echo "options nvidia NVreg_EnableGpuFirmware=0" | tee --append /etc/modprobe.d/nvidia-gsp.conf | ||
| echo "Running dracut" | ||
| dracut -f |
There was a problem hiding this comment.
This will have no effect until reboot if the module was not successfully unloaded before we edit the kernel params
There was a problem hiding this comment.
Is EKS fine with this?
How much latency are we adding here?
There was a problem hiding this comment.
Is EKS fine with this?
Does not cause any problems as such but the problem is this works now but if some config changes under our feet then it would stop working.
How much latency are we adding here?
Maybe 1-2s, I can measure it out
This is a hacky but working (so far) solution to NVIDIA/open-gpu-kernel-modules#446 and awslabs/amazon-eks-ami#1523