-
Notifications
You must be signed in to change notification settings - Fork 50
Update CUDA configuration guide #816
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -43,14 +43,18 @@ to store the symlinks to the drivers. For example, to store these under `/opt/ee | |
| sudo bash -c "echo 'EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia' >> /etc/cvmfs/default.local" | ||
| ``` | ||
|
|
||
| Keep in mind that values stored in `/etc/cvmfs/default.local` can be overridden by values stored in `/etc/cvmfs/domain.d/eessi.io.local`. | ||
|
|
||
| *Step 2 (EESSI 2023.06, optional):* Change the location in which the symlinks will end up by configuring `EESSI_HOST_INJECTIONS` explicitly (default: `/opt/eessi`): | ||
|
|
||
| ```{ .bash copy } | ||
| ```{ .bash .copy } | ||
|
ocaisa marked this conversation as resolved.
|
||
| sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/desired/path/to/host/injections' >> /etc/cvmfs/default.local" | ||
| ``` | ||
|
|
||
| Keep in mind that values stored in `/etc/cvmfs/default.local` can be overridden by values stored in `/etc/cvmfs/domain.d/eessi.io.local`. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See previous comment. |
||
|
|
||
| *Step 3:* To actually reconfigure the variant symlinks, reload the updated CernVM-FS configuration using: | ||
| ```{ .bash copy } | ||
| ```{ .bash .copy } | ||
| sudo cvmfs_config reload software.eessi.io | ||
| ``` | ||
|
|
||
|
|
@@ -66,7 +70,7 @@ sudo cvmfs_config reload software.eessi.io | |
|
|
||
| !!! tip "Maintaining different driver versions for each EESSI version" | ||
| The standard approach for EESSI >= 2025.06 means that the drivers may be found by any EESSI version. If you prefer to create one set of symlinks per EESSI | ||
| version, instead of defining a single location through EESSI_NVIDIA_OVERRIDE_DEFAULT, you can define one per EESSI version, by setting EESSI_<VERSION>_NVIDIA_OVERRIDE. | ||
| version, instead of defining a single location through `EESSI_NVIDIA_OVERRIDE_DEFAULT`, you can define one per EESSI version, by setting `EESSI_<VERSION>_NVIDIA_OVERRIDE`. | ||
|
ocaisa marked this conversation as resolved.
|
||
| For example: | ||
| ```{ .bash .copy} | ||
| sudo bash -c "echo 'EESSI_202506_NVIDIA_OVERRIDE=/opt/eessi/2025.06/nvidia' >> /etc/cvmfs/default.local" | ||
|
|
@@ -96,6 +100,10 @@ Then, change to the correct directory: | |
| - For EESSI 2023.06: `/cvmfs/software.eessi.io/host_injections/${EESSI_VERSION}/compat/${EESSI_OS_TYPE}/${EESSI_CPU_FAMILY}/lib` | ||
|
|
||
| Then, manually create the symlinks for each of the files in the aforementioned list (if they exist on your system) to the current directory. | ||
| You can use the command `locate libcuda.so` to determine the absolute path of the libraries on your system. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not too familiar with Not that I have a better alternative, but maybe we should phrase it a bit more careful (like 'if available, you can use a command like ...').
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is known to the OS, it should appear with |
||
| Sometimes, these libraries have versioned variants (e.g. with extension `.so.1`), which need to be linked as well. | ||
| If a CUDA application fails at runtime due to a missing shared library, you can determine which one | ||
| is missing with `compute-sanitizer --tool memcheck ./myapp` (requires `module load CUDA/<cuda_version>`). | ||
|
|
||
| #### Runtime support when using EESSI in a container {: #nvidia_eessi_container } | ||
|
|
||
|
|
@@ -136,6 +144,11 @@ Third, run the helper script to install the CUDA and cuDNN versions that are use | |
| /cvmfs/software.eessi.io/versions/${EESSI_VERSION}/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh | ||
| ``` | ||
|
|
||
| The script may ask for additional command line options, e.g. to accept the terms of the EULA. | ||
| Be sure to work in a clean environment and outside your local EasyBuild configuration. | ||
| If the installation is interrupted by the error message "Files missing CUDA PTX code", | ||
| re-run the command with environment variable `EASYBUILD_CUDA_SANITY_CHECK_ACCEPT_MISSING_PTX=1` set. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting. I don't have this exactly in my mind, but I guess the EasyBuild we load in I'm wondering if we should put this in the docs - if we tell people to do this any way (and I think it's reasonable...) maybe we better just adjust |
||
|
|
||
| Note that this script uses EasyBuild in order to install CUDA and cuDNN - and EasyBuild does not allow running as root by default. | ||
| The recommended approach is to change ownership of the `host_injections` directory to a non-root user, and perform the installation with | ||
| that user. Alternatively (but not recommended), you can override EasyBuild's behaviour and install as root by setting | ||
|
|
@@ -166,11 +179,13 @@ Thus, you may want to periodically run this script to pick up on new CUDA and cu | |
|
|
||
| The quickest way to test if software installations included in EESSI can access and use your GPU is to run the | ||
| `deviceQuery` executable that is part of the `CUDA-Samples` module: | ||
| ``` | ||
| ```{ .bash .copy } | ||
| module load CUDA-Samples | ||
| deviceQuery | ||
|
Comment on lines
167
to
184
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will work with the 2025.06 stack once EESSI/software-layer#1541 is merged.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That looks like it will take a little longer that hoped...but I will get there! |
||
| ``` | ||
| If both are successful, you should see information about your GPU printed to your terminal. | ||
| If the symlinks are not properly set up, Lmod will fail to find the CUDA runtime environment. | ||
| If `libcuda.so` and its versioned variants are missing, `deviceQuery` will print `cudaGetDeviceCount returned 35`. | ||
|
|
||
| ## Support for using AMD GPUs {: #amd } | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if we should actually modify the instruction here alltogether, and say that they should do:
I'm not sure why we ever decided to put this in
default.local, but it doesn't make much sense. This variant symlink is only relevant for EESSI anyway, it should be in the EESSI-specific (local) config.One could even argue if it should be under
config.d, i.e. the repository-specific files. But... that might bite us later if we ever rename the repo or set up another repo under the same domain in which we want to reuse the same variant symlinks, etc. For now, I'd saydomain.d/eessi.io.localis specific enough.I'd like to get one other opinion on this though. @ocaisa what do you think?