Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions docs/site_specific_config/gpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,18 @@ to store the symlinks to the drivers. For example, to store these under `/opt/ee
sudo bash -c "echo 'EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia' >> /etc/cvmfs/default.local"
```

Keep in mind that values stored in `/etc/cvmfs/default.local` can be overridden by values stored in `/etc/cvmfs/domain.d/eessi.io.local`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we should actually modify the instruction here alltogether, and say that they should do:

sudo bash -c "echo 'EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia' >> /etc/cvmfs/domain.d/eessi.io.local

I'm not sure why we ever decided to put this in default.local, but it doesn't make much sense. This variant symlink is only relevant for EESSI anyway, it should be in the EESSI-specific (local) config.

One could even argue if it should be under config.d, i.e. the repository-specific files. But... that might bite us later if we ever rename the repo or set up another repo under the same domain in which we want to reuse the same variant symlinks, etc. For now, I'd say domain.d/eessi.io.local is specific enough.

I'd like to get one other opinion on this though. @ocaisa what do you think?


*Step 2 (EESSI 2023.06, optional):* Change the location in which the symlinks will end up by configuring `EESSI_HOST_INJECTIONS` explicitly (default: `/opt/eessi`):

```{ .bash copy }
```{ .bash .copy }
Comment thread
ocaisa marked this conversation as resolved.
sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/desired/path/to/host/injections' >> /etc/cvmfs/default.local"
```

Keep in mind that values stored in `/etc/cvmfs/default.local` can be overridden by values stored in `/etc/cvmfs/domain.d/eessi.io.local`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment.


*Step 3:* To actually reconfigure the variant symlinks, reload the updated CernVM-FS configuration using:
```{ .bash copy }
```{ .bash .copy }
sudo cvmfs_config reload software.eessi.io
```

Expand All @@ -66,7 +70,7 @@ sudo cvmfs_config reload software.eessi.io

!!! tip "Maintaining different driver versions for each EESSI version"
The standard approach for EESSI >= 2025.06 means that the drivers may be found by any EESSI version. If you prefer to create one set of symlinks per EESSI
version, instead of defining a single location through EESSI_NVIDIA_OVERRIDE_DEFAULT, you can define one per EESSI version, by setting EESSI_<VERSION>_NVIDIA_OVERRIDE.
version, instead of defining a single location through `EESSI_NVIDIA_OVERRIDE_DEFAULT`, you can define one per EESSI version, by setting `EESSI_<VERSION>_NVIDIA_OVERRIDE`.
Comment thread
ocaisa marked this conversation as resolved.
For example:
```{ .bash .copy}
sudo bash -c "echo 'EESSI_202506_NVIDIA_OVERRIDE=/opt/eessi/2025.06/nvidia' >> /etc/cvmfs/default.local"
Expand Down Expand Up @@ -96,6 +100,10 @@ Then, change to the correct directory:
- For EESSI 2023.06: `/cvmfs/software.eessi.io/host_injections/${EESSI_VERSION}/compat/${EESSI_OS_TYPE}/${EESSI_CPU_FAMILY}/lib`

Then, manually create the symlinks for each of the files in the aforementioned list (if they exist on your system) to the current directory.
You can use the command `locate libcuda.so` to determine the absolute path of the libraries on your system.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too familiar with locate, but this instruction might not very generically work. Not all systems have locate installed. Also, as far as I know, it searches a database, which may or may not have indexed the relevant path. So... your mileage may vary.

Not that I have a better alternative, but maybe we should phrase it a bit more careful (like 'if available, you can use a command like ...').

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is known to the OS, it should appear with ldconfig -p | grep cuda

Sometimes, these libraries have versioned variants (e.g. with extension `.so.1`), which need to be linked as well.
If a CUDA application fails at runtime due to a missing shared library, you can determine which one
is missing with `compute-sanitizer --tool memcheck ./myapp` (requires `module load CUDA/<cuda_version>`).

#### Runtime support when using EESSI in a container {: #nvidia_eessi_container }

Expand Down Expand Up @@ -136,6 +144,11 @@ Third, run the helper script to install the CUDA and cuDNN versions that are use
/cvmfs/software.eessi.io/versions/${EESSI_VERSION}/scripts/gpu_support/nvidia/install_cuda_and_libraries.sh
```

The script may ask for additional command line options, e.g. to accept the terms of the EULA.
Be sure to work in a clean environment and outside your local EasyBuild configuration.
If the installation is interrupted by the error message "Files missing CUDA PTX code",
re-run the command with environment variable `EASYBUILD_CUDA_SANITY_CHECK_ACCEPT_MISSING_PTX=1` set.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I don't have this exactly in my mind, but I guess the EasyBuild we load in install_cuda_and_libraries.sh uses the same config as EESSI-extend - and I'm guessing that set a --cuda-compute-capabilities value if you're on a GPU-enabled node?

I'm wondering if we should put this in the docs - if we tell people to do this any way (and I think it's reasonable...) maybe we better just adjust install_cuda_and_libraries.sh and make sure it gets set there (as environment variable, or even as a command line option).


Note that this script uses EasyBuild in order to install CUDA and cuDNN - and EasyBuild does not allow running as root by default.
The recommended approach is to change ownership of the `host_injections` directory to a non-root user, and perform the installation with
that user. Alternatively (but not recommended), you can override EasyBuild's behaviour and install as root by setting
Expand Down Expand Up @@ -166,11 +179,13 @@ Thus, you may want to periodically run this script to pick up on new CUDA and cu

The quickest way to test if software installations included in EESSI can access and use your GPU is to run the
`deviceQuery` executable that is part of the `CUDA-Samples` module:
```
```{ .bash .copy }
module load CUDA-Samples
deviceQuery
Comment on lines 167 to 184

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will work with the 2025.06 stack once EESSI/software-layer#1541 is merged.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks like it will take a little longer that hoped...but I will get there!

```
If both are successful, you should see information about your GPU printed to your terminal.
If the symlinks are not properly set up, Lmod will fail to find the CUDA runtime environment.
If `libcuda.so` and its versioned variants are missing, `deviceQuery` will print `cudaGetDeviceCount returned 35`.

## Support for using AMD GPUs {: #amd }

Expand Down
2 changes: 1 addition & 1 deletion docs/systems.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ MareNostrum 5 is the EuroHPC JU supercomputer hosted by the [Barcelona Supercomp

### MeluXina (Luxembourg)

MeluXina is the EuroHPC JU supercomputer hosted by the [Barcelona Supercomputing Center (BSC)](https://www.bsc.es/).
MeluXina is the EuroHPC JU supercomputer hosted by [LuxProvide](https://www.luxprovide.lu/).

* [Website](https://www.luxprovide.lu/meluxina/)
* [General documentation](https://docs.lxp.lu/)
Expand Down
Loading