Skip to content
Snippets Groups Projects

AI Containers

Goal: compile definition files of containers for AI use cases. Also provide documentation on how to use them on our HPC systems.

Getting started

We use Apptainer to build/run containers on our HPC systems. You will need a Linux system to run Apptainer natively on your machine, and it’s easiest to install if you have root access.

But it is also easy to use or convert docker images with Apptainer.

For a nice introduction to Apptainer on our HPC systems, have a look at the awesome presentation by Michele. You can also browse our documentation.

Building containers

Containers are built via a definition file and the apptainer build command.

In each folder of this repo you will find a definition .def file and a README.md that describes the exact build command.

Pull from Dokcer Hub

You can easily pull containers from the Docker Hub or other OCI registries:

$ apptainer pull my_apptainer.sif docker://sylabsio/lolcow:latest

Convert from Docker Daemon or Docker Archive files

You can also convert images/containers running in your Docker Daemon:

$ apptainer build my_apptainer.sif docker-daemon:sylabsio/lolcow:latest

Or convert a Docker Archive file:

$ sudo docker images
REPOSITORY                        TAG               IMAGE ID       CREATED          SIZE
sylabsio/lolcow                   latest            5a15b484bc65   2 hours ago      188MB

$ docker save 5a15b484bc65 -o lolcow.tar
$ apptainer build my_apptainer.sif docker-archive:lolcow.tar

Running containers

TODO:

  • mention important flags, like --nv for example
  • how to run the containers on our SLURM cluster

Using containers with RVS

The Remote Visualisation Service (RVS) allows you to run Jupyter sessions on the HPC systems. You can use your container as a kernel within such a session by providing a kernel.json spec file.

1. Setting up the container

Make sure you install ipython and ipykernel in your container:

pip install ipython ipykernel

2. Setting up RVS

Load apptainer module when initializing your RVS session.

3. Creating the kernel

Create a kernel spec file

vim ~/.local/share/jupyter/kernels/my-kernel/kernel.json

that should look something like this

{
 "argv": [
  "apptainer",
  "exec",
  "--nv",
  "--bind",
  "{connection_file}:/tmp/connection_spec,/ptmp/<your user name>",
  "/absolute/path/to/your/container.sif",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "/tmp/connection_spec"
 ],
 "display_name": "Name of your kernel",
 "language": "python",
 "metadata": {
  "debugger": true
 },
 "env": {
  "PYTHONPATH": "/add/custom/packages/here/if/you/want"
 }
}

The next time you request a jupyter session, you can choose the generic jupyter version, and use your custom kernel. Keep in mind that you are inside the container. If you want to access files outside your home directory, you have to bind them explicitly in the kernel spec file when calling the apptainer command. For example, in the kernel spec file above we bind your ptmp folder.

Local-to-HPC Workflow

TODO: The sandbox option does not work 100% correctly for VSCode or PyCharm, use docker images instead! Need to update this guide!

A nice workflow to develop a python library locally and deploy it on our HPO systems (sharing exactly the same environment) is to use the sandbox feature of Apptainer.

We are still investigating if something similar is possible with Docker (please let us know if you find a way :) ).

1. Create a definition file

In the root directory of your library (repository) create a definition *.def file. This definition file should reflect your environment in which you want your library to develop and use.

You can leverage base environments, such as docker images on DockerHub, or existing apptainers.

2. Build the sandbox

Build the sandbox (container in a directory) instead of the default SIF format:

apptainer build --fakeroot --sandbox my_container my_container.def

3. Install your library in the sandbox

Now we can add our library that we develop to the sandbox environment and install it in editable mode:

apptainer exec --writable my_container python -m pip install -e .

4. Point your IDE's interpreter to the sandbox

You should be able to point the interpreter of your IDE (VSC, PyCharm, etc.) to the python executable inside the sandbox folder.

5. Add your developed library to the my_container.def file

While in principle you could build a SIF container directly from your sandbox, it is better to modify your definition *.def file to include your library/package. In this way, your container is fully reproducible using only the definition file.

6. Build your *.sif apptainer, deploy on our HPC systems

Once you built the SIF container, you can copy it to our HPC systems and use it there.

apptainer build --fakeroot my_container.sif my_container.def