Skip to content
Snippets Groups Projects
Commit 16e178f8 authored by David Carreto Fidalgo's avatar David Carreto Fidalgo
Browse files

improve toc + remove local to hpc workflow

parent a64243e3
No related branches found
No related tags found
1 merge request!2Feat/readme
......@@ -215,18 +215,18 @@ You can apply overlays with the `run`, `exec`, `shell` and `instance start` comm
The [Remote Visualisation Service (RVS)](https://docs.mpcdf.mpg.de/doc/visualization/index.html) allows you to run Jupyter sessions on the HPC systems.
You can use your container as a kernel within such a session by providing a `kernel.json` spec file.
### 1. Setting up the container
**1. Setting up the container**
Make sure you install ipython and ipykernel in your container:
```
pip install ipython ipykernel
```
### 2. Setting up RVS
**2. Setting up RVS**
Load apptainer module when initializing your RVS session.
### 3. Creating the kernel
**3. Creating the kernel**
Create a kernel spec file
```bash
......@@ -263,52 +263,3 @@ The next time you request a jupyter session, you can choose the generic jupyter
Keep in mind that you are inside the container.
If you want to access files outside your home directory, you have to bind them explicitly in the kernel spec file when calling the apptainer command.
For example, in the kernel spec file above we bind your `ptmp` folder.
## Local-to-HPC Workflow
**TODO: The sandbox option does not work 100% correctly for VSCode or PyCharm, use docker images instead! Need to update this guide!**
A nice workflow to develop a python library locally and deploy it on our HPO systems (sharing exactly the same environment) is to use the [*sandbox* feature](https://apptainer.org/docs/user/main/build_a_container.html#sandbox) of Apptainer.
We are still investigating if something similar is possible with `Docker` (please let us know if you find a way :) ).
### 1. Create a definition file
In the root directory of your library (repository) create a *definition* `*.def` file.
This definition file should reflect your environment in which you want your library to develop and use.
You can leverage base environments, such as docker images on DockerHub, or existing apptainers.
### 2. Build the sandbox
Build the sandbox (container in a directory) instead of the default SIF format:
```shell
apptainer build --fakeroot --sandbox my_container my_container.def
```
### 3. Install your library in the sandbox
Now we can add our library that we develop to the sandbox environment and install it in [`editable`](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) mode:
```shell
apptainer exec --writable my_container python -m pip install -e .
```
### 4. Point your IDE's interpreter to the sandbox
You should be able to point the interpreter of your IDE (VSC, PyCharm, etc.) to the python executable inside the sandbox folder.
### 5. Add your developed library to the my_container.def file
While in principle you could build a SIF container directly from your sandbox, it is better to modify your *definition* `*.def` file to include your library/package.
In this way, your container is fully reproducible using only the definition file.
### 6. Build your *.sif apptainer, deploy on our HPC systems
Once you built the SIF container, you can copy it to our HPC systems and use it there.
```shell
apptainer build --fakeroot my_container.sif my_container.def
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment