Skip to content
Snippets Groups Projects
Commit 2ae04646 authored by Lukas Bentkamp's avatar Lukas Bentkamp
Browse files

WIP: installation instructions

parent 80f9952e
No related branches found
No related tags found
1 merge request!38Feature/readme
Pipeline #120496 passed
Stage: build
Stage: test
Show whitespace changes
Inline Side-by-side
README.rst
+ 98
− 70
View file @ 2ae04646
......@@ -60,9 +60,9 @@ TurTLE contains contributions from:
.. _sec-installation:
------------
---------------------------------------------------
Installation for postprocessing only
------------
---------------------------------------------------
The Python 3 package may be installed directly if only post-processing
of existing data is desired:
......@@ -75,9 +75,9 @@ of existing data is desired:
(add `--user` or `sudo` as appropriate).
`setup.py` uses the `setuptools` package for dependency resolution.
------------
----------------------
Full installation
------------
----------------------
The C++ library requires a number of dependencies, that `CMake` will
search for before compilation and installation.
......@@ -87,9 +87,9 @@ We provide instructions for local compilation of FFTW and HDF5, because
default versions packaged with Linux variants are typically inadequately
configured.
These installation steps assume that you have a working MPI compiler,
These installation steps assume that you have a working C/C++ MPI compiler,
properly configured on your system (i.e. the various configure scripts
are able to find it), as well as the `cmake` tool.
are able to find it), as well as an installation of Python 3.
The list is a work in progress, please contact us
(Cristian.Lalescu@mpcdf.mpg.de) if the procedure fails at any step of the
process.
......@@ -101,7 +101,7 @@ this in detail. Some default OpenMPI3 installations will not work.
Detailed full installation instructions:
**Preparation**
**Installation of requirements**
TurTLE has the following requirements:
......@@ -111,9 +111,11 @@ TurTLE has the following requirements:
3. cmake version > 3.12
We recommend installing these requirements on a local fast partition. In the following, we refer to their locations as `FFTW_DIR`, `HDF5_DIR` and `CMAKE_DIR`.
1. **Installation of FFTW**
Download latest version from http://www.fftw.org/.
To compile and install in custom location FFTW_DIR, execute the following commands in order, feel free to customize
To compile and install it in custom location FFTW_DIR, execute the following commands in order, feel free to customize
optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):
.. code:: bash
......@@ -124,6 +126,7 @@ TurTLE has the following requirements:
./configure --prefix=FFTW_DIR --enable-sse2 --enable-avx512 --enable-mpi --enable-openmp --enable-threads
make
make install
..
TurTLE will try to find FFTW using the FindFFTW from the Morse project.
If the package is installed in a non standard location, you should
......@@ -131,29 +134,30 @@ TurTLE has the following requirements:
`FFTW_LIBDIR`, `FFTW_OPEMMP_LIBDIR` and `FFTW_MPI_LIBDIR`.
2. **Installation of HDF5**
Download HDF5 from https://www.hdfgroup.org/downloads/hdf5/. To compile and install in custom location HDF5_ROOT, execute the following commands in order:
Download HDF5 from https://www.hdfgroup.org/downloads/hdf5/. To compile and install it in custom location HDF5_DIR, execute the following commands in order:
.. code:: bash
./configure --prefix=HDF5_ROOT --enable-parallel
./configure --prefix=HDF5_DIR --enable-parallel
make
make install
Since TurTLE is using parallel I/O, HDF5 needs to be compiled with the "--enable-parallel" flag.
..
TurTLE will try to find HDF5 using the regular FindHDF5, which
searches system folders, or `HDF5_ROOT`.
searches system folders, or `HDF5_DIR`.
3. **Installation of cmake**
TurTLE requires `cmake` version > 3.12, which should be available
from your default package manager.
If not possible, then download cmake at https://cmake.org/cmake/resources/software.html. To compile and install in custom location CMAKE_DIR, execute the following commands in order:
Check if cmake (version >3.12) is available from your default package manager.
If not, then download cmake at https://cmake.org/cmake/resources/software.html. To compile and install it in custom location CMAKE_DIR, execute the following commands in order:
.. code:: bash
./bootstrap --prefix=CMAKE_DIR
make
make install
..
The value of `PREFIX` used above is only relevant to later executing
the `cmake` binary (which can be found under `${PREFIX}/bin` after
......@@ -172,46 +176,55 @@ TurTLE has the following requirements:
* `MPICXX` - Preferred MPI C++ compiler
* `HDF5_ROOT` - Base directory of HDF5
**Installation**
**Installation of TurTLE**
1. We recommend creating a virtual environment for TurTLE. To do this,
choose an installation location INSTALL_LOC on a local fast partition
(Under unix systems, this could be e.g. `~/.local`) and a name for the
environment VENV_NAME (e.g. `turtle-production`). To create the virtual environment, execute:
environment VENV_NAME (e.g. `turtle-production`). To cCC=${GCC_HOME}/bin/gcc
In the following, we refer to the path `INSTALL_LOC/VENV_NAME` as `TURTLE_DIR`.
.. code:: bash
python -m venv INSTALL_LOC/VENV_NAME
In the following, we refer to the path `INSTALL_LOC/VENV_NAME` as `TurTLE_DIR`.
2. Clone turtle repository.
2. Choose a location for the source code, enter it and clone turtle repository by
.. code:: bash
git clone git@gitlab.mpcdf.mpg.de:TurTLE/turtle.git
3. Go into TurTLE repository, execute
3. Execute
.. code:: bash
cd turtle
mkdir build
cd build
cp ../pc_host_info.py ./host_info.py
Edit the `host_info.py` file according to the instructions in the file.
This is strictly required before installing on a cluster.
4. In the build folder, create a file `bash_setup_for_TurTLE.sh`. It will set all required environment variables.
4.
Optional.
*PINCHECK* is available from https://gitlab.mpcdf.mpg.de/khr/pincheck.
If you'd like to check whether TurTLE MPI processes and OpenMP
threads are pinned properly to hardware threads, you can simply
place the pincheck headers under `${PREFIX}/include`.
.. code:: bash
5.
In the build folder, edit the file `bash_setup_for_TurTLE.sh` so that it
exports the following environment variables with apropriate values:
#! /bin/bash
# module-dependent variables
export CC=
export CXX=
export MPI_HOME=
export FFTW_ROOT=FFTW_DIR
export FFTW_DIR=FFTW_DIR
export HDF5_ROOT=HDF5_DIR
# custom location for gcc compilation
export TURTLE_ROOT=TURTLE_DIR
# location-dependent variables
export CUSTOM_INSTALL_PATH=${TURTLE_ROOT}
export PATH=${CUSTOM_INSTALL_PATH}/bin:${PATH}
export CMAKE_PREFIX_PATH=${CUSTOM_INSTALL_PATH}/lib
# activate python virtual environment
source ${TURTLE_ROOT}/bin/activate
..
* `MPICXX` - Preferred MPI C++ compiler
* `FFTW_DIR` - Base directory of FFTW
* `FFTW_INCDIR` - Directory containing FFTW header files
......@@ -219,15 +232,52 @@ TurTLE has the following requirements:
* `FFTW_OPENMP_LIBDIR` - Directory containing OpenMP FFTW library files (please define if different from FFTW_LIBDIR)
* `FFTW_MPI_LIBDIR` - Directory containing MPI FFTW library files (please define if different from FFTW_LIBDIR)
* `HDF5_ROOT` - Base directory of HDF5
* `PINCHECK_ROOT` - Directory under which
`include/pincheck.hpp` may be found (OPTIONAL)
* `TurTLE_DIR` - Directory under which TurTLE will be installed (same value from step 2).
I.e. library will go under `${TurTLE_DIR}/lib`, headers under
`${TurTLE_DIR}/include`, etc.
This script should also contain the line
`source TurTLE_DIR/lib/bash_setup_for_TurTLE.sh`, as well as the
line `source TurTLE_DIR/bin/activate`.
* `TURTLE_DIR` - Directory under which TurTLE will be installed (same value from step 2).
I.e. library will go under `${TURTLE_DIR}/lib`, headers under
`${TURTLE_DIR}/include`, etc.
This script should also contain the
line `source TURTLE_DIR/bin/activate`.
5. Rename the file `pc_host_info.py` in the following way
.. code:: bash
cp ../pc_host_info.py ./host_info.py
This file contains information about the machine on which TurTLE will run.
On desktop machines, no further steps are required.
On clusters, please edit the `host_info.py` file according to the instructions in the file.
6.
Finally, we are ready to install TurTLE. Within the build folder, execute
.. code:: bash
source bash_setup_for_TurTLE.sh
cmake .. -DCMAKE_INSTALL_PREFIX=TURTLE_DIR
Then compile TurTLE with `N` cores using
.. code:: bash
make -jN
make install
Congratulations, you have installed TurTLE!
**Optional**
5.
Optional.
*PINCHECK* is available from https://gitlab.mpcdf.mpg.de/khr/pincheck.
If you'd like to check whether TurTLE MPI processes and OpenMP
threads are pinned properly to hardware threads, you can simply
place the pincheck headers under `${PREFIX}/include`.
* `PINCHECK_ROOT` - Directory under which `include/pincheck.hpp` may be found (OPTIONAL)
1.
We also recommend that an environment variable `TURTLE_COMPILATION_FLAGS`
is defined appropriately.
In particular, for clusters of unknown architecture it helps to log into
......@@ -251,22 +301,7 @@ TurTLE has the following requirements:
*Note*: in principle it is possible to add this information to your
`.bashrc`, but we recommend against it.
6.
In the previously created build folder, execute
.. code:: bash
source bash_setup_for_TurTLE.sh
cmake .. -DCMAKE_INSTALL_PREFIX=TurTLE_DIR
make
# to get a verbose compilation process, use
# VERBOSE=1 make
# to use more than one core for the compilation use
# make -jN
# (where N is the number of available cores)
make install
7.
8.
Using TurTLE from an external project.
TurTLE creates and installs 3 files alongside the C++ headers and
library:
......@@ -277,17 +312,10 @@ TurTLE has the following requirements:
TurTLE_EXPORT.cmake
TurTLE_EXPORT-noconfig.cmake
These files are installed under `${TurTLE_DIR}/lib`.
These files are installed under `${TURTLE_DIR}/lib`.
In case these files provide incomplete information, it is necessary to update
the cmake input config file: `turtle/cmake/TurTLEConfig.cmake.in`.
8.
For clusters.
In the `extra_slurm_lines` list from `host_info.py` one should add a
corresponding `source INSTALL_DIR/lib/bash_setup_for_TurTLE.sh`
line (where you should write the explicit install folder instead of
`INSTALL_DIR`), such that the appropriate setup is loaded in the job
script.
**Uninstall**
......
......
pc_host_info.py
+ 2
− 0
View file @ 2ae04646
......@@ -65,6 +65,8 @@ host_info = {'type' : 'pc'}
# info_template_extra_slurm_lines, relevant for 'SLURM' clusters,
# is a list of strings, written as extra options within the SLURM file.
# for example something like `#SBATCH --get-user-env`
# Here, also the virtual environment and environment variables need to be loaded,
# e.g. by `source TURTLE_DIR/lib/bash_setup_for_TurTLE.sh`
# info_template_explicit_slurm_environment, relevant for `SLURM` clusters,
# is a bool specifying whether the environment must be mentioned explicitly
......
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment