Skip to content
Snippets Groups Projects

Turbulence Tools: Lagrangian and Eulerian

Introduction

TurTLE implements a number of standard functionality of Fourier-based pseudo-spectral numerical simulations, as well as the corresponding numerical particle-tracking functionality. The package also contains a Navier-Stokes solver, as well as a small number of post-processing tools. The solver is production-ready, having already been used for a number of publications.

TurTLE is written in C++ and it uses a hybrid MPI/OpenMP programming paradigm, relying on FFTW3 for an efficient Fourier transform implementation. HDF5 is used for I/O, including for parameter input and output of statistics. CMake is used for compilation and installation.

A Python 3 wrapper is used to prepare parameters and initial conditions, as well as to generate job submission scripts for clusters and to perform basic post-processing of simulation results.

The primary aim of TurTLE is to provide optimal performance, while reducing the time spent on setting up and supervising ensembles of DNS, with the added benefit of a unified launch-postprocess approach through the use of the Python wrapper.

Background and documentation

Partial documentation is available at http://TurTLE.pages.mpcdf.de/turtle/sphinx_html; the C++ API is described at http://TurTLE.pages.mpcdf.de/turtle/html (doxygen-generated).

Authors

TurTLE is developed and maintained by the Wilczek group at the Max Planck Institute for Dynamics and Self-Organization and the University of Bayreuth in collaboration with the Application Support Group of the Max Planck Computing and Data Facility.

TurTLE contains contributions from:

Installation

TurTLE can be used on various machines, with laptops routinely being used for development and testing, but large production runs using tens of thousands of CPU cores on large computing clusters.

Postprocessing only

The Python 3 package may be installed directly if only post-processing of existing data is desired:

bash configure_python.sh
python setup.py install

(add --user or sudo as appropriate). setup.py uses the setuptools package for dependency resolution.

Full installation

The C++ library requires a number of dependencies, that CMake will search for before compilation and installation. In brief, an MPI compiler is required, as well as the HDF5 C library compiled with parallel support and FFTW >= 3.3.4. Detailed instructions are provided at the end of this document.

These installation steps assume that you have a working MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as the cmake tool. The list is a work in progress, please contact us (Cristian.Lalescu@mpcdf.mpg.de) if the procedure fails at any step of the process.

Note: the HDF5 1.8x library may require MPI standard v1, we haven't tested this in detail. Some default OpenMPI3 installations will not work.

Detailed full installation instructions:

  1. Export the following environment variables with apropriate values:
    • MPICXX - Preferred MPI C++ compiler
    • FFTW_DIR - Base directory of FFTW
    • FFTW_INCDIR - Directory containing FFTW header files
    • FFTW_LIBDIR - Directory containing FFTW library files
    • FFTW_OPENMP_LIBDIR - Directory containing OpenMP FFTW library files (please define if different from FFTW_LIBDIR)
    • FFTW_MPI_LIBDIR - Directory containing MPI FFTW library files (please define if different from FFTW_LIBDIR)
    • HDF5_ROOT - Base directory of HDF5
    • CMAKE_PREFIX_PATH - Directory containing local cmake configuration files
    • PINCHECK_ROOT - Directory under which include/pincheck.hpp may be found (OPTIONAL)

    We also recommend that an environment variable TURTLE_COMPILATION_FLAGS is defined appropriately. In particular, for clusters of unknown architecture it helps to log into individual nodes and run the following command:

    gcc -march=native -Q --help=target

    Note that an example file example.bashrc is included in the repository. There is at least one recorded case of a cluster where different FFTW libraries (serial, MPI and OpenMP) were located in different folders, hence the options of specifying the details. See also lines 143 through 160 of CMakeLists.txt.

    If using openmpi you may need to recompile it with the --enable-mpi1-compatibility flag, in order for HDF5 to compile and link (step 4).

    PINCHECK is available from https://gitlab.mpcdf.mpg.de/khr/pincheck

  2. Make directory PREFIX on a local fast partition.

  3. Download, compile, install FFTW (latest version 3.x from http://www.fftw.org/). Execute the following commands in order, feel free to customize optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):

    ./configure --prefix=PREFIX --enable-float --enable-sse --enable-mpi --enable-openmp --enable-threads
    make
    make install
    ./configure --prefix=PREFIX  --enable-sse2 --enable-avx512 --enable-mpi --enable-openmp --enable-threads
    make
    make install

    TurTLE will try to find FFTW using the FindFFTW from the Morse project. If the package is installed in a non standard location, it is recommanded to setup the environment variables: FFTW_DIR, FFTW_INCDIR, FFTW_LIBDIR, FFTW_OPEMMP_LIBDIR and FFTW_MPI_LIBDIR.

  4. Download, compile, install HDF5 (version 1.8.x, currently available at https://portal.hdfgroup.org/display/support/HDF5+1.8.20#files). We are using parallel I/O, therefore we must use the plain C interface of HDF5:

    ./configure --prefix=PREFIX --enable-parallel
    make
    make install

    TurTLE will try to find HDF5 using the regular FindHDF5. Therefore, if the package is installed in a non standard location, it is recommanded to setup the environment variable: HDF5_ROOT.

  5. TurTLE requires cmake version > 3.12. If required, download, compile and install cmake, currently available at https://cmake.org/cmake/resources/software.html.

    ./bootstrap --prefix=PREFIX
    make
    make install
  1. Optional. We recommend the creation of a virtual python3 environment (also under PREFIX) that will be used for installing TurTLE and dependencies. Please see https://docs.python-guide.org/dev/virtualenvs/.

  2. Clone turtle repository.

    git clone git@gitlab.mpcdf.mpg.de:TurTLE/turtle.git
  3. Copy the pc_host_info.py file to host_info.py file and edit according to the instructions in the file. This must be done before installing on a cluster.

  4. Go into TurTLE repository, execute

    mkdir build
    cd build
    cmake .. -DCMAKE_INSTALL_PREFIX=INSTALL_DIR
    make
    # to get a verbose compilation process, use
    # VERBOSE=1 make
    # to use more than one core for the compilation use
    # make -jN
    # (where N is the number of available cores)
    make install
  5. If you used a custom install location (i.e. CMAKE_INSTALL_PREFIX) you must include this location in the environment variable CMAKE_PREFIX_PATH. This ensures that the required TurTLEConfig.cmake file is accessible for future use by the package. Alternatively you may set the TurTLE_DIR environment variable to include the location instead.

  6. Using TurTLE from an external project. TurTLE creates and installs 3 files alongside the C++ headers and library:

    -- Installing: install/lib/TurTLEConfig.cmake
    -- Installing: install/lib/TurTLE_EXPORT.cmake
    -- Installing: install/lib/TurTLE_EXPORT-noconfig.cmake

    In case these files provide incomplete information, it is necessary to update the cmake input config file: turtle/cmake/TurTLEConfig.cmake.in.

  7. Optional: building for multiple target systems. When creating several different build folders for different targets, it is recommended to create build-specific bash settings in a file named bash_setup_for_TurTLE.sh, as well as a build-specific copy of the host_info.py file. In the extra_slurm_lines list from host_info one should then add a corresponding source INSTALL_DIR/lib/bash_setup_for_TurTLE.sh line, such that the appropriate setup is loaded in the job script.

Documentation

A local build of the documentation is possible where necessary. Doxygen is used for the C++ source code, and the Sphinx and breathe Python packages are used to merge the Python wrapper documentation with the C++ documentation. The optional CMake targets doc_doxygen, doc_html and doc_latex generate the required documents in the build directory. As long as the full cmake installation is possible (see below), one would proceed as follows to generate the documentation locally:

mkdir build-doc
cd build-doc
cmake ..
make doc_doxygen
make doc_html
cd sphinx_latex
make

After these steps, the (HTML) manual is available under build-doc/sphinx_html/index.html, and a PDF version of the manual can be found at build-doc/sphinx_latex/TurTLE.pdf. The stand-alone doxygen-generated documentation is present at build-doc/html/index.html.

Reference publication

Please see https://arxiv.org/abs/2107.01104 for a description of TurTLE, as well as a detailed discussion of the novel particle tracking approach. This is also the publication to be cited by works that made use of TurTLE.

Contact

If you have any questions, comments or suggestions, please contact Dr. Cristian C. Lalescu (Cristian.Lalescu@mpcdf.mpg.de).

Comments