Turbulence Tools: Lagrangian and Eulerian

Introduction

TurTLE implements a number of standard functionality of Fourier-based pseudo-spectral numerical simulations, as well as the corresponding numerical particle-tracking functionality. The package also contains a Navier-Stokes solver, as well as a small number of post-processing tools. The solver is production-ready, having already been used for a number of publications.

TurTLE is written in C++ and it uses a hybrid MPI/OpenMP programming paradigm, relying on FFTW3 for an efficient Fourier transform implementation. HDF5 is used for I/O, including for parameter input and output of statistics. CMake is used for compilation and installation.

A Python 3 wrapper is used to prepare parameters and initial conditions, as well as to generate job submission scripts for clusters and to perform basic post-processing of simulation results.

The primary aim of TurTLE is to provide optimal performance, while reducing the time spent on setting up and supervising ensembles of DNS, with the added benefit of a unified launch-postprocess approach through the use of the Python wrapper.

Installation

TurTLE can be used on various machines, with laptops routinely being used for development and testing, but large production runs using tens of thousands of CPU cores on large computing clusters.

Postprocessing only

The Python 3 package may be installed directly if only post-processing of existing data is desired:

python setup.py install

(add --user or sudo as appropriate). setup.py uses the setuptools package for dependency resolution.

Full installation

The C++ library requires a number of dependencies, that CMake will search for before compilation and installation. In brief, an MPI compiler is required, as well as the HDF5 C library compiled with parallel support and FFTW >= 3.4. Detailed instructions are provided at the end of this document.

Documentation

Partial documentation is available (link to online documentation coming soon). Doxygen is used for the C++ source code, and the Sphinx and breathe Python packages are used to merge the Python wrapper documentation with the C++ documentation. The optional CMake targets doc_doxygen, doc_html and doc_latex generate the required documents in the build directory.

Comments

• the cmake folder contains files extracted from https://gitlab.inria.fr/solverstack/morse_cmake, a separate project licensed under the "CeCILL-C" license, please see http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html for details.
• particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail.
• code is only known to work with HDF5 1.8.x.
• code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x

Full installation

These installation steps assume that you have a working MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as the cmake tool. The list is a work in progress, please contact us (Cristian.Lalescu@ds.mpg.de) if the procedure fails at any step of the process.

Note: the HDF5 1.8x library may require MPI standard v1, we haven't tested this in detail. Some default OpenMPI3 installations will not work.

Detailed full installation instructions:

1. Export the following environment variables with apropriate values:

   • MPICXX - Preferred MPI C++ compiler
   • FFTW_DIR - Base directory of FFTW
   • FFTW_INCDIR - Directory containing FFTW header files
   • FFTW_LIBDIR - Directory containing FFTW library files
   • FFTW_OPENMP_LIBDIR - Directory containing OpenMP FFTW library files (please define if different from FFTW_LIBDIR)
   • FFTW_MPI_LIBDIR - Directory containing MPI FFTW library files (please define if different from FFTW_LIBDIR)
   • HDF5_ROOT - Base directory of HDF5
   • CMAKE_PREFIX_PATH - Directory containing local cmake configuration files

   We also recommend that an environment variable TURTLE_COMPILATION_FLAGS is defined appropriately. In particular, for clusters of unknown architecture it helps to log into individual nodes and run the following command:

   gcc -march=native -Q --help=target

   Note that an example file example.bashrc is included in the repository. There is at least one recorded case of a cluster where different FFTW libraries (serial, MPI and OpenMP) were located in different folders, hence the options of specifying the details. See also lines 143 through 160 of CMakeLists.txt.

   If using openmpi you may need to recompile it with the --enable-mpi1-compatibility flag, in order for HDF5 to compile and link (step 4).

2. Make directory PREFIX on a local fast partition.

3. Download, compile, install FFTW (latest version 3.x from http://www.fftw.org/). Execute the following commands in order, feel free to customize optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):

   ./configure --prefix=PREFIX --enable-float --enable-sse --enable-mpi --enable-openmp --enable-threads
   make
   make install
   ./configure --prefix=PREFIX  --enable-sse2 --enable-avx512 --enable-mpi --enable-openmp --enable-threads
   make
   make install

   TurTLE will try to find FFTW using the FindFFTW from the Morse project. If the package is installed in a non standard location, it is recommanded to setup the environment variables: FFTW_DIR, FFTW_INCDIR, FFTW_LIBDIR, FFTW_OPEMMP_LIBDIR and FFTW_MPI_LIBDIR.

4. Download, compile, install HDF5 (version 1.8.x, currently available at https://portal.hdfgroup.org/display/support/HDF5+1.8.20#files). We are using parallel I/O, therefore we must use the plain C interface of HDF5:

   ./configure --prefix=PREFIX --enable-parallel
   make
   make install

   TurTLE will try to find HDF5 using the regular FindHDF5. Therefore, if the package is installed in a non standard location, it is recommanded to setup the environment variable: HDF5_ROOT.

5. TurTLE requires cmake version > 3.12. If required, download, compile and install cmake, currently available at https://cmake.org/cmake/resources/software.html.

   ./bootstrap --prefix=PREFIX
   make
   make install

5. Optional. We recommend the creation of a virtual python3 environment (also under PREFIX) that will be used for installing TurTLE and dependencies. Please see https://docs.python-guide.org/dev/virtualenvs/.

6. Clone turtle repository.

   git clone git@gitlab.mpcdf.mpg.de:clalescu/turtle.git

7. Copy the pc_host_info.py file to host_info.py file and edit according to the instructions in the file. This must be done before installing on a cluster.

8. Go into TurTLE repository, execute

   mkdir build
   cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=INSTALL_DIR
   make
   # to get a verbose compilation process, use
   # VERBOSE=1 make
   # to use more than one core for the compilation use
   # make -jN
   # (where N is the number of available cores)
   make install

9. If you used a custom install location (i.e. CMAKE_INSTALL_PREFIX) you must include this location in the environment variable CMAKE_PREFIX_PATH. This ensures that the required TurTLEConfig.cmake file is accessible for future use by the package. Alternatively you may set the TurTLE_DIR environment variable to include the location instead.

10. Using TurTLE from an external project. TurTLE creates and installs 3 files alongside the C++ headers and library:

    -- Installing: install/lib/TurTLEConfig.cmake
    -- Installing: install/lib/TurTLE_EXPORT.cmake
    -- Installing: install/lib/TurTLE_EXPORT-noconfig.cmake

    In case these files provide incomplete information, it is necessary to update the cmake input config file: turtle/cmake/TurTLEConfig.cmake.in.

11. Optional: building for multiple target systems. When creating several different build folders for different targets, it is recommended to create build-specific bash settings in a file named bash_setup_for_TurTLE.sh, as well as a build-specific copy of the host_info.py file. In the extra_slurm_lines list from host_info one should then add a corresponding source INSTALL_DIR/lib/bash_setup_for_TurTLE.sh line, such that the appropriate setup is loaded in the job script.

Scaling tests.

Initial scaling data is available at (url coming soon). Please copy the files to the location TURTLE_FIELD_DATABASE.

Separately, please recompile TurTLE with the TIMING_OUTPUT cmake option switched to ON.

Afterwards, please run variations of the following command:

python ${TURTLE_REPOSITORY}/tests/DNS/test_scaling.py D \
    -n 128 \
    --nprocesses 4 \
    --ncores 1 \
    --src-wd ${TURTLE_FIELD_DATABASE} \
    --src-iteration 8192

Available iterations for

• n = 128: 8192