Turbulence Tools: Lagrangian and Eulerian
In brief, this code runs pseudospectral direct numerical simulations (DNS) of the incompressible Navier-Stokes equations, using FFTW 3, and it can integrate particle trajectories in the resulting fields.
The Navier-Stokes solver has been extensively tested (tests are included in the repository), and it is working as expected. Parameters and statistics are stored in HDF5 format, together with code information, so simulation data should be "future proof" --- suggestions of possible improvements to the current approach are always welcome.
The primary aim of TurTLE is to reduce the time spent on setting up and baby sitting DNS, as well as simplify the analysis of the generated data. The wish is that this Python package provides an easy and general way of constructing efficient specialized DNS C++ codes for different turbulence problems encountered in research. At the same time, the package should provide a unified way of postprocessing, and accessing the postprocessing results. The code therefore consists of two main parts: the pure C++ code, a set of loosely related "building blocks", and the Python code, which can generate C++ code using the pure classes, but with a significant degree of flexibility.
The code user is expected to write a small python script that will properly define the DNS they are interested in running. That code will generate an executable that can then be run directly on the user's machine, or submitted to a queue on a cluster.
Installation
So far, the code has been run on laptops, desktops, and a couple of clusters (biggest run so far was 1536^3 on 16 nodes of 32 cores each, with about 11 seconds per time step, for a simple incompressible Navier-Stokes problem). Postprocessing data may not be very computationally intensive, depending on the amount of data involved.
Postprocessing only
Use a console; navigate to the turtle
folder, and type:
python setup.py install
(add --user or sudo as appropriate). setup.py should tell you about the various packages you need.
Full installation
If you want to run simulations on the machine where you're installing, you will need to use cmake to compile and install the full library. Your machine needs to have an MPI compiler installed, the HDF5 C library and FFTW >= 3.4 --- detailed instructions are included at the end of this document.
Documentation
While the code is not fully documented yet, basic information is already available, and it is recommended that you generate the manual and go through it carefully. Please do ask for specific improvements to the current text where it is found lacking. In order to generate the manual, navigate to the repository folder, and execute the following commands:
cd documentation
make latexpdf
Optionally, html documentation can be generated instead if needed, just
type make html
instead of make latexpdf
.
Comments
- the cmake folder contains files extracted from https://gitlab.inria.fr/solverstack/morse_cmake, a separate project licensed under the "CeCILL-C" license, please see http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html for details.
- particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail.
- code is only known to work with HDF5 1.8.x.
- code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x
Installation with prerequisites
These installation steps assume that you have a working MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as the cmake tool. The list is a work in progress, please contact us (Cristian.Lalescu@ds.mpg.de) if the procedure fails at any step of the process.
Detailed full installation instructions:
-
- Export the following environment variables with apropriate values:
-
- MPICXX - Preferred MPI C++ compiler
- FFTW_DIR - Base directory of FFTW
- FFTW_INCDIR - Directory containing FFTW header files
- FFTW_LIBDIR - Directory containing FFTW library files
- HDF5_ROOT - Base directory of HDF5
- CMAKE_PREFIX_PATH - Directory containing local cmake configuration files
We also recommend that an environment variable TURTLE_OPTIMIZATION_FLAGS is defined appropriately. In particular, for clusters of unknown architecture it helps to log into individual nodes and run the following command:
gcc -march=native -Q --help=target
-
Make directory PREFIX on a local fast partition.
-
Download, compile, install FFTW (latest version 3.x from http://www.fftw.org/). Execute the following commands in order, feel free to customize optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):
./configure --prefix=PREFIX --enable-float --enable-sse --enable-mpi --enable-openmp --enable-threads make make install ./configure --prefix=PREFIX --enable-sse2 --enable-avx512 --enable-mpi --enable-openmp --enable-threads make make install
TurTLE will try to find FFTW using the FindFFTW from the Morse project. If the package is installed in a non standard location, it is recommanded to setup the environment variables: FFTW_DIR, FFTW_INCDIR and FFTW_LIBDIR.
-
Download, compile, install HDF5 (version 1.8.x, currently available at https://portal.hdfgroup.org/display/support/HDF5+1.8.20#files). We are using parallel I/O, therefore we must use the plain C interface of HDF5:
./configure --prefix=PREFIX --enable-parallel make make install
TurTLE will try to find HDF5 using the regular FindHDF5. Therefore, if the package is installed in a non standard location, it is recommanded to setup the environment variable: HDF5_ROOT.
-
Optional. We recommend the creation of a virtual python3 environment (also under PREFIX) that will be used for installing TurTLE and dependencies. Please see https://docs.python-guide.org/dev/virtualenvs/.
-
Clone turtle repository.
git clone git@gitlab.mpcdf.mpg.de:clalescu/turtle.git
-
Copy the pc_host_info.py file to host_info.py file and edit according to the instructions in the file. This must be done before installing on a cluster.
-
Go into TurTLE repository, execute
mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=INSTALL_DIR make # to get a verbose compilation process, use # VERBOSE=1 make # to use more than one core for the compilation use # make -jN # (where N is the number of available cores) make install
-
If you used a custom install location (i.e. CMAKE_INSTALL_PREFIX) you must include this location in the environment variable CMAKE_PREFIX_PATH. This ensures that the required TurTLEConfig.cmake file is accessible for future use by the package.
-
Using TurTLE from an external project. TurTLE creates and installs 3 files alongside the C++ headers and library:
-- Installing: install/lib/TurTLEConfig.cmake -- Installing: install/lib/TurTLE_EXPORT.cmake -- Installing: install/lib/TurTLE_EXPORT-noconfig.cmake
In case these files provide incomplete information, it is necessary to update the cmake input config file: turtle/cmake/TurTLEConfig.cmake.in.