Big Fluid and Particle Simulator

In brief, this code runs pseudospectral direct numerical simulations (DNS) of the incompressible Navier-Stokes equations, using FFTW 3, and it can integrate particle trajectories in the resulting fields.

The Navier-Stokes solver has been extensively tested (tests are included in the repository), and it is working as expected. Parameters and statistics are stored in HDF5 format, together with code information, so simulation data should be "future proof" --- suggestions of possible improvements to the current approach are always welcome.

The primary aim of bfps is to reduce the time spent on setting up and baby sitting DNS, as well as simplify the analysis of the generated data. The wish is that this Python package provides an easy and general way of constructing efficient specialized DNS C++ codes for different turbulence problems encountered in research. At the same time, the package should provide a unified way of postprocessing, and accessing the postprocessing results. The code therefore consists of two main parts: the pure C++ code, a set of loosely related "building blocks", and the Python code, which can generate C++ code using the pure classes, but with a significant degree of flexibility.

The code user is expected to write a small python script that will properly define the DNS they are interested in running. That code will generate an executable that can then be run directly on the user's machine, or submitted to a queue on a cluster.

Installation

So far, the code has been run on laptops, desktops, and a couple of clusters (biggest run so far was 1536^3 on 16 nodes of 32 cores each, with about 11 seconds per time step, for a simple incompressible Navier-Stokes problem). Postprocessing data may not be very computationally intensive, depending on the amount of data involved.

Postprocessing only

Use a console; navigate to the bfps folder, and type:

python setup.py install

(add --user or sudo as appropriate). setup.py should tell you about the various packages you need.

Full installation

If you want to run simulations on the machine where you're installing, you will need to use cmake to compile and install the full library. Your machine needs to have an MPI compiler installed, the HDF5 C library and FFTW >= 3.4 --- detailed instructions are included at the end of this document.

Documentation

While the code is not fully documented yet, basic information is already available, and it is recommended that you generate the manual and go through it carefully. Please do ask for specific improvements to the current text where it is found lacking. In order to generate the manual, navigate to the repository folder, and execute the following commands:

cd documentation
make latexpdf

Optionally, html documentation can be generated instead if needed, just type make html instead of make latexpdf.

Comments

• the cmake folder contains files extracted from https://gitlab.inria.fr/solverstack/morse_cmake, a separate project licensed under the "CeCILL-C" license, please see http://www.cecill.info/licences/Licence_CeCILL-C_V1-en.html for details.
• particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail.
• code is only known to work with HDF5 1.8.x.
• code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x

Installation with prerequisites

These installation steps assume that you have a working MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as the cmake tool. We recommend to specify the desired MPI C++ compiler by exporting the environment variable MPICXX --- the BFPS cmake configuration looks for this variable. We also recommend that an environment variable BFPS_OPTIMIZATION_FLAGS is defined appropriately. In particular, for clusters of unknown architecture it helps to log into individual nodes and run the following command:

gcc -march=native -Q --help=target

Detailed full installation instructions:

1. Make directory PREFIX on a local fast partition.

2. Download, compile, install FFTW (latest version 3.x from http://www.fftw.org/). Execute the following commands in order, feel free to customize optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):

   ./configure --prefix=PREFIX --enable-float --enable-sse --enable-mpi --enable-openmp --enable-threads
   make
   make install
   ./configure --prefix=PREFIX  --enable-sse2 --enable-avx512 --enable-mpi --enable-openmp --enable-threads
   make
   make install

   BFPS will try to find FFTW using the FindFFTW from the Morse project. If the package is installed in a non standard location, it is recommanded to setup the environment variables: FFTW_DIR (or FFTW_INCDIR and FFTW_LIBDIR).

3. Download, compile, install HDF5 (version 1.8.x, currently available at https://portal.hdfgroup.org/display/support/HDF5+1.8.20#files). We are using parallel I/O, therefore we must use the plain C interface of HDF5:

   ./configure --prefix=PREFIX --enable-parallel
   make
   make install

   BFPS will try to find HDF5 using the regular FindHDF5. Therefore, if the package is installed in a non standard location, it is recommanded to setup the environment variable: HDF5_ROOT.

3. Optional. We recommend the creation of a virtual python3 environment (also under PREFIX) that will be used for installing bfps and dependencies. Please see https://docs.python-guide.org/dev/virtualenvs/.

4. Clone bfps repository.

   git clone git@gitlab.mpcdf.mpg.de:clalescu/bfps.git

5. Go into bfps repository, execute

   mkdir build
   cd build
   cmake ..
   # possibly : cmake .. -DCMAKE_INSTALL_PREFIX=INSTALL_DIR
   make
   # to get a verbose compilation process, use
   VERBOSE=1 make
   make install

6. If you used a custom install location (i.e. CMAKE_INSTALL_PREFIX) you must include this location in the environment variable CMAKE_PREFIX_PATH. This ensures that the required BFPSConfig.cmake file is accessible for future use by the package.

7. Using BFPS from an external project. BFPS creates and installs 3 files alongside the C++ headers and library:

   -- Installing: install/lib/BFPSConfig.cmake
   -- Installing: install/lib/BFPS_EXPORT.cmake
   -- Installing: install/lib/BFPS_EXPORT-noconfig.cmake

   In case these files provide incomplete information, it is necessary to update the cmake input config file: bfps/cmake/BFPSConfig.cmake.in.