Big Fluid and Particle Simulator

In brief, this code runs pseudospectral direct numerical simulations (DNS) of the incompressible Navier-Stokes equations, using FFTW 3, and it can integrate particle trajectories in the resulting fields.

The Navier-Stokes solver has been extensively tested (tests are included in the repository), and it is working as expected. Parameters and statistics are stored in HDF5 format, together with code information, so simulation data should be "future proof" --- suggestions of possible improvements to the current approach are always welcome.

The primary aim of bfps is to reduce the time spent on setting up and baby sitting DNS, as well as simplify the analysis of the generated data. The wish is that this Python package provides an easy and general way of constructing efficient specialized DNS C++ codes for different turbulence problems encountered in research. At the same time, the package should provide a unified way of postprocessing, and accessing the postprocessing results. The code therefore consists of two main parts: the pure C++ code, a set of loosely related "building blocks", and the Python code, which can generate C++ code using the pure classes, but with a significant degree of flexibility.

The code user is expected to write a small python script that will properly define the DNS they are interested in running. That code will generate an executable that can then be run directly on the user's machine, or submitted to a queue on a cluster.

Installation

So far, the code has been run on laptops, desktops, and a couple of clusters (biggest run so far was 1536^3 on 16 nodes of 32 cores each, with about 11 seconds per time step, for a simple incompressible Navier-Stokes problem). Postprocessing data may not be very computationally intensive, depending on the amount of data involved.

Postprocessing only

Use a console; navigate to the bfps folder, and type:

python setup.py install

(add --user or sudo as appropriate). setup.py should tell you about the various packages you need.

Full installation

If you want to run simulations on the machine where you're installing, you will need to call compile_library before installing. Your machine needs to have an MPI compiler installed, the HDF5 C library and FFTW >= 3.4 --- a detailed prerequisite installation list is included at the end of this document. The file machine_settings_py.py should be modified appropriately for your machine (otherwise the compile_library command will most likely fail). This file will be copied the first time you run setup.py into $HOME/.config/bfps/machine_settings.py, where it will be imported from afterwards --- any future edits must be made to the new file. You may, obviously, edit it afterwards and rerun the compile_library command as needed.

python setup.py compile_library
python setup.py install

For machine_settings.py, please keep in mind to turn on optimizations for production environments. In particular, for clusters of unknown architecture it helps to log into individual nodes and run the following command:

gcc -march=native -Q --help=target

The results can be used to then compile on the frontend node(s) without using -march=native (since the frontend node may have different architecture).

Documentation

While the code is not fully documented yet, basic information is already available, and it is recommended that you generate the manual and go through it carefully. Please don't be shy about asking for specific improvements to the current text. In order to generate the manual, navigate to the repository folder, and execute the following commands:

cd documentation
make latexpdf

Optionally, html documentation can be generated instead if needed, just type make html instead of make latexpdf.

Comments

• particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail.
• Code is only known to work with HDF5 1.8.x.
• Code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x

Installation with prerequisites

These installation steps assume that you have a working MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it). If this is not the case, please consult the FFTW and HDF5 compilation instructions for detailed instructions (./configure --help should be enough).

1. Make directory PREFIX on local fast partition.
2. Download, compile, install FFTW (latest version 3.x from http://www.fftw.org/). Execute the following commands in order, feel free to customize optimisation flags for your own computer (see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html):

./configure --prefix=PREFIX --enable-float --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-avx-128-fma --enable-avx512 --enable-mpi --enable-openmp --enable-threads
make
make install
./configure --prefix=PREFIX  --enable-sse --enable-sse2 --enable-avx --enable-avx2 --enable-avx-128-fma --enable-avx512 --enable-mpi --enable-openmp --enable-threads
make
make install

BFPS will try to find HDF5 using the FindFFTW from the Morse project.
If the package is installed in a non standard location, it is recommanded
to setup the environment variables: FFTW_DIR (or FFTW_INCDIR and FFTW_LIBDIR)

3. Download, compile, install HDF5 (version 1.8.x, currently available at https://portal.hdfgroup.org/display/support/HDF5+1.8.20#files). We are using parallel I/O, therefore we use the plain C interface of HDF5:

./configure --prefix=PREFIX --enable-parallel
make
make install

BFPS will try to find HDF5 using the regular FindHDF5.
Therefore, if the package is installed in a non standard location, it is recommanded
to setup the environment variable: HDF5_ROOT

3. This step may be ommited. I recommend the creation of a virtual python3 environment (also under PREFIX) that will be used for installing bfps and dependencies. Please see https://docs.python-guide.org/dev/virtualenvs/.
4. Clone bfps repository.

git clone git@gitlab.mpcdf.mpg.de:clalescu/bfps.git

5. Go into bfps repository, execute

mkdir build
cd build
cmake ..
# possibly : cmake .. -DCMAKE_INSTALL_PREFIX=INSTALL_DIR
make ..
# to get a verbose compilation process, use VERBOSE=1 make
# make install

6. Using BFPS from an external project. Along with all the BFPS files (lib and headers) can be found different cmake files that contains all the information related to the compilation of BFPS. It is possible to load this file in another CMake based application to know the dependencies and paths.

   For instance, the installation will create these files: