Commit 8afad291 authored by Rainer Weinberger's avatar Rainer Weinberger
Browse files

User guide

parent 0f84a567
# Minimal makefile for Sphinx documentation
# You can set these variables from the command line.
SPHINXBUILD = sphinx-build
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
\ No newline at end of file
\ No newline at end of file
# -*- coding: utf-8 -*-
# Configuration file for the Sphinx documentation builder.
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'Arepo'
copyright = '2019, Max-Planck Institute for Astrophysics'
author = 'Arepo developers'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = '1.0'
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.md'
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'Arepodoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
# Latex figure (float) alignment
# 'figure_align': 'htbp',
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'Arepo.tex', 'Arepo Documentation',
'Arepo developers', 'manual'),
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'arepo', 'Arepo Documentation',
[author], 1)
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'Arepo', 'Arepo Documentation',
author, 'Arepo', 'One line description of project.',
\ No newline at end of file
This diff is collapsed.
Code development
We strongly encourage further development of the code by other people. The idea
with this public version is to provide a well tested stable version of AREPO to
the community as a basis of individual model development. Changes to the code can
then be made in two ways: bugfixes and code extensions. The former is organized in
the issue-tracking system of the repository, while for the
second one, the developers should be contacted.
Issue reporrting
Problems with the code will in generally be reported to the issue tracker of the repository.
Therefore, if a problem occurs, the first thing to check is whether there already exists
an open issue, i.e. whether this is a known problem. If not, there are two ways to create
a new issue.
* The issue tracking system of the gitlab repository requires log-in to ````.
In general, AREPO users will not have this access, however, active developers may request
a guest account there, and can then create own issues. These issues need to be specific
and reproducible, or directly point to the problematic part of the code.
* AREPO users without such an account should contact the authors in case of problems with the
code. Also in this case examples that reproduce the issue greatly help and accellerate
the problem-solving process.
Code extensions
We welcome code extensions by other people, however with certain requirements
for the implemented modules.
* The modules are under the same terms of use as the AREPO code, i.e. a GPLv3 license.
All modules in this version of AREPO are free to use for everyone.
* The number of interfaces to the main code should be as few as possible.
* The module should be of general interest. This implies in particular that
the implementation needs to work in a fully scalable fashion in massively-parallel
runs, with a minimum of shared/dublicated data.
* The module should come with some additional examples for verification of its
Developers interested to share their module with the community as a part of
AREPO should contact the authors.
Major code updates
Attached a list of important bugfixes and additions with the date and id of the
| **Description** | **date (** | **commit** |
| Public version complete | 20.03.2019 | |
This diff is collapsed.
.. Arepo documentation master file
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Arepo documentation
AREPO is a massively parallel code for gravitational n-body systems and
magnetohydrodynamics, both on Newtonian as well as cosmological background.
It is a flexible code that can be applied to a variety of different types of
simulations, offering a number of sophisticated simulation algorithms. A
description of the numerical algorithms employed by the code is given in the
original code paper and subsequent publications. This documentation addresses
the question how to use the different numerical algorithms.
AREPO was written by Volker Springel ( with further
development by Rüdiger Pakmor ( and contributions by
many other authors. The public version of the code was compiled by Rainer
Weinberger (
The AREPO code was initially developed to combine the advantages of
finite-volume hydrodynamics schemes with the Lagrangian invariance of
Smoothed Particle Hydrodynamics schemes. To this end, Arepo makes use of an
unstructured Voronoi-mesh which is, in its standard setting, moving with
the fluid in an quasi-Lagrangian fashion. The fluxes between cells are computed
using a finite-volume approach, and further spatial adaptivity is
provided by the possibility to add and remove cells from the
mesh according to defined criteria. In addition to gas, AREPO allows for a
number of additional particle types which interact only gravitationally, as
well as for forces from external gravitational potentials.
AREPO is optimized for, but not limited to, cosmological simulations of galaxy
formation and consequently for simulations with very high dynamic ranges in
space and time. Therefore, AREPO employes an adaptive timestepping for each
individual cell and particle as well as a dynamic, on-the fly load and memory
balancing scheme. In its current version, AREPO is fully MPI parallel, and tested
to run with >10,000 MPI tasks. The exact performance is, however, highly problem and
machine dependent.
It is important to note that the performance and accuracy of the code is a
sensitive function of some of the code parameters. We also stress that AREPO
comes without any warranty, and without any guarantee that it produces correct
results. If in doubt about something, reading (and potentially improving) the
source code is always the best strategy to understand what is going on!
**Please also note the following:**
The numerical parameter values used in the examples contained in the code
distribution do not represent a specific recommendation by the authors! In
particular, we do not endorse these parameter settings in any way as standard
values, nor do we claim that they will provide fully converged results for the
example problems, or for other initial conditions. We think that it is extremely
difficult to make general statements about what accuracy is sufficient for
certain scientific goals, especially when one desires to achieve it with the
smallest possible computational effort. For this reason we refrain from making
such recommendations. We encourage every simulator to find out for
herself/himself what integration settings are needed to achieve sufficient
accuracy for the system under study. We strongly recommend to make convergence
and resolution studies to establish the range of validity and the uncertainty of
any numerical result obtained with AREPO.
Table of contents
.. toctree::
Related codes
The general code structure of AREPO is similar to the GADGET code, but with
significant further developments and improvements. However, keeping in mind
that the code architecture originates from an Tree-PM-SPH code helps to
understand why certain things are done in certain ways.
One of the obvious signs of this are the name of some global structures and
functions. For example the array holding the information for gas cells only is
called SphP. Another example would be the routines in ``src/init/density.c``,
which used to be an SPH density calculation, but now only used for an estimate
for the typical distance to neighbouring cells. Another aspect is the
presence of the neighbour tree, a search-tree structure on gas cells. In
practice this means that many of the routines that depend on the state of
neighbouring cells, including sub-grid feedback models etc., can be based on
this neighbour search. This makes it possible to port these models from earlier
versions of GADGET to AREPO.
However, it is important to realize that routines written for the GADGET code
will in general not work directly when ported to Arepo. We encourage developers
to carefully test that the implementation is still having the same effect.
Apart from the differences in implementation, it is also important to realize
that a grid-based finite volume scheme might react different to a particular to
a sub-grid implementation.
Differences to the development Version
This public version of AREPO was branched off from the development version in
November 2017, substantially cut down, cleand up and the documentation completed
by Rainer Weinberger. The main reason for doing this is to provide a code to the
community that researchers are able to understand and use without the need of
an extensive introduction by one of the authors. We belive that the public
version, being better documented on a developer level, also provides a better
starting point for new developments.
The general idea for this public verson was to preserve the well-tested code
with a limited number of changes. However, some compile-time options have been
eliminated and to recover the same running mode **in the development version
of AREPO**, the following compile-time flags need to be set in ``
* ``PERIODIC`` (has become obsolete over the years)
* ``VORONOI`` (the development version also has an AMR mode)
* ``VORONOI_DYNAMIC_UPDATE`` (in practice now a default)
* ``AUTO_SWAP_ENDIAN_READIC`` (only relevant for file formats 1 and 2)
* the switch ``EXACT_GRAVITY_FOR_PARTICLE_TYPE`` automatically activates ``NO_SELFGRAVITY_TYPE``, ``NO_GRAVITY_TYPE`` and ``EXACT_GRAVITY_REACTION`` (see src/main/allvars.h) in the public version. This is not the case in the development version where all these flags need to be set individually in
This diff is collapsed.
Getting started
Library requirements
AREPO requires the following libraries for compilation:
Allways needed:
1. **mpi** The `Message Passing Interface`, version 2.0 or higher.
In practice AREPO is well-tested on the open source libraries
MPICH <> and OpenMPI <>.
For some machines, vendor supplied versions may exist and can also be used.
AREPO relies exclusively on MPI for its parallelization.
2. **gsl** `The GNU Scientific Library`. This open-source library is available
under <>.
Needed in some cases:
3. **fftw3** `Fastest Fourier Transform in the West` (<>)
This free software package for Fast-Fourier-Transformations is used in the
particle-mesh algorithm. Consequently, it is only needed if ``PMGRID`` is
set in ````.
4. **hdf5** `Hierarchical Data Format` <>
is a data format specification that is used for the input/output format 3
in Arepo, which is the de-facto standard I/O format (even though the old
GADGET formats are supported). The library is needed if the ``HAVE_HDF5``
flag is set in ``.
5. **hwloc** : The `hardware locality library` is useful for allowing
the code to probe the processor topology it is running on and enable a
pinning of MPI threads to individual cores. This library is optional and
only required if the ``IMPOSE_PINNING`` flag is set.
AREPO needs a working c compiler supporting at least the ``c11`` standard, as
well as GNU-Make and Python, which are used in the compilation.
Building the code
Systype variable
The first thing after obtaining a local copy of the code is to set a ``SYSTYPE``
variable, either by typing::
export SYSYTPE="MySystype"
or by copying the ``Template-Makefile.systype`` file::
cp Template-Makefile.systype Makefile.systype
and uncommenting the intended ``SYSTYPE`` variable in ``Makefile.systype``.
The ``SYSTYPE`` is a variable defining a specific machine. Each machine has its
specific settings and library paths defined in the Makefile. New ``SYSTYPE`` can
be added as needed. -- Compile time flags
The next step is to set the compile-time configuration of the code. This is
done best by copying the ```` file::
Note that the compile-time configuration file does not need to be ````,
however, if it is not, this needs to be specified when calling the Makefile.
An example if this are the code examples.
The ```` file contains the possible running modes of AREPO, and the
flags not outcommented with a '#' will be processed and included by a
``#define MyModule`` in a header file ``arepoconfig.h`` that is included in the
code build directory.
Compiling the code
AREPO can be compiled with::
which will first execute a check on compile time flags (see below) followed
by the actual build. It is also possible to skip the check by typing::
make build
The compilation can be accelerated by compiling files in parallel with the
option ``-j 8``, which will use 8 tasks for the compilation.
Compile-time checks of flags
The normal ``make`` command will first execute a check using the python script
````. This will check that all flags used in the code via
``#if defined(MyFlag)`` or ``#ifdef MyFlag`` occur either in ````
or ``defines_extra``. The main reason for doing this is to prevent typos in the
code such as ``#if defined(MyFlg)``, which are generally very hard to find
because the code might compile normally, but the code followed by
``#if defined(MyFlg)`` just does not appear in the executable.
New flags that can be actvated should be included to ````,
while internally defined flags are listed in ``defines_extra``.
Starting the code
param.txt -- Runtime settings
After compiling AREPO, the executable ``Arepo`` should be present in the main
directory. To start a simulation, AREPO needs be handed over the path to
a file containing the runtime parameters, usually called ``param.txt``.
The settings that need to be set generally depend on the compile-time flags
used. All possible parameters are listed under :ref:`Parameterfile`, including the
compile-time flags they belong to. Starting AREPO with a parameter file that has
too few or obsolete parameter settings will result in an output telling which
parameters are required or still needed, respectively.
Starting the simulation
A simulation is started with::
mpirun -np 32 Arepo param.txt
which will start Arepo on 32 MPI ranks using the ``param.txt`` file for the
run-time paramter input. Note that one some systems, mpirun has a different
syntax. We refer to the user guide of individual machines for such
Interrupting a run
AREPO supports running a simulations in chunks and resuming an already started
simulation via reading in so-called restart files, which are memory dumps
written to the output directory. The code is written in such a way that the
result is completely independent of the number of restarts performed, which is
a cruicial aspect for debugging the code.
A (regular) code termination always results in the writing of such restart
files, and is triggered whenever 85% of the specified maximum runtime has been
reached or the final simulation time has been reached.
It is important, in particular in larger simulations, that the code is able to
complete the output of the restart file before, e.g. a job time-limit is
reached. Therefore the specified maximum runtime should never exceed the
runtime limits of the machine.
It is also possible to trigger a (regular) code termination by introducing a
file called ``stop`` in the output directory.::
touch stop
Note that AREPO usually produces restart-files in regular user specified time
intervals to ensure runs can be resumed without a major loss of computation
time even if a run was terminated by the operating system.
Restarting a run
A simulaion can be restarted in two ways in AREPO, which is indicated with the
additional flags 1 or 2 in the execution call. The flag 1 is the restarting
from a restart file and generally recommended if possible. This, however,
has the restriction that a run needs to be resumed with **exactly as many MPI
ranks** as it was originally started from. The restart-command then looks
the following:
mpirun -np 32 ./Arepo param.txt 1
The flag 2 denotes a restart from a regular snapshot output. This is possible,
however the simulation generally will not be binary identical to a run without
this restart. In particular in runs where the fields (especially positions) are
written in single precision, this can have effects on the overall quality of
the simulation. A command to restart AREPO from a snapshot looks the following::
mpirun -np 32 ./Arepo param.txt 2 7
in which case AREPO would read in the snapshot 7 and resume the simulation from
there. This second method should only be used if the regular start from
restart files is not possible for some reason.
Starting postprocessing
There are a limited number of posprocessing options in the public version of
Arepo. The main ones are
* to run the halo and/or subhalo finder in postprocessing (flag 3)
* to convert a snapshot to another format (flag 6),
* to write the Voronoi-mesh data (flag 14)
* to calculate and output the gradients of hydrodynamical quantities (flag 17)
* to calculate and output the gravitational potential (flag 18)
Note that the non-contiguity of the numbers originates from the consistency
with the developer version, which has more options. Other flags are not
supported in the current implementation of Arepo.
Simulation examples
AREPO is a multi-purpose code that supports a number of different types of
simulations. Here, we present some examples that in our opinion are
particularly relevant. This list is by no means complete, and highly biased
to the fields of interest of the authors.
We encourage users that have other simulation setups to make them (or small
test examples) availlable to us, in order to provide a more exhaustive list of
examples in future code versions.
General Execution Structure of Examples
Examples can be run using the ```` shell script (typing ``bash``).
The script itself, as the name suggests, is there it run a couple
of verification simulations and report whether the simulation
results agree with specifications. In particular, if this is the case,
the script will delete the directory in which the example run in, which
might not be desired when getting started with the code.
The general procedure to execute an example is (replace `myexample` by the desired one):
* create a run-directory, in which the run will be executed::
mkdir -p ./run/examples/myexample
* copy the entire content of the example directory of interest to the run directory::
cp -r ./examples/myexample/* ./run/examples/myexample/
* execute the ```` file, with the run directory as first argument::
python ./run/examples/myexample/ \
This creates initial conditions and possible other files required by the simulation.
* compile AREPO, using the provided file::
make CONFIG=./run/examples/myexample/ \
BUILD_DIR=./run/examples/myexample/build \
* change to run directory::
cd ./run/examples/myexample/
* execute Arepo (here on 8 mpi ranks)::
mpiexec -np 8 ./Arepo param.txt
* change back to root directory::
cd ../../../
* execute the ```` script to do analysis on the simulation output (again specifying the relative path to the run directory)::