Update diagnosticfiles authored by Rainer Weinberger's avatar Rainer Weinberger
# Diagnostic output *****************
Diagnostic output
*****************
Arepo will not only output the simulation snapshot and reduced data via Arepo will not only output the simulation snapshot and reduced data via
the halo-finder files, but also a number of (mostly ascii) diagnostic log- the halo-finder files, but also a number of (mostly ascii) diagnostic log-
files which contian important information about the code performance and files which contain important information about the code performance and
runtime behavior. runtime behavior.
In practice, to quickly check the performance of large In practice, to quickly check the performance of large
...@@ -14,9 +15,7 @@ simulated, the latter provides information about the computational ...@@ -14,9 +15,7 @@ simulated, the latter provides information about the computational
time spent in each part of the code, which can be influenced to some time spent in each part of the code, which can be influenced to some
degree by the values of the code parameters. degree by the values of the code parameters.
For ongoing simulations, these can be checked via For ongoing simulations, these can be checked via::
.. code-block :: python
tail -n 30 timebins.txt tail -n 30 timebins.txt
tail -n 200 cpu.txt tail -n 200 cpu.txt
...@@ -36,9 +35,7 @@ balance.txt ...@@ -36,9 +35,7 @@ balance.txt
Output of fractional cpu time used in each individual step, optimized to be Output of fractional cpu time used in each individual step, optimized to be
machine readable (while cpu.txt is more human readable). machine readable (while cpu.txt is more human readable).
Symbol key Symbol key::
.. code-block :: python
total = '-' / '-' total = '-' / '-'
treegrav = 'a' / ')' treegrav = 'a' / ')'
...@@ -87,9 +84,7 @@ Symbol key ...@@ -87,9 +84,7 @@ Symbol key
restart = '(' / 'b' restart = '(' / 'b'
misc = ')' / 'a' misc = ')' / 'a'
example: example::
.. code-block :: python
Step= 13147 sec= 0.212 Nsync-grv= 5181 Nsync-hyd= Step= 13147 sec= 0.212 Nsync-grv= 5181 Nsync-hyd=
342 dhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhttwwwwwxxzBBBBBBBBBBBBBBB 342 dhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhttwwwwwxxzBBBBBBBBBBBBBBB
...@@ -103,7 +98,7 @@ cpu.txt ...@@ -103,7 +98,7 @@ cpu.txt
======= =======
Each sync-point, such a block is written. This file Each sync-point, such a block is written. This file
reports the result of the different timers built into AREPO. Each reports the result of the different timers built into Arepo. Each
computationally expensive operation has a different timer attached to it and computationally expensive operation has a different timer attached to it and
this way allows to closely monitor what the computational time is spent on. this way allows to closely monitor what the computational time is spent on.
Some of the timers (e.g. treegrav) have sub-timers for individual operations. Some of the timers (e.g. treegrav) have sub-timers for individual operations.
...@@ -113,12 +108,10 @@ possible to identify inefficient parts of the overall algorithm and optimize ...@@ -113,12 +108,10 @@ possible to identify inefficient parts of the overall algorithm and optimize
only the most time-consuming parts of the code. There is the option only the most time-consuming parts of the code. There is the option
``OUTPUT_CPU_CSV`` which also returns this data as a ``cpu.csv`` file. ``OUTPUT_CPU_CSV`` which also returns this data as a ``cpu.csv`` file.
The different colums are: The different columns are:
name; wallclock-time (in s) this step; percentage this step; wallclock-time name; wallclock time (in s) this step; percentage this step; wallclock time
(in s) cumulative; percentage up to this step. A typical block of cpu.txt looks (in s) cumulative; percentage up to this step. A typical block of cpu.txt looks
the following (here a gravity-only, tree-only run): the following (here a gravity-only, tree-only run)::
.. code-block :: python
Step 131, Time: 0.197266, CPUs: 1, MultiDomains: 8, HighestActiveTim Step 131, Time: 0.197266, CPUs: 1, MultiDomains: 8, HighestActiveTim
eBin: 20 eBin: 20
...@@ -174,9 +167,7 @@ domain.txt ...@@ -174,9 +167,7 @@ domain.txt
The load-balancing (cpu work and memory) both in gravity and hydro calculation The load-balancing (cpu work and memory) both in gravity and hydro calculation
are reported for each timebin individually. Reported every sync-point. are reported for each timebin individually. Reported every sync-point.
Ideally balanced runs have the value 1, the higher the value, the more Ideally balanced runs have the value 1, the higher the value, the more
imbalanced the simulation. imbalanced the simulation.::
.. code-block :: python
DOMAIN BALANCE, Sync-Point 13314, Time: 0.997486 DOMAIN BALANCE, Sync-Point 13314, Time: 0.997486
Timebins: Gravity Hydro cumulative grav-balance Timebins: Gravity Hydro cumulative grav-balance
...@@ -202,9 +193,7 @@ In specified intervals (in simulation time, specified by the parameter ...@@ -202,9 +193,7 @@ In specified intervals (in simulation time, specified by the parameter
written into `energy.txt`. This file also contains the cumulative energy written into `energy.txt`. This file also contains the cumulative energy
that had to be injected into the system to ensure positivity in thermal energy. that had to be injected into the system to ensure positivity in thermal energy.
All output in code units. Note: this only works with up to 6 particle types. All output in code units. Note: this only works with up to 6 particle types.
The columns are The columns are::
.. code-block :: python
1. simulation time/ scalefactor 1. simulation time/ scalefactor
2. total thermal energy 2. total thermal energy
...@@ -237,9 +226,7 @@ The columns are ...@@ -237,9 +226,7 @@ The columns are
29. total injected energy due to positivity enforcement of thermal e 29. total injected energy due to positivity enforcement of thermal e
nergy nergy
Two example lines: Two example lines::
.. code-block :: python
0.96967 3.29069e+06 0 4.27406e+07 3.29069e+06 0 1.65766e+06 0 0 3.93 0.96967 3.29069e+06 0 4.27406e+07 3.29069e+06 0 1.65766e+06 0 0 3.93
02e+07 0 0 0 0 0 0 0 0 1.78097e+06 0 0 0 503.489 3047.89 0 0 65.5756 02e+07 0 0 0 0 0 0 0 0 1.78097e+06 0 0 0 503.489 3047.89 0 0 65.5756
...@@ -252,9 +239,7 @@ info.txt ...@@ -252,9 +239,7 @@ info.txt
======== ========
Every sync-point, the time-bins, time, timestep and number of active particles Every sync-point, the time-bins, time, timestep and number of active particles
are written into this file, e.g. are written into this file, e.g.::
.. code-block :: python
Sync-Point 13327, TimeBin=16, Time: 0.999408, Redshift: 0.000592464, Sync-Point 13327, TimeBin=16, Time: 0.999408, Redshift: 0.000592464,
Systemstep: 0.000147974, Dloga: 0.000148072, Nsync-grv: 17679, Systemstep: 0.000147974, Dloga: 0.000148072, Nsync-grv: 17679,
...@@ -263,16 +248,14 @@ are written into this file, e.g. ...@@ -263,16 +248,14 @@ are written into this file, e.g.
memory.txt memory.txt
========== ==========
AREPO internally uses an own memory manager. This means that one large chunk of Arepo internally uses an own memory manager. This means that one large chunk of
memory is reserved initially for AREPO (specified by the parameter memory is reserved initially for Arepo (specified by the parameter
`MaxMemSize`) and allocation for individual arrays is handeled internally. `MaxMemSize`) and allocation for individual arrays is handled internally.
The reason for introducing this was to avoid memory fragmentation during The reason for introducing this was to avoid memory fragmentation during
runtime on some machines, but also to have detailed information about how much runtime on some machines, but also to have detailed information about how much
memory AREPO actually needs and to terminate if this exceeds a pre-defined memory Arepo actually needs and to terminate if this exceeds a pre-defined
treshold. ``memory.txt`` reports this internal memory usage, and how much memory threshold. ``memory.txt`` reports this internal memory usage, and how much memory
is actually needed by the simulation. is actually needed by the simulation.::
.. code-block :: python
MEMORY: Largest Allocation = 816.742 Mbyte | Largest Allocation W MEMORY: Largest Allocation = 816.742 Mbyte | Largest Allocation W
ithout Generic = 132.938 Mbyte ithout Generic = 132.938 Mbyte
...@@ -460,7 +443,7 @@ is actually needed by the simulation. ...@@ -460,7 +443,7 @@ is actually needed by the simulation.
sfr.txt sfr.txt
======= =======
In case ``USE_SFR`` is active, AREPO will create a ``sfr.txt`` file, which reports In case ``USE_SFR`` is active, Arepo will create a ``sfr.txt`` file, which reports
the stars created in every call of the star-formation routine. the stars created in every call of the star-formation routine.
The individual columns are: The individual columns are:
...@@ -470,9 +453,7 @@ The individual columns are: ...@@ -470,9 +453,7 @@ The individual columns are:
* instantaneous star formation rate of all cells (Msun/yr), * instantaneous star formation rate of all cells (Msun/yr),
* instantaneous star formation rate of active cells (Msun/yr), * instantaneous star formation rate of active cells (Msun/yr),
* total mass in stars formed in this timestep (after sampling) (code units), * total mass in stars formed in this timestep (after sampling) (code units),
* cumulative stellar mass formed (code units). * cumulative stellar mass formed (code units).::
.. code-block :: python
4.373019e-01 9.714635e-03 1.100743e+02 1.405136e+02 2.2809 4.373019e-01 9.714635e-03 1.100743e+02 1.405136e+02 2.2809
41e-02 2.752464e+01 41e-02 2.752464e+01
...@@ -492,11 +473,9 @@ gravitational interactions on the largest possible timestep that is allowed by ...@@ -492,11 +473,9 @@ gravitational interactions on the largest possible timestep that is allowed by
the timestep criterion and allowed by the binary hierarchy of time steps. the timestep criterion and allowed by the binary hierarchy of time steps.
Each for each timestep, a linked list of particles on this particular Each for each timestep, a linked list of particles on this particular
integration step exists, and their statistics are reported in `timebins.txt`. integration step exists, and their statistics are reported in `timebins.txt`.
In this file, the number of gas cells and collisionless-particles in each In this file, the number of gas cells and collisionless particles in each
timebin (i.e. integration timestep) is reported for each sync-point, as well timebin (i.e. integration timestep) is reported for each sync-point, as well
as the cpu time and fraction spent on each timebin. A typical bock looks like as the cpu time and fraction spent on each timebin. A typical bock looks like::
.. code-block :: python
Sync-Point 2658, Time: 0.307419, Redshift: 2.25289, Systemstep: 9.10 Sync-Point 2658, Time: 0.307419, Redshift: 2.25289, Systemstep: 9.10
27e-05, Dloga: 0.000296144 27e-05, Dloga: 0.000296144
...@@ -518,9 +497,7 @@ timings.txt ...@@ -518,9 +497,7 @@ timings.txt
The performance of the gravitational tree algorithm is reported in The performance of the gravitational tree algorithm is reported in
`timings.txt` for each sync-point. An example of a single sync-point looks `timings.txt` for each sync-point. An example of a single sync-point looks
the following the following::
.. code-block :: python
Step(*): 372, t: 0.0455302, dt: 0.000215226, highest active timebin: Step(*): 372, t: 0.0455302, dt: 0.000215226, highest active timebin:
19 (lowest active: 19, highest occupied: 19) 19 (lowest active: 19, highest occupied: 19)
... ...
......