Changes
Page history
Update diagnosticfiles
authored
Aug 28, 2019
by
Rainer Weinberger
Show whitespace changes
Inline
Side-by-side
userguide/diagnosticfiles.md
View page @
6547ec3e
# Diagnostic output
*****************
Diagnostic output
*****************
Arepo will not only output the simulation snapshot and reduced data via
Arepo will not only output the simulation snapshot and reduced data via
the halo-finder files, but also a number of (mostly ascii) diagnostic log-
the halo-finder files, but also a number of (mostly ascii) diagnostic log-
files which cont
i
an important information about the code performance and
files which conta
i
n important information about the code performance and
runtime behavior.
runtime behavior.
In practice, to quickly check the performance of large
In practice, to quickly check the performance of large
...
@@ -14,9 +15,7 @@ simulated, the latter provides information about the computational
...
@@ -14,9 +15,7 @@ simulated, the latter provides information about the computational
time spent in each part of the code, which can be influenced to some
time spent in each part of the code, which can be influenced to some
degree by the values of the code parameters.
degree by the values of the code parameters.
For ongoing simulations, these can be checked via
For ongoing simulations, these can be checked via::
.. code-block :: python
tail -n 30 timebins.txt
tail -n 30 timebins.txt
tail -n 200 cpu.txt
tail -n 200 cpu.txt
...
@@ -36,9 +35,7 @@ balance.txt
...
@@ -36,9 +35,7 @@ balance.txt
Output of fractional cpu time used in each individual step, optimized to be
Output of fractional cpu time used in each individual step, optimized to be
machine readable (while cpu.txt is more human readable).
machine readable (while cpu.txt is more human readable).
Symbol key
Symbol key::
.. code-block :: python
total = '-' / '-'
total = '-' / '-'
treegrav = 'a' / ')'
treegrav = 'a' / ')'
...
@@ -87,9 +84,7 @@ Symbol key
...
@@ -87,9 +84,7 @@ Symbol key
restart = '(' / 'b'
restart = '(' / 'b'
misc = ')' / 'a'
misc = ')' / 'a'
example:
example::
.. code-block :: python
Step= 13147 sec= 0.212 Nsync-grv= 5181 Nsync-hyd=
Step= 13147 sec= 0.212 Nsync-grv= 5181 Nsync-hyd=
342 dhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhttwwwwwxxzBBBBBBBBBBBBBBB
342 dhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhttwwwwwxxzBBBBBBBBBBBBBBB
...
@@ -103,7 +98,7 @@ cpu.txt
...
@@ -103,7 +98,7 @@ cpu.txt
=======
=======
Each sync-point, such a block is written. This file
Each sync-point, such a block is written. This file
reports the result of the different timers built into A
REPO
. Each
reports the result of the different timers built into A
repo
. Each
computationally expensive operation has a different timer attached to it and
computationally expensive operation has a different timer attached to it and
this way allows to closely monitor what the computational time is spent on.
this way allows to closely monitor what the computational time is spent on.
Some of the timers (e.g. treegrav) have sub-timers for individual operations.
Some of the timers (e.g. treegrav) have sub-timers for individual operations.
...
@@ -113,12 +108,10 @@ possible to identify inefficient parts of the overall algorithm and optimize
...
@@ -113,12 +108,10 @@ possible to identify inefficient parts of the overall algorithm and optimize
only the most time-consuming parts of the code. There is the option
only the most time-consuming parts of the code. There is the option
``OUTPUT_CPU_CSV``
which also returns this data as a
``cpu.csv``
file.
``OUTPUT_CPU_CSV``
which also returns this data as a
``cpu.csv``
file.
The different colums are:
The different colum
n
s are:
name; wallclock
-
time (in s) this step; percentage this step; wallclock
-
time
name; wallclock
time (in s) this step; percentage this step; wallclock
time
(in s) cumulative; percentage up to this step. A typical block of cpu.txt looks
(in s) cumulative; percentage up to this step. A typical block of cpu.txt looks
the following (here a gravity-only, tree-only run):
the following (here a gravity-only, tree-only run)::
.. code-block :: python
Step 131, Time: 0.197266, CPUs: 1, MultiDomains: 8, HighestActiveTim
Step 131, Time: 0.197266, CPUs: 1, MultiDomains: 8, HighestActiveTim
eBin: 20
eBin: 20
...
@@ -174,9 +167,7 @@ domain.txt
...
@@ -174,9 +167,7 @@ domain.txt
The load-balancing (cpu work and memory) both in gravity and hydro calculation
The load-balancing (cpu work and memory) both in gravity and hydro calculation
are reported for each timebin individually. Reported every sync-point.
are reported for each timebin individually. Reported every sync-point.
Ideally balanced runs have the value 1, the higher the value, the more
Ideally balanced runs have the value 1, the higher the value, the more
imbalanced the simulation.
imbalanced the simulation.::
.. code-block :: python
DOMAIN BALANCE, Sync-Point 13314, Time: 0.997486
DOMAIN BALANCE, Sync-Point 13314, Time: 0.997486
Timebins: Gravity Hydro cumulative grav-balance
Timebins: Gravity Hydro cumulative grav-balance
...
@@ -202,9 +193,7 @@ In specified intervals (in simulation time, specified by the parameter
...
@@ -202,9 +193,7 @@ In specified intervals (in simulation time, specified by the parameter
written into
`energy.txt`
. This file also contains the cumulative energy
written into
`energy.txt`
. This file also contains the cumulative energy
that had to be injected into the system to ensure positivity in thermal energy.
that had to be injected into the system to ensure positivity in thermal energy.
All output in code units. Note: this only works with up to 6 particle types.
All output in code units. Note: this only works with up to 6 particle types.
The columns are
The columns are::
.. code-block :: python
1. simulation time/ scalefactor
1. simulation time/ scalefactor
2. total thermal energy
2. total thermal energy
...
@@ -237,9 +226,7 @@ The columns are
...
@@ -237,9 +226,7 @@ The columns are
29. total injected energy due to positivity enforcement of thermal e
29. total injected energy due to positivity enforcement of thermal e
nergy
nergy
Two example lines:
Two example lines::
.. code-block :: python
0.96967 3.29069e+06 0 4.27406e+07 3.29069e+06 0 1.65766e+06 0 0 3.93
0.96967 3.29069e+06 0 4.27406e+07 3.29069e+06 0 1.65766e+06 0 0 3.93
02e+07 0 0 0 0 0 0 0 0 1.78097e+06 0 0 0 503.489 3047.89 0 0 65.5756
02e+07 0 0 0 0 0 0 0 0 1.78097e+06 0 0 0 503.489 3047.89 0 0 65.5756
...
@@ -252,9 +239,7 @@ info.txt
...
@@ -252,9 +239,7 @@ info.txt
========
========
Every sync-point, the time-bins, time, timestep and number of active particles
Every sync-point, the time-bins, time, timestep and number of active particles
are written into this file, e.g.
are written into this file, e.g.::
.. code-block :: python
Sync-Point 13327, TimeBin=16, Time: 0.999408, Redshift: 0.000592464,
Sync-Point 13327, TimeBin=16, Time: 0.999408, Redshift: 0.000592464,
Systemstep: 0.000147974, Dloga: 0.000148072, Nsync-grv: 17679,
Systemstep: 0.000147974, Dloga: 0.000148072, Nsync-grv: 17679,
...
@@ -263,16 +248,14 @@ are written into this file, e.g.
...
@@ -263,16 +248,14 @@ are written into this file, e.g.
memory.txt
memory.txt
==========
==========
A
REPO
internally uses an own memory manager. This means that one large chunk of
A
repo
internally uses an own memory manager. This means that one large chunk of
memory is reserved initially for A
REPO
(specified by the parameter
memory is reserved initially for A
repo
(specified by the parameter
`MaxMemSize`
) and allocation for individual arrays is hand
e
led internally.
`MaxMemSize`
) and allocation for individual arrays is handled internally.
The reason for introducing this was to avoid memory fragmentation during
The reason for introducing this was to avoid memory fragmentation during
runtime on some machines, but also to have detailed information about how much
runtime on some machines, but also to have detailed information about how much
memory AREPO actually needs and to terminate if this exceeds a pre-defined
memory Arepo actually needs and to terminate if this exceeds a pre-defined
treshold.
``memory.txt``
reports this internal memory usage, and how much memory
threshold.
``memory.txt``
reports this internal memory usage, and how much memory
is actually needed by the simulation.
is actually needed by the simulation.::
.. code-block :: python
MEMORY: Largest Allocation = 816.742 Mbyte | Largest Allocation W
MEMORY: Largest Allocation = 816.742 Mbyte | Largest Allocation W
ithout Generic = 132.938 Mbyte
ithout Generic = 132.938 Mbyte
...
@@ -460,7 +443,7 @@ is actually needed by the simulation.
...
@@ -460,7 +443,7 @@ is actually needed by the simulation.
sfr.txt
sfr.txt
=======
=======
In case
``USE_SFR``
is active, A
REPO
will create a
``sfr.txt``
file, which reports
In case
``USE_SFR``
is active, A
repo
will create a
``sfr.txt``
file, which reports
the stars created in every call of the star-formation routine.
the stars created in every call of the star-formation routine.
The individual columns are:
The individual columns are:
...
@@ -470,9 +453,7 @@ The individual columns are:
...
@@ -470,9 +453,7 @@ The individual columns are:
*
instantaneous star formation rate of all cells (Msun/yr),
*
instantaneous star formation rate of all cells (Msun/yr),
*
instantaneous star formation rate of active cells (Msun/yr),
*
instantaneous star formation rate of active cells (Msun/yr),
*
total mass in stars formed in this timestep (after sampling) (code units),
*
total mass in stars formed in this timestep (after sampling) (code units),
*
cumulative stellar mass formed (code units).
*
cumulative stellar mass formed (code units).::
.. code-block :: python
4.373019e-01 9.714635e-03 1.100743e+02 1.405136e+02 2.2809
4.373019e-01 9.714635e-03 1.100743e+02 1.405136e+02 2.2809
41e-02 2.752464e+01
41e-02 2.752464e+01
...
@@ -492,11 +473,9 @@ gravitational interactions on the largest possible timestep that is allowed by
...
@@ -492,11 +473,9 @@ gravitational interactions on the largest possible timestep that is allowed by
the timestep criterion and allowed by the binary hierarchy of time steps.
the timestep criterion and allowed by the binary hierarchy of time steps.
Each for each timestep, a linked list of particles on this particular
Each for each timestep, a linked list of particles on this particular
integration step exists, and their statistics are reported in
`timebins.txt`
.
integration step exists, and their statistics are reported in
`timebins.txt`
.
In this file, the number of gas cells and collisionless
-
particles in each
In this file, the number of gas cells and collisionless
particles in each
timebin (i.e. integration timestep) is reported for each sync-point, as well
timebin (i.e. integration timestep) is reported for each sync-point, as well
as the cpu time and fraction spent on each timebin. A typical bock looks like
as the cpu time and fraction spent on each timebin. A typical bock looks like::
.. code-block :: python
Sync-Point 2658, Time: 0.307419, Redshift: 2.25289, Systemstep: 9.10
Sync-Point 2658, Time: 0.307419, Redshift: 2.25289, Systemstep: 9.10
27e-05, Dloga: 0.000296144
27e-05, Dloga: 0.000296144
...
@@ -518,9 +497,7 @@ timings.txt
...
@@ -518,9 +497,7 @@ timings.txt
The performance of the gravitational tree algorithm is reported in
The performance of the gravitational tree algorithm is reported in
`timings.txt`
for each sync-point. An example of a single sync-point looks
`timings.txt`
for each sync-point. An example of a single sync-point looks
the following
the following::
.. code-block :: python
Step(*): 372, t: 0.0455302, dt: 0.000215226, highest active timebin:
Step(*): 372, t: 0.0455302, dt: 0.000215226, highest active timebin:
19 (lowest active: 19, highest occupied: 19)
19 (lowest active: 19, highest occupied: 19)
...
...
...
...