adds text on how to launch jobs

documentation/index.rst

@@ -15,6 +15,7 @@ Contents:
     sphinx_static/overview
     sphinx_static/convergence
     sphinx_static/bandpass
+    sphinx_static/launch
     sphinx_static/development
     chapters/api
     chapters/cpp_doxygen

documentation/sphinx_static/launch.rst

+Tutorial: job launching methods
+===============================
+
+**Note**: This document assumes you are familiar with TurTLE
+installation, and you've tested it on your machine.
+
+Context
+-------
+
+A TurTLE direct numerical simulation is identified by a simulation name, and
+a work directory.
+This information directly corresponds to the HDF5 file `simname.h5` located
+in the working directory and containing DNS parameters and basic diagnostics.
+
+A TurTLE DNS **job** consists of actually executing the C++ executable for
+`niter_todo` iterations.
+Each successful job will typically update the `iteration` value
+stored in the DNS file, (as well as the checkpoint and diagnostics).
+
+A TurTLE **job** in general may be a DNS job, or it may be a
+post-processing job, or it may be a test job.
+In all three cases, **job** refers to the actual execution of the
+binary code (generated from C++ code).
+
+TurTLE C++ executables typically require a `simname` command line
+argument, that they use to find the relevant HDF5 file (different for
+DNS and postprocessing jobs).
+Depending on parameters and type of DNS, if `iteration` is 0 a DNS job may
+also require a checkpoint file from which to read its initial condition;
+otherwise the executable will generate its own initial condition and write
+the 0 checkpoint as well.
+If `iteration` is larger than 0 (it should be a multiple of
+`niter_todo`, by the way), then any DNS job will require a corresponding
+checkpoint file from which to read the DNS state.
+
+User friendly launch
+--------------------
+
+The default Python wrapper can do quite a lot.
+I.e. you may execute from the command line
+
+.. code:: bash
+
+     turtle DNS NSVE -n 64 --simname test64 --njobs 4
+
+This will write the DNS HDF5 file, and it will generate 4 separate
+DNS jobs (on an interactive computer, that means 4 subsequent
+executions of the C++ executable with the same simulation name).
+
+If you decide that you would like to run this same DNS for longer,
+all you need to do is navigate to the same folder and execute
+
+.. code:: bash
+
+     turtle DNS NSVE --simname test64 --njobs 8
+
+**Note** the "cubic box size" no longer needs to be specified,
+because we are continuing an existing DNS, and the Python wrapper
+will never overwrite parameters already present in the DNS file.
+To clarify, the arguments present in these commands are arguments to
+the Python wrapper `turtle`, which the Python wrapper interprets.
+The argument to the C++ executable is then the simulation name, and
+all other DNS parameters are passed through the HDF5 file (which is
+silently generated by the Python wrapper if it does not exist).
+In general all "physical" parameters follow this rule.
+
+**Note** the "HPC job parameters" may be different between different
+job launches with the same DNS.
+I.e. we may first execute
+
+.. code:: bash
+
+     turtle DNS NSVE -n 64 --simname test64 --njobs 2 --np 2 --ntpp 2
+
+If we decide that this is too slow, when we continue the DNS we can request
+a different number of MPI processes (`np` Python wrapper argument) or a
+different number of OpenMP threads per MPI process (`ntpp` Python wrapper
+argument):
+
+.. code:: bash
+
+     turtle DNS NSVE --simname test64 --njobs 2 --np 4 --ntpp 2
+
+Technically the python wrapper can be used from the shell as any
+other shell command: more advanced users may generate custom scripts
+that call the wrapper from loops for different simulation names for
+example, changing other job/DNS parameters as required.
+
+**Main benefit**: on HPC clusters, the Python wrapper will generate
+appropriate SLURM scripts, and issue the requires `sbatch` commands.
+When multiple jobs are requested, it will also create the job
+dependencies appropriately.
+
+Python hack of bash tools
+-------------------------
+
+Rather than executing the `turtle` command from bash scripts, one may
+create TurTLE DNS objects within Python scripts and use the same
+mechanism directly from Python, writing out bash arguments as Python
+lists of strings.
+For example:
+
+.. code:: python
+
+     from TurTLE import DNS
+     c = DNS()
+     c.launch([
+         'NSVE',
+         '-n', '64',
+         '--simname', 'test64',
+         '--njobs', '4'])
+
+The above Python code can also be turned into an elaborate mechanism
+to sample different values of some parameter, or to manage some other
+"multiple DNS" scenario.
+
+Depending on the specific DNS tool being used (i.e. something other
+than `NSVE`), users may prefer Python scripts to bash scripts because
+Python can also generate custom initial conditions before the call to
+the `launch` method is made.
+
+Python solution
+---------------
+
+For some use-cases it is simpler to duplicate (then modify) the
+`launch` method discussed above.
+I.e. one may use the code below:
+
+.. code:: python
+
+     from TurTLE import DNS
+     c = DNS()
+     c.simname = 'test64'
+     c.dns_type = 'NSVE'
+     c.name = 'cpp_executable_name'
+     c.parameters['nx'] = 64
+     c.parameters['ny'] = 64
+     c.parameters['nz'] = 64
+     c.parameters['forcing_type'] = 'fixed_energy_injection_rate'
+     c.parameters['injection_rate'] = 0.4
+     # also set viscosity etc
+     c.write_par(iter0 = 0)
+     # generate initial conditions
+     c.run(
+         nb_processes = 2,
+         nb_threads_per_process = 2,
+         njobs = 4)
+
+**Note**: the `run` method is defined the `TurTLE/_code.py` file, and
+it is technically undocumented.
+In brief, it writes the C++ code for the executable, it compiles and
+links the binary, and then it launches it via an HPC queueing system,
+or directly on interactive machines (as determined from the
+`host_info.py` file).
+
+**Note**: The `write_par` method does not modify existing
+simulation files. The "parameters" group
+of any production `simname.h5` file should be treated as read-only after the
+first job has finished.
+Calls to `write_par` with the `force` argument turned on are
+performed at the user's risk.
+
+**Main benefit**: this approach will always work. When used for new
+codes (new C++ codes, and/or children/alternate Python classes to
+DNS),
+this approach may be modified/debugged without prior understanding of the relatively
+elaborate construction of the "user friendly" solution (which is meant to be
+used from the command line, and makes a number of limiting
+assumptions about the physical problem, as well as the C++ executable
+itself).