diff --git a/documentation/index.rst b/documentation/index.rst index 36c60cd39a51e02fe717312b31da22ef1107032a..3da78b9ef38eb8db0b24d64966d6cdc2975157c7 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -15,6 +15,7 @@ Contents: sphinx_static/overview sphinx_static/convergence sphinx_static/bandpass + sphinx_static/launch sphinx_static/development chapters/api chapters/cpp_doxygen diff --git a/documentation/sphinx_static/launch.rst b/documentation/sphinx_static/launch.rst new file mode 100644 index 0000000000000000000000000000000000000000..49a0612b358541eb8ca085af9f7b84d354a085c0 --- /dev/null +++ b/documentation/sphinx_static/launch.rst @@ -0,0 +1,170 @@ +Tutorial: job launching methods +=============================== + +**Note**: This document assumes you are familiar with TurTLE +installation, and you've tested it on your machine. + +Context +------- + +A TurTLE direct numerical simulation is identified by a simulation name, and +a work directory. +This information directly corresponds to the HDF5 file `simname.h5` located +in the working directory and containing DNS parameters and basic diagnostics. + +A TurTLE DNS **job** consists of actually executing the C++ executable for +`niter_todo` iterations. +Each successful job will typically update the `iteration` value +stored in the DNS file, (as well as the checkpoint and diagnostics). + +A TurTLE **job** in general may be a DNS job, or it may be a +post-processing job, or it may be a test job. +In all three cases, **job** refers to the actual execution of the +binary code (generated from C++ code). + +TurTLE C++ executables typically require a `simname` command line +argument, that they use to find the relevant HDF5 file (different for +DNS and postprocessing jobs). +Depending on parameters and type of DNS, if `iteration` is 0 a DNS job may +also require a checkpoint file from which to read its initial condition; +otherwise the executable will generate its own initial condition and write +the 0 checkpoint as well. +If `iteration` is larger than 0 (it should be a multiple of +`niter_todo`, by the way), then any DNS job will require a corresponding +checkpoint file from which to read the DNS state. + +User friendly launch +-------------------- + +The default Python wrapper can do quite a lot. +I.e. you may execute from the command line + +.. code:: bash + + turtle DNS NSVE -n 64 --simname test64 --njobs 4 + +This will write the DNS HDF5 file, and it will generate 4 separate +DNS jobs (on an interactive computer, that means 4 subsequent +executions of the C++ executable with the same simulation name). + +If you decide that you would like to run this same DNS for longer, +all you need to do is navigate to the same folder and execute + +.. code:: bash + + turtle DNS NSVE --simname test64 --njobs 8 + +**Note** the "cubic box size" no longer needs to be specified, +because we are continuing an existing DNS, and the Python wrapper +will never overwrite parameters already present in the DNS file. +To clarify, the arguments present in these commands are arguments to +the Python wrapper `turtle`, which the Python wrapper interprets. +The argument to the C++ executable is then the simulation name, and +all other DNS parameters are passed through the HDF5 file (which is +silently generated by the Python wrapper if it does not exist). +In general all "physical" parameters follow this rule. + +**Note** the "HPC job parameters" may be different between different +job launches with the same DNS. +I.e. we may first execute + +.. code:: bash + + turtle DNS NSVE -n 64 --simname test64 --njobs 2 --np 2 --ntpp 2 + +If we decide that this is too slow, when we continue the DNS we can request +a different number of MPI processes (`np` Python wrapper argument) or a +different number of OpenMP threads per MPI process (`ntpp` Python wrapper +argument): + +.. code:: bash + + turtle DNS NSVE --simname test64 --njobs 2 --np 4 --ntpp 2 + +Technically the python wrapper can be used from the shell as any +other shell command: more advanced users may generate custom scripts +that call the wrapper from loops for different simulation names for +example, changing other job/DNS parameters as required. + +**Main benefit**: on HPC clusters, the Python wrapper will generate +appropriate SLURM scripts, and issue the requires `sbatch` commands. +When multiple jobs are requested, it will also create the job +dependencies appropriately. + +Python hack of bash tools +------------------------- + +Rather than executing the `turtle` command from bash scripts, one may +create TurTLE DNS objects within Python scripts and use the same +mechanism directly from Python, writing out bash arguments as Python +lists of strings. +For example: + +.. code:: python + + from TurTLE import DNS + c = DNS() + c.launch([ + 'NSVE', + '-n', '64', + '--simname', 'test64', + '--njobs', '4']) + +The above Python code can also be turned into an elaborate mechanism +to sample different values of some parameter, or to manage some other +"multiple DNS" scenario. + +Depending on the specific DNS tool being used (i.e. something other +than `NSVE`), users may prefer Python scripts to bash scripts because +Python can also generate custom initial conditions before the call to +the `launch` method is made. + +Python solution +--------------- + +For some use-cases it is simpler to duplicate (then modify) the +`launch` method discussed above. +I.e. one may use the code below: + +.. code:: python + + from TurTLE import DNS + c = DNS() + c.simname = 'test64' + c.dns_type = 'NSVE' + c.name = 'cpp_executable_name' + c.parameters['nx'] = 64 + c.parameters['ny'] = 64 + c.parameters['nz'] = 64 + c.parameters['forcing_type'] = 'fixed_energy_injection_rate' + c.parameters['injection_rate'] = 0.4 + # also set viscosity etc + c.write_par(iter0 = 0) + # generate initial conditions + c.run( + nb_processes = 2, + nb_threads_per_process = 2, + njobs = 4) + +**Note**: the `run` method is defined the `TurTLE/_code.py` file, and +it is technically undocumented. +In brief, it writes the C++ code for the executable, it compiles and +links the binary, and then it launches it via an HPC queueing system, +or directly on interactive machines (as determined from the +`host_info.py` file). + +**Note**: The `write_par` method does not modify existing +simulation files. The "parameters" group +of any production `simname.h5` file should be treated as read-only after the +first job has finished. +Calls to `write_par` with the `force` argument turned on are +performed at the user's risk. + +**Main benefit**: this approach will always work. When used for new +codes (new C++ codes, and/or children/alternate Python classes to +DNS), +this approach may be modified/debugged without prior understanding of the relatively +elaborate construction of the "user friendly" solution (which is meant to be +used from the command line, and makes a number of limiting +assumptions about the physical problem, as well as the C++ executable +itself).