Config refactor
Our configuration mechanism has currently several drawbacks:
- Validation is happening too early, causing configuration to be ignored
- Configurations defaults are sprinkled in several places.
- We are not using pydantic validators properly
- The config variables partly shadow python modules (e.g.
nomad.config.north
)
Steps to fix:
-
Most of the default values should go into a new nomad/config/default.yaml
file that is loaded first. This way it is very clear where to find the defaults, whereas currently the defaults are sprinkled as model defaults with no clear rules on whether parents of children should define the defaults. Note that derived values are still controlled by the models, and should not be included indefault.yaml
. -
Before loading the configuration options into a Pydantic model, we should merge the config values from default.yaml
,nomad.yaml
and the environment variables with a custom stragegy. This will allow us to delay the model validation to the very end where all data has been loaded and merged. -
Refactor config tests: they are very shallow at the moment. At least the following should be tested: -
Env variables are read correctly -
Loading config from custom location works -
YAML file is read correctly -
The priority over default.yaml
, YAML and env is correct -
Merging of nested structures happens correctly: dictionaries are merged, everything else overridden. -
Test that it is possible to provide incomplete models in config (validation should only happen at the very end).
-
-
Refactor code structure: currently there is nomad.config
subpackage which by default imports the config fields. This is quite messy as it is impossible to e.g. say ifnomad.config.fs
is a python module, or a config parameter. We also don't have a root model for our config. This means that we can't automatically iterate over models field when creating our docs, or add pydantic validators for the root config. To solve this problem, we should move all config models intonomad.config.models
, and then innomad/config/__init__.py
create and expose a config model instance + subfields for backwards compatibility. The default way to access config should becomefrom nomad.config import config
, as e.g. fromnomad.config import archive
will not makemypy
happy. -
Harmonize model validation. Currently, parts of the validation happen outside the pydantic validator framework: these should be moved into validators
orroot_validators
. -
Harmonize the way validation errors and warnings in the config are handled. The stragegy for now is: Pydantic validation errors will cause exceptions that halt the program, any extra fields that are not part of the models will cause a warning to be logged.
Edited by Lauri Himanen