Preparing a local environment for the sbt tool
-
Follow the instructions in nomad-lab-base/Setup.markdown to install the needed software to run the tools that are available through sbt
-
Preferably create a new python virtualenv, e.g. with
mkvirtualenv labEnv3 -p python3
Install the needed dependencies by following the README.md file in the python-common repository, or in short:
cd nomad-lab-base/python-common workon labEnv3 pip install -r requirements.txt pip install -e .
-
Modify the file reference.conf in core/src/main/resources to use the localDir configuration
configurationToUse="localDir"
and possibly update the configuration "localDir.nomad_lab" to use the correct python binary from a virtual environment folder, e.g. ${?HOME}"/.virtualenvs/<env_name>/bin/python
Setting up nomad-lab-base to detect and run a new normalizer
The normalizer should have it's own git repository within the nomad-lab group in gitlab. You can ask one of the group owners to create a new repository.
Once the repository is created, you need to add it as a submodule to the nomad-lab-base repository. You can do this with the following command:
git submodule add git@gitlab.mpcdf.mpg.de:nomad-lab/<normalizer_name>.git ./normalizers/<normalizer_name>
@berko says: We might want to use the following command instead of the above one because nomad-lab-base is also a directory of nomad-lab and skip the manual modification at .gitmodules file since we might want to avoid breaking git repo by manually editing its files.
git submodule add ../<normalizer_name>.git ./normalizers/<normalizer_name>
The sbt-tool also requires that the project has at least one tag specified, so you will need to create a new tag with the following command within the normalizer repository:
git tag 0.0.1
git push --tags
You will need to also modify the .gitmodules file to use a local git file for the new normalizer. By default the submodule will use the remote url, but you can change it to local with something like this:
[submodule "normalizers/<normalizer_name>"]
path = normalizers/<normalizer_name>
url = ../normalizer-<normalizer_name>.git
Next we need to tell sbt to recognize the new normalizer when running the normalize-command. First add a new variable declaration to build.sbt (look for other normalizer declarations and follow the same syntax):
lazy val <normalizer_name> = (project in file("normalizers/<normalizer_name>")).
dependsOn(core).
enablePlugins(BuildInfoPlugin).
settings(gitVersionSettings: _*).
settings(commonSettings: _*).
settings(
name := "<normalizer_name>",
(unmanagedResourceDirectories in Compile) += baseDirectory.value / "normalizer"
).
settings(Revolver.settings: _*)
Then we need to add it as a part of the normalize-command:
lazy val normalize = (project in file("normalize")).
dependsOn(<normalizer_name>).
...
Finally add it to the root within the aggregate function:
aggregate(
<normalizer_name>, ...
If you want to activate the new normalizer by default, you will also need declare it in "nomad-lab-base/normalize/src/main/scala/eu/nomad_lab/normalizers/AllNormalizers.scala".
/**
* All active normalizers
*/
val activeNormalizers: Map[String, NormalizerGenerator] = {
listToMap(Seq(
<normalizer_name>,
Running the Normalizer
Open up sbt from the directory nomad-lab-base, where the file build.sbt is:
sbt
To run a specific normalizer on a HDF file, run the following command:
tool/run normalize --archive-uri nmd://S1Sz7mm16YsS6qTg0LZ36RaUXfgHW --normalizers <NameOfNormalizer>
The name of the normalizer is case-sensitive and should be written exactly as it was written in AllNormalizers.scala.
The name of the archive should start with the letter S, which identifies a parsed file. The abbreviation nmd://S1Sz7mm16YsS6qTg0LZ36RaUXfgHW will tell the normalizer to look for the path parsed/productionH5/S1S/S1Sz7mm16YsS6qTg0LZ36RaUXfgHW.h5. The root folder parsed is defined by the settings in reference.conf