Commit d70ce4ea authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Moved from dependencies.py to git submodules.

parent 773b8278
......@@ -5,7 +5,6 @@ __pycache__
*.pyc
*.egg-info/
/data/
.dependencies/
.volumes/
.pytest_cache/
.coverage
......
[submodule "submodules/python-common"]
path = submodules/python-common
url = git@gitlab.mpcdf.mpg.de:nomad-lab/python-common.git
[submodule "submodules/meta-info"]
path = submodules/nomad-meta-info
url = git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-meta-info.git
[submodule "submodules/parsers/vasp"]
path = submodules/parsers/vasp
url = git@gitlab.mpcdf.mpg.de:nomad-lab/parser-vasp.git
[submodule "dependencies/nomad-meta-info"]
path = dependencies/nomad-meta-info
url = https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-meta-info.git
branch = nomad-fair
[submodule "dependencies/python_common"]
path = dependencies/python_common
url = https://gitlab.mpcdf.mpg.de/nomad-lab/python-common.git
branch = nomad-fair
[submodule "dependencies/parsers/vasp"]
path = dependencies/parsers/vasp
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-vasp.git
branch = nomad-fair
[submodule "dependencies/parsers/exciting"]
path = dependencies/parsers/exciting
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-exciting.git
branch = nomad-fair
[submodule "dependencies/parsers/fhi-aims"]
path = dependencies/parsers/fhi-aims
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-fhi-aims.git
branch = nomad-fair
[submodule "dependencies/parsers/cp2k"]
path = dependencies/parsers/cp2k
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-cp2k
branch = nomad-fair
[submodule "dependencies/parsers/crystal"]
path = dependencies/parsers/crystal
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-crystal
branch = nomad-fair
[submodule "dependencies/parsers/cpmd"]
path = dependencies/parsers/cpmd
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-cpmd
branch = nomad-fair
[submodule "dependencies/parsers/nwchem"]
path = dependencies/parsers/nwchem
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-nwchem
branch = nomad-fair
[submodule "dependencies/parsers/bigdft"]
path = dependencies/parsers/bigdft
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-big-dft
branch = nomad-fair
[submodule "dependencies/parsers/wien2k"]
path = dependencies/parsers/wien2k
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-wien2k
branch = nomad-fair
\ No newline at end of file
......@@ -19,10 +19,11 @@ You can access the running system and its documentation here:
### Generate the docs from the source
First, clone this repo:
First, clone this repo and init its submodules:
```
git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
git submodules init --depth 1
```
Second, create and source your own virtual python environment:
......
#!/bin/sh
set -e
git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
while read path_key path
do
echo $path
[ -f $path/requirements.txt ] && pip install -r $path/requirements.txt
[ -f $path/setup.py ] && pip install $1 $path
done
Subproject commit 716f3495427527446b693e62bfdf8bd30290d1d0
Subproject commit 4114f96be4985a288b293be1ef26f830d6065eec
Subproject commit b87cd29be9fe1637c20a23087ce98855ff1fea44
Subproject commit a8b323f8f8b24a5cead4c7e0cca15a78193b12b5
Subproject commit a824f0f72303bb8e39c8ae7ee710210edc287331
Subproject commit 3028201afe3f5b839afe3212769f193c9f634069
Subproject commit cb54c37d0c1b85cd57486f572de7bc2dbc294123
Subproject commit 179c4ce3e0990b431869a7d644590689e91f8fe2
Subproject commit 05e5f608178df11fe5563dcb88c3e475d51cd230
Subproject commit 03899751624694c7099d5bc3e5b48971d924bc32
Subproject commit f03817034886e2c7225806e284a1eb8ecd8b3e25
......@@ -115,16 +115,14 @@ said entities.
NOMAD-coe Dependencies
----------------------
We currently clone and install NOMAD-coe dependencies *"outside"* the nomad-FAIRDI project
(see :py:mod:`nomad.dependencies`). The installed projects become part of the python
environment and all dependencies are used like regular pipy packages and python modules.
This allows us to target (e.g. install) individual commits. In theory, these might
change during runtime, allowing to update parsers or normalizers on a running nomad.
More importantly, we can address commit hashes to identify exact parser/normalizer versions.
On the downside, common functions for all dependencies (e.g. the python-common package,
or nomad_meta_info) cannot be part of the nomad-FAIRDI project. In general, it is hard
to simultaneously develop nomad-FAIRDI and NOMAD-coe dependencies.
We currently use git submodules to maintain references to NOMAD-coe dependencies.
All dependencies are python packages and installed via pip to your python environement.
This allows us to target (e.g. install) individual commits. More importantly, we can address c
ommit hashes to identify exact parser/normalizer versions. On the downside, common functions
for all dependencies (e.g. the python-common package, or nomad_meta_info) cannot be part
of the nomad-FAIRDI project. In general, it is hard to simultaneously develop nomad-FAIRDI
and NOMAD-coe dependencies.
Another approach is to integrate the NOMAD-coe sources with nomad-FAIRDI. The lacking
availability of individual commit hashes, could be replaces with hashes of source-code
......
......@@ -315,21 +315,16 @@ You a writing a python program. You know what to do.
## Added the parser to nomad@FAIRDI
First, you add your project to the list of :py:mod:`nomad.dependencies`:
```python
dependencies = [
PythonGit(
name='parsers/vasp',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-vasp.git',
git_branch='nomad-fair'),
]
First, you add your parser to the dependencies. Put it into the dependencies folder, then:
```
git submodule add dependencies/parsers/vasp
```
Second, you add your parser to the list of parsers :py:mod:`nomad.parsing`:
```python
parsers = [
LegacyParser(
python_git=dependencies['parsers/vasp'],
name='parsers/vasp',
parser_class_name='vaspparser.VaspOutcarParser',
main_file_re=r'^OUTCAR(\.[^\.]*)?$',
main_contents_re=(r'^\svasp\..*$')
......
......@@ -11,11 +11,6 @@ nomad.infrastructure
.. automodule:: nomad.infrastructure
:members:
nomad.dependencies
------------------
.. automodule:: nomad.dependencies
:members:
nomad.datamodel
---------------
.. automodule:: nomad.datamodel
......
......@@ -63,10 +63,10 @@ different parser, normalizer versions from within the running nomad infrastructu
To run the dependencies script and install all dependencies into your environment:
```
python nomad/dependencies.py --dev
./dependencies.sh
```
This will checkout the proper version of the respective NOMAD-coe modules, install
further requirements, and install the modules themselves. The `--dev` option will install
further requirements, and install the modules themselves. The `-e` option will install
the NOMAD-coe dependencies with symbolic links allowing you to change the downloaded
dependency code without having to reinstall after.
......
# Copyright 2018 Markus Scheidgen
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an"AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
This module allows to configure and install all necessary legecy nomad GIT repositories
to process (parser, normalizer, etc.) uploaded calculations.
Parsers are developed as independent, individual python programs in their own GIT repositories.
They are built on a common modules called *python-common*, also in a separate GIT.
All parsers depend on the *meta-info*, which is also maintained in its own GIT.
Using dependencies
------------------
To install all dependencies (clone, install requirements, install the dependency as python package)
use the following:
.. code-block:: sh
python -m nomad.dependencies --dev
With the `--dev` parameter *pip* will use `-e` to install. The dependencies are cloned
into `.dependencies` and installed into your current python environment.
Adding dependencies
----------------------
To make GIT maintained python modules available, we use:
.. autoclass:: PythonGit
:members:
Dependencies are listed in
.. autodata:: dependencies
Dependencies can be programmatically installed with:
.. autofunction:: prepare
"""
import sys
import os
import os.path
import logging
import subprocess
import shutil
_logger = logging.getLogger(__name__)
base_dir = './.dependencies'
class PythonGitError(Exception):
def __init__(self, msg, repo):
msg = '%s [%s]' % (msg, repo)
super().__init__(msg)
class PythonGit():
"""
Represents a python module in a git repository. It allows to fetch a specific commit,
install all requirements to the current python environment, and check the installation
via module import.
This is only useful before you want to use the respective module in a different
python process, because it will not try to reload any already loaded modules into
the current python process.
Arguments:
name: A name that determines the download path, can contain '/' for sub dirs.
Names are important, because modules might use relatives paths between
them.
git_url: A publically available and fetchable url to the GIT repository.
git_branch: The branch that should be used.
"""
def __init__(self, name: str, git_url: str, git_branch: str) -> None:
self.name = name
self.git_url = git_url
self.git_branch = git_branch
def _run_pip_install(self, *args):
pipcode = 0
# some weird interaction of pip and virtualenv causes a bug that does
# not allow to install in docker due to a wrong PIP_REQ_TRACKER path. This
# is a workarround.
pip_req_tracker_key = 'PIP_REQ_TRACKER'
env = dict(os.environ)
if pip_req_tracker_key in env:
del(env['PIP_REQ_TRACKER'])
pipcode = subprocess.call(
[sys.executable, '-m', 'pip', 'install'] + list(args),
env=env)
if pipcode != 0:
raise PythonGitError(
'Could not install (pip return code=%s)' % pipcode, repo=self)
def prepare(self, dev: bool = False) -> None:
"""
Makes sure that the repository is fetched, at the right commit, and installed.
Arguments:
dev (bool): Indicate dev install (uses pip with -e). Default is False.
Raises:
PythonGitError: if something went wrong.
"""
# import late because git will not be available in production
from git import Repo, Git
# check/change working directory
old_cwd = os.getcwd()
try:
cwd = os.path.join(base_dir, self.name)
if not os.path.exists(cwd):
os.makedirs(cwd)
os.chdir(cwd)
_logger.info('check git/do init with origin %s for %s' % (self.git_url, self.name))
if os.path.exists('.git'):
git = Repo('./')
else:
git_cmd = Git('./')
git_cmd.init()
git = Repo('./')
origin = git.create_remote('origin', self.git_url)
_logger.info('pull %s for %s' % (self.git_branch, self.name))
origin = git.remote('origin')
origin.pull(self.git_branch, depth=1)
if git.head.name != self.git_branch:
new_branch = git.create_head(self.git_branch)
git.head.reference = new_branch
try:
git.submodule_update(init=True)
except Exception:
pass
if os.path.exists('requirements.txt'):
_logger.info('install requirements.txt for %s' % self.name)
self._run_pip_install('-r', 'requirements.txt')
if os.path.exists('setup.py'):
_logger.info('install setup.py for %s' % self.name)
if dev:
self._run_pip_install('-e', '.')
else:
self._run_pip_install('.')
except PythonGitError as e:
raise e
except Exception as e:
raise PythonGitError(
'Unexpected exception during preparation: %s' % e, repo=self)
finally:
os.chdir(old_cwd)
pass
def __repr__(self):
return self.name
dependencies = [
# repository api is not really usuable, because it is written in python 2.x
# PythonGit(
# name='repository-api',
# git_url='https://gitlab.mpcdf.mpg.de/NoMaD/NomadRepositoryParser.git',
# git_branch='v2.1'
# ),
# not strictly necessary or useful for the common build
# PythonGit(
# name='nomad-lab-base',
# git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-lab-base.git',
# git_branch='nomad-FAIR'),
PythonGit(
name='nomad-meta-info',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-meta-info.git',
git_branch='nomad-fair'),
PythonGit(
name='python_common',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/python-common.git',
git_branch='nomad-fair'),
PythonGit(
name='parsers/vasp',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-vasp.git',
git_branch='nomad-fair'),
PythonGit(
name='parsers/exciting',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-exciting.git',
git_branch='nomad-fair'),
PythonGit(
name='parsers/fhi-aims',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-fhi-aims.git',
git_branch='nomad-fair'),
PythonGit(
name='parsers/cp2k',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-cp2k',
git_branch='nomad-fair'),
PythonGit(
name='parsers/crystal',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-crystal',
git_branch='nomad-fair'),
PythonGit(
name='parsers/cpmd',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-cpmd',
git_branch='nomad-fair'),
PythonGit(
name='parsers/nwchem',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-nwchem',
git_branch='nomad-fair'),
PythonGit(
name='parsers/bigdft',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-big-dft',
git_branch='nomad-fair'),
PythonGit(
name='parsers/wien2k',
git_url='https://gitlab.mpcdf.mpg.de/nomad-lab/parser-wien2k',
git_branch='nomad-fair')
]
dependencies_dict = {dependency.name: dependency for dependency in dependencies}
def prepare(*args, **kwargs) -> None:
"""
Installs all dependencies from :data:`dependencies` and :data:`parsers`.
"""
for python_git in dependencies:
python_git.prepare(*args, **kwargs)
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser(description='Install dependencies from NOMAD-coe.')
parser.add_argument('--dev', help='pip install with -e', action='store_true')
parser.add_argument('--clean', help='remove old deps first', action='store_true')
args = parser.parse_args()
_logger.setLevel(logging.DEBUG)
if args.clean and os.path.exists(base_dir):
shutil.rmtree(base_dir)
prepare(dev=args.dev)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment