diff --git a/README.md b/README.md index 67258e1cc9a019fe88e7fc92d898f24ad39aa944..19a40e90365d0e94cb8e946bbf2f20a285c38bab 100644 --- a/README.md +++ b/README.md @@ -1,99 +1,36 @@ This project tries and test approaches that might lead to an improved architecture for NOMAD (XT). -## Getting started +## Generate the docs (and continue there) -### Install the legacy NOMAD submoduels. -This has to be done differently in the future. For no init the submodules and checkout -working branches/tags: -- submodules/parsers/parser-vasp master -- submodules/python-common master -- submodules/nomad-meta-info 1.6.0 - -To checkout a tag use: -``` -git fetch --all --tags --prune -git checkout tags/1.6.0 -b 1.6.0 -``` - -`pip install -r requirements` in `python-common`, and `pip install -e .` in `python-common` and -`parsers/parser-vasp`. Futhermore, there are some dependency issues in `python-commons` requirments. - -### Install the python in your own virtual environment. - -``` -pip install -r requirements.txt -pip install -e . -``` - -### Run dev infrastructure with docker. -You can do it with or without the ELK stack. - -To run is without (default): -``` -cd ./infrastructure -docker-compose build -sh up-wo-elk.sh -``` - -To run with ELK, enable `logstash` in nomad.config:logstash, and start the docker compose with +First, clone this repo: ``` -docker-compose up +git clone git@gitlab.mpcdf.mpg.de:mscheidg/nomad-xt.git +cd nomad-xt ``` -You can reach the Kibana with [localhost:5601](http://localhost:5601). -The index prefix for logs is `logstash-`. -Optionally register the infrastructue minio host to the minio client (mc). +Second, create and source your own virtual python environment: ``` -mc config host add minio http://localhost:9007 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +pip install virtualenv +virtualenv -p `which phyton3` .pyenv +source .pyenv/bin/activate ``` -### Run the celery worker (should be moved to docker TODO) - +Third, install the documentation system [sphinx](http://www.sphinx-doc.org/en/master/index.html): ``` -celery -A nomad.processing worker -l info +pip install sphinx +pip install recommonmark ``` -You can use different debug level (e.g. switch `info` to `debug`) -Use watchdog during development. Install (i.e. [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version fo MacOS) +Forth, generate the documentation: ``` -pip install git+https://github.com/gorakhargosh/watchdog.git +cd docs +make html ``` -Now use this to auto relead worker: +Conintue with reading the documentation for further setup and contribution guidelines: ``` -watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing -``` - -### Run tests. +cd .build/html +python -m SimpleHTTPServer .build -p 8888 ``` -python tests/test_files.py -``` - -### Generate the documentation - -Install [pdoc](https://pypi.org/project/pdoc/) if necessary, and run this from the project root: -``` -pdoc --html --html-dir=./build/docs --overwrite nomad -``` - -Do serve the current docs via HTTP during development, run -``` -pdoc --http --http-port 8080 -``` -and open [http://localhost:8080/nomad](http://localhost:8080/nomad). - -We will probably move to Sphynx at an apropriate moment. - -## Contributing - -### Code quality - -- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) to enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting). -- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish. -- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/). Use sphynx references a lot! -- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch [this](https://www.youtube.com/watch?v=wf-BqAjZb8M). -- The project structure is according to [this](https://docs.python-guide.org/writing/structure/) guide. Keep it! -- Test the public API of each submodule (i.e. python file) - - - +Open [http://localhost:8888/html/setup.html](http://localhost:8888/html/setup.html) in +your browser. diff --git a/docs/.DS_Store b/docs/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..efcf6d48ca6cbe1f32bb3708f624e8a39f3a1ada Binary files /dev/null and b/docs/.DS_Store differ diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..9d57e044758a3bbe670edc972057ad460e63957e --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,2 @@ +.build/ +.static/ \ No newline at end of file diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..0f345d76f36cedb102b50dbef04b2b56f16d2114 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +SPHINXPROJ = NOMAD +SOURCEDIR = . +BUILDDIR = .build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000000000000000000000000000000000000..585bbed46debd3f2d4d93d31bc089011cef2cda8 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,181 @@ +# -*- coding: utf-8 -*- +# +# Configuration file for the Sphinx documentation builder. +# +# This file does only contain a selection of the most common options. For a +# full list see the documentation: +# http://www.sphinx-doc.org/en/master/config + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +import os +import sys +from recommonmark.transform import AutoStructify + +sys.path.insert(0, os.path.abspath('../nomad')) + + +# -- Project information ----------------------------------------------------- + +project = 'NOMAD-XT' +copyright = '2018, the NOMAD developers' +author = 'the NOMAD developers' + +# The short X.Y version +version = '' +# The full version, including alpha/beta/rc tags +release = '' + + +# -- General configuration --------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.todo', + 'sphinx.ext.coverage', + 'sphinx.ext.ifconfig', + 'sphinx.ext.napoleon' +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['.templates'] + +source_parsers = { + '.md': 'recommonmark.parser.CommonMarkParser', +} + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +source_suffix = ['.rst', '.md'] + +# The master toctree document. +master_doc = 'index' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path . +exclude_patterns = ['.build', 'Thumbs.db', '.DS_Store'] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['.static'] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# The default sidebars (for documents that don't match any pattern) are +# defined by theme itself. Builtin themes are using these templates by +# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', +# 'searchbox.html']``. +# +# html_sidebars = {} + + +# -- Options for HTMLHelp output --------------------------------------------- + +# Output file base name for HTML help builder. +htmlhelp_basename = 'NOMADdoc' + + +# -- Options for LaTeX output ------------------------------------------------ + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'NOMAD-XT.tex', 'NOMAD-XT Documentation', + 'the NOMAD developers', 'manual'), +] + + +# -- Options for manual page output ------------------------------------------ + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'nomad', 'NOMAD-XT Documentation', + [author], 1) +] + + +# -- Options for Texinfo output ---------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'NOMAD-XT', 'NOMAD-XT Documentation', + author, 'NOMAD-XT', 'One line description of project.', + 'Miscellaneous'), +] + + +# -- Extension configuration ------------------------------------------------- + +# -- Options for todo extension ---------------------------------------------- + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = True + + +# Enably sphinx specifc markdown features +def setup(app): + app.add_config_value('recommonmark_config', { + 'enable_auto_doc_ref': True, + 'enable_eval_rst': True + }, True) + app.add_transform(AutoStructify) diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000000000000000000000000000000000000..f1549fa1cabd81461e6cfe7501a8a7fcba94de37 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,26 @@ +# Contributing + +The are some *rules* or better strong *guidelines* + +- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) or otherwise automatically +enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting). + +- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) +compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish. + +- Test the public API of each submodule (i.e. python file) + +- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch +[this](https://www.youtube.com/watch?v=wf-BqAjZb8M). + +- Document any *public* API of each submodule (e.g. python file). Public meaning API that +is exposed to other submodules (i.e. other python files). + +- Use google [docstrings](http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html). + +- Add your docstrings to the sphinx documentation in `docs`. Use .md, follow the example. +Markdown in sphix is supported via [recommonmark](https://recommonmark.readthedocs.io/en/latest/index.html#autostructify) +and [AutoStructify](http://recommonmark.readthedocs.io/en/latest/auto_structify.html) + +- The project structure is according to [this](https://docs.python-guide.org/writing/structure/) +guide. Keep it! diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000000000000000000000000000000000000..04d4ecf6931e2a62d54adc26fb66478d6f65c817 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,19 @@ +Welcome to NOMAD's documentation! +================================= + +This project tries and test approaches that might lead to an improved architecture for NOMAD (XT). + +.. toctree:: + :maxdepth: 2 + + setup + contributing + modules + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000000000000000000000000000000000000..3cda6c138e596c652a9e03f1f2d003de20392b5e --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,36 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=.build +set SPHINXPROJ=NOMAD + +if "%1" == "" goto help + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% + +:end +popd diff --git a/docs/modules.md b/docs/modules.md new file mode 100644 index 0000000000000000000000000000000000000000..60e908ef5dfb3c5e3f146c989cf5933d1afbfc1b --- /dev/null +++ b/docs/modules.md @@ -0,0 +1,6 @@ +# Modules + +## Files +```eval_rst +.. automodule:: nomad.files +``` diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000000000000000000000000000000000000..4e85514ac4f9a2a629bf18a2ca738f5c4df4e8e8 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,68 @@ +# Setup + +### Install the legacy NOMAD submoduels. +This has to be done differently in the future. For no init the submodules and checkout +working branches/tags: +- submodules/parsers/parser-vasp master +- submodules/python-common master +- submodules/nomad-meta-info 1.6.0 + +To checkout a tag use: +``` +git fetch --all --tags --prune +git checkout tags/1.6.0 -b 1.6.0 +``` + +`pip install -r requirements` in `python-common`, and `pip install -e .` in `python-common` and +`parsers/parser-vasp`. Futhermore, there are some dependency issues in `python-commons` requirments. + +### Install the python in your own virtual environment. + +``` +pip install -r requirements.txt +pip install -e . +``` + +### Run dev infrastructure with docker. +You can do it with or without the ELK stack. + +To run is without (default): +``` +cd ./infrastructure +docker-compose build +sh up-wo-elk.sh +``` + +To run with ELK, enable `logstash` in nomad.config:logstash, and start the docker compose with +``` +docker-compose up +``` +You can reach the Kibana with [localhost:5601](http://localhost:5601). +The index prefix for logs is `logstash-`. + +Optionally register the infrastructue minio host to the minio client (mc). +``` +mc config host add minio http://localhost:9007 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +``` + +### Run the celery worker (should be moved to docker TODO) + +``` +celery -A nomad.processing worker -l info +``` +You can use different debug level (e.g. switch `info` to `debug`) + +Use watchdog during development. Install (i.e. [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version fo MacOS) +``` +pip install git+https://github.com/gorakhargosh/watchdog.git +``` + +Now use this to auto relead worker: +``` +watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing +``` + +### Run tests. +``` +python tests/test_files.py +``` diff --git a/nomad/files.py b/nomad/files.py index 1844b5930603f4a2fe59aeb2bd3e6d7d8982d38d..5ab1f61d2bfc92c198f97fc6bd6a6e1690657472 100644 --- a/nomad/files.py +++ b/nomad/files.py @@ -13,11 +13,30 @@ # limitations under the License. """ -This module (and its main class :class:`Files`) represents an abstraction for NOMAD -file storage system. +NOMAD's file storage implementation +=================================== -Responsibilities: create, access files; create, receive, notify on, and access uploads. +This file storage abstraction currently uses the object storage API +http://minio.io to manage and organize files. Object storage +organizes files in *buckets/ids*, with small amounts of *buckets* and virtually +unlimited numbers of *ids*. *Ids* can contain delimiters like `/` to mimic +filesystem structure. There is a 1024 utf-8 character limit on *id* length. + +The file storage is organized in multiple buckets: + +* *uploads*: used for uploaded user code input/output archives. Currently only .zip files \ +are suported + +Presigned URLs +-------------- +Users (or GUI clients) can upload files directly to the object storage system. To avoid +authentication hassly, presigned URLs can be created that can be used directly to safely +*PUT* files. + +.. autofunction:: nomad.files.get_presigned_upload_url +.. autofunction:: nomad.files.create_curl_upload_cmd """ +import sys import os from os.path import join from zipfile import ZipFile, BadZipFile @@ -31,24 +50,52 @@ import nomad.config as config logger = logging.getLogger(__name__) -_client = Minio('%s:%s' % (config.minio.host, config.minio.port), - access_key=config.minio.accesskey, - secret_key=config.minio.secret, - secure=False) +_client = None + +if _client is None and 'sphinx' not in sys.modules: + _client = Minio('%s:%s' % (config.minio.host, config.minio.port), + access_key=config.minio.accesskey, + secret_key=config.minio.secret, + secure=False) + + # ensure all neccessary buckets exist + try: + _client.make_bucket(bucket_name=config.s3.uploads_bucket) + logger.info("Created uploads bucket with name %s." % config.s3.uploads_bucket) + except minio.error.BucketAlreadyOwnedByYou: + logger.debug( + "Uploads bucket with name %s already existed." % config.s3.uploads_bucket) -# ensure all neccessary buckets exist -try: - _client.make_bucket(bucket_name=config.s3.uploads_bucket) - logger.info("Created uploads bucket with name %s." % config.s3.uploads_bucket) -except minio.error.BucketAlreadyOwnedByYou: - logger.debug("Uploads bucket with name %s already existed." % config.s3.uploads_bucket) +def get_presigned_upload_url(upload_id: str) -> str: + """Generates a presigned upload URL. -def get_presigned_upload_url(upload_id): + Presigned URL allows users (and their client programs) to safely *PUT* + a single file without further authorization or API to the *uploads* bucket + using the given ``upload_id``. Example usages for presigned URLs include + browser based uploads or simple *curl* commands (see also :func:`create_curl_upload_cmd`). + + Args: + upload_id: The upload id for the uploaded file. + + Returns: + The presigned URL string. + """ return _client.presigned_put_object(config.s3.uploads_bucket, upload_id) def create_curl_upload_cmd(presigned_url, file_dummy='<ZIPFILE>'): + """Creates a readymade curl command for uploading. + + Args: + presigned_url: The presigned URL to base the command on. + + Kwargs: + file_dummy: A placeholder for the file that the user/client has to replace. + + Returns: + The curl shell command with correct method, url, headers, etc. + """ headers = 'Content-Type: application/octet-steam' return 'curl -X PUT "%s" -H "%s" -F file=@%s' % (presigned_url, headers, file_dummy)