Commit a64d4b13 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Added a documentation system.

parent 9097f91e
This project tries and test approaches that might lead to an improved architecture for NOMAD (XT).
## Getting started
## Generate the docs (and continue there)
### Install the legacy NOMAD submoduels.
This has to be done differently in the future. For no init the submodules and checkout
working branches/tags:
- submodules/parsers/parser-vasp master
- submodules/python-common master
- submodules/nomad-meta-info 1.6.0
To checkout a tag use:
```
git fetch --all --tags --prune
git checkout tags/1.6.0 -b 1.6.0
```
`pip install -r requirements` in `python-common`, and `pip install -e .` in `python-common` and
`parsers/parser-vasp`. Futhermore, there are some dependency issues in `python-commons` requirments.
### Install the python in your own virtual environment.
```
pip install -r requirements.txt
pip install -e .
```
### Run dev infrastructure with docker.
You can do it with or without the ELK stack.
To run is without (default):
```
cd ./infrastructure
docker-compose build
sh up-wo-elk.sh
```
To run with ELK, enable `logstash` in nomad.config:logstash, and start the docker compose with
First, clone this repo:
```
docker-compose up
git clone git@gitlab.mpcdf.mpg.de:mscheidg/nomad-xt.git
cd nomad-xt
```
You can reach the Kibana with [localhost:5601](http://localhost:5601).
The index prefix for logs is `logstash-`.
Optionally register the infrastructue minio host to the minio client (mc).
Second, create and source your own virtual python environment:
```
mc config host add minio http://localhost:9007 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
pip install virtualenv
virtualenv -p `which phyton3` .pyenv
source .pyenv/bin/activate
```
### Run the celery worker (should be moved to docker TODO)
Third, install the documentation system [sphinx](http://www.sphinx-doc.org/en/master/index.html):
```
celery -A nomad.processing worker -l info
pip install sphinx
pip install recommonmark
```
You can use different debug level (e.g. switch `info` to `debug`)
Use watchdog during development. Install (i.e. [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version fo MacOS)
Forth, generate the documentation:
```
pip install git+https://github.com/gorakhargosh/watchdog.git
cd docs
make html
```
Now use this to auto relead worker:
Conintue with reading the documentation for further setup and contribution guidelines:
```
watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing
```
### Run tests.
cd .build/html
python -m SimpleHTTPServer .build -p 8888
```
python tests/test_files.py
```
### Generate the documentation
Install [pdoc](https://pypi.org/project/pdoc/) if necessary, and run this from the project root:
```
pdoc --html --html-dir=./build/docs --overwrite nomad
```
Do serve the current docs via HTTP during development, run
```
pdoc --http --http-port 8080
```
and open [http://localhost:8080/nomad](http://localhost:8080/nomad).
We will probably move to Sphynx at an apropriate moment.
## Contributing
### Code quality
- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) to enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting).
- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/). Use sphynx references a lot!
- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch [this](https://www.youtube.com/watch?v=wf-BqAjZb8M).
- The project structure is according to [this](https://docs.python-guide.org/writing/structure/) guide. Keep it!
- Test the public API of each submodule (i.e. python file)
Open [http://localhost:8888/html/setup.html](http://localhost:8888/html/setup.html) in
your browser.
File added
.build/
.static/
\ No newline at end of file
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = NOMAD
SOURCEDIR = .
BUILDDIR = .build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import os
import sys
from recommonmark.transform import AutoStructify
sys.path.insert(0, os.path.abspath('../nomad'))
# -- Project information -----------------------------------------------------
project = 'NOMAD-XT'
copyright = '2018, the NOMAD developers'
author = 'the NOMAD developers'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = ''
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.ifconfig',
'sphinx.ext.napoleon'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['.templates']
source_parsers = {
'.md': 'recommonmark.parser.CommonMarkParser',
}
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_suffix = ['.rst', '.md']
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = ['.build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['.static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'NOMADdoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'NOMAD-XT.tex', 'NOMAD-XT Documentation',
'the NOMAD developers', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'nomad', 'NOMAD-XT Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'NOMAD-XT', 'NOMAD-XT Documentation',
author, 'NOMAD-XT', 'One line description of project.',
'Miscellaneous'),
]
# -- Extension configuration -------------------------------------------------
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True
# Enably sphinx specifc markdown features
def setup(app):
app.add_config_value('recommonmark_config', {
'enable_auto_doc_ref': True,
'enable_eval_rst': True
}, True)
app.add_transform(AutoStructify)
# Contributing
The are some *rules* or better strong *guidelines*
- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) or otherwise automatically
enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting).
- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/)
compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
- Test the public API of each submodule (i.e. python file)
- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch
[this](https://www.youtube.com/watch?v=wf-BqAjZb8M).
- Document any *public* API of each submodule (e.g. python file). Public meaning API that
is exposed to other submodules (i.e. other python files).
- Use google [docstrings](http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html).
- Add your docstrings to the sphinx documentation in `docs`. Use .md, follow the example.
Markdown in sphix is supported via [recommonmark](https://recommonmark.readthedocs.io/en/latest/index.html#autostructify)
and [AutoStructify](http://recommonmark.readthedocs.io/en/latest/auto_structify.html)
- The project structure is according to [this](https://docs.python-guide.org/writing/structure/)
guide. Keep it!
Welcome to NOMAD's documentation!
=================================
This project tries and test approaches that might lead to an improved architecture for NOMAD (XT).
.. toctree::
:maxdepth: 2
setup
contributing
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=.build
set SPHINXPROJ=NOMAD
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
# Modules
## Files
```eval_rst
.. automodule:: nomad.files
```
# Setup
### Install the legacy NOMAD submoduels.
This has to be done differently in the future. For no init the submodules and checkout
working branches/tags:
- submodules/parsers/parser-vasp master
- submodules/python-common master
- submodules/nomad-meta-info 1.6.0
To checkout a tag use:
```
git fetch --all --tags --prune
git checkout tags/1.6.0 -b 1.6.0
```
`pip install -r requirements` in `python-common`, and `pip install -e .` in `python-common` and
`parsers/parser-vasp`. Futhermore, there are some dependency issues in `python-commons` requirments.
### Install the python in your own virtual environment.
```
pip install -r requirements.txt
pip install -e .
```
### Run dev infrastructure with docker.
You can do it with or without the ELK stack.
To run is without (default):
```
cd ./infrastructure
docker-compose build
sh up-wo-elk.sh
```
To run with ELK, enable `logstash` in nomad.config:logstash, and start the docker compose with
```
docker-compose up
```
You can reach the Kibana with [localhost:5601](http://localhost:5601).
The index prefix for logs is `logstash-`.
Optionally register the infrastructue minio host to the minio client (mc).
```
mc config host add minio http://localhost:9007 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
### Run the celery worker (should be moved to docker TODO)
```
celery -A nomad.processing worker -l info
```
You can use different debug level (e.g. switch `info` to `debug`)
Use watchdog during development. Install (i.e. [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version fo MacOS)
```
pip install git+https://github.com/gorakhargosh/watchdog.git
```
Now use this to auto relead worker:
```
watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing
```
### Run tests.
```
python tests/test_files.py
```
......@@ -13,11 +13,30 @@
# limitations under the License.
"""
This module (and its main class :class:`Files`) represents an abstraction for NOMAD
file storage system.
NOMAD's file storage implementation
===================================
Responsibilities: create, access files; create, receive, notify on, and access uploads.
This file storage abstraction currently uses the object storage API
http://minio.io to manage and organize files. Object storage
organizes files in *buckets/ids*, with small amounts of *buckets* and virtually
unlimited numbers of *ids*. *Ids* can contain delimiters like `/` to mimic
filesystem structure. There is a 1024 utf-8 character limit on *id* length.
The file storage is organized in multiple buckets:
* *uploads*: used for uploaded user code input/output archives. Currently only .zip files \
are suported
Presigned URLs
--------------
Users (or GUI clients) can upload files directly to the object storage system. To avoid
authentication hassly, presigned URLs can be created that can be used directly to safely
*PUT* files.
.. autofunction:: nomad.files.get_presigned_upload_url
.. autofunction:: nomad.files.create_curl_upload_cmd
"""
import sys
import os
from os.path import join
from zipfile import ZipFile, BadZipFile
......@@ -31,24 +50,52 @@ import nomad.config as config
logger = logging.getLogger(__name__)
_client = Minio('%s:%s' % (config.minio.host, config.minio.port),
access_key=config.minio.accesskey,
secret_key=config.minio.secret,
secure=False)
_client = None
if _client is None and 'sphinx' not in sys.modules:
_client = Minio('%s:%s' % (config.minio.host, config.minio.port),
access_key=config.minio.accesskey,
secret_key=config.minio.secret,
secure=False)
# ensure all neccessary buckets exist
try:
_client.make_bucket(bucket_name=config.s3.uploads_bucket)
logger.info("Created uploads bucket with name %s." % config.s3.uploads_bucket)
except minio.error.BucketAlreadyOwnedByYou:
logger.debug(
"Uploads bucket with name %s already existed." % config.s3.uploads_bucket)
# ensure all neccessary buckets exist
try:
_client.make_bucket(bucket_name=config.s3.uploads_bucket)
logger.info("Created uploads bucket with name %s." % config.s3.uploads_bucket)
except minio.error.BucketAlreadyOwnedByYou:
logger.debug("Uploads bucket with name %s already existed." % config.s3.uploads_bucket)
def get_presigned_upload_url(upload_id: str) -> str:
"""Generates a presigned upload URL.
def get_presigned_upload_url(upload_id):
Presigned URL allows users (and their client programs) to safely *PUT*
a single file without further authorization or API to the *uploads* bucket
using the given ``upload_id``. Example usages for presigned URLs include
browser based uploads or simple *curl* commands (see also :func:`create_curl_upload_cmd`).
Args:
upload_id: The upload id for the uploaded file.
Returns:
The presigned URL string.
"""
return _client.presigned_put_object(config.s3.uploads_bucket, upload_id)
def create_curl_upload_cmd(presigned_url, file_dummy='<ZIPFILE>'):
"""Creates a readymade curl command for uploading.
Args:
presigned_url: The presigned URL to base the command on.
Kwargs:
file_dummy: A placeholder for the file that the user/client has to replace.
Returns:
The curl shell command with correct method, url, headers, etc.
"""
headers = 'Content-Type: application/octet-steam'
return 'curl -X PUT "%s" -H "%s" -F file=@%s' % (presigned_url, headers, file_dummy)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment