Commit 0d580e9c authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Experiments with celery for processing uploads.

parent 4e781c19
.pyenv/
__pycache__
*.pyc
*.egg-info/
data
\ No newline at end of file
## Installing
This project tries and test approaches that might lead to an improved architecture for NOMAD (XT).
## Getting started
### Install the python in your own virtual environment.
```
pip install -r requirements.txt
pip install -e .
```
## Contributing
### Code quality
- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/).
### Tests
To run the test execute this from the project root:
### Run dev infrastructure with docker.
```
cd ./infrastructure
docker-compose build
docker-compose up
```
### Run tests.
```
python tests/test_files.py
```
### Generate the documentation
......@@ -32,3 +34,18 @@ pdoc --http --http-port 8080
```
and open [http://localhost:8080/nomad](http://localhost:8080/nomad).
We will probably move to Sphynx at an apropriate moment.
## Contributing
### Code quality
- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) to enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting).
- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/). Use sphynx references a lot!
- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch [this](https://www.youtube.com/watch?v=wf-BqAjZb8M).
- The project structure is according to [this](https://docs.python-guide.org/writing/structure/) guide. Keep it!
- Test the public API of each submodule (i.e. python file)
......@@ -14,16 +14,14 @@ services:
- "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE"
- "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
command: server /data
# links:
# - rabbitmq
# rabbitmq:
# image: rabbitmq
# hostname: "rabbitmq"
# environment:
# - "RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG"
# - "RABBITMQ_DEFAULT_USER=rabbitmq"
# - "RABBITMQ_DEFAULT_PASS=rabbitmq"
# - "RABBITMQ_DEFAULT_VHOST=/"
# ports:
# - 5672:5672
rabbitmq:
image: rabbitmq
hostname: "rabbitmq"
environment:
- "RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG"
- "RABBITMQ_DEFAULT_USER=rabbitmq"
- "RABBITMQ_DEFAULT_PASS=rabbitmq"
- "RABBITMQ_DEFAULT_VHOST=/"
ports:
- 5672:5672
from celery import Celery, chain, chord, group
import nomad.config as config
import nomad.files as files
broker_url = 'pyamqp://%s:%s@localhost//' % (config.rabbitmq.user, config.rabbitmq.password)
backend_url = 'rpc://localhost'
app = Celery('nomad.processing', backend=backend_url, broker=broker_url)
@app.task()
def process(upload):
mainfiles = [('a', 'pa'), ('b', 'pb'), ('c', 'pc')]
parsers = group([parse.s(mainfile, parser) for mainfile, parser in mainfiles]).delay()
return parsers
@app.task()
def parse(mainfile, parser):
return 'parsed %s with %s' % (mainfile, parser)
if __name__ == '__main__':
print(~process.s('test'))
minio
pika
unittest
requests
\ No newline at end of file
requests
celery
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment