Experiments with celery for processing uploads.

.gitignore

 .pyenv/
 __pycache__
+*.pyc
 *.egg-info/
 data
\ No newline at end of file

README.md

-## Installing
+This project tries and test approaches that might lead to an improved architecture for NOMAD (XT).
+
+## Getting started
+
+### Install the python in your own virtual environment.
 
 ```
 pip install -r requirements.txt
 pip install -e .
 ```
 
-## Contributing
-
-### Code quality
-
-- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
-- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/).
-
-### Tests
-
-To run the test execute this from the project root:
+### Run dev infrastructure with docker.
+```
+cd ./infrastructure
+docker-compose build
+docker-compose up
 ```
 
+### Run tests.
+```
+python tests/test_files.py
 ```
 
 ### Generate the documentation
@@ -32,3 +34,18 @@ pdoc --http --http-port 8080
 ```
 and open [http://localhost:8080/nomad](http://localhost:8080/nomad).
 
+We will probably move to Sphynx at an apropriate moment.
+
+## Contributing
+
+### Code quality
+
+- Use an IDE (e.g. [vscode](https://code.visualstudio.com/)) to enforce code [formatting and linting](https://code.visualstudio.com/docs/python/linting).
+- There is a style guide to python. Write [pep-8](https://www.python.org/dev/peps/pep-0008/) compliant python code. An exception is the line cap at 79, which can be broken but keep it 90-ish.
+- Use docstrings and document any *public* API of each submodule (e.g. python file). Public meaning API that is exposed to other submodules (i.e. other python files). Keep docstrings compliant to [pep-257](https://www.python.org/dev/peps/pep-0257/). Use sphynx references a lot!
+- Be [pythonic](https://docs.python-guide.org/writing/style/) and watch [this](https://www.youtube.com/watch?v=wf-BqAjZb8M).
+- The project structure is according to [this](https://docs.python-guide.org/writing/structure/) guide. Keep it!
+- Test the public API of each submodule (i.e. python file)
+
+
+

infrastructure/docker-compose.yml

@@ -14,16 +14,14 @@ services:
       - "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE"
       - "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
     command: server /data
-    # links:
-    #   - rabbitmq
 
-  # rabbitmq:
-  #   image: rabbitmq
-  #   hostname: "rabbitmq"
-  #   environment:
-  #     - "RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG"
-  #     - "RABBITMQ_DEFAULT_USER=rabbitmq"
-  #     - "RABBITMQ_DEFAULT_PASS=rabbitmq"
-  #     - "RABBITMQ_DEFAULT_VHOST=/"
-  #   ports:
-  #     - 5672:5672
+  rabbitmq:
+    image: rabbitmq
+    hostname: "rabbitmq"
+    environment:
+      - "RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG"
+      - "RABBITMQ_DEFAULT_USER=rabbitmq"
+      - "RABBITMQ_DEFAULT_PASS=rabbitmq"
+      - "RABBITMQ_DEFAULT_VHOST=/"
+    ports:
+      - 5672:5672

nomad/processing.py

+from celery import Celery, chain, chord, group
+import nomad.config as config
+import nomad.files as files
+
+broker_url = 'pyamqp://%s:%s@localhost//' % (config.rabbitmq.user, config.rabbitmq.password)
+backend_url = 'rpc://localhost'
+app = Celery('nomad.processing', backend=backend_url, broker=broker_url)
+
+
+@app.task()
+def process(upload):
+  mainfiles = [('a', 'pa'), ('b', 'pb'), ('c', 'pc')]
+  parsers = group([parse.s(mainfile, parser) for mainfile, parser in mainfiles]).delay()
+  return parsers
+
+@app.task()
+def parse(mainfile, parser):
+  return 'parsed %s with %s' % (mainfile, parser)
+
+if __name__ == '__main__':
+  print(~process.s('test'))

requirements.txt

 minio
 pika
 unittest
-requests
\ No newline at end of file
+requests
+celery
\ No newline at end of file