Commit 73333905 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Added ELK to infra, experimented with python-logstash an celery workers.

parent ed197a70
......@@ -2,6 +2,22 @@ This project tries and test approaches that might lead to an improved architectu
## Getting started
### Install the legacy NOMAD submoduels.
This has to be done differently in the future. For no init the submodules and checkout
working branches/tags:
- submodules/parsers/parser-vasp master
- submodules/python-common master
- submodules/nomad-meta-info 1.6.0
To checkout a tag use:
```
git fetch --all --tags --prune
git checkout tags/1.6.0 -b 1.6.0
```
`pip install -r requirements` in `python-common`, and `pip install -e .` in `python-common` and
`parsers/parser-vasp`. Futhermore, there are some dependency issues in `python-commons` requirments.
### Install the python in your own virtual environment.
```
......
......@@ -19,6 +19,7 @@ services:
minio:
restart: always
image: minio/minio:RELEASE.2018-06-08T03-49-38Z
hostname: "minio"
# image: minio/minio
ports:
- 9007:9000
......@@ -32,6 +33,7 @@ services:
# broker for celery
rabbitmq:
restart: always
image: rabbitmq
hostname: "rabbitmq"
environment:
......@@ -50,3 +52,14 @@ services:
- './data/redis:/data'
ports:
- '6379:6379'
# used for central logging
elk:
restart: always
image: "sebp/elk"
hostname: "elk"
ports:
- 5601:5601 # kibana web
# - 9200:9200 # elastic search api
# - 9300:9300 # elastic transport api
- 5044:5044 # logstash
......@@ -16,11 +16,26 @@ from celery import Celery, group, subtask
from celery.result import result_from_tuple
import logging
import time
import logstash
import nomad.config as config
import nomad.files as files
from nomad.parsers import parsers, parser_dict
# def initialize_logstash(logger=None, loglevel=logging.INFO, **kwargs):
# handler = logstash.TCPLogstashHandler(
# 'localhost', 5044, tags=['celery'], message_type='celery', version=1)
# handler.setLevel(loglevel)
# logger.addHandler(handler)
# return logger
# from celery.signals import after_setup_task_logger
# after_setup_task_logger.connect(initialize_logstash)
# from celery.signals import after_setup_logger
# after_setup_logger.connect(initialize_logstash)
broker_url = 'pyamqp://%s:%s@localhost//' % (config.rabbitmq.user, config.rabbitmq.password)
backend_url = 'redis://localhost/0'
app = Celery('nomad.processing', backend=backend_url, broker=broker_url)
......
Subproject commit 7fb26fa234f426f3d9568f3ac9a64cc18e5044c3
Subproject commit c99a30a907e617fb39e900d3516790311c3e5c74
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment