setup.md 11.9 KB
Newer Older
Markus Scheidgen's avatar
Markus Scheidgen committed
1
# Development Setup
2

Markus Scheidgen's avatar
Markus Scheidgen committed
3
4
5
## Introduction
The nomad infrastructure consists of a series of nomad and 3rd party services:
- nomad worker (python): task worker that will do the processing
6
- nomad app (python): the nomad app and it's REST APIs
Markus Scheidgen's avatar
Markus Scheidgen committed
7
8
9
10
11
- nomad gui: a small server serving the web-based react gui
- proxy: an nginx server that reverse proxyies all services under one port
- elastic search: nomad's search and analytics engine
- mongodb: used to store processing state
- rabbitmq: a task queue used to distribute work in a cluster
12

Markus Scheidgen's avatar
Markus Scheidgen committed
13
All 3rd party services should be run via *docker-compose* (see blow). The
14
15
nomad python  services can be run with python to develop them.
The gui can be run with a development server via yarn.
Markus Scheidgen's avatar
Markus Scheidgen committed
16
17

Below you will find information on how to install all python dependencies and code
18
manually. How to use *docker*/*docker-compose*. How run 3rd-party services with *docker-compose*.
Markus Scheidgen's avatar
Markus Scheidgen committed
19
20
21
22
23

Keep in mind the *docker-compose* configures all services in a way that mirror
the configuration of the python code in `nomad/config.py` and the gui config in
`gui/.env.development`.

24
25
26
To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in
production, go (here)(/app/docs/ops.html).

Markus Scheidgen's avatar
Markus Scheidgen committed
27
28
29
30
31
32
## Install python code and dependencies

### Cloning and development tools
If not already done, you should clone nomad and create a python virtual environment.

To clone the repository:
33
```
34
35
git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
36
37
```

38
39
40
### C libs

Even though the NOMAD infrastructure is written in python, there is a C library
41
required by one of our python dependencies.
42
43
44
45
46
47
48
49
50
51
52
53
54

#### libmagic

Libmagic allows to determine the MIME type of files. It should be installed on most
unix/linux systems. It can be installed on MacOS with homebrew:

```
brew install libmagic
```

### Virtual environment

#### pyenv
55
56
57
The nomad code currently targets python 3.7. If you host machine has an older version installed,
you can use [pyenv](https://github.com/pyenv/pyenv) to use python 3.7 in parallel to your
system's python.
58

59
#### virtualenv
60
We strongly recommend to use *virtualenv* to create a virtual environment. It will allow you
Markus Scheidgen's avatar
Markus Scheidgen committed
61
to keep nomad and its dependencies separate from your system's python installation.
62
63
Make sure to base the virtual environment on Python 3.
To install *virtualenv*, create an environment and activate the environment use:
64
65
66
67
68
69
```
pip install virtualenv
virtualenv -p `which python3` .pyenv
source .pyenv/bin/activate
```

70
71
72
73
#### Conda
If you are a conda user, there is an equivalent, but you have to install pip and the
right python version while creating the environment.
```
74
conda create --name nomad_env pip python=3.7
75
76
77
78
79
conda activate nomad_env
```

To install libmagick for conda, you can use (other channels might also work):
```
80
conda install -c conda-forge --name nomad_env libmagic
81
82
```

83
84
85
86
87
88
89
90
#### pip
Make sure you have the most recent version of pip:
```
pip install --upgrade pip
```


The next steps can be done using the `setup.sh` script. If you prefer to understand all
91
92
the steps and run them manually, read on:

93

Markus Scheidgen's avatar
Markus Scheidgen committed
94
95
### Install NOMAD-coe dependencies.
Nomad is based on python modules from the NOMAD-coe project.
96
97
This includes parsers, python-common and the meta-info. These modules are maintained as
their own GITLab/git repositories. To clone and initialize them run:
98

99
```
100
git submodule update --init
101
```
102

103
104
105
All requirements for these submodules need to be installed and they need to be installed
themselves as python modules. Run the `dependencies.sh` script that will install
everything into your virtual environment:
106
```
107
./dependencies.sh -e
108
```
109
110
111

The `-e` option will install the NOMAD-coe dependencies with symbolic links allowing you
to change the downloaded dependency code without having to reinstall after.
112

Markus Scheidgen's avatar
Markus Scheidgen committed
113
### Install nomad
114
Finally, you can add nomad to the environment itself (including all extras)
115
```
116
pip install -e .[all]
117
118
```

Markus Scheidgen's avatar
Setup    
Markus Scheidgen committed
119
120
121
122
123
### Generate GUI artifacts
The NOMAD GUI requires static artifacts that are generated from the NOMAD Python codes.
```
nomad dev metainfo > gui/src/metainfo.json
nomad dev searchQuantities > gui/src/searchQuantities.json
124
nomad dev units > gui/src/units.js
Markus Scheidgen's avatar
Setup    
Markus Scheidgen committed
125
126
127
./gitinfo.sh
```

Markus Scheidgen's avatar
Markus Scheidgen committed
128
129
130
In additional, you have to do some more steps to prepare your working copy to run all
the tests. See below.

Markus Scheidgen's avatar
Markus Scheidgen committed
131
132
133
## Build and run the infrastructure with docker

### Docker and nomad
134
Nomad depends on a set of databases, search engines, and other services. Those
Markus Scheidgen's avatar
Markus Scheidgen committed
135
136
137
138
must run to make use of nomad. We use *docker* and *docker-compose* to create a
unified environment that is easy to build and to run.

You can use *docker* to run all necessary 3rd-party components and run all nomad
139
140
services manually from your python environment. You can also run nomad in docker,
but using Python is often preferred during development, since it allows
Markus Scheidgen's avatar
Markus Scheidgen committed
141
you change things, debug, and re-run things quickly. The later one brings you
142
143
closer to the environment that will be used to run nomad in production. For
development we recommend to skip the next step.
144

Markus Scheidgen's avatar
Markus Scheidgen committed
145
### Docker images for nomad
146
Nomad comprises currently two services,
147
the *worker* (does the actual processing), and the *app*. Those services can be
Markus Scheidgen's avatar
Markus Scheidgen committed
148
run from one image that have the nomad python code and all dependencies installed. This
149
150
is covered by the `Dockerfile` in the root directory
of the nomad sources. The gui is served also served from the *app* which entails the react-js frontend code.
151

152
Before building the image, make sure to execute
153
154
155
```
./gitinfo.sh
```
156
This allows the app to present some information about the current git revision without
157
having to copy the git itself to the docker build context.
Markus Scheidgen's avatar
Markus Scheidgen committed
158

159
### Run necessary 3-rd party services with docker-compose
Markus Scheidgen's avatar
Markus Scheidgen committed
160

161
You can run all containers with:
162
```
163
cd ops/docker-compose/infrastructure
164
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq
165
```
166

Markus Scheidgen's avatar
Markus Scheidgen committed
167
168
169
170
171
172
173
To shut down everything, just `ctrl-c` the running output. If you started everything
in *deamon* mode (`-d`) use:
```
docker-compose down
```

Usually these services only used by the nomad containers, but sometimes you also
174
need to check something or do some manual steps.
Markus Scheidgen's avatar
Markus Scheidgen committed
175

176
177
178
179
The *docker-compose* can be overriden with additional seetings. See documentation section on
operating NOMAD for more details. The override `docker-compose.override.yml` will
expose all database ports to the hostmachine and should be used in development. To use
it run docker-compose with `-f docker-compose.yml -f docker-compose.override.yml`.
Markus Scheidgen's avatar
Markus Scheidgen committed
180
181
182

### ELK (elastic stack)

183
184
If you run the ELK stack (and enable logstash in nomad/config.py),
you can reach the Kibana with [localhost:5601](http://localhost:5601).
185
186
The index prefix for logs is `logstash-`. The ELK is only available with the
`docker-compose.dev-elk.yml` override.
187

Markus Scheidgen's avatar
Markus Scheidgen committed
188
### mongodb and elastic search
Markus Scheidgen's avatar
Markus Scheidgen committed
189

190
You can access mongodb and elastic search via your preferred tools. Just make sure
Markus Scheidgen's avatar
Markus Scheidgen committed
191
192
to use the right ports (see above).

193
## Run nomad services
Markus Scheidgen's avatar
Markus Scheidgen committed
194

195
### API and worker
196

197
198
To simply run a worker with the installed nomad cli, do (from the root)
```
199
nomad admin run worker
200
201
```

202
To run it directly with celery, do (from the root)
203
204
205
206
```
celery -A nomad.processing worker -l info
```

207
You can also run worker and app together:
208
```
209
nomad admin run appworker
210
211
```

212
### GUI
213
When you run the gui on its own (e.g. with react dev server below), you have to have
Markus Scheidgen's avatar
Markus Scheidgen committed
214
the API running manually also. This *inside docker* API is configured for ngingx paths
215
216
and proxies, which are run by the gui container. But you can run the *production* gui
in docker and the dev server gui in parallel with an API in docker.
217
218
219
220
221
222
223
Either with docker, or:
```
cd gui
yarn
yarn start
```

Markus Scheidgen's avatar
Markus Scheidgen committed
224
## Run the tests
Markus Scheidgen's avatar
Markus Scheidgen committed
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251

### additional settings and artifacts
To run the tests some additional settings and files are necessary that are not part
of the code base.

First you need to create a `nomad.yaml` with the admin password for the user management
system:
```
keycloak:
  password: <the-password>
```

Secondly, you need to provide the `springer.msg` Springer materials database. It can
be copied from `/nomad/fairdi/db/data/springer.msg` on our servers and should
be placed at `nomad/normalizing/data/springer.msg`.

Thirdly, you have to provide static files to serve the docs and NOMAD distribution:
```
cd docs
make html
cd ..
python setup.py compile
python setup.py sdist
cp dist/nomad-lab-*.tar.gz dist/nomad-lab.tar.gz
```

### run the necessary infrastructure
252
You need to have the infrastructure partially running: elastic, rabbitmq.
253
The rest should be mocked or provided by the tests. Make sure that you do no run any
254
worker, as they will fight for tasks in the queue.
255
```
Markus Scheidgen's avatar
Markus Scheidgen committed
256
cd ops/docker-compose
257
docker-compose up -d elastic rabbitmq
Markus Scheidgen's avatar
Markus Scheidgen committed
258
259
cd ../..
pytest -svx tests
260
```
Markus Scheidgen's avatar
Markus Scheidgen committed
261
262
263

We use pylint, pycodestyle, and mypy to ensure code quality. To run those:
```
264
nomad dev qa --skip-test
Markus Scheidgen's avatar
Markus Scheidgen committed
265
266
267
268
```

To run all tests and code qa:
```
269
nomad dev qa
Markus Scheidgen's avatar
Markus Scheidgen committed
270
271
272
```

This mimiques the tests and checks that the GitLab CI/CD will perform.
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299


## Setup your (I)DE

The documentation section on development guidelines details how the code is organized,
tested, formatted, and documented. To help you meet these guidelines, we recomment to
use a proper IDE for development and ditch any VIM/Emacs (mal-)practices.

### Visual Studio Code

Here are some VSCode settings that will enable features for linting, some auto formating,
line size ruler, etc.
```json
{
    "python.venvPath": "${workspaceFolder}/.pyenv",
    "python.pythonPath": "${workspaceFolder}/.pyenv/bin/python",
    "git.ignoreLimitWarning": true,
    "editor.rulers": [90],
    "editor.renderWhitespace": "all",
    "editor.tabSize": 4,
    "[javascript]": {
        "editor.tabSize": 2
    },
    "files.trimTrailingWhitespace": true,
    "git.enableSmartCommit": true,
    "eslint.autoFixOnSave": true,
    "python.linting.pylintArgs": [
300
        "--load-plugins=pylint_mongoengine,nomad/metainfo/pylint_plugin",
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
    ],
    "python.linting.pep8Path": "pycodestyle",
    "python.linting.pep8Enabled": true,
    "python.linting.pep8Args": ["--ignore=E501,E701"],
    "python.linting.mypyEnabled": true,
    "python.linting.mypyArgs": [
        "--ignore-missing-imports",
        "--follow-imports=silent",
        "--no-strict-optional"
    ],
    "workbench.colorCustomizations": {
        "editorError.foreground": "#FF2222",
        "editorOverviewRuler.errorForeground": "#FF2222",
        "editorWarning.foreground": "#FF5500",
        "editorOverviewRuler.warningForeground": "#FF5500",
        "activityBar.background": "#4D2111",
        "titleBar.activeBackground": "#6B2E18",
        "titleBar.activeForeground": "#FDF9F7"
    },
    "files.watcherExclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/*/**": true,
        "**/.pyenv/*/**": true,
        "**/__pycache__/*/**": true,
        "**/.mypy_cache/*/**": true,
        "**/.volumes/*/**": true,
        "**/docs/.build/*/**": true
    }
}
```

Here are some example launch configs for VSCode:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "chrome",
      "request": "launch",
      "name": "Launch Chrome against localhost",
      "url": "http://localhost:3000",
      "webRoot": "${workspaceFolder}/gui"
    },
    {
      "name": "Python: API Flask (0.11.x or later)",
      "type": "python",
      "request": "launch",
      "module": "flask",
      "env": {
352
        "FLASK_APP": "nomad/app/__init__.py"
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
      },
      "args": [
        "run",
        "--port",
        "8000",
        "--no-debugger",
        "--no-reload"
      ]
    },
    {
      "name": "Python: some test",
      "type": "python",
      "request": "launch",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/.pyenv/bin/pytest",
      "args": [
        "-sv",
        "tests/test_cli.py::TestClient::test_mirror"
      ]
    },
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}"
    },
    {
      "name": "Python: Attach",
      "type": "python",
      "request": "attach",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "${workspaceFolder}",
      "port": 3000,
      "secret": "my_secret",
      "host": "localhost"
    }
  ]
}
391
```