... | ... | @@ -8,6 +8,378 @@ Technical details going beyond this can be found here. This is supplementary inf |
|
|
... submitted report will be downloadable here as soon as submitted
|
|
|
|
|
|
# Infrastructure setup
|
|
|
Connecting to NOMAD virtual machine
|
|
|
-----------------------------------
|
|
|
|
|
|
~~~
|
|
|
$ ssh USER@con01.rzg.mpg.de
|
|
|
$ ssh root@enc-devel-nomad
|
|
|
~~~
|
|
|
|
|
|
Add a new user
|
|
|
--------------
|
|
|
|
|
|
- the new user will have access to our Nomad database and run the python application:
|
|
|
|
|
|
**user**: nomadapp **pass**: NoMaD_App
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # useradd -m nomadapp
|
|
|
enc-devel-nomad:~ # passwd nomadapp
|
|
|
~~~
|
|
|
|
|
|
- test to see if user was created and we can login
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # su - nomadapp
|
|
|
nomadapp@enc-devel-nomad:~> whoami
|
|
|
nomadapp
|
|
|
nomadapp@enc-devel-nomad:~> pwd
|
|
|
/home/nomadapp
|
|
|
nomadapp@enc-devel-nomad:~>
|
|
|
~~~
|
|
|
|
|
|
|
|
|
PostgreSQL installation:
|
|
|
------------------------
|
|
|
|
|
|
|
|
|
- let's install the necessary packages
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # zypper install postgresql94-server postgresql94 postgresql94-contrib
|
|
|
Loading repository data...
|
|
|
Reading installed packages...
|
|
|
Resolving package dependencies...
|
|
|
|
|
|
The following 5 NEW packages are going to be installed:
|
|
|
libpq5 postgresql-init postgresql94 postgresql94-contrib postgresql94-server
|
|
|
|
|
|
5 new packages to install.
|
|
|
Overall download size: 5.2 MiB. Already cached: 0 B. After the operation, additional 23.2 MiB will be used.
|
|
|
Continue? [y/n/? shows all options] (y): y
|
|
|
[ ... ]
|
|
|
enc-devel-nomad:~ #
|
|
|
~~~
|
|
|
|
|
|
- check to see if the packages are installed and running properly
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # postgres --version
|
|
|
postgres (PostgreSQL) 9.4.6
|
|
|
~~~
|
|
|
|
|
|
- let's start PostgreSQL server once and check if its status
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # service postgresql start
|
|
|
enc-devel-nomad:~ # service postgresql status
|
|
|
postgresql.service - LSB: Start the PostgreSQL master daemon
|
|
|
Loaded: loaded (/etc/init.d/postgresql)
|
|
|
Active: active (exited) since Fri 2016-03-04 16:25:29 CET; 6s ago
|
|
|
Process: 29987 ExecStart=/etc/init.d/postgresql start (code=exited, status=0/SUCCESS)
|
|
|
~~~
|
|
|
|
|
|
- enable PostgreSQL to start at boot time (after a restart):
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # chkconfig postgresql on
|
|
|
enc-devel-nomad:~ # chkconfig --list
|
|
|
postgresql 0:off 1:off 2:off 3:on 4:off 5:on 6:off
|
|
|
enc-devel-nomad:~ # systemctl is-enabled postgresql
|
|
|
~~~
|
|
|
|
|
|
- a *postgres* user is automatically created when the packages were installed, so, let's see if is working
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # su - postgres
|
|
|
postgres@enc-devel-nomad:~> whoami
|
|
|
postgres
|
|
|
postgres@enc-devel-nomad:~> psql
|
|
|
psql (9.4.6)
|
|
|
Type "help" for help.
|
|
|
|
|
|
postgres=# \conninfo
|
|
|
You are connected to database "postgres" as user "postgres" via socket in "/var/run/postgresql" at port "5432".
|
|
|
postgres=#
|
|
|
|
|
|
postgres-# \l
|
|
|
List of databases
|
|
|
Name | Owner | Encoding | Collate | Ctype | Access privileges
|
|
|
-----------+----------+----------+-------------+-------------+-----------------------
|
|
|
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
|
|
|
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
|
|
|
| | | | | postgres=CTc/postgres
|
|
|
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
|
|
|
| | | | | postgres=CTc/postgres
|
|
|
(3 rows)
|
|
|
|
|
|
postgres-#
|
|
|
postgres=# SELECT * FROM pg_catalog.pg_tables;
|
|
|
schemaname | tablename | tableowner | tablespace | hasindexes | hasrules | hastriggers
|
|
|
--------------------+-------------------------+------------+------------+------------+----------+-------------
|
|
|
pg_catalog | pg_statistic | postgres | | t | f | f
|
|
|
pg_catalog | pg_type | postgres | | t | f | f
|
|
|
|
|
|
postgres-# \q
|
|
|
~~~
|
|
|
|
|
|
|
|
|
- let’s create a new user role for our database, using the same login name as the user created for running python app:
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # su - postgres
|
|
|
postgres@enc-devel-nomad:~> createuser --interactive
|
|
|
Enter name of role to add: nomadapp
|
|
|
Shall the new role be a superuser? (y/n) y
|
|
|
postgres@enc-devel-nomad:~>
|
|
|
~~~
|
|
|
|
|
|
- test the new user/role for database
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # su - nomadapp
|
|
|
nomadapp@enc-devel-nomad:~> psql -d repodump
|
|
|
repodump=# \conninfo
|
|
|
You are connected to database "repodump" as user "nomadapp" via socket in "/var/run/postgresql" at port "5432".
|
|
|
repodump=#
|
|
|
repodump=# \password nomadapp
|
|
|
|
|
|
~~~
|
|
|
**Note:** the password added is *Nomad_Flask*
|
|
|
|
|
|
- in order to have authentifcation enabled for our webapp is needed to replace *ident* with *md5* in */var/lib/pgsql/data/pg_hba.conf* config file. The important section should look like (after our changes):
|
|
|
|
|
|
~~~
|
|
|
# IPv4 local connections:
|
|
|
host all all 127.0.0.1/32 md5
|
|
|
# IPv6 local connections:
|
|
|
host all all ::1/128 md5
|
|
|
~~~
|
|
|
|
|
|
- and then restarting Postgres is needed to reread the config file
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # service postgresql restart
|
|
|
~~~
|
|
|
|
|
|
**Note:** PostgreSQL LOG files are in: */var/lib/pgsql/data/pg_log/*
|
|
|
|
|
|
|
|
|
Nginx installation
|
|
|
------------------
|
|
|
|
|
|
- Nginx is not present in the default SLES12 repository, we need to add it's repository to our system
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # zypper addrepo -G -t yum -c 'http://nginx.org/packages/sles/12' nginx
|
|
|
Adding repository 'nginx' ......................................................................................[done]
|
|
|
Warning: GPG checking is disabled in configuration of repository 'nginx'. Integrity and origin of packages cannot be verified.
|
|
|
Repository 'nginx' successfully added
|
|
|
Enabled : Yes
|
|
|
Autorefresh : No
|
|
|
GPG Check : No
|
|
|
Priority : 99
|
|
|
URI : http://nginx.org/packages/sles/12
|
|
|
~~~
|
|
|
|
|
|
- let's install the package
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # zypper install nginx
|
|
|
~~~
|
|
|
|
|
|
- start the webserver once, to check if it is installed and working
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # systemctl start nginx
|
|
|
enc-devel-nomad:~ # systemctl status nginx
|
|
|
nginx.service - nginx - high performance web server
|
|
|
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled)
|
|
|
Active: active (running) since Wed 2016-03-09 10:14:21 CET; 2s ago
|
|
|
~~~
|
|
|
|
|
|
- check if Nginx is enabled at boot time
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # systemctl list-unit-files | grep nginx
|
|
|
nginx.service disabled
|
|
|
~~~
|
|
|
|
|
|
- is not enabled, so we have to enable it
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # systemctl enable nginx
|
|
|
ln -s '/usr/lib/systemd/system/nginx.service' '/etc/systemd/system/multi-user.target.wants/nginx.service'
|
|
|
~~~
|
|
|
|
|
|
- check again if indeed is enabled
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # systemctl list-unit-files | grep nginx
|
|
|
nginx.service enabled
|
|
|
~~~
|
|
|
|
|
|
- add/prepare a config file for our application
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # cd /etc/nginx/conf.d
|
|
|
enc-devel-nomad:/etc/nginx/conf.d # pwd
|
|
|
/etc/nginx/conf.d
|
|
|
|
|
|
enc-devel-nomad:/etc/nginx/conf.d # mv default.conf default.conf.orig
|
|
|
|
|
|
enc-devel-nomad:/etc/nginx/conf.d # cat nomadapp.conf
|
|
|
server {
|
|
|
listen 80;
|
|
|
server_name localhost;
|
|
|
|
|
|
location / {
|
|
|
proxy_pass http://127.0.0.1:5000;
|
|
|
}
|
|
|
}
|
|
|
~~~
|
|
|
|
|
|
- restart the webserver to read the new config file
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:/etc/nginx/conf.d # systemctl restart nginx
|
|
|
~~~
|
|
|
|
|
|
- all right, our webserver is up and running
|
|
|
|
|
|
**Note:** The Nginx config file is very minimal but for first Deliverable D2.1 is sufficient
|
|
|
|
|
|
Python and Flask environment setup
|
|
|
----------------------------------
|
|
|
|
|
|
- install the necessary python packages:
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # zypper install python-setuptools
|
|
|
enc-devel-nomad:~ # easy_install pip
|
|
|
enc-devel-nomad:~ # pip install virtualenv
|
|
|
~~~
|
|
|
|
|
|
- in order to use *psycopg2* (to connect to database using python, we don't use SQLAlchemy right now) we have to install *postgresql94-devel* package:
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # zypper install postgresql94-devel
|
|
|
~~~
|
|
|
|
|
|
- let’s install *supervisord* package which will monitor our python app. (not used for Deliverable D2.1)
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # pip install supervisor
|
|
|
Collecting supervisor
|
|
|
Downloading supervisor-3.2.2.tar.gz (410kB)
|
|
|
100% |...........................| 419kB 2.1MB/s
|
|
|
~~~
|
|
|
|
|
|
- let’s generate the supervisor config file and place it in *nomadapp* user’s home folder (we can also put it in /etc/)
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # su - nomadapp
|
|
|
nomadapp@enc-devel-nomad:~> echo_supervisord_conf > supervisord.conf
|
|
|
|
|
|
nomadapp@enc-devel-nomad:~> ls -ltr supervisord.conf
|
|
|
-rw-r--r-- 1 nomadapp users 8658 Mar 9 14:31 supervisord.conf
|
|
|
~~~
|
|
|
|
|
|
|
|
|
- we need to create a virtual environment for python where to install all necessary modules for our app (Flask, Werkzeug, gunicorn, ...) and to run the app in this virtualenv
|
|
|
|
|
|
~~~
|
|
|
nc-devel-nomad:~ # su - nomadapp
|
|
|
nomadapp@enc-devel-nomad:~> pwd
|
|
|
/home/nomadapp
|
|
|
|
|
|
nomadapp@enc-devel-nomad:~> virtualenv nomadvenv
|
|
|
New python executable in /home/nomadapp/nomadvenv/bin/python
|
|
|
Installing setuptools, pip, wheel...done.
|
|
|
|
|
|
nomadapp@enc-devel-nomad:~> source nomadvenv/bin/activate
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~>
|
|
|
~~~
|
|
|
|
|
|
- add the python modules to our virtualenv
|
|
|
|
|
|
~~~
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~> pip install flask gunicorn
|
|
|
Collecting flask
|
|
|
Using cached Flask-0.10.1.tar.gz
|
|
|
Collecting gunicorn
|
|
|
Downloading gunicorn-19.4.5-py2.py3-none-any.whl (112kB)
|
|
|
100% |...........................| 122kB 2.4MB/s
|
|
|
Collecting Werkzeug>=0.7 (from flask)
|
|
|
Downloading Werkzeug-0.11.4-py2.py3-none-any.whl (305kB)
|
|
|
100% |...........................| 307kB 1.9MB/s
|
|
|
Collecting Jinja2>=2.4 (from flask)
|
|
|
Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)
|
|
|
100% |...........................| 266kB 6.1MB/s
|
|
|
Collecting itsdangerous>=0.21 (from flask)
|
|
|
Downloading itsdangerous-0.24.tar.gz (46kB)
|
|
|
100% |...........................| 51kB 10.6MB/s
|
|
|
Collecting MarkupSafe (from Jinja2>=2.4->flask)
|
|
|
Downloading MarkupSafe-0.23.tar.gz
|
|
|
Building wheels for collected packages: flask, itsdangerous, MarkupSafe
|
|
|
Running setup.py bdist_wheel for flask ... done
|
|
|
Stored in directory: /home/nomadapp/.cache/pip/wheels/d2/db/61/cb9b80526b8f3ba89248ec0a29d6da1bb6013681c930fca987
|
|
|
Running setup.py bdist_wheel for itsdangerous ... done
|
|
|
Stored in directory: /home/nomadapp/.cache/pip/wheels/97/c0/b8/b37c320ff57e15f993ba0ac98013eee778920b4a7b3ebae3cf
|
|
|
Running setup.py bdist_wheel for MarkupSafe ... done
|
|
|
Stored in directory: /home/nomadapp/.cache/pip/wheels/94/a7/79/f79a998b64c1281cb99fa9bbd33cfc9b8b5775f438218d17a7
|
|
|
Successfully built flask itsdangerous MarkupSafe
|
|
|
Installing collected packages: Werkzeug, MarkupSafe, Jinja2, itsdangerous, flask, gunicorn
|
|
|
Successfully installed Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.11.4 flask-0.10.1 gunicorn-19.4.5 itsdangerous-0.24
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~>
|
|
|
~~~
|
|
|
|
|
|
- for Deliverable D2.1 we run the python app only using *gunicorn*, without configuring *supervisord*
|
|
|
|
|
|
~~~
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~/app> pwd
|
|
|
/home/nomadapp/app
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~/app> gunicorn -c gunicorn.conf app:app -b localhost:5000
|
|
|
(nomadvenv) nomadapp@enc-devel-nomad:~/app>
|
|
|
~~~
|
|
|
|
|
|
- check if our application is running:
|
|
|
|
|
|
~~~
|
|
|
enc-devel-nomad:~ # ps aux | grep gunicorn
|
|
|
nomadapp 13510 0.0 0.1 93044 13060 ? S 13:31 0:00 /home/nomadapp/nomadvenv/bin/python /home/nomadapp/nomadvenv/bin/gunicorn -c gunicorn.conf app:app -b localhost:5000
|
|
|
nomadapp 13515 0.0 0.2 140168 19588 ? S 13:31 0:00 /home/nomadapp/nomadvenv/bin/python /home/nomadapp/nomadvenv/bin/gunicorn -c gunicorn.conf app:app -b localhost:5000
|
|
|
root 13560 0.0 0.0 9236 928 pts/1 S+ 13:33 0:00 grep --color=auto gunicorn
|
|
|
enc-devel-nomad:~ #
|
|
|
~~~
|
|
|
|
|
|
**All DONE**
|
|
|
|
|
|
**Notes:**
|
|
|
|
|
|
- there is no automatic deployment for miniAPI and miniGUI
|
|
|
- the miniAPI is in */home/nomadapp/app/* and the miniGUI is in */home/nomadapp/app/static/*
|
|
|
- gunicorn's config file:
|
|
|
|
|
|
~~~
|
|
|
daemon = True
|
|
|
accesslog = "logs/gunicorn_access.log"
|
|
|
errorlog = "logs/gunicorn_error.log"
|
|
|
~~~
|
|
|
|
|
|
- nginx config file (really minimal file, *nginx* is acting as a reverse proxy for our python app):
|
|
|
|
|
|
~~~
|
|
|
server {
|
|
|
listen 80;
|
|
|
server_name localhost;
|
|
|
|
|
|
location / {
|
|
|
proxy_pass http://127.0.0.1:5000;
|
|
|
}
|
|
|
}
|
|
|
~~~
|
|
|
|
|
|
# Software implemented
|
|
|
## Mini-API
|
... | ... | |