Skip to content
Snippets Groups Projects
Commit ce2bc7be authored by Luigi's avatar Luigi
Browse files

Resolve conflicts merge cmlkit

parents f3ef5bec e5d24521
No related branches found
No related tags found
2 merge requests!123testing develop merge,!23Various tutorials
Hide whitespace changes
Inline Side-by-side
.gitmodules
+ 6
− 0
View file @ ce2bc7be
......@@ -22,6 +22,12 @@
[submodule "tutorials/analytics-tcmi"]
path = tutorials/analytics-tcmi
url = https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-tcmi.git
[submodule "tutorials/analytics-cmlkit"]
path = tutorials/analytics-cmlkit
url = https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-cmlkit.git
[submodule "3rdparty/qmmlpack"]
path = 3rdparty/qmmlpack
url = https://gitlab.com/qmml/qmmlpack.git
[submodule "tutorials/analytics-query-nomad-repository"]
path = tutorials/analytics-query-nomad-repository
url = https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-query-nomad-repository
......
qmmlpack @ 1c66caff
Subproject commit 1c66caff7dfb1d206e32796692f7381d45d06522
Dockerfile
+ 36
− 4
View file @ ce2bc7be
......@@ -16,6 +16,7 @@ RUN apt-get update \
netcdf-bin \
libxpm-dev \
libgsl-dev \
lsof \
vim \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
......@@ -23,6 +24,7 @@ USER $NB_UID
# Dependecies:
# - quippy: gcc gfortran liblapack-dev libblas-dev libnetcdf-dev netcdf-bin libxpm-dev libgsl-dev
# - qmmlpack: gcc libgsl-dev lsof
# ================================================================================
# Python environment
......@@ -41,6 +43,8 @@ RUN conda install --quiet --yes \
'jupyter_nbextensions_configurator==0.4.1' \
'ase' \
'asap3' \
'dscribe' \
'pytest' \
&& conda clean -tipsy \
&& pip install https://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing-major/dist/nomad-0.8.0.tar.gz \
&& jupyter nbextensions_configurator enable --user \
......@@ -51,6 +55,36 @@ RUN conda install --quiet --yes \
&& fix-permissions $CONDA_DIR \
&& fix-permissions /home/$NB_USER
# Special dependecies:
# - grain-boundaries: asap3
# - qmmlpack: dscribe pytest
# ================================================================================
# QMMLPACK
# ================================================================================
# for the test TMPDIR has to be defined
ENV TMPDIR "/tmp/"
WORKDIR /opt/qmmlpack
COPY 3rdparty/qmmlpack .
USER root
RUN ./make --debug --verbose cpp --include-path /usr/include/gsl/ --library-path /usr/lib/
RUN ./make --debug --verbose python --include-path /usr/include/gsl/ --library-path /usr/lib/
RUN chown -R $NB_USER:$NB_GID $PWD
USER $NB_USER
#RUN make -v install
ENV PYTHONPATH "/opt/qmmlpack/python":$PYTHONPATH
# The CML_PLUGINS environment variable tells cmlkit to register the cscribe plugin, which provides the dscribe interface.
ENV CML_PLUGINS "cscribe"
ENV CML_DATASET_PATH $HOME/tutorials/data/cmlkit/
# Dependecies:
# - ase: many
......@@ -106,16 +140,13 @@ RUN pip install ./analytics-gap-si-surface
RUN pip install ./analytics-grain-boundaries
RUN pip install ./analytics-kaggle-competition
RUN pip install ./analytics-soap-atomic-charges
RUN pip install ./analytics-cmlkit
RUN pip install ./analytics-tcmi
RUN pip install -r ./analytics-query-nomad-repository/requirements.txt
<<<<<<< HEAD
RUN pip install ./analytics-descriptor-role
RUN pip install ./analytics-decision-tree
RUN pip install ./analytics-nn-regression
RUN pip install ./analytics-perovskites-tolerance-factor
=======
>>>>>>> master
USER root
......@@ -144,6 +175,7 @@ RUN mkdir data && ln -s /opt/tutorials/*/data/* .
RUN fix-permissions $TUTORIALS_HOME \
&& fix-permissions $HOME/.local/share/jupyter
# ================================================================================
# Switch back to jovyan to avoid accidental container runs as root
# ================================================================================
......
README.md
+ 60
− 55
View file @ ce2bc7be
# Single-user Jupyter notebook image for analytics
**Please note that this documentation is just a draft. Some of the instructions may be incomplete.**
This notebook folder contains Dockerfile for a single user notebook image which based on Jupyter docker stack.
In addition of the base notebook it contains QUIP with GAP and all of the necessary packages for the tools.
The docker image is suitable for running/testing locally and for deploying it by the jupyterhub on a kubernetes cluster.
## Short description
- Based on the jupyter/minimal-notebook docker image
- conda: python 2 and 3 environments
- compiled quip with gap
## Tutorials
- Based on the [jupyter/tensorflow-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-tensorflow-notebook) docker image
- conda: python 3 environments
- compiled quip with gap, qmmlpack
Each tutorial could have it's own repository. For practical reason please use "analytics-" as prefix for your tutorial
You can add a new repository to the tutorial folder like:
```bash
cd tutorials
git submodule add https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-grain-boundaries.git grain-boundaries
```
Link for the old tutorials:tutorials
- https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-lab-base/tree/master/analysis-tools
- https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-toolkit-tutorials
## Building/using your own docker image
......@@ -46,7 +31,7 @@ git submodule update --init --recursive
### Building the docker image locally
```bash
docker build -t analytics-notebook:latest .
docker build -t analytics:latest .
```
......@@ -59,16 +44,62 @@ docker build -t analytics-notebook:latest .
-e GRANT_SUDO=yes \
--user root \
--name analytics-notebook \
analytics-notebook:latest
analytics:latest
```
- To have bash in the container use the following command:
```bash
docker exec -it analytics-notebook start.sh
docker exec -it analytics start.sh
```
more info: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html?highlight=root#alternative-commands
### Using the docker image from the registry
Beside the latest tag there will be a unique tags (same that as the value of the git commit) available for explicitly tracking the version of the notebook for a cluster deployment. Each PR has its own tag (name of the source branch) which used for staging.
1. Install docker on your machine
2. Login to the image repository
```bash
docker login gitlab-registry.mpcdf.mpg.de
```
3. Pull the image:
```bash
docker pull gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics:latest
```
4. Create a container:
```bash
docker run -p 8888:8888 \
-v $PWD/tutorials:/home/jovyan/tutorials \
gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics:latest
```
Note: The latest tag can be find on the following page:
https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/container_registry
## How to add a new tutorial - Basic workflow
1) Create a branch and add all the necessary changes
2) Create PR to 'master' branch
- CI tries to build the image and deploys staging hub
3) Manually test the staging hub
4) Merge PR to 'master' branch
- CI deploys production hub
### Template
Each tutorial could have it's own repository and stored in the tutorials folder as a submodule:
```bash
cd tutorials
git submodule add https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-{tutorial}.git
```
To create a new tutorial you can use the skeleton project in the `template`folder.
### Useful tricks for Linux
......@@ -77,8 +108,8 @@ docker build -t analytics-notebook:latest .
docker run --rm \
-p 8889:8888 \
-v $PWD/tutorials:/home/jovyan/tutorials \
--name analytics-notebook \
analytics-notebook:latest \
--name analytics \
analytics:latest \
start-notebook.sh \
--NotebookApp.token=''
```
......@@ -89,8 +120,8 @@ docker build -t analytics-notebook:latest .
--rm \
-p 8888:8888 \
-v $PWD/tutorials:/home/jovyan/tutorials \
--name analytics-notebook \
analytics-notebook:latest
--name analytics \
analytics:latest
```
- For mounting a folder, you may need to use absolute path or other tricks if the PWD environmental variable is not accessible in your shell:
......@@ -99,8 +130,8 @@ docker build -t analytics-notebook:latest .
--rm \
-p 8888:8888 \
-v /path/for/the/tutorials:/home/jovyan/tutorials \
--name analytics-notebook \
analytics-notebook:latest
--name analytics \
analytics:latest
```
- you may need to change the user id in the container - by adding "-e NB_UID=1001" and "--user root" options to your command - to have access for the mounted folders:
......@@ -111,36 +142,10 @@ docker build -t analytics-notebook:latest .
-v $PWD/tutorials:/home/jovyan/tutorials \
--user root \
-e NB_UID=1001 \
--name analytics-notebook \
analytics-notebook:latest
--name analytics \
analytics:latest
```
More information about the command line options: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#notebook-options
## Continuous integration
Each commit triggers a build process on GitLab Runner. Beside the latest tag there will be a unique tag (same that as the value of the git commit) available for explicitly tracking the version of the notebook for a cluster deployment.
### Using the docker image from the registry
1. Install docker on your machine
2. Login to the image repository
```bash
docker login gitlab-registry.mpcdf.mpg.de
```
3. Pull the image:
```bash
docker pull gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics:latest
```
4. Create a container:
```bash
docker run -p 8888:8888 \
-v $PWD/tutorials:/home/jovyan/tutorials \
gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics:latest
```
Note: The latest tag can be find on the following page:
https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/container_registry
analytics-cmlkit @ 10780cad
Subproject commit 10780cad1b97e3c1e5e1d8852a97f6f67372d4c1
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment