Single-user Jupyter notebook image for analytics

Please note that this documentation is just a draft. Some of the instructions may be incomplete.

This notebook folder contains Dockerfile for a single user notebook image which based on Jupyter docker stack. In addition of the base notebook it contains QUIP with GAP and all of the necessary packages for the tools. The docker image is suitable for running/testing locally and for deploying it by the jupyterhub on a kubernetes cluster.

Short description

• Based on the jupyter/minimal-notebook docker image
• conda: python 2 and 3 environments
• compiled quip with gap

Tutorials

Each tutorial could have it's own repository. For practical reason please use "analytics-" as prefix for your tutorial

You can add a new repository to the tutorial folder like:

cd notebook/tutorials
git submodule add https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-grain-boundaries.git grain-boundaries

Link for the old tutorials:

• https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-lab-base/tree/master/analysis-tools
• https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-toolkit-tutorials

Building/using your own docker image

Building the docker image locally

docker build -t analytics-notebook:latest notebook

Testing/Developing the notebooks locally

• Use the following command to run the docker image locally:

  docker run --rm \
             -p 8888:8888 \
             -v $PWD/notebook/tutorials:/home/jovyan/tutorials \
             --name analytics-notebook \
             analytics-notebook:latest 

• To have a password-less sudo access in the container use the following command:

   docker exec -it --user root analytics-notebook start.sh

  more info: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html?highlight=root#alternative-commands

Useful tricks for Linux

• avoiding tokens for development

  docker run --rm \
    -p 8889:8888 \
    -v $PWD/notebook/tutorials:/home/jovyan/tutorials \
    --name analytics-notebook \
    analytics-notebook:latest \
      start-notebook.sh \
        --NotebookApp.token=''

• you may consider to use the '--rm' option which deletes the container when it stop. Be careful in this way you will loose everything which was not stored into the mounted folder.

  docker run \
         --rm \
         -p 8888:8888  \
         -v $PWD/notebook/tutorials:/home/jovyan/tutorials  \
         --name analytics-notebook \
         analytics-notebook:latest

• For mounting a folder, you may need to use absolute path or other tricks if the PWD environmental variable is not accessible in your shell:

  docker run \
         --rm \
         -p 8888:8888  \
         -v /path/for/the/tutorials:/home/jovyan/tutorials  \
         --name analytics-notebook \
         analytics-notebook:latest

• you may need to change the user id in the container - by adding "-e NB_UID=1001" and "--user root" options to your command - to have access for the mounted folders:

  docker run \
         --rm \
         -p 8888:8888  \
         -v $PWD/notebook/tutorials:/home/jovyan/tutorials  \
         --user root \
         -e NB_UID=1001 \
         --name analytics-notebook \
         analytics-notebook:latest

More information about the command line options: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#notebook-options

Continuous integration

Each commit triggers a build process on GitLab Runner. Beside the latest tag there will be a unique tag (same that as the value of the git commit) available for explicitly tracking the version of the notebook for a cluster deployment.

Using the docker image from the registry

1. Install docker on your machine

2. Login to the image repository

   docker login gitlab-registry.mpcdf.mpg.de 

3. Pull the image:

   docker pull gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics-jupyterhub/notebook:<tag> 

4. Create a container:

   docker run -p 8888:8888 \
              -v $PWD/notebook/tutorials:/home/jovyan/tutorials \
              gitlab-registry.mpcdf.mpg.de/nomad-lab/analytics-jupyterhub/notebook:<tag>

Note: The latest tag can be find on the following page: https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-jupyterhub/container_registry

Jupyterhub deployment

This part is based on excellent tutorial (https://zero-to-jupyterhub.readthedocs.io/en/stable/index.html). All the default values are available on the following link: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/0.8.0/jupyterhub/values.yaml

Goals

Features:

• single user requirements (1-4 CPU, 1-2 GB RAM, 1 GB home space)
• custom docker single user image (which contains all the necessary packages)
• reloading the tutorials for each users

Continuous integration/delivery

Sensitive parameters can be passed as environmental variables. All of these variables can be stored in the settings (More Settings > CI / CD) of GitLab. In the config file the keys with empty values are necessary and usually they are defined as environmental variables and applied by the helm command.

Tips and tricks

1. URL forwarding: You can use "/hub/user-redirect/" to access the tutorials without explicitly defining the username. eg: https://labdev-nomad.esc.rzg.mpg.de/jupyterhub/hub/user-redirect/tree/tutorials