Skip to content
Snippets Groups Projects
New look: Off

Paraprobe in a container

    • Closed Issue created by Sandor Brockhauser @sanbrock

    Designs

      Child items ...

      2. Activity

      • Sandor Brockhauser added Backlog label

        added Backlog label

      • Sandor Brockhauser changed time estimate to 1d

        changed time estimate to 1d

      • Sandor Brockhauser added 1 deleted label and removed Backlog label

        added 1 deleted label and removed Backlog label

      • Sandor Brockhauser added In Progress label and removed 1 deleted label

        added In Progress label and removed 1 deleted label

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        update: ubuntu-based container with all dependencies were build, paraprobe tools were configured and compiled, resulted in successful image, user management was modified to execute the container not as root, example apt dataset was loaded into the container and test analyses executed. tests executed the tools successfully using one mpi process and eight OpenMP threads, file clean up steps were taken, resulted in 3.6 gb container size, container export into file tested, resulted in 1.4 gb gz-compressed size

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        Source code of the docker container for first review by Sherjeel, Docker file follows

        Edited by Markus Kuehbach
      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        This is an example of a Dockerfile mostly relying on Linux cmds

        Ubuntu 20.04 LTS as base operation system

        FROM ubuntu:focal

        Do not enable interactive queries

        ENV DEBIAN_FRONTEND noninteractive

        ENV TZ=Europe/Berlin RUN apt-get update RUN apt-get install -y tzdata RUN apt-get install m4 -y RUN apt-get install file -y RUN apt-get install unzip -y RUN apt-get install vim -y RUN apt-get install wget -y

        Install GNU C/C++/Fortran compiler and make

        RUN apt-get install build-essential -y && gcc --version && g++ --version && make -v

        Install Message Passing Interface library

        RUN apt-get install mpich -y RUN apt-get install libmpich-dev RUN mpiexec --version && apt-get clean

        Use this version to compile a new cmake from scratch

        RUN mkdir -p code/thirdparty/mandatory WORKDIR code/thirdparty/mandatory RUN mkdir -p cmake WORKDIR cmake RUN wget https://cmake.org/files/v3.19/cmake-3.19.4.tar.gz && tar -xvf cmake-3.19.4.tar.gz WORKDIR cmake-3.19.4 RUN mkdir -p localinstallation && chmod +x bootstrap RUN ./bootstrap --prefix=localinstallation/ -- -DCMAKE_BUILD_TYPE=Release -DCMAKE_USE_OPENSSL=OFF RUN make -j8 && make install ENV PATH="/code/thirdparty/mandatory/cmake/cmake-3.19.4/localinstallation/bin/:${PATH}" RUN cmake --version WORKDIR /code/thirdparty/mandatory/cmake RUN rm cmake-3.19.4.tar.gz WORKDIR /code/thirdparty/mandatory RUN pwd

        Get and install HDF5

        RUN mkdir -p hdf5 #often the tar.gz archives from the HDF5 group are double gz compressed, quite cumbersome, therefore use the gzip version shipped with paraprobe COPY code/thirdparty/mandatory/hdf5/CMake-hdf5-1.12.1.zip hdf5 WORKDIR hdf5 RUN unzip CMake-hdf5-1.12.1.zip WORKDIR CMake-hdf5-1.12.1

        download link for now seems cumbersome to resolve for me RUN wget https://www.hdfgroup.org/package/cmake-hdf5-1-12-1-tar-gz

        RUN ./build-unix.sh && ./HDF5-1.12.1-Linux.sh --include-subdir --skip-license
        WORKDIR /code/thirdparty/mandatory/hdf5 RUN rm CMake-hdf5-1.12.1.zip WORKDIR /code/thirdparty/mandatory

        Install GNU multi-precision arithmetic library (GMP), alternatively for apt-get install libgmp3-dev

        RUN mkdir -p gmp WORKDIR gmp RUN wget https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz && tar -xvf gmp-6.2.1.tar.xz WORKDIR gmp-6.2.1 RUN mkdir -p localinstallation RUN ./configure && make -j8 && make check && make install ENV PATH="/code/thirdparty/mandatory/gmp/gmp-6.2.1/:${PATH}" WORKDIR /code/thirdparty/mandatory/gmp RUN rm gmp-6.2.1.tar.xz WORKDIR /code/thirdparty/mandatory

        Install MPFR

        RUN mkdir -p mpfr WORKDIR mpfr RUN wget https://www.mpfr.org/mpfr-current/mpfr-4.1.0.tar.xz && tar -xvf mpfr-4.1.0.tar.xz WORKDIR mpfr-4.1.0 RUN mkdir -p localinstallation RUN ./configure --with-gmp-build=/code/thirdparty/mandatory/gmp/gmp-6.2.1 && make -j8 && make check && make install #RUN make check-gmp-symbols ENV PATH="/code/thirdparty/mandatory/mpfr/mpfr-4.1.0/:${PATH}" WORKDIR /code/thirdparty/mandatory/mpfr RUN rm mpfr-4.1.0.tar.xz WORKDIR /code/thirdparty/mandatory

        Get and compile TetGen library, compile (by default in TetGen makefile CXX set to use GNU)

        RUN mkdir -p tetgen #TetGen has a complicated license issue so we should better take the version from the paraprobe repository COPY code/thirdparty/mandatory/tetgen/tetgen1.6.0.tar.gz tetgen WORKDIR tetgen RUN tar -xvf tetgen1.6.0.tar.gz WORKDIR tetgen1.6.0 RUN make -j8 tetlib #add to path WORKDIR /code/thirdparty/mandatory

        Get FFTW fast Fourier transform library

        RUN mkdir -p fftw WORKDIR fftw RUN wget https://www.fftw.org/fftw-3.3.10.tar.gz && tar -xvf fftw-3.3.10.tar.gz WORKDIR fftw-3.3.10 RUN ./configure --enable-float --enable-sse && make -j8 && make check && make install WORKDIR /code/thirdparty/mandatory/fftw RUN rm fftw-3.3.10.tar.gz ##add to path WORKDIR /code/thirdparty/mandatory

        GMP and MPFR get installed in weird locations likely this is related to the resolving of local paths, etx

        it is super relevant to check the result of the make install scripts

        Get source code of Voro++ tessellation library

        RUN mkdir -p voroxx #Voro++ is no longer maintained and the download link cumbersome so take code from paraprobe repository COPY code/thirdparty/mandatory/voroxx/voro++-0.4.6.tar.xz voroxx WORKDIR voroxx RUN tar -xvf voro++-0.4.6.tar.xz WORKDIR /code/thirdparty/mandatory/voroxx RUN rm voro++-0.4.6.tar.xz #add to path WORKDIR /code/thirdparty/mandatory

        Install boost

        RUN mkdir -p boost WORKDIR boost RUN wget https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.gz && tar -xvf boost_1_77_0.tar.gz WORKDIR boost_1_77_0 ENV PATH="/code/thirdparty/mandatory/boost/boost_1_77_0/:${PATH}" WORKDIR /code/thirdparty/mandatory/boost RUN rm boost_1_77_0.tar.gz WORKDIR /code/thirdparty/mandatory

        Install eigen

        RUN mkdir -p eigen WORKDIR eigen RUN wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz && tar -xvf eigen-3.4.0.tar.gz WORKDIR eigen-3.4.0 ENV PATH="/code/thirdparty/mandatory/eigen/eigen-3.4.0/:${PATH}" WORKDIR /code/thirdparty/mandatory/eigen RUN rm eigen-3.4.0.tar.gz WORKDIR /code/thirdparty/mandatory

        Get computational geometry algorithms library

        RUN mkdir -p cgal WORKDIR cgal RUN wget https://github.com/CGAL/cgal/releases/download/v5.3/CGAL-5.3.tar.xz && tar -xvf CGAL-5.3.tar.xz #remove maybe the examples to reduce the size of the image WORKDIR /code/thirdparty/mandatory/cgal RUN rm CGAL-5.3.tar.xz #add to path WORKDIR /code/thirdparty/mandatory

        Unpack Lewiner2002, and additional source code portions marching cubes code common files

        RUN mkdir -p lewiner2002/marching_cubes_jgt/MarchingCubes COPY code/thirdparty/mandatory/lewiner2002/marching_cubes_jgt/MarchingCubes/* lewiner2002/marching_cubes_jgt/MarchingCubes/ #add to path RUN mkdir -p hdf5cxxwrapper/src/cxx COPY code/thirdparty/mandatory/hdf5cxxwrapper/src/cxx/* hdf5cxxwrapper/src/cxx/ #add to path RUN mkdir -p hornus2017/intersection-detection-master COPY code/thirdparty/mandatory/hornus2017/intersection-detection-master/* hornus2017/intersection-detection-master/ #add to path WORKDIR /code/thirdparty/mandatory

        This concludes the installation and configuration of prerequisities, now we can start compiling paraprobe from its source code

        WORKDIR /code RUN ls

        paraprobe-utils tool, copy source code in there, replace later a pull from the GitLab repo of the docker branch

        RUN pwd COPY code/PARAPROBE.ExternalLibraries.cmake . RUN ls .cmake RUN mkdir -p paraprobe-utils/build COPY code/paraprobe-utils/CMakeLists.txt paraprobe-utils/ COPY code/paraprobe-utils/src/ paraprobe-utils/src/cxx/ WORKDIR paraprobe-utils/build RUN cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-ranger tool

        RUN mkdir -p paraprobe-ranger/build COPY code/paraprobe-ranger/CMakeLists.txt paraprobe-ranger/ COPY code/paraprobe-ranger/src/* paraprobe-ranger/src/cxx/ WORKDIR paraprobe-ranger/build RUN cmake -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-surfacer tool

        RUN mkdir -p paraprobe-surfacer/build COPY code/paraprobe-surfacer/CMakeLists.txt paraprobe-surfacer/ COPY code/paraprobe-surfacer/src/* paraprobe-surfacer/src/cxx/ WORKDIR paraprobe-surfacer/build RUN cmake -DBOOST_ROOT=/code/thirdparty/mandatory/boost/boost_1_77_0 -DEIGEN3_INCLUDE_DIR=/code/thirdparty/mandatory/eigen/eigen-3.4.0 -DCGAL_DIR=/code/thirdparty/mandatory/cgal/CGAL-5.3 -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-distancer tool

        RUN mkdir -p paraprobe-distancer/build COPY code/paraprobe-distancer/CMakeLists.txt paraprobe-distancer/ COPY code/paraprobe-distancer/src/* paraprobe-distancer/src/cxx/ WORKDIR paraprobe-distancer/build RUN cmake -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-tessellator tool

        RUN mkdir -p paraprobe-tessellator/build COPY code/paraprobe-tessellator/CMakeLists.txt paraprobe-tessellator/ COPY code/paraprobe-tessellator/src/* paraprobe-tessellator/src/cxx/ WORKDIR paraprobe-tessellator/build RUN cmake -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-nanochem tool

        RUN mkdir -p paraprobe-nanochem/build COPY code/paraprobe-nanochem/CMakeLists.txt paraprobe-nanochem/ COPY code/paraprobe-nanochem/src/* paraprobe-nanochem/src/cxx/ WORKDIR paraprobe-nanochem/build RUN cmake -DBOOST_ROOT=/code/thirdparty/mandatory/boost/boost_1_77_0 -DEIGEN3_INCLUDE_DIR=/code/thirdparty/mandatory/eigen/eigen-3.4.0 -DCGAL_DIR=/code/thirdparty/mandatory/cgal/CGAL-5.3 -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-intersector tool

        RUN mkdir -p paraprobe-intersector/build COPY code/paraprobe-intersector/CMakeLists.txt paraprobe-intersector/ COPY code/paraprobe-intersector/src/* paraprobe-intersector/src/cxx/ WORKDIR paraprobe-intersector/build RUN cmake -DBOOST_ROOT=/code/thirdparty/mandatory/boost/boost_1_77_0 -DEIGEN3_INCLUDE_DIR=/code/thirdparty/mandatory/eigen/eigen-3.4.0 -DCGAL_DIR=/code/thirdparty/mandatory/cgal/CGAL-5.3 -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-crystalstructure tool

        RUN mkdir -p paraprobe-crystalstructure/build COPY code/paraprobe-crystalstructure/CMakeLists.txt paraprobe-crystalstructure/ COPY code/paraprobe-crystalstructure/src/* paraprobe-crystalstructure/src/cxx/ WORKDIR paraprobe-crystalstructure/build RUN cmake -DBOOST_ROOT=/code/thirdparty/mandatory/boost/boost_1_77_0 -DMYGITSHA=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=mpicxx .. && make -j8 WORKDIR /code

        paraprobe-spatstat tool

        to be added when properly tested

        prep a directory structure to perform execution test and example data

        WORKDIR / RUN mkdir -p research/database/usa_portland_wang COPY PARAPROBE.Transcoder.Results.SimID.0.zip research/database/usa_portland_wang/

        copy created executables into example analysis folder

        RUN mv /code/paraprobe-ranger/build/paraprobe_ranger research/database/usa_portland_wang/ &&
        mv /code/paraprobe-surfacer/build/paraprobe_surfacer research/database/usa_portland_wang/ &&
        mv /code/paraprobe-distancer/build/paraprobe_distancer research/database/usa_portland_wang/ &&
        mv /code/paraprobe-tessellator/build/paraprobe_tessellator research/database/usa_portland_wang/ &&
        mv /code/paraprobe-nanochem/build/paraprobe_nanochem research/database/usa_portland_wang/ &&
        mv /code/paraprobe-intersector/build/paraprobe_intersector research/database/usa_portland_wang/ &&
        mv /code/paraprobe-crystalstructure/build/paraprobe_crystalstructure research/database/usa_portland_wang/

        create a user so that the container is not running as root upon usage

        RUN groupadd -g 1001 aptuser && useradd -r -u 1001 -g aptuser atomprober WORKDIR /research/database RUN chown -cR atomprober * WORKDIR usa_portland_wang RUN chmod +x paraprobe_*

        USER atomprober

        ENV DEBIAN_FRONTEND newt

        #ENTRYPOINT ["mpirun", "-n", "1", "paraprobe"] #default parameters!!!! #CMD define parameters to what user to possibly change #docker run --> would run #CMD ["Config.h5", "Input.h5"] #appending additional commandds default parameters!!!! #pseudocode: in case docker run $entrypoint + $cmd #docker run MyAwesomeConfig.h5 MyAwesomeInput.h5

      • Markus Kuehbach assigned to @sshabih

        assigned to @sshabih

      • Markus Kuehbach assigned to @kuehbachm and unassigned @sshabih

        assigned to @kuehbachm and unassigned @sshabih

      • Markus Kuehbach added Review label and removed In Progress label

        added Review label and removed In Progress label

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        Pushed public tools from the paraprobe toolbox and corresponding tools to gitlab and modified the docker build recipe such that it is now fully self-contained, i.e. no files have to be transferred from my local computer, currently testing it, will then share the build recipe with sherjeel so that he can test it

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        Dockerfile

        This draft can be reviewed

        Edited by Markus Kuehbach
      • Markus Kuehbach created merge request !26 (closed) to address this issue

        created merge request !26 (closed) to address this issue

      • Markus Kuehbach mentioned in merge request !26 (closed)

        mentioned in merge request !26 (closed)

      • Sherjeel Shabih
        Sherjeel Shabih @sshabih ·
        Developer

        I had a talk with Markus about this. If his tools are to be used from within a Jupyter notebook, then the best way will be to base the image off something like: jupyter/datascience-notebook. Then add the C++ tools on top of that. This will make them available to the user from the jupyter notebook without using webtop or anything like that.

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        I have modified the dockerfile for paraprobe using now successfully the webtop ubuntu, also all versions for the apt-get and other tools were added. I will update the repository during the weekend so that i can test to hook into Sherjeels buttons in north

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        Please check merge request for branch markus-link-paraprobe-with-north. In my history and displayed branches it is unclear where Sherjeel added the buttons for paraprobe and pyARPES, the last slack message

        Friday, October 29th Sandor Brockhauser 1:20 PM @Sherjeel Shabih can you add buttons for pyARPES and Paraprobe images when @m.kuehbach finalizes them. And also add jupyterHUB links behind in a different branch (or under a new button)

        Sherjeel Shabih 1:21 PM Yeah. I can do those.

        Sandor Brockhauser 1:22 PM Great I will be able to test them after the Coordinator's meeting...

        m.kuehbach 1:26 PM as I am anyway implementing right now the spec arch and specific versions in these build recipes, I will test to use webtop ubuntu for paraprobe instead of focal in this way we can be sure that we have jupyter and conda installable, which we would need for paraprobe as well

        Sandor Brockhauser 2:03 PM We just finished the meeting. Please just slack me if I can test something...

        Sandor Brockhauser 4:58 PM @m.kuehbach: I have done the arpes move, so the example data with ipynb-s are finalised for all techniques in this sprint. What is the status with the docker images? I have seen that you got the commit tag for the pyarpes source and I have accepted it...

      • Markus Kuehbach
        Markus Kuehbach @kuehbachm ·
        Developer

        Would be happy to pick up on this Monday morning again after the stand-up maybe with after a short discussion with Sherjeel, I am confined it is now only the marriage between the guacd showing jupyter-lab with the right files and rights and have a button for this in north

      • Markus Kuehbach closed

        closed

      • Markus Kuehbach reopened

        reopened

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply