analytics issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues2022-07-08T20:30:06Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/18Docker images taking too much space2022-07-08T20:30:06ZNicolas FabasDocker images taking too much spaceHi all!
We noticed here at MPCDF that Docker images are taking ~300GB on the container registry of this project and really start to fill up the partition. We are talking about images with name like
'develop' + git commit id
Is there a...Hi all!
We noticed here at MPCDF that Docker images are taking ~300GB on the container registry of this project and really start to fill up the partition. We are talking about images with name like
'develop' + git commit id
Is there a way that you could reassess the usefulness of these images?
Thank you in advance,
Nicolas Fabashttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/29Notebook: NequIP tutorial2023-03-24T00:00:33ZAdam FeketeNotebook: NequIP tutorial@lsbailo, @sshlu, @lucamghi
Repository:
- https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-nequip
Major dependencies:
- LAMMPS - stable release 23/06/22. Install LAMMPS as a shared library using cmake, as described here https://docs.la...@lsbailo, @sshlu, @lucamghi
Repository:
- https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-nequip
Major dependencies:
- LAMMPS - stable release 23/06/22. Install LAMMPS as a shared library using cmake, as described here https://docs.lammps.org/Howto_pylammps.html, to use the 'lammps' Python package.
- torch, nequip, ase, sklearn
Known issue:
- [ ] running lammps using lammps package or as a subprocess
- [x] the major problem is that the pairing NequIP-Lammps package uses Cudahttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/27The cleanup process of container images doesn't work2022-07-08T20:30:06ZAdam FeketeThe cleanup process of container images doesn't workhttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/26Refactoring the infrastructure2022-07-08T13:42:38ZAdam FeketeRefactoring the infrastructurehttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/25Add lint to notebooks2021-11-29T10:55:37ZLuigi SbailoAdd lint to notebooksNotebooks now do not follow any architectureNotebooks now do not follow any architectureLuigi SbailoLuigi Sbailohttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/24Add password to sensitive notebooks2021-08-31T09:16:35ZLuigi SbailoAdd password to sensitive notebooksTo protect sensitive data, notebooks in develop should have the option to be protected with a password.
Submodules storing sensitive notebooks should also be private.
Currently, submodules need to be public repositories.To protect sensitive data, notebooks in develop should have the option to be protected with a password.
Submodules storing sensitive notebooks should also be private.
Currently, submodules need to be public repositories.https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/23Documentation on notebook development roadmap2021-08-30T13:50:54ZLuca Massimiliano GhiringhelliDocumentation on notebook development roadmapIt would be good to have a documentation (written + tutorial video(s)) on the steps to be taken to develop a new tutorial (and update an existing one). This should include
- fully online. Show how to create a new (empty notebook), modif...It would be good to have a documentation (written + tutorial video(s)) on the steps to be taken to develop a new tutorial (and update an existing one). This should include
- fully online. Show how to create a new (empty notebook), modify and save the modified version of an existing notebook. Show also how to manage the personal "Work" directory, (e.g., create a data directory, upload local files, rename, delete, etc.). Show how to go from the files saved in Work to commit the change into gitlab (upload the modified files to gitlab, etc.)
- via docker. For now, assume people can install docker without further help, just document how to use local IDE (e.g., anaconda) for development, docker for preliminary testing, upload to gitlab, etc.
- point out that metainfo.json is also to be created by the developers as part of the development processLuigi SbailoLuigi Sbailohttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/22Update date and other notebook's metadata2021-12-21T14:19:55ZLuca Massimiliano GhiringhelliUpdate date and other notebook's metadataIt would be good to have
- the update date of a notebook (metadata "update" in metainfo.json) appearing in the info box of the notebook (AIToolkit homepage or future list-of-notebooks page). This date should be automatically updated as ...It would be good to have
- the update date of a notebook (metadata "update" in metainfo.json) appearing in the info box of the notebook (AIToolkit homepage or future list-of-notebooks page). This date should be automatically updated as part of continuous integration whenever a new version (from gitlab) of that notebook is committed.
- have title, author (+affiliations), and logos (which can be optional), as appearing in the header of a notebook, treated as metadata, which should correspond to the metadata in metainfo.json. If possible, also the update date should be treated the same way, i.e., it appears in the the notebook' s header but it is consistent with metainfo.json and therefore (see point above) automatically updated.Luigi SbailoLuigi Sbailohttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/21Usage of local data in the AI toolkit2021-08-31T09:16:15ZLuigi SbailoUsage of local data in the AI toolkitThe AI toolkit is currently structured to allow for direct analysis of data retrieved from the Archive, but a straightforward approach to analyze data stored locally is missing.The AI toolkit is currently structured to allow for direct analysis of data retrieved from the Archive, but a straightforward approach to analyze data stored locally is missing.Luigi SbailoLuigi Sbailohttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/20nomad-lab package2021-04-15T06:07:42ZMarkus Scheidgennomad-lab package@lsbailo I hope this is the right project to post this.
We updated NOMAD to a new version. While the archive query notebook still works with the new version, there are two things that should be done.
- The nomad-lab package (old 0.9.1...@lsbailo I hope this is the right project to post this.
We updated NOMAD to a new version. While the archive query notebook still works with the new version, there are two things that should be done.
- The nomad-lab package (old 0.9.10 and new 0.10.1 version) has a wrong default config and users can't authenticate via `ArchiveQuery(username='*', password='*')`. I try to fix this and release a new version ASAP (0.10.2).
- You should try to build an image with the new version of the nomad-lab package (0.10.1) right away, just to see if the install and everything works as expected. I am not sure if and how I should trigger the CI/CD; maybe you can do it?https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/19Docker daemon failing2021-02-16T10:51:18ZLuigi SbailoDocker daemon failingDocker building fails as it can't connects to the Docker daemon.
The message error is the following:
```
Running with gitlab-runner 13.3.1 (738bbe5a)
on fairdi-nomad-analytics-jupyterhub E6vDzAsq
Preparing the "docker" executor
ERROR:...Docker building fails as it can't connects to the Docker daemon.
The message error is the following:
```
Running with gitlab-runner 13.3.1 (738bbe5a)
on fairdi-nomad-analytics-jupyterhub E6vDzAsq
Preparing the "docker" executor
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:978:120s)
Will be retried in 3s ...
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:978:120s)
Will be retried in 3s ...
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:978:120s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:978:120s)
```
It firstly appeared here:
https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/jobs/1260940
where it was not possible to download some packages. After retrying CI cannot even connect to the daemon. Some network problems occurred overnight?Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/17Visualizer improvements2021-02-08T15:11:20ZLuigi SbailoVisualizer improvements- Implement a feature to avoid plotting overlapping points.
- Add structure visualization as optional requirement- Implement a feature to avoid plotting overlapping points.
- Add structure visualization as optional requirementLuigi SbailoLuigi Sbailohttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/16Size of the image is quite big2020-11-12T07:06:42ZAdam FeketeSize of the image is quite bigIt would make sense to try to reduce the size of these images.
You can find some summary of the size of the latest develop image:
```bash
$ du -shL tutorials/data/*
448M tutorials/data/cmlkit
21M tutorials/data/co2
736K tutor...It would make sense to try to reduce the size of these images.
You can find some summary of the size of the latest develop image:
```bash
$ du -shL tutorials/data/*
448M tutorials/data/cmlkit
21M tutorials/data/co2
736K tutorials/data/compressed_sensing
11M tutorials/data/convolutional_nn
0 tutorials/data/data
118M tutorials/data/decision_tree
880K tutorials/data/descriptor_role
109M tutorials/data/domain_of_applicability
123M tutorials/data/error_estimates
180K tutorials/data/exploratory_analysis
137M tutorials/data/grain_boundaries
14M tutorials/data/hisisso_perovskites
250M tutorials/data/kaggle_competition
1.1G tutorials/data/krr4mat
253M tutorials/data/nn_regression
252K tutorials/data/perovskites_tolerance_factor
15M tutorials/data/query_nomad_archive
3.4M tutorials/data/soap_atomic_charges
1.2M tutorials/data/tcmi
209M tutorials/data/tetradymite_PRM2020
```
```bash
$ du -sh /opt/*
3.8G /opt/conda
1.1G /opt/cpp_sisso
75M /opt/qmmlpack
877M /opt/quip
2.8G /opt/tutorials
```
```bash
$ du -sh /usr/*
173M /usr/bin
0 /usr/games
28M /usr/include
833M /usr/lib
28K /usr/local
3.0M /usr/sbin
1.6G /usr/share
0 /usr/src
```
Although the size of the tutorials is getting bigger and bigger is quite normal (for about 20 tutorial, 3GB shouldn't be a problem).
As far as I can see one of the possible ways to reduce the size is to do a multi-stage build.https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/15CO2 tutorial improvements2020-09-21T12:56:27ZLuigi SbailoCO2 tutorial improvementsA few things to be addressed in the future:
- printing a list of top k (k is user given) subgroups, not just the top one
- allowing selection of features in TR. I do not think I got an answer, here: which feature are offered to the TR tr...A few things to be addressed in the future:
- printing a list of top k (k is user given) subgroups, not just the top one
- allowing selection of features in TR. I do not think I got an answer, here: which feature are offered to the TR training, in the notebook and in the learning done for the manuscript? All the features listed in the section "physical features"?
- remove hard-coded data in the script and all bad-programming choice such as relative change of directories
- a tough but very important task: providing the scripts to extract the data from the raw FHI-aims calculations, essentially a meta-notebook that provides the input data to this notebook. As usual, if the processing is slow (to be defined), the data table is pre-loaded like now, but each entry points back to a FHI-aims calculation and a script for reproducing the reported values.
This task will very probably tackled by a new guy who will be 100% in publishing notebooks, but they will need assistance from Alex.https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/14Publishing the cmlkit tutorial2020-04-20T20:15:02ZMarcel LangerPublishing the cmlkit tutorialHello!
So, I've gotten the official permission from Matthias to publish the tutorial and the paper. I've pushed the most recent version of the tutorial to my repository. (See #13, I'll just hope the visualisation works!).
I assume you ...Hello!
So, I've gotten the official permission from Matthias to publish the tutorial and the paper. I've pushed the most recent version of the tutorial to my repository. (See #13, I'll just hope the visualisation works!).
I assume you need `qmmlpack` to be public before you can add the tutorial to the public build infrastructure?
If everything is merged and built, I assume the way to publish is to set the "isPublic" flag to `true`? I'd like to make sure the tutorial is live not too much before I submit the paper, to avoid confusion.
Thanks! Hope y'all are well in this time of crisis!https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/13Add `qmmlpack`2020-03-24T17:07:28ZMarcel LangerAdd `qmmlpack`Hello everyone,
For an upcoming paper and the release of associated software, I've been asked by Matthias Scheffler to prepare a tutorial. While most of the requirements are available on `pypi` and so no problem to handle with `setup.py...Hello everyone,
For an upcoming paper and the release of associated software, I've been asked by Matthias Scheffler to prepare a tutorial. While most of the requirements are available on `pypi` and so no problem to handle with `setup.py`, one package is not yet: [`qmmlpack`](https://gitlab.com/qmml/qmmlpack/-/tree/development) (development branch). Can you manually add it somehow?
`qmmlpack` is not yet public, but if you tell me the your username on `gitlab`, I can get you added to the repository.
Cheers!https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/123D visualisation for molecules2020-07-14T14:09:11ZAdam Fekete3D visualisation for moleculesThere multiple options to visualise 3D molecules in a web browser. There is a nice list of the available libraries here:
https://en.m.wikipedia.org/wiki/List_of_molecular_graphics_systems
Currently, a few tutorials use the JSmol (http:/...There multiple options to visualise 3D molecules in a web browser. There is a nice list of the available libraries here:
https://en.m.wikipedia.org/wiki/List_of_molecular_graphics_systems
Currently, a few tutorials use the JSmol (http://wiki.jmol.org/index.php/JSmol) package which works but a little bit outdated.
There are a few alternatives for the future which even have python interface:
- https://github.com/3dmol/3Dmol.js
- https://github.com/arose/nglviewhttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/11Use images that have the nomad python packages installed2020-09-08T07:44:30ZMarkus ScheidgenUse images that have the nomad python packages installedThe fundamental idea behind the analytics toolkit is that we want to allow scientist to analyse NOMAD data without barriers. Locally, the notebooks should support NOMAD client libraries that allow to easily access NOMAD data.
How can t...The fundamental idea behind the analytics toolkit is that we want to allow scientist to analyse NOMAD data without barriers. Locally, the notebooks should support NOMAD client libraries that allow to easily access NOMAD data.
How can this be achieved? Currently there are no pypi packages for NOMAD, but we are looking into this. Otherwise, NOMAD source-code can be simply checked out and pip installed directly. There is also the NOMAD docker image, which is a plain Python3.6 image with all required packages installed.
Currently, NOMAD sources are still very monolithic. We are planning to break this up into a reasonably hierarchy of standalone setuptools/pypi packages. This should allow to only have an isolated metainfo/API package.https://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/10Py3 version of QUIP/QUIPPY/GAP2020-04-20T20:17:32ZAdam FeketePy3 version of QUIP/QUIPPY/GAPThe following tutorials had been converted to python3:
- gap-si-surface
- grain-boundaries
- soap-atomic-chargesThe following tutorials had been converted to python3:
- gap-si-surface
- grain-boundaries
- soap-atomic-chargeshttps://gitlab.mpcdf.mpg.de/nomad-lab/analytics/-/issues/9Trusted notebooks by default2020-08-05T18:06:27ZAdam FeketeTrusted notebooks by defaultThere is a way to make a notebook trusted by default (https://jupyter-notebook.readthedocs.io/en/stable/security.html#explicit-trust). In short:
```
jupyter trust /path/to/notebook.ipynb
```
This command has to be run inside the user's c...There is a way to make a notebook trusted by default (https://jupyter-notebook.readthedocs.io/en/stable/security.html#explicit-trust). In short:
```
jupyter trust /path/to/notebook.ipynb
```
This command has to be run inside the user's container like part of a startup script