nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2024-03-07T12:14:23Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1925HDF5 support2024-03-07T12:14:23ZMarkus ScheidgenHDF5 support- theory parsers use HDF for large data (e.g. MD)
- as a HDF "sibling"-file to the archive .msg, but placed in the raw files
- written via the ServerContext
- if numpy quantity has HDF5Reference type, the values are written to HDF...- theory parsers use HDF for large data (e.g. MD)
- as a HDF "sibling"-file to the archive .msg, but placed in the raw files
- written via the ServerContext
- if numpy quantity has HDF5Reference type, the values are written to HDF5
- if numpy quantity has annotation, the values are written to HDF5
- for both annotations and HDF5Reference the archive browser opens H5Web
- problems
- we create more raw files even though it should be part of the archive
- if published h5groove cannot work with the HDF5 raw files anymore
- entries with these HDF5 files cannot be reprocessed, because the raw files are immutable
- transparently resolving HDF5 references in the archive api
- solution
- move archive HDF5 actually into the archive folder
- publish the archive in a way that the HDF5 stays usable (h5groove)
- publish the raw-files in a way the the HDF5 stays usable (h5groove)
- bigger idea: replace the archive with hdf5 ..., even bigger idea: replacing files with s3, object storage via HSDS
- fully replace
- hybrid
- whole entry, whole upload
- always have both + transparent NOMAD API
unpublished-upload:
raw/** (/<entry_id>.hdf5)
archive/<entry_id>.msg
published-upload:
raw-...-.zip
raw/**/*.hdf
archive-...-.msg
- archive/<entry_id>.hdf5
- archive-...-.hdf5
- <entry-id> / ...
architure:
h5web | nomad GUI |
h5grove | HSDS | archive API |
hd5 | msg | parques
s3, fs | fs | fs
tasks:
- suite of benchmark
- generic benchmark
- impl for .msg/archive
- impl for .hdf5
- impl for .parquet
- investigate if h5groove could read HDF from within a .zip filehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1885AchiveQuery gives error when querying archive on Oasis2024-03-04T08:02:25ZSarthak KapoorAchiveQuery gives error when querying archive on OasisUsing the ArchiveQuery for the same entry archive gives different behavior on local installation and Oasis (http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1).
On the local installation, the query is successful, whereas on the Oasis...Using the ArchiveQuery for the same entry archive gives different behavior on local installation and Oasis (http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1).
On the local installation, the query is successful, whereas on the Oasis it runs into the following error:
```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[9], line 19
9 required = {
10 'resolve-inplace': True,
11 'data': 'include-resolved',
12 'workflow2': '*',
13 }
14 aquery = ArchiveQuery(
15 url=url,
16 query=query,
17 required=required,
18 )
---> 19 response = await aquery.async_download()
21 print(response)
24 # for r in response:
25 # print(r.data)
26 # try:
27 # print(r.data.inputs[0].reference)
28 # except AttributeError:
29 # pass
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:442, in ArchiveQuery.async_download(self, number)
438 elif pending_size < number:
439 # if not sufficient fetched entries, fetch first
440 await self.async_fetch(number - pending_size)
--> 442 return await self._download_async(number)
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:324, in ArchiveQuery._download_async(self, number)
320 async with AsyncClient(timeout=Timeout(timeout=300)) as session:
321 tasks = [asyncio.create_task(
322 self._acquire(
323 ids, session, semaphore, bar)) for ids in self._entries[:number]]
--> 324 results = await asyncio.gather(*tasks)
326 return [result for result in results if result]
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:366, in ArchiveQuery._acquire(self, ids, session, semaphore, bar)
364 # successfully downloaded data
365 context = ClientContext(self._url, upload_id=upload_id, auth=self._auth)
--> 366 result = EntryArchive.m_from_dict(response.json()['data'][0]['archive'], m_context=context)
368 if not result:
369 print(f'No result returned for id {entry_id}, is the query proper?')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2371, in MSection.m_from_dict(cls, data, **kwargs)
2363 @classmethod
2364 def m_from_dict(cls: Type[MSectionBound], data: Dict[str, Any], **kwargs) -> MSectionBound:
2365 ''' Creates a section from the given serializable data dictionary.
2366
2367 This is the 'opposite' of :func:`m_to_dict`. It takes a deserialized dict, e.g.
2368 loaded from JSON, and turns it into a proper section, i.e. instance of the given
2369 section class.
2370 '''
-> 2371 return MSection.from_dict(data, cls=cls, **kwargs)
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2436, in MSection.from_dict(dct, cls, m_parent, m_context, **kwargs)
2433 section.m_annotations.update(m_annotations)
2434 section.m_parse_annotations()
-> 2436 section.m_update_from_dict(dct)
2437 return section
File /opt/conda/lib/python3.10/site-packages/nomad/datamodel/datamodel.py:940, in EntryArchive.m_update_from_dict(self, dct)
939 def m_update_from_dict(self, dct) -> None:
--> 940 super().m_update_from_dict(dct)
941 if self.definitions is not None:
942 self.definitions.archive = self
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2326, in MSection.m_update_from_dict(self, dct)
2324 section.m_add_sub_section(sub_section_def, sub_section)
2325 else:
-> 2326 sub_section = sub_section_cls.m_from_dict(
2327 sub_section_value, m_parent=self, m_context=m_context)
2328 section.m_add_sub_section(sub_section_def, sub_section)
2330 if isinstance(property_def, Quantity):
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2371, in MSection.m_from_dict(cls, data, **kwargs)
2363 @classmethod
2364 def m_from_dict(cls: Type[MSectionBound], data: Dict[str, Any], **kwargs) -> MSectionBound:
2365 ''' Creates a section from the given serializable data dictionary.
2366
2367 This is the 'opposite' of :func:`m_to_dict`. It takes a deserialized dict, e.g.
2368 loaded from JSON, and turns it into a proper section, i.e. instance of the given
2369 section class.
2370 '''
-> 2371 return MSection.from_dict(data, cls=cls, **kwargs)
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2407, in MSection.from_dict(dct, cls, m_parent, m_context, **kwargs)
2405 m_def_proxy = SectionReference.deserialize(context_section, None, m_def)
2406 m_def_proxy.m_proxy_context = m_context
-> 2407 cls = m_def_proxy.section_cls
2409 # if 'm_def_id' exists, check if id matches
2410 # in case of mismatch, retrieve the Package and use the corresponding section definition
2411 if 'm_def_id' in dct:
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:172, in MProxy.__getattr__(self, key)
171 def __getattr__(self, key):
--> 172 if self.m_proxy_resolve() is not None:
173 return getattr(self.m_proxy_resolved, key)
175 raise MetainfoReferenceError(f'could not resolve {self.m_proxy_value}')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:248, in SectionProxy.m_proxy_resolve(self)
246 current = self.m_proxy_section
247 for name in python_name:
--> 248 current = self._resolve_name(name, current)
250 if current is None:
251 raise MetainfoReferenceError(
252 f'could not resolve {self.m_proxy_value} from scope {self.m_proxy_section}')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:193, in SectionProxy._resolve_name(self, name, context)
190 if context is None:
191 return None
--> 193 if context.name == name and context != self.m_proxy_section:
194 return context
196 if isinstance(context, Section):
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:1308, in MSection.__getattr__(self, name)
1304 raise AttributeError(name)
1306 return self.m_get_annotations(annotation_name)
-> 1308 raise AttributeError(name)
AttributeError: name
```
Following is the code used on local installation. For authentication on the local installation, I am adding the following in `nomad.yaml`:
```
tests:
assume_auth_for_username: <<my username>>
```
```
from nomad.client import ArchiveQuery
import asyncio
async def my_query():
url = 'http://127.0.0.1:8000/fairdi/nomad/latest/api/v1/'
query = {
'entry_id:any': [
'4A82Wn-dzXJIheOBm94WNDjfSUts',
],
}
required = {
'resolve-inplace': True,
'data': 'include-resolved',
'workflow2': '*',
}
aquery = ArchiveQuery(
url=url,
query=query,
required=required,
)
response = await aquery.async_download()
print(response)
if __name__ == '__main__':
asyncio.run(my_query())
```
Following is the code used on the Oasis. The code runs on a jupyter notebook on the Oasis's JupterHub through which `nomad.client.Auth` can get the authentication token.
```
from nomad.client import ArchiveQuery
url = "http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1"
query = {
'entry_id:any': [
'rfd_xidMgJBJkKMymNscvN9mwejq',
],
}
required = {
'resolve-inplace': True,
'data': 'include-resolved',
'workflow2': '*',
}
aquery = ArchiveQuery(
url=url,
query=query,
required=required,
)
response = await aquery.async_download()
print(response)
```
The schema of the entry archive is available here: https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas/blob/main/analysis_plugin/src/analysis/schema.py#L391Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1807Harmonize electron count with charge2023-12-21T15:59:03ZNathan DaelmanHarmonize electron count with chargeThere is currently some inconsistency between with dimensions are used for denoting electrons.
There are quantities which count electrons (dimensionless) and those that measure their (combined) charge.
I suggest that we settle onto a sin...There is currently some inconsistency between with dimensions are used for denoting electrons.
There are quantities which count electrons (dimensionless) and those that measure their (combined) charge.
I suggest that we settle onto a single choice (to be determined), or allow for both when designing base classes.
The advantage of electron count is that it will remain consistent across unit systems (i.e. e- charge is unambiguously preferable).
The downside is that in many cases, experimentalists will want to see charge in their favorite units.
Would appreciate input from @jrudz @pizarroj @lucamghihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1779python schema gui error2023-12-21T15:55:26ZAdam Feketepython schema gui errorPython schema fails on the frontend when name is not defined as ELNAnnotation of a MSection.
Previously the nema was autogenerated from the classname and stored as a str. This causes error on the fronend because str doesn't have label a...Python schema fails on the frontend when name is not defined as ELNAnnotation of a MSection.
Previously the nema was autogenerated from the classname and stored as a str. This causes error on the fronend because str doesn't have label attriburte.
to reproduce:
```
from nomad.metainfo import Quantity, Package, Section, MEnum, Datetime, MSection, SubSection
from nomad.datamodel.data import EntryData, ArchiveSection
from nomad.datamodel.metainfo.annotations import ELNAnnotation, ELNComponentEnum
import numpy as np
m_package = Package(name='Jupyter Notebook Schema')
class Reference(MSection):
m_def = Section()
url = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.URLEditQuantity, label='URL'),
description='For testing subsection quantity.'
)
```
this one works:
```
from nomad.metainfo import Quantity, Package, Section, MEnum, Datetime, MSection, SubSection
from nomad.datamodel.data import EntryData, ArchiveSection
from nomad.datamodel.metainfo.annotations import ELNAnnotation, ELNComponentEnum
import numpy as np
m_package = Package(name='Jupyter Notebook Schema')
class Reference(MSection):
m_def = Section()
name = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.StringEditQuantity, label='Name'),
description='For testing subsection quantity.'
)
url = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.URLEditQuantity, label='URL'),
description='For testing subsection quantity.'
)
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1775Plugin links and descriptions not loaded into defintion sections2023-12-21T15:59:03ZMichael GöttePlugin links and descriptions not loaded into defintion sectionsDear Nomad,
unfortuantely the defintion sections do not contain annotations of the plugin components, such as links.
![grafik](/uploads/20af89159236738f775f1180cbe8d0bd/grafik.png)
Here you can see that the Process class (which is fro...Dear Nomad,
unfortuantely the defintion sections do not contain annotations of the plugin components, such as links.
![grafik](/uploads/20af89159236738f775f1180cbe8d0bd/grafik.png)
Here you can see that the Process class (which is from the nomad branch) displays the description and links while the inherited classes from the plugin do not.
Best
Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1747Change management strategies and policies2023-12-21T15:40:08ZMohammad NakhaeeChange management strategies and policiesI think at the moment we have lack of cahnge mangement. These are some conducting strategies that comes to my mind for user onboarding
- Warning when using deprecated features in a not published upload (If it is publishe we can show the...I think at the moment we have lack of cahnge mangement. These are some conducting strategies that comes to my mind for user onboarding
- Warning when using deprecated features in a not published upload (If it is publishe we can show the feature without any warning)
- Warning or even preventing publishing of outdated features
- Warnings or even preventing publishing of Beta version features
We had the same problem about tabular parser that changed a lot before approaching this version and hopefully we did not have any published schema.
In the case of plot functuionality I would say
- plotly_graph_object annotation [official feature]
- python PlotSection [official feature]
- plotly_express annotation [beta feature]
- plotly_subplot annotation [beta feature]
- plot annotation [deprecated feature]
@mscheidg @himanel1https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1735Files with filenames starting with a number cannot be opened2023-10-23T12:12:02ZMichael KriegerFiles with filenames starting with a number cannot be openedWhen a filename starts with a number (e.g. "1.txt"), the file cannot be opened from the FILES page in an upload. The error message shown ("ERROR The item ":1.txt" could not be found.") indicates that a colon is added before the actual fi...When a filename starts with a number (e.g. "1.txt"), the file cannot be opened from the FILES page in an upload. The error message shown ("ERROR The item ":1.txt" could not be found.") indicates that a colon is added before the actual filename; this only happends if the filename starts with a number.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1725Some feedback from sol group on gui/api2023-11-29T10:48:58ZAlvin Noe LadinesSome feedback from sol group on gui/apiFollowing are some of the thoughts I gathered from our users during the Arnsberg retreat:
- changing to 'dft' dashboard is not intuitive
- dft dashboard should include periodic table by default
- system info should be on entry table qua...Following are some of the thoughts I gathered from our users during the Arnsberg retreat:
- changing to 'dft' dashboard is not intuitive
- dft dashboard should include periodic table by default
- system info should be on entry table quantities
- querying via the api is not intuitive
- query results should somehow be sorted according to 'reliability' of entries
- metadata should include the github commit hash of the program used if available
- the 'linear' scaling of plots is not intuitive
- protocol for updating published dataset should be communicated
-
@himanel1 @mscheidghttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1717Log tab doesnt update when error is corrected2023-12-21T15:55:25ZMichael GötteLog tab doesnt update when error is correctedDear nomad,
I am using nomad to provide a quite complex tool to plan experiments. I nwant to communicate tot he user, if some input is wrong through the logs.
![grafik](/uploads/9f2666dbd0eb721a5bab3aad459cc2e2/grafik.png)
When correct...Dear nomad,
I am using nomad to provide a quite complex tool to plan experiments. I nwant to communicate tot he user, if some input is wrong through the logs.
![grafik](/uploads/9f2666dbd0eb721a5bab3aad459cc2e2/grafik.png)
When correcting the error I get:
![grafik](/uploads/fe30c3ba5171fd9181af03e048cf88eb/grafik.png)
Note that the logs tab is not red anymore.
When i reload i get:
![grafik](/uploads/5d9a296312a8689c1813f84f14d46189/grafik.png)
As i would expect it. The same happens, if another error occurs then the old ne is shown til reload.
Is this know @himanel1
Best
Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1714Remote debug options for mongoDB database2023-10-19T10:16:52ZAdam FeketeRemote debug options for mongoDB databaseThis issue is just collect and discuss all of the possibilities:
- web based clients:
- [ ] [mongo-express](https://github.com/mongo-express/mongo-express): most popular and commonly used (using web basic authentication)
- [ ] [mongo...This issue is just collect and discuss all of the possibilities:
- web based clients:
- [ ] [mongo-express](https://github.com/mongo-express/mongo-express): most popular and commonly used (using web basic authentication)
- [ ] [mongo-gui](https://github.com/arunbandari/mongo-gui)
- [ ] [MongoDB-PHP-GUI](https://github.com/SamuelTallet/MongoDB-PHP-GUI)
- exposing the port of mongodb:
- [ ] exposing through ingress controller (restricted access with username and password but it is publicly available)
- [ ] using kubernetes port forward (you need to have access to the cluster to use this way)
- [ ] [VS Code plugin](https://www.mongodb.com/products/tools/vs-code): super impressivehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1523Integrate GIMS (as a NORTH tool)2023-12-21T16:06:51ZMarkus ScheidgenIntegrate GIMS (as a NORTH tool)The "Graphical Interface for Materials Simulations" (GIMS) is a Python-based web-enabled tool to visualize fhi-aims and exciting output.
It is imaginable that we could containerise this and offer it as a NORTH tool on NOMAD. So if you ...The "Graphical Interface for Materials Simulations" (GIMS) is a Python-based web-enabled tool to visualize fhi-aims and exciting output.
It is imaginable that we could containerise this and offer it as a NORTH tool on NOMAD. So if you uploaded some exciting or aims data, you can run this to have a look.
If the tool provides value, we can think about improving it.
GIMS gitlab: https://gitlab.com/gims-developers/gims
GIMS docs: https://gims-developers.gitlab.io/gims/developerManual/developerManual.htmlhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1409Elemental Composition in Results2023-05-11T08:56:30ZJose Marquez PrietoElemental Composition in ResultsFor many experimental applications and apps, we need to have a place in `results` to store the elemental composition of a sample or a material.
My proposal would be to have it under `results.material` and under each system in `results.m...For many experimental applications and apps, we need to have a place in `results` to store the elemental composition of a sample or a material.
My proposal would be to have it under `results.material` and under each system in `results.material.topology`.
The section would have the following form, which is adapted from the one in our eln `System` base class that @hnaesstroem designed:
```
class ElementalComposition(MSection):
'''A section for describing the elemental composition of a system, i.e. the element
and its atomic fraction.
'''
m_def = Section(label_quantity='element')
element = Quantity(
type=MEnum(chemical_symbols[1:]),
description='''
The symbol of the element, e.g. 'Pb'.
''')
atomic_fraction = Quantity(
type=np.float64,
description='''
The atomic fraction of the element in the system it is contained within.
Per definition a positive value less than or equal to 1.
'''
)
```
This will allow us to search these values, use these in search widgets like the Scatter Plot once #1325 gets fixed, and build new widgets like ternary plot diagrams (important for #1296).
We will also update the Formula normalizer to populate this automatically from the formulas so that upon reprocessing the entries everything gets populated too.
@lucamghi, @himanel1 and Team C (@jrudz, @pizarroj and @ndaelman). Please do let us know as soon as possible if you have any objections to this. We would need this to be implemented with high priority for several upcoming presentations and use cases.Jose Marquez PrietoJose Marquez Prieto