nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2024-03-07T12:14:23Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1925HDF5 support2024-03-07T12:14:23ZMarkus ScheidgenHDF5 support- theory parsers use HDF for large data (e.g. MD)
- as a HDF "sibling"-file to the archive .msg, but placed in the raw files
- written via the ServerContext
- if numpy quantity has HDF5Reference type, the values are written to HDF...- theory parsers use HDF for large data (e.g. MD)
- as a HDF "sibling"-file to the archive .msg, but placed in the raw files
- written via the ServerContext
- if numpy quantity has HDF5Reference type, the values are written to HDF5
- if numpy quantity has annotation, the values are written to HDF5
- for both annotations and HDF5Reference the archive browser opens H5Web
- problems
- we create more raw files even though it should be part of the archive
- if published h5groove cannot work with the HDF5 raw files anymore
- entries with these HDF5 files cannot be reprocessed, because the raw files are immutable
- transparently resolving HDF5 references in the archive api
- solution
- move archive HDF5 actually into the archive folder
- publish the archive in a way that the HDF5 stays usable (h5groove)
- publish the raw-files in a way the the HDF5 stays usable (h5groove)
- bigger idea: replace the archive with hdf5 ..., even bigger idea: replacing files with s3, object storage via HSDS
- fully replace
- hybrid
- whole entry, whole upload
- always have both + transparent NOMAD API
unpublished-upload:
raw/** (/<entry_id>.hdf5)
archive/<entry_id>.msg
published-upload:
raw-...-.zip
raw/**/*.hdf
archive-...-.msg
- archive/<entry_id>.hdf5
- archive-...-.hdf5
- <entry-id> / ...
architure:
h5web | nomad GUI |
h5grove | HSDS | archive API |
hd5 | msg | parques
s3, fs | fs | fs
tasks:
- suite of benchmark
- generic benchmark
- impl for .msg/archive
- impl for .hdf5
- impl for .parquet
- investigate if h5groove could read HDF from within a .zip filehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1885AchiveQuery gives error when querying archive on Oasis2024-03-04T08:02:25ZSarthak KapoorAchiveQuery gives error when querying archive on OasisUsing the ArchiveQuery for the same entry archive gives different behavior on local installation and Oasis (http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1).
On the local installation, the query is successful, whereas on the Oasis...Using the ArchiveQuery for the same entry archive gives different behavior on local installation and Oasis (http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1).
On the local installation, the query is successful, whereas on the Oasis it runs into the following error:
```
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[9], line 19
9 required = {
10 'resolve-inplace': True,
11 'data': 'include-resolved',
12 'workflow2': '*',
13 }
14 aquery = ArchiveQuery(
15 url=url,
16 query=query,
17 required=required,
18 )
---> 19 response = await aquery.async_download()
21 print(response)
24 # for r in response:
25 # print(r.data)
26 # try:
27 # print(r.data.inputs[0].reference)
28 # except AttributeError:
29 # pass
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:442, in ArchiveQuery.async_download(self, number)
438 elif pending_size < number:
439 # if not sufficient fetched entries, fetch first
440 await self.async_fetch(number - pending_size)
--> 442 return await self._download_async(number)
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:324, in ArchiveQuery._download_async(self, number)
320 async with AsyncClient(timeout=Timeout(timeout=300)) as session:
321 tasks = [asyncio.create_task(
322 self._acquire(
323 ids, session, semaphore, bar)) for ids in self._entries[:number]]
--> 324 results = await asyncio.gather(*tasks)
326 return [result for result in results if result]
File /opt/conda/lib/python3.10/site-packages/nomad/client/archive.py:366, in ArchiveQuery._acquire(self, ids, session, semaphore, bar)
364 # successfully downloaded data
365 context = ClientContext(self._url, upload_id=upload_id, auth=self._auth)
--> 366 result = EntryArchive.m_from_dict(response.json()['data'][0]['archive'], m_context=context)
368 if not result:
369 print(f'No result returned for id {entry_id}, is the query proper?')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2371, in MSection.m_from_dict(cls, data, **kwargs)
2363 @classmethod
2364 def m_from_dict(cls: Type[MSectionBound], data: Dict[str, Any], **kwargs) -> MSectionBound:
2365 ''' Creates a section from the given serializable data dictionary.
2366
2367 This is the 'opposite' of :func:`m_to_dict`. It takes a deserialized dict, e.g.
2368 loaded from JSON, and turns it into a proper section, i.e. instance of the given
2369 section class.
2370 '''
-> 2371 return MSection.from_dict(data, cls=cls, **kwargs)
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2436, in MSection.from_dict(dct, cls, m_parent, m_context, **kwargs)
2433 section.m_annotations.update(m_annotations)
2434 section.m_parse_annotations()
-> 2436 section.m_update_from_dict(dct)
2437 return section
File /opt/conda/lib/python3.10/site-packages/nomad/datamodel/datamodel.py:940, in EntryArchive.m_update_from_dict(self, dct)
939 def m_update_from_dict(self, dct) -> None:
--> 940 super().m_update_from_dict(dct)
941 if self.definitions is not None:
942 self.definitions.archive = self
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2326, in MSection.m_update_from_dict(self, dct)
2324 section.m_add_sub_section(sub_section_def, sub_section)
2325 else:
-> 2326 sub_section = sub_section_cls.m_from_dict(
2327 sub_section_value, m_parent=self, m_context=m_context)
2328 section.m_add_sub_section(sub_section_def, sub_section)
2330 if isinstance(property_def, Quantity):
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2371, in MSection.m_from_dict(cls, data, **kwargs)
2363 @classmethod
2364 def m_from_dict(cls: Type[MSectionBound], data: Dict[str, Any], **kwargs) -> MSectionBound:
2365 ''' Creates a section from the given serializable data dictionary.
2366
2367 This is the 'opposite' of :func:`m_to_dict`. It takes a deserialized dict, e.g.
2368 loaded from JSON, and turns it into a proper section, i.e. instance of the given
2369 section class.
2370 '''
-> 2371 return MSection.from_dict(data, cls=cls, **kwargs)
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:2407, in MSection.from_dict(dct, cls, m_parent, m_context, **kwargs)
2405 m_def_proxy = SectionReference.deserialize(context_section, None, m_def)
2406 m_def_proxy.m_proxy_context = m_context
-> 2407 cls = m_def_proxy.section_cls
2409 # if 'm_def_id' exists, check if id matches
2410 # in case of mismatch, retrieve the Package and use the corresponding section definition
2411 if 'm_def_id' in dct:
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:172, in MProxy.__getattr__(self, key)
171 def __getattr__(self, key):
--> 172 if self.m_proxy_resolve() is not None:
173 return getattr(self.m_proxy_resolved, key)
175 raise MetainfoReferenceError(f'could not resolve {self.m_proxy_value}')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:248, in SectionProxy.m_proxy_resolve(self)
246 current = self.m_proxy_section
247 for name in python_name:
--> 248 current = self._resolve_name(name, current)
250 if current is None:
251 raise MetainfoReferenceError(
252 f'could not resolve {self.m_proxy_value} from scope {self.m_proxy_section}')
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:193, in SectionProxy._resolve_name(self, name, context)
190 if context is None:
191 return None
--> 193 if context.name == name and context != self.m_proxy_section:
194 return context
196 if isinstance(context, Section):
File /opt/conda/lib/python3.10/site-packages/nomad/metainfo/metainfo.py:1308, in MSection.__getattr__(self, name)
1304 raise AttributeError(name)
1306 return self.m_get_annotations(annotation_name)
-> 1308 raise AttributeError(name)
AttributeError: name
```
Following is the code used on local installation. For authentication on the local installation, I am adding the following in `nomad.yaml`:
```
tests:
assume_auth_for_username: <<my username>>
```
```
from nomad.client import ArchiveQuery
import asyncio
async def my_query():
url = 'http://127.0.0.1:8000/fairdi/nomad/latest/api/v1/'
query = {
'entry_id:any': [
'4A82Wn-dzXJIheOBm94WNDjfSUts',
],
}
required = {
'resolve-inplace': True,
'data': 'include-resolved',
'workflow2': '*',
}
aquery = ArchiveQuery(
url=url,
query=query,
required=required,
)
response = await aquery.async_download()
print(response)
if __name__ == '__main__':
asyncio.run(my_query())
```
Following is the code used on the Oasis. The code runs on a jupyter notebook on the Oasis's JupterHub through which `nomad.client.Auth` can get the authentication token.
```
from nomad.client import ArchiveQuery
url = "http://nomad.support.ikz-berlin.de/nomad-oasis/api/v1"
query = {
'entry_id:any': [
'rfd_xidMgJBJkKMymNscvN9mwejq',
],
}
required = {
'resolve-inplace': True,
'data': 'include-resolved',
'workflow2': '*',
}
aquery = ArchiveQuery(
url=url,
query=query,
required=required,
)
response = await aquery.async_download()
print(response)
```
The schema of the entry archive is available here: https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas/blob/main/analysis_plugin/src/analysis/schema.py#L391Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1807Harmonize electron count with charge2023-12-21T15:59:03ZNathan DaelmanHarmonize electron count with chargeThere is currently some inconsistency between with dimensions are used for denoting electrons.
There are quantities which count electrons (dimensionless) and those that measure their (combined) charge.
I suggest that we settle onto a sin...There is currently some inconsistency between with dimensions are used for denoting electrons.
There are quantities which count electrons (dimensionless) and those that measure their (combined) charge.
I suggest that we settle onto a single choice (to be determined), or allow for both when designing base classes.
The advantage of electron count is that it will remain consistent across unit systems (i.e. e- charge is unambiguously preferable).
The downside is that in many cases, experimentalists will want to see charge in their favorite units.
Would appreciate input from @jrudz @pizarroj @lucamghihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1779python schema gui error2023-12-21T15:55:26ZAdam Feketepython schema gui errorPython schema fails on the frontend when name is not defined as ELNAnnotation of a MSection.
Previously the nema was autogenerated from the classname and stored as a str. This causes error on the fronend because str doesn't have label a...Python schema fails on the frontend when name is not defined as ELNAnnotation of a MSection.
Previously the nema was autogenerated from the classname and stored as a str. This causes error on the fronend because str doesn't have label attriburte.
to reproduce:
```
from nomad.metainfo import Quantity, Package, Section, MEnum, Datetime, MSection, SubSection
from nomad.datamodel.data import EntryData, ArchiveSection
from nomad.datamodel.metainfo.annotations import ELNAnnotation, ELNComponentEnum
import numpy as np
m_package = Package(name='Jupyter Notebook Schema')
class Reference(MSection):
m_def = Section()
url = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.URLEditQuantity, label='URL'),
description='For testing subsection quantity.'
)
```
this one works:
```
from nomad.metainfo import Quantity, Package, Section, MEnum, Datetime, MSection, SubSection
from nomad.datamodel.data import EntryData, ArchiveSection
from nomad.datamodel.metainfo.annotations import ELNAnnotation, ELNComponentEnum
import numpy as np
m_package = Package(name='Jupyter Notebook Schema')
class Reference(MSection):
m_def = Section()
name = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.StringEditQuantity, label='Name'),
description='For testing subsection quantity.'
)
url = Quantity(
type=str,
a_eln=ELNAnnotation(component=ELNComponentEnum.URLEditQuantity, label='URL'),
description='For testing subsection quantity.'
)
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1775Plugin links and descriptions not loaded into defintion sections2023-12-21T15:59:03ZMichael GöttePlugin links and descriptions not loaded into defintion sectionsDear Nomad,
unfortuantely the defintion sections do not contain annotations of the plugin components, such as links.
![grafik](/uploads/20af89159236738f775f1180cbe8d0bd/grafik.png)
Here you can see that the Process class (which is fro...Dear Nomad,
unfortuantely the defintion sections do not contain annotations of the plugin components, such as links.
![grafik](/uploads/20af89159236738f775f1180cbe8d0bd/grafik.png)
Here you can see that the Process class (which is from the nomad branch) displays the description and links while the inherited classes from the plugin do not.
Best
Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1725Some feedback from sol group on gui/api2023-11-29T10:48:58ZAlvin Noe LadinesSome feedback from sol group on gui/apiFollowing are some of the thoughts I gathered from our users during the Arnsberg retreat:
- changing to 'dft' dashboard is not intuitive
- dft dashboard should include periodic table by default
- system info should be on entry table qua...Following are some of the thoughts I gathered from our users during the Arnsberg retreat:
- changing to 'dft' dashboard is not intuitive
- dft dashboard should include periodic table by default
- system info should be on entry table quantities
- querying via the api is not intuitive
- query results should somehow be sorted according to 'reliability' of entries
- metadata should include the github commit hash of the program used if available
- the 'linear' scaling of plots is not intuitive
- protocol for updating published dataset should be communicated
-
@himanel1 @mscheidghttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1717Log tab doesnt update when error is corrected2023-12-21T15:55:25ZMichael GötteLog tab doesnt update when error is correctedDear nomad,
I am using nomad to provide a quite complex tool to plan experiments. I nwant to communicate tot he user, if some input is wrong through the logs.
![grafik](/uploads/9f2666dbd0eb721a5bab3aad459cc2e2/grafik.png)
When correct...Dear nomad,
I am using nomad to provide a quite complex tool to plan experiments. I nwant to communicate tot he user, if some input is wrong through the logs.
![grafik](/uploads/9f2666dbd0eb721a5bab3aad459cc2e2/grafik.png)
When correcting the error I get:
![grafik](/uploads/fe30c3ba5171fd9181af03e048cf88eb/grafik.png)
Note that the logs tab is not red anymore.
When i reload i get:
![grafik](/uploads/5d9a296312a8689c1813f84f14d46189/grafik.png)
As i would expect it. The same happens, if another error occurs then the old ne is shown til reload.
Is this know @himanel1
Best
Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1714Remote debug options for mongoDB database2023-10-19T10:16:52ZAdam FeketeRemote debug options for mongoDB databaseThis issue is just collect and discuss all of the possibilities:
- web based clients:
- [ ] [mongo-express](https://github.com/mongo-express/mongo-express): most popular and commonly used (using web basic authentication)
- [ ] [mongo...This issue is just collect and discuss all of the possibilities:
- web based clients:
- [ ] [mongo-express](https://github.com/mongo-express/mongo-express): most popular and commonly used (using web basic authentication)
- [ ] [mongo-gui](https://github.com/arunbandari/mongo-gui)
- [ ] [MongoDB-PHP-GUI](https://github.com/SamuelTallet/MongoDB-PHP-GUI)
- exposing the port of mongodb:
- [ ] exposing through ingress controller (restricted access with username and password but it is publicly available)
- [ ] using kubernetes port forward (you need to have access to the cluster to use this way)
- [ ] [VS Code plugin](https://www.mongodb.com/products/tools/vs-code): super impressive