Parsing nexus files generates a lot of warnings and errors
When I parse a nexus file into nomad a lot of error and warnings are generated. The metainfo gets populated but some of the fields and attributes are missing.
This can be reproduced by running the mpes example on staging. Make sure that the MoTe2.mpes.nxs
file was generated (you might need to reprocess once after the upload has been created and all of the files downloaded). While this happens for this example the same error exists for every nexus file we try to process.
Here is the error log I get while processing: mpes_error.log
These are the two most common errors I get from the processing:
nomad_oasis_worker | ERROR nomad.processing 2024-03-13T14:45:33 could not normalize section
nomad_oasis_worker | - exception: Traceback (most recent call last):
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/normalizing/metainfo.py", line 35, in normalize_section
nomad_oasis_worker | normalize(self.entry_archive, logger)
nomad_oasis_worker | TypeError: 'MSubSectionList' object is not callable
nomad_oasis_worker | WARNING nomad.processing 2024-03-13T14:45:31 Error while setting attribute.
nomad_oasis_worker | - nomad.commit:
nomad_oasis_worker | - nomad.deployment: oasis
nomad_oasis_worker | - nomad.entry_id: Da5PJyBHOYH_drwzEfokahG1Cf3e
nomad_oasis_worker | - nomad.mainfile: MoTe2.mpes.nxs
nomad_oasis_worker | - nomad.processing.exe_info: Cannot find a proper definition for name depends_on__field
nomad_oasis_worker | - nomad.processing.logger: nomad.processing
nomad_oasis_worker | - nomad.processing.parser: parsers/nexus
nomad_oasis_worker | - nomad.processing.proc: Entry
nomad_oasis_worker | - nomad.processing.process: process_entry
nomad_oasis_worker | - nomad.processing.process_status: RUNNING
nomad_oasis_worker | - nomad.processing.process_worker_id: 6XXPTPE8S7ehdxNZRoy0Lg
nomad_oasis_worker | - nomad.processing.step: parsers/nexus
nomad_oasis_worker | - nomad.processing.target_name: depends_on__attribute
nomad_oasis_worker | - nomad.service: unknown nomad service
nomad_oasis_worker | - nomad.upload_id: N_uam4l-Q1-zRhkMNmZ35A
nomad_oasis_worker | - nomad.version: 1.2.2.dev497+g150c1828d
Also when I try to parse the IV_temp example in this MR !1717 (merged) (this deactivates the nexus obj file which produced another error with loading the file) I get an additional error. This fails the whole processing and no metainfo is generated for this file.
nomad_oasis_worker | ERROR nomad.metainfo 2024-03-13T14:54:52 error in indexing dynamic quantity
nomad_oasis_worker | - exception: Traceback (most recent call last):
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/processing/data.py", line 1488, in parsing
nomad_oasis_worker | parser.parse(
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/parsing/parser.py", line 460, in parse
nomad_oasis_worker | self.mainfile_parser.parse(mainfile, archive, logger)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/parsing/nexus/nexus.py", line 346, in parse
nomad_oasis_worker | nexus_helper.process_nexus_master_file(self.__nexus_populate)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 733, in process_nexus_master_file
nomad_oasis_worker | self.full_visit(self.in_file, self.in_file, "", self.visit_node)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 717, in full_visit
nomad_oasis_worker | self.full_visit(root, child, full_name, func)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 717, in full_visit
nomad_oasis_worker | self.full_visit(root, child, full_name, func)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 717, in full_visit
nomad_oasis_worker | self.full_visit(root, child, full_name, func)
nomad_oasis_worker | [Previous line repeated 2 more times]
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 712, in full_visit
nomad_oasis_worker | func(name, hdf_node)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 656, in visit_node
nomad_oasis_worker | process_node(hdf_node, "/" + hdf_name, self.parser, self.logger)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/pynxtools/nexus/nexus.py", line 361, in process_node
nomad_oasis_worker | parser(
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/parsing/nexus/nexus.py", line 306, in __nexus_populate
nomad_oasis_worker | self._populate_data(depth, nx_path, nx_def, hdf_node, current)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/parsing/nexus/nexus.py", line 239, in _populate_data
nomad_oasis_worker | if metainfo_def.use_full_storage:
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/metainfo/metainfo.py", line 3501, in __getattr__
nomad_oasis_worker | return super().__getattr__(name)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/metainfo/metainfo.py", line 1471, in __getattr__
nomad_oasis_worker | raise AttributeError(name)
nomad_oasis_worker | AttributeError: use_full_storage
nomad_oasis_worker |
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker |
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/processing/base.py", line 969, in proc_task
nomad_oasis_worker | rv = unwrapped_func(proc, *args, **kwargs)
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/processing/data.py", line 1247, in process_entry
nomad_oasis_worker | self._process_entry_local()
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/processing/data.py", line 1341, in _process_entry_local
nomad_oasis_worker | self.parsing()
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/processing/data.py", line 1496, in parsing
nomad_oasis_worker | raise ProcessFailure(
nomad_oasis_worker | nomad.processing.base.ProcessFailure: parser failed with exception
nomad_oasis_worker |
nomad_oasis_worker | During handling of the above exception, another exception occurred:
nomad_oasis_worker |
nomad_oasis_worker | Traceback (most recent call last):
nomad_oasis_worker | File "/usr/local/lib/python3.9/site-packages/nomad/metainfo/elasticsearch_extension.py", line 1614, in create_searchable_quantity
nomad_oasis_worker | value = float(value)
nomad_oasis_worker | TypeError: float() argument must be a string or a number, not 'dict'
nomad_oasis_worker | - exception_hash: 17kZjSVVAOoLB3FaSS-eAoOrnW5H
nomad_oasis_worker | - nomad.commit:
nomad_oasis_worker | - nomad.deployment: oasis
nomad_oasis_worker | - nomad.metainfo.path_archive: nexus.NXiv_temp.ENTRY.0.DATA.0.DATA__field
nomad_oasis_worker | - nomad.service: unknown nomad service
nomad_oasis_worker | - nomad.version: 1.2.2.dev497+g150c1828d