Problems in new parsers (#484) · Issues · nomad-lab / nomad-FAIR

Problems in new parsers

These are the top 20 exception from the last reprocessing of aims, exciting, abinit, crystal.

In addition to these, there are many calculations that produce incomplete results. This is most easily to spot, if you look at the system metadata. Many calculations have no formula. This indicates they have no system. This indicates the parser (even through running successfully) did not match much. Probably these calculation require slightly more relaxed regexps due to code version differences.

16879, parsers/fhi-aims, xx9izr2yRputru0JleV7-Q:t-ZPO5zE2Jw9DkP0m8fGQgR6vC-7

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1475, in parse
    self.parse_method()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1302, in parse_method
    sec_method.x_fhi_aims_controlInOut_number_of_spin_channels = self.out_parser.get_number_of_spin_channels()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 710, in get_number_of_spin_channels
    return array_size.get('Number of spin channels', 1)
AttributeError: 'NoneType' object has no attribute 'get'

5891, in normalizer, qhx0VdcgS3inMmKECuN3KA:zw5o6RH9ZBAyDlsc4CXhWml09CLc

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 558, in normalizing
    normalizer(self._parser_results).normalize(logger=logger)
  File "/app/nomad/normalizing/encyclopedia/encyclopedia.py", line 286, in normalize
    self.fill(context)
  File "/app/nomad/normalizing/encyclopedia/encyclopedia.py", line 188, in fill
    struct.normalize(context)
  File "/app/nomad/normalizing/encyclopedia/material.py", line 739, in normalize
    self.periodicity(ideal, prim_atoms)
  File "/app/nomad/normalizing/encyclopedia/material.py", line 577, in periodicity
    raise ValueError("Could not detect the periodic dimensions in a 1D system.")
ValueError: Could not detect the periodic dimensions in a 1D system.

3084, parsers/lammps, DDZKox_nQ_CLTPPaGSr0_w:zzStifnyE7SwZuV3SGagFAfCtekQ

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/lammpsparser/lammps_parser.py", line 787, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/lammpsparser/lammps_parser.py", line 766, in parse
    sec_md.with_trajectory = self.traj_parser.with_trajectory()
  File "/usr/local/lib/python3.7/site-packages/lammpsparser/lammps_parser.py", line 199, in with_trajectory
    return self.get('atoms_info') is not None
  File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get
    val = self.results.get(key, None)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results
    self.parse(self._key, **self._kwargs)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 501, in parse
    if self.file_mmap is None:
  File "/app/nomad/parsing/file_parser/text_parser.py", line 345, in file_mmap
    with open(self.mainfile) as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

2711, parsers/fhi-aims, o-8XYG0JS5OJU59uchgA7w:zjS-6QqvVUrBNsQgaQGNEHRcipV1

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1473, in parse
    sec_parallel_task_assignement.x_fhi_aims_parallel_task_host = task_hosts[i]
  File "/app/nomad/metainfo/metainfo.py", line 849, in __setattr__
    return super().__setattr__(name, value)
  File "/app/nomad/metainfo/metainfo.py", line 2221, in __set__
    (value, type(value), self, self.type))
TypeError: The value ['Task' '0' 'of' '512' '(0,0,0,0,0,0)' 'R1B-M0-N00-J00'] with type <class 'numpy.ndarray'> for quantity fhiaimsparser.metainfo.fhi_aims.x_fhi_aims_section_parallel_task_assignement.x_fhi_aims_parallel_task_host:Quantity is not of type <class 'str'>

2271, parsers/fhi-aims, zb79w_T9Q_qbstffWBW46g:JHXpfS4E6piQw8ohphNqL8VglHyr

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1475, in parse
    self.parse_method()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1342, in parse_method
    sec_xc_func.XC_functional_weight = weight(hybrid_coeff)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 766, in <lambda>
    'name': 'GGA_X_PBE', 'weight': lambda x: 0.75 if x is None else 1.0 - x},
TypeError: unsupported operand type(s) for -: 'float' and 'str'

1105, parsers/exciting, pvRdA8EBQ--7UQdVou5s-w:xrP0AfYb8-JsPChh0t0OthNP7K1X

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2153, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2128, in parse
    self.parse_configurations()
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2053, in parse_configurations
    self.parse_file(f, sec_scc)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1332, in parse_file
    parser_function(section)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1171, in _parse_eigenvalues
    sec_eigenvalues.eigenvalues_values = get_data('eigenvalues')
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1164, in get_data
    res = np.hstack([v[key] for v in data])
  File "<__array_function__ internals>", line 6, in hstack
  File "/usr/local/lib/python3.7/site-packages/numpy/core/shape_base.py", line 346, in hstack
    return _nx.concatenate(arrs, 1)
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 2 and the array at index 18 has size 1

1080, in normalizer, uCSlNMEWQfOS5TMvsQXz1A:ziSEudHjaTYIe3Jzj-MKRZn4VkDt

Traceback (most recent call last):
  File "/app/nomad/normalizing/dos.py", line 149, in normalize
    n_atoms=number_of_atoms
  File "/usr/local/lib/python3.7/site-packages/nomad_dos_fingerprints/DOSfingerprint.py", line 25, in calculate
    self.indices, self.bins = self._calculate_bytes(raw_energies, raw_dos, grid)
  File "/usr/local/lib/python3.7/site-packages/nomad_dos_fingerprints/DOSfingerprint.py", line 88, in _calculate_bytes
    energy, dos = np.transpose([(e,d) for e,d in zip(energy, dos) if (e >= grid_array[0][0] and e <= grid_array[-1][0])])
ValueError: not enough values to unpack (expected 2, got 0)

768, parsers/fhi-aims, iZGJdsGSRm6DYERVpo74fA:0iEfV-_AQA6wlw6mneY5ol8ia1vj

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1475, in parse
    self.parse_method()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1259, in parse_method
    for key, val in self.control_parser.items():
  File "/app/nomad/parsing/file_parser/text_parser.py", line 365, in items
    yield key, self.get(key)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get
    val = self.results.get(key, None)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results
    self.parse(self._key, **self._kwargs)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 501, in parse
    if self.file_mmap is None:
  File "/app/nomad/parsing/file_parser/text_parser.py", line 345, in file_mmap
    with open(self.mainfile) as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

341, parsers/fhi-aims, wKiS4s-NS9Ky5o1OtYKI2w:2IIcVSSBI5ktFmjUyfMISbWYaOiI

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1475, in parse
    self.parse_method()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1315, in parse_method
    self.parse_topology()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1424, in parse_topology
    species = self.out_parser['control_inout']['species']
TypeError: 'NoneType' object is not subscriptable

278, parsers/fhi-aims, qSCiRCcYSaifBVn00EFOQA:yHb3bEu31Sb24V5eRXcOJ63XX3bk

Traceback (most recent call last):
  File "/app/nomad/parsing/file_parser/text_parser.py", line 406, in _parse_quantities
    value_processed = quantities[i].to_data(values)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 220, in to_data
    val_out = [process(val) for val in val_in]
  File "/app/nomad/parsing/file_parser/text_parser.py", line 220, in <listcomp>
    val_out = [process(val) for val in val_in]
  File "/app/nomad/parsing/file_parser/text_parser.py", line 158, in process
    val = self.str_operation(val)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 480, in str_to_energy_components
    unit = {'Ha': 'hartree', 'eV': 'eV'}.get(vi[1], None)
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1459, in parse
    value = self.out_parser.get(key)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get
    val = self.results.get(key, None)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results
    self.parse(self._key, **self._kwargs)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 524, in parse
    self._parse_quantity(quantity)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 463, in _parse_quantity
    value.append(sub_parser.parse())
  File "/app/nomad/parsing/file_parser/text_parser.py", line 516, in parse
    self._parse_quantities(quantities_findall)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 419, in _parse_quantities
    self.logger.warn('Error setting value', data=dict(quantity=quantities[i].name))
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1982, in warn
    warning(msg, *args, **kwargs)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1977, in warning
    root.warning(msg, *args, **kwargs)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1390, in warning
    self._log(WARNING, msg, args, **kwargs)
TypeError: _log() got an unexpected keyword argument 'data'

213, parsers/crystal, gddy5rHgRceN2sZZ6yTnrA:zbuT8J3-vcIknvze4gbDSWQDPEfP

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/crystalparser/crystal_parser.py", line 566, in parse
    dist, minor = distribution.split(" : ", 1)
AttributeError: 'NoneType' object has no attribute 'split'

199, parsers/phonopy, z369XxTOT5S3db4Q2vrbqg:xeYjLT2hhwWskUCXdeqVg3bkND6h

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 498, in process_phonon
    raise ValueError("No method information available in referenced calculation.")
ValueError: No method information available in referenced calculation.

171, parsers/exciting, vDg3jiguSsau7B2R9XjxMQ:o9pLQY2RAWuFd58rYJVgrzvO9SuO

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2153, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse
    self.parse_method()
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1802, in parse_method
    xc_functional_names = self.info_parser.get_xc_functional_name()
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1002, in get_xc_functional_name
    xc_functional = self.get('initialization').get('x_exciting_xc_functional', None)
AttributeError: 'NoneType' object has no attribute 'get'

160, parsers/phonopy, win2cjTvRF6t_KjDmfVlZw:IxR6Cf1HLzo8zeedPFUHzO0t74BN

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/phonopyparser/phonopy_parser.py", line 185, in parse
    phonopy_obj = self.phonopy_obj
  File "/usr/local/lib/python3.7/site-packages/phonopyparser/phonopy_parser.py", line 50, in phonopy_obj
    self._build_phonopy_object_fhi_aims()
  File "/usr/local/lib/python3.7/site-packages/phonopyparser/phonopy_parser.py", line 77, in _build_phonopy_object_fhi_aims
    cell_obj, supercell_matrix, displacement, sym)
  File "/usr/local/lib/python3.7/site-packages/phonopyparser/fhiaims_io.py", line 137, in read_forces_aims
    raise Exception("No phonon calculations found")
Exception: No phonon calculations found

144, in normalizer, uCSlNMEWQfOS5TMvsQXz1A:xp4iSuT8IXqePP_KNuQaR0pOeC9z

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 558, in normalizing
    normalizer(self._parser_results).normalize(logger=logger)
  File "/app/nomad/normalizing/band_structure.py", line 63, in normalize
    self.add_band_gaps(band, energy_reference)
  File "/app/nomad/normalizing/band_structure.py", line 181, in add_band_gaps
    energies = np.concatenate(energies, axis=2)
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 9 and the array at index 6 has size 27

126, parsers/fhi-aims, jSSVtYWfTZacc7HjWPSqHQ:a8r8KkvKXWams50UhzMGCxY0IGqH

TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1477, in parse
    self.parse_configurations()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1190, in parse_configurations
    parse_section(section)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1147, in parse_section
    forces_raw[-len(forces):], 'eV/angstrom')
  File "/app/nomad/metainfo/metainfo.py", line 849, in __setattr__
    return super().__setattr__(name, value)
  File "/app/nomad/metainfo/metainfo.py", line 2111, in __set__
    obj.m_set(self, value)
  File "/app/nomad/metainfo/metainfo.py", line 956, in m_set
    value = self.__to_np(quantity_def, value)
  File "/app/nomad/metainfo/metainfo.py", line 920, in __to_np
    value = value.astype(quantity_def.type)
  File "/usr/local/lib/python3.7/site-packages/pint/quantity.py", line 1654, in _numpy_method_wrap
    value = func(*args, **kwargs)
ValueError: setting an array element with a sequence.

107, parsers/exciting, pzD88LYnTty-c_Xg07dqwg:wRtykAr2p9qUNTGDT-fZabQozNB4

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2153, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2128, in parse
    self.parse_configurations()
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2061, in parse_configurations
    sec_scc.x_exciting_geometry_optimization_method = optimization_step.get('method')
AttributeError: 'NoneType' object has no attribute 'x_exciting_geometry_optimization_method'

101, parsers/fhi-aims, ZFNMJDbJQ9aUq3o-aMq7lw:n43gSkg2cIhEM60wbmipsKQo1rig

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1477, in parse
    self.parse_configurations()
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1190, in parse_configurations
    parse_section(section)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1163, in parse_section
    parse_dos(section)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 941, in parse_dos
    energies, dos = read_projected_dos(proj_dos_files)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 897, in read_projected_dos
    energies = pint.Quantity(data[0], 'eV')
TypeError: 'NoneType' object is not subscriptable

61, parsers/exciting, ff8C1I9LTuiNVu5fyPVxnw:xHhm3RgE-DzKTJ8X1qYc2xm01Ld6

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2153, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2132, in parse
    self.parse_xs()
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1631, in parse_xs
    self.parse_file('input.xml', sec_method)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1332, in parse_file
    parser_function(section)
  File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1368, in _parse_input_xs
    sec_method.x_exciting_xs_screening_rgkmax = rgkmax
  File "/app/nomad/metainfo/metainfo.py", line 849, in __setattr__
    return super().__setattr__(name, value)
  File "/app/nomad/metainfo/metainfo.py", line 2111, in __set__
    obj.m_set(self, value)
  File "/app/nomad/metainfo/metainfo.py", line 956, in m_set
    value = self.__to_np(quantity_def, value)
  File "/app/nomad/metainfo/metainfo.py", line 917, in __to_np
    (quantity_def, value))
nomad.metainfo.metainfo.MetainfoError: The quantity excitingparser.metainfo.exciting.section_method.x_exciting_xs_screening_rgkmax:Quantity has not a unit, but value 3.5 bohr has.

15, parsers/crystal, llZrblH4THCuJvVjEBXBFA:VdOUqF-aHTwlDSRcuoubmNBPFxHo

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/crystalparser/crystal_parser.py", line 618, in parse
    atomic_numbers,
UnboundLocalError: local variable 'atomic_numbers' referenced before assignment

13, parsers/fhi-aims, w8VbMKDwSb27iSM2oCE4Kg:kEp6pGbAJfjwn5r1PuvciCi1IIPd

Traceback (most recent call last):
  File "/app/nomad/parsing/file_parser/text_parser.py", line 406, in _parse_quantities
    value_processed = quantities[i].to_data(values)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 220, in to_data
    val_out = [process(val) for val in val_in]
  File "/app/nomad/parsing/file_parser/text_parser.py", line 220, in <listcomp>
    val_out = [process(val) for val in val_in]
  File "/app/nomad/parsing/file_parser/text_parser.py", line 158, in process
    val = self.str_operation(val)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 467, in str_to_eigenvalues
    eig, occ = eigs[1:3]
ValueError: not enough values to unpack (expected 2, got 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/nomad/processing/data.py", line 453, in parsing
    self._parser_results, logger=logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1499, in parse
    parser.parse(filepath, archive, logger)
  File "/usr/local/lib/python3.7/site-packages/fhiaimsparser/fhiaims_parser.py", line 1459, in parse
    value = self.out_parser.get(key)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get
    val = self.results.get(key, None)
  File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results
    self.parse(self._key, **self._kwargs)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 524, in parse
    self._parse_quantity(quantity)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 463, in _parse_quantity
    value.append(sub_parser.parse())
  File "/app/nomad/parsing/file_parser/text_parser.py", line 516, in parse
    self._parse_quantities(quantities_findall)
  File "/app/nomad/parsing/file_parser/text_parser.py", line 419, in _parse_quantities
    self.logger.warn('Error setting value', data=dict(quantity=quantities[i].name))
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1982, in warn
    warning(msg, *args, **kwargs)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1977, in warning
    root.warning(msg, *args, **kwargs)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1390, in warning
    self._log(WARNING, msg, args, **kwargs)
TypeError: _log() got an unexpected keyword argument 'data'

Old issues description: These are exceptions thrown by parsers during reprocessing all non staging exciting and abinit code runs. I only checked the first one, but it reproduces with nomad parse ....

exception_hash.keyword: Descending	#	calc_id	upload_id	mainfile	parser	exception
s7J4tcDHcGPdLgK2ZnnzqhE56hd3	1105	xrP0AfYb8-JsPChh0t0OthNP7K1X	pvRdA8EBQ--7UQdVou5s-w	volume-1.5/0.965489384605630/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse self.parse_configurations() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2058, in parse_configurations self.parse_file(f, sec_scc) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1176, in _parse_eigenvalues sec_eigenvalues.eigenvalues_values = get_data('eigenvalues') File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1169, in get_data res = np.hstack([v[key] for v in data]) File "<array_function internals>", line 6, in hstack File "/usr/local/lib/python3.7/site-packages/numpy/core/shape_base.py", line 346, in hstack return _nx.concatenate(arrs, 1) File "<array_function internals>", line 6, in concatenate ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 2 and the array at index 18 has size 1
hwe7MyzCotT17K4FWIlLLnLyII3_	171	yEqUiNBD7iu3P4YZmzgvpj2CbFV6	QLFt5odgS1S1LQ_MDbZgpQ	silver-tddft/silver-loss/6/INFOXS.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2124, in parse self.parse_method() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1807, in parse_method xc_functional_names = self.info_parser.get_xc_functional_name() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1002, in get_xc_functional_name xc_functional = self.get('initialization').get('x_exciting_xc_functional', None) AttributeError: 'NoneType' object has no attribute 'get'
MEQ6y79UyB-M8DAu-NR4SBHzEJIq	107	wRtykAr2p9qUNTGDT-fZabQozNB4	pzD88LYnTty-c_Xg07dqwg	home1/UTE/ETERRNAL/DATA/FINISHED/CGeSn/000016/GG/L_GAMMA/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse self.parse_configurations() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2066, in parse_configurations sec_scc.x_exciting_geometry_optimization_method = optimization_step.get('method') AttributeError: 'NoneType' object has no attribute 'x_exciting_geometry_optimization_method'
HXW2tu3a-mtJmMCuvzjUTwTvISba	62	xHhm3RgE-DzKTJ8X1qYc2xm01Ld6	ff8C1I9LTuiNVu5fyPVxnw	AB@Graphene/azobenzene/cis/BSE/rpa/INFO_QMT001.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2130, in parse self.parse_xs() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1636, in parse_xs self.parse_file('input.xml', sec_method) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1373, in _parse_input_xs sec_method.x_exciting_xs_screening_rgkmax = rgkmax File "/app/nomad/metainfo/metainfo.py", line 849, in setattr return super().setattr(name, value) File "/app/nomad/metainfo/metainfo.py", line 2111, in set obj.m_set(self, value) File "/app/nomad/metainfo/metainfo.py", line 956, in m_set value = self.__to_np(quantity_def, value) File "/app/nomad/metainfo/metainfo.py", line 917, in __to_np (quantity_def, value)) nomad.metainfo.metainfo.MetainfoError: The quantity excitingparser.metainfo.exciting.section_method.x_exciting_xs_screening_rgkmax:Quantity has not a unit, but value 3.5 bohr has.
3kmtj-uvUs_AtxPSd_mgAoWmUY5i	14	LuH6KeBENcz3_mwE8HOwvYKJyqLw	0NR0E6gqQ2yVicJCMEP4SA	c-Si-OCTSF/12x12x12-BSE-PBE/INFO_QMT001.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2130, in parse self.parse_xs() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1636, in parse_xs self.parse_file('input.xml', sec_method) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1360, in _parse_input_xs sec_method.x_exciting_xs_rgkmax = rgkmax File "/app/nomad/metainfo/metainfo.py", line 849, in setattr return super().setattr(name, value) File "/app/nomad/metainfo/metainfo.py", line 2111, in set obj.m_set(self, value) File "/app/nomad/metainfo/metainfo.py", line 956, in m_set value = self.__to_np(quantity_def, value) File "/app/nomad/metainfo/metainfo.py", line 917, in __to_np (quantity_def, value)) nomad.metainfo.metainfo.MetainfoError: The quantity excitingparser.metainfo.exciting.section_method.x_exciting_xs_rgkmax:Quantity has not a unit, but value 8.0 bohr has.
2khWqBfkrqi65UqODhX0_U-Dp5qs	6	fwG3jU4_PBV1uWdAzFtXotFs293m	dkZ_BRDqSA2DeGsH3ItFyw	conv_ecut/Si.out0006	parsers/abinit	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/abinitparser/abinit_parser.py", line 983, in parse self.parse_dataset(dataset) File "/usr/local/lib/python3.7/site-packages/abinitparser/abinit_parser.py", line 847, in parse_dataset sec_method = self.parse_method(dataset) File "/usr/local/lib/python3.7/site-packages/abinitparser/abinit_parser.py", line 759, in parse_method sec_method.number_of_spin_channels = int(self.out_parser.get_input_var('nsppol', nd, 1)) TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
h04dOeanRAxBsfNPaVUnydXG8kSu	5	mVpv7QL-IkSnnbPdZB0CxrPjhtwG	ML2qGfRqSnmxDLvPKkrLMQ	LiH-pbc/130/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse self.parse_configurations() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2051, in parse_configurations sec_scc = parse_configuration(self.info_parser.get('groundstate')) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2042, in parse_configuration sec_system = self.parse_system(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1995, in parse_system setattr(sec_system, name, val) File "/app/nomad/metainfo/metainfo.py", line 849, in setattr return super().setattr(name, value) File "/app/nomad/metainfo/metainfo.py", line 2111, in set obj.m_set(self, value) File "/app/nomad/metainfo/metainfo.py", line 956, in m_set value = self.__to_np(quantity_def, value) File "/app/nomad/metainfo/metainfo.py", line 934, in __to_np value = quantity_def.type.type(value) ValueError: invalid literal for int() with base 10: '********'
cfgruvgumKUmo3jBj8wCYAPfsg7e	2	Ukv9vOpmfindZocfvucn3eYOFFZq	bBwLURrsQzyT-avIDAsNhA	silicon-gw/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2128, in parse self.parse_gw() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1712, in parse_gw self.parse_file(f, sec_method) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1665, in _parse_input_gw 'gw/selfenergy/npol', 0) File "/app/nomad/metainfo/metainfo.py", line 849, in setattr return super().setattr(name, value) File "/app/nomad/metainfo/metainfo.py", line 2221, in set (value, type(value), self, self.type)) TypeError: The value 2 with type <class 'numpy.int64'> for quantity nomad.datamodel.metainfo.common_dft.Method.gw_self_energy_c_number_of_poles:Quantity is not of type <class 'int'>
QptEjMdzQgru7wm-iUM0sYRQktG0	1	9UM_CpSzaystbWPZoghgjg3dZmC8	CgtgzCMXTUyY0tjfBYXAnA	TiO2/anatese/K/nomad/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse self.parse_configurations() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2058, in parse_configurations self.parse_file(f, sec_scc) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1106, in _parse_dos if self.dos_parser.get('totaldos', None) is None: File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get val = self.results.get(key, None) File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results self.parse(self._key, **self._kwargs) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 505, in parse if not self.total_dos: File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 471, in total_dos self._total_dos = self.root.findall('./%s/%s' % (self._totaldos_key, self._diagram_key)) File "/app/nomad/parsing/file_parser/xml_parser.py", line 52, in root self._file_handler = ElementTree.parse(self.mainfile).getroot() File "/usr/local/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse tree.parse(source, parser) File "/usr/local/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse self._root = parser._parse_whole(source) File "", line None xml.etree.ElementTree.ParseError: no element found: line 111468, column 0
g4xYYjGHqxmbahc5Xk0jQrB2BvLv	1	B7YX4kB9f1PSwYbeJeUcExy-UKZw	cdAK5_-gQ4GLFbZrgBRpQQ	conv/workdir/ngridk_08-rgkmax_05/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2126, in parse self.parse_configurations() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2058, in parse_configurations self.parse_file(f, sec_scc) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1337, in parse_file parser_function(section) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1162, in _parse_eigenvalues if self.eigval_parser.get('eigenvalues_occupancies', None) is None: File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get val = self.results.get(key, None) File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results self.parse(self._key, **self._kwargs) File "/app/nomad/parsing/file_parser/text_parser.py", line 501, in parse if self.file_mmap is None: File "/app/nomad/parsing/file_parser/text_parser.py", line 348, in file_mmap offset=self._file_offset) ValueError: cannot mmap an empty file
iwirABQpTlKQWqHEg7jS-2l-oA5w	1	qQbcfYsilb40jkYEN5hoW9PUdUrY	mcyQ9xEwQu2uNrkEtKNobg	volumes-rgkmax13/volume-2.1/1.01315940382018/INFO.OUT	parsers/exciting	Traceback (most recent call last): File "/app/nomad/processing/data.py", line 453, in parsing self._parser_results, logger=logger) File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 2124, in parse self.parse_method() File "/usr/local/lib/python3.7/site-packages/excitingparser/exciting_parser.py", line 1813, in parse_method correlation = self.input_xml_parser.get('libxc/correlation', None) File "/app/nomad/parsing/file_parser/file_parser.py", line 78, in get val = self.results.get(key, None) File "/app/nomad/parsing/file_parser/file_parser.py", line 47, in results self.parse(self._key, **self._kwargs) File "/app/nomad/parsing/file_parser/xml_parser.py", line 76, in parse if not self.root: File "/app/nomad/parsing/file_parser/xml_parser.py", line 52, in root self._file_handler = ElementTree.parse(self.mainfile).getroot() File "/usr/local/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse tree.parse(source, parser) File "/usr/local/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse self._root = parser._parse_whole(source) File "", line None xml.etree.ElementTree.ParseError: no element found: line 1, column 0

In addition, there are plenty of ABINIT calc that do not have a section_system, but no parser errors: https://nomad-lab.eu/prod/rae/reprocess/gui/search?visualization=system&dft.code_name=ABINIT&dft.system=unavailable

I am not sure if this is all, because the numbers do not add up. There are roughly ~2500 exciting runs with missing formulas/systems, but only ~2000 exceptions. We will see after bug fixes.

Edited Jan 26, 2021 by Alvin Noe Ladines