ArchiveQuery multiprocess errors
There is an issue with a set of query error pairs: git.physik.hu-berlin.de
The original issue description is:
Using nomad.client.ArchiveQuery
fails when downloading some search queries, here are examples of queries with the errors.
query = ArchiveQuery(
query = {
'domain': 'dft',
'dft.system': 'bulk',
'dft.spacegroup': '225',
'dft.code_name': 'VASP',
'dft.searchable_quantities': ['electronic_dos'],
'atoms': ['Ge']
},
required={
'section_workflow': {
'calculation_result_ref' : '*'
},
'section_metadata' : {
'calc_id' : '*',
'formula' : '*'
},
},
per_page=1000,
parallel=10,
max=17000
)
print(query)
Output:
>>>MaybeEncodingError: Error sending result: '[<nomad.client.ProcState object at 0x7f7d9e2de190>]'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647")'
---------------------------------------------------------------------------
MaybeEncodingError Traceback (most recent call last)
/tmp/ipykernel_26715/3156984789.py in <module>
----> 1 print(query)
/home1/similarity_search_development/venv/lib/python3.7/site-packages/nomad/client.py in __repr__(self)
514 def __repr__(self):
515 if self._total == -1:
--> 516 self.call_api()
517
518 return str(self._statistics)
/home1/similarity_search_development/venv/lib/python3.7/site-packages/nomad/client.py in call_api(self)
477 elif nproc_states > 1:
478 with multiprocessing.Pool(nproc_states) as pool:
--> 479 self._proc_states = pool.map(_run_proc, self._proc_states)
480 else:
481 assert False, 'archive query was not stopped before running out of things to query'
~/anaconda3/lib/python3.7/multiprocessing/pool.py in map(self, func, iterable, chunksize)
266 in a list that is returned.
267 '''
--> 268 return self._map_async(func, iterable, mapstar, chunksize).get()
269
270 def starmap(self, func, iterable, chunksize=None):
~/anaconda3/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
655 return self._value
656 else:
--> 657 raise self._value
658
659 def _set(self, i, obj):
MaybeEncodingError: Error sending result: '[<nomad.client.ProcState object at 0x7f7d9e2de190>]'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647")'
First guess seems that certain exceptions that become part of ProcState
are not pickable and are causing all the encode/decode errors.