NOMAD v1 (re-)processing causes elasticsearc client Out of Memory errors
Something about the v1 changes seems to tick off elastic. Reprocessing standard Aflow/MP VASP calculations in bulk causes the elasticsearch client pods to fail with OutOfMemory on the java heap space.
We need more information and better error handling:
-
log stats on search index operations -
log/cap size of materials (in terms of #entries) -
configurable timeouts on ES operations -
configurable materials indexing -
configurable bulk index op sizes -
handling of ES index errors -
additional ES materials index creation/update (i.e. skip materials during processing and create index separately)