NOMAD v1 (re-)processing causes elasticsearc client Out of Memory errors

Something about the v1 changes seems to tick off elastic. Reprocessing standard Aflow/MP VASP calculations in bulk causes the elasticsearch client pods to fail with OutOfMemory on the java heap space.

We need more information and better error handling:

  • log stats on search index operations
  • log/cap size of materials (in terms of #entries)
  • configurable timeouts on ES operations
  • configurable materials indexing
  • configurable bulk index op sizes
  • handling of ES index errors
  • additional ES materials index creation/update (i.e. skip materials during processing and create index separately)
Edited Nov 22, 2021 by Markus Scheidgen
Assignee Loading
Time tracking Loading