NOMAD v1 (re-)processing causes elasticsearc client Out of Memory errors
Something about the v1 changes seems to tick off elastic. Reprocessing standard Aflow/MP VASP calculations in bulk causes the elasticsearch client pods to fail with OutOfMemory on the java heap space.
We need more information and better error handling:
- log stats on search index operations
- log/cap size of materials (in terms of #entries)
- configurable timeouts on ES operations
- configurable materials indexing
- configurable bulk index op sizes
- handling of ES index errors
- additional ES materials index creation/update (i.e. skip materials during processing and create index separately)