Large amounts of dynamic search quantities
I just reprocessed the solar cell database. There were issues with the indexing because it requires much more resources and time as usual. I managed to run the indexing, but only with reduce bulk insert size and a lot of patience.
The issue is that the database has ~300 search quantities per entry + ~200 files. In the end the elastic docs aggregate to 4.4GB, which is quite large for the relatively "small" dataset.
I suggest that we somehow limit the number of search quantities or the overall elastic doc size. As an alternative, we would need some annotation that allows schema authors to select quantities, if the automatically discovered search quantities have to be capped.