Real-time updates for Encyclopedia
The materials index was separated from the calculations index due to performance limitations. These two indices should, however, be synced from time-to-time. Currently the contents of the Encyclopedia (=material entries) are updated manually by running a CLI command (takes > 10 hours currently). This could change so that entries in the materials index would be updated in automatically, or possibly even in real-time.
The updates should be triggered in several different scenarios, including at least the following:
- A new calculation has been processed and has been linked with a material
- Calculation is published
- Calculation is re-processed, possibly changing the material or material properties
- The calculation ownership is changed (
shared_with
,owner
) - Embargo is released
- Calculation is removed (done by admin)
- Uploads are migrated?
Possibly the stablest solution for achieving this could be the following:
- New uploads are indexes straight away into the materials index (small change in the processing pipeline).
- A periodic job checks for updated entries (based on a timestamp field which would record the latest update time of a calculation) and re-indexes them in the materials index. This could run e.g. every night.