Skip to content

Refactoring metadata: using a consistent flat structure in MongoDB

David Sikter requested to merge flatten-mongo-fields into v1.0.0

Getting rid of Calc.metadata and instead storing the fields directly on the Calc object, plus some minor name changes for consistency and cleanup of old code.

Metadata refactoring:

  • Dataset.name -> Dataset.dataset_name
  • Dataset.created -> Dataset.dataset_create_time
  • Dataset.modified -> Dataset.dataset_modified_time
  • Calc.metadata.last_processing -> Calc.last_processing_time
  • Calc.metadata.last_edit -> Calc.last_edit_time
  • Fields calc_hash, pid, last_processing_time, last_edit_time, external_id, nomad_version, nomad_commit, external_db, comment, references, coauthors, shared_with, datasets are moved from Calc.metadata to Calc
  • Calc.metadata (and Calc.metadata.raw_id) are removed

Additionally:

  • Metadata files are only read when an entry is first processed, not when reprocessing or when importing bundles. Note, if the initial processing fails we will not try to read the metadata files again. Maybe this could be smarter...
  • Functionality connected to import of metadata from oasis metadata files is removed. Testing of new bundle-based oasis export/import is extended.

Merge request reports