GitLab

These features would be super helpful for the data-migration:

• add a check if any dataset used in entries does actually exists. I migrated some entries that had a dataset_id in metadata.datasets which did not exist in the source db's datasets collection. The --fix option should auto remove these non existing datasets from the entries. There should be an error log of course.
• parameters entry-update and upload-update that both take a dict and are applied to the upload and entry documents at the end. This would allow me to overwrite e.g. process status, nomad version, etc. Things that would make it easier to query for new additions
• a flag that allows to run with the same query twice without even considering those uploads that are already present on the other side. This helps to quickly fetch new uploads or uploads that were skipped for some reason.