Extending published datasets blocked

A SOL user reported that they can no longer add data to an already published dataset. They get a 422 error message, saying "DatasetType.owned is not a value of this enumeration." with error type "value_error.exception". See the added screenshot for how this renders in the front-end.

Edit: this issue is only reproducible with older published datasets, and does NOT apply to newer published datasets. This was first pointed out by @ahmil, and later on traced by @thchang and myself. Below is a screenshot from Mongo Express that shows how in the column dataset_type

• old datasets have the incorrect value "DatasetType.owned".
• new datasets have the correct value "owned". Note that in the Python code base, these values are the attribute name and value, respectively.

added datamodel label

From the error message, it looks like the wrong value is set for the enumeration type DatasetType.owned in datasets.py.

added app label and removed datamodel label

changed the description

@mscheidg think this is relevant for you. This issue traces back to faulty values under the dataset.dataset_type key. It isn't clear how many datasets are affected.

I was just looking at the central nomad production datasets. Only very new datasets (last couple days) have owned instead of DatasetType.owned.

This indicates a data backwards compatibility problem in our latest release.

@thchang Due to the timing, I am pretty sure that was introduced when you reworked the types. Did the default serialisation of enum typed quantities changed?

None of the commit that introduces the new system, the current https://nomad-lab.eu/prod/v1/gui/ or the current develop branch shows this behavior. I do not think it is related to thew typing system.

But since I cannot reproduce it, I cannot pinpoint the commit that shows similar thing.

Before the commit 85d5eb1b that introduces the new typing system, the value serialized and stored in mongo is DatasetType.owned.

The typing system corrects this bug and exports the correct value owned.

So for the current codebase, there is nothing needs to be done as the old one is wrong while the current one is correct.

The mongo has to be updated such that all wrong values need to be corrected. This wrong value cannot pass the enum validation, clearly DatasetType.owned is not a valid value for the Enum('owned', 'foreign').

I see the good intentions, but what is "right" or "wrong", a "bug" or a "fix" is blurred, if the system cannot work with the existing data anymore. We have to be careful with such changes as asking all Oases to migrate their data is an expensive endeavour and not warranted for such a simple fix. This leaves us with two choices. We either roll the enum serialisation back to the wrong behaviour or we make the system backwards compatible and support both serialisations. The later one being preferrable. How hard would it be to also support existing datasets serialised with DatasetType.owned?

This specific case may not be the only faulty case. Other enums may also be affected. Two workarounds:

1. Disable the whole validation, so that it behaves just like the previous implementation. No need to deal with other usages.
2. Redefine this enum to Enum('owned', 'foreign', 'DatasetType.owned', 'DatasetType.foreign'). And do the same thing when another faulty usage of enum is reported.

if the system cannot work with the existing data anymore.

This is likely going to happen anyway since previously populated data would likely contain some quantities with wrong type/shape.

Maybe it is not going to affect GUI functionalities, but for sure deserialization via m_from_dict() is going to fail.

I don't think we need to disable the validation. You could simply add a clause that would also accept <TypeName>.<enumValue> as a valid value.

How to predict the correct <TypeName>? Or maybe both A.<enumValue> and B.<enumValue> are valid?

I would say

if '.' in value:
   value = value.split('.')[-1]

may be better as at least the wrong value does not propagate further.

created branch 2104-extending-published-datasets-blocked to address this issue

mentioned in merge request !2046 (merged)

mentioned in commit c8a81848

closed with merge request !2046 (merged)

@ndaelman tomorrow i will deploy this fix on the central NOMAD and test with the datasets there. I would then include this in our v1.3.5 pre-release and you could initiate a respective update on the SOL Oasis. I keep you posted.

I can still reproduce the problem on our production, but it is fixed on develop. There are some other issues, I hope to still fix in 1.3.5. But next week the latest, we should have a new release tag that could be used to deploy the fix.