Draft: Updating archive json based on changes from normalize method

This MR is not a fix, but an issue and includes some example code to recreate the issue.

If some changes are made to the quantities of an entry through its normalize method, they are not updated in the archive.json main file of the entry when reprocessing the upload.

For reproduction: I wrote a test section with a normalizer that modifies its quantities and a data file that can be dropped in the upload to create an entry.

Dropping the data file creates an entry with bool_quantity=True and string_quantity="initialized".

Expected behavior: Upon processing the entry, the normalize method should modify these values to bool_quantity=False and string_quantity="modified". But this doesn't happen. Can we fix this?

Some observations:

1. While these modifications show up in the ELN, they are not updated in the archive.json file. If you try reprocessing the whole upload again (or multiple times), the archive.json is still not updated. This means that every time the upload is reprocessed the starting state of the quantities is the initialized one, never the modified one.
2. If you go inside the ELN, and trigger a reprocess from there (using the save button), the updates are reflected in the archive.json. So the updation only seems to work if reprocessing is triggered from inside the entry.
3. If you now change the states in the ELN to bool_quantity=True and string_quantity="modified again" and save the entry, the ELN will update to bool_quantity=False and string_quantity="modified" based on the normalization, but the archive.json is updated to bool_quantity=True and string_quantity="modified again". So it seems that the state of the ELN before hitting save is what is updated in the archive.json. Modifications done by the normalizer are not saved.

@himanel1

nomad/datamodel/metainfo/eln/__init__.py

@@ -36,6 +36,7 @@ from nomad.datamodel.data import (
 )
 from nomad.datamodel.metainfo.annotations import (
     ELNAnnotation,
+    ELNComponentEnum,
 )
 from nomad.datamodel.metainfo.basesections.v1 import (
     Activity,
@@ -61,6 +62,7 @@ from nomad.datamodel.metainfo.basesections.v1 import (
     CompositeSystem as Ensemble,  # For legacy support
 )
 from nomad.datamodel.metainfo.basesections.v1 import (
+    BaseSection,
     SystemComponent as Component,
 )
 from nomad.datamodel.metainfo.common import ProvenanceTracker
@@ -1740,4 +1742,31 @@ class SolarCellEQE(PlotSection):
         add_band_gap(archive, self.bandgap_eqe)
 
 
+class TestSection(BaseSection, EntryData):
+    """
+    Section for testing schema.
+    """
+
+    bool_quantity = Quantity(
+        type=bool,
+        a_eln=ELNAnnotation(
+            component=ELNComponentEnum.BoolEditQuantity,
+        ),
+    )
+    string_quantity = Quantity(
+        type=str,
+        a_eln=ELNAnnotation(
+            component=ELNComponentEnum.StringEditQuantity,
+        ),
+    )
+
+    def normalize(self, archive, logger):
+        if self.bool_quantity:
+            logger.info('bool_quantity is True')
+            self.bool_quantity = False
+            self.string_quantity = 'modified'
+
+        super().normalize(archive, logger)
+
+
 m_package.__init_metainfo__()