Commit edc65261 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Added references to quantities. #471

parent c2810c34
Pipeline #92201 passed with stages
in 28 minutes and 17 seconds
......@@ -190,9 +190,69 @@ and a *sub-section* is a set of *sections* (contents) contained in another *sect
.. autoclass:: SubSection
.. _metainfo-categories:
References and Proxies
----------------------
Beside creating hierarchies (e.g. tree structures) with ``SubSection``, the metainfo
also allows to create cross references between sections and other sections or quantity
values:
.. code-block:: python
class Calculation(MSection):
system = Quantity(type=System.m_def)
atom_labels = Quantity(type=System.atom_labels)
calc = Calculation()
calc.system = run.systems[-1]
calc.atom_labels = run.systems[-1]
To define a reference, you define a normal quantity and simply use the section or quantity
that you want to reference as type. Then you can assign respective section instances
as values.
When in Python memory, quantity values that reference other sections simply contain a
Python reference to the respective `section instance`. However, upon serializing/storing
metainfo data, these references have to be represented differently.
Value references are a little different. When you read a value references it behaves like
the references value. Internally, we do not store the values, but a reference to the
section that holds the referenced quantity. Therefore, when you want to
assign a value reference, you use the section with the quantity and not the value itself.
Currently this metainfo implementation only supports references within a single
section hierarchy (e.g. the same JSON file). References are stored as paths from the
root section, over sub-sections, to the referenced section or quantity value. Each path segment is
the name of the sub-section or an index in a repeatable sub-section:
``/system/0`` or ``/system/0/atom_labels``.
References are automatically serialized by :py:meth:`MSection.m_to_dict`. When de-serializing
data with :py:meth:`MSection.m_from_dict` these references are not resolved right away,
because the references section might not yet be available. Instead references are stored
as :class:`MProxy` instances. These objects are automatically replaced by the referenced
object when a respective quantity is accessed.
.. autoclass:: MProxy
If you want to defined references, it might not be possible to define the referenced
section or quantity before hand, due to how Python definitions and imports work. In these
cases, you can use a proxy to reference the reference type:
.. code-block:: python
class Calculation(MSection):
system = Quantity(type=MProxy('System')
atom_labels = Quantity(type=MProxy('System/atom_labels')
The strings given to ``MProxy`` are paths within the available definitions. The above example
works, if ``System`` and ``System/atom_labels`` are eventually defined in the same package.
Categories
----------
......@@ -252,27 +312,6 @@ quantity definitions are unknown when writing code.
.. _metainfo-urls:
References and metainfo URLs
----------------------------
When in Python memory, quantity values that reference other sections simply contain a
Python reference to the respective `section instance`. However, upon serializing/storing
metainfo data, these references have to be represented differently.
Currently this metainfo implementation only supports references within a single
section hierarchy (e.g. the same JSON file). References are stored as paths from the
root section, over sub-sections, to the references section. Each path segment is
the name of the sub-section or an index in a repeatable sub-section:
``/system/0/symmetry``.
References are automatically serialized by :py:meth:`MSection.m_to_dict`. When de-serializing
data with :py:meth:`MSection.m_from_dict` these references are not resolved right away,
because the references section might not yet be available. Instead references are stored
as :class:`MProxy` instances. These objects are automatically replaced by the referenced
object when a respective quantity is accessed.
.. autoclass:: MProxy
Resources
---------
......@@ -308,6 +347,7 @@ from .metainfo import (
MetainfoReferenceError,
DataType,
Reference,
QuantityReference,
Datetime,
JSON,
MResource,
......
......@@ -293,6 +293,9 @@ class _QuantityType(DataType):
if section is not None:
return Reference(section)
if isinstance(value, Quantity):
return QuantityReference(value)
if isinstance(value, MProxy):
value.m_proxy_section = section
value.m_proxy_quantity = quantity_def
......@@ -387,6 +390,26 @@ class Reference(DataType):
return MProxy(value, m_proxy_section=section, m_proxy_quantity=quantity_def)
class QuantityReference(Reference):
''' Datatype used for reference quantities that reference other quantities. '''
def __init__(self, quantity_def: Union['Quantity']):
super().__init__(cast(Section, quantity_def.m_parent))
self.target_quantity_def = quantity_def
def get_normalize(self, section: 'MSection', quantity_def: 'Quantity', value: Any) -> Any:
section = super().get_normalize(section, quantity_def, value)
return getattr(section, self.target_quantity_def.name)
def serialize(self, section: 'MSection', quantity_def: 'Quantity', value: Any) -> Any:
section_path = super().serialize(section, quantity_def, value)
return f'{section_path}/{self.target_quantity_def.name}'
def deserialize(self, section: 'MSection', quantity_def: 'Quantity', value: Any) -> Any:
section_path = value.rsplit('/', 1)[0]
return MProxy(section_path, m_proxy_section=section, m_proxy_quantity=quantity_def)
class _Datetime(DataType):
def _parse(self, datetime_str: str) -> datetime:
......
......@@ -618,53 +618,6 @@ class TestM1:
scc.m_to_dict()
test_utils.assert_log(caplog, 'WARN', 'wrong shape')
def test_proxy(self):
class OtherSection(MSection):
name = Quantity(type=str)
class ReferencingSection(MSection):
proxy = Quantity(type=Reference(OtherSection.m_def))
sub = SubSection(sub_section=OtherSection.m_def)
obj = ReferencingSection()
referenced = obj.m_create(OtherSection)
referenced.name = 'test_value'
obj.proxy = referenced
assert obj.proxy == referenced
assert obj.m_to_dict()['proxy'] == '/sub'
assert obj.m_resolve('sub') == referenced
assert obj.m_resolve('/sub') == referenced
obj.proxy = MProxy('doesnotexist', m_proxy_section=obj, m_proxy_quantity=ReferencingSection.proxy)
with pytest.raises(ReferenceError):
obj.proxy.name
obj.proxy = MProxy('sub', m_proxy_section=obj, m_proxy_quantity=ReferencingSection.proxy)
assert obj.proxy.name == 'test_value'
assert not isinstance(obj.proxy, MProxy)
obj = ReferencingSection.m_from_dict(obj.m_to_dict(with_meta=True))
assert obj.proxy.name == 'test_value'
def test_ref_with_section_proxy(self):
package = Package(name='test_package')
class OtherSection(MSection):
name = Quantity(type=str)
class ReferencingSection(MSection):
reference = Quantity(type=Reference(SectionProxy('OtherSection')))
package.m_add_sub_section(Package.section_definitions, OtherSection.m_def)
package.m_add_sub_section(Package.section_definitions, ReferencingSection.m_def)
referencing = ReferencingSection()
reference = OtherSection()
referencing.reference = reference
assert referencing.reference == reference
def test_copy(self):
run = Run()
run.m_create(Parsing).parser_name = 'test'
......
#
# Copyright The NOMAD Authors.
#
# This file is part of NOMAD. See https://nomad-lab.eu for further info.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import pytest
from nomad.metainfo import MSection, Quantity, SubSection, MProxy, Reference, QuantityReference
class Referenced(MSection):
str_quantity = Quantity(type=str)
class Referencing(MSection):
section_reference = Quantity(type=Referenced)
section_reference_list = Quantity(type=Referenced, shape=['*'])
quantity_reference = Quantity(type=Referenced.str_quantity)
class Root(MSection):
referenced = SubSection(sub_section=Referenced)
referenceds = SubSection(sub_section=Referenced, repeats=True)
referencing = SubSection(sub_section=Referencing)
@pytest.fixture(scope='function', params=['reference', 'proxy'])
def definitions(request):
''' Alters the type of the references used in the reference properties definitions. '''
reference_type = request.param
if reference_type == 'reference':
Referencing.section_reference.type = Reference(Referenced.m_def)
Referencing.section_reference_list.type = Reference(Referenced.m_def)
Referencing.quantity_reference.type = QuantityReference(Referenced.str_quantity)
elif reference_type == 'proxy':
Referencing.section_reference.type = Reference(Referenced.m_def)
Referencing.section_reference_list.type = Reference(Referenced.m_def)
Referencing.quantity_reference.type = QuantityReference(Referenced.str_quantity)
else:
raise NotImplementedError()
@pytest.fixture(scope='function')
def example_data(definitions):
def create_referenced():
referenced = Referenced()
referenced.str_quantity = 'test_value'
return referenced
referenced = create_referenced()
referenced_1 = create_referenced()
referenced_2 = create_referenced()
root = Root()
root.referenced = referenced
root.m_add_sub_section(Root.referenceds, referenced_1)
root.m_add_sub_section(Root.referenceds, referenced_2)
referencing = Referencing()
referencing.section_reference = referenced
referencing.section_reference_list = [referenced_1, referenced_2]
referencing.quantity_reference = referenced
root.referencing = referencing
return root
def assert_data(example_data):
def assert_properties(example_data):
assert example_data.referencing.section_reference.m_resolved() == example_data.referenced
assert example_data.referencing.m_to_dict()['section_reference'] == '/referenced'
assert example_data.referencing.section_reference_list[1].m_resolved() == example_data.referenceds[1]
assert example_data.referencing.m_to_dict()['section_reference_list'] == ['/referenceds/0', '/referenceds/1']
assert example_data.referencing.quantity_reference == 'test_value'
assert example_data.referencing.m_to_dict()['quantity_reference'] == '/referenced/str_quantity'
assert_properties(example_data)
example_data_serialized = example_data.m_to_dict(with_meta=True)
example_data = Root.m_from_dict(example_data_serialized)
assert_properties(example_data)
def test_references(example_data):
assert_data(example_data)
def test_section_proxy(example_data):
example_data.referencing.section_reference = MProxy(
'doesnotexist',
m_proxy_section=example_data.referencing,
m_proxy_quantity=Referencing.section_reference)
with pytest.raises(ReferenceError):
example_data.referencing.section_reference.str_quantity
example_data.referencing.section_reference = MProxy(
'/referenced',
m_proxy_section=example_data.referencing,
m_proxy_quantity=Referencing.section_reference)
assert_data(example_data)
def test_quantity_proxy(example_data):
example_data.referencing.quantity_reference = MProxy(
'doesnotexist',
m_proxy_section=example_data.referencing,
m_proxy_quantity=Referencing.section_reference)
with pytest.raises(ReferenceError):
example_data.referencing.quantity_reference
example_data.referencing.quantity_reference = MProxy(
'/referenced',
m_proxy_section=example_data.referencing,
m_proxy_quantity=Referencing.section_reference)
assert example_data.referencing.quantity_reference == 'test_value'
assert_data(example_data)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment