README.md

 # __python-common__
 
-
 This repository contains the common python files that are
 part of the [NOMAD Laboratory](http://nomad-lab.eu).
 The official version lives at
@@ -18,54 +17,20 @@ The simplest way to have this is to check out nomad-lab-base recursively:
 
 then this will be in python-common within it.
 
-# Local Install
-
-The following instructions were tested on Ubuntu 14.04. With these instructions
-you can install the package for the local user (doesn't need root privileges).
-First make sure you have [pip](https://pypi.python.org/pypi/pip) available. If
-not, you can install it for python 2.x with:
-
-```sh
-wget https://bootstrap.pypa.io/get-pip.py
-python get-pip.py --user
-```
-
-or for python3 with:
+# Installation
+The code is python>=2.7 and python>=3.4 compatible. First download and install
+this package:
 
 ```sh
-wget https://bootstrap.pypa.io/get-pip.py
-python3 get-pip.py --user
+git clone https://gitlab.mpcdf.mpg.de/nomad-lab/python-common.git
+cd python-common
+pip install -r requirements.txt
+pip install -e .
 ```
 
-The modules used for parser development are located under the 'nomadcore'
-package. If you wish to setup this package locally, you can do so by first
-installing the required dependencies (use pip3 for python3 setup)
+Then download the metainfo definitions to the same folder where the
+'python-common' repository was cloned:
 
 ```sh
-pip install -r requirements.txt --user
-#pip3 install -r requirements.txt --user
+git clone https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-meta-info.git
 ```
-
-and then installing the package itself with the provided installation
-script(use python3 for python3 setup)
-
-```sh
-python setup.py develop --user
-#python3 setup.py develop --user
-```
-
-This will install a development version, which means that if you update the
-source code, all the changes will be available immediately without reinstall of
-the package. The current setup also assumes a certain location for the metainfo
-repository. If you place all the repositories (python-common, nomad-meta-info,
-parser repository) in the same folder, things should work.
-
-After this the package will be available to import by simply calling
-
-```python
-import nomadcore
-```
-
-in python. The development mode also means that the latest version of the code
-is used, so any updates from git will automatically be available. You can
-install a static snapshot by using 'install' instead of 'develop'.

common/python/nomadcore/ActivateLogging.py

 import logging
 
 # create console handler with a higher log level
-ch = logging.StreamHandler()
-ch.setLevel(logging.INFO)
+# ch = logging.StreamHandler()
+# ch.setLevel(logging.INFO)
 
 logger=logging.getLogger("nomadcore")
 logger.setLevel(logging.WARNING)
-logger.addHandler(ch)
+# logger.addHandler(ch)
 
 logger2=logging.getLogger("nomad")
 logger2.setLevel(logging.WARNING)
-logger2.addHandler(ch)
+# logger2.addHandler(ch)
 
 def debugToFile():
     "makes a full log to a file named detailed.log"

common/python/nomadcore/archive.py

@@ -9,6 +9,12 @@ import h5py
 import numpy as np
 from abc import ABCMeta, abstractmethod
 from io import open
+import re
+
+import logging
+
+
+LOGGER = logging.getLogger(__name__)
 
 
 class ArchiveSection(object):
@@ -31,6 +37,21 @@ class ArchiveSection(object):
         self._path = path
         self._archive = archive
 
+    def is_filtered(self, name):
+        """Used to filter out unnecessary information when recursing the
+        information. This unnecessary information includes e.g. the gIndex,
+        references, etc.
+        """
+        filtered = set([
+            "gIndex",
+            "name",
+            "references",
+            "type",
+        ])
+        if name not in filtered:
+            return name
+        return None
+
     def get_by_path(self, path):
         """Used to query the ArchiveSection recursively with a simple syntax
         that also allows indexing.
@@ -59,7 +80,7 @@ class ArchiveSection(object):
                 information within the metainfo hierarchy. See the examples.
 
         Returns:
-            ArchiveSection, a list of ArchiveSections, a concrete value
+            One of the following: ArchiveSection, a list of ArchiveSections or a concrete value
             corresponding to the given query,
         """
         parts, _, _ = self.get_path_parts(path)
@@ -76,6 +97,10 @@ class ArchiveSection(object):
 
             current_data = current_data.get_child(part)
             if i_part == n_parts - 1:
+
+                filtered = self.is_filtered(part)
+                if filtered is None or filtered != part:
+                    raise KeyError("Value for '{}' could not be found".format(current_path))
                 return current_data
 
     def get(self, key, default=None):
@@ -186,6 +211,7 @@ class ArchiveSection(object):
         # Check that the value has not been deleted
         full_path = "{}/{}".format(self._path, key)
         deleted = self.check_deletions(full_path)
+
         if deleted:
             raise KeyError("Value for '{}' has not been set.".format(full_path))
         try:
@@ -451,6 +477,16 @@ class ArchiveSectionHDF5(ArchiveSection):
     def __len__(self):
         return len(self.keys())
 
+    def is_filtered(self, key):
+        # return key
+        if key.endswith("-index"):
+            return None
+        key_without_size = key.rsplit(".", 1)[0]
+        if key_without_size.endswith("-v"):
+            return key_without_size[:-2]
+        else:
+            return key
+
     def __contains__(self, key):
         try:
             self[key]
@@ -461,6 +497,9 @@ class ArchiveSectionHDF5(ArchiveSection):
     def items(self):
         local_index = 0
         for key, value in self._data.items():
+            key_filtered = self.is_filtered(key)
+            if key_filtered is None:
+                continue
             if isinstance(value, h5py.Group):
                 index_datas = self._index_datas[:]
                 index_data = self._data.get("{}-index".format(key))
@@ -468,7 +507,7 @@ class ArchiveSectionHDF5(ArchiveSection):
                     index_datas = []
                 else:
                     index_datas.append(index_data)
-                yield (key, ArchiveSectionHDF5(
+                yield (key_filtered, ArchiveSectionHDF5(
                     value,
                     "{}/{}".format(self._path, key),
                     self._archive,
@@ -476,15 +515,21 @@ class ArchiveSectionHDF5(ArchiveSection):
                     local_index)
                 )
             else:
-                yield (key, value)
+                yield (key_filtered, value)
             local_index += 1
 
     def keys(self):
-        return self._data.keys()
+        for key in self._data.keys():
+            key_filtered = self.is_filtered(key)
+            if key_filtered is not None:
+                yield key_filtered
 
     def values(self):
         local_index = 0
         for key, value in self._data.items():
+            key_filtered = self.is_filtered(key)
+            if key_filtered is None:
+                continue
             if isinstance(value, h5py.Group):
                 index_datas = self._index_datas[:]
                 index_data = self._data.get("{}-index".format(key))
@@ -656,17 +701,16 @@ class ArchiveSectionHDF5(ArchiveSection):
                     # .format(child_path)
                 # )
 
-            index_rows = index_data[test_index]
-
             # If the value can have multiple shapes, the values are split into
             # different tables. For each table there is a local index in the
             # second column of the index table that we must use.
             data = []
-            for index_row in index_rows:
+            for row_i in test_index:
+                index_row = index_data[row_i]
                 if index_row.shape != (1,):
                     data_index = index_row[1]
                 else:
-                    data_index = test_index[0]
+                    data_index = row_i
 
                 # The data name may depend on the shape, and if so, the
                 # shape is appended to the name as base64 fields
@@ -691,7 +735,7 @@ class ArchiveSectionHDF5(ArchiveSection):
                     data.append(i_data)
 
             # If one object returned, remove the outermost list
-            if len(index_rows) == 1:
+            if len(test_index) == 1:
                 if data[0].shape == ():
                     data = np.array([data[0]])
                 else:
@@ -734,12 +778,7 @@ class ArchiveJSON(Archive):
 
             # Get the repository name from mainFileUri
             mainfile_uri = json_root["mainFileUri"]
-            if not mainfile_uri.startswith("nmd://"):
-                raise ValueError(
-                    "The mainFileUri in the JSON Archive file '{}' is invalid."
-                    .format(filepath)
-                )
-            repository_name = mainfile_uri[6:]
+            repository_name = mainfile_uri.split("://", 1)[1]
             repository_name = repository_name.split("/", 1)[0]
 
             root_section = {
@@ -763,6 +802,8 @@ class ArchiveJSON(Archive):
 
 
 class ArchiveSectionJSON(ArchiveSection):
+    """Represents a section inside a JSON-file.
+    """
     def __len__(self):
         return len(self._data)
 
@@ -775,6 +816,8 @@ class ArchiveSectionJSON(ArchiveSection):
 
     def items(self):
         for key, value in self._data.items():
+            if self.is_filtered(key) is None:
+                continue
             if isinstance(value, dict):
                 yield (key, ArchiveSectionJSON(value, "{}/{}".format(self._path, key), self._archive))
             else:
@@ -787,10 +830,15 @@ class ArchiveSectionJSON(ArchiveSection):
                 yield (key, value)
 
     def keys(self):
-        return self._data.keys()
+        for key in self._data.keys():
+            if self.is_filtered(key) is None:
+                continue
+            yield key
 
     def values(self):
         for key, value in self._data.items():
+            if self.is_filtered(key) is None:
+                continue
             if isinstance(value, dict):
                 yield ArchiveSectionJSON(value, "{}/{}".format(self._path, key), self._archive)
             else:
@@ -849,6 +897,9 @@ class ArchiveSectionJSON(ArchiveSection):
         is_section = False
         if path.startswith("section"):
             is_section = True
+        elif re.match(r'^x_\S+_section', path):
+            # code-specific section
+            is_section = True
 
         # If no index specified, try to get as concrete value or as a list of
         # sections

common/python/nomadcore/atoms2nomad.py

+import numpy as np
+
+from nomadcore.unit_conversion.unit_conversion import convert_unit
+
+
+def ase_atoms_to_section_system(backend, atoms, new_section=True):
+    """Add ASE Atoms object as metainfo to section_system.
+
+    If new_section is True, open and close a new section_system,
+    returning its gIndex."""
+
+    if new_section:
+        gIndex = backend.openSection('section_system')
+
+    backend.addArrayValues('atom_labels',
+                           np.array(atoms.get_chemical_symbols()))
+    backend.addArrayValues('atom_positions',
+                           convert_unit(atoms.positions, 'angstrom'))
+    backend.addArrayValues('simulation_cell',
+                           convert_unit(atoms.cell, 'angstrom'))
+    backend.addArrayValues('configuration_periodic_dimensions',
+                           np.array(atoms.pbc))
+
+    # Return system ref if we opened it, else None:
+    if new_section:
+        backend.closeSection('section_system', gIndex)
+        return gIndex

common/python/nomadcore/baseclasses.py

 """
-This module contains base classes that might help in building parsers for the
-NoMaD project.
+This module contains base classes for building parsers for the NoMaD project
+with an object-oriented approach.
 """
 from builtins import str
 from builtins import object
@@ -9,7 +9,9 @@ import os
 import copy
 import numpy as np
 import logging
+from future.utils import with_metaclass
 from abc import ABCMeta, abstractmethod
+
 from nomadcore.unit_conversion import unit_conversion
 from nomadcore.simple_parser import mainFunction
 from nomadcore.local_backend import LocalBackend
@@ -17,12 +19,10 @@ from nomadcore.local_meta_info import load_metainfo
 from nomadcore.caching_backend import CachingLevel
 from nomadcore.simple_parser import extractOnCloseTriggers, extractOnOpenTriggers
 from nomadcore.caching_backend import ActiveBackend
-import nomadcore.ActivateLogging
-from future.utils import with_metaclass
-logger = logging.getLogger("nomad")
+
+logger = logging.getLogger(__file__)
 
 
-#===============================================================================
 class ParserInterface(with_metaclass(ABCMeta, object)):
     """This class provides an interface for parsing. The end-user will
     typically only interact with an object that derives from this class. All
@@ -31,6 +31,10 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
     is given in the constructor as a dependency. If no backend is specified, a
     local backend that outputs results into a dictionary will be used.
 
+    This class controls if a specific parser version should be instantiated and
+    provides basic information about the parser, such as the parser-info
+    -dictionary and the name of the metainfo file.
+
     Attributes:
         main_parser: Object that actually does the parsing and is
             setup by this class based on the given contents.
@@ -38,55 +42,61 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
             This is contructed here and then passed onto the different
             subparsers.
     """
-    metainfo_env = None
 
-    def __init__(self, main_file, metainfo_to_keep=None, backend=None, default_units=None, metainfo_units=None, debug=False, log_level=logging.ERROR, store=True):
+    def __init__(
+            self, metainfo_to_keep=None, backend=None, default_units=None,
+            metainfo_units=None, debug=False, log_level=logging.ERROR, store=True):
         """
-    Args:
-        main_file: A special file that can be considered the main file of the
-            calculation.
-        metainfo_to_keep: A list of metainfo names. This list is used to
-            optimize the parsing process as optimally only the information
-            relevant to these metainfos will be parsed.
-        backend: An object to which the parser will give all the parsed data.
-            The backend will then determine where and when to output that data.
+        Arguments:
+            metainfo_to_keep: A list of metainfo names. This list is used to
+                optimize the parsing process as optimally only the information
+                relevant to these metainfos will be parsed.
+            backend: An object to which the parser will give all the parsed data.
+                The backend will then determine where and when to output that data.
         """
-        self.debug = debug
-        logger.setLevel(log_level)
+        try:
+            logger.setLevel(log_level)
+        except Exception:
+            # might fail on custom loggers
+            pass
+
         self.store = store
-        self.initialize(main_file, metainfo_to_keep, backend, default_units, metainfo_units)
+        self.debug = debug
+        self.metainfo_env = None
+        self.metaInfoEnv = None
+        self.initialize(metainfo_to_keep, backend, default_units, metainfo_units)
+
+    def setup_logger(self, new_logger):
+        global logger
+        logger = new_logger
 
-    def initialize(self, main_file, metainfo_to_keep, backend, default_units, metainfo_units):
+        # tell tests about received logger
+        new_logger.debug('received logger')
+
+    def initialize(self, metainfo_to_keep, backend, default_units, metainfo_units):
         """Initialize the parser with the given environment.
         """
         self.parser_context = ParserContext()
         self.parser_context.metainfo_to_keep = metainfo_to_keep
-        self.parser_context.main_file = main_file
         self.parser_context.file_service = FileService()
         self.parser_context.cache_service = CacheService()
         self.parser_context.parser_info = self.get_parser_info()
         self.main_parser = None
-
-        # Check that the main file exists
-        if not os.path.isfile(main_file):
-            logger.error("Couldn't find the main file {}. Check that the path is valid and the file exists on this path.".format(main_file))
-
-        # Setup the metainfo environment. All parsers that inherit from this
-        # class will have a static class attribute that will store the metainfo
-        # environment. This way every instance of a parser doesn't have to load
-        # the environment separately because it is identical for each instance.
-        if type(self).metainfo_env is None:
-            metainfo_env, warn = load_metainfo(self.get_metainfo_filename())
-            type(self).metainfo_env = metainfo_env
-            self.parser_context.metainfo_env = metainfo_env
-        else:
-            self.parser_context.metainfo_env = type(self).metainfo_env
-
-        # Initialize the backend. Use local backend if none given
-        if backend is not None:
-            self.parser_context.super_backend = backend(type(self).metainfo_env)
-        else:
-            self.parser_context.super_backend = LocalBackend(type(self).metainfo_env, debug=self.debug, store=self.store)
+        self.backend = backend
+
+        if self.metainfo_env is None:
+            metainfo_filename = os.path.basename(self.get_metainfo_filename())
+            from nomad.metainfo.legacy import python_package_mapping
+            import importlib
+            python_package_name, _ = python_package_mapping(metainfo_filename)
+            python_package_name = '.'.join(python_package_name.split('.')[:-1])
+            python_module = importlib.import_module(python_package_name)
+            metainfo = getattr(python_module, 'm_env')
+            self.metainfo_env = metainfo
+            self.metaInfoEnv = self.metainfo_env.legacy_info_env()
+
+        # Setup the metainfo environment.
+        self.parser_context.metainfo_env = self.metaInfoEnv
 
         # Check the list of default units
         default_unit_map = {}
@@ -106,7 +116,7 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
                 unit_conversion.ureg(unit)
 
                 # Check that the metaname is OK
-                meta = ParserInterface.metainfo_env.infoKinds.get(metaname)
+                meta = self.metaInfoEnv.infoKinds.get(metaname)
                 if meta is None:
                     raise KeyError("The metainfo name '{}' could not be found. Check for typos or try updating the metainfo repository.".format(metaname))
 
@@ -136,6 +146,14 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
         """
         pass
 
+    def get_mainfile_regex(self):
+        """Used to return the regular expression that is used to match the main
+        file.
+
+        Returns:
+            str: regular expression as a string.
+        """
+
     @abstractmethod
     def get_metainfo_filename(self):
         """This function should return the name of the metainfo file that is
@@ -160,15 +178,30 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
         """
         return None
 
-    def parse(self):
+    def parse(self, main_file):
         """Starts the actual parsing process, and outputs the results to the
         backend specified in the constructor.
         """
+        # Initialize the backend.
+        if self.backend is not None:
+            self.parser_context.super_backend = self.backend(self.metainfo_env)
+        else:
+            from nomad.parsing.legacy import Backend
+            self.parser_context.super_backend = Backend(self.metainfo_env)
+
+        # Check that the main file exists
+        if not os.path.isfile(main_file):
+            raise ValueError(
+                "Couldn't find the main file '{}'. Check that the path is valid "
+                "and the file exists on this path.".format(main_file)
+            )
+
+        self.parser_context.main_file = main_file
         self.setup_version()
         if not self.main_parser:
             logger.error("The main parser has not been set up.")
 
-        self.main_parser.parse()
+        self.main_parser.parse(main_file)
 
         # If using a local backend, the results will have been saved to a
         # separate results dictionary which should be returned.
@@ -178,7 +211,6 @@ class ParserInterface(with_metaclass(ABCMeta, object)):
             return None
 
 
-#===============================================================================
 class FileService(object):
     """Provides the interface to accessing files related to a calculation.
 
@@ -268,7 +300,6 @@ class FileService(object):
             return path
 
 
-#===============================================================================
 class RegexService(object):
     """
     Stores basic regex definitions that can be reused on multiple parsers.
@@ -281,7 +312,6 @@ class RegexService(object):
         self.eol = "[^\n]+"  # Regex for a single alphabetical letter
 
 
-#===============================================================================
 class AbstractBaseParser(with_metaclass(ABCMeta, object)):
     """A base class for all objects that parse contents from files.
 
@@ -299,8 +329,7 @@ class AbstractBaseParser(with_metaclass(ABCMeta, object)):
             caching backend.
     """
 
-    def __init__(self, file_path, parser_context):
-        self.file_path = file_path
+    def __init__(self, parser_context):
         self.parser_context = parser_context
         self.backend = parser_context.caching_backend
         self.super_backend = parser_context.super_backend
@@ -346,16 +375,18 @@ class AbstractBaseParser(with_metaclass(ABCMeta, object)):
             metainfo_units=self.parser_context.metainfo_units)
 
     def print_json_header(self):
-        self.super_backend.fileOut.write("[")
-        uri = "file://" + self.parser_context.main_file
-        self.backend.startedParsingSession(uri, self.parser_context.parser_info)
+        # self.super_backend.fileOut.write("[")
+        # uri = "file://" + self.parser_context.main_file
+        # self.backend.startedParsingSession(uri, self.parser_context.parser_info)
+        pass
 
     def print_json_footer(self):
-        self.backend.finishedParsingSession("ParseSuccess", None)
-        self.super_backend.fileOut.write("]\n")
+        # self.backend.finishedParsingSession("ParseSuccess", None)
+        # self.super_backend.fileOut.write("]\n")
+        pass
 
     @abstractmethod
-    def parse(self):
+    def parse(self, filepath):
         """Used to do the actual parsing. Inside this function you should push
         the parsing results into the Caching backend, or directly to the
         superBackend. You will also have to open new sections, but remember
@@ -363,7 +394,6 @@ class AbstractBaseParser(with_metaclass(ABCMeta, object)):
         """
 
 
-#===============================================================================
 class MainHierarchicalParser(AbstractBaseParser):
     """A base class for all parsers that parse a file using a hierarchy of
     SimpleMatcher objects.
@@ -388,19 +418,20 @@ class MainHierarchicalParser(AbstractBaseParser):
             onClose triggers, SimpleMatchers, caching levels, etc.
 
     """
-    def __init__(self, file_path, parser_context):
+    def __init__(self, parser_context):
         """
         Args:
             file_path: Path to the main file as a string.
             parser_context: The ParserContext object that contains various
                 in-depth information about the parsing environment.
         """
-        super(MainHierarchicalParser, self).__init__(file_path, parser_context)
+        super(MainHierarchicalParser, self).__init__(parser_context)
         self.root_matcher = None
         self.regexs = RegexService()
         self.cm = None
+        self.super_context = self
 
-    def parse(self):
+    def parse(self, filepath):
         """Starts the parsing. By default uses the SimpleParser scheme, if you
         want to use something else or customize the process just override this
         method in the subclass.
@@ -409,16 +440,16 @@ class MainHierarchicalParser(AbstractBaseParser):
                 mainFileDescription=self.root_matcher,
                 metaInfoEnv=self.parser_context.metainfo_env,
                 parserInfo=self.parser_context.parser_info,
-                outF=self.parser_context.super_backend.fileOut,
+                # outF=self.parser_context.super_backend.fileOut,
                 cachingLevelForMetaName=self.caching_levels,
-                superContext=self,
+                superContext=self.super_context,
                 onClose=self.on_close,
                 onOpen=self.on_open,
                 default_units=self.parser_context.default_units,
                 metainfo_units=self.parser_context.metainfo_units,
                 superBackend=self.parser_context.super_backend,
                 metaInfoToKeep=self.parser_context.metainfo_to_keep,
-                mainFile=self.parser_context.main_file)
+                mainFile=filepath)
 
     def startedParsing(self, fInName, parser):
         """Function is called when the parsing starts.
@@ -450,7 +481,6 @@ class MainHierarchicalParser(AbstractBaseParser):
         self.caching_levels.update(common_matcher.caching_levels)
 
 
-#===============================================================================
 class CommonParser(object):
     """
     This class is used as a base class for objects that store and instantiate
@@ -490,7 +520,6 @@ class CommonParser(object):
         return onOpen
 
 
-#===============================================================================
 class ParserContext(object):
     """A container class for storing and moving information about the parsing
     environment. A single ParserContext object is initialized by the Parser
@@ -510,7 +539,6 @@ class ParserContext(object):
         self.cache_service = None
 
 
-#===============================================================================
 class CacheObject(object):
     """Wraps a value stored inside a CacheService.
     """
@@ -536,7 +564,6 @@ class CacheObject(object):
         self._value = self.default_value
 
 
-#===============================================================================
 class CacheService(object):
     """A class that can be used to store intermediate results in the parsing
     process. This is quite similar to the caching system that is used by the

common/python/nomadcore/caching_backend.py

@@ -176,7 +176,7 @@ class CachingSectionManager(object):
         self.preOpened = preOpened
 
     def setSectionInfo(self, gIndex, references):
-        self.openSections[gIndex].references = [reference[x] for x in self.parentSectionNames]
+        self.openSections[gIndex].references = [references[x] for x in self.parentSectionNames]
 
     def get_latest_section(self):
         """Returns the latest opened section if it is still open.
@@ -261,14 +261,19 @@ class CachingSectionManager(object):
         except:
             raise Exception("Cannot add array values for metadata %s to section %d (%d) of %s, as it is not open" % (valueMetaInfo.name, gI, gIndex, self.metaInfo.name))
 
+
 class CachingDataManager(object):
     def __init__(self, metaInfo, superSectionManager, cachingLevel):
         self.metaInfo = metaInfo
         self.superSectionManager = superSectionManager
         self.cachingLevel = cachingLevel
 
+
 class ActiveBackend(object):
-    def __init__(self, metaInfoEnv, sectionManagers, dataManagers, superBackend, propagateStartFinishParsing = True, default_units=None, metainfo_units=None):
+    def __init__(
+            self, metaInfoEnv, sectionManagers, dataManagers, superBackend,
+            propagateStartFinishParsing=True, default_units=None, metainfo_units=None):
+
         self.__metaInfoEnv = metaInfoEnv
         self.sectionManagers = sectionManagers
         self.dataManagers = dataManagers
@@ -278,36 +283,62 @@ class ActiveBackend(object):
         self.metainfo_units = metainfo_units  # A mapping between metaname and an unit definition.
 
     @classmethod
-    def activeBackend(cls, metaInfoEnv, cachingLevelForMetaName = {}, defaultDataCachingLevel = CachingLevel.ForwardAndCache, defaultSectionCachingLevel = CachingLevel.Forward, superBackend = None,
-                      onClose = {}, onOpen = {}, propagateStartFinishParsing = True, default_units=None, metainfo_units=None):
+    def activeBackend(
+            cls,
+            metaInfoEnv,
+            cachingLevelForMetaName={},
+            defaultDataCachingLevel=CachingLevel.ForwardAndCache,
+            defaultSectionCachingLevel=CachingLevel.Forward,
+            superBackend=None,
+            onClose={}, onOpen={},
+            propagateStartFinishParsing=True,
+            default_units=None,
+            metainfo_units=None):
+
         for sectionName in onClose.keys():
-            if not sectionName in metaInfoEnv:
-                raise Exception("Found trigger for non existing section %s" % sectionName)
+            if sectionName not in metaInfoEnv:
+                raise Exception(
+                    "Found trigger for non existing section %s" % sectionName)
+
             elif metaInfoEnv.infoKinds[sectionName].kindStr != "type_section":
-                raise Exception("Found trigger for %s which is not a section but %s" %
-                                (sectionName, json.dumps(metaInfoEnv.infoKinds[sectionName].toDict(), indent=2)))
+                raise Exception(
+                    "Found trigger for %s which is not a section but %s" %
+                    (sectionName, json.dumps(metaInfoEnv.infoKinds[sectionName].toDict(), indent=2)))
+
         for sectionName in onOpen.keys():
-            if not sectionName in metaInfoEnv:
-                raise Exception("Found trigger for non existing section %s" % sectionName)
+            if sectionName not in metaInfoEnv:
+                raise Exception(
+                    "Found trigger for non existing section %s" % sectionName)
+
             elif metaInfoEnv.infoKinds[sectionName].kindStr != "type_section":
-                raise Exception("Found trigger for %s which is not a section but %s" %
-                                (sectionName, json.dumps(metaInfoEnv.infoKinds[sectionName].toDict(), indent=2)))
+                raise Exception(
+                    "Found trigger for %s which is not a section but %s" %
+                    (sectionName, json.dumps(metaInfoEnv.infoKinds[sectionName].toDict(), indent=2)))
+
         sectionManagers = {}
-        for ikNames, ik in metaInfoEnv.infoKinds.items():
+        for ik in metaInfoEnv.infoKinds.values():
             if ik.kindStr == "type_section":
-                parentS, parentO = list(metaInfoEnv.firstAncestorsByType(ik.name).get("type_section", [[],[]]))
+                parentS, parentO = list(
+                    metaInfoEnv.firstAncestorsByType(ik.name).get("type_section", [[], []]))
                 parentS.sort()
-                cachingLevel = reduce(CachingLevel.restrict, [cachingLevelForMetaName.get(x, defaultSectionCachingLevel) for x in ([ik.name] + parentS + parentO)])
+
+                cachingLevel = reduce(
+                    CachingLevel.restrict,
+                    [
+                        cachingLevelForMetaName.get(x, defaultSectionCachingLevel)
+                        for x in ([ik.name] + parentS + parentO)])
+
                 sectionManagers[ik.name] = CachingSectionManager(
-                    metaInfo           = ik,
-                    parentSectionNames = parentS,
-                    storeInSuper       = (cachingLevel == CachingLevel.ForwardAndCache or cachingLevel == CachingLevel.Cache or cachingLevel == CachingLevel.PreOpenedCache),
-                    forwardOpenClose   = (cachingLevel == CachingLevel.Forward or cachingLevel == CachingLevel.ForwardAndCache),
-                    preOpened = (cachingLevel == CachingLevel.PreOpenedCache or cachingLevel == CachingLevel.PreOpenedIgnore),
-                    onClose = onClose.get(ik.name, []),
-                    onOpen = onOpen.get(ik.name, []))
+                    metaInfo=ik,
+                    parentSectionNames=parentS,
+                    storeInSuper=(cachingLevel == CachingLevel.ForwardAndCache or cachingLevel == CachingLevel.Cache or cachingLevel == CachingLevel.PreOpenedCache),
+                    forwardOpenClose=(cachingLevel == CachingLevel.Forward or cachingLevel == CachingLevel.ForwardAndCache),
+                    preOpened=(cachingLevel == CachingLevel.PreOpenedCache or cachingLevel == CachingLevel.PreOpenedIgnore),
+                    onClose=onClose.get(ik.name, []),
+                    onOpen=onOpen.get(ik.name, []))
+
         dataManagers = {}
-        for ikNames, ik in metaInfoEnv.infoKinds.items():
+        for ik in metaInfoEnv.infoKinds.values():
             if ik.kindStr == "type_document_content" or ik.kindStr == "type_dimension":
                 superSectionNames = metaInfoEnv.firstAncestorsByType(ik.name).get("type_section", [[]])[0]
                 if not superSectionNames:
@@ -316,9 +347,15 @@ class ActiveBackend(object):
                     raise Exception("MetaInfo of concrete value %s has multiple superSections (%s)" %
                                     (ik.name, superSectionNames))
                 sectionManager = sectionManagers[superSectionNames[0]]
-                dataManagers[ik.name] = CachingDataManager(ik, sectionManager,
-                                                           CachingLevel.restrict(cachingLevelForMetaName.get(ik.name, defaultDataCachingLevel), CachingLevel.Forward if sectionManager.forwardOpenClose or sectionManager.preOpened else CachingLevel.Ignore))
-        return ActiveBackend(metaInfoEnv, sectionManagers, dataManagers, superBackend, propagateStartFinishParsing, default_units, metainfo_units)
+                dataManagers[ik.name] = CachingDataManager(
+                    ik, sectionManager,
+                    CachingLevel.restrict(
+                        cachingLevelForMetaName.get(ik.name, defaultDataCachingLevel),
+                        CachingLevel.Forward if sectionManager.forwardOpenClose or sectionManager.preOpened else CachingLevel.Ignore))
+
+        return ActiveBackend(
+            metaInfoEnv, sectionManagers, dataManagers, superBackend,
+            propagateStartFinishParsing, default_units, metainfo_units)
 
     def appendOnClose(self, sectionName, onClose):
         self.sectionManagers.onClose.append(onClose)
@@ -500,9 +537,9 @@ class ActiveBackend(object):
         cachingLevel = dataManager.cachingLevel
         if cachingLevel == CachingLevel.Forward or cachingLevel == CachingLevel.ForwardAndCache:
             if self.superBackend:
-                self.superBackend.addArrayValues(metaName, values, gIndex)
+                self.superBackend.addArrayValues(metaName, values, gIndex=gIndex)
         if cachingLevel == CachingLevel.ForwardAndCache or cachingLevel == CachingLevel.Cache or cachingLevel == CachingLevel.PreOpenedCache:
-            dataManager.superSectionManager.addArrayValues(dataManager.metaInfo, values, gIndex)
+            dataManager.superSectionManager.addArrayValues(dataManager.metaInfo, values, gIndex=gIndex)
 
     def convertScalarStringValue(self, metaName, strValue):
         """converts a scalar string value of the given meta info to a python value"""

common/python/nomadcore/compact_sha.py

@@ -3,6 +3,7 @@ import hashlib
 import base64
 
 class CompactHash(object):
+    """compact sha can be used to calculate nomad gids"""
     def __init__(self, proto):
         self._proto = proto
 
@@ -18,14 +19,27 @@ class CompactHash(object):
             data=data.encode("utf-8")
         return self._proto.update(data)
 
+    def gid(self, prefix=""):
+        """returns a nomad gid with the given prefix"""
+        return prefix + self.b64digest()[:28]
+
     def __getattr__(self, name):
         return getattr(self._proto, name)
 
 def sha224(*args, **kwargs):
+    """CompactSha using sha224 for the checksums (non standard)"""
     return CompactHash(hashlib.sha224(*args,**kwargs))
 
-def sha512(*args, **kwargs):
-    return CompactHash(hashlib.sha512(*args,**kwargs))
+def sha512(baseStr=None,*args,**kwargs):
+    """Uses sha512 to calculate the gid (default in nomad)
+
+    If you pass and argument it is immediately added to the checksum.
+    Thus sha512("someString").gid("X") creates a gid in one go"""
+    sha=CompactHash(hashlib.sha512(*args,**kwargs))
+    if baseStr is not None:
+        sha.update(baseStr)
+    return sha
 
 def md5(*args, **kwargs):
+    """CompactSha using md5 for the checksums (non standard)"""
     return CompactHash(hashlib.md5(*args,**kwargs))

common/python/nomadcore/configurationreading.py

 import ase.io
 import ase.io.formats
-import mdtraj as md
-import mdtraj.formats
 import numpy as np
 import logging
+
+try:
+    import mdtraj as md
+    import mdtraj.formats
+except ImportError:
+    logging.getLogger("nomad").warn('MDTraj is not installed.')
+
 logger = logging.getLogger("nomad")
 
 

common/python/nomadcore/cube_reader.py

+import numpy as np
+from ase.io import read
+
+from nomadcore.atoms2nomad import ase_atoms_to_section_system
+
+class CubeError(OSError):
+    pass
+
+
+def read_cube_file(backend, file_name):
+    try:
+        d = read(file_name, format = 'cube', full_output = True)
+
+    except Exception as err:
+        raise CubeError(err)
+
+    data = d['data']
+    atoms = d['atoms']
+    origin = d['origin']
+    nx, ny, nz = data.shape
+    displacements = np.array([atoms.cell[i]/data.shape[i] for i in range(3)])
+
+    system = ase_atoms_to_section_system(backend, atoms)
+
+    singleconfig = backend.openSection('section_single_configuration_calculation')
+    volumetric = backend.openSection('section_volumetric_data')
+
+    backend.addValue('volumetric_data_nx', nx)
+    backend.addValue('volumetric_data_ny', ny)
+    backend.addValue('volumetric_data_nz', nz)
+
+    backend.addArrayValues('volumetric_data_origin', origin)
+    backend.addArrayValues('volumetric_data_displacements', displacements)
+
+    backend.addValue('volumetric_data_multiplicity', 1)
+    backend.addArrayValues('volumetric_data_values', data[None])
+    backend.closeSection('section_volumetric_data', volumetric)
+
+    backend.addValue('single_configuration_calculation_to_system_ref', system)
+    backend.closeSection('section_single_configuration_calculation', singleconfig)

common/python/nomadcore/local_meta_info.py

@@ -10,17 +10,20 @@ import json
 import os, re
 from nomadcore.json_support import jsonCompactS, jsonCompactD, jsonIndentD
 from io import open
+# We removed the old nomad-meta-info project and the metainfo module respectively. Most of the
+# functionality in this module wont work anymore.
+# import metainfo
 """objects to handle a local InfoKinds with unique name (think self written json)"""
 
 class InfoKindEl(object):
     """Info kind (tipically from a file, without shas but with locally unique names)"""
-    __slots__ = ["name","description","kindStr","units","superNames","dtypeStr", "repeats", "shape", "extra_args"]
+    __slots__ = ["name","description","kindStr","units","superNames","dtypeStr", "repeats", "shape", "extra_args", "package"]
     IGNORE_EXTRA_ARGS = 1
     ADD_EXTRA_ARGS = 2
     RAISE_IF_EXTRA_ARGS = 3
 
     def __init__(self, name, description, kindStr = "type_document_content", units = None, superNames = None,
-            dtypeStr = None, shape = None, extraArgsHandling = ADD_EXTRA_ARGS, repeats = None, **extra_args):
+            dtypeStr = None, shape = None, package = None, extraArgsHandling = ADD_EXTRA_ARGS, repeats = None, **extra_args):
         if superNames is None:
             superNames = []
         self.name = name
@@ -29,6 +32,7 @@ class InfoKindEl(object):
         self.superNames = superNames
         self.units = units
         self.dtypeStr = dtypeStr
+        self.package = package
         if dtypeStr in ["None", "null"]:
             self.dtypeStr = None
         self.shape = shape
@@ -183,14 +187,20 @@ class RelativeDependencySolver(object):
         self.deps = {}
 
     def __call__(self, infoKindEnv, source, dep):
-        if "relativePath" not in dep:
-            raise Exception('Invalid dependency for relativeDependencySolver there must be a relativePath')
-        basePath = source.get('path')
+        if "metainfoPath" in dep:
+            basePath = metainfo.__file__
+            path = dep["metainfoPath"]
+        elif "relativePath" in dep:
+            basePath = source.get('path')
+            path = dep["relativePath"]
+        else:
+            raise Exception('Invalid dependency for relativeDependencySolver there must be a relativePath or metainfoPath')
+
         if basePath:
             baseDir = os.path.dirname(os.path.abspath(basePath))
         else:
             baseDir = os.getcwd()
-        dPath = os.path.realpath(os.path.join(baseDir, dep['relativePath']))
+        dPath = os.path.realpath(os.path.join(baseDir, path))
         if dPath in self.deps:
             return self.deps[dPath]
         depInfo = None
@@ -264,7 +274,7 @@ class InfoKindEnv(object):
     def addInfoKindEl(self, infoKind):
         if infoKind.name in self.infoKinds and infoKind != self.infoKinds[infoKind.name]:
             raise Exception('InfoKindEnv has collision for name {0}: {1} vs {2}'
-                    .format(infoKind.name, infoKind, self.infoKinds[infoKind.name]))
+                    .format(infoKind.name, infoKind.package, self.infoKinds[infoKind.name].package))
         self.infoKinds[infoKind.name] = infoKind
 
     def addDependenciesFrom(self, infoKindEnv):
@@ -456,7 +466,7 @@ class InfoKindEnv(object):
 
     def toJsonList(self, withGids):
         infoKinds = list(self.infoKinds.keys())
-        infoKinds.sort(lambda x, y: self.compareKeys(x.name, y.name))
+        # infoKinds.sort(lambda x, y: self.compareKeys(x.name, y.name))
         return [self.infoKinds[x].toDict(self,
                 self if withGids else None) for x in infoKinds]
 
@@ -522,6 +532,7 @@ class InfoKindEnv(object):
                 gidToCheck[ii["name"]] = toCheck
                 del val['superGids']
             val['extraArgsHandling'] = extraArgsHandling
+            val['package'] = self.name
             ikEl = InfoKindEl(**val)
             if not oldVal is None and ikEl != oldVal:
                 overwritten.append((oldVal, ikEl))
@@ -668,7 +679,10 @@ def load_metainfo(filename, dependencyLoader=None, extraArgsHandling=InfoKindEl.
         Tuple containing the metainfo environment, and any possible warnings
         that were encountered in the loading.
     """
-    path = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../../nomad-meta-info/meta_info/nomad_meta_info/{}".format(filename)))
+    if os.path.isfile(filename):
+        path = filename
+    else:
+        path = os.path.join(os.path.dirname(metainfo.__file__), filename)
     return loadJsonFile(path, dependencyLoader, extraArgsHandling, uri)
 
 def loadJsonStream(fileStream, name = None, dependencyLoader = None, extraArgsHandling = InfoKindEl.ADD_EXTRA_ARGS, filePath = None, uri = None):

common/python/nomadcore/md_data_access/LICENSE

+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2017 Berk Onat
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.