Parsing

Parsing goal is to extract the interesting pieces of information from an output and prepare them for further processing.

It is necessarily dependent on the source output (i.e. on the code used for the calculation). Writing a parser is a bit tedious and needs specialized knowledge of the code (depending on how well it is documented), its units, and the ability to map its output to the common model we use.

Being so code dependent it means that any change to the parser infrastructure is multiplied by 50 (or however many we have), and has to be done keeping in account the unique way n that code to get the different quantities.

Finally parsing performance is very important, as it has to go through basically all the data we have.

Thus it should be clear that

1. we do not want to do changes that touch all the parsers at once, we might want to update a single parser from time to time, but we definitely want to avoid the factor 50 or more when doing changes
2. parsers should be fast
3. when we need a parser we should try to be able to use our parsers (again to avoid the factor 50, or having access to just a small subset)

Standardized data

A common parser usage looks for few quantities and then does some computation with them. The idea conversion layer is to prepare basically all the possible quantities performing all conversions to give them in a good standardized form to be used for the further processing.

Here we have already an important issue: how you get the data out of this standardized form? In a way this will probably entail some form of parsing, but is should be much faster than the original parsing and conversion for it to make sense.

Here we choose hdf5 as a good compromise: it can store large amounts of data, and has indexes, so that skipping to a given piece of data is quick. It has two main drawbacks:

1. it is relatively slow to create/write a single arrays, so storing many small arrays (maybe with only a single value) is a bit expensive
2. it needs the hdf/netcdf library to be read (which can be used basically from any language, on any machine)

So we have json to complement it, as it is human readable, it can be parsed even by hand, and plays well with the web browser.

Direct parsing

If the data was already parsed to this standarized form, then further analysis is faster, but if you start from the raw data it adds an extra step. It is too early to see all applications but direct parsing (for example if one wants to some data now) is so commonly used that will probably still be relevant some time.

Ideally we want to be able to use our parser also for this kind of applications. The major hurdle in this is the time lost in parsing and writing out data that is not relevant for the current application (think for example parsing the wavefunctions). If one parses only the quantities needed there is still a bit of overhead, but it is small as the parsing of the output format is optimized to be fast, and well offset by the flexibility that one gains (in some cases one can remove also this overhead).

What we do is having a way for the parser to tell us what he can extract, and one to tell the parser to prepare a specialized parser just for the pieces we are interested it.

Metadata

To define what is extracted, and model it from the conceptual point of view we use metadata. Our metadata model is pretty flexible, can accommodate extensions and is described in NomadMetaInfo.

This is a central thing to make the whole system interoperable.

Large changes of the metadata model will require changes to the parsers, but description changes, and additions will not need any real changes. In any case all the parser always specify the metadata they use, so changes can be detected and either handled automatically or added as warning.

Reducing the factor 50

To reduce the work that needs to be repeated a classical approach is to write a library and reusable components.

Unfortunately between a parser for Xml and one for free text there is little in common. Also some want to write in python (as it is a nice language, but can easily become ~10 times slower than C/java/scala), others (like octopus) have already a C/yacc based parser for their input that they want to reuse.

Still some things can be abstracted (for example metadata handling, and writing to the common format), and for a class of parsers one can have a common infrastructure.

The ideal approach is to have declarative parsers, that specify the structure and position of the things to extract and their mapping. Then the actual parser is "built" reading this specification. This allows lot of flexibility on the optimization of the parsers and to evolve them (one can improve the parser generation independently from the actual description of the things to parse.

Such an approach is what I did try to use for fhi-aims (it is not perfectly declarative, but close), and it will hope allow its evolution and improvement. The code written should also be reusable to parse other outputs that have a "free form" output like fhi-aims.

Xml parsing can also make the declarative approach easy, by having a mapping of the entities to our abstract model.

Indeed one can then even write a parser generator that calls functions with the parsed values without ever writing them out, removing the overhead of passing through a standarized format.

Conversion layer

The parsers are an important component of the conversion layer, but are not the conversion layer. Indeed (again to avoid the factor 50) the parsers should do the minimal conversion possible to bring the data to a model that is well defined, further conversions should be done by a common code, so they need to implemented only once.

For the conversion layer this driver is written in scala and takes care of identifying the files up to the point in which it knows which parser to use, then take the parsing output and normalize it.

Practical parsing

Event based parser

A parser is expected to be at lowest level a push parser, that calls a series of callbacks (or if you prefer that emits a series of parsing events).

The callbacks available to the parser are defined in eu.nomad_lab.parsing.PaserBackendExternal, and the corresponding possible parsing events are those defined in eu.nomad_lab.parsing.ParseEvents.

In python this is in python-common parser_backend.py.

The idea is that you connect the backend in such a way that it writes to stdout (and take care of writing eventual error messages to stderr).

backend = JsonParseEventsWriterBackend(metaInfoEnv, sys.stdout)

then you start with

backend.startParsingSession(fileUri, filePath, { "name":"<parser_name>", "version":"<parser_version>"})

and probably you open some sections

runId = backend.openSection("section_run") #place for a whole run
methodId = backend.openSection("section_method") # description of the method used
sysDescId = backend.openSection("section_system_description") # description of the system
spId = backend.openSection("section_single_point_evaluation")

then you add values backend.addValue("program_name", "bbb") backend.addValue(" program_version", "xx") they can be in any order of any open section (run, single point, ....)

you might open further sections

scfId = backend.openSection("section_scf_iteration")

and later close it with

backend.closeSection("section_scf_iteration", scfId)

currently you have to pass the Id, but we might decide to make it superfluous if there is only one section of that type open (most cases).

Python Api

Free form parser

It is possible to define nested sections and using regular expressions. This gives a relatively declarative definition of the parser can be done. This allows one to compile different parsers and reaching good performance.

simple_parser.py takes this approach.

The idea for it is to have matchers that actually consist of a regular expression, possibly some sub matchers that match subsequent lines and finally an optional closing regular expression.

These give a way to organize a group of regular expressions, specifying the sequence they are expected to be matched.

The actual parsing by default tries to match forward, or fall back at the to an upper level, this copes reasonably well with different verbosity and optional values, or extra debugging.

There are flags to allow for repetitions or matching lines in any order

Once a match is found the name of the group is used to get the corresponding metadata and the value in the group is assumed to correspond to it.

Limitations: algorithmic parsers (if you read x parse y otherwise,...) have to be handled adHoc.

"Compilation" of finite state machine to match it can be optimized, or generate debugging parsers

Xml Parsing

If an xml schema is available it can be used to associate properties to simple transformations of some entries, but probably

Current Status

FHI parser was written with this approach in mind, but has to be fixed for the changes that were done when implementing in practice.