Add section defs

assigned to @thchang

Updated data arrangement. Please see if it suits the needs. @dsikter

Search is working.

requested review from @mscheidg

@mscheidg Please have a look.

What we discussed was slightly different:

create a section_defs instance for each compatible (the used definitions + all recursive base sections)
each instance has definition_id:str, definition_qualified_name: str, used_directly: bool)
used_directly is True if there is a direct instance of this def in the archive, False if there def is only in the list because its the base section of another used def

Was just thinking of the use case where we have a parent definition A and a child-def B, and we're opening the new search dialogue to find data candidates of type A. With this format, it would be possible to, with a single query, determine (rather easily) if an entry contains something matching A, but not exactly what or where. So in the end you would still need to import the schemas, and I guess also the archive data (all of it?) to find all the matches it contains. But it's maybe hard to avoid this?

The qualified name is not always meaningful if it is definition file with pure definitions (with no data).

With definition_id, Won't it suffice to make another API call to metainfo endpoint to get definition ? Loading files from disk is feasible for sure.

Hm, these are two different cases, when looking for data and when looking for definitions. An entry containing only definitions and no data would have a very short section_defs list, basically just the "schema-schema sections", right? I.e.

nomad.metainfo.metainfo.Package
nomad.metainfo.metainfo.Section
nomad.metainfo.metainfo.Quantity
nomad.metainfo.metainfo.SubSection
nomad.metainfo.metainfo.Category
nomad.metainfo.metainfo.Attribute

Maybe we need to put some info about an entry's definitions in a separate field with some other structure.

Not sure if I understand this discussion.

This metadata (section_defs) is only there to find instances of definitions. So the "reference target search dialog" uses this to find entries that contain possible targets (instances of the reference type). The dialog still needs to show a list of the actual instances and let the user choose. But this cannot come from section_defs; this information can only come from the archive itself.

For me this thread is solved? Objections?

Yeah, I think we can close it. My point was just that with this impl it will be easy to find entries that contain matches, but to actually locate the matches in the data we need to load and look through the entire archive, and we've talked about how we want to avoid loading potentially very big archives completely. But I think it's hard to avoid needing to scan the entire archive to find these matches. The only way I can see this could be avoided is by creating some huge index containing all archive data (not just the metadata), which I assume we don't want to do.