Nexus examples downloads

I replaced the downloads in generate_example_uplods.sh with respective downloads that are done during processing. We are about to drastically reduce the size of our images and I do not want to loose these gains immediately. This also fixes some of the issues in the examples and triggers .nxs processing while processing ELNs.

added 1 commit

• 9f2b0da5 - Added a download section to automatize adding urls to uploads.

Compare with previous version

changed target branch from nexus-eln-examples to develop

added 4 commits

• 004bb21d - Disable celery stdout log forwarding.
• 43438d4f - Bugfixes for downloading and generating mainfiles.
• 78614bcf - Adding download archives to replace example files in nomad image.
• 8aa34586 - Use canonical setup strategy for nexus parser.

Compare with previous version

@sanbrock @sshabih

What has been done:

• This adds a schema that allows to create simple entries with download instructions.
• This adds download instructions to all the example uploads with large files. There is basically a small downloads.archive.yaml in most the examples. This triggers a download of the files and a reprocessing of the ELNs that depend on the downloaded files.
• Clean the generate_example_uplods.sh script
• Removed specific upload ids from some of the example ELN files
• The nexus_data_converter schema now triggers processing for generated .nxs files

Problems:

• Please rebase !866 (merged) with develop to make it easier to compare this with !866 (merged) and potentially merge this into !866 (merged)
• There are some unit related problems with parsing some of the generated nexus files with the new nexus parser. Someone in Area B should test and discuss with @thchang
• There are related PRs/MRs on the north and nexus parser projects that should be merged after this MR
• I am still fixing the install tests
• Once !795 (merged) is done, we probably need to adapt the requirements/setup of nexus parser project again

The rebase is done on !866 (merged).

changed the description

In case of missed, I had the issue drafted on github: https://github.com/nomad-coe/nomad-parser-nexus/issues/53

changed target branch from develop to nexus-eln-examples

mentioned in merge request !866 (merged)

added 20 commits

• 8aa34586...456bf071 - 15 commits from branch nexus-eln-examples
• 65ef767e - Added a download section to automatize adding urls to uploads.
• b9ab624d - Disable celery stdout log forwarding.
• 93e4b88e - Bugfixes for downloading and generating mainfiles.
• 44637bfd - Adding download archives to replace example files in nomad image.
• 7752b49a - Use canonical setup strategy for nexus parser.

Compare with previous version

assigned to @sshabih

@sshabih @dflor @sanbrock Please have a look and let me know if this works for you.

Worked for me after I updated the nexus parser submodule to the version in nexus-eln-examples. The change somehow wasn't picked up by the rebase.

Not sure what revision you need. Can you add it as a commit to this branch?

However, this https://github.com/nomad-coe/nomad-parser-nexus/pull/55 is important to make the CI here pass.

Yes I will add it. However, I think the branch for PR 55 is outdated as we switched to pyproject.toml w/o setup.py in master. The CI/CD seems to fail on generating the docs only (https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1880278) in the nexus-eln-examples branch (which works fine locally for me) - or is this the actual problem from setup.py?

What exactly is needed in nomad? Then I can try to add it to the toml file.

Edit: I've just seen that the exact same command is fine in the build stage in the other branch as well, which is failing here because of the older submodule. So I don't really understand why mkdocs is failing in dev setup tests there.

We just manually merged all the needed changes from this branch into !866 (merged). This, https://github.com/nomad-coe/nomad-parser-nexus/pull/55, is reverting all the steps we made towards the newer install mechanism. To get the image to build until we have the new install system up in Nomad, we can keep the manual install steps for nexusutils in the Docker image and CI tests for now.

We'll get !866 (merged) up and running with the changes from here. Then we can sync this branch to if you like.

Well, when @afekete manages to finish his work on the nomad-fair side of the new install mechanism, https://github.com/nomad-coe/nomad-parser-nexus/pull/55 will hopefully not be needed anymore.

Can't we just keep the manual installs we added here? https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/blob/4ea3010bbe417b9172214e055be7cc3da9f1d284/Dockerfile#L72

It really doesn't change the final Docker image much for now. It should also be simple to remove this bit here than adding a setup.py to nexusutils. I mean I've made the build script in Nomad work and have nexusutils sort of ready for the new system.

If you get it working yes, feel free to try. I replaced them because I could not build the image this way.

deleted the nexus-eln-examples branch. This merge request now targets the develop branch

closed