Transparent, restriction aware staging and public file structures
We need an abstraction for handling all files related to an upload.
There are two different structures for uploads in two different states: staging and public. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a almost readonly (beside metadata) storage.
fs/staging/<upload>/metadata/<calc>.json /raw/** /archive/<calc>.hdf5 /.frozen /.public /.restricted fs/public/<upload>/metadata.json.gz /raw-public.bagit.zip /raw-restricted.bagit.zip /archive-public.hdf5.zip /archive-restricted.hdf5.zip
This is an complex issue with multiple sub-tasks
- implementing/testing an respective the uploads.py module
- adopt processing and api accordingly
- make the raw file API restriction aware
- do potential adaptations to the staging GUI
This is a prerequisite for building reasonable mirror functionality. It would allow to move calc (and potentially also user) metadata out of the repository db, which also requires to adopt the repoTool.