Transparent, restriction aware staging and public file structures
We need an abstraction for handling all files related to an upload.
There are two different structures for uploads in two different states: staging and public. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a almost readonly (beside metadata) storage.
fs/staging/<upload>/metadata/<calc>.json
/raw/**
/archive/<calc>.hdf5
/.frozen
/.public
/.restricted
fs/public/<upload>/metadata.json.gz
/raw-public.bagit.zip
/raw-restricted.bagit.zip
/archive-public.hdf5.zip
/archive-restricted.hdf5.zip
This is an complex issue with multiple sub-tasks
-
implementing/testing an respective the uploads.py module -
adopt processing and api accordingly -
make the raw file API restriction aware -
do potential adaptations to the staging GUI
This is a prerequisite for building reasonable mirror functionality. It would allow to move calc (and potentially also user) metadata out of the repository db, which also requires to adopt the repoTool.