Transparent, restriction aware staging and public file structures
We need an abstraction for handling all files related to an upload.
There are two different structures for uploads in two different states: staging and public. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a almost readonly (beside metadata) storage.
fs/staging/<upload>/metadata/<calc>.json
/raw/**
/archive/<calc>.hdf5
/.frozen
/.public
/.restricted
fs/public/<upload>/metadata.json.gz
/raw-public.bagit.zip
/raw-restricted.bagit.zip
/archive-public.hdf5.zip
/archive-restricted.hdf5.zip
This is an complex issue with multiple sub-tasks
-
implementing/testing an respective the uploads.py module -
adopt processing and api accordingly -
make the raw file API restriction aware -
do potential adaptations to the staging GUI
This is a prerequisite for building reasonable mirror functionality. It would allow to move calc (and potentially also user) metadata out of the repository db, which also requires to adopt the repoTool.
Edited by Markus Scheidgen