Skip to content

GitLab

Closed
Created by Markus Scheidgen@mscheidgOwner4 of 4 tasks completed4/4 tasks

Transparent, restriction aware staging and public file structures

We need an abstraction for handling all files related to an upload.

There are two different structures for uploads in two different states: staging and public. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a almost readonly (beside metadata) storage.

fs/staging/<upload>/metadata/<calc>.json
                    /raw/**
                    /archive/<calc>.hdf5
                    /.frozen
                    /.public
                    /.restricted
fs/public/<upload>/metadata.json.gz
                  /raw-public.bagit.zip
                  /raw-restricted.bagit.zip
                  /archive-public.hdf5.zip
                  /archive-restricted.hdf5.zip

This is an complex issue with multiple sub-tasks

  • implementing/testing an respective the uploads.py module
  • adopt processing and api accordingly
  • make the raw file API restriction aware
  • do potential adaptations to the staging GUI

This is a prerequisite for building reasonable mirror functionality. It would allow to move calc (and potentially also user) metadata out of the repository db, which also requires to adopt the repoTool.

Edited by Markus Scheidgen