Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • nomad-FAIR nomad-FAIR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 296
    • Issues 296
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 41
    • Merge requests 41
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
    • Model experiments
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • nomad-labnomad-lab
  • nomad-FAIRnomad-FAIR
  • Issues
  • #80
Closed
Open
Issue created Dec 23, 2018 by Markus Scheidgen@mscheidgOwner

Transparent, restriction aware staging and public file structures

We need an abstraction for handling all files related to an upload.

There are two different structures for uploads in two different states: staging and public. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a almost readonly (beside metadata) storage.

fs/staging/<upload>/metadata/<calc>.json
                    /raw/**
                    /archive/<calc>.hdf5
                    /.frozen
                    /.public
                    /.restricted
fs/public/<upload>/metadata.json.gz
                  /raw-public.bagit.zip
                  /raw-restricted.bagit.zip
                  /archive-public.hdf5.zip
                  /archive-restricted.hdf5.zip

This is an complex issue with multiple sub-tasks

  • implementing/testing an respective the uploads.py module
  • adopt processing and api accordingly
  • make the raw file API restriction aware
  • do potential adaptations to the staging GUI

This is a prerequisite for building reasonable mirror functionality. It would allow to move calc (and potentially also user) metadata out of the repository db, which also requires to adopt the repoTool.

Edited Jan 14, 2019 by Markus Scheidgen
Assignee
Assign to
Time tracking