Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • nomad-FAIR nomad-FAIR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 217
    • Issues 217
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 28
    • Merge requests 28
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • nomad-labnomad-lab
  • nomad-FAIRnomad-FAIR
  • Issues
  • #161
Closed
Open
Issue created Jun 25, 2019 by Markus Scheidgen@mscheidgOwner5 of 5 checklist items completed5/5 checklist items

A "script" that quickly allows to migrate new data to a certain deployment.

The scenario is that we use a deployment of nomad-FAIRDI alongside the old python API. Once the migration of old data is completed and we want to discontinue the old python API, we need a script to move new data (produced by either the old python API or the mimicing FARIDI deployment) to the fully migrated deployment.

How to do it

requirements

  • a CLI admin command to pull data from a different source deployment via the source deployments API
  • only available for the admin user (for now) and he can also pull restricted data
  • it is enough if the upload files are provides via path; the command can make local copy (or even mv); no need to stream the actual data yet
  • this can be extended to a actual mirror functionality later

tasks

The 'mirror' API endpoint to export data

  • endpoint uses upload as param
  • endpoint provides metadata of upload + all calcs + paths of upload files

The admin command to import data

  • the admin command that uses the search endpoint to aggregate over all uploads
  • the admin command uses the 'mirror' endpoint to get data
  • the admin command puts the data into mongo, es, and the file system

'mirror' endpoint results

{
    "upload_id": "<id>",
    "upload": {
       /* ... */
    ),
    "calcs": [
       /* ... */
    ]
    "upload_files": "<path to public upload files>"
}
Edited Jul 25, 2019 by Markus Scheidgen
Assignee
Assign to
Time tracking