A "script" that quickly allows to migrate new data to a certain deployment.

The scenario is that we use a deployment of nomad-FAIRDI alongside the old python API. Once the migration of old data is completed and we want to discontinue the old python API, we need a script to move new data (produced by either the old python API or the mimicing FARIDI deployment) to the fully migrated deployment.

How to do it

requirements

• a CLI admin command to pull data from a different source deployment via the source deployments API
• only available for the admin user (for now) and he can also pull restricted data
• it is enough if the upload files are provides via path; the command can make local copy (or even mv); no need to stream the actual data yet
• this can be extended to a actual mirror functionality later

tasks

The 'mirror' API endpoint to export data

• endpoint uses upload as param
• endpoint provides metadata of upload + all calcs + paths of upload files

The admin command to import data

• the admin command that uses the search endpoint to aggregate over all uploads
• the admin command uses the 'mirror' endpoint to get data
• the admin command puts the data into mongo, es, and the file system

'mirror' endpoint results

{
    "upload_id": "<id>",
    "upload": {
       /* ... */
    ),
    "calcs": [
       /* ... */
    ]
    "upload_files": "<path to public upload files>"
}