A "script" that quickly allows to migrate new data to a certain deployment.
The scenario is that we use a deployment of nomad-FAIRDI alongside the old python API. Once the migration of old data is completed and we want to discontinue the old python API, we need a script to move new data (produced by either the old python API or the mimicing FARIDI deployment) to the fully migrated deployment.
How to do it
requirements
- a CLI admin command to pull data from a different source deployment via the source deployments API
- only available for the admin user (for now) and he can also pull restricted data
- it is enough if the upload files are provides via path; the command can make local copy (or even mv); no need to stream the actual data yet
- this can be extended to a actual mirror functionality later
tasks
The 'mirror' API endpoint to export data
-
endpoint uses upload as param -
endpoint provides metadata of upload + all calcs + paths of upload files
The admin command to import data
-
the admin command that uses the search endpoint to aggregate over all uploads -
the admin command uses the 'mirror' endpoint to get data -
the admin command puts the data into mongo, es, and the file system
'mirror' endpoint results
{
"upload_id": "<id>",
"upload": {
/* ... */
),
"calcs": [
/* ... */
]
"upload_files": "<path to public upload files>"
}