In this page is explained how the data are loaded to elasticsearch and how the api/(query,download,autocomplete) works:
- Extraction of information from Parsed data;
- In order extract the information and ingest it by elasticsearch, for each parsed file we run this command:
cd /nomad/nomadlab/elasticsearch/elasticsearch-5.5.0/nomad
sh runElastic.sh archiveGID
################################
##hdf5lib=lib
#jar=nomadElastic.jar
#java -Djava.library.path=$hdf5lib -jar $jar $1
################################The nomadElastic.jar is generated from sbt nomad-lab-base
cd nomad-lab-base
sbt elasticsearch/assembly The source code of this jar can be find at:
nomad-lab/nomad-lab-base/core/src/main/scala/eu/nomad_lab/normalize/ElasticsearchData.scala- After running the command a new folder with the new json files is generated at:
/nomad/nomadlab/normalized/elastic/archiveGID3. The new data can be ingested by elasticsearch running the following command:cd /nomadtmp/elasticsearch/elasticsearch-5.5.0/nomad
sh runUploadJsonBulk.sh archiveGID4. There is a script to generate and ingest a set of archiveGID:cd /nomadtmp/elasticsearch/elasticsearch-5.5.0/nomad
sh runElasticByFile.sh index.txt # (index.txt contains all the archiveGID, one per line)Elasticsearch recepies:
- Get all the values of variable (e.g. atom_symbols):
curl -s -XGET "http://nomad-flink-01:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
"size": 0,
"aggs" : {
"counter" : {
"terms" : { "field" : "atom_symbols", "size" : 4000 }
}
}
}
'Add more information metadata to elasticsearch:
We have to add some code to: nomad-lab/nomad-lab-base/core/src/main/scala/eu/nomad_lab/normalize/ElasticsearchData.scala
- We have to create the metadata on "Section: Creating Metadata" (see comments on code)
- We have to add the new metadata to the json file on "Section: Creating Json" (see comments on code)
Deployment on Kubernets
- Create the docker image
cd nomad-lab-base
sbt webservice/docker- Push the docker image (be sure that a new version number is generated)
docker push labdev-nomad.esc.rzg.mpg.de:5000/nomadlab/nomadwebservice:<version>
#current version v1.8.0-119-gb50befb3-dirty- Create the kubernetes config files
sbt "kubernetes/run --config flink --all"- Adjust the number of replicas (file: kubeGen/api-server-rc.yaml):
"replicas": 1 - Load new kubernetes services:
/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 delete rc api-rc-flink
/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 create -f kubeGen/api-server-rc.yaml- Check that our service is running
/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 get services- Check that nginx is redirected properly on labtest.