Archive/elasticsearch/elasticsearch-query.md

+In this page is explained how the data are loaded to elasticsearch and
+how the api/(query,download,autocomplete)  works:
+
+
+1. Extraction of information from Parsed data;
+   1. In order extract the information and ingest it by elasticsearch, for each parsed file we run this command:
+```
+cd /nomad/nomadlab/elasticsearch/elasticsearch-5.5.0/nomad
+sh runElastic.sh archiveGID
+################################
+##hdf5lib=lib
+#jar=nomadElastic.jar 
+#java -Djava.library.path=$hdf5lib -jar $jar  $1
+################################
+```
+The nomadElastic.jar is generated from sbt nomad-lab-base
+```
+cd nomad-lab-base
+sbt elasticsearch/assembly 
+```
+The source code of this jar can be find at:
+```
+nomad-lab/nomad-lab-base/core/src/main/scala/eu/nomad_lab/normalize/ElasticsearchData.scala
+```
+1. After running the command a new folder with the new json files is generated at:
+```
+/nomad/nomadlab/normalized/elastic/archiveGID
+```
+    3. The new data can be ingested by elasticsearch running the following command:
+```
+cd /nomadtmp/elasticsearch/elasticsearch-5.5.0/nomad
+sh runUploadJsonBulk.sh archiveGID
+```
+    4. There is a script to generate and ingest a set of archiveGID:
+```
+cd /nomadtmp/elasticsearch/elasticsearch-5.5.0/nomad
+sh runElasticByFile.sh index.txt  # (index.txt contains all the archiveGID, one per line)
+```
+
+# Elasticsearch recepies:
+
+1. Get all the values of variable (e.g. atom_symbols):
+
+```
+curl -s -XGET "http://nomad-flink-01:9200/_search?pretty" -H 'Content-Type: application/json' -d'
+{
+  "size": 0,
+  "aggs" : {
+      "counter" : {
+          "terms" : { "field" : "atom_symbols",  "size" : 4000 }
+      }
+   }
+}
+'
+```
+
+# Add more information metadata to elasticsearch:
+We have to add some code to:
+nomad-lab/nomad-lab-base/core/src/main/scala/eu/nomad_lab/normalize/ElasticsearchData.scala
+1. We have to create the metadata on "Section: Creating Metadata" (see comments on code)
+1. We have to add the new metadata to the json file on "Section: Creating Json" (see comments on code)
+
+# Deployment on Kubernets
+1. Create the docker image
+```
+cd nomad-lab-base
+sbt webservice/docker
+```
+1. Push the docker image (be sure that a new version number is generated)
+```
+docker push labdev-nomad.esc.rzg.mpg.de:5000/nomadlab/nomadwebservice:<version>
+#current version v1.8.0-119-gb50befb3-dirty
+```
+1. Create the kubernetes config files
+```
+sbt "kubernetes/run --config flink --all"
+``` 
+1. Adjust the number of replicas (file: kubeGen/api-server-rc.yaml):
+```
+"replicas": 1  
+```
+1. Load new kubernetes services:
+```
+/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 delete rc api-rc-flink
+/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 create -f kubeGen/api-server-rc.yaml
+```
+1. Check that our service is running
+```
+/nomad/nomadlab/kubernetes/kubectl --kubeconfig /nomad/nomadlab/kubernetes/admin.conf -s https://130.183.207.112:6443 get services
+```
+1. Check that nginx is redirected properly on labtest.
+
+# Using api Examples
+1. Query:
+https://analytics-toolkit.nomad-coe.eu/api/query/section_run?filter=program_name:FHI-aims
+1. Download:
+https://analytics-toolkit.nomad-coe.eu/api/downlad/hashQuery?file=datestatamp_hashquery.zip
+1. Autocomplete:
+https://analytics-toolkit.nomad-coe.eu/api/autocomplete/uniquevalues?field=uploader
+