updating labbook

BioEM_labbook.org

@@ -15,9 +15,21 @@
    - State "TODO"       from              [2017-06-13 Tue 16:28]
    :END:
    - Probably in OpenMP4, but still need to discuss about this part
+** TODO [#C] Intel KNL
+   :LOGBOOK:
+   - State "TODO"       from              [2017-06-23 Fri 15:52]
+   :END:
+   - Try to compile on Intel KNL
+   - Investigate the performance
 ** Algorithmic improvements (new functionality)
-** ?Xeon Phi
-* Ideas
+* REMINDER Ideas
+  :LOGBOOK:
+  - State "REMINDER"   from              [2017-06-23 Fri 15:54]
+  :END:
+** Hyperthreading
+   - Initial tests on draco showed no benefits from hyperthreading (on the contrary)
+   - Markus has a feeling (and experience) that hyperthreading cant help much codes such as BioEM
+   - However, with some changes to the code, it might be interesting to investigate again this option
 * Important codes
 ** Recipes for compilations and runs
 *** [partially working] dvl machine
@@ -580,12 +592,17 @@ module load boost/intel/1.61
 module load cuda/7.5
 
 # Environment variables
-export OMP_NUM_THREADS=32
+export OMP_NUM_THREADS=16
 export OMP_PLACES=cores
 export FFTALGO=1
 export GPU=1
 export GPUDEVICE=-1
 
+# For hyperthreading, but actually brings bad performance
+#export OMP_NUM_THREADS=32
+#export OMP_PLACES=threads
+#export SLURM_HINT=multithread 
+
 # Environment variable to tune
 export GPUWORKLOAD=90
 export BIOEM_DEBUG_OUTPUT=0
@@ -1612,8 +1629,13 @@ ggplot(df, aes(x=Workload, y=Time, fill=Workload)) + geom_bar(stat="identity") +
 #+RESULTS:
 [[file:analysis/results1_analysis.pdf]]
 ** 2017-06-19
-*** TODO Autotuning [3/7]:
+*** TODO Autotuning [6/9]:
     :LOGBOOK:
+    - State "TODO"       from "TODO"       [2017-06-23 Fri 15:46]
+    - State "TODO"       from "TODO"       [2017-06-23 Fri 12:58]
+    - State "TODO"       from "TODO"       [2017-06-23 Fri 09:52]
+    - State "TODO"       from "TODO"       [2017-06-22 Thu 17:27]
+    - State "TODO"       from "TODO"       [2017-06-22 Thu 17:27]
     - State "TODO"       from "TODO"       [2017-06-22 Thu 09:59]
     - State "TODO"       from "TODO"       [2017-06-21 Wed 16:21]
     - State "TODO"       from "TODO"       [2017-06-21 Wed 16:21]
@@ -1631,28 +1653,39 @@ ggplot(df, aes(x=Workload, y=Time, fill=Workload)) + geom_bar(stat="identity") +
       + phys machine is using SunGrid job scheduler (/qstat/), but it is not well documented for MPCDF machines. Markus will send me a template for running jobs on this machine
     - [X] Only later realized that the influence of the pipelining is not so big, so actually measuring the time on "GPU only" and "CPU only" might be better approach. Implementing another autotuning approach might be interesting. Could even evaluate both and present results
     - [ ] Markus thinks that enabling autonuning with GPUWORKLOAD=-1 or without stating GPUWORKLOAD might be the best idea
+      + Pilar agrees, and this should be activated by default (GPUWORKLOAD='')
+      + Final code should look like this before merging (and keeping only one Autotuning algorithm)
     - [ ] Discuss on following meeting the possibility to recalibrate the workload during the execution. For now there seems to be no need for such a thing, but it can be added quite easily and cheaply in terms of the performance overhead and code development
+      + Pilar believes that this can be useful, check with Markus
+      + Propose an environment variable for this, as well as for the number of comparisons to get stable performance (STABILIZER)
     - [X] Check if there is another approach for Autotuning, described in Numerical Recipes book
       + Yes with bisection, performance results will show if this is better than other approaches
-    - [ ] Report error regarding documentation of the computation of GPUWORKLOAD
+    - [X] Report error regarding documentation of the computation of GPUWORKLOAD
       + Done, waiting for the reply and someone to correct this
       + If Pilar doesnt reply by Wednesday 21st, commit in main project and add tag 1.0.2
-    - [ ] Discuss with Markus possibility of hyperthreading and whether they generally use it
+      + Fixed and pushed
+    - [X] Discuss with Markus possibility of hyperthreading and whether they generally use it
+      + Could be a thing to test, although Markus doesnt expect a significant improvements from it
+      + On draco, hyperthreading (if I configured it correctly), seems to bring negative influence on the performance
+    - [X] Create another type of the visulization (Timeline vs Time for projection), possibly add events for the optimal workload (or colors for workload)
 ** 2017-06-20
-*** TODO Other code things to add for BioEM [1/5]
+*** TODO Other code things to add for BioEM [2/5]
     :LOGBOOK:
+    - State "TODO"       from "TODO"       [2017-06-22 Thu 11:59]
     - State "TODO"       from "TODO"       [2017-06-20 Tue 16:23]
     - State "TODO"       from "TODO"       [2017-06-20 Tue 16:13]
     - State "TODO"       from              [2017-06-20 Tue 10:19]
     :END:
     - [X] Problems with installing BioEM on hydra machine with Intel recipe that used to work. Issues seem unrelated to autotuning changes, but something regarding bio:configure and reading the parameters. Actually, before Luka was not using the right Intel compilers, with the correct one everything compiles fine
-    - [ ] Need to find a proper way of handling errors in the code
+    - [X] Need to find a proper way of handling errors in the code
     - [ ] Need to do a nice cleanup before merging into the main project
+    - [ ] Add nice printf for writing the Optimal Workload
     - [ ] Add more profoud CUDA profiling, possibly using specialized CUDA tools for that. We will certainly need it in future when doing more developments in BioEM
     - [ ] Ensure that pinning is done correctly (in Intel case there shouldnt be any problem)
 
-*** TODO Simple analysis of the result2
+*** DONE Simple analysis of the result2
     :LOGBOOK:
+    - State "DONE"       from "TODO"       [2017-06-22 Thu 13:55]
     - State "TODO"       from              [2017-06-20 Tue 15:17]
     :END:
 
@@ -1669,23 +1702,24 @@ ggplot(df, aes(x=Algorithm, y=Time, fill=Workload)) + geom_bar(stat="identity")
 #+end_src
 
 #+RESULTS:
-[[file:/tmp/babel-22888QGl/figure22888bVT.png]]
+[[file:/tmp/babel-22888QGl/figure22888ofZ.png]]
 
         - Maybe it would be good to save these results
 
-#+begin_src R :results output graphics :file analysis/results2_analysis.pdf :exports both :width 6 :height 4 :session org-R
+#+begin_src R :results output graphics :file analysis/results2_analysis.pdf :exports both :width 8 :height 4 :session org-R
 library(ggplot2)
 
-df <- read.table("data/results2.csv", header=TRUE, sep=", ")
+df <- read.table("data/results2.csv", header=TRUE, sep=",", strip.white=TRUE)
 
 df$Algorithm <- factor(df$Algorithm, levels=c("Ref", "Static", "Auto1", "Auto2", "Auto3"))
 df$GPUs <- as.factor(as.character(paste("#GPUs = ", df$GPUs)))
 df$OMP_THREADS <- as.factor(as.character(paste("#OMPs = ", df$OMP_THREADS)))
 
-ggplot(df, aes(x=Algorithm, y=Time, fill=Algorithm)) + geom_bar(stat="identity") + facet_wrap( ~ Machine) + theme_bw() + ylab("Overall execution time [s]") + xlab("Workload algorithm")
+ggplot(df, aes(x=Algorithm, y=Time, fill=Workload)) + geom_bar(stat="identity") + scale_fill_gradientn(colours=rainbow(4)) + facet_wrap( ~ Machine) + theme_bw() + ylab("Overall execution time [s]") + xlab("Workload algorithm")
 #+end_src
 
 #+RESULTS:
+[[file:analysis/results2_analysis.pdf]]
 ** 2017-06-21
 *** MOVED Autotuning [0/1]
     :LOGBOOK:
@@ -1709,3 +1743,141 @@ ggplot(df, aes(x=Algorithm, y=Time, fill=Algorithm)) + geom_bar(stat="identity")
       + If this is true, additional code in BioEM is needed, to make sure it can work on similar machines
       + We checked it, and indeed the problem on dvl machines was caused by the EXCLUSIVE_PROCESS mode of GPUs. In DEFAULT mode everything works smoothly, even with the GPU testing code
       + I need to investigate why EXCLUSIVE_PROCESS mode is causing troubles for BioEM
+** 2017-06-23
+*** Analyzing workload choice for Autotuning 3 on dvl
+
+#+begin_src R :results output graphics :file :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session org-R 
+  library(ggplot2)
+
+  df1 <- read.table("data/workload_test.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df1$Exp <- "Exp1"
+  df2 <- read.table("data/workload_test2.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df2$Exp <- "Exp2"
+  df3 <- read.table("data/workload_test3.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df3$Exp <- "Exp3"
+
+  df <- rbind(df1, df2, df3)
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+  base <- floor(max(df$Proj)/(max(df$MPI_Rank)+1))
+  df$Comp2 <- (df$Proj-base*df$MPI_Rank)*base + (df$Comp - base*df$MPI_Rank)
+
+  ggplot(df, aes(x=Comp2, y=Duration, color=factor(Workload))) + geom_point()  + facet_wrap( Exp ~ MPI_Rank, ncol=2) + theme_bw() + ylab("Comparison duration [s]") + xlab("Comparison timeline")
+
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-22888QGl/figure22888dQt.png]]
+
+#+begin_src R :results output graphics :file :file (org-babel-temp-file "figure" ".png") :exports both :width 600 :height 400 :session org-R 
+  library(ggplot2)
+  library(plyr)
+
+  df1 <- read.table("data/workload_test.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df1$Exp <- "Exp1"
+  df2 <- read.table("data/workload_test2.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df2$Exp <- "Exp2"
+  df3 <- read.table("data/workload_test3.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df3$Exp <- "Exp3"
+
+  df <- rbind(df1, df2, df3)
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+
+  df <- ddply(df, .(MPI_Rank, Workload, Exp), summarise, Duration = max(Duration))
+
+  ggplot(df, aes(x=Workload, y=Duration, color=factor(Workload))) + geom_point()  + facet_wrap( Exp ~ MPI_Rank, ncol=2) + theme_bw() + ylab("Max comparison duration [s]") + xlab("Workload value")
+
+#+end_src
+
+#+RESULTS:
+[[file:/tmp/babel-22888QGl/figure22888QNb.png]]
+*** Analyzing workload choice for Autotuning 3 on draco
+
+#+begin_src R :results output graphics :file analysis/draco_workload1.pdf :exports both :width 9 :height 12 :session org-R 
+  library(ggplot2)
+
+  df1 <- read.table("data/draco_workload_test.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df1$Exp <- "Experiment 1"
+  df2 <- read.table("data/draco_workload_test2.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df2$Exp <- "Experiment 2"
+  df <- rbind(df1, df2)
+
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+
+  base <- floor(max(df$Proj)/(max(df$MPI_Rank)+1))
+  df$Comp2 <- (df$Proj-base*df$MPI_Rank)*base + (df$Comp - base*df$MPI_Rank)
+
+  df$MPI_Rank <- as.factor(as.character(paste("MPI_Rank = ", df$MPI_Rank)))
+
+  ggplot(df, aes(x=Comp2, y=Duration, color=factor(Workload))) + geom_point()  + facet_grid( MPI_Rank ~ Exp) + theme_bw() + ylab("Comparison duration [s]") + xlab("Comparison timeline")+xlim(2000000,2100000)
+
+#+end_src
+
+#+RESULTS:
+[[file:analysis/draco_workload1.pdf]]
+
+#+begin_src R :results output graphics :file analysis/draco_workload2.pdf :exports both :width 9 :height 12 :session org-R 
+  library(ggplot2)
+  library(plyr)
+
+  df1 <- read.table("data/draco_workload_test.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df1$Exp <- "Experiment 1"
+  df2 <- read.table("data/draco_workload_test2.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df2$Exp <- "Experiment 2"
+  df <- rbind(df1, df2)
+
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+
+  #df <- ddply(df, .(MPI_Rank, Workload, Exp), summarise, Duration = max(Duration))
+
+  df$MPI_Rank <- as.factor(as.character(paste("MPI_Rank = ", df$MPI_Rank)))
+
+  ggplot(df, aes(x=Workload, y=Duration, color=factor(Workload))) + geom_point()  + facet_grid( MPI_Rank ~ Exp) + theme_bw() + ylab("Comparison duration [s]") + xlab("Workload value")
+
+#+end_src
+
+#+RESULTS:
+[[file:analysis/draco_workload2.pdf]]
+
+#+begin_src R :results output graphics :file analysis/draco_workload1_zoom.pdf :exports both :width 6 :height 6 :session org-R 
+  library(ggplot2)
+
+  df1 <- read.table("data/draco_workload_test.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df1$Exp <- "Experiment 1"
+  df2 <- read.table("data/draco_workload_test2.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df2$Exp <- "Experiment 2"
+  df <- rbind(df1, df2)
+
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+
+  base <- floor(max(df$Proj)/(max(df$MPI_Rank)+1))
+  df$Comp2 <- (df$Proj-base*df$MPI_Rank)*base + (df$Comp - base*df$MPI_Rank)
+
+  df$MPI_Rank <- as.factor(as.character(paste("MPI_Rank = ", df$MPI_Rank)))
+
+  ggplot(df[df$MPI_Rank=="MPI_Rank =  4" & df$Exp=="Experiment 1",], aes(x=Comp2, y=Duration, color=factor(Workload))) + geom_point()  + facet_grid( MPI_Rank ~ Exp) + theme_bw() + ylab("Comparison duration [s]") + xlab("Comparison timeline")+xlim(2010000,2100000) + ggtitle("Zoomed view on bogus performance")
+
+#+end_src
+
+#+RESULTS:
+[[file:analysis/draco_workload1_zoom.pdf]]
+
+#+begin_src R :results output graphics :file analysis/draco_workload0.pdf :exports both :width 4.5 :height 12 :session org-R 
+  library(ggplot2)
+
+  df <- read.table("data/draco_workload_test0.csv", header=FALSE, sep=",", strip.white=TRUE)
+  df$Exp <- "Static experiment 87%"
+  names(df) <- c("Name", "MPI_Rank", "Proj", "Comp", "Duration", "Workload", "Exp")
+
+  base <- floor(max(df$Proj)/(max(df$MPI_Rank)+1))
+  df$Comp2 <- (df$Proj-base*df$MPI_Rank)*base + (df$Comp - base*df$MPI_Rank)
+
+  df$MPI_Rank <- as.factor(as.character(paste("MPI_Rank = ", df$MPI_Rank)))
+
+  df$Workload <- 87
+
+  ggplot(df, aes(x=Comp2, y=Duration, color=factor(Workload))) + geom_point()  + facet_grid( MPI_Rank ~ Exp) + theme_bw() + ylab("Comparison duration [s]") + xlab("Comparison timeline")
+
+#+end_src
+
+#+RESULTS:
+[[file:analysis/draco_workload0.pdf]]