Creating of ELPA_development_version_OpenMP

Based on the ELPA 2013.08.001 a development version for ELPA OpenMP
has been created

ELPA_development_version_OpenMP/COPYING/COPYING

+Licensing and copyright terms for the ELPA library:
+ELPA Consortium (2011)
+
+****
+
+Copyright of the original code rests with the authors inside the ELPA
+consortium. The copyright of any additional modifications shall rest
+with their original authors, but shall adhere to the licensing terms
+set forth below.
+
+****
+
+The code is distributed under the terms of the GNU Lesser General Public 
+License version 3 (LGPL). The full text can be found in the file "lgpl.txt" 
+in this repository. "lgpl.txt" makes reference to the GPL v3, which can also 
+be found in this repository ("gpl.txt").
+
+****
+ELPA reflects a substantial effort on the part of the original
+ELPA consortium, and we ask you to respect the spirit of the
+license that we chose: i.e., please contribute any changes you
+may have back to the original ELPA library distribution, and keep
+any derivatives of ELPA under the same license that we chose for
+the original distribution, the GNU Lesser General Public License.
+
+When in doubt, talk to us. What we would like to ensure is that the ELPA
+code can be used as needed, while providing a strong incentive for
+others to contribute their modifications back to the original development.
+
+****

ELPA_development_version_OpenMP/COPYING/lgpl.txt

+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

ELPA_development_version_OpenMP/INSTALL

+How to install ELPA:
+----------------------
+
+ELPA is shipped with a typical "configure" and "make" procedure. It is
+recommended to use this way to install ELPA, see (A). If you do not want to 
+install ELPA as library, but to include it in your source code, please refer 
+to point (B)
+
+
+
+you can find a 
+"Makefile.example" in ./test, to see how this is done. Please distibute then 
+all files of ELPA with your code.
+
+
+(A): Installing ELPA as library with configure
+===================================================
+
+The configure installation is best done in four steps
+
+1) run configure: 
+   
+   Check the available options with "configure --help". 
+   ELPA is shipped with several different versions of the
+   elpa2-kernel, each optimized and tuned for a different
+   architecture. Since, at the moment we do not want to
+   define a default, which works but maybe with less then
+   the optimal performance, it is MANDATORY that you choose
+   (real and complex kernels) for your system, via the
+   "--with-..." option. For hints which kernels might be best
+   for you, please refer to the file 
+   "./src/elpa2_kernels/README_elpa2_kernels.txt".
+
+   Please point to your blacs/scalapack installation and the
+   linkline with the variables "BLACS_LDFLAGS" and "BLACS_FCFLAGS".
+   "BLACS_LDFLAGS" should then contain the correct linkline for your
+   blacs/scalapack installation and "BLACS_FCFLAGS" the include path
+   and any other flags you need at compile time.
+
+   It is recommended that you use the "rpath functionality" in the linkline,
+   otherwise it will be necessary to update the LD_LIBRARY_PATH environment
+   variable.
+
+   You can either specify your own builds of lapack/blacs/scalapack
+   or use specialized Vendor packages, e.g. if available you can use 
+   Intel's MKL. If you do not set these variables ELPA will not be
+   build!
+
+
+   Please set the optimisation that you would like with the
+   variable "FCFLAGS", "CFLAGS", and "CXXFLAGS", e.g. FCFLAGS="-O3 -xAVX".
+   For some elpa2-kernels, it is MANDATORY to set a few options,
+   please see "./src/elpa2_kernels/README_elpa2_kernels.txt".
+
+   Set the "prefix" - flag, if you wish another installation location than
+   the default "/usr/local/".
+
+   Note, that at the moment we do not officially support "cross compilation"
+   allthough it should work.
+
+2) run "make"
+
+3) run "make check"
+   a simple test of ELPA is done. At the moment the usage of "mpiexec"
+   is required. If this is not possible at your system, you can run the
+   binaries "test_real", "test_real2", "test_complex", and "test_complex2"
+   yourself. At the moment the tests check whether the residual and the
+   orthogonality of the found eigenvectors are lower than a threshold of
+   5e-12. If this test fails, or if you believe the threshold should be
+   even lower, please talk to us.
+
+
+4) run "make install"
+   Note that a pckconfig file for ELPA is produced. You should then be
+   able to link the ELPA library to your own applications.
+
+
+B) Installing ELPA without the autotools procedure
+===================================================
+    You can find an example makefile "Makefile.example" in "./test",
+    to see how you can use ELPA directly in your code, and not as library. 
+    If you do so, please distibute then all files of ELPA with your code.
+
+    However, this is not the recommended way for several reasons:
+     - from the last release, ELPA has grown substantially in performance
+       optimizations but also complexity. The simple "just use elpa source
+       files in your code" approach is becoming more and more difficult.
+    - you still have to choose an elpa2-kernel (see at (A)). Getting them
+      build from hand might be tedious.
+    - the file elpa2.F90 uses preprocessor defines for the different kernels.
+      you will have to do this by hand, if you do not use the autotools 
+      infrastructure.
+
+
+
+How to use ELPA:
+-----------------
+Using ELPA should be quite simple. It is similiar to ScalaPack but the API
+is different. See the examples in the directory "./test". There is shown how
+to evoke ELPA from a Fortran code.
+
+If you installed ELPA with the build procedure a pk-config file is produced,
+which will tell you how to link against ELPA.

ELPA_development_version_OpenMP/Makefile.am

+ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS} -I m4
+
+AM_FCFLAGS = @AM_FCFLAGS@ @FC_MODINC@modules @FC_MODOUT@modules 
+
+AM_LDFLAGS = @AM_LDFLAGS@ @BLACS_LDFLAGS@
+
+BLACS_LDFLAGS = @BLACS_LDFLAGS@
+
+# libelpa
+lib_LTLIBRARIES = libelpa.la
+
+##rule to produce fortran config file:
+#config_f90.h: ./config.h
+#	grep "^#define" ./config.h > $@
+
+
+libelpa_la_SOURCES = src/elpa1.f90 src/elpa2.F90
+
+
+if WITH_GENERIC_SIMPLE
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_complex_simple.f90 \
+                        src/elpa2_kernels/elpa2_kernels_real_simple.f90
+endif
+
+if WITH_GENERIC
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_complex.f90 \
+                       src/elpa2_kernels/elpa2_kernels_real.f90
+endif
+
+if WITH_BGP
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_bgp.f90 \
+                        src/elpa2_kernels/elpa2_kernels_complex.f90 
+endif
+
+if WITH_BGQ
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_bgq.f90 \
+                        src/elpa2_kernels/elpa2_kernels_complex.f90 
+endif
+
+if WITH_SSE_AS
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_asm_x86_64.s
+endif
+
+if WITH_AVX_SANDYBRIDGE
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_sse-avx_2hv.c \
+                        src/elpa2_kernels/elpa2_kernels_complex_sse-avx_1hv.cpp
+endif
+
+if WITH_AMD_BULLDOZER
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_sse-avx_4hv.c \
+                        src/elpa2_kernels/elpa2_kernels_real_sse-avx_2hv.c \
+                        src/elpa2_kernels/elpa2_kernels_complex_sse-avx_1hv.cpp
+endif
+if WITH_AVX_COMPLEX_BLOCK1
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_complex_sse-avx_1hv.cpp
+endif
+
+if WITH_AVX_COMPLEX_BLOCK2
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_complex_sse-avx_2hv.cpp \
+                        src/elpa2_kernels/elpa2_kernels_complex_sse-avx_1hv.cpp
+endif
+
+if WITH_AVX_REAL_BLOCK2
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_sse-avx_2hv.c
+endif
+
+if WITH_AVX_REAL_BLOCK4
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_sse-avx_4hv.c
+endif
+
+if WITH_AVX_REAL_BLOCK6
+  libelpa_la_SOURCES += src/elpa2_kernels/elpa2_kernels_real_sse-avx_6hv.c
+endif
+
+
+libelpa_la_LDFLAGS = -version-info $(ELPA_SO_VERSION)
+
+# install any .mod files in the include/ dir
+elpa_includedir = $(includedir)/elpa
+nobase_elpa_include_HEADERS = $(wildcard modules/*)
+
+# other files to distribute
+filesdir = $(datarootdir)
+files_DATA = \
+	test/read_real.f90 \
+	test/read_real_gen.f90 \
+	test/test_complex2.f90 \
+	test/test_complex.f90 \
+	test/test_complex_gen.f90 \
+	test/test_real2.f90 \
+	test/test_real.f90 \
+	test/test_real_gen.f90
+
+# pkg-config stuff
+pkgconfigdir = $(libdir)/pkgconfig
+pkgconfig_DATA = elpa.pc
+
+# test programs
+noinst_bindir = $(abs_top_builddir)
+noinst_bin_PROGRAMS = test_real test_real2 test_complex test_complex2
+
+test_real_SOURCES = test/test_real.f90
+test_real_LDADD = libelpa.la 
+
+test_real2_SOURCES = test/test_real2.f90
+test_real2_LDADD = libelpa.la 
+
+test_complex_SOURCES = test/test_complex.f90
+test_complex_LDADD = libelpa.la
+
+test_complex2_SOURCES = test/test_complex2.f90
+test_complex2_LDADD = libelpa.la
+
+
+check_SCRIPTS = test_real.sh test_real2.sh test_complex.sh test_complex2.sh   
+TESTS = $(check_SCRIPTS)
+
+test_real.sh:
+	echo "mpiexec -n 2 ./test_real > /dev/null 2>&1" > test_real.sh
+	chmod +x test_real.sh	
+
+test_real2.sh:
+	echo "mpiexec -n 2 ./test_real2 > /dev/null 2>&1" > test_real2.sh
+	chmod +x test_real2.sh	
+
+test_complex.sh:
+	echo "mpiexec -n 2 ./test_complex > /dev/null 2>&1" > test_complex.sh
+	chmod +x test_complex.sh	
+
+test_complex2.sh:
+	echo "mpiexec -n 2 ./test_complex2 > /dev/null 2>&1" > test_complex2.sh
+	chmod +x test_complex2.sh	
+
+
+CLEANFILES = test_real.sh test_real2.sh test_complex.sh test_complex2.sh

ELPA_development_version_OpenMP/README

+Welcome to the git-based distribution of the ELPA eigensolver library.
+
+If you are reading this file, you have obtained the ELPA library
+through the git repository that hosts the source code and also allows
+you to contribute improvements to the project if necessary.
+
+In your use of ELPA, please respect the copyright restrictions
+found below and in the "COPYING" directory in this repository. In a
+nutshell, if you make improvements to ELPA, copyright for such
+improvements remains with you, but we request that you relicense any
+such improvements under the same exact terms of the (modified) LGPL v3
+that we are using here. Please do not simply absorb ELPA into your own
+project and then redistribute binary-only without making your exact
+version of the ELPA source code (unmodified or MODIFIED) available as
+well. 
+
+
+*** Citing:
+
+  A description of some algorithms present in ELPA can be found in:
+
+  T. Auckenthaler, V. Blum, H.-J. Bungartz, T. Huckle, R. Johanni,
+  L. Kr\"amer, B. Lang, H. Lederer, and P. R. Willems, 
+  "Parallel solution of partial symmetric eigenvalue problems from
+  electronic structure calculations", 
+  Parallel Computing 37, 783-794 (2011).
+  doi:10.1016/j.parco.2011.05.002. 
+
+  Please cite this paper when using ELPA. We also intend to publish an
+  overview description of the ELPA library as such, and ask you to
+  make appropriate reference to that as well, once it appears.
+
+
+*** Copyright: 
+
+Copyright of the original code rests with the authors inside the ELPA
+consortium. The code is distributed under the terms of the GNU Lesser General 
+Public License version 3 (LGPL).
+
+Please also note the express "NO WARRANTY" disclaimers in the GPL.
+
+Please see the file "COPYING" for details, and the files "gpl.txt" and
+"lgpl.txt" for further information.
+
+
+*** Using ELPA: 
+
+ELPA is designed to be compiled (Fortran) on its own, to be later
+linked to your own application. In order to use ELPA, you must still
+have a set of separate libraries that provide
+
+  - Basic Linear Algebra Subroutines (BLAS)
+  - Lapack routines
+  - Basic Linear Algebra Communication Subroutines (BLACS)
+  - Scalapack routines
+  - a working MPI library
+
+Appropriate libraries can be obtained and compiled separately on many
+architectures as free software. Alternatively, pre-packaged libraries
+are usually available from any HPC proprietary compiler vendors.
+
+For example, Intel's ifort compiler contains the "math kernel library"
+(MKL), providing BLAS/Lapack/BLACS/Scalapack functionality. (except on
+Mac OS X, where the BLACS and Scalapack part must still be obtained
+and compiled separately). 
+
+A very usable general-purpose MPI library is OpenMPI (ELPA was tested
+with OpenMPI 1.4.3 for example). Intel MPI seems to be a very well
+performing option on Intel platforms.
+
+Examples of how to use ELPA are included in the accompanying
+test_*.f90 subroutines in the "test" directory. An example makefile 
+"Makefile.example" is also included as a minimal example of how to 
+build and link ELPA to any other piece of code. In general, however,
+we suggest to use the build environment in order to install ELPA
+as library to your system. 
+
+
+*** Structure of this repository:
+
+* README file - this file. Please also consult the ELPA Wiki, and
+  consider adding any useful information that you may have.
+
+* COPYING directory - the copyright and licensing information for ELPA.
+
+* src directory - contains all the files that are needed for the
+  actual ELPA subroutines. If you are attempting to use ELPA in your
+  own application, these are the files which you need. 
+
+- elpa1.f90 contains routines for the one-stage solver,
+  The 1 stage solver (elpa1.f90) can be used standalone without elpa2.
+
+- elpa2.f90 - ADDITIONAL routines needed for the two-stage solver
+  elpa2.f90 requires elpa1.f90 and a version of elpa2_kernels.f90, so
+  always compile them together.
+
+- elpa2_kernels.f90 - optimized linear algebra kernels for ELPA.
+  This file is a generic version of optimized linear algebra kernels
+  for use with the ELPA library. The standard elpa2_kernels.f90 runs
+  on every platform but it is optimized for the Intel SSE instruction
+  set. Best perfomance is achieved with the Intel ifort compiler and 
+  compile flags -O3 -xSSE4.2
+
+  For optimum performance on special architectures, you may wish to
+  investigate whether hand-tuned versions of this file give additional
+  gains. If so, simply remove elpa2_kernels.f90 from your compilation
+  and replace with the version of your choice. It would be great if
+  you could contribute such hand-tuned versions back to the
+  repository. (LGPL requirement for redistribution holds in any case)
+
+- elpa2_kernels_bg.f90
+  Example of optimized ELPA kernels for the BlueGene/P
+  architecture. Use instead of the standard elpa2_kernels.f90
+  file. elpa2_kernels_bg.f90 contains assembler instructions for the
+  BlueGene/P processor which IBM's xlf Fortran compiler can handle.
+
+* test directory
+
+- Contains the Makefile that demonstrates how to compile and link to
+  the ELPA routines
+
+- All files starting with test_... are for demonstrating the use
+  of the elpa library (but not needed for using it).
+
+- All test programs solve a eigenvalue problem and check the correctnes
+  of the result by evaluating || A*x - x*lamba || and checking the
+  orthogonality of the eigenvectors
+
+  test_real         Real eigenvalue problem, 1 stage solver
+  test_real_gen     Real generalized eigenvalue problem, 1 stage solver
+  test_complex      Complex eigenvalue problem, 1 stage solver
+  test_complex_gen  Complex generalized eigenvalue problem, 1 stage solver
+  test_real2        Real eigenvalue problem, 2 stage solver
+  test_complex2     Complex eigenvalue problem, 2 stage solver
+
+- There are two programs which read matrices from a file, solve the
+  eigenvalue problem, print the eigenvalues and check the correctness
+  of the result (all using elpa1 only)
+
+  read_real         for the real eigenvalue problem
+  read_real_gen     for the real generalized eigenvalue problem
+                    A*x - B*x*lambda = 0
+
+  read_real has to be called with 1 command line argument (the file
+  containing the matrix). The file must be in ASCII (formatted) form.
+
+  read_real_gen has to be called with 3 command line arguments. The
+  first argument is either 'asc' or 'bin' (without quotes) and
+  determines the format of the following files. 'asc' refers to ASCII
+  (formatted) and 'bin' to binary (unformatted). Command line
+  arguments 2 and 3 are the names of the files which contain matrices
+  A and B.
+
+  The structure of the matrix files for read_real and read_real_gen
+  depends on the format of the files:
+
+  * ASCII format (both read_real and read_real_gen):
+
+    The files must contain the following lines:
+
+      - 1st line containing the matrix size
+      - then following the upper half of the matrix in column-major
+        (i.e. Fortran) order, one number per line:
+        a(1,1)
+        a(1,2)
+        a(2,2)
+        ...
+        a(1,i)
+        ...
+        a(i,i)
+        ...
+        a(1,n)
+        ...
+        a(n,n)
+
+
+  * Binary format (read_real_gen only):
+
+    The files must contain the following records:
+
+      - 1st record:  matrix size  (type integer)
+      - 2nd record:  a(1,1)
+      - 3rd record:  a(1,2)  a(2,2)
+      - ...
+      - ...          a(1,i)   ...   a(i,i)
+      - ...
+      - ...          a(1,n)      ...         a(n,n)