Merge branch 'do_cleanup' into 'NIFTy_6'

Remove standard MPI parallelization

See merge request !387

.gitlab-ci.yml

@@ -39,17 +39,10 @@ test_serial:
   script:
     - pytest-3 -q --cov=nifty6 test
     - >
-      python3 -m coverage report --omit "*plot*,*distributed_do*" | tee coverage.txt
+      python3 -m coverage report --omit "*plot*" | tee coverage.txt
     - >
       grep TOTAL coverage.txt | awk '{ print "TOTAL: "$4; }'
 
-test_mpi:
-  stage: test
-  variables:
-    OMPI_MCA_btl_vader_single_copy_mechanism: none
-  script:
-    - mpiexec -n 2 --bind-to none pytest-3 -q test
-
 pages:
   stage: release
   script:

ChangeLog

@@ -16,3 +16,24 @@ but it is hard to make explicit tests since the two approaches cannot be mapped
 onto each other exactly. We experienced that preconditioning in the `MetricGaussianKL`
 via `napprox` breaks the inference scheme with the new model so `napprox` may not
 be used here.
+
+Removal of the standard parallelization scheme:
+===============================================
+
+When several MPI tasks are present, NIFTy5 distributes every Field over these
+tasks by splitting it along the first axis. This approach to parallelism is not
+very efficient, and it has not been used by anyone for several years, so we
+decided to remove it, which led to many simplifications within NIFTy.
+User-visible changes:
+- the methods `to_global_data`, `from_global_data`, `from_local_data` and
+  the property `local_data` have been removed from `Field` and `MultiField`.
+  Instead there are now the property `val` (returning a read-only numpy.ndarray
+  for `Field` and a dictionary of read-only numpy.ndarrays for `MultiField`) and
+  the method `val_rw()` (returning the same structures with writable copies of
+  the arrays). Fields and MultiFields can be created from such structures using
+  the static method `from_raw`
+- the functions `from_global_data` and `from_local_data` in `sugar` have been
+  replaced by a single function called `makeField`
+- the property `local_shape` has been removed from `Domain` (and subclasses)
+  and `DomainTuple`.
+

demos/Wiener_Filter.ipynb

 %% Cell type:markdown id: tags:
 
 # A NIFTy demonstration
 
 %% Cell type:markdown id: tags:
 
 ## IFT: Big Picture
 IFT starting point:
 
 $$d = Rs+n$$
 
 Typically, $s$ is a continuous field, $d$ a discrete data vector. Particularly, $R$ is not invertible.
 
 IFT aims at **inverting** the above uninvertible problem in the **best possible way** using Bayesian statistics.
 
 
 ## NIFTy
 
 NIFTy (Numerical Information Field Theory) is a Python framework in which IFT problems can be tackled easily.
 
 Main Interfaces:
 
 - **Spaces**: Cartesian, 2-Spheres (Healpix, Gauss-Legendre) and their respective harmonic spaces.
 - **Fields**: Defined on spaces.
 - **Operators**: Acting on fields.
 
 %% Cell type:markdown id: tags:
 
 ## Wiener Filter: Formulae
 
 ### Assumptions
 
 - $d=Rs+n$, $R$ linear operator.
 - $\mathcal P (s) = \mathcal G (s,S)$, $\mathcal P (n) = \mathcal G (n,N)$ where $S, N$ are positive definite matrices.
 
 ### Posterior
 The Posterior is given by:
 
 $$\mathcal P (s|d) \propto P(s,d) = \mathcal G(d-Rs,N) \,\mathcal G(s,S) \propto \mathcal G (s-m,D) $$
 
 where
 $$\begin{align}
 m &= Dj \\
 D^{-1}&= (S^{-1} +R^\dagger N^{-1} R )\\
 j &= R^\dagger N^{-1} d
 \end{align}$$
 
 Let us implement this in NIFTy!
 
 %% Cell type:markdown id: tags:
 
 ## Wiener Filter: Example
 
 - We assume statistical homogeneity and isotropy. Therefore the signal covariance $S$ is diagonal in harmonic space, and is described by a one-dimensional power spectrum, assumed here as $$P(k) = P_0\,\left(1+\left(\frac{k}{k_0}\right)^2\right)^{-\gamma /2},$$
 with $P_0 = 0.2, k_0 = 5, \gamma = 4$.
 - $N = 0.2 \cdot \mathbb{1}$.
 - Number of data points $N_{pix} = 512$.
 - reconstruction in harmonic space.
 - Response operator:
 $$R = FFT_{\text{harmonic} \rightarrow \text{position}}$$
 
 %% Cell type:code id: tags:
 
 ``` python
 N_pixels = 512     # Number of pixels
 
 def pow_spec(k):
     P0, k0, gamma = [.2, 5, 4]
     return P0 / ((1. + (k/k0)**2)**(gamma / 2))
 ```
 
 %% Cell type:markdown id: tags:
 
 ## Wiener Filter: Implementation
 
 %% Cell type:markdown id: tags:
 
 ### Import Modules
 
 %% Cell type:code id: tags:
 
 ``` python
 import numpy as np
 np.random.seed(40)
 import nifty6 as ift
 import matplotlib.pyplot as plt
 %matplotlib inline
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Implement Propagator
 
 %% Cell type:code id: tags:
 
 ``` python
 def Curvature(R, N, Sh):
     IC = ift.GradientNormController(iteration_limit=50000,
                                     tol_abs_gradnorm=0.1)
     # WienerFilterCurvature is (R.adjoint*N.inverse*R + Sh.inverse) plus some handy
     # helper methods.
     return ift.WienerFilterCurvature(R,N,Sh,iteration_controller=IC,iteration_controller_sampling=IC)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Conjugate Gradient Preconditioning
 
 - $D$ is defined via:
 $$D^{-1} = \mathcal S_h^{-1} + R^\dagger N^{-1} R.$$
 In the end, we want to apply $D$ to $j$, i.e. we need the inverse action of $D^{-1}$. This is done numerically (algorithm: *Conjugate Gradient*).
 
 <!--
 - One can define the *condition number* of a non-singular and normal matrix $A$:
 $$\kappa (A) := \frac{|\lambda_{\text{max}}|}{|\lambda_{\text{min}}|},$$
 where $\lambda_{\text{max}}$ and $\lambda_{\text{min}}$ are the largest and smallest eigenvalue of $A$, respectively.
 
 - The larger $\kappa$ the slower Conjugate Gradient.
 
 - By default, conjugate gradient solves: $D^{-1} m = j$ for $m$, where $D^{-1}$ can be badly conditioned. If one knows a non-singular matrix $T$ for which $TD^{-1}$ is better conditioned, one can solve the equivalent problem:
 $$\tilde A m = \tilde j,$$
 where $\tilde A = T D^{-1}$ and $\tilde j = Tj$.
 
 - In our case $S^{-1}$ is responsible for the bad conditioning of $D$ depending on the chosen power spectrum. Thus, we choose
 
 $$T = \mathcal F^\dagger S_h^{-1} \mathcal F.$$
 -->
 
 %% Cell type:markdown id: tags:
 
 ### Generate Mock data
 
 - Generate a field $s$ and $n$ with given covariances.
 - Calculate $d$.
 
 %% Cell type:code id: tags:
 
 ``` python
 s_space = ift.RGSpace(N_pixels)
 h_space = s_space.get_default_codomain()
 HT = ift.HarmonicTransformOperator(h_space, target=s_space)
 
 # Operators
 Sh = ift.create_power_operator(h_space, power_spectrum=pow_spec)
 R = HT #*ift.create_harmonic_smoothing_operator((h_space,), 0, 0.02)
 
 # Fields and data
 sh = Sh.draw_sample()
 noiseless_data=R(sh)
 noise_amplitude = np.sqrt(0.2)
 N = ift.ScalingOperator(s_space, noise_amplitude**2)
 
 n = ift.Field.from_random(domain=s_space, random_type='normal',
                           std=noise_amplitude, mean=0)
 d = noiseless_data + n
 j = R.adjoint_times(N.inverse_times(d))
 curv = Curvature(R=R, N=N, Sh=Sh)
 D = curv.inverse
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Run Wiener Filter
 
 %% Cell type:code id: tags:
 
 ``` python
 m = D(j)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Signal Reconstruction
 
 %% Cell type:code id: tags:
 
 ``` python
 # Get signal data and reconstruction data
-s_data = HT(sh).to_global_data()
-m_data = HT(m).to_global_data()
-d_data = d.to_global_data()
+s_data = HT(sh).val
+m_data = HT(m).val
+d_data = d.val
 
 plt.figure(figsize=(15,10))
 plt.plot(s_data, 'r', label="Signal", linewidth=3)
 plt.plot(d_data, 'k.', label="Data")
 plt.plot(m_data, 'k', label="Reconstruction",linewidth=3)
 plt.title("Reconstruction")
 plt.legend()
 plt.show()
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 plt.figure(figsize=(15,10))
 plt.plot(s_data - s_data, 'r', label="Signal", linewidth=3)
 plt.plot(d_data - s_data, 'k.', label="Data")
 plt.plot(m_data - s_data, 'k', label="Reconstruction",linewidth=3)
 plt.axhspan(-noise_amplitude,noise_amplitude, facecolor='0.9', alpha=.5)
 plt.title("Residuals")
 plt.legend()
 plt.show()
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Power Spectrum
 
 %% Cell type:code id: tags:
 
 ``` python
-s_power_data = ift.power_analyze(sh).to_global_data()
-m_power_data = ift.power_analyze(m).to_global_data()
+s_power_data = ift.power_analyze(sh).val
+m_power_data = ift.power_analyze(m).val
 plt.figure(figsize=(15,10))
 plt.loglog()
 plt.xlim(1, int(N_pixels/2))
 ymin = min(m_power_data)
 plt.ylim(ymin, 1)
 xs = np.arange(1,int(N_pixels/2),.1)
 plt.plot(xs, pow_spec(xs), label="True Power Spectrum", color='k',alpha=0.5)
 plt.plot(s_power_data, 'r', label="Signal")
 plt.plot(m_power_data, 'k', label="Reconstruction")
 plt.axhline(noise_amplitude**2 / N_pixels, color="k", linestyle='--', label="Noise level", alpha=.5)
 plt.axhspan(noise_amplitude**2 / N_pixels, ymin, facecolor='0.9', alpha=.5)
 plt.title("Power Spectrum")
 plt.legend()
 plt.show()
 ```
 
 %% Cell type:markdown id: tags:
 
 ## Wiener Filter on Incomplete Data
 
 %% Cell type:code id: tags:
 
 ``` python
 # Operators
 Sh = ift.create_power_operator(h_space, power_spectrum=pow_spec)
 N = ift.ScalingOperator(s_space, noise_amplitude**2)
 # R is defined below
 
 # Fields
 sh = Sh.draw_sample()
 s = HT(sh)
 n = ift.Field.from_random(domain=s_space, random_type='normal',
                       std=noise_amplitude, mean=0)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Partially Lose Data
 
 %% Cell type:code id: tags:
 
 ``` python
 l = int(N_pixels * 0.2)
 h = int(N_pixels * 0.2 * 2)
 
 mask = np.full(s_space.shape, 1.)
 mask[l:h] = 0
-mask = ift.Field.from_global_data(s_space, mask)
+mask = ift.Field.from_raw(s_space, mask)
 
 R = ift.DiagonalOperator(mask)(HT)
-n = n.to_global_data_rw()
+n = n.val_rw()
 n[l:h] = 0
-n = ift.Field.from_global_data(s_space, n)
+n = ift.Field.from_raw(s_space, n)
 
 d = R(sh) + n
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 curv = Curvature(R=R, N=N, Sh=Sh)
 D = curv.inverse
 j = R.adjoint_times(N.inverse_times(d))
 m = D(j)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Compute Uncertainty
 
 %% Cell type:code id: tags:
 
 ``` python
 m_mean, m_var = ift.probe_with_posterior_samples(curv, HT, 200)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Get data
 
 %% Cell type:code id: tags:
 
 ``` python
 # Get signal data and reconstruction data
-s_data = s.to_global_data()
-m_data = HT(m).to_global_data()
-m_var_data = m_var.to_global_data()
+s_data = s.val
+m_data = HT(m).val
+m_var_data = m_var.val
 uncertainty = np.sqrt(m_var_data)
-d_data = d.to_global_data_rw()
+d_data = d.val_rw()
 
 # Set lost data to NaN for proper plotting
 d_data[d_data == 0] = np.nan
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 fig = plt.figure(figsize=(15,10))
 plt.axvspan(l, h, facecolor='0.8',alpha=0.5)
 plt.fill_between(range(N_pixels), m_data - uncertainty, m_data + uncertainty, facecolor='0.5', alpha=0.5)
 plt.plot(s_data, 'r', label="Signal", alpha=1, linewidth=3)
 plt.plot(d_data, 'k.', label="Data")
 plt.plot(m_data, 'k', label="Reconstruction", linewidth=3)
 plt.title("Reconstruction of incomplete data")
 plt.legend()
 ```
 
 %% Cell type:markdown id: tags:
 
 # 2d Example
 
 %% Cell type:code id: tags:
 
 ``` python
 N_pixels = 256      # Number of pixels
 sigma2 = 2.         # Noise variance
 
 def pow_spec(k):
     P0, k0, gamma = [.2, 2, 4]
     return P0 * (1. + (k/k0)**2)**(-gamma/2)
 
 s_space = ift.RGSpace([N_pixels, N_pixels])
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 h_space = s_space.get_default_codomain()
 HT = ift.HarmonicTransformOperator(h_space,s_space)
 
 # Operators
 Sh = ift.create_power_operator(h_space, power_spectrum=pow_spec)
 N = ift.ScalingOperator(s_space, sigma2)
 
 # Fields and data
 sh = Sh.draw_sample()
 n = ift.Field.from_random(domain=s_space, random_type='normal',
                       std=np.sqrt(sigma2), mean=0)
 
 # Lose some data
 
 l = int(N_pixels * 0.33)
 h = int(N_pixels * 0.33 * 2)
 
 mask = np.full(s_space.shape, 1.)
 mask[l:h,l:h] = 0.
-mask = ift.Field.from_global_data(s_space, mask)
+mask = ift.Field.from_raw(s_space, mask)
 
 R = ift.DiagonalOperator(mask)(HT)
-n = n.to_global_data_rw()
+n = n.val_rw()
 n[l:h, l:h] = 0
-n = ift.Field.from_global_data(s_space, n)
+n = ift.Field.from_raw(s_space, n)
 curv = Curvature(R=R, N=N, Sh=Sh)
 D = curv.inverse
 
 d = R(sh) + n
 j = R.adjoint_times(N.inverse_times(d))
 
 # Run Wiener filter
 m = D(j)
 
 # Uncertainty
 m_mean, m_var = ift.probe_with_posterior_samples(curv, HT, 20)
 
 # Get data
-s_data = HT(sh).to_global_data()
-m_data = HT(m).to_global_data()
-m_var_data = m_var.to_global_data()
-d_data = d.to_global_data()
+s_data = HT(sh).val
+m_data = HT(m).val
+m_var_data = m_var.val
+d_data = d.val
 uncertainty = np.sqrt(np.abs(m_var_data))
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 cm = ['magma', 'inferno', 'plasma', 'viridis'][1]
 
 mi = np.min(s_data)
 ma = np.max(s_data)
 
 fig, axes = plt.subplots(1, 2, figsize=(15, 7))
 
 data = [s_data, d_data]
 caption = ["Signal", "Data"]
 
 for ax in axes.flat:
     im = ax.imshow(data.pop(0), interpolation='nearest', cmap=cm, vmin=mi,
                    vmax=ma)
     ax.set_title(caption.pop(0))
 
 fig.subplots_adjust(right=0.8)
 cbar_ax = fig.add_axes([0.85, 0.15, 0.05, 0.7])
 fig.colorbar(im, cax=cbar_ax)
 ```
 
 %% Cell type:code id: tags:
 
 ``` python
 mi = np.min(s_data)
 ma = np.max(s_data)
 
 fig, axes = plt.subplots(3, 2, figsize=(15, 22.5))
-sample = HT(curv.draw_sample(from_inverse=True)+m).to_global_data()
-post_mean = (m_mean + HT(m)).to_global_data()
+sample = HT(curv.draw_sample(from_inverse=True)+m).val
+post_mean = (m_mean + HT(m)).val
 
 data = [s_data, m_data, post_mean, sample, s_data - m_data, uncertainty]
 caption = ["Signal", "Reconstruction", "Posterior mean", "Sample", "Residuals", "Uncertainty Map"]
 
 for ax in axes.flat:
     im = ax.imshow(data.pop(0), interpolation='nearest', cmap=cm, vmin=mi, vmax=ma)
     ax.set_title(caption.pop(0))
 
 fig.subplots_adjust(right=0.8)
 cbar_ax = fig.add_axes([.85, 0.15, 0.05, 0.7])
 fig.colorbar(im, cax=cbar_ax)
 ```
 
 %% Cell type:markdown id: tags:
 
 ### Is the uncertainty map reliable?
 
 %% Cell type:code id: tags:
 
 ``` python
 precise = (np.abs(s_data-m_data) < uncertainty)
 print("Error within uncertainty map bounds: " + str(np.sum(precise) * 100 / N_pixels**2) + "%")
 
 plt.figure(figsize=(15,10))
 plt.imshow(precise.astype(float), cmap="brg")
 plt.colorbar()
 ```
 
 %% Cell type:markdown id: tags:
 
 # Start Coding
 ## NIFTy Repository + Installation guide
 
 https://gitlab.mpcdf.mpg.de/ift/NIFTy
 
 NIFTy v5 **more or less stable!**

demos/bench_gridder.py

@@ -24,16 +24,16 @@ for ii in range(10, 26):
 
     img = np.random.randn(nu*nv)
     img = img.reshape((nu, nv))
-    img = ift.from_global_data(uvspace, img)
+    img = ift.makeField(uvspace, img)
 
     t0 = time()
     GM = ift.GridderMaker(uvspace, eps=1e-7, uv=uv)
-    vis = ift.from_global_data(visspace, vis)
+    vis = ift.makeField(visspace, vis)
     op = GM.getFull().adjoint
     t1 = time()
-    op(img).to_global_data()
+    op(img).val
     t2 = time()
-    op.adjoint(vis).to_global_data()
+    op.adjoint(vis).val
     t3 = time()
     print(t2-t1, t3-t2)
     N0s.append(N)

demos/bernoulli_demo.py

@@ -61,9 +61,9 @@ if __name__ == '__main__':
     # Generate mock data
     p = R(sky)
     mock_position = ift.from_random('normal', harmonic_space)
-    tmp = p(mock_position).to_global_data().astype(np.float64)
+    tmp = p(mock_position).val.astype(np.float64)
     data = np.random.binomial(1, tmp)
-    data = ift.Field.from_global_data(R.target, data)
+    data = ift.Field.from_raw(R.target, data)
 
     # Compute likelihood and Hamiltonian
     position = ift.from_random('normal', harmonic_space)

demos/find_amplitude_parameters.py

@@ -57,7 +57,7 @@ if __name__ == '__main__':
         for _ in range(n_samps):
             fld = pspec(ift.from_random('normal', pspec.domain))
             klengths = fld.domain[0].k_lengths
-            ycoord = fld.to_global_data_rw()
+            ycoord = fld.val_rw()
             ycoord[0] = ycoord[1]
             ax.plot(klengths, ycoord, alpha=1)
 
@@ -80,7 +80,7 @@ if __name__ == '__main__':
         foo = []
         for ax in axs:
             pos = ift.from_random('normal', correlated_field.domain)
-            fld = correlated_field(pos).to_global_data()
+            fld = correlated_field(pos).val
             foo.append((ax, fld))
         mi, ma = np.inf, -np.inf
         for _, fld in foo:
@@ -106,7 +106,7 @@ if __name__ == '__main__':
         flds = []
         for _ in range(n_samps):
             pos = ift.from_random('normal', correlated_field.domain)
-            ax.plot(correlated_field(pos).to_global_data())
+            ax.plot(correlated_field(pos).val)
 
     plt.savefig('correlated_fields.png')
     plt.close()

demos/getting_started_1.py

@@ -42,7 +42,7 @@ def make_random_mask():
     # Random mask for spherical mode
     mask = ift.from_random('pm1', position_space)
     mask = (mask + 1)/2
-    return mask.to_global_data()
+    return mask.val
 
 
 if __name__ == '__main__':
@@ -95,7 +95,7 @@ if __name__ == '__main__':
     # and harmonic transformaion
 
     # Masking operator to model that parts of the field have not been observed
-    mask = ift.Field.from_global_data(position_space, mask)
+    mask = ift.Field.from_raw(position_space, mask)
     Mask = ift.MaskOperator(mask)
 
     # The response operator consists of

demos/getting_started_2.py

@@ -40,7 +40,7 @@ def exposure_2d():
     exposure[:, x_shape*4//5:x_shape] *= .1
     exposure[:, x_shape//2:x_shape*3//2] *= 3.
 
-    return ift.Field.from_global_data(position_space, exposure)
+    return ift.Field.from_raw(position_space, exposure)
 
 
 if __name__ == '__main__':
@@ -94,8 +94,8 @@ if __name__ == '__main__':
     lamb = R(sky)
     mock_position = ift.from_random('normal', domain)
     data = lamb(mock_position)
-    data = np.random.poisson(data.to_global_data().astype(np.float64))
-    data = ift.Field.from_global_data(d_space, data)
+    data = np.random.poisson(data.val.astype(np.float64))
+    data = ift.Field.from_raw(d_space, data)
     likelihood = ift.PoissonianEnergy(data)(lamb)