Commit 01cf9dfc authored by Philipp Arras's avatar Philipp Arras
Browse files

Merge remote-tracking branch 'origin/NIFTy_7' into nifty627

parents 0b7b4e88 de268998
Pipeline #76978 failed with stages
in 45 seconds
......@@ -37,7 +37,7 @@ build_docker_from_cache:
test_serial:
stage: test
script:
- pytest-3 -q --cov=nifty6 test
- pytest-3 -q --cov=nifty7 test
- >
python3 -m coverage report --omit "*plot*" | tee coverage.txt
- >
......
Changes since NIFTy 5:
Changes since NIFTy 6
=====================
Naming of operator tests
------------------------
The implementation tests for nonlinear operators are now available in
`ift.extra.check_operator()` and for linear operators
`ift.extra.check_linear_operator()`.
MetricGaussianKL interface
--------------------------
Users do not instanciate `MetricGaussianKL` by its constructor anymore. Rather
`MetricGaussianKL.make()` shall be used.
Changes since NIFTy 5
=====================
Minimum Python version increased to 3.6
=======================================
---------------------------------------
New operators
=============
-------------
In addition to the below changes, the following operators were introduced:
......@@ -20,14 +38,14 @@ In addition to the below changes, the following operators were introduced:
* IntegrationOperator: Integrates over subspaces of fields
FFT convention adjusted
=======================
-----------------------
When going to harmonic space, NIFTy's FFT operator now uses a minus sign in the
exponent (and, consequently, a plus sign on the adjoint transform). This
convention is consistent with almost all other numerical FFT libraries.
Interface change in EndomorphicOperator.draw_sample()
=====================================================
-----------------------------------------------------
Both complex-valued and real-valued Gaussian probability distributions have
Hermitian and positive endomorphisms as covariance. Just by looking at an
......@@ -43,7 +61,7 @@ In order to dive into those subtleties I suggest running the following code and
playing around with the dtypes.
```
import nifty6 as ift
import nifty7 as ift
import numpy as np
dom = ift.UnstructuredDomain(5)
......@@ -72,30 +90,30 @@ does not give 1, but sqrt(2) as a result.
MPI parallelisation over samples in MetricGaussianKL
====================================================
----------------------------------------------------
The classes `MetricGaussianKL` and `MetricGaussianKL_MPI` have been unified
into one `MetricGaussianKL` class which has MPI support built in.
New approach for random number generation
=========================================
-----------------------------------------
The code now uses `numpy`'s new `SeedSequence` and `Generator` classes for the
production of random numbers (introduced in numpy 1.17. This greatly simplifies
the generation of reproducible random numbers in the presence of MPI parallelism
and leads to cleaner code overall. Please see the documentation of
`nifty6.random` for details.
`nifty7.random` for details.
Interface Change for from_random and OuterProduct
=================================================
-------------------------------------------------
The sugar.from_random, Field.from_random, MultiField.from_random now take domain
as the first argument and default to 'normal' for the second argument.
Likewise OuterProduct takes domain as the first argument and a field as the second.
Interface Change for non-linear Operators
=========================================
-----------------------------------------
The method `Operator.apply()` takes a `Linearization` or a `Field` or a
`MultiField` as input. This has not changed. However, now each non-linear
......@@ -112,7 +130,7 @@ behaviour since both `Operator._check_input()` and
fulfilled.
Special functions for complete Field reduction operations
=========================================================
---------------------------------------------------------
So far, reduction operations called on Fields (like `vdot`, `sum`, `integrate`,
`mean`, `var`, `std`, `prod` etc.) returned a scalar when the reduction was
......@@ -124,7 +142,7 @@ operate over all subdomains and therefore don't take a `spaces` argument; they
are named `s_vdot`, `s_sum` etc. and always return a scalar.
Updates regarding correlated fields
===================================
-----------------------------------
The most commonly used model for homogeneous and isotropic correlated fields in
nifty5 has been `SLAmplitude` combined with `CorrelatedField`. This model
......@@ -141,7 +159,7 @@ via `napprox` breaks the inference scheme with the new model so `napprox` may no
be used here.
Removal of the standard MPI parallelization scheme:
===================================================
---------------------------------------------------
When several MPI tasks are present, NIFTy5 distributes every Field over these
tasks by splitting it along the first axis. This approach to parallelism is not
......
NIFTy - Numerical Information Field Theory
==========================================
[![pipeline status](https://gitlab.mpcdf.mpg.de/ift/nifty/badges/NIFTy_6/pipeline.svg)](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commits/NIFTy_6)
[![coverage report](https://gitlab.mpcdf.mpg.de/ift/nifty/badges/NIFTy_6/coverage.svg)](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commits/NIFTy_6)
[![pipeline status](https://gitlab.mpcdf.mpg.de/ift/nifty/badges/NIFTy_7/pipeline.svg)](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commits/NIFTy_7)
[![coverage report](https://gitlab.mpcdf.mpg.de/ift/nifty/badges/NIFTy_7/coverage.svg)](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commits/NIFTy_7)
**NIFTy** project homepage:
[http://ift.pages.mpcdf.de/nifty](http://ift.pages.mpcdf.de/nifty)
......@@ -59,8 +59,8 @@ Optional dependencies:
### Sources
The current version of NIFTy6 can be obtained by cloning the repository and
switching to the NIFTy_6 branch:
The current version of NIFTy7 can be obtained by cloning the repository and
switching to the NIFTy_7 branch:
git clone https://gitlab.mpcdf.mpg.de/ift/nifty.git
......@@ -69,10 +69,10 @@ switching to the NIFTy_6 branch:
In the following, we assume a Debian-based distribution. For other
distributions, the "apt" lines will need slight changes.
NIFTy6 and its mandatory dependencies can be installed via:
NIFTy7 and its mandatory dependencies can be installed via:
sudo apt-get install git python3 python3-pip python3-dev
pip3 install --user git+https://gitlab.mpcdf.mpg.de/ift/nifty.git@NIFTy_6
pip3 install --user git+https://gitlab.mpcdf.mpg.de/ift/nifty.git@NIFTy_7
Plotting support is added via:
......@@ -108,7 +108,7 @@ To run the tests, additional packages are required:
Afterwards the tests (including a coverage report) can be run using the
following command in the repository root:
pytest-3 --cov=nifty6 test
pytest-3 --cov=nifty7 test
### First Steps
......
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2019 Max-Planck-Society
# Copyright(C) 2019-2020 Max-Planck-Society
# Author: Martin Reinecke
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -21,70 +21,75 @@ from time import time
import matplotlib.pyplot as plt
import numpy as np
import nifty6 as ift
N0s, a0s, b0s, c0s = [], [], [], []
for ii in range(10, 26):
nu = 1024
nv = 1024
N = int(2**ii)
print('N = {}'.format(N))
rng = ift.random.current_rng()
uv = rng.uniform(-.5, .5, (N,2))
vis = rng.normal(0., 1., N) + 1j*rng.normal(0., 1., N)
uvspace = ift.RGSpace((nu, nv))
visspace = ift.UnstructuredDomain(N)
img = rng.standard_normal((nu, nv))
img = ift.makeField(uvspace, img)
t0 = time()
GM = ift.GridderMaker(uvspace, eps=1e-7, uv=uv)
vis = ift.makeField(visspace, vis)
op = GM.getFull().adjoint
t1 = time()
op(img).val
t2 = time()
op.adjoint(vis).val
t3 = time()
print(t2-t1, t3-t2)
N0s.append(N)
a0s.append(t1 - t0)
b0s.append(t2 - t1)
c0s.append(t3 - t2)
print('Measure rest operator')
sc = ift.StatCalculator()
op = GM.getRest().adjoint
for _ in range(10):
t0 = time()
res = op(img)
sc.add(time() - t0)
t_fft = sc.mean
print('FFT shape', res.shape)
plt.scatter(N0s, a0s, label='Gridder mr')
plt.legend()
# no idea why this is necessary, but if it is omitted, the range is wrong
plt.ylim(min(a0s), max(a0s))
plt.ylabel('time [s]')
plt.title('Initialization')
plt.loglog()
plt.savefig('bench0.png')
plt.close()
plt.scatter(N0s, b0s, color='k', marker='^', label='Gridder mr times')
plt.scatter(N0s, c0s, color='k', label='Gridder mr adjoint times')
plt.axhline(sc.mean, label='FFT')
plt.axhline(sc.mean + np.sqrt(sc.var))
plt.axhline(sc.mean - np.sqrt(sc.var))
plt.legend()
plt.ylabel('time [s]')
plt.title('Apply')
plt.loglog()
plt.savefig('bench1.png')
plt.close()
import nifty7 as ift
def main():
N0s, a0s, b0s, c0s = [], [], [], []
for ii in range(10, 26):
nu = 1024
nv = 1024
N = int(2**ii)
print('N = {}'.format(N))
rng = ift.random.current_rng()
uv = rng.uniform(-.5, .5, (N, 2))
vis = rng.normal(0., 1., N) + 1j*rng.normal(0., 1., N)
uvspace = ift.RGSpace((nu, nv))
visspace = ift.UnstructuredDomain(N)
img = rng.standard_normal((nu, nv))
img = ift.makeField(uvspace, img)
t0 = time()
GM = ift.GridderMaker(uvspace, eps=1e-7, uv=uv)
vis = ift.makeField(visspace, vis)
op = GM.getFull().adjoint
t1 = time()
op(img).val
t2 = time()
op.adjoint(vis).val
t3 = time()
print(t2-t1, t3-t2)
N0s.append(N)
a0s.append(t1 - t0)
b0s.append(t2 - t1)
c0s.append(t3 - t2)
print('Measure rest operator')
sc = ift.StatCalculator()
op = GM.getRest().adjoint
for _ in range(10):
t0 = time()
res = op(img)
sc.add(time() - t0)
print('FFT shape', res.shape)
plt.scatter(N0s, a0s, label='Gridder mr')
plt.legend()
# no idea why this is necessary, but if it is omitted, the range is wrong
plt.ylim(min(a0s), max(a0s))
plt.ylabel('time [s]')
plt.title('Initialization')
plt.loglog()
plt.savefig('bench0.png')
plt.close()
plt.scatter(N0s, b0s, color='k', marker='^', label='Gridder mr times')
plt.scatter(N0s, c0s, color='k', label='Gridder mr adjoint times')
plt.axhline(sc.mean, label='FFT')
plt.axhline(sc.mean + np.sqrt(sc.var))
plt.axhline(sc.mean - np.sqrt(sc.var))
plt.legend()
plt.ylabel('time [s]')
plt.title('Apply')
plt.loglog()
plt.savefig('bench1.png')
plt.close()
if __name__ == '__main__':
main()
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2013-2019 Max-Planck-Society
# Copyright(C) 2013-2020 Max-Planck-Society
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -24,9 +24,10 @@
import numpy as np
import nifty6 as ift
import nifty7 as ift
if __name__ == '__main__':
def main():
# Set up the position space of the signal
mode = 2
if mode == 0:
......@@ -84,3 +85,7 @@ if __name__ == '__main__':
plot.add(GR.adjoint_times(data), title='data')
plot.add(sky(mock_position), title='truth')
plot.output(nx=3, xsize=16, ysize=9, title="results", name="bernoulli.png")
if __name__ == '__main__':
main()
......@@ -80,11 +80,11 @@
%% Cell type:code id: tags:
``` python
import numpy as np
import nifty6 as ift
import nifty7 as ift
import matplotlib.pyplot as plt
%matplotlib inline
```
%% Cell type:markdown id: tags:
......
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2013-2019 Max-Planck-Society
# Copyright(C) 2013-2020 Max-Planck-Society
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -25,7 +25,7 @@ import sys
import numpy as np
import nifty6 as ift
import nifty7 as ift
def make_checkerboard_mask(position_space):
......@@ -38,14 +38,14 @@ def make_checkerboard_mask(position_space):
return mask
def make_random_mask():
def make_random_mask(domain):
# Random mask for spherical mode
mask = ift.from_random(position_space, 'pm1')
mask = ift.from_random(domain, 'pm1')
mask = (mask + 1)/2
return mask.val
if __name__ == '__main__':
def main():
# Choose space on which the signal field is defined
if len(sys.argv) == 2:
mode = int(sys.argv[1])
......@@ -63,7 +63,7 @@ if __name__ == '__main__':
else:
# Sphere with half of its pixels randomly masked
position_space = ift.HPSpace(128)
mask = make_random_mask()
mask = make_random_mask(position_space)
# Specify harmonic space corresponding to signal
harmonic_space = position_space.get_default_codomain()
......@@ -148,3 +148,7 @@ if __name__ == '__main__':
title='Residuals')
plot.output(nx=2, ny=2, xsize=10, ysize=10, name=filename)
print("Saved results as '{}'.".format(filename))
if __name__ == '__main__':
main()
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2013-2019 Max-Planck-Society
# Copyright(C) 2013-2020 Max-Planck-Society
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -25,25 +25,23 @@ import sys
import numpy as np
import nifty6 as ift
import nifty7 as ift
def exposure_2d():
def exposure_2d(domain):
# Structured exposure for 2D mode
x_shape, y_shape = position_space.shape
exposure = np.ones(position_space.shape)
x_shape, y_shape = domain.shape
exposure = np.ones(domain.shape)
exposure[x_shape//3:x_shape//2, :] *= 2.
exposure[x_shape*4//5:x_shape, :] *= .1
exposure[x_shape//2:x_shape*3//2, :] *= 3.
exposure[:, x_shape//3:x_shape//2] *= 2.
exposure[:, x_shape*4//5:x_shape] *= .1
exposure[:, x_shape//2:x_shape*3//2] *= 3.
return ift.Field.from_raw(position_space, exposure)
return ift.Field.from_raw(domain, exposure)
if __name__ == '__main__':
def main():
# Choose space on which the signal field is defined
if len(sys.argv) == 2:
mode = int(sys.argv[1])
......@@ -57,7 +55,7 @@ if __name__ == '__main__':
elif mode == 1:
# Two-dimensional regular grid with inhomogeneous exposure
position_space = ift.RGSpace([512, 512])
exposure = exposure_2d()
exposure = exposure_2d(position_space)
else:
# Sphere with uniform exposure of 100
position_space = ift.HPSpace(128)
......@@ -118,3 +116,7 @@ if __name__ == '__main__':
plot.add(reconst - signal, title='Residuals')
plot.output(xsize=12, ysize=10, name=filename)
print("Saved results as '{}'.".format(filename))
if __name__ == '__main__':
main()
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2013-2019 Max-Planck-Society
# Copyright(C) 2013-2020 Max-Planck-Society
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -29,7 +29,7 @@ import sys
import numpy as np
import nifty6 as ift
import nifty7 as ift
def random_los(n_los):
......@@ -44,7 +44,7 @@ def radial_los(n_los):
return starts, ends
if __name__ == '__main__':
def main():
# Choose between random line-of-sight response (mode=0) and radial lines
# of sight (mode=1)
if len(sys.argv) == 2:
......@@ -60,10 +60,10 @@ if __name__ == '__main__':
# see 'getting_started_4_CorrelatedFields.ipynb'.
cfmaker = ift.CorrelatedFieldMaker.make(
offset_mean = 0.0, # 0.
offset_std_mean = 1e-3, # 1e-3
offset_std_std = 1e-6, # 1e-6
prefix = '')
offset_mean= 0.0,
offset_std_mean= 1e-3,
offset_std_std= 1e-6,
prefix='')
fluctuations_dict = {
# Amplitude of field fluctuations
......@@ -72,7 +72,7 @@ if __name__ == '__main__':
# Exponent of power law power spectrum component
'loglogavgslope_mean': -2.0, # -3.0
'loglogavgslope_stddev': 0.5, # 0.5
'loglogavgslope_stddev': 0.5, # 0.5
# Amplitude of integrated Wiener process power spectrum component
'flexibility_mean': 2.5, # 1.0
......@@ -131,7 +131,7 @@ if __name__ == '__main__':
# Draw new samples to approximate the KL five times
for i in range(5):
# Draw new samples and minimize KL
KL = ift.MetricGaussianKL(mean, H, N_samples)
KL = ift.MetricGaussianKL.make(mean, H, N_samples)
KL, convergence = minimizer(KL)
mean = KL.position
......@@ -144,7 +144,7 @@ if __name__ == '__main__':
name=filename.format("loop_{:02d}".format(i)))
# Draw posterior samples
KL = ift.MetricGaussianKL(mean, H, N_samples)
KL = ift.MetricGaussianKL.make(mean, H, N_samples)
sc = ift.StatCalculator()
for sample in KL.samples:
sc.add(signal(sample + KL.position))
......@@ -163,3 +163,7 @@ if __name__ == '__main__':
linewidth=[1.]*len(powers) + [3., 3.])
plot.output(ny=1, nx=3, xsize=24, ysize=6, name=filename_res)
print("Saved results as '{}'.".format(filename_res))
if __name__ == '__main__':
main()
......@@ -19,11 +19,11 @@
## Setup code
%% Cell type:code id: tags:
``` python
import nifty6 as ift
import nifty7 as ift
import matplotlib.pyplot as plt
import numpy as np
ift.random.push_sseq_from_seed(43)
n_pix = 256
......@@ -33,11 +33,11 @@
%% Cell type:code id: tags:
``` python
# Plotting routine
def plot(fields, spectra, title=None):
# Plotting preparation is normally handled by nifty6.Plot
# Plotting preparation is normally handled by nifty7.Plot
# It is done manually here to be able to tweak details
# Fields are assumed to have identical domains
fig = plt.figure(tight_layout=True, figsize=(12, 3.5))
if title is not None:
fig.suptitle(title, fontsize=14)
......
......@@ -11,7 +11,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Copyright(C) 2013-2019 Max-Planck-Society
# Copyright(C) 2013-2020 Max-Planck-Society
#
# NIFTy is being developed at the Max-Planck-Institut fuer Astrophysik.
......@@ -29,7 +29,7 @@ import sys
import numpy as np
import nifty6 as ift
import nifty7 as ift
class SingleDomain(ift.LinearOperator):
......@@ -55,7 +55,7 @@ def radial_los(n_los):
return starts, ends
if __name__ == '__main__':
def main():
# Choose between random line-of-sight response (mode=0) and radial lines
# of sight (mode=1)
if len(sys.argv) == 2:
......@@ -83,7 +83,7 @@ if __name__ == '__main__':
A2 = cfmaker.normalized_amplitudes[1]
DC = SingleDomain(correlated_field.target, position_space)
## Apply a nonlinearity
# Apply a nonlinearity
signal = DC @ ift.sigmoid(correlated_field)
# Build the line-of-sight response and define signal response
......@@ -118,7 +118,7 @@ if __name__ == '__main__':
ic_newton.enable_logging()
minimizer = ift.NewtonCG(ic_newton, enable_logging=True)
## number of samples used to estimate the KL
# number of samples used to estimate the KL
N_samples = 20
# Set up likelihood and information Hamiltonian
......@@ -131,7 +131,7 @@ if __name__ == '__main__':
for i in range(10):
# Draw new samples and minimize KL
KL = ift.MetricGaussianKL(mean, H, N_samples)
KL = ift.MetricGaussianKL.make(mean, H, N_samples)
KL, convergence = minimizer(KL)
mean = KL.position
......@@ -157,8 +157,7 @@ if __name__ == '__main__':
name=filename.format("loop_{:02d}".format(i)))
# Done, draw posterior samples
Nsamples = 20
KL = ift.MetricGaussianKL(mean, H, N_samples)