ift issueshttps://gitlab.mpcdf.mpg.de/groups/ift/-/issues2022-01-27T11:18:26Zhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/345sample list average crashes for mpi run with map estimator2022-01-27T11:18:26ZJakob Rothsample list average crashes for mpi run with map estimatorWhen computing a map estimate with multiple MPI tasks the sample list average method crashes.
Here is an example code that works if executed normally with python, but crashes in an MPI run
```
import numpy as np
import nifty8 as ift
ift...When computing a map estimate with multiple MPI tasks the sample list average method crashes.
Here is an example code that works if executed normally with python, but crashes in an MPI run
```
import numpy as np
import nifty8 as ift
ift.random.push_sseq_from_seed(27)
try:
from mpi4py import MPI
comm = MPI.COMM_WORLD
master = comm.Get_rank() == 0
except ImportError:
comm = None
master = True
position_space = ift.RGSpace([128, 128])
op = ift.makeOp(ift.full(position_space, 10.))
noise = 0.1
N = ift.ScalingOperator(position_space, noise, np.float64)
mock_position = ift.from_random(op.domain)
data = op(mock_position) + N.draw_sample()
lh = ift.GaussianEnergy(mean=data, inverse_covariance=N.inverse) @ op
ic_sampling = ift.AbsDeltaEnergyController(
name="Sampling (linear)", deltaE=0.05, iteration_limit=10
)
ic_newton = ift.AbsDeltaEnergyController(
name="Newton", deltaE=0.5, convergence_level=2, iteration_limit=5
)
minimizer = ift.NewtonCG(ic_newton)
def callback(samples, i):
plot = ift.Plot()
mean = samples.average(op)
plot.add(mean, title="Reconstruction", zmin=0, zmax=1)
if master:
plot.output()
n_iterations = 3
n_samples = lambda iiter: 0 if iiter < 1 else 2
samples = ift.optimize_kl(
lh,
n_iterations,
n_samples,
minimizer,
ic_sampling,
None,
overwrite=True,
comm=comm,
callback=callback,
)
```https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/341Correlated Field model, `total_N != 0`: Sample statistic is not correct2021-12-08T10:00:54ZPhilipp Arrasparras@mpa-garching.mpg.deCorrelated Field model, `total_N != 0`: Sample statistic is not correctIf I compare prior samples generated by `SimpleCorrelatedField` and `CorrelatedFieldMaker` with `total_N > 0`, the statistics looks different.
```
import nifty8 as ift
offset_mean = 0.
offset_std = None
fluctuations = (0.1, 0.05)
loglo...If I compare prior samples generated by `SimpleCorrelatedField` and `CorrelatedFieldMaker` with `total_N > 0`, the statistics looks different.
```
import nifty8 as ift
offset_mean = 0.
offset_std = None
fluctuations = (0.1, 0.05)
loglogslope = (-4, 1)
dom = ift.RGSpace(2625, 20)
op = ift.SimpleCorrelatedField(dom, offset_mean, offset_std, fluctuations, None, None, loglogslope)
ift.single_plot([op(ift.from_random(op.domain)) for _ in range(20)], name="cfm_single.png")
N = 1
cfm = ift.CorrelatedFieldMaker("", N)
cfm.add_fluctuations(dom, fluctuations, None, None, loglogslope)
cfm.set_amplitude_total_offset(offset_mean, offset_std)
op = cfm.finalize(0)
prior_samples = [op(ift.from_random(op.domain)) for _ in range(10)]
p = ift.Plot()
for ii in range(N):
p.add([ift.DomainTupleFieldInserter(pp.domain, 0, (ii,)).adjoint(pp) for pp in prior_samples])
p.output(name="cfm_multi.png")
```
Prior samples for SimpleCorrelatedField:
![cfm_single](/uploads/3f9db21b292bccab96ec48c94e3296c6/cfm_single.png)
Prior samples for non-trivial CorrelatedFieldMaker with `N_total = 1`
![cfm_multi](/uploads/564ca8466f3e955ab7218bfe8ff8149f/cfm_multi.png)
ping: @jrothhttps://gitlab.mpcdf.mpg.de/ift/resolve/-/issues/3setup.py: multi-line descriptions are not allowed since 59.1.02021-12-01T08:37:32ZPhilipp Arrasparras@mpa-garching.mpg.desetup.py: multi-line descriptions are not allowed since 59.1.0This has to be fixed in line 104 of `setup.py`. I suggest that you, @mtr, do this yourself but if you like I can make a proposal.This has to be fixed in line 104 of `setup.py`. I suggest that you, @mtr, do this yourself but if you like I can make a proposal.https://gitlab.mpcdf.mpg.de/ift/resolve/-/issues/4Calibration Operator in Imaging Likelihood2021-12-01T08:35:37ZJakob RothCalibration Operator in Imaging LikelihoodThe resolve imaging Likelihood accepts a calibration operator as a keyword argument ([https://gitlab.mpcdf.mpg.de/ift/resolve/-/blob/devel/resolve/likelihood.py#L107](https://gitlab.mpcdf.mpg.de/ift/resolve/-/blob/devel/resolve/likelihoo...The resolve imaging Likelihood accepts a calibration operator as a keyword argument ([https://gitlab.mpcdf.mpg.de/ift/resolve/-/blob/devel/resolve/likelihood.py#L107](https://gitlab.mpcdf.mpg.de/ift/resolve/-/blob/devel/resolve/likelihood.py#L107)), but as far as I see this calibration operator is never used.
I think what should be there is something like:
```
model_data = cal_op * R @ sky
res = data - model_data
lh = 0.5 * res^dagger @ inv_cov @ res
```
@parras do you see a mistake here? If not I could implement this fix.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/339optimize_kl: Improvement list2021-11-26T11:52:41ZPhilipp Arrasparras@mpa-garching.mpg.deoptimize_kl: Improvement list- [x] Fix sample plot with ground truth
- [x] Option for storing all intermediate latent positions
- [x] String of model and domain to hdf5
- [x] Save nifty random state
- [x] Pseudo code of optimize_kl into doc string
- [x] plottable sp...- [x] Fix sample plot with ground truth
- [x] Option for storing all intermediate latent positions
- [x] String of model and domain to hdf5
- [x] Save nifty random state
- [x] Pseudo code of optimize_kl into doc string
- [x] plottable spelling?
- [x] Check documentaiton on callback (iglobal)Philipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/337Possible error in assert_equal and assert_allclose for MultiFields2021-09-24T08:03:42ZPhilipp FrankPossible error in assert_equal and assert_allclose for MultiFieldsIn their current form, both methods `assert_equal` and `assert_allclose` only perform a one sided test for `MultiFields`. This leads to some unexpected behaviour when comparing two Fields which are only equal one some sub-domain. In part...In their current form, both methods `assert_equal` and `assert_allclose` only perform a one sided test for `MultiFields`. This leads to some unexpected behaviour when comparing two Fields which are only equal one some sub-domain. In particular the following test
```
import nifty8 as ift
d = ift.RGSpace(10)
d = ift.MultiDomain.make({'a' : d, 'b' : d})
f = ift.from_random(d)
fp = f.extract_by_keys(['a',])
ift.extra.assert_equal(fp, f)
```
will pass, but
```
ift.extra.assert_equal(f, fp)
```
produces an error, as only the keys of the first input are looped over. The same goes for `assert_allclose`.
This does not look like an intended behaviour to me. @mtr, @parras, am I missing something here?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/336Sampling dtypes in sandwich operators2021-09-22T08:26:46ZPhilipp Arrasparras@mpa-garching.mpg.deSampling dtypes in sandwich operatorsI think (but I am not sure), that it is relatively important to fix the FIXME that @pfrank has put here:
https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/operators/sandwich_operator.py#L77I think (but I am not sure), that it is relatively important to fix the FIXME that @pfrank has put here:
https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/operators/sandwich_operator.py#L77https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/335CorrelatedFieldMaker amplitude normalization breaks fluctuation amplitude2021-08-24T10:52:00ZLukas PlatzCorrelatedFieldMaker amplitude normalization breaks fluctuation amplitudeIf one adds more than one amplitude operator to a `CorrelatedFieldMaker` and sets a very small/large offset standard deviation, the field fluctuation amplitude gets distorted.
This is because in `get_normalized_amplitudes()` every ampli...If one adds more than one amplitude operator to a `CorrelatedFieldMaker` and sets a very small/large offset standard deviation, the field fluctuation amplitude gets distorted.
This is because in `get_normalized_amplitudes()` every amplitude operator gets devided by `self.azm` and the results of this function are then multiplied with an outer product to create the joint amplitude operator.
In CFs with more than one amplitude operator, this results in the joint amplitude operator being divided by multiples of the zeromode operator and thus an incorrect output scaling behavior.
To demonstrate this effect, I created correlated field operators for identical 2d fields, but one with a single amplitude operator for `ift.RGSpace((N, N)` and one with two amplitude operators on `ift.RGSpace(N)`. From top to bottom, I varied the `offset std` setting passed to `set_amplitude_total_offset()` and in each case plotted a histogram of the sampled fields' standard deviation.
![output](/uploads/6779601e231148c36a77a91aa8e7fd62/bug.png)
One can clearly see the output fluctuation amplitude is independant from the `offset_std` for the 'single' case, as it should be, but not for the 'dual' case.
A fix for this is proposed in merge requests !669 and !670.
To reproduce, run the following code:
```
import nifty7 as ift
import matplotlib.pyplot as plt
fluct_pars = {
'fluctuations': (1.0, 0.1),
'flexibility': None,
'asperity': None,
'loglogavgslope': (-2.0, 0.2),
'prefix': 'flucts_',
'dofdex': None
}
n_offsets = 5
n_samples = 1000
dom_single = ift.RGSpace((25, 25))
dom_dual = ift.RGSpace(25)
fig, axs = plt.subplots(nrows=n_offsets, ncols=2, figsize=(12, 4 * n_offsets))
for i in range(n_offsets):
cfmaker_single = ift.CorrelatedFieldMaker(prefix='str(i)_')
cfmaker_dual = ift.CorrelatedFieldMaker(prefix='str(i)_')
cfmaker_single.add_fluctuations(dom_single, **fluct_pars)
cfmaker_dual.add_fluctuations(dom_dual, **fluct_pars)
cfmaker_dual.add_fluctuations(dom_dual, **fluct_pars)
offset_std = 10 ** -i
zm_pars = {'offset_mean': 0.,
'offset_std': (offset_std, offset_std / 10.)}
cfmaker_single.set_amplitude_total_offset(**zm_pars)
cfmaker_dual.set_amplitude_total_offset(**zm_pars)
cf_single = cfmaker_single.finalize(prior_info=0)
cf_dual = cfmaker_dual.finalize(prior_info=0)
samples_single = [cf_single(ift.from_random(cf_single.domain)).val.std() for _ in range(n_samples)]
samples_dual = [cf_dual(ift.from_random(cf_dual.domain)).val.std() for _ in range(n_samples)]
_ = axs[i, 0].hist(samples_single, bins=20, density=True)
_ = axs[i, 1].hist(samples_dual, bins=20, density=True)
axs[i, 0].set_title(f'offset_std = {offset_std:1.0e}, single')
axs[i, 1].set_title(f'offset_std = {offset_std:1.0e}, dual')
axs[i, 0].set_xlabel('sample std')
axs[i, 1].set_xlabel('sample std')
plt.tight_layout()
plt.show()
```
Any comments?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/333New feature: Add flag to minisanity that disables terminal colors2021-08-05T12:39:13ZPhilipp Arrasparras@mpa-garching.mpg.deNew feature: Add flag to minisanity that disables terminal colorsPhilipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/334`MetricGaussianKL` changed its interface (there is not `.make`) but there is ...2021-07-01T15:34:54ZGordian Edenhofer`MetricGaussianKL` changed its interface (there is not `.make`) but there is no corresponding changelog entryhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/323Final steps for release2021-06-11T19:59:51ZPhilipp Arrasparras@mpa-garching.mpg.deFinal steps for release- [x] Check documentation generation for warnings and errors -> !636
- [x] Check pytest output for warnings -> #329 + !635, !634
- [x] Run `isort` on whole repository
```sh
find . -type f -name '*.py' | xargs isort
```
- [x] Update aut...- [x] Check documentation generation for warnings and errors -> !636
- [x] Check pytest output for warnings -> #329 + !635, !634
- [x] Run `isort` on whole repository
```sh
find . -type f -name '*.py' | xargs isort
```
- [x] Update author list again
```sh
git shortlog -s origin/NIFTy_6..origin/NIFTy_7 | cut -f2-
```
- [x] Fast-forward `NIFTy_8` to `origin/NIFTy_7` branch
- [x] Rename nifty7 -> nifty8 on NIFTy_8 branch
```sh
rm nifty7
ln -s src nifty8
find . -type f -exec sed -i 's/nifty7/nifty8/g' {} \;
find . -type f -exec sed -i 's/NIFTy_7/NIFTy_8/g' {} \;
find . -type f -exec sed -i 's/NIFTy7/NIFTy8/g' {} \; # Only partially!
```
- [x] Rename NIFTy_6 -> NIFTy_7 in `.gitlab-ci.yml` (for doc generation)
- [x] NIFTy_8 subsection in `README.md` in contributors section
- [x] NIFTy_7 section in `ChangeLog.md`
- [x] Make NIFTy_7 the default branch of the gitlab repository
- [x] Add daily pipeline schedule for `NIFTy_8` branch and remove the one for `NIFTy_6`NIFTy7 releasePhilipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/330Write guide how to define custom non-linearities2021-06-10T11:39:25ZPhilipp Arrasparras@mpa-garching.mpg.deWrite guide how to define custom non-linearitiesNIFTy7 releasehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/332Rename `Likelihood` -> `LogLikelihood`2021-06-10T10:44:36ZPhilipp Arrasparras@mpa-garching.mpg.deRename `Likelihood` -> `LogLikelihood`NIFTy7 releasehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/331Build html coverage report of both release and develop branch2021-06-09T15:51:21ZPhilipp Arrasparras@mpa-garching.mpg.deBuild html coverage report of both release and develop branchNIFTy7 releasehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/329Performance check (in ift.check_operator) broken for BernoulliEnergy2021-06-07T10:42:14ZLukas PlatzPerformance check (in ift.check_operator) broken for BernoulliEnergyThe operator performance check introduced in [`a50ec021`](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commit/a50ec021e457dc8f4e199c6331c67167eec0fdb7) changes the operator test position [by a factor of two](https://gitlab.mpcdf.mpg.de/ift/ni...The operator performance check introduced in [`a50ec021`](https://gitlab.mpcdf.mpg.de/ift/nifty/-/commit/a50ec021e457dc8f4e199c6331c67167eec0fdb7) changes the operator test position [by a factor of two](https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/a50ec021e457dc8f4e199c6331c67167eec0fdb7/nifty6/extra.py#L146) for the 'application to a linearization' test. This breaks the test for the `BernoulliEnergy`, as it creates input values outside the interval of `[0, 1)`.
@reimar, since you coded this function: does the factor of two serve a valuable purpose, or can we safely delete it?NIFTy7 releasehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/325Update general documentation to GeoVI2021-05-31T08:43:03ZPhilipp Arrasparras@mpa-garching.mpg.deUpdate general documentation to GeoVIAt the end of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_7/docs/source/ift.rst MGVI is explicitly discussed. Let's add a discussion on GeoVI here or alternatively add a new file to the documentation for GeoVI alone.At the end of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_7/docs/source/ift.rst MGVI is explicitly discussed. Let's add a discussion on GeoVI here or alternatively add a new file to the documentation for GeoVI alone.NIFTy7 releasehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/326Check NIFTy_7 documentation generation for error messages2021-05-30T17:30:29ZPhilipp Arrasparras@mpa-garching.mpg.deCheck NIFTy_7 documentation generation for error messagesNIFTy7 releasePhilipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/322Update contributors list2021-05-30T17:28:48ZPhilipp Arrasparras@mpa-garching.mpg.deUpdate contributors listNIFTy7 releasePhilipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/324Update `demos/mgvi_visualized.py` to GeoVI2021-05-30T14:49:38ZPhilipp Arrasparras@mpa-garching.mpg.deUpdate `demos/mgvi_visualized.py` to GeoVINIFTy7 releasePhilipp Arrasparras@mpa-garching.mpg.dePhilipp Arrasparras@mpa-garching.mpg.dehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/321Windows compatibility2021-05-05T17:08:08ZLukas PlatzWindows compatibilityA collaborator of mine just tried to install Nifty on a Windows machine in an Anaconda environment and had the problem that the symlink from `nifty7` to `src` was apparently breaking the setup. Once she removed it and renamed `src` to `n...A collaborator of mine just tried to install Nifty on a Windows machine in an Anaconda environment and had the problem that the symlink from `nifty7` to `src` was apparently breaking the setup. Once she removed it and renamed `src` to `nifty7`, the setup worked.
I have not checked if this is the general behavior under Windows or if it is just her setup, but assume it is the former.
Has anybody else experience with this and can weigh in?
What was the rationale behind changing the source location and introducing the symlink?
Cheers,
Lukas