ift issueshttps://gitlab.mpcdf.mpg.de/groups/ift/-/issues2017-09-15T23:25:14Zhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/182Completely remove inverse smoothing2017-09-15T23:25:14ZMartin ReineckeCompletely remove inverse smoothingThe inverse_times() of both `SmoothingOerator`s is currently broken.
`DirectSmoothingOperator` considers the inverse operation of "smoothing with sigma" to be "smoothing with 1/sigma", which is simply wrong.
`FFTSmoothingOperator` does...The inverse_times() of both `SmoothingOerator`s is currently broken.
`DirectSmoothingOperator` considers the inverse operation of "smoothing with sigma" to be "smoothing with 1/sigma", which is simply wrong.
`FFTSmoothingOperator` does in principle perform the correct operations, but does not check for division by zero, which occurs for most sigma values and indicates that the smoothing is in fact non-invertible with finite precision arithmetics.
Unfortunately the test `test_inverse_adjoint_times` does not recognize this, because the results it compares are both NaN and therefore the same...
Since this operation appears to be unused, I'd suggest to remove it ... doing it properly may be possible (at least for FFT smoothing), but is only worth the effort if someone really needs it.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/181Logarithmic smoothing on non-PowerSpaces?2017-09-15T23:25:48ZMartin ReineckeLogarithmic smoothing on non-PowerSpaces?Currently, NIFTy allows to construct a `SmoothingOperator` like this:
```
import nifty as ift
op=ift.SmoothingOperator(ift.RGSpace(10),sigma=1.,log_distances=True)
```
I do not think this makes sense, as the RGSace is periodic and ther...Currently, NIFTy allows to construct a `SmoothingOperator` like this:
```
import nifty as ift
op=ift.SmoothingOperator(ift.RGSpace(10),sigma=1.,log_distances=True)
```
I do not think this makes sense, as the RGSace is periodic and there are no absolute distances.
This combination of parameters also leads to overflows because the code in FFTSmoothingOperator tries to compute log(0.).
Therefore I propose to ensure that `log_distance==False` whenever an FFTSmoothingOperator is constructed and raise an exception otherwise.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/180Use MPCDF's runners for continuous integration?2017-11-13T09:39:16ZMartin ReineckeUse MPCDF's runners for continuous integration?It seems that MPCDF now offers shared runners (http://www.mpcdf.mpg.de/about-mpcdf/publications/bits-n-bytes?BB-View=196&BB-Doc=187).
Should we try to use these instead of Theo's machine?It seems that MPCDF now offers shared runners (http://www.mpcdf.mpg.de/about-mpcdf/publications/bits-n-bytes?BB-View=196&BB-Doc=187).
Should we try to use these instead of Theo's machine?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/179Field.from_random crashes mpi2018-02-15T14:31:18ZJakob KnollmuellerField.from_random crashes mpiHi,
I found another bug with mpi:
[minimal_example.py](/uploads/3863c7e25ac0661caab65a64bd964264/minimal_example.py)
the error I get is:
```
[MainThread][ERROR ] root Uncaught exception
Traceback (most recent call last):
F...Hi,
I found another bug with mpi:
[minimal_example.py](/uploads/3863c7e25ac0661caab65a64bd964264/minimal_example.py)
the error I get is:
```
[MainThread][ERROR ] root Uncaught exception
Traceback (most recent call last):
File "minimal_example-3.py", line 16, in <module>
std=some_diagonal.val)
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.1.0-py2.7.egg/nifty/field.py", line 251, in from_random
lambda shape: generator_function(dtype=f.dtype,
File "/usr/local/lib/python2.7/dist-packages/d2o/distributed_data_object.py", line 469, in apply_generator
self.set_local_data(generator(self.distributor.local_shape), copy=copy)
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.1.0-py2.7.egg/nifty/field.py", line 253, in <lambda>
**random_arguments))
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.1.0-py2.7.egg/nifty/random.py", line 50, in normal
x *= dtype.type(std)
ValueError: operands could not be broadcast together with shapes (16,64,64) (64,64,64) (16,64,64)
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[29272,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
I only get it if the std in from_random is a array, for numbers it seems to work
Jakobhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/178Replace the current implementation of MPI FFTs2018-01-11T14:25:44ZMartin ReineckeReplace the current implementation of MPI FFTsThe current way of performing MPI-parallel FFTs has several problems:
- it relies on a fork of `pyfftw` that may or may not be merged with the official version in the future, causing potential confusion for users
- it does not work for ...The current way of performing MPI-parallel FFTs has several problems:
- it relies on a fork of `pyfftw` that may or may not be merged with the official version in the future, causing potential confusion for users
- it does not work for all array sizes due to FFTW limitations.
The second point already makes broad regression testing of many different FFT sizes quite tricky.
My suggestion to overcome both these drawbacks is based on the fact that MPI communication during an FFT is _only_ needed if the first field dimension needs to be transformed, and that a multi-D FFT can be separated in to FFTs along individual axes.
If an FFT along the first axis is required, NIFTy (or D2O) could just do the following:
- MPI-tanspose the field in such a way that the first dimension is no longer distributed across CPUs
- perform the FFT along this dimension (no MPI required)
- revert the transpose again
- (perform FFTs along the other requested axes)
Internally this is exactly how FFTW handles this problem as well.
The transposition algorithm is not trivial, but certainly implementable without too many difficulties.
Doing this would also get rid of the special "fftw" distribution strategy.
Additional bonus: MPI FFTs would then also work with numpy FFT. The dependence on MPI-enabled FFTW would vanish, making NIFTy configuration simpler.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/177Documentation for FFTSmoothingOperator and DirectSmoothingOperator missing2018-02-21T21:18:52ZPhilipp Arrasparras@mpa-garching.mpg.deDocumentation for FFTSmoothingOperator and DirectSmoothingOperator missinghttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/176mpi crashes with power_analyze2017-08-14T22:04:56ZJakob Knollmuellermpi crashes with power_analyzeCalling power_analyze of a fftw distributed field crashes the mpirun. I attached a minimal example:
[minimal_example.py](/uploads/3253d1e327fb18e6ec1c6326779cea6f/minimal_example.py)
I run it with:
mpirun -n 2 python minimal_example.py ...Calling power_analyze of a fftw distributed field crashes the mpirun. I attached a minimal example:
[minimal_example.py](/uploads/3253d1e327fb18e6ec1c6326779cea6f/minimal_example.py)
I run it with:
mpirun -n 2 python minimal_example.py
and I get:
```
[MainThread][ERROR ] root Uncaught exception
Traceback (most recent call last):
File "minimal_example.py", line 22, in <module>
pp=sh.power_analyze()
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.0.4-py2.7.egg/nifty/field.py", line 377, in power_analyze
for part in parts]
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.0.4-py2.7.egg/nifty/field.py", line 413, in _single_power_analyze
axes=work_field.domain_axes[space_index])
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.0.4-py2.7.egg/nifty/field.py", line 439, in _calculate_power_spectrum
axes=axes)
File "/usr/local/lib/python2.7/dist-packages/ift_nifty-3.0.4-py2.7.egg/nifty/field.py", line 469, in _shape_up_pindex
semiscaled_local_data = local_data.reshape(semiscaled_shape)
ValueError: cannot reshape array of size 131072 into shape (64,64,64)
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[12289,1],0]
Exit code: 1
--------------------------------------------------------------------------
```https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/175Indexing bug in Laplace operator2017-08-17T23:22:06ZMartin ReineckeIndexing bug in Laplace operatorI'm running the following test case on current master:
```
from nifty import *
s_space = RGSpace([256,256])
FFT = FFTOperator(s_space)
h_space = FFT.target[0]
p_space = PowerSpace(h_space, logarithmic=True, distribution_strategy='not')...I'm running the following test case on current master:
```
from nifty import *
s_space = RGSpace([256,256])
FFT = FFTOperator(s_space)
h_space = FFT.target[0]
p_space = PowerSpace(h_space, logarithmic=True, distribution_strategy='not')
t0 = Field(p_space, val=log(0.1/(1+p_space.kindex)**3))
sop = SmoothnessOperator(p_space)
#print t0.val.get_full_data()
print t0.vdot(sop(t0))
#print t0.val.get_full_data()
print t0.vdot(sop(t0))
```
This produces the output:
```
0.645594411887
139746.578501
```
which is definitely incorrect, since the same number should be reported twice.
Things become even weirder. If I remove the commenting "#" from the first print statement, I get:
```
[ -2.30258509 -4.38202663 -4.94670585 -5.75181152 -6.49001612
-7.17883085 -8.05624052 -8.99425111 -9.93125436 -10.87947582
-11.85503513 -12.85994974 -16.24599361]
2.93684941121
139746.578501
```
So the print statement changes the result of the next statement?!
Removing the next "#" as well produces:
```
[ -2.30258509 -4.38202663 -4.94670585 -5.75181152 -6.49001612
-7.17883085 -8.05624052 -8.99425111 -9.93125436 -10.87947582
-11.85503513 -12.85994974 -16.24599361]
2.93684941121
[ -2.30258509 -4.38202663 -4.94670585 -5.75181152 -6.49001612
-7.17883085 -8.05624052 -8.99425111 -9.93125436 -10.87947582
-11.85503513 -12.85994974 -16.24599361]
2.93684941121
```
First question: can anyone reproduce this, or is it a bug in my local Python installation?
If this is reproducible, it definitely needs to be fixed quickly, but I'm completely at my wits' end ...https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/174More zerocenter weirdness...2017-08-15T00:50:33ZMartin ReineckeMore zerocenter weirdness...```
from nifty import *
s=RGSpace(3,zerocenter=True)
f=Field(s,val=[0.,1.,2.])
op=FFTOperator(s)
print "op domain:", op.domain[0]
print "op target:", op.target[0]
print "op times real field", op.times(f).val
```
This prints
```
op doma...```
from nifty import *
s=RGSpace(3,zerocenter=True)
f=Field(s,val=[0.,1.,2.])
op=FFTOperator(s)
print "op domain:", op.domain[0]
print "op target:", op.target[0]
print "op times real field", op.times(f).val
```
This prints
```
op domain: RGSpace(shape=(3,), zerocenter=(True,), distances=(0.33333333333333331,), harmonic=False)
op target: RGSpace(shape=(3,), zerocenter=(True,), distances=(1.0,), harmonic=True)
op times real field [-0.33333333+0.j -0.16666667+0.8660254j 0.16666667+0.8660254j]
```
What I don't understand: if the target domain of the operator is zero-centered, why is the first value of the result real-only, and not the middle one?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/173Remove one of the WienerFilter demos2017-08-15T08:09:44ZPhilipp Arrasparras@mpa-garching.mpg.deRemove one of the WienerFilter demosWas ist der Unterschied zwischen den beiden Wiener Filter demos?
`demos/wiener_filter_via_curvature.py` macht Folgendes:
```
wiener_curvature = WienerFilterCurvature(S=S, N=N, R=R_harmonic)
m = wiener_curvature.inverse_times(j)
```
`de...Was ist der Unterschied zwischen den beiden Wiener Filter demos?
`demos/wiener_filter_via_curvature.py` macht Folgendes:
```
wiener_curvature = WienerFilterCurvature(S=S, N=N, R=R_harmonic)
m = wiener_curvature.inverse_times(j)
```
`demos/wiener_filter_via_hamiltonian.py` macht:
```
m0 = Field(h_space, val=.0)
energy = WienerFilterEnergy(position=m0, d=d, R=R, N=N, S=S, inverter=inverter)
D0 = energy.curvature
m0 = D0.inverse_times(j)
```
Soweit ich das verstehe, ist der einzige Unterschied zwischen den beiden Demos, dass bei der Zweiten eine Position und ein Invertierer definiert werden, die dann aber gar nicht verwendet werden (weil `energy.curvature` nur R, N und S braucht und keine Position).https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/172Overflow using lognormal2017-08-14T22:05:30ZMatevz, Sraml (sraml)Overflow using lognormalIn my project I use constantly lognormal model and since the normalization of the gradient in the SteepestDescent was removed I get overflow. I fix that with normalizing my gradient.
The overflow can be reproduced with this:
[test.py](...In my project I use constantly lognormal model and since the normalization of the gradient in the SteepestDescent was removed I get overflow. I fix that with normalizing my gradient.
The overflow can be reproduced with this:
[test.py](/uploads/5f5ce380ae65b699b5e350fd54ba8b18/test.py)https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/171create_power_operator() returns operator with incorrect data type2017-08-15T08:17:02ZMartin Reineckecreate_power_operator() returns operator with incorrect data typeIf I understood the purpose of `create_power_operator()` correctly, it should produce a `DiagonalOperator` which imprints the supplied power spectrum onto a given field in harmonic space.
I would expect that this returned operator conta...If I understood the purpose of `create_power_operator()` correctly, it should produce a `DiagonalOperator` which imprints the supplied power spectrum onto a given field in harmonic space.
I would expect that this returned operator contains values of real type, but its data type is complex, even if I explicitly ask for real numbers:
```
from nifty import *
import numpy as np
s=RGSpace(10,harmonic=True)
def ps(x): return x
x=create_power_operator(s,ps,dtype=np.float64)
print x.diagonal().dtype
```
What am I missing?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/170CriticalPowerEnergy has incomplete at() method2017-07-19T11:57:02ZMartin ReineckeCriticalPowerEnergy has incomplete at() methodThe `at()` method in `CriticalPowerEnergy` does not pass the `logarithmic` property to the newly built object, which results in a (potentially) different T inside that object.
What's the best solution to fix this? Store `logarithmic` ex...The `at()` method in `CriticalPowerEnergy` does not pass the `logarithmic` property to the newly built object, which results in a (potentially) different T inside that object.
What's the best solution to fix this? Store `logarithmic` explicitly to have it available in the `at()` method?
@kjako, any preferences?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/169destroying distribution_strategy2017-07-26T08:27:00ZJakob Knollmuellerdestroying distribution_strategyHi,
I am currently trying to get my code running using the full capabilities of d2o, but some things in nifty seem to destroy the distribution_strategy, namely the InvertibleOperatorMixin and power_analyze (I tracked it down to the pind...Hi,
I am currently trying to get my code running using the full capabilities of d2o, but some things in nifty seem to destroy the distribution_strategy, namely the InvertibleOperatorMixin and power_analyze (I tracked it down to the pindex.bincount, see the example)
[minimal_example.py](/uploads/95bab8b0159517babf98330ce2a9ae5c/minimal_example.py)
Another problem is in the InvertibleOperatorMixin:
If no initial guess x0 is specified it is set to
x0 = Field(self.target, val=0., dtype=x.dtype)
setting the distribution to the default of "not" so the result also has this distribution, I would suggest setting
x0 = Field(self.target, val=0., dtype=x.dtype, distribution_strategy=x.distribution_strategy)
Jakobhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/168Bugs in line_search_strong_wolfe2017-08-16T12:25:13ZMartin ReineckeBugs in line_search_strong_wolfeThere are several strange "features" in the line search algorithm:
- if the _zoom procedure does not converge, the outer minimization loop repeats the (failing) _zoom search in identical fashion until it reaches its own iteration limit....There are several strange "features" in the line search algorithm:
- if the _zoom procedure does not converge, the outer minimization loop repeats the (failing) _zoom search in identical fashion until it reaches its own iteration limit. This can't be intentional.
- The Wolfe conditions look a bit different to the ones mentioned in Wikipedia.
Do we have a reference paper for this implementation?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/167Field data types must be enforced consistently2017-08-15T08:05:09ZMartin ReineckeField data types must be enforced consistentlyI'm trying to debug the line searching algorithm and noticed that some of the input quantities have complex type.
This is not really acceptable, because there are no useful comparison operators defined for complex numbers, and the code ...I'm trying to debug the line searching algorithm and noticed that some of the input quantities have complex type.
This is not really acceptable, because there are no useful comparison operators defined for complex numbers, and the code does lots of comparisons. I don't understand why no exceptions are thrown; pure Python immediately complains if complex numbers are compared via <,>, etc., but numpy somehow doesn't.
We really need strict enforcement that search positions, gradients and energy values have real type.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/166MPI-enabled tests2017-07-29T11:50:06ZMartin ReineckeMPI-enabled testsMostly for fun, I enabled MPI nosetests in out continuous integration, and it seems that many tests actually continue to work!
But unfortunately I think I managed to freeze the test VM, so the pipeline is stuck now :(Mostly for fun, I enabled MPI nosetests in out continuous integration, and it seems that many tests actually continue to work!
But unfortunately I think I managed to freeze the test VM, so the pipeline is stuck now :(https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/165Critical power energy doesn't work for product spaces2017-08-02T21:20:47ZTheo SteiningerCritical power energy doesn't work for product spacesIn principle it is no problem to port the existing code to product-space functionality. The biggest thing is that we then would need a n-dimensional SmoothnessOperator. Any volunteers?In principle it is no problem to port the existing code to product-space functionality. The biggest thing is that we then would need a n-dimensional SmoothnessOperator. Any volunteers?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/164Another Field.vdot() bug...2017-07-11T08:18:02ZMartin ReineckeAnother Field.vdot() bug...```
from nifty import *
import numpy as np
s=RGSpace((10,))
f1=Field.from_random("normal",domain=s,dtype=np.complex128)
f2=Field.from_random("normal",domain=s,dtype=np.complex128)
print f1.vdot(f2)
print f1.vdot(f2,spaces=0)
``````
from nifty import *
import numpy as np
s=RGSpace((10,))
f1=Field.from_random("normal",domain=s,dtype=np.complex128)
f2=Field.from_random("normal",domain=s,dtype=np.complex128)
print f1.vdot(f2)
print f1.vdot(f2,spaces=0)
```https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/163Descent minimizers: stopping criterion is broken2017-09-28T01:18:15ZMartin ReineckeDescent minimizers: stopping criterion is brokenCurrently, descent_minimizer.py contains the following lines:
```
elif delta < self.convergence_tolerance:
convergence += 1
self.logger.info("Updated convergence level to: %u" %
...Currently, descent_minimizer.py contains the following lines:
```
elif delta < self.convergence_tolerance:
convergence += 1
self.logger.info("Updated convergence level to: %u" %
convergence)
```
The problem is that `delta` never goes down far enough for the algorithms to stop in there, even though the energy is tiny compared to the starting energy (example below).
This can't be a valid stopping criterion.
```
VL_BFGS: DEBUG: Iteration:00000001 step_length=3.7E+01 delta=1.4E+00 energy=5.1E+01
VL_BFGS: DEBUG: Iteration:00000002 step_length=7.5E+00 delta=1.5E+00 energy=4.6E+00
VL_BFGS: DEBUG: Iteration:00000003 step_length=2.6E+00 delta=1.9E+00 energy=6.5E-01
VL_BFGS: DEBUG: Iteration:00000004 step_length=1.4E+00 delta=3.4E+00 energy=2.3E-02
VL_BFGS: DEBUG: Iteration:00000005 step_length=3.1E-01 delta=5.2E+00 energy=8.8E-03
VL_BFGS: DEBUG: Iteration:00000006 step_length=1.1E-01 delta=2.9E+00 energy=2.3E-04
VL_BFGS: DEBUG: Iteration:00000007 step_length=1.5E-02 delta=2.5E+00 energy=5.0E-05
VL_BFGS: DEBUG: Iteration:00000008 step_length=1.3E-02 delta=6.7E+00 energy=1.2E-06
VL_BFGS: DEBUG: Iteration:00000009 step_length=2.4E-03 delta=6.4E+00 energy=6.2E-07
VL_BFGS: DEBUG: Iteration:00000010 step_length=9.0E-04 delta=3.0E+00 energy=1.5E-08
VL_BFGS: DEBUG: Iteration:00000011 step_length=1.2E-04 delta=2.7E+00 energy=2.7E-09
VL_BFGS: DEBUG: Iteration:00000012 step_length=8.7E-05 delta=7.7E+00 energy=9.6E-11
VL_BFGS: DEBUG: Iteration:00000013 step_length=1.9E-05 delta=6.2E+00 energy=1.4E-11
VL_BFGS: DEBUG: Iteration:00000014 step_length=5.5E-06 delta=3.9E+00 energy=1.3E-12
VL_BFGS: DEBUG: Iteration:00000015 step_length=1.4E-06 delta=3.2E+00 energy=6.4E-14
VL_BFGS: DEBUG: Iteration:00000016 step_length=2.5E-07 delta=4.3E+00 energy=1.4E-14
VL_BFGS: DEBUG: Iteration:00000017 step_length=2.2E-07 delta=1.0E+01 energy=1.8E-16
VL_BFGS: DEBUG: Iteration:00000018 step_length=1.6E-08 delta=3.6E+00 energy=1.4E-17
VL_BFGS: DEBUG: Iteration:00000019 step_length=3.3E-09 delta=1.7E+00 energy=3.5E-18
VL_BFGS: DEBUG: Iteration:00000020 step_length=3.2E-09 delta=4.8E+00 energy=9.6E-20
VL_BFGS: DEBUG: Iteration:00000021 step_length=6.2E-10 delta=4.6E+00 energy=3.0E-20
VL_BFGS: DEBUG: Iteration:00000022 step_length=2.0E-10 delta=3.6E+00 energy=1.0E-21
VL_BFGS: DEBUG: Iteration:00000023 step_length=3.3E-11 delta=2.6E+00 energy=2.0E-22
VL_BFGS: DEBUG: Iteration:00000024 step_length=2.6E-11 delta=6.8E+00 energy=6.3E-24
VL_BFGS: DEBUG: Iteration:00000025 step_length=5.8E-12 delta=8.6E+00 energy=4.5E-24
VL_BFGS: DEBUG: Iteration:00000026 step_length=1.0E-12 delta=1.5E+00 energy=1.7E-24
VL_BFGS: DEBUG: Iteration:00000027 step_length=1.6E-12 delta=3.6E+00 energy=2.4E-26
VL_BFGS: DEBUG: Iteration:00000028 step_length=1.5E-13 delta=4.8E+00 energy=5.9E-27
VL_BFGS: DEBUG: Iteration:00000029 step_length=1.5E-13 delta=1.1E+01 energy=3.0E-29
VL_BFGS: DEBUG: Iteration:00000030 step_length=7.0E-15 delta=3.2E+00 energy=4.3E-30
VL_BFGS: WARNING: Reached iteration limit. Stopping.
```