ift issueshttps://gitlab.mpcdf.mpg.de/groups/ift/-/issues2023-11-27T15:01:57Zhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/394remove .envrc2023-11-27T15:01:57ZVincent Eberleremove .envrchttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/393Register `Likelihood` as PyTree2024-01-21T17:49:36ZGordian EdenhoferRegister `Likelihood` as PyTreeIf `Likelihood` would be a registered PyTree, we could compile `draw_sample(likelihood, primals, *a, **k)` and the data would be traced instead of inlined (I think)!If `Likelihood` would be a registered PyTree, we could compile `draw_sample(likelihood, primals, *a, **k)` and the data would be traced instead of inlined (I think)!Gordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/391any check for lognormal_moments2023-11-24T23:43:23ZJulian Rüstigany check for lognormal_momentsInside the stats_distributions, a check ensuring positivity for the mean and the std for the lognormal_distribution, throws an error if one supplies an array for the mean or the standard deviation.Inside the stats_distributions, a check ensuring positivity for the mean and the std for the lognormal_distribution, throws an error if one supplies an array for the mean or the standard deviation.Julian RüstigJulian Rüstighttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/390update likelihood sum domains2023-11-25T16:11:30ZJulian Rüstigupdate likelihood sum domainsWhen adding two likelihoods in nifty.re the domains should be updated.When adding two likelihoods in nifty.re the domains should be updated.Julian RüstigJulian Rüstighttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/389Incorrect number of samples in optimize_kl2023-11-27T10:15:22ZAymeric GalanIncorrect number of samples in optimize_klAfter I updated my script to the last changes of `re.optimize_kl`, it seems that when I provide a varying number of samples (i.e. `n_samples` is a callable), `optimize_kl()` only returns a number of samples that corresponds to the first ...After I updated my script to the last changes of `re.optimize_kl`, it seems that when I provide a varying number of samples (i.e. `n_samples` is a callable), `optimize_kl()` only returns a number of samples that corresponds to the first iteration defined in `n_samples`. Not sure if the during optimization it actually uses the right number of samples though (it seems to do so).https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/387remove flake.lock2023-11-27T15:01:48ZVincent Eberleremove flake.lockhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/386remove flake.nix2023-11-27T15:01:24ZVincent Eberleremove flake.nixhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/385update setup.py2023-11-17T18:19:11ZVincent Eberleupdate setup.pyVincent EberleVincent Eberlehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/384update pyproject.toml2023-11-17T18:19:03ZVincent Eberleupdate pyproject.tomlVincent EberleVincent Eberlehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/383Embrace SPEC 0 Minimum Supported Version https://scientific-python.org/specs/...2024-01-12T17:22:59ZGordian EdenhoferEmbrace SPEC 0 Minimum Supported Version https://scientific-python.org/specs/spec-0000/Akin to Numpy and Scipy, I propose to only support recent versions of python, numpy, and scipy (released within the last 3 years) in development branches. See https://scientific-python.org/specs/spec-0000/ for details on the joint effort...Akin to Numpy and Scipy, I propose to only support recent versions of python, numpy, and scipy (released within the last 3 years) in development branches. See https://scientific-python.org/specs/spec-0000/ for details on the joint effort of the scientific community to standardize this. We would continue to support older versions of NIFTy which work with older versions of python, numpy, and scipy but the development version need only to work with SPEC 0 compliant versions. Specifically, this would drop python3.8 support. @allhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/378Matern-kernel fluctuations in NIFTy.re2023-11-29T20:14:47ZMatteo GuardianiMatern-kernel fluctuations in NIFTy.reImplement `add_fluctuations_matern` in `correlated_field.py`.Implement `add_fluctuations_matern` in `correlated_field.py`.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/374Restructure NIFTy.re.likelihoods2024-01-21T17:49:35ZGordian EdenhoferRestructure NIFTy.re.likelihoodshttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/369Thread safety in optimize_kl2023-11-07T11:27:57ZLandman BesterThread safety in optimize_klThe following code attempts to run a number of optimize_kl instances in parallel, first with a process pool and then with a thread pool.
```python
import numpy as np
import nifty8 as ift
import concurrent.futures as cf
def function_ca...The following code attempts to run a number of optimize_kl instances in parallel, first with a process pool and then with a thread pool.
```python
import numpy as np
import nifty8 as ift
import concurrent.futures as cf
def function_calling_nifty(seed):
# set up correlated field model
npix = 512
pospace = ift.RGSpace((npix,))
spfreq = ift.RGSpace((npix,))
cfmaker = ift.CorrelatedFieldMaker('amplitude')
cfmaker.add_fluctuations(spfreq, (0.1, 1e-2), None, None, (-3, 1),
'f')
cfmaker.set_amplitude_total_offset(0., (1e-2, 1e-6))
cf = cfmaker.finalize()
normalized_amp = cfmaker.get_normalized_amplitudes()
pspec = normalized_amp[0]**2
# signal + fake data
signal = ift.exp(cf.real)
mask = np.random.binomial(1, 0.5, size=npix).astype(bool)
tmp = ift.makeField(signal.target, ~mask)
R = ift.MaskOperator(tmp)
signal_response = R(signal)
mock_position = ift.from_random(signal_response.domain, 'normal')
dspace = R.target
noise = .001
N = ift.ScalingOperator(dspace, noise, float)
data = signal_response(mock_position) + N.draw_sample()
# Minimization parameters
ic_sampling = ift.AbsDeltaEnergyController(deltaE=0.01, iteration_limit=50)
ic_newton = ift.AbsDeltaEnergyController(deltaE=0.01, iteration_limit=15)
minimizer = ift.NewtonCG(ic_newton, enable_logging=False)
# Set up likelihood energy and information Hamiltonian
likelihood_energy = ift.GaussianEnergy(data, inverse_covariance=N.inverse) @ signal_response
n_samples = 3
n_iterations = 5
with ift.random.Context(seed):
samples = ift.optimize_kl(likelihood_energy,
n_iterations,
n_samples,
minimizer,
ic_sampling,
None, # for GeoVI
plot_energy_history=False,
plot_minisanity_history=False)
return samples
if __name__=='__main__':
# processes
print("___________________________________Running with processes___________________________________")
nrun = 8
futures = []
with cf.ProcessPoolExecutor(max_workers=nrun) as executor:
for i in range(nrun):
future = executor.submit(function_calling_nifty, i)
futures.append(future)
for f in cf.as_completed(futures):
samples = f.result()
# threads
print("___________________________________Running with threads___________________________________")
futures = []
with cf.ThreadPoolExecutor(max_workers=nrun) as executor:
for i in range(nrun):
future = executor.submit(function_calling_nifty, i)
futures.append(future)
for f in cf.as_completed(futures):
samples = f.result()
```
The first run with processes is successful but the second one falls over with
```bash
Traceback (most recent call last):
File "/home/landman/software/scratch/nifty_threadsafety.py", line 43, in function_calling_nifty
samples = ift.optimize_kl(likelihood_energy,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/optimize_kl.py", line 368, in optimize_kl
e = SampledKLEnergy(
^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/kl_energies.py", line 290, in SampledKLEnergy
sample_list = draw_samples(position, ham_sampling, minimizer_sampling, n_samples,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/kl_energies.py", line 142, in draw_samples
with random.Context(sseq[i]):
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/random.py", line 290, in __exit__
raise RuntimeError("inconsistent RNG usage detected")
RuntimeError: inconsistent RNG usage detected
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/landman/software/scratch/nifty_threadsafety.py", line 78, in <module>
samples = f.result()
^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/software/scratch/nifty_threadsafety.py", line 42, in function_calling_nifty
with ift.random.Context(seed):
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/random.py", line 290, in __exit__
raise RuntimeError("inconsistent RNG usage detected")
RuntimeError: inconsistent RNG usage detected
```
I tried using the Context class as suggested in ```src/random.py``` but this doesn't seem to have the desired effect. Any suggestions on how to get this working with a thread pool would be much appreciated.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/368Dictionary with .update instead of | for python 3.7 compatibility2023-10-24T14:46:50ZVincent EberleDictionary with .update instead of | for python 3.7 compatibility# Broken Interface for Python < 3.9 and Jax installed
If Jax is installed the Correlated Field of nifty tries to make an instance of the nifty.re.correlated field using |= which is a python 3.9 syntax. This brakes the interface for user...# Broken Interface for Python < 3.9 and Jax installed
If Jax is installed the Correlated Field of nifty tries to make an instance of the nifty.re.correlated field using |= which is a python 3.9 syntax. This brakes the interface for users having an older python version and jax as well.
IMHO we should go to the .update syntax instead. If we want to use this syntax, we can consider to update the minimal python version required for using NIFTy8.
@gedenhof @wmarg @pfrank
```python
fluctuations = WrappedCall(fluctuations, name=prefix + "fluctuations")
ptree = fluctuations.domain
loglogavgslope = WrappedCall(loglogavgslope, name=prefix + "loglogavgslope")
ptree |= loglogavgslope.domain
if flexibility is not None:
flexibility = WrappedCall(flexibility, name=prefix + "flexibility")
ptree |= flexibility.domain
# Register the parameters for the spectrum
_safe_assert(log_vol is not None)
_safe_assert(rel_log_mode_len.ndim == log_vol.ndim == 1)
ptree |= {prefix + "spectrum": ShapeWithDtype((2, ) + log_vol.shape)}
if asperity is not None:
asperity = WrappedCall(asperity, name=prefix + "asperity")
ptree |= asperity.domain
```
[Go to File](src/re/correlated_field.py#L133-146)Gordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/367Inverse Hartley transform for creating a GP covariance kernel based on a Four...2023-08-21T23:36:22ZJohannes BuchnerInverse Hartley transform for creating a GP covariance kernel based on a Fourier power spectrumThe demo of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/demos/re/hmc_wiener_filter.py
uses `jft.correlated_field.hartley` to create a covariance kernel from a powerlaw Fourier power spectrum.
My understanding is that an inverse...The demo of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/demos/re/hmc_wiener_filter.py
uses `jft.correlated_field.hartley` to create a covariance kernel from a powerlaw Fourier power spectrum.
My understanding is that an inverse Hartley transform would be needed to make the connection, since the
Fourier power spectrum is obtained by a Fourier transform.
However, in https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/re/correlated_field.py#L23
the "hartley" function is implemented with a (forward) FFT and summing real and imaginary components.
Why is that equivalent to an inverse Hartley transform?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/365Update PyPi minor release2024-02-18T14:50:58ZMatteo GuardianiUpdate PyPi minor releaseThe last minor NIFTy8 release on PyPi (https://pypi.org/project/nifty8/#history) is 8.4 (from 29.9.22).
Since many commits have been merged in between (including many bugfixs), it would be good to release a new minor stable version to ke...The last minor NIFTy8 release on PyPi (https://pypi.org/project/nifty8/#history) is 8.4 (from 29.9.22).
Since many commits have been merged in between (including many bugfixs), it would be good to release a new minor stable version to keep the PiPy version more or less up to date.
@pfrank @gedenhof @mtrhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/363Wrong result of FFT.jax_expr2023-04-19T08:07:45ZJakob RothWrong result of FFT.jax_exprThe inverse application of the `FFT.jax_expr` seems to give wrong results. Here is a snippet to reproduce the error:
```
import numpy as np
import nifty8 as ift
sp = ift.RGSpace((100, 100), distances=0.1)
FFT = ift.FFTOperator(sp)
in...The inverse application of the `FFT.jax_expr` seems to give wrong results. Here is a snippet to reproduce the error:
```
import numpy as np
import nifty8 as ift
sp = ift.RGSpace((100, 100), distances=0.1)
FFT = ift.FFTOperator(sp)
inp = ift.from_random(FFT.domain)
res_nifty = FFT(inp)
res_jax = FFT.jax_expr(inp.val)
assert(np.allclose(res_nifty.val, res_jax)) # works
res_nifty_inv = FFT.inverse(res_nifty)
res_jax_inv = FFT.jax_expr(inp.val, inverse=True)
assert(np.allclose(res_nifty_inv.val, res_jax_inv)) # fails
res_nifty_inv = FFT.inverse(res_nifty)
res_jax_inv = FFT.inverse.jax_expr(inp.val, inverse=True)
assert(np.allclose(res_nifty_inv.val, res_jax_inv)) # fails
assert(np.allclose(res_nifty_inv.val, inp.val)) # works
assert(np.allclose(res_jax, inp.val)) # fails
```https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/361Failing docs pipeline2023-11-09T12:51:08ZPhilipp FrankFailing docs pipelineRecently the docs stage of the pipeline fails due to some ownership issues regarding the docs folder (see https://gitlab.mpcdf.mpg.de/ift/nifty/-/jobs/2023129#L497).
I am not entirely sure what is going on here, I suspect it may be somet...Recently the docs stage of the pipeline fails due to some ownership issues regarding the docs folder (see https://gitlab.mpcdf.mpg.de/ift/nifty/-/jobs/2023129#L497).
I am not entirely sure what is going on here, I suspect it may be something related to using a cached docker image.
@veberle, @gedenhof: Do you have any thoughts on this?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/358Mean / Variance confusion in save_fits function of optimize_KL2023-01-13T09:35:36ZVincent EberleMean / Variance confusion in save_fits function of optimize_KLWhile finishing some plots I was surprised by a very high standard deviation and wanted to check if this could be true.
Then I found this:
```python
if mean or std:
m, s = self.sample_stat(op)
if mean:
se...While finishing some plots I was surprised by a very high standard deviation and wanted to check if this could be true.
Then I found this:
```python
if mean or std:
m, s = self.sample_stat(op)
if mean:
self._save_fits_2d(m, file_name_base + "_mean.fits", overwrite)
if std:
self._save_fits_2d(s, file_name_base + "_std.fits", overwrite)
```
here: https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/minimization/sample_list.py#L224
So in the 'if std:' we save something called standard deviation.
But sample_stat returns the mean and the variance:
```python
def sample_stat(self, op=None):
"""Compute mean and variance of samples after applying `op`.
Parameters
----------
op : callable or None
Callable that is applied to each item in the :class:`SampleListBase`
before it is used to compute mean and variance.
Returns
-------
tuple
A tuple with two items: the mean and the variance.
"""
from ..probing import StatCalculator
if self.n_samples == 1:
res = self.average(op)
return res, 0*res
sc = StatCalculator()
for ss in self.iterator(op):
sc.add(ss)
return sc.mean, sc.var
```
here: https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/minimization/sample_list.py#L348
So I think we should take the square root here.... @gedenhof @pfrank
I address also @parras, since he worked a lot on the whole optmize-kl and sample-list part of nifty.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/357Un-deprecate CMAP registration2023-01-02T12:50:38ZGordian EdenhoferUn-deprecate CMAP registration```
test/test_plot.py::test_plots
/builds/ift/nifty/nifty8/plot.py:266: PendingDeprecationWarning: The register_cmap function will be deprecated in a future version. Use ``matplotlib.colormaps.register(name)`` instead.
```
See https:/...```
test/test_plot.py::test_plots
/builds/ift/nifty/nifty8/plot.py:266: PendingDeprecationWarning: The register_cmap function will be deprecated in a future version. Use ``matplotlib.colormaps.register(name)`` instead.
```
See https://gitlab.mpcdf.mpg.de/ift/nifty/-/jobs/1948616#L528