ift issueshttps://gitlab.mpcdf.mpg.de/groups/ift/-/issues2023-11-17T18:19:03Zhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/384update pyproject.toml2023-11-17T18:19:03ZVincent Eberleupdate pyproject.tomlVincent EberleVincent Eberlehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/383Embrace SPEC 0 Minimum Supported Version https://scientific-python.org/specs/...2024-01-12T17:22:59ZGordian EdenhoferEmbrace SPEC 0 Minimum Supported Version https://scientific-python.org/specs/spec-0000/Akin to Numpy and Scipy, I propose to only support recent versions of python, numpy, and scipy (released within the last 3 years) in development branches. See https://scientific-python.org/specs/spec-0000/ for details on the joint effort...Akin to Numpy and Scipy, I propose to only support recent versions of python, numpy, and scipy (released within the last 3 years) in development branches. See https://scientific-python.org/specs/spec-0000/ for details on the joint effort of the scientific community to standardize this. We would continue to support older versions of NIFTy which work with older versions of python, numpy, and scipy but the development version need only to work with SPEC 0 compliant versions. Specifically, this would drop python3.8 support. @allhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/382RFC Make all convergence criteria per DOF2023-11-16T16:06:28ZGordian EdenhoferRFC Make all convergence criteria per DOFInstead of doing tricks like `absdelta = 1e-4 * jnp.prod(jnp.array(dims))`. Make all convergence criteria to be per degree of freedom in the parameter space. WDYT @pfrank @jroth ?Instead of doing tricks like `absdelta = 1e-4 * jnp.prod(jnp.array(dims))`. Make all convergence criteria to be per degree of freedom in the parameter space. WDYT @pfrank @jroth ?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/381Implement LogNormal CF2024-02-06T11:48:29ZGordian EdenhoferImplement LogNormal CFImplement a thin wrapper around an exponentiated CFM which converts the desired statistics for exp(CF) to parameters for the "normal" CFM @pfrank @jroth .Implement a thin wrapper around an exponentiated CFM which converts the desired statistics for exp(CF) to parameters for the "normal" CFM @pfrank @jroth .https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/380RFC: Make offset a tuple of mean and std. in CF2023-11-17T14:14:17ZGordian EdenhoferRFC: Make offset a tuple of mean and std. in CFWDYT about making the `offset` parameter of the CFM a tuple of a mean and a standard deviation akin to all other parameters of the CFM? @pfrank @jroth @veberle @matteani
I imagine the implementation would look something like `A[0] = zm...WDYT about making the `offset` parameter of the CFM a tuple of a mean and a standard deviation akin to all other parameters of the CFM? @pfrank @jroth @veberle @matteani
I imagine the implementation would look something like `A[0] = zm_std; cf += zm_mean` with `A` the amplitude/power-spectrum and `cf` the correlated field realization, i.e. we would learn the zero-mode via the zeroth excitation. I am well aware that this is already feasible right now by specifying a custom zero-mode operator. My proposal is to make the zero-mode a scalar by default without custom code.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/379Sunset jax_expr in numpy NIFTy2023-11-29T19:43:48ZGordian EdenhoferSunset jax_expr in numpy NIFTyI think it is fair to say that `Operator.jax_expr` never got that much traction. Furthermore, it is not the kind of model building that we want to encourage. @all WDYT about removing `jax_expr`s from NIFTy operators again? It was a nice ...I think it is fair to say that `Operator.jax_expr` never got that much traction. Furthermore, it is not the kind of model building that we want to encourage. @all WDYT about removing `jax_expr`s from NIFTy operators again? It was a nice idea when I started developing NIFTy.re but it turned out to be an ineffective aproach.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/378Matern-kernel fluctuations in NIFTy.re2023-11-29T20:14:47ZMatteo GuardianiMatern-kernel fluctuations in NIFTy.reImplement `add_fluctuations_matern` in `correlated_field.py`.Implement `add_fluctuations_matern` in `correlated_field.py`.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/377? RFC: Incorporate the data into likelihoods in such a way that jitting does ...2023-12-07T15:26:19ZGordian Edenhofer? RFC: Incorporate the data into likelihoods in such a way that jitting does not inline them (e.g. insert them patially e.g. via `jax.tree_util.Partial` and special-case partials in likelihood)https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/376? RFC: Pull out the forward model of the likelihood into its own dedicated at...2023-11-09T12:48:11ZGordian Edenhofer? RFC: Pull out the forward model of the likelihood into its own dedicated attribute of the likelihoodhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/375Parametrize likelihoods s.t. all variables that could be filled in are filled...2023-11-09T12:47:36ZGordian EdenhoferParametrize likelihoods s.t. all variables that could be filled in are filled in by default, i.e. set the mean of the Gaussian likelihood to zero etc.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/374Restructure NIFTy.re.likelihoods2024-01-21T17:49:35ZGordian EdenhoferRestructure NIFTy.re.likelihoodshttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/373Implement a JIT-compileable minimizer2023-11-09T12:41:47ZGordian EdenhoferImplement a JIT-compileable minimizerSee !875.See !875.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/372Implement constants for minimization (and optimize_kl)2023-11-09T12:41:03ZGordian EdenhoferImplement constants for minimization (and optimize_kl)Implement constants for NIFTy.re, i.e. parts of the pytree that are potentially sampled but *NOT* optimized.Implement constants for NIFTy.re, i.e. parts of the pytree that are potentially sampled but *NOT* optimized.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/371Implement HEALPix CF in NIFTy.re2023-12-19T11:09:31ZGordian EdenhoferImplement HEALPix CF in NIFTy.reImplement a power distributor for HEALPix spheres akin to `get_fourier_mode_distributor`. While at it, we should probably refactor the `domain` variable in `nifty.re.correlated_field` into a `namedtuple`.
* [ ] Implement CPU-only versio...Implement a power distributor for HEALPix spheres akin to `get_fourier_mode_distributor`. While at it, we should probably refactor the `domain` variable in `nifty.re.correlated_field` into a `namedtuple`.
* [ ] Implement CPU-only version by wrapping ducc
* [ ] ? Done / Bind external GPU versionhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/370Implement Evidence Calculation2024-03-25T09:30:45ZGordian EdenhoferImplement Evidence CalculationImplement the evidence calculation in NIFTy.re akin to NIFTy using the existing tools in `NIFTy.re.num.lanczos`.
* [ ] Enable `NIFTy.re.num.lanczos` to work with pytrees
* [ ] ...Implement the evidence calculation in NIFTy.re akin to NIFTy using the existing tools in `NIFTy.re.num.lanczos`.
* [ ] Enable `NIFTy.re.num.lanczos` to work with pytrees
* [ ] ...Matteo GuardianiMatteo Guardiani2023-12-31https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/369Thread safety in optimize_kl2023-11-07T11:27:57ZLandman BesterThread safety in optimize_klThe following code attempts to run a number of optimize_kl instances in parallel, first with a process pool and then with a thread pool.
```python
import numpy as np
import nifty8 as ift
import concurrent.futures as cf
def function_ca...The following code attempts to run a number of optimize_kl instances in parallel, first with a process pool and then with a thread pool.
```python
import numpy as np
import nifty8 as ift
import concurrent.futures as cf
def function_calling_nifty(seed):
# set up correlated field model
npix = 512
pospace = ift.RGSpace((npix,))
spfreq = ift.RGSpace((npix,))
cfmaker = ift.CorrelatedFieldMaker('amplitude')
cfmaker.add_fluctuations(spfreq, (0.1, 1e-2), None, None, (-3, 1),
'f')
cfmaker.set_amplitude_total_offset(0., (1e-2, 1e-6))
cf = cfmaker.finalize()
normalized_amp = cfmaker.get_normalized_amplitudes()
pspec = normalized_amp[0]**2
# signal + fake data
signal = ift.exp(cf.real)
mask = np.random.binomial(1, 0.5, size=npix).astype(bool)
tmp = ift.makeField(signal.target, ~mask)
R = ift.MaskOperator(tmp)
signal_response = R(signal)
mock_position = ift.from_random(signal_response.domain, 'normal')
dspace = R.target
noise = .001
N = ift.ScalingOperator(dspace, noise, float)
data = signal_response(mock_position) + N.draw_sample()
# Minimization parameters
ic_sampling = ift.AbsDeltaEnergyController(deltaE=0.01, iteration_limit=50)
ic_newton = ift.AbsDeltaEnergyController(deltaE=0.01, iteration_limit=15)
minimizer = ift.NewtonCG(ic_newton, enable_logging=False)
# Set up likelihood energy and information Hamiltonian
likelihood_energy = ift.GaussianEnergy(data, inverse_covariance=N.inverse) @ signal_response
n_samples = 3
n_iterations = 5
with ift.random.Context(seed):
samples = ift.optimize_kl(likelihood_energy,
n_iterations,
n_samples,
minimizer,
ic_sampling,
None, # for GeoVI
plot_energy_history=False,
plot_minisanity_history=False)
return samples
if __name__=='__main__':
# processes
print("___________________________________Running with processes___________________________________")
nrun = 8
futures = []
with cf.ProcessPoolExecutor(max_workers=nrun) as executor:
for i in range(nrun):
future = executor.submit(function_calling_nifty, i)
futures.append(future)
for f in cf.as_completed(futures):
samples = f.result()
# threads
print("___________________________________Running with threads___________________________________")
futures = []
with cf.ThreadPoolExecutor(max_workers=nrun) as executor:
for i in range(nrun):
future = executor.submit(function_calling_nifty, i)
futures.append(future)
for f in cf.as_completed(futures):
samples = f.result()
```
The first run with processes is successful but the second one falls over with
```bash
Traceback (most recent call last):
File "/home/landman/software/scratch/nifty_threadsafety.py", line 43, in function_calling_nifty
samples = ift.optimize_kl(likelihood_energy,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/optimize_kl.py", line 368, in optimize_kl
e = SampledKLEnergy(
^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/kl_energies.py", line 290, in SampledKLEnergy
sample_list = draw_samples(position, ham_sampling, minimizer_sampling, n_samples,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/minimization/kl_energies.py", line 142, in draw_samples
with random.Context(sseq[i]):
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/random.py", line 290, in __exit__
raise RuntimeError("inconsistent RNG usage detected")
RuntimeError: inconsistent RNG usage detected
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/landman/software/scratch/nifty_threadsafety.py", line 78, in <module>
samples = f.result()
^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/landman/software/scratch/nifty_threadsafety.py", line 42, in function_calling_nifty
with ift.random.Context(seed):
File "/home/landman/venvs/qcal/lib/python3.11/site-packages/nifty8/random.py", line 290, in __exit__
raise RuntimeError("inconsistent RNG usage detected")
RuntimeError: inconsistent RNG usage detected
```
I tried using the Context class as suggested in ```src/random.py``` but this doesn't seem to have the desired effect. Any suggestions on how to get this working with a thread pool would be much appreciated.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/368Dictionary with .update instead of | for python 3.7 compatibility2023-10-24T14:46:50ZVincent EberleDictionary with .update instead of | for python 3.7 compatibility# Broken Interface for Python < 3.9 and Jax installed
If Jax is installed the Correlated Field of nifty tries to make an instance of the nifty.re.correlated field using |= which is a python 3.9 syntax. This brakes the interface for user...# Broken Interface for Python < 3.9 and Jax installed
If Jax is installed the Correlated Field of nifty tries to make an instance of the nifty.re.correlated field using |= which is a python 3.9 syntax. This brakes the interface for users having an older python version and jax as well.
IMHO we should go to the .update syntax instead. If we want to use this syntax, we can consider to update the minimal python version required for using NIFTy8.
@gedenhof @wmarg @pfrank
```python
fluctuations = WrappedCall(fluctuations, name=prefix + "fluctuations")
ptree = fluctuations.domain
loglogavgslope = WrappedCall(loglogavgslope, name=prefix + "loglogavgslope")
ptree |= loglogavgslope.domain
if flexibility is not None:
flexibility = WrappedCall(flexibility, name=prefix + "flexibility")
ptree |= flexibility.domain
# Register the parameters for the spectrum
_safe_assert(log_vol is not None)
_safe_assert(rel_log_mode_len.ndim == log_vol.ndim == 1)
ptree |= {prefix + "spectrum": ShapeWithDtype((2, ) + log_vol.shape)}
if asperity is not None:
asperity = WrappedCall(asperity, name=prefix + "asperity")
ptree |= asperity.domain
```
[Go to File](src/re/correlated_field.py#L133-146)Gordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/367Inverse Hartley transform for creating a GP covariance kernel based on a Four...2023-08-21T23:36:22ZJohannes BuchnerInverse Hartley transform for creating a GP covariance kernel based on a Fourier power spectrumThe demo of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/demos/re/hmc_wiener_filter.py
uses `jft.correlated_field.hartley` to create a covariance kernel from a powerlaw Fourier power spectrum.
My understanding is that an inverse...The demo of https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/demos/re/hmc_wiener_filter.py
uses `jft.correlated_field.hartley` to create a covariance kernel from a powerlaw Fourier power spectrum.
My understanding is that an inverse Hartley transform would be needed to make the connection, since the
Fourier power spectrum is obtained by a Fourier transform.
However, in https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/re/correlated_field.py#L23
the "hartley" function is implemented with a (forward) FFT and summing real and imaginary components.
Why is that equivalent to an inverse Hartley transform?https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/366Potential waste of MPI communicators2023-06-19T11:33:54ZMartin ReineckePotential waste of MPI communicators@kjako observed a "too many MPI communicators" error in a calculation. I think this is caused by the constructor of `SampleListBase`, which unconditionally creates a new communicator and stores it in `self._active_comm`. When either too ...@kjako observed a "too many MPI communicators" error in a calculation. I think this is caused by the constructor of `SampleListBase`, which unconditionally creates a new communicator and stores it in `self._active_comm`. When either too many sampe lists are generated, or if the garbage collector does not recycle them quickly enough, MPI will run out of comunicators (typical limits are 2048 or 4096).
I don't know a really good solution for this, but we may need a global communicator cache so that we don't recreate the "same" communicator over and over.https://gitlab.mpcdf.mpg.de/ift/nifty/-/issues/365Update PyPi minor release2024-02-18T14:50:58ZMatteo GuardianiUpdate PyPi minor releaseThe last minor NIFTy8 release on PyPi (https://pypi.org/project/nifty8/#history) is 8.4 (from 29.9.22).
Since many commits have been merged in between (including many bugfixs), it would be good to release a new minor stable version to ke...The last minor NIFTy8 release on PyPi (https://pypi.org/project/nifty8/#history) is 8.4 (from 29.9.22).
Since many commits have been merged in between (including many bugfixs), it would be good to release a new minor stable version to keep the PiPy version more or less up to date.
@pfrank @gedenhof @mtr