NIFTy merge requestshttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests2023-08-07T09:30:56Zhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/859Update gitlab-ci to un-deprecated global variables2023-08-07T09:30:56ZGordian EdenhoferUpdate gitlab-ci to un-deprecated global variablesGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/851Rethink NIFTy.re.Field (Field -> Vector)2023-07-04T16:36:39ZGordian EdenhoferRethink NIFTy.re.Field (Field -> Vector)Gordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/856Switch to debain:bullseye-slim for CI2023-06-16T10:13:05ZPhilipp FrankSwitch to debain:bullseye-slim for CIPhilipp FrankPhilipp Frankhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/855add hashing for MPI equality check for fields in optimize_kl2023-06-16T09:36:03ZJakob Knollmuelleradd hashing for MPI equality check for fields in optimize_klchecking MPI equality of large fields explicitly can crash and lead to segmentation faults, that's why we introduced the possibility to just compare hashes. We didn't add it to the check in the `_single_value_sample_list` in optimize_kl....checking MPI equality of large fields explicitly can crash and lead to segmentation faults, that's why we introduced the possibility to just compare hashes. We didn't add it to the check in the `_single_value_sample_list` in optimize_kl.py and I run into an issue there.https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/852Extract values form field at given indices2023-06-12T08:13:19ZJakob RothExtract values form field at given indicesExtracts values from a field at given indices and puts them into an unstructured field. Useful when some indices appear twice (e.g. several measurements at the same location).Extracts values from a field at given indices and puts them into an unstructured field. Useful when some indices appear twice (e.g. several measurements at the same location).Philipp FrankPhilipp Frankhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/848Fix potential color rendering error in multifrequency plotting2023-05-09T16:38:01ZLukas PlatzFix potential color rendering error in multifrequency plottingWhile refactoring the multifrequency plotting, I think I discoverd an error in the wavelength-color mapping.
To test the routine, I created an mf image in which each column contains contribution from only one wavelength. Below are imag...While refactoring the multifrequency plotting, I think I discoverd an error in the wavelength-color mapping.
To test the routine, I created an mf image in which each column contains contribution from only one wavelength. Below are images how the nifty plotting routine in the current `NIFTy_8` branch renders this image ("original") and how my (supposedly) fixed routine does ("new_fixed").
![original](/uploads/91f9a2b3ac0d3fbe3d17c69d9a0a8db8/original.png)
![new_fixed](/uploads/e5dbe3706f7e6b12734eebacb9f6bec5/new_fixed.png)
I expect a test image like described to be rendered into the familiar rainbow colors known of visualizations of the visibible light spectrum, but the original RGB mapping instead produces some red, lots of yellow, some turquise, and lots of violet.
The problem seems to be that the original routine first applies the rgb mapping and *then* the logarithmic "sensitivity" correction, while it should be the other way around. The source branch of this MR contains the code producing the "new_fixed" output.
@mtr @pfrank What do you think? Was the original behavior desireable, or is the new behavior better? I lean towards the latter, if our goal is to map spectral data to the visible spectrum without color distortion.https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/849OperatorAdapter.jax_expr: Fix casting in recent JAX2023-05-02T14:31:08ZGordian EdenhoferOperatorAdapter.jax_expr: Fix casting in recent JAXGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/842Introduce a more flexible sequential map2023-05-02T13:50:13ZGordian EdenhoferIntroduce a more flexible sequential mapGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/843Various small tweaks2023-05-01T18:56:19ZGordian EdenhoferVarious small tweaksGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/846Import `TransposeOperator` into NIFTy namespace2023-04-17T07:33:09ZLukas PlatzImport `TransposeOperator` into NIFTy namespaceFor some reason the openly documented `TransposeOperator` is not imported into the NIFTy namespace.
I suspect this is an oversight, therefore this merge request.For some reason the openly documented `TransposeOperator` is not imported into the NIFTy namespace.
I suspect this is an oversight, therefore this merge request.https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/845Fix pipeline2023-03-17T17:58:30ZGordian EdenhoferFix pipelineGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/840Reintroduce `plt.figure()` to protect user figures from closing2023-02-22T14:11:28ZLukas PlatzReintroduce `plt.figure()` to protect user figures from closingPhilipp FrankPhilipp Frankhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/820Fix `ift.Field.vdot` breaking with np.uint32 and other integer types2023-02-21T16:03:20ZLukas PlatzFix `ift.Field.vdot` breaking with np.uint32 and other integer typesHi Martin,
I just stumbled over `ducc0.misc.vdot` not multiplying doubles and integers, which is required in the `PoissonianEnergy`. It seems Philipp Arras did so, too, a while ago, but unfortunately his mitigation was incomplete, as it...Hi Martin,
I just stumbled over `ducc0.misc.vdot` not multiplying doubles and integers, which is required in the `PoissonianEnergy`. It seems Philipp Arras did so, too, a while ago, but unfortunately his mitigation was incomplete, as it did only catch `np.int64`s.
Is there a `double*integer` `vdot` in `ducc0` that we could call instead of the conversion?
While at it: I noticed [`ift.PoissonianEnergy`](https://ift.pages.mpcdf.de/nifty/_modules/nifty8/operators/energy_operators.html#PoissonianEnergy) enforces integer-typed data fields (sensible for Poisson counts), but then passes them directly to `vdot`, triggering an `int->double` conversion in each application. Should we cast the data field to `double` in the initialization of `PoissonianEnergy` after the integer check to avoid this?
Cheers,
LukasMartin ReineckeMartin Reineckehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/821User Experience: Fix `optimize_kl` minisanity plotting for runs with many par...2023-02-21T15:43:03ZLukas PlatzUser Experience: Fix `optimize_kl` minisanity plotting for runs with many parametersCurrently, the reduced-ChiĀ² plots produced by `optimize_kl` become unreadable when reconstructing a model with many parameter keys or when doing many MGVI/GeoVI iterations.
This patch relocates the minisanity plot legend to outside of t...Currently, the reduced-ChiĀ² plots produced by `optimize_kl` become unreadable when reconstructing a model with many parameter keys or when doing many MGVI/GeoVI iterations.
This patch relocates the minisanity plot legend to outside of the graph area and resizes the figure adaptively to make it readable in both named scenarios. Adaptive resizing is also added to the energy history plot.
@mtr: Are you the right maintainer to ask to merge this? Has somebody been assigned to take over @parras position as maintainer?
Cheers, Lukas
# Before:
![minisanity_history_last](/uploads/65f87d0a8b13f3906f317d6f322736b5/minisanity_history_last.png)
# After:
![minisanity_history_last](/uploads/6c8419dbfd52f28d31c9e5b3f56add2a/minisanity_history_last.png)https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/838don't open new fig in every iteration2023-02-13T09:36:48ZJakob Rothdon't open new fig in every iterationCurrently, opimize_kl opens a new figure, when plotting the minisanity history, in every iteration of the optimisation (See this line in the code: https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/minimization/optimize_kl.py#L613)...Currently, opimize_kl opens a new figure, when plotting the minisanity history, in every iteration of the optimisation (See this line in the code: https://gitlab.mpcdf.mpg.de/ift/nifty/-/blob/NIFTy_8/src/minimization/optimize_kl.py#L613). After the minisanity plot the figure is cleared but not closed. Thus after an `optimize_kl` run with n iterations, there are n open matplotlib figures. If one calls `plt.show()` after `optimize_kl` n empty figures will open. Here is a small demo:
```
import nifty8 as ift
import matplotlib.pyplot as plt
import numpy as np
from mpi4py import MPI
comm = MPI.COMM_WORLD
master = comm.Get_rank() == 0
############ some nifty with optimize_kl ##############
sp = ift.RGSpace(1)
op = ift.makeOp(ift.full(sp, 1)).ducktape('blub')
d = ift.full(sp, 1.)
n = ift.ScalingOperator(sp, 0.1, np.float64)
lh = ift.GaussianEnergy(d, inverse_covariance=n.inverse) @ op
n_iterations = 3
n_samples = 2
ic_sampling = ift.AbsDeltaEnergyController(name="Sampling (linear)",
deltaE=0.05, iteration_limit=100)
ic_newton = ift.AbsDeltaEnergyController(name='Newton', deltaE=0.5,
convergence_level=2, iteration_limit=35)
ic_sampling_nl = ift.AbsDeltaEnergyController(name='Sampling (nonlin)',
deltaE=0.5, iteration_limit=15,
convergence_level=2)
minimizer = ift.NewtonCG(ic_newton)
minimizer_sampling = ift.NewtonCG(ic_sampling_nl)
samples = ift.optimize_kl(lh, n_iterations, 2,
minimizer, ic_sampling, minimizer_sampling,
output_directory="folder",
comm=comm, plot_energy_history=True,
plot_minisanity_history=True)
#######################################################
x = np.linspace(1,10,100)
plt.plot(x,x**2)
if master:
plt.show()
```
Is there a reason to create a new figure every time? I guess this is a bug. I have removed the `plt.figure()`. Note: also in the energy history we don't have a `plt.figure()`.https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/839change from last to latest2023-02-02T16:41:18ZVincent Eberlechange from last to latest#359#359Martin ReineckeMartin Reineckehttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/836remove note mpi tasks in sample_list2023-01-25T12:09:28ZJakob Rothremove note mpi tasks in sample_listIn the Sample List documentation, we have a note that a sample list has to be loaded with the sample number of MPI tasks with which the Sample List was saved. Actually, this condition is not necessary. The number of MPI tasks does neithe...In the Sample List documentation, we have a note that a sample list has to be loaded with the sample number of MPI tasks with which the Sample List was saved. Actually, this condition is not necessary. The number of MPI tasks does neither affect the order in which the samples are stored nor the order in which the samples are loaded again. To convince yourself, here are two code snippets:
Save a sample list:
```
import nifty8 as ift
from mpi4py import MPI
comm = MPI.COMM_WORLD
n_tasks = comm.Get_size()
my_rank = comm.Get_rank()
master = my_rank == 0
n_samples = 5
sp = ift.RGSpace(1)
mean = ift.full(sp, 0)
# create samples with "value of sample=index in list"
# create samples mpi parallel according to nifty convention
loc_inds = ift.utilities.shareRange(n_samples, n_tasks, my_rank)
smps = [ift.full(sp, ii) for ii in range(*loc_inds)]
neg = [False for _ in range(*loc_inds)]
samples = ift.ResidualSampleList(mean, smps, neg, comm)
samples.save('smp')
if master:
print(f'saved sample list with {n_samples} samples distributed on {n_tasks} tasks')
```
Read the sample list and verify the number of samples and their order:
```
import nifty8 as ift
from mpi4py import MPI
comm = MPI.COMM_WORLD
n_tasks = comm.Get_size()
my_rank = comm.Get_rank()
master = my_rank == 0
samples = ift.ResidualSampleList.load('smp', comm)
n_samples = samples.n_samples
# check order of samples
for ii, smp in enumerate(samples.iterator()):
assert(ii == smp.val[0])
if master:
print(f'read sample list with {n_samples} samples, and distribute on {n_tasks} tasks')
print('verified order of samples')
```
Thanks for noticing this Maximilian!https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/837add missing "not" in changelog2023-01-25T11:07:14ZJakob Rothadd missing "not" in changelogAdd missing "not" in the changelog description of minisanity output.Add missing "not" in the changelog description of minisanity output.https://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/835Sampling: Do not raise on not converged samples2023-01-13T11:59:35ZGordian EdenhoferSampling: Do not raise on not converged samplesGordian EdenhoferGordian Edenhoferhttps://gitlab.mpcdf.mpg.de/ift/nifty/-/merge_requests/832Better Lanczos interface2023-01-13T11:29:43ZGordian EdenhoferBetter Lanczos interfaceWhile this path contains the changes for a working geomap implementation, it is mostly about minor changes to the interface of the Lanczos algorithm.While this path contains the changes for a working geomap implementation, it is mostly about minor changes to the interface of the Lanczos algorithm.Gordian EdenhoferGordian Edenhofer