Expensive sanity checks in optimize_kl.py?
The file optimize_KL.py
contains several calls to check_MPI_equality
, to ensure that certain objects are identical on all MPI tasks. While this is definitely useful, some of these calls try to compare objects that can become huge (which leads to a lot of additional memory/time consumption) and will in extreme cases exceed 2GB, causing MPI failures.
Would it be possible to either remove these cases or simplify the comparison somehow, e.g. by hashing the data and comparing the hash across MPI tasks?