Add logic to remove old maintainers

Otherwise this failed with a rather cryptic 404, on needed manual
intervention