removing duplicates did not remove every duplicate now it does
Also standarized the function somewhat