Automated recognition of atypical nuclei in breast cancer cytology specimens by iterative image transformations