Analysis of the finite element heterogeneous multiscale method for nonmonotone elliptic homogenization problems