During ion sputtering of GaSb(100) surfaces a transient behavior from initial smoothing to roughening accompanied by self-organized pattern formation has been observed using in situ x-ray reflectivity and grazing incidence small angle scattering. The induced patterns show hexagonally ordered nanodot arrays with a spatial periodicity of 30 nm. The correlation length of the pattern increases with ion fluence. In the framework of the Bradley-Harper model [R. M. Bradley and J. M. E. Harper, J. Vac. Sci. Technol. A 6, 2390 (1988)], where the dot pattern formation results from an interplay of surface roughening due to sputtering and surface smoothing due to diffusion, the initial smoothing behavior is explained by the same surface diffusion processes as the pattern formation.
