In this work, we have carried out an extensive TEM investigation of misfit dislocations and strain relaxation in Sb-based III-V epitaxial layers on the GaAs and GaP substrates. On GaAs, we have investigated the influence of AlSb interlayer thickness and substrate surface treatment on the strain relaxation and threading dislocation density inside GaSb layers. Similarly, we studied the growth parameters, such as substrate surface treatment, growth rate, and growth temperature on the strain relaxation of 10 MLs GaSb on GaP. With the optimized GaSb buffer layers (600 nm), high mobility AlSb/InAs hetero-structures with room temperature mobility of 30000 cm2V-1s-1 (25500 cm2V-1s-1) on GaAs (GaP) substrates have been achieved. A growth mode dependence of the misfit dislocation has been observed: a 2D growth of GaSb promotes the generation of Lomer dislocations; in contrast 60o dislocations and closely spaced 60o pairs are predominantly generated in 3D growth mode. Consequently, a 60° dislocation glide model in combination with surface effects is able to account for the formation of Lomer, 60o, and 60o dislocation pairs at these hetero-interfaces. The core structures of the misfit dislocations and their stability have been investigated by atomic resolution HAADF and molecular dynamic simulation. The dislocation density tensor analysis was next used to quantify the burgers vector of the misfit dislocations. This precise measurement revealed the misfit dislocation formation mechanism in agreement with our proposed model.
