Coherent optical detection and digital signal processing are at the heart of the new generation optical transmission systems. These systems use spectrally efficient modulation formats and are very robust to the propagation impairments of the optical channel. Thus, they allow telecommunications carriers to increase dramatically the capacity of their transport networks while keeping their existing fiber infrastructures. This thesis develops various issues related to the design of digital functions for coherent optical receivers operating at 100 Gb/s and beyond. It first addresses the computational complexity of digital equalization of the optical channel, and evaluates the contribution of frequency-domain techniques in order to make the processing compatible with the implementation constraints at very high speed. The second point of the thesis concerns the potential difficulties with the use of polarization multiplexing. The final part deals with the joint equalization and carrier synchronization for application to highly spectrally efficient modulation formats.
