Modulation and detection methods have been studied to improve the performance of optical long distance communication systems. Polarization division multiplexed modulation and multilevel phase modulation format have been evaluated. Coherent detection associated with digital signal processing has been selected at the receiver side to optimize system performance. This receiver design allows especially to compensate linear distortions induced by fiber optics propagation very efficiently. After long distance propagation, optical signal to noise ratio is a major limitation. The other main limitation comes from non linear interactions of light and fiber optics which put a limit on the maximum power to be injected within fiber optics.The first wavelength division multiplexed transmission over long distances (>1000km) using coherent detection has been demonstrated.Various strategies have been proposed to minimize the impact of non linear effect. The impact of polarisation multiplexing has been quantified in configuration where chromatic dispersion is either regularly optically compensated within the line or digitally compensated at the receiver end.The proposition to use a polarization multiplexed format using only 2 phase levels as well as the algorithms capable to detect them has been done. The performance gain obtained compared to the convention solution using 4 phase levels has been demonstrated.At 100Gb/s bit rate, the demonstration of the first transmission over trans-oceanic distance has also been done.These research work have been used to define and then develop Alcatel-Lucent products which are now commercialized, at 40Gb/s by using a polarization multiplexed and 2 phase levels modulation format, and at 100Gb/s by using a polarization multiplexed 4 phase levels format.
