Currently deployed optical transmission systems use coherent detection for data rates of 40 Gb/s and 100 Gb/s. Quadrature phase shift keying (QPSK) modulation using four phase levels in combination with polarization division multiplexing (PDM) allows transmitting four bits per symbol. The use of more complex modulation formats, such as 16 level quadrature amplitude modulation (16QAM) allows increasing the data rate. However, this method reduces dramatically the transmission reach. For example, when passing from 100 Gb/s PDM-QPSK to 200 Gb/s PDM-16QAM, the reach is reduced by a factor of five. A new and disruptive approach in order to increase the capacity is mode division multiplexing (MDM), and this approach is investigated in the frame of my thesis. I start my thesis with some generalities on optical transmission systems followed by a presentation of their historical evolution against the background of the increasing capacity demand in the worldwide telecommunication networks. Afterwards I show some ways to continue the capacity growth in optical transmission systems before focusing on MDM. I describe the new key elements, notably the few-mode fiber and the few-mode amplifier, the mode-multiplexer / -demultiplexer and the new receiver system. I finish with a presentation of some experiments using entire MDM systems, which allowed us to be among the first research teams worldwide to realize a successful MDM transmission
