Advanced mobile communication systems are characterized by a dense deployment of different types of wireless access points. Since these systems are primarily limited by interference, multiple-input multiple-output (MIMO) techniques as well as coordinated transmission and detection schemes are necessary to mitigate this limitation. Thus, mobile communication systems become more complex which requires that also the mathematical tools for their theoretical analysis must evolve. These must be able to take the most important system characteristics into account, such as fading, path loss, and interference. The aim of this thesis is to develop such tools based on large random matrix theory and to demonstrate their usefulness with the help of several practical applications, such as the performance analysis of network MIMO and large-scale MIMO systems, the design of low-complexity polynomial expansion detectors, and the study of random beamforming techniques as well as multi-hop relay and double-scattering channels. The methods developed in this work provide deterministic approximations of the system performance which become arbitrarily tight in the large system regime with an unlimited number of transmitting and receiving devices. This leads in many cases to simple and close approximations of the finite-size system performance and allows one to draw relevant conclusions about the most significant parameters. One can think of these methods as a way to provide a deterministic abstraction of the physical layer which substantially reduces the system complexity. Due to this complexity reduction, it is possible to carry out a system optimization which would be otherwise intractable.
