In systems through which flows of energy or matter propagate, it is possible to observe self-organization phenomena. The system can leave its thermodynamical equilibrium state. Its components self-organize themselves in « dissipative structures », also called « patterns ». In optics, we observe such patterns in the transverse dimensions of laser beams during their propagation in certain nonlinear materials.This thesis aims to study the patterns observed in a photorefractive single feedback system. The forward beam and the beam reflected by the mirror interfere in the photorefractive crystal and modify its electro-optical properties. This modification influences in return the propagation of the beams. If the incident beam is sufficiently powerful, the system reaches the « modulation instability » threshold : the observation of the backward beam reveals that the intensity has self-organized in patterns.Particularly, we deal in depth with two axes of research. Firstly, we study the influence of an orbital angular momentum of the input beam (therefore called a « vortex » beam) on the pattern formation process. This property influences the self-organization phenomenon and the dynamics of the transverse structures. Moreover the results provided by a numerical model of the wave mixing process are in a good accordance with the experimental observations. Secondly, we study the highly nonlinear regime obtained with a classical gaussian pump but very powerful. We show by a statistical analysis that the turbulent state far from the instability threshold contains some extreme events, also called « rogue waves ».
