This thesis deals with the theoretical and experimental study of the dynamics of a fluid inside a precessing cylinder. It is a rotating flow problem which has many applications in aeronautics (precession of a flying object with a propellant liquid) and in geophysics (influence of the Earth precession on the dynamics of the Earth liquid core). At moderate Reynolds numbers, the precessing flow is the sum of a solid body rotation and waves called Kelvin modes. These waves are resonant when their eigenfrequency equals the precessing frequency. In order to predict the amplitude of a resonant Kelvin mode, a viscous and weakly nonlinear theory has been carried out and confirmed experimentally. When the Reynolds number is increased, the base flow becomes unstable. PIV measurements have shown that this instability is due to an interaction between the Kelvin mode forced by precession and two free Kelvin modes. A linear analysis of stability based on a mechanism of triadic resonance between Kelvin modes predicts correctly the threshold and the growth rate of the instability. This instability is saturated by a geostrophic mode due to the nonlinear and viscous interaction of the free Kelvin modes with themselves. In agreement with experiments, a weakly nonlinear theory shows that the Kelvin modes amplitudes are stationary or oscillating (cycles of intermittency) depending on the Reynolds number. Finally, the fixed point of the nonlinear equations (dynamical system with four degrees of freedom) predicts fairly well the mean flow inside the cylinder, even for very high Reynolds numbers (i.e. Re > 50000).
