This thesis has two goals: the first one is to improve the control techniques in quantum mechanics, and more specifically in NMR, by using the tools of geometric optimal control. The second one is the study of the influence of Hamiltonian singularities in controlled systems. The chapter about optimal control study three classical problems of NMR : the inversion problem, the influence of the radiation damping term, and the steady state technique. Then, we apply the geometric optimal control to the problem of the population transfert in a three levels quantum system to recover the STIRAP scheme.The two next chapters study Hamiltonian singularities. We show that they allow to control the polarization in different type of optical fibers. Then, we show the existence of generalized hamiltonian monodromy in the vibrational spectrum of the HOCl molecule. Finally, we propose a method to measure dynamically the monodromy in two different nonlinear optics systems : the Bragg model and the three waves mixing model.
