We study the deformations of twisted harmonic maps f with respect to a representation. After constructing a continuous "universal" twisted harmonic map, we give a construction of every first order deformation of f in terms of Hodge theory; we apply this result to the moduli space of reductive representations of a Kähler group, to show that the critical points of the energy functional E coincide with the monodromy representations of polarized complex variations of Hodge structure. We then proceed to second order deformations, where obstructions arise; we investigate the existence of such deformations, and give a method for constructing them, as well. Applying this to the energy functional as above, we prove (for every finitely presented group) that the energy functional is strictly pluri sub-harmonic on the moduli space of representations; assuming furthermore that the group is Kähler, we study the eigenvalues of the Hessian of E at critical points.