We present an optical-inertial tracking system which consists of two stationary cameras and a Sensor Unit with optical markers and an inertial measurement unit (IMU). This Sensor Unit is attached to the object being tracked and its position and orientation are determined by a data fusion algorithm.The tracking system is to be used for servo-controlling a handheld tool in a navigated or computer-assisted surgery system.The data fusion algorithm integrates data from the different sensors, that is optical data from the cameras and inertial data from accelerometers and gyroscopes. We present different algorithms which ensure high-bandwidth tracking with at least 200Hz with low latencies by using a direct approach and so-called invariant filters which take into account system symmetries. Through these features, the tracking system meets the requirements for being used in the desired application.The system was successfully implemented and tested with an experimental setup.