System state estimation is an essential part for robot navigation and control. A combination of Inertial Navigation Systems (INS) and further exteroceptive sensors such as cameras or laser scanners is widely used. On small robotic systems with limitations in payload, power consumption and computational resources the processing of exteroceptive sensor data often introduces time delays which have to be considered in the sensor data fusion process. These time delays are especially critical in the estimation of system velocity. In this paper we present a state estimation framework fusing an INS with time delayed, relative exteroceptive sensor measurements. We evaluate its performance for a highly dynamic flight system trajectory including a flip. The evolution of velocity and position errors for varying measurement frequencies from 15Hz to 1Hz and time delays up to 1s is shown in Monte Carlo simulations. The filter algorithm with key frame based odometry permits an optimal, local drift free navigation while still being computationally tractable on small onboard computers. Finally, we present the results of the algorithm applied to a real quadrotor by flying from inside a house out through the window.