"Augmented reality over video stream acquired from UAVs for operations support" 

Susana Ruano

E.T.S. Ing. Telecomunicación, Universidad Politécnica de Madrid, November 2018, "Cum Laude".

Ph.D. thesis Directors: Carlos Cuevas Rodríguez y Guillermo Gallego Bonet .

Augmented reality (AR) has become, due to recent technology developments, a fast-growing discipline. The potential of AR supports its study not only for specific devices such as glasses or helmets, but for anything equipped with a camera. Following this idea, Airbus promoted an innovation project, Situational Awareness Virtual EnviRonment (SAVIER), to incorporate AR in their ground control stations, thus allowing the enhancement of the video stream captured from Unmanned Aerial Vehicles (UAVs). This thesis is framed in that project and explores different approaches to improve the situational awareness of the UAV operators during a mission. Initially, the thesis is focused on geo-registration, a strategy used for the localization of the UAV in GPS-denied environments. This is of interest because knowing the position of the UAV is essential to provide information about the surroundings. For this reason, we proposed two key systems for geo-registration with different reference data.

First, a multi-view stereo processing pipeline for building a dense terrain model from images of the UAV video feed. This is helpful when a reference terrain model is needed for geo-registration but it is unavailable, outdated, or it has low resolution. The proposed variational method enforces continuity not only along epipolar lines but also across them, in the full image domain. Second, the thesis proposed a joint geometric and photometric image registration method that can deal with generic types of distortion: parametric warpings (such as homographies) and non-linear photometric transformations. It is built on top of area-based registration methods to be able to operate in scenarios where feature-based geo-registration methods are not reliable. Finally, the general case was considered, in which every sensor measurement is known with enough accuracy and the thesis focused on displaying virtual elements over the video stream acquired by the UAV.

An AR tool to improve the situational awareness of UAV operators during intelligence and surveillance missions was developed. The AR system provides information about the flying path and the targets, so that the operator can reduce the time to find them even in the presence of occlusions. The usability of the proposed AR tool was proved by the adoption of NATO standards and it was fully integrated with the Airbus SAVIER demonstrator, in Getafe, Madrid. 

Tweet