"2D to 3D Image and Video Conversion using Machine Learning" 

José Luis Herrera

E.T.S. Ing. Telecomunicación, Universidad Politécnica de Madrid, March 2021, "Sobresaliente".

Ph.D. Thesis Advisors: Narciso García and Carlos Roberto del Blanco.

Despite the concept of three-dimensional (3D) image and video was introduced many years ago, there has been recently a significant increase in the number of available 3D displays and players in the last years. Nevertheless, the amount of 3D content has not increased in the same magnitude, creating a gap between 3D offer and demand. To reduce this difference, many algorithms have appeared that perform 2D-to-3D image and video conversion. While many of this techniques usually required several images from the same scene to perform the conversion, the most recent family of these techniques, which corresponds to machine learning- based algorithms, is not restricted by this limitation computing the 3D image with a single view of the scene. Machine learning-based methods require the use of databases of 2D and 3D images to learn how to perform this conversion. Since the number of available datasets has recently increased significantly, these algorithms have become very popular. However, the quality achieved by the current 2D to 3D conversion techniques is far to be fully satisfying and they need to be improved in order be used for producing good quality 3D content.

This thesis proposes two systems for 2D-to-3D conversion, one for image and another one for video, belonging to the machine learning family of methods. With respect to the image conversion system, a new approach is proposed that makes the algorithm more robust and adaptive to different types of scenarios by using a combination of feature descriptors. At the same time, the proposed clustering of the dataset makes the solution faster and more efficient to deal with large datasets. The system also learns how to automatically adapt the value of the different parameters involved in the conversion, resulting in a fully automatic solution.

With respect to video conversion, the learning based approach for images is extended to video sequences. The algorithm is divided in three main parts. In the first one, a depth estimation of the background is computed using the previous presented approach. Then, the foreground is segmented analyzing the optical flow to manage the different objects individually. Finally, the depth estimation of the background is combined with the foreground information and filtered to obtain the final depth estimation.

The developed algorithms have been tested with different publicly available datasets of 3D images and video sequences.

Tweet