FVV Live is a real-time, low-latency, end-to-end free viewpoint video system including capture, transmission, synthesis on an edge server, and visualisation and control on a mobile terminal.

The following video shows an application where the advanced features of FVV Live are used (free viewpoint, user control, real-time, low-latency, ...) while explaining the technical developments:

For more info, contact us:  This email address is being protected from spambots. You need JavaScript enabled to view it.

SYSTEM DESCRIPTION

The following video presents the laboratory initial setup (cameras on rig, not on individual tripods) as well as the low-latency real-time performance:

CAPTURE

- Consumer-grade stereo cameras over standard USB connections, no genlock required.

- Software synchronization procedure over a shared clock source distributed using PTP (IEEE 1588-2002). 

- Real-time depth estimation and segmentation to enable layered synthesis and bandwidth savings. 

TRANSMISSION 

- Real-time encoding all (necessary) camera streams, both color and depth, RTP transmission.

- Depth information means structure, but video lossy codecs are designed for natural images and the HVS: lossless coding.

- Extended precision to deliver depth data for the synthesis: adaptation of 4:2:0 video structures to get 12 bits per pixel.

SYNTHESIS AND VISUALIZATION

- Real-time synthesis, too much data from all cameras, therefore only the cameras nearest to the virtual viewpoint are used. 

- Very complex task, and also unreliable depth data. Soft transitions mix contributions from closest reference cameras. 

- Dense off-line background model with Retinex+AKAZE+SfM+MVS. 

- Layered synthesis to integrate reliable off-line background model with on-line foreground to combat noisy on-line depth estimation in the background.  

- Cell phone controls the position of the virtual camera.  

For more info, contact us:  This email address is being protected from spambots. You need JavaScript enabled to view it. 

Tweet