The "Friends" footage you haven't seen, but the AI has made up for it｜ECCV 2022

Eh? This is the "Old Friends" out of undisclosed footage? Or the "bullet time" effect?

The moment the characters are framed, the camera moves silky smoothly and a multi-angle kitchen appears in front of you, as if I was standing on the scene. You know, in the main film there are actually only these two images.

That's right, it's AI doing its "magic" again. After watching "Friends", AI can directly recreate the 3D scene as if it were a real scene, and fill in the pose of the characters at different angles between two switching scenes. It can reconstruct all the angles that have never been filmed.

It can also turn a close up shot into a large telephoto.

At first glance, it's really hard to tell that the generated effect is actually a complete fabrication. "Save your TV series from the future?" This is the new method of reconstructing 3D characters and environments proposed by UC Berkeley researchers.

The netizens had a brainstorm.

Maybe in less than 10 years, you'll be able to put your own VR image into your favorite show.

This method has now been accepted by ECCV 2022.

Dedicated to TV show scene reconstruction

The researchers said that the proposed new method is specifically for TV drama scenes. In addition to "Friends", they have also 3D reconstructed scenes from 7 TV series, including "The Big Bang Theory".

It is still difficult to reconstruct 3D scenes using individual videos, but TV shows often have multiple shots of the same scene, which provides a rich source of image data for AI learning.

The method in this paper can run automatically over the whole season, calculate the camera position information, static 3D scene structure and character body information of each shot, and then integrate them into a 3D scene.

Specifically, the method is divided into two main aspects of processing scene information character information. On the scene, based on different frames, the method estimates the camera position at the time of shooting by SfM (Structure-from-Motion). This method means that in the case of only a single camera, the 3D scene information is determined by analyzing the scene captured when the camera moves.

Then, by analyzing the position relationship between the camera and the character, we can determine the area where the character is located, and then integrate and analyze the images from two different angles to perform triangulation, so as to determine the real position of the character.