A technology that converts objects in video into real-time CG?

Transformer MR is an AI system that detects and removes objects such as cars and people from images shot with a camera and replaces them with CG models in real time.

This technology was jointly announced by the University of Duisburg-Essen, the Technical University of Zurich, and the Porsche AI team, an automobile manufacturer, and the application of augmented reality technology is expected because visible objects can be edited on the spot.

Transformer MR is a technology that removes objects in real time from the video captured by the camera and replaces them with a CG model prepared in advance. The two processes of recognizing an object and chasing the movement of the object are performed simultaneously in real time. It is a structure that displays the CG model linked to the movement of the object by recognizing the object and preserving the cut place according to the background.

The processed image is displayed on the user client such as a PC or smartphone, but the main processing of Transformer MR is done on the backend server. In order to transmit data to the cloud, it is said that more than 4G network is required to run Transformer MR.

Alternate CG models are loaded from utt. For example, take a picture of a car running on an iPad. A car is running in front of you, but on the iPad, the car is replaced with a SF-like vehicle. A woman walking in front and a car running from the inside are transformed into robots and sci-fi vehicles, respectively. It is also possible to replace humans with bears. Because it is processed in real time, it is possible to change the CG model on the spot.

Porsche AI researcher Mohamed Kali points out that pose detection is possible as one of the important points of Transformer MR. Being able to perform pose detection means, for example, that when a person is found, the joints of the body can be identified and the CG model movements can be applied to the target model. Using this system, it is possible to conduct live broadcasts by replacing the actors’ movements with virtual character CG models outdoors and in places without special facilities.

However, at present, there are technical limitations because it requires a lot of calculations to run Transformer MR at a large resolution. The demo is a small 512×512 pixel video, so you can ignore the traffic. The video frame rate is about 15 fps, the delay is 50 to 100 milliseconds, and the alternative CG model is not the best quality either. It is still in the development stage.

The reason Porsche is involved in the development of this technology is to improve the passenger and driver experience. In the future, it is explained that Transformer MR can be applied to entertain people caught in traffic jams. Porsche is also developing a system called SoundRide that detects changes in the surrounding landscape and plays appropriate music while driving. Related information can be found here.