DeepFake How do you break a face swap scam? Make him turn sideways

DeepFake is being used in a telecoms scam, how do you break it?

Just ask him to turn his head and look at the side of his face.

Why does this happen? Here's an article on the Extranet that explains why facial forgery is much less effective when the face is turned sideways.

Reasons for side face distortion

Lateral limitations

With DeepFake face swapping, the realism drops dramatically when the face is at a side angle. This is because most 2D-based face alignment algorithms only capture 50%-60% of the number of feature points in the side view as in the main view.

Take the Multi-view Hourglass face alignment model from 'Joint Multi-view Face Alignment in the Wild' as an example.

The model is trained by identifying feature points from the face and using this as learning data.

As can be seen in the figure, 68 feature points were identified for frontal alignment, while only 39 feature points were identified for side alignment.

How to solve the DeepFake face swap scam? Make him turn sideways

The side profile view hides 50% of the feature points, which not only hinders recognition, but also interferes with the accuracy of training and subsequent face composition.

According to DeepFake expert Dr Siwei Lyu.

Side faces are really a big problem for current DeepFake technology. Facial alignment networks are very good for fronts, but not so good for sides.

There is a fundamental limitation to these algorithms: if you only cover part of your face, the alignment mechanism works well and is very powerful in this case, but when you turn around, more than half of the feature points are lost.

Face swapping has to be trained extensively to achieve a more realistic result, which means that there needs to be enough training data.

On the Internet, a large amount of data was trained to replace Jerry Seinfeld's face with that of a scene from "Lowlife" (1994). The resulting side face image is also difficult to see through.

But to achieve such realistic results, a large amount of data was used to train the actor, in this case the TV show "Seinfeld", which provided 66 hours of usable footage for the training.

In contrast, there is very little footage of ordinary people, apart from film actors, and very few people record a full 90° side view when they are photographed.

As a result, DeepFake's faked faces can easily be faked in profile.