Unveiling Truth Through Gestures: A Multimodal Deepfake Detection System

Abstract: 

Over the past few years, generative artificial intelligence and other deepfake technologies have been improving every day. Although these technologies have many useful applications, they are also being used maliciously to spread fake news, blackmail, scam, and cause cyberattacks. Thus, it is important to accurately detect if these videos are real or if they are the creation of artificial intelligence. Much of the recent work on these uses face images and Generative Adversarial Network (GAN) artifacts to detect deepfakes and generated videos. Instead, we propose to use a multimodal architecture that uses gesture information and the whole body in order to detect deepfakes. In addition, this model will also use EfficientNet to extract deep features from an image and assist in deepfake detection. On the Deepfake Dataset Challenge (DFDC) and the FakeAVCeleb dataset, the system has achieved an AUC of 0.9658 and 0.9714 respectively.