Applying multisensor in-car situations to detect violence

Abstract

Violence recognition is challenging because it can be presented in very different forms. For example, it can be present in an image by a person hitting another person or present in audio by a person being rude to another. Thus, audio and video are essential features to be analysed. In the audio approach, speech processing, music, and ambient sound are some of the main points of this problem since finding similarities and differences between these domains is necessary. Human activity can be classified into four different categories in the video approach, depending on the complexity and the number of body parts involved in the action. Examples of Human activity categories are considered: gestures, actions, interactions and activities. Recognizing human actions in the video becomes a challenge with this varied set of human activities. Furthermore, in the last years, the growth of deep learning techniques applied to this area has been enormous, and the reason is that their results surpass traditional signal processing on a large scale. This article is based on audio and video signals inside a vehicle to detect violence. Furthermore, the architecture used was ResNet model with Mel-spectrogram methodology for audio signals. The proposed method for video signal representation was RGB, which applied four different models: C2D, I3D, X3D, and Flow-Gated. Finally, multimodal fusion was applied at the end of the process.