Human behaviour is inherently complex, encompassing a wide range of possibilities. These actions can typically be categorized as either normal or abnormal, with the latter spanning a broad spectrum. In this study, abnormal behaviour is specifically defined as instances of human aggression or car accidents on roadways. Such behaviours pose significant risks to surrounding traffic participants, including vehicles and pedestrians, highlighting the importance of effective monitoring. With the proliferation of various types of cameras, these technologies can be leveraged for behaviour classification and surveillance. This work introduces an optimized model featuring a novel integrated wavelet-based channel augmentation unit, designed to classify human behaviour in diverse scenarios. The model includes 5.3 million trainable parameters and achieves an average inference time of 0.09 seconds. It has been trained and evaluated on four public datasets—Real-Life Violence Situations (RLVS), Highway Incident Detection (HWID), Movie Fights, and Hockey Fights—achieving accuracies ranging from 92% to 99.5%. Comprehensive analyses and comparisons with state-of-the-art models demonstrate its superior performance, offering an average accuracy improvement of 4.97% while reducing the number of parameters by approximately 139.1 million compared to other approaches.