Facial emotion recognition (FER) is a significant subject in computer vision and artificial intelligence because of its tremendous academic and commercial potential, such as in cognitive science, health care, virtual reality, and video conferencing in various domains. While FER could be carried out using multiple sensors, this review includes research that exclusively uses face images, since facial expressions are among the key pieces of knowledge in human interactions. We give a brief review of FER research carried out in recent years. We divided the techniques of FER mainly into two parts, i.e., based on the type of approach and based on the type of frame, which is further subdivided into two sub-parts of each classification for more detailed division. First, it explains traditional FER approaches together with a description of the representative classes of FER systems. Deep-learning FER strategies are then addressed using deep networks that allow “end-to-end” learning. This review is also directed toward an up-to-date deep learning strategy, which is a trending topic nowadays. A brief overview of publicly accessible assessment metrics is provided in the later part of this paper, as well as a comparison with the baseline results is presented, which is a norm for quantitative analysis of FER research. The whole analysis also could act as a concise field guide for beginners in the FER sector, providing general details and a common understanding of both the recent state-of-the-art research and established researchers searching for fruitful areas for further research.

Guest editors Amit Kumar Singh and Stefano Berretti introduce the Special Section on Recent Advances in Multimedia Information Security.

Skin cancer is one of the world’s most common cancers. Among different skin cancer types, melanoma is very dangerous. Its treatment is possible if detected at an early stage. Detection of skin cancer is still challenging, even for experienced dermatologists, due to the high similarity between benign and malignant lesions. Deep learning technologies help in diagnosing skin lesions and classifying them. They are used to detect lesions at an early stage. However, it still struggles with complex skin lesions because of complicated properties such as fuzzy boundaries, artifacts, low background contrast, and insufficient training datasets. Medical images require information security also. This paper proposes a method using particle swarm optimization (PSO) with the ensemble of pretrained convolutional neural network PSOCNN architecture for skin disease classification with information security. To evaluate the proposed architecture, 10,015 samples of the publicly available HAM1000 (human against machine) dataset comprise seven distinct classes. It is the case of a multiclass skin lesion classification problem. This method achieves 97.82% accuracy and a macroaverage ROC curve area of 0.99. The average F1 score obtained is 0.98. This work has been compared with different studies present in the literature. The result shows that this work has achieved comparable classification accuracy. It helps in achieving better performance for all classes of multi-class classification.

The development of artificial intelligence, especially the application and development of deep learning, brings not only great convenience to people but also some challenges. Some face forgery techniques can confuse the false with the true in deep learning. With the widespread popularity of social applications and streaming media, people pay more and more attention to portrait privacy and security. To solve the great threat to personal privacy caused by deep face forgery technology, we propose an effective face forgery method to enrich the face antispoofing datasets. We first explore the representation of identity information in latent space. Based on this foundation, we use the identity feature replacement module to edit the latent codes in the latent space and finally generate the images with photo-realistic results with the help of the powerful generation ability of StyleGAN. The face forgery method can further expand the face antispoofing dataset, provide rich data for the learning-based forgery detection methods to improve their generalization and to ensure the security in preserving portrait privacy.

The numerous technologies and procedures used to safeguard private cloud infrastructure are private cloud security. Despite multitenant public cloud environments, resources are dedicated to individual companies in a private cloud. Cloud computing uses several data security mechanisms, including authentication and identification, access control, encryption, secure deletion, integrity checking, and data masking. Cloud computing relies heavily on security and traffic management. Because data must be transferred from a local computer to a remote computer when using a cloud computing service, data protection is becoming an increasingly important security problem. Research on cloud computing-based secure dynamic bit standard (CC-SDBS) technology has increased data security. This technique’s cryptographic algorithms (CA) improve cloud computing security. As a result of the private cloud security level algorithm (PCSLA), the client’s private data can be easily accessed in the cloud. For the client’s private cloud data, PCSLA provides dynamic security measures. Cloud computing applications can use the tests’ better data security in the future. The proposed algorithm’s testing results have shown a noticeable improvement in cipher size and execution time compared with other commonly used cloud computing. Numerical results revealed that CC-SDBS improved average energy consumption by 20%, average network latency by 25%, average key generation time analysis by 4.3%, and average network use by 15%.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

AMS, SDC, SCB, and SP either did not respond directly or could not be reached.

Anomaly detection is one of the most researched topics in computer vision and machine learning. Manual detection of an oddity in a video costs significant time and money, so there is a need for an autonomous detection system that can analyze the process and detect the anomaly in the majority of captured video datasets. Through an in-depth study on the recently published works on anomaly detection, a review is prepared to highlight the various tasks performed in abnormal behavior detection. Descriptions along with the pros and cons of various machine-learning and non-machine-learning techniques are discussed in depth. Similarly, more concentration is given to the generation adversarial network (GAN), and a comprehensive description of its design for achieving a better abnormality detection rate is provided. Moreover, a comparison of various state-of-the-art approaches on the basis of their methodologies, advantages, and disadvantages is given. We further quantitatively analyze some of the recent robust approaches at the frame level on the UCSD Ped1 dataset, with the GAN-based model achieving an astonishing performance. We provide various suggestions on how to further increase the performance of GAN for abnormal behavior detection in surveillance videos.

A transform domain digital image watermarking technique is proposed using bacterial foraging optimization (BFO) for telemedicine applications. To make a trade-off between robustness, imperceptibility, and authenticity is a significant focus of our work. The field of telemedicine is one of the critical areas for an intruder to make unwanted changes. The size of medical data is exponentially increasing since the pandemic. So, the security of medical data is also a big concern. One of the most important techniques to safeguard medical data is image watermarking. The digital imaging and communications in medical images of liver ultrasound are taken as input host images. Peak signal-to-noise ratio, Kullback–Leibler distance, and root-mean-square difference are used for BFO. The proposed BacterialWmark technique shows a significant improvement in robustness and imperceptibility compared with other existing techniques. The computational complexity of the proposed work is also less. The authentication of the watermarked image is successfully executed as the features are perfectly matched with the aid of the MinEigen algorithm. So, the BacterialWmark algorithm can be used in the field of telemedicine efficiently.

Workflows can be optimized with the use of three-dimensional (3D) computer vision (CV) by monitoring processes from many angles. The potential for increased efficiency and cost savings is clear here. Input scenarios are always being monitored by security and surveillance systems to recognize and classify items, human faces, and moving things. The input reconstruction in these surveillance systems is problematic because of the mapping from three dimensions to two dimensions in the form of pixels. We present a new 3D CV paradigm-based framework for facial recognition and reconstruction. In this setup, deep neural networks are used to distinguish between mapped and unmapped pixels during a 2D to 3D or 3D to 2D transformation. To use correlation analysis to recognize human faces, it is necessary to first examine the input image for textural properties. The mapping procedure continues with the extraction of textural elements using dimensional contours. Then, continuous contours are used to do the 3D mapping and reduce the number of false positives. Instead, this framework isolates individual contours of moving objects or faces to increase the number of training iterations. Finally, missing pixels in the dimension conversion are filled to rebuild the human face utilizing discrete and continuous outlines. Hence, the suggested framework minimizes the false rate (10%), increases the false positive rate as (13.93%), and minimizes the error by (9.89%) while maximizing the recognition accuracy for (13.93%) and the precision is (13.21%).

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

HA did not agree with the retraction.

In multimedia forensics, several methods have been developed for the authentication of digital images. However, the detection and localization of removed objects from an image has always been a challenging problem. Image forgery, for the removal of objects, can be done in many ways. Among them, image inpainting performs object removal and fills the empty region with surrounding patches. The clues of inpainted region are visually imperceptible. Till date, limited work has been done for image inpainting detection. Hence, a convolutional neural network-based model for the detection of inpainted regions in an image is presented in this research. A hybrid encoder–decoder-based architecture is proposed, where a segment of DenseNet-121 architecture is adopted as an encoder. The primary goal of this architecture is to use spatial maps to explore the distinguishing features between inpainted and uninpainted regions. Inpainted image dataset created by using the exemplar-based image inpainting method is used to train and validate the proposed model. The performance of the proposed model is evaluated using various performance metrics. Experimental results show that the proposed model outperformed existing methods for a variety of inpainted images.

Self-attention has been proven to be a quite powerful yet calculation-intensive method for scene semantic segmentation. Even though many efforts have been made to explore more effective and resource-saving ways to apply self-attention, there is still space in reducing the calculation consumption. Meanwhile, since self-attention is good at fusing information, its application should be extended to multi-scale-feature-fusion, which is barely researched while the information exchange paths between features in different resolutions are mostly addition and concatenation. A special partition method decreasing the computational complexity of self-attention is investigated, and a multi-scale-feature-attention (MFA) module fusing low-resolution features containing semantic information with high-resolution features having detailed information is presented at the same time. To be specific, the proposed multi-scale-partition-attention (MPA) module and MFA module are inserted into the backbone in sequence to fuse information among all the pixels in one highly extracted feature and the pixels from features with different resolutions, respectively. Extensive experiments are carried out on semantic segmentation benchmarks including PASCAL-Context and Cityscapes to demonstrate that these two improved modules can improve the performance of the backbone in scene semantic segmentation tasks that contain multiple classes and objects in both big and small sizes.

At present, there has been much focus on information hiding because it acts as an alternative to secure communication. Secret content is embodied in an image using various image embodiment techniques, with the least significant bit (LSB) substitution being one of the preferred strategies. Asserting the originality and quality of the content embodiment is equally challenging. A scrambling mechanism using a Sudoku scheme is applied on all red, green, and blue layer blocks of the image. The image layers are devised into blocks, and over the blocks unique scrambling procedures are applied to acquire the scrambled image. The scrambled image is then exposed to the confidential stream embodiment using the traditional LSB substitution scheme by following the chess game-based queen tour traversal pattern for locating the pixels. After the embedding process, the scrambled image is fragmented into blocks, and the exact descrambling of the blocks is performed to get back the image, which visually resembles the original image called the stego image. The stego image gets split into blocks at the receiver end to apply exact scrambling on the corresponding blocks, and using the queen tour pattern, the secret stream is to be extracted. As a result of the experiments, it is found that the projected approach is unlikely to be very predictable and unique in its patterns.

Affected by organic matter and suspended particles in the underwater environment, underwater images suffer from information loss and low contrast, and the underwater images obtained by traditional visible light imaging techniques are not effective. Based on this problem, a non-subsampled contourlet transform (NSCT)-based underwater polarization image fusion method is proposed in this paper. First, we use a division of time polarimeter and polarization imaging technique to acquire underwater target images. Then, the contrast limited adaptive histogram equalization algorithm and the two-dimensional median filtering algorithm preprocess the visible image and the degree of polarization image, respectively. After that, the low-frequency and high-frequency sub-bands of the images are decomposed by the non-subsampled contourlet transform. The fusion rule for the low-frequency subband adopts the adaptive fuzzy method. For the high-frequency sub-bands, the fusion rule of taking the larger absolute value in pixels is used. In the experiment, popular methods were used to compare the advantages of the proposed method. The experimental results show that the proposed method is effective in improving the evaluation indexes, such as information entropy, enhancement measure evaluation, contrast, average gradient, and standard deviation. And the visual effect is much more comfortable to observe. The method is an effective fusion algorithm that can be applied to multiple material targets to improve the quality of underwater polarization imaging. The code is available at a Github repository at: https://github.com/Si-Yu12/Underwater-polarization-image-fusion-based-on-NSCT.

As the number of government and commercial satellites increases, there is a large increase in Earth observation (EO) imagery. Using different locations and tools, images can be taken from more than one satellite. Manipulations are carried out on these images using a variety of different methods. The number of studies that have been done on the manipulation of EO images is very small. In recent years, generative adversarial networks (GANs), a major breakthrough in deep learning, have made it very easy to obtain fake images. In this study, scene-by-scene fake images were obtained with the deep convolutional GAN on the EuroSAT dataset, which is one of the EO image sets, and fake scene images were obtained from the original scenes. In this study, a dataset called RF-EuroSAT was created. It consists of 14 classes and 36,000 images. Five transfer learning models (VGG-16, DenseNet201, MobileNetV2, RegNetY320, and ResNet152V2) were used to classify this dataset. Using these models as feature extraction and ensemble models (XGBoost, CatBoost, and LightGBM) as classifiers, the classification process was performed using our proprietary transferemble model. The best result was obtained with an accuracy of 91.55% using our transferemble model, which is developed in a modular structure.

Few-shot image classification aims to perform image classification on new categories with only a small amount of labeled training data. However, it is difficult to complete such a task under the existing conditions. Therefore, the current few-shot learning methods are built on the paradigm of transfer learning, where the core idea is to learn a priori that can solve unknown new tasks. The idea of meta-learning is fast learning, which is similar to the idea of few-shot learning. Therefore, conventional few-shot learning methods generally adopt a meta-training approach with episodic training to construct knowledge prior. However, research work indicates that an embedding model with powerful feature representations is simpler and more effective than the existing sophisticated few-shot learning methods. Inspired by this insight, we propose a few-shot image classification method based on the transductive clustering optimization learning. First, memory block is introduced to store the internal feature structure of each category, namely comprehensive representation of various features. Second, during the training process, sample features are compared with the memory content to further expand the differences between different categories and improve the performance of feature extractor. Finally, in the transductive few-shot setting, a clustering optimization module has been introduced to select appropriate samples for clustering to alleviate the fundamental problem of sample scarcity. A large number of experimental results show that the proposed few-shot image classification method based on the transductive clustering optimization learning effectively improves the classification accuracy under various training conditions.

Image steganography is a technique for hiding secret data within a cover image and thus making the data immune to typical steganalysis attacks. Recently, there is a great concern about color distortion in most commonly used color-based image steganography methods. To address this issue, we first propose a steganography method that hides secret images in grayscale image and creates a color image. Specifically, we design a colorable information hiding mechanism to convert the image steganography task to gray-to-color conversion. In addition, we exploit the invertible neural network to realize image hiding and recovering in the forward and backward propagation, respectively. Meanwhile, we develop a perceptual loss function to effectively improve the quality of the stego image for increasing steganography security. Experiments indicate that the proposed method outperforms other state-of-the-art image steganography on secret image recovery, embedding capacity, and steganography security over different datasets.

Style transfer is a challenging computer vision task. As an important cultural heritage in the Tibetan region, Thangka murals cover history, culture, and religion. The stylistic recreation of Thangka images not only promotes better appreciation of Thangka art but also provides a new art form for Thangka. Two challenges prevent the existing methods from solving Thangka stylization problems: (1) feature point based attention networks ignore manifold alignment of features between content images and style images and are unable to achieve regional consistency of stylized Thangka images well. (2) Due to the lack of comprehensive understanding of content and style losses in previous studies, the style-content trade-off of the stylized Thangka is unable to be guaranteed. To solve these problems, we propose a progressive style-attentional network (PSANet) and a multi-level loss function strategy for Thangka style transfer. The proposed method consists of two parts: (1) a PSANet to align content manifolds and style manifolds. (2) A multi-level loss function strategy to achieve a balance between content and style of the stylized Thangka images and to ensure better content preservation. Qualitative and quantitative experiments show that our proposed method is able to achieve satisfactory stylized effects on Thangka images.

In the agricultural field environment, in order to meet the strict requirements of modern intelligent agriculture on the accuracy and speed of pest detection algorithms, we proposed a lightweight farmland pest detection algorithm based on an improved YOLO v5. First, we constructed a new backbone by combining PPLCNet and BotNet to reduce network parameters and enhance the ability of image feature extraction. Second, the soft non-maximum suppression (Soft-NMS) algorithm is used instead of the NMS algorithm to improve the detection of target feature areas and minimize duplicate detection of the same areas. Then, the scylla intersection over union is applied to replace the original loss function complete intersection over union in the model to obtain higher quality anchors. Besides, mosaic data enhancement, scaling, translation, rotation, and color transformation methods are used to enhance the data volume, and letterbox is employed to unify data set sizes. Finally, we performed performance analysis experiments on a dataset containing 15 species of agricultural pests. From a large number of experimental results, we can conclude that our model improves the precision by 4.3% compared with YOLOv5n, while reducing the reasoning speed and model size by 10.1% and 50%. Furthermore, compared with faster R-CNN, cascade R-CNN, CenterNet, RetinaNet, YOLOv3, YOLOv4, YOLOv5, and YOLOx, our model shows the highest detection accuracy and the fastest reasoning speed. The mAP@0.5 is 95.3%, and the reasoning time of each image is only 1.7 ms.

The task of classifying small objects is still challenging for current deep learning classification models [such as convolutional neural networks (CNNs) and vision transformers (ViTs)]. We believe that these algorithms are not designed specifically for small targets, so their feature extraction abilities for small targets are insufficient. To improve the classification capabilities of CNN-based and ViT-based classification models for small objects, two multidomain feature fusion (MDFF) frameworks are proposed to increase the amount of feature information derived from images and they are called MDFF-ConvMixer and MDFF-ViT. Compared with the basic model, the uniquely added design includes frequency domain feature extraction and MDFF processes. In the frequency domain feature extraction part, the input image is first transformed into a frequency domain form through discrete cosine transform (DCT) transformation and then a three-dimensional matrix containing the frequency domain information is obtained via channel splicing and reshaping. In the MDFF part, MDFF-ConvMixer splices the spatial and frequency domain features by channel, whereas MDFF-ViT uses a cross-attention mechanism to fuse the spatial and frequency domain features. When targeting small target classification tasks, these two frameworks obviously improve the utilized classification algorithm. On the DOTA dataset and the CIFAR10 dataset with two downsampling operations, the accuracies of MDFF-ConvMixer relative to ConvMixer increase from 87.82% and 62.14% to 90.14% and 66.00%, respectively, and the accuracies of MDFF-ViT relative to the ViT increase from 79.22% and 36.2% to 88.15% and 59.23%, respectively.

A reversible data hiding (RDH) method with prediction error histogram (PEH) decomposition and hybrid dimensional histogram modification is proposed. First, a global one-dimensional (1D) PEH is generated by pixel-based pixel value ordering predictor. The proposed method fully utilizes the pixels inside the range of the minimum and maximum of neighbors to decompose 1D-PEH into several sub 1D-PEHs. Then, each sub 1D-PEH is separated into two parts. The part with small-magnitude is used to construct a reduced two-dimensional (2D) PEH, whereas the rest is shifted for reversibility. Finally, data embedding is performed by modifying each reduced 2D-PEH to reduce distortion. Experimental results demonstrates that the proposed method outperforms some high-fidelity RDH methods.

The prevalence of deep learning has attracted interest in the face image manipulation domain, especially in face editing with disentanglement representation. However, how to realize controllable disentangled representation of face images still remains challenging. Current methods require extensive supervision and training, or images will have a significantly impaired quality. We present an approach that learns how to represent data in an ideal disentangled way, with minimal supervision. Specifically, we use a swapping autoencoder with identity and attribute branches to learn identity and attribute representations, respectively. In addition, we separate the process of disentanglement and synthesis by an advanced pre-trained unsupervised StyleGAN2 image generator to make the entire network structure focus on learning data disentanglement. The identity and attribute vectors from different images are combined into a new representation that is mapped by a linear mapper into the generator’s latent space to generate a new hybrid image. In this way, we take advantage of StyleGAN2’s most advanced quality and its expressive latent space without the pressure of training a decoder. Experimental results prove that our method successfully separates the identity and other attributes of face images, outperforms existing methods, and requires less training and supervision.

In low-light-level detection, glow and hot pixels in some imaging sensors become visible due to long-exposure time, leading to image quality degradation. To solve the problem of glow and hot pixels in a single image, an improved extraction algorithm based on the idea of robust principal component analysis is proposed to remove them. The image is divided into three terms in our algorithm: a low-rank matrix (image without glow and hot pixels), an extremely sparse matrix (hot pixels), and a sparse and spatially smooth matrix (glow). Specifically, the total variation norm and ℓ₁-norm are exploited to describe the property of glow. Moreover, a top-hat filter and a boundary-searching method are introduced into the soft threshold operator to improve accuracy. The superiority of the proposed approach is demonstrated with evaluations on simulated datasets, quantitative metrics, and real data.

Unmanned missions have become more and more popular in recent years. The related technologies of unmanned ground vehicles and unmanned aerial vehicles are growing rapidly, but research on unmanned surface vehicles (USVs) is rare. Water surface object detection algorithms play a crucial role in the field of USVs. However, achieving an object detection algorithm that balances speed and accuracy in the presence of interference is a difficult challenge. We proposed a network, DBCR-YOLO, that improved the detection accuracy while meeting real-time requirements. Based on YOLOv5, we added an additional detection head for detecting tiny objects. Then, we replaced the downsampling in YOLOv5’s backbone network with the proposed double sampling mechanism to solve the problem that paying attention to the key features of objects cannot be done in the downsampling process of YOLOv5. Finally, we substituted the proposed BCR neck for YOLOv5’s neck, thus improving the fusion of features between different scales based on fewer parameters and fewer calculations. We tested our network on the water surface object detection dataset. Compared with YOLOv5, DBCR-YOLO improved the detection accuracy by 3.4%. At the same time, DBCR-YOLO achieved the highest accuracy in comparison with other networks.

Insulators that connect high-voltage transmission lines may experience various faults due to long-term exposure to the natural environment, leading to safety degradation and reliability issues in power grids. Therefore, detecting defective insulators through daily maintenance and long-term overhaul is crucial. We propose an insulator defect detection method to achieve this objective, using a novel neural network named more information-you only look once (MI-YOLO). MI-YOLO includes several modifications compared to YOLOv5s: a skip connection module with down-sampling in the backbone, a spatial pyramid dilated convolution module in the neck, and a novel serial-parallel spatial-channel attention module in between. These changes enhance the feature abstraction process by providing more information (MI) through the YOLO-like structure, hence the name MI-YOLO. To evaluate the superiority of MI-YOLO, multiple experiments are performed. First, an ablation study demonstrates the effectiveness of novel substructures and their combinations compared to YOLOv5s. Then, MI-YOLO is compared to baseline structures, including faster R-CNN, SSD, and five YOLOs. Furthermore, the performances of MI-YOLO and 10 state-of-the-art object detection methods are evaluated. Lastly, a second dataset is adopted to evaluate the generalization property and heat maps are presented. The experimental results show that the new designs substantially improve the accuracy of insulator fault detection by abstracting more information from the features.

In this digital era, images are the essential medium of communication. They may be found on magazine covers, legal evidence, newspapers, and social media. With the advancement of numerous editing tools, image modification will pose a serious concern due to the spread of information. Image splicing is one of the leading strategies in image forgery. While there are several approaches to solving the problem, some have flaws, such as the accuracy of faked region localization. We present an effective image-splicing forgery localization model called VGG16Unet. The VGG16Unet is an encoder-decoder architecture for the localization of image-spliced forgery. The pre-trained VGG16 is used as an encoder part of the architecture, and our network’s decoder part is inspired by UNet architecture. The faked images and their respective masks are sent over the network in form of patches to train the model. This model is tested on the widely used dataset of image splicing forgery, i.e., NIST and CASIA v2. The experimental result illustrates that VGG16Unet’s performance of localizing the image-spliced forged regions is more effective than prior methods.

Recent studies have shown that the method combining the back-projection mechanism and deep convolutional neural network can well solve the problem of interdependence between high- and low-resolution images in single image super-resolution (SISR). However, this method still suffers from problems, such as large number of parameters and blurred detail textures. To address these problems, we propose a progressive residual feedback network with additional constraints for SISR. Specifically, we design a progressive residual structure that improves the densely connected network. This structure not only ensures feature reuse between each layer of the dense network through residual connections but also reduces the network complexity through progressive up-sampling reconstruction. In addition to enhance the detailed texture of the reconstructed image, we construct a spatial feature projection block based on the coordinate attention mechanism, so the extracted features contain spatial information that is relevant to the image reconstruction. Finally, we simulate the image degradation process with a degradation block to provide additional degradation constraints for the up-sampling network and optimize the convergence state-of-the-network. Numerous experimental results have proven the effectiveness of our method, which have advantages in both recovery quality and network complexity compared with other state-of-the-art SISR methods.

As the issue of garbage pollution becomes increasingly urgent, accurate and effective garbage detection can play a crucial role in minimizing environmental pollution and facilitating the conversion of waste into energy. However, the accuracy of current waste detection algorithms is limited. We propose an approach called the adaptive activate decoupled head for YOLOv5 (AADH-YOLOv5) for detecting garbage. Our model improves localization accuracy without significantly increasing floating point of operations (FLOPs) or parameters. Extensive testing on the newly created domestic garbage dataset-6 demonstrates that AADH-YOLOv5 outperforms YOLOv5l and Cascade R-CNN, achieving a 5.4% and 4.2% improvement in mAP@0.5, respectively. Our study also contributes to the field by providing a new dataset for garbage detection research.

Selective encryption algorithms have emerged as a popular technique for protecting the privacy of images during real-time transmission. For selectively encrypted images, it is necessary to evaluate their security and usability with visual security indices, and there have been a series of studies in this area. However, those proposed visual security indices (VSI) are often ineffective. We present a multi-directional structure and content-aware features-based visual security index (MCVSI) to perform an objective assessment on selectively encrypted images. Considering that selectively encrypted images prevent the main contents from being easily identified, stable local features in the images are extracted to indicate the degree of image content leakage. Meanwhile, we extract spatial structure information that closely aligns with human visual perception to indicate the level of variation in the overall image skeleton. Next, these features are subjected to similarity measurements to produce two types of similarity, content perception feature similarity and structure feature similarity. Finally, our visual security index is built by connecting all feature similarities and their corresponding visual security scores using the regression module. The experimental results and analyses indicate that the proposed MCVSI outperforms many existing mainstream VSI in terms of higher performance and stronger robustness, particularly on low and medium quality images.

Video denoising is a low-level task that uses the information of neighbor frames to denoise polluted frames. Using the information provided by neighboring frames, noise can be effectively removed and the denoising effect is improved. Through continuous development, single-stage and multi-stage denoising methods have been proposed. However, these existing denoising methods simply use the information provided by neighbor frames to denoise the polluted frames, paying insufficient attention to the polluted frames and making insufficient use of the information of the polluted frames. We propose a two-stage video denoising network called multiple forward-backward strategies for two-stage video denoising, which improves the denoising performance and retains more detailed information by enhancing the utilization of information of polluted frames. Extensive experiments have been conducted, and the experimental results show that the proposed method outperforms the state-of-the-art denoising methods in both qualitative and quantitative comparisons.

At present, most iris segmentation methods based on deep learning have poor robustness in the face of iris images collected in non-cooperative environments (with partial occlusion, distortion, etc.), and their performance decreases to varying degrees. Inspired by the visual transformer (ViT), we combined the advantages of the ViT and ConvNeXts networks to propose a deep learning-based robust iris segmentation method called CINet. Specifically, we introduced global region aware (GRA) in ConvNeXts to capture global spatial information. It increases the sensitivity of the model to the inner and outer boundaries of the iris, achieving efficient iris segmentation. In addition, it also suppresses noise information irrelevant to the iris region, thus improving the robustness of the model. We used global channel normalization instead of batch normalization, which suppresses some unimportant channel information, further enhancing the network’s performance. Experimental results demonstrate that GRA provides important feature information, which is essential for efficient iris segmentation. We verify the effectiveness of the proposed method on three benchmark iris datasets.

In recent times, the volume of video surveillance data has been rapidly increasing, and processing is required to detect various kinds of activities in these data. Most human activity recognition methods primarily focus on detecting normal human activities; however, identifying and detecting suspicious activities is equally important. We propose a two-stream framework for detecting abnormal human activities in real-time videos. The proposed method uses the first stream to extract features related to spatial characteristics and the other stream to extract the temporal features from the videos. The spatial features focus on the contextual details in the frame, including color, texture, and other robust features. The temporal features focus on the features related to the changes in frames with respect to the time, which play a vital role in the detection of activity. The proposed framework leverages the Xception model to extract the features in the spatial and temporal streams. The extracted features are integrated for further processing and detection. To process the integrated features and detect the activity, we train a multi-layer bidirectional long short term memory network using the extracted features. The proposed framework is trained in an end-to-end manner to accurately estimate human activity by learning patterns and forming correlations. The effectiveness of the proposed approach is evaluated by assessing the popular UCF Crime dataset, which encompasses a variety of criminal activities, and using performance measures such as recall, precision, and accuracy. Experimental results indicate that the proposed approach outperforms other contemporary methods, achieving an accuracy of 86%.

Hard exudate is one of the early symptoms of diabetic retinopathy (DR). If hard exudates are detected promptly, the doctor can provide timely treatment advice for the patient’s condition. However, due to the specific nature of hard exudates in the fundus, there are significant differences in the size of the lesions. To solve this problem, a dual-branch transformer network is proposed, which can effectively balance the size difference of lesions and improve the segmentation performance. First, the image is segmented into small pieces of the same size and the same image is fed into two branches. Then, the features are extracted with CNN, trained with transformer, sampled on the decoder, and fused with the features obtained from CNN. The segmentation of lesions is improved by adjusting the lesion sizes and their weights between different branches to gradually focus on small lesions. This method is tested on the e-ophtha and IDRiD datasets to evaluate the network performance. The results show that our method can achieve better performance than the most advanced methods at present.

Latent representation features in deep learning (DL) exhibit excellent potential for visual data applications. For example, in traffic monitoring and video surveillance, the features simultaneously perform image analysis for machine vision and image reconstruction for human viewing. However, the existing deep features that appeal to machine and human receivers are always combinations of separated pieces and specific features. Due to these features being extracted from different branches in collaboration frameworks, the inherent relations between machine and human vision are insufficiently explored. Therefore, to obtain one set of representative and generic features, we propose a dynamic groupwise splitting network based on image content to explore and extract generic features for the two different receivers. First, we analyze the characteristics of the latent features and adopt intermediate features as the base features. Then, a feature classification and transformation mechanism based on image content is proposed to enhance the base features for further image reconstruction and analysis. Consequently, an end-to-end model with multimodel cascading and multistage training realizes both machine and human vision tasks. Extensive experiments show that our human–machine vision collaboration framework has high practical value and performance.

Images taken under low light conditions tend to suffer from poor visibility, which can decrease image quality and even reduce the performance of downstream tasks. It is hard for a CNN-based method to learn generalized features that can recover normal images from the ones under various unknown low light conditions. We propose to incorporate the contrastive learning into an illumination correction network to learn abstract representations to distinguish various low light conditions in the representation space, with the purpose of enhancing the generalizability of the network. Considering that light conditions can change the frequency components of the images, the representations are learned and compared in both spatial and frequency domains to make full advantage of the contrastive learning. Additionally, a grayscale self-weight perception method is used to preproccess the images to reduce the complexity of the model in coping with the uneven distribution of image illumination. The proposed method is evaluated on LOL and LOL-V2 datasets, and the results show that the proposed method achieves better qualitative and quantitative results compared with other state-of-the-art methods.

Underwater image restoration is of great significance to underwater vision operation. We propose an underwater image restoration method based on the independent color channel of a single image. In this method, the relationship between the transmittance parameters and the second-order statistics of the scattered light underwater is established through a multi-layer continuous model of the water. First, the region with approximate depth is obtained by super-pixel segmentation, then the sub-color channel statistical calculation is carried out for each super-pixel independently, and finally the transmittance of each color channel is estimated combined with the relational expression. Experimental results show that this method has better effect on color fidelity of underwater image restoration.

Deep learning-based halftoning methods have demonstrated their abilities in dithering images with desired blue-noise properties. However, tons of parameters and computations come along with the 2D convolutional neural networks (CNNs), which hinder their practical usage, particularly in the cost-sensitive printing imaging system equipped with low-end computing devices. We devise a fast halftoning policy network to alleviate the problem. Motivated by classic error diffusion algorithms, we introduce an error feedback mechanism to model the dependencies between halftone lines. Consequently, the problem collapses to 1D deep halftoning within each line, which simplifies the learning task. Furthermore, a very simple convolutional layer is applied to capture the texture details from the continuous-tone image. We find that the proposed halftoning model can stably converge under the reinforcement learning-based framework. Experiments show that our model achieves a 10.6× speedup with only 1.7% parameters compared to a previous 2D CNN-based work while maintaining similar halftone quality. We have also conducted experiments on a digital signal processor to demonstrate how the proposed model saves expensive off-chip memory accesses and improves processing throughput in real scenarios.

Knowledge distillation, which transfers knowledge from a large model (the teacher) to a small model (the student), is a promising way in the field of lightweight model design. Existing distillation methods based on intermediate features mainly focus on knowledge transfer between the same stages of teacher and student networks, which may lead to student networks receiving semantically mismatched knowledge and missing contextual knowledge. To solve this problem, a cross-stage distillation with inverted bottleneck projectors is proposed. A cross-stage connection structure is designed that enables the student network to access the teacher network’s most matched stages in semantics and further obtain rich contextual knowledge from multiple stages of the teacher. After establishing the cross-stage connection between the teacher network and the student network, inverted bottleneck projectors are employed to extract useful knowledge from multiple stages of the teacher. A stack structure with three-layer convolutions is proposed in every projector, which can make the knowledge of the teacher network more easily understood by the student network. The inverted bottleneck structure can reduce information loss, ensuring the integrity of knowledge transmission. Further, ReLU activations are incorporated into the projectors to remove features with lower response values, thereby filtering redundant elements. Extensive experiments on CIFAR100, ImageNet, Tiny-ImageNet, and STL-10 datasets demonstrate the effectiveness of the proposed approach.

Two-stream networks have been widely used in action recognition by integrating the appearance information from RGB frames with the motion-rich optical flow data, resulting in impressive recognition accuracy. However, the drawbacks of two-stream networks using 2D convolutional neural network (CNN) are apparent. The computation of optical flow is resource intensive and time consuming, and 2D CNN cannot model temporal information. To alleviate these problems, we propose a temporal interaction and excitation (TIE) module that can be embedded into existing 2D CNN in a plug-and-play manner. It comprises two components: the temporal interaction fusion (TIF) module and the motion excitation (ME) module. The TIF module employs channel-wise temporal convolution to adaptively fuse information from adjacent video frames, facilitating information exchange between the adjacent frames while maintaining its spatial feature learning capability. The ME module is a lightweight motion extraction module that leverages the differences between adjacent frames to model feature-level motion information, replacing the need for traditional optical flow. This enhancement helps the model better understand human actions in videos. The TIE module is designed to improve the network performance while adding minimal computational cost. Experimental results demonstrate that our proposed method adds only a few parameters and computational costs while achieving competitive results on HMDB-51 and UCF-101 datasets.

Image synthesis is a critical technique in the image processing field. Recently, generative adversarial networks (GANs) have played a significant role in synthesis tasks. However, the issue of mode collapse remains a major challenge in GANs, which limits their potential applications. We propose a method to address the mode collapse problem. Our approach focuses on minimizing the divergence between the distributions of real and generated features, thereby reducing the learning pressure on the discriminator. An advantage of our method is that it does not require prior knowledge or manual design. Additionally, it can be easily incorporated into state-of-the-art frameworks across various domains. Experimental results demonstrate the effectiveness and competitive performance of our proposed method.

The accurate detection and location of a human foot moving target is the prerequisite for a small robot to realize short-range robust tracking of human moving targets. An F-YOLOv3 (FlowNet2-YOLOv3) algorithm based on deep optical flow and neural network spatiotemporal information fusion that can accurately detect and locate foot moving targets is proposed. The algorithm extracts the motion information of the human foot target in the time domain and the position information of the human foot target in the space domain through the FlowNet2 and YOLOv3 networks, respectively, for foot detection. Then, according to the established fusion strategy, the temporal and spatial information of the foot target extracted by the two networks is fused to realize the foot target location. Five groups of human walking videos of typical scenes are used to simulate the detection and location of foot targets. The results show that the target detection precision of the F-YOLOv3 fusion algorithm is 85% in the scene of a human turning and foot occlusion and 90% in the scene with drastic illumination change. Compared with the YOLOv3 algorithm, the average mAP value obtained by F-YOLOv3 is increased by 3.89%. Finally, under the robot operating system, the F-YOLOv3 algorithm is applied to the turtlebot robot, and foot detection and positioning in the process of human walking are successfully completed.

Real-time semantic segmentation has been challenging, and the fusion of features from different branches remains crucial to improvement. The two-branch structure has shown promising results in real-time semantic segmentation. However, upsampling feature maps from the semantic branch to match the detail branch leads to a loss of object feature information and compromises segmentation accuracy. We propose a deep bilateral fusion and bilateral embedded network (BFBE-Net) based on the encoder–decoder structure for real-time semantic segmentation to address these issues. The BFBE-Net adopts a two-branch design in the encoder, with a top-down fusion module and a bottom-up fusion module designed to integrate multi-scale context information in the channel dimension, and assigns different weights to detailed information and semantic information to enhance information characteristics. In the decoder, a bilateral embedded attention module under the guidance of spatial and channel attention integrates semantic and spatial features, gradually upsampling feature maps to reduce the loss of feature information. In addition, an enhanced aggregation pyramid pooling module is designed to efficiently extract contextual information by combining depth-wise asymmetric convolution. The proposed algorithm is evaluated on two benchmark datasets, Cityscapes and CamVid, achieving 78.5% mean intersection over union (mIoU) at 82 frames per second (FPS) on the Cityscapes test set and 79.2% mIoU at 131 FPS on the CamVid test set. The proposed BFBE-Net not only improves segmentation accuracy but also ensures real-time performance.

Encrypted traffic classification (ETC) plays an important role in network management. In most research, the statistical features, transformed traffic images, or text are used for classification. However, the statistical features’ design is time-consuming and labor-intensive, and the transformed traffic data lack spatial or semantic features. Considering that the headers of traffic packets have a uniform structure and are independent of each other, traffic data are most similar to tabular data. Thus we propose a data processing approach to convert packet headers into traffic tables in which each field is viewed as a column (feature). In addition, traffic data are hard to label in real traffic environments, and each field contributes differently to the classification. Therefore, a self-supervised learning algorithm, SubTab, is used as the baseline network to reduce the reliance on labeled data and assign different weights to different fields. To the best of our knowledge, this is the first time that the ETC problem is solved from the tabular domain. Experimental results on two real-world datasets, ISCX VPN-nonVPN and the self-collected dataset SHU-ET, demonstrate that our method surpasses state-of-the-art methods based on traffic images or text and proves that traffic tables are more suitable for ETC problems. In addition, our method achieves a great performance with only 10% of labeled data and reduces the reliance on labeling data.

Achieving real-time performance while maintaining high accuracy in semantic segmentation can be a challenging task. Many existing methods adopt multi-branch architectures to extract both spatial and semantic information, resulting in increased computational complexity and a lack of communication between branches. We propose a pseudo bilateral segmentation network (PBSNet) that can extract rich spatial and semantic features from a single path, without incurring additional computational cost or time consumption. Our proposed PBSNet utilizes a semantic enhancement module to explore the relationship between high-level semantic features, an interchange module to enhance feature representation through bi-directional vertical propagation and adaptive spatial attention, and an attention fusion module to aggregate multi-scale features to produce the final segmentation prediction. Our results on the Cityscapes dataset demonstrate the superiority of PBSNet over state-of-the-art methods, achieving a balance of accuracy and efficiency with 74.52% mean intersection over union and 82.5 frames per second.

With the rapid development of computer vision, the omnidirectional image (OI) with a broader view and a higher resolution provides humans with an immersive visual experience and has attracted more researchers’ attention. But current image quality assessment models cannot perform well due to neglecting the difference between panoramic images and traditional images, limiting OIs’ further development. Inspired by the above problem, we propose a blind OI quality assessment model based on multi-frequency features, color features, and human visual system-based saliency features. Specifically, we utilize the high- and low-frequency information mapped to the frequency domain to measure the frequency domain loss. The local binary pattern operator is applied to encode the different color channel information. Then the local natural scene statistics and entropy features are extracted using the input image to weight the obtained saliency image by our model to reduce the model’s computational complexity. Finally, support vector regression is employed to predict the OIs’ quality scores. Experimental results on CVIQD and OIQA databases prove that our work performs better than state-of-the-art OIQA models.

Object detection models are at the core of various computer vision tasks and have shown excellent performance on public datasets, but they also inherit the disadvantage of neural networks that they are vulnerable to adversarial example attacks. Adversarial patches are specific forms of adversarial examples that, as shown in previous studies, can only make specific objects (such as pedestrians and traffic signs), but not all objects, disappear. In addition, a patch must be placed on every object to deceive the detector. To solve the above problems, we propose a location-independent adversarial patch generation method that can attack objects in the range to be detected with a single patch. By attacking the confidence loss of the object detector, we creatively assign a greater weight to the foreground region, which makes its confidence decrease faster and effectively guides the convergence direction of the adversarial patch in the training process. Furthermore, we glue the patches randomly on the images to make them less sensitive to location during patch training. Experimental results indicate that the patches generated using our proposed method are not restricted to specific areas of the image and provide a minimum recall of 29.5%.

Rumination plays a pivotal role in assessing the health status of ruminants. However, conventional contact devices such as ear tags and pressure sensors raise animal welfare concerns during rumination behavior detection. Deep learning offers a promising solution for non-contact rumination recognition by training neural networks on datasets. We introduce UD-YOLOv5s, an approach for bovine rumination recognition that incorporates jaw skeleton feature extraction techniques. Initially, a skeleton feature extraction method is proposed for the upper and lower jaws, employing skeleton heatmap descriptors and the Kalman filter algorithm. Subsequently, the UD-YOLOv5s method is developed for rumination recognition. To optimize the UD-YOLOv5s model, the traditional intersection over the union loss function is replaced with the generalized one. A self-built bovine rumination dataset is used to compare the performance of three deep learning techniques: mean shift algorithm, mask region-based convolutional neural network, and you only look once version 3 (YOLOv3). The results of the ablation experiment demonstrate that UD-YOLOv5s achieves impressive precision (98.25%), recall (97.75%), and a mean average precision of 93.43%. We conducted a generalization performance evaluation in a controlled experimental environment to ensure fairness, indicating that UD-YOLOv5s converges faster than other models while maintaining comparable recognition accuracy. Moreover, our work reveals that when convergence speed is equal, UD-YOLOv5s outperforms other models regarding recognition accuracy. These findings provide robust support for accurately identifying cattle rumination behavior, showcasing the potential of the UD-YOLOv5s method in advancing ruminant health assessment.

As the demand for long-term health evaluation grows, researchers show increased interest in remote photoplethysmography studies. However, conventional methods are vulnerable to noise interference caused by non-rigid facial movements (facial expression, talking, etc.). Consequently, avoiding these interferences and improving the remote photoplethysmography (rPPG) signal quality become important tasks during heart rate (HR) estimation. We propose an approach that extracts high-quality rPPG signals from various subregions of the face by fusing static and dynamic weights and then employs the convolutional neural network to estimate HR value by converting the 1D rPPG signal into 2D time-frequency analysis maps. Specifically, chrominance features from various regions of interest are used to generate the raw subregion rPPG signal set that is further utilized to estimate the static weights of different regions through a clustering method. Additionally, a measurement method called enclosed area distance is proposed to perform static weights estimation. The dynamic weights of different regions are calculated using the 3D-gradient descriptor to eliminate motion interference, which evaluates the inactivation degree under regional movement situations. The final rPPG signal is reconstructed by combining the rPPG signals from the different subregions using the static and dynamic weights. The experiments are conducted on two widely used public datasets, i.e., MAHNOB-HCI and PURE. The results demonstrate that the proposed method achieves 3.12 MAE and 3.78 SD on MAHNOB-HCI and the best r on the PURE, which significantly outperforms state-of-the-art methods.

Growing reliance on digital communications has necessitated development of dependable and secure technologies to ensure that the transmission and reception of images over the Internet do not pose a risk to the data of individuals and governments. We propose developing a hybrid image encryption and compression algorithm by combining compressive sensing, the gray wolf algorithm, and multi-dimensional chaotic systems. It aims to generate a highly secure encrypted image while conserving transmission and storage resources. This algorithm overlaps several stages designed to protect vital images while minimizing size. First, the image is converted to the frequency domain using the discrete wavelet transform. Then, the discrete wavelet transform coefficients are scrambled globally using the Waleed-Ali Map and the gray wolf algorithm. Second, the confused image is measured by a parameters-controlled matrix to reduce transmission costs. The final encrypted image is obtained after performing the diffusion operation with a bitstream derived from the Nahrain chaotic map. The average peak signal-to-noise ratio score was 53.1995, and the average mean squared error score was 0.6130, demonstrating that the plaintext and decrypted images are identical. The average correlation coefficient score was −0.010095; the average entropy analysis was 7.9987; and the average number of pixel change rate and unified average changing intensity analyses were 99.60 and 33.52, respectively. The experimental results demonstrate the algorithm’s efficiency and robustness, as well as the high quality of the reconstructed image.