Guest Editors Keping Yu, Wei Wang, Moayad Aloqaily, Muhammad Tariq introduce the Special Section on Biologically Inspired Computer Vision and Image Processing.

The iris segmentation algorithm plays a vital role in iris recognition systems and directly controls the authentication and recognition results. However, traditional iris segmentation techniques have reduced performance, and they are not robust enough when used in noisy iris databases captured under immeasurable conditions. In addition, there is currently no extensive iris database. Therefore, the iris segmentation algorithms cannot reap the benefits of the convolution neural networks. Iris segmentation plays an essential role in maintaining iris accuracy by restricting the defects in iris images. Under these nonideal conditions, presented segmentation based on local operations cannot see the actual iris boundary, and the iris segmentation will fail. The proposed solution contributes to the effect of the distribution of the most irregular iris images under visible light. Gray level co-occurring matrix is proposed for segmentation, and VGG 16 and VGG 19 are applied for iris classification. The proposed iris recognition method divides the iris segmentation process into two steps: localization of the iris area of the eye and segmentation. Well-designed VGG 16 and VGG 19 networks are used to distinguish and locate the eyes. Performance for both the proposed models shows improvement over other techniques in terms of accuracy, precision, recall, E1, and E2.

Image quality is greatly affected by noise in various applications such as machine vision and computer vision. Noisy images lead to inaccurate results as well as segmentation and enhancement errors, making denoising vital to efficient image processing. Existing denoising methods do not work effectively on both grayscale and color images. Thus an effective denoising method that works well on both image types is proposed. The proposed two-stage method involves: (i) noisy pixel detection and (ii) noisy pixel restoration. In the first stage, the conglomerate method detects the noisy pixel position; in the second stage, the corrupted value of the noisy pixel is restored using a conglomerate method. The proposed method was used to process a variety of grayscale and color images. The result analysis of image denoising was done in two stages: in the first stage, results are carried out using large standard datasets; comparison of these results with those of five other denoising methods showed that the proposed conglomerate denoising method outperformed these existing ones. The second stage of result analysis considers the state-of-the-art methods in comparison with the proposed method. The proposed method outperforms the state-of-the-art methods for images affected with noise ranging from 10% to 90%.

Texture classification has been one of the problem-solving strategies. The feature selection technique is critical for increasing the efficiency of machine learning because it significantly improves the performance of texture classification by removing irrelevant features from the original collection. The majority of methods for selecting features are statistical in nature. Accuracy in database classification is primarily achieved by selecting attributes and simultaneously increasing the classification rate. The task’s primary objective is to select the most significant features from the feature database for execution. We use the cluster-based feature selection method to derive features for statistical and structural approaches. The proposed strategy should be implemented in three stages. To begin, the function can be extracted using the local binary pattern (LBP), the gray level co-occurrence matrix, or the Gabor filter. Following that, a modified K-means clustering method is used to select features using four standard distance measurements: Euclidean, Minkowski, Chebychev, and City Block. Finally, benchmark classifiers such as naive bayes (NB), support vector machine (SVM), and K-nearest neighbor (KNN) were used to effectively compute the classification accuracy of the research. The proposed approach achieves accuracy of 99.4328 for KNN, 98.6654 for NB, and 99.6712 for SVM with the highest classification rate.

As a challenging task in computer vision, instance segmentation has attracted extensive attention in recent years. Able to obtain very rich and refined object information, this technology shows important application value in many fields, such as intelligent driving, medical health, and remote sensing detection. Instance segmentation technology should not only identify the positions of objects but should also accurately mark the boundary of any single instance, which can be defined as solving object detection and semantic segmentation at the same time. Our study gives a detailed introduction to the background of instance segmentation technology, its development and the common datasets in this field, and further deeply discusses key issues appearing in the development of this field, with the future development direction of instance segmentation technology proposed. Our study provides an important reference for future research on this technology

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

NS did not agree with the retraction. CC either did not respond or could not be reached.

We introduce a robust watermarking method to efficiently embed multiple marks into ECG signals using redundant discrete wavelet transform and generalized singular value decomposition. Multiple marks are embedded within the signal to eliminate any ownership conflict and provide copyright protection. The hybrid optimization technique is also used to balance the invisibility and robustness. Further, wavelet-based compression is adopted to compress the marked signal before transmission reduces the bandwidth demand. To evaluate the proposed algorithm, the standard ECG dataset is used. The proposed algorithm yields very good invisibility and robustness against several attacks, and it is also suitable for applications in the cloud.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

TY either did not respond directly or could not be reached.

In recent years, reflection is a kind of noise in images that is frequently generated by reflections from windows, glasses, and so on, when you take pictures or movies. The reflection does not only degrade the image quality but also affects computer vision tasks, such as the accuracy of object detection and segmentation. In the task of single image reflection removal (SIRR), deep learning models play a key role for solving the problems of various patterns and the versatility. The challenge of SIRR is the influence of image quality and low precision of the method. We propose a deep learning model for the task of SIRR. The assumed scenes of reflection are varying, and there is little training data because it is difficult to obtain true values. We focus on the latter and propose an SIRR based on meta-learning. We adopt model agnostic meta-learning (MAML), and we propose an SIRR using a deep learning model with MAML, both of which are methods of meta-learning. The deep learning model includes the iterative boost convolutional long short-term memory network, which is adopted as the deep learning model. Experimental results show that the proposed method improves accuracy compared with conventional state-of-the-art methods.

Biometric technologies, such as handwritten signature verification, are extremely useful for identifying individuals inside an organization or finance department. The improvement of picture categorization using deep learning (DL) neural networks has offered an opportunity to exhibit computer vision in contemporary research applications by applying image processing approaches. Manual signature verification is inefficient, error-prone, time-consuming, and inconvenient; therefore, it is critical to create an automatic signature verification recognition system. This research offers an automatic recognition method based on DL that makes use of the Grupo de Procesado Digital de Seales. The biggest publicly accessible handwritten signature dataset, the synthetic signature dataset, was used to classify the signatures of 100 people, each of whom possessed 24 genuine signatures and 30 forged signatures. An inception V3 transfer learning (TL) model is proposed by hyper-tuning different layers from the middle of its architecture and this model is fine-tuned by adding layers, such as flatten, dense (1024), dropout (0.5), and dense (1). The suggested model was tested against six well-known pre-trained TL convolutional neural network models: VGG 16, VGG 19, ResNet 50, ResNet 101, MobileNet, and EfficientNet. The suggested model surpasses the pre-trained models. Precision, sensitivity, and F1-score are likewise outperformed by the model, with the values of 88%, 88%, and 87%, respectively. The accuracy of the pre-trained models was evaluated as 80%, 81%, 77%, 73%, 71%, and 74%, respectively. The suggested fine-tuned inception V3 gives the highest accurate classifications, distinguishing between genuine and forged signatures with an accuracy of 88%. This study will aid researchers in developing more effective CNN-based models for offline signature verification with application to computer vision.

Iris recognition has been considered as a secure and reliable biometric technology. However, iris images are prone to off-angle or are partially occluded when captured with fewer user cooperations. As a consequence, iris recognition especially iris segmentation suffers a serious performance drop. To solve this problem, we propose a multitask deep active contour model for off-angle iris image segmentation. Specifically, the proposed approach combines the coarse and fine localization results. The coarse localization detects the approximate position of the iris area and further initializes the iris contours through a series of robust preprocessing operations. Then, iris contours are represented by 40 ordered isometric sampling polar points and thus their corresponding offset vectors are regressed via a convolutional neural network for multiple times to obtain the precise inner and outer boundaries of the iris. Next, the predicted iris boundary results are regarded as a constraint to limit the segmentation range of noise-free iris mask. Besides, an efficient channel attention module is introduced in the mask prediction to make the network focus on the valid iris region. A differentiable, fast, and efficient SoftPool operation is also used in place of traditional pooling to keep more details for more accurate pixel classification. Finally, the proposed iris segmentation approach is combined with off-the-shelf iris feature extraction models including traditional OM and deep learning-based FeatNet for iris recognition. The experimental results on two NIR datasets CASIA-Iris-off-angle, CASIA-Iris-Africa, and a VIS dataset SBVPI show that the proposed approach achieves a significant performance improvement in the segmentation and recognition for both regular and off-angle iris images.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of compromised peer review and determined the paper is unrelated to the special section. The Editor and publisher no longer have confidence in the results and conclusions of the article.

MFS disagrees with the retraction. AB and MS either did not respond or could not be reached.

The visual appearance of images changes in line with varying environmental light conditions. Adjusting the exposure of various distorted images is a highly complex process. Previous approaches have addressed this issue from different viewpoints and attained remarkable progress. However, they either failed to achieve visually pleasing results or were suitable for a single class of images (e.g., underexposed or nonuniform images). To fully consider the exposure in various distorted images, we proposed a diverse image enhancement model that improved the brightness and contrast, processed the colors, and eliminated the hazy effect. Accordingly, an input red green blue color image was transformed into a hue, saturation, value color image. The V component was inverted and enhanced using three steps. In the first step, the hyperbolic and statistical methods were applied, and then their results were combined using an adjusted logarithmic methodology. This method properly adjusted the high-contrast and low-contrast impact while preserving the vital image information. In the second step, the output of the first step was inverted back and fed into a complete optimization algorithm to estimate the illumination map. Then, the exposure ratio map was estimated using an illumination map, which was adjusted using the camera response function. In the third step, a nonlinear stretching function was introduced to control brightness and contrast. For instance, a lower value of α yielded maximum stretching, and a higher value of α eliminated haze in the image to a great extent. Finally, an empirical evaluation and comparison of the most recent state-of-the-art approaches on eight datasets revealed that the proposed model efficiently addressed the exposure in various degraded images.

Crowd counting is a considerable yet challenging task in intelligent video surveillance and urban security systems. The performance has been significantly boosted along with the springing up of the convolutional neural networks (CNNs). However, accurate and efficient crowd counting in congested scenes remains under-explored due to scale variation and cluttered background. To address these problems, we propose a biologically inspired crowd counting method named group-split attention network (GSANet). The GSANet consists of three principal modules, namely GS module, dual-aware attention module, and aggregation module. The GS module processes the subfeatures of each group in parallel, and groups the input feature map to reduce the computational cost. The dual-aware attention module synergies the spatial and channel dimensional information to alleviate the estimation error in background regions. The aggregation module adopts a learning-based cross-group strategy to aggregate and facilitate the fusion of feature maps along different channel dimensions. Extensive experimental results on five benchmark crowd datasets demonstrate that the GSANet achieves superior performances in terms of accuracy and efficiency.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of compromised peer review and determined the paper is not relevant to the special section. The Editor and publisher no longer have confidence in the results and conclusions of the article.

CG did not agree with the retraction. MPG either did not respond or could not be reached.

Iris segmentation is an important task in the iris recognition system. Traditional iris segmentation methods are susceptible to irrelevant noise. At present, segmentation methods based on convolutional neural networks (CNNs) have become the main trend in addressing the task. Although these networks achieve promising segmentation performance on the advanced graphics processor, they require a lot of computation and time, which are far from meeting the standards of real-time iris segmentation. To this end, a precise and fast iris segmentation algorithm is proposed. First, an efficient feature extraction network that combines depth-wise separable convolution with dilated convolution is designed to reduce model parameters while maintaining segmentation accuracy. Then, an attention mechanism is introduced to suppress noise interference and enhance the discriminability of the iris region. Finally, an auxiliary training branch is proposed to overcome the vanishing gradient problem. Experimental results show that the proposed method not only achieves state-of-the-art performance but also is more efficient in terms of required parameters, calculated load, and storage space. Specifically, the model parameter is only 0.1 million (M), and the average prediction time is <0.03 seconds (S).

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

NL, JX, and SG either did not respond directly or could not be reached.

Image noise removal (denoising) is a significant preprocessing step in image processing research domain. In past decades, various noise removal algorithms have been developed, and they are categorized into conventional and deep learning denoising methods. Most recently, deep learning-based denoising methods outperform the conventional methods. However, deep learning methods have the limitations of requiring large training sample images, difficult to train the deep convolution neural network (DeCNN), and the majority of the convolutional neural network suffers from the performance saturation. Hence, using combination of block matching (BM) and dilated convolutional neural network (DCNN), a unique method for reducing noise in still images is proposed. In this approach, initially, existing algorithm used to preprocess the image. Then, BM technique uses a sliding window size of a four-by-four block and moves it across the image to select for blocks that are similar in the image. The matching blocks are highly correlated with one another. The matching blocks are then fed to the deep DCNN to remove noise and obtain a better noiseless image. Finally, the noiseless image performances are evaluated using standard metrics. From the experimental results, it is observed that the proposed BM and deep DCNN yielded substantially better results than existing state-of-the-art methods to remove the image’s noise.

Slender objects are more difficult to detect than conventional objects with regular shapes, and the rotation of these objects makes detecting them even more challenging. To address these challenges, we proposed SRPAR, an anchor-free detector with aspect ratio priority for slender objects. The aspect ratio priority factor is designed based on the object’s aspect ratio and rotation angle. The aspect ratio priority factor guides the regression of slender rotating objects and improves the regression accuracy. In addition, through the multi-level prediction of feature pyramid network, the range of bounding boxes and corresponding angular regressions at each level is limited, and the regression of overlapping object prediction boxes is accelerated. To better evaluate the SRPAR’s detection performance of slender rotating objects having an aspect ratio of at least 3:1, some images of sticks are supplemented into the baseball bat subset of the common objects in context (COCO) dataset to form a new self-made COCO-Stick dataset. Experimental results on the dataset of object detection in aerial images dataset and the self-made COCO-Stick dataset show that, compared with state-of-the-art detectors, the proposed method has some advantages in detection accuracy.

Single-image deraining is a classical problem in the field of low-level computer tasks. Most of the recent state-of-the-art image rain removal methods are trained on synthetic images, which have the problems of incomplete rain removal on real images and inability to process complex rain conditions. Based on these limitations, we propose a single-image deraining network based on multistage feature fusion (SIDNMFF). The network performs rain removal in four stages, with the first three stages using an improved encoder–decoder subnetwork for feature fusion to extract global and local information from the image, resulting in more detailed texture information of the obtained image. In the last stage, the network fuses the feature information extracted in the first three stages, performs feature extraction on the original resolution image, and finally outputs the final result of multistage deraining. The proposed method conducts a large number of comparative experiments on synthetic and real datasets as well as experiments on real datasets using no-reference metrics and the target detection method as the evaluation basis. Experimental results confirm that the proposed method achieves a satisfactory rain removal effect and outperforms the other methods.

Cameras convert optical signals into electronic information to enable image acquisition. However, cameras are susceptible to environmental factors such as light and weather conditions. The information captured by cameras in rainy condition tends to degrade, resulting in reduced contrast and visibility of the images. Deraining aims to restore the image degraded by raindrops and rain accumulation in rainy weather. Most state-of-the-art, end-to-end neural network-based deraining methods have achieved satisfactory results. By considering the features from different scales, we proposed a further improved multiscale information exchange module to extract features from different scales. As a result, the structure achieves a competitive effect on rain mask detection. Moreover, experiments have found that the rain layer-based prediction models tend to leave rain streak residuals in specific image patches. Therefore, we further developed the Feature Attention Compensation module (FAC) to better utilize the deraining image and rain layer information obtained from the multiscale model, which is demonstrated to boost deraining performance. To summarize, we design a coarse-to-fine rain removal model, starting with a rain detection network to get a coarse rain-free image, which is further refined with the FAC module to eventually create a fine rain-free image. We conduct our experiments on both synthetic and real-world datasets. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms the state-of-the-art deraining methods. Source code will be available at https://github.com/Jstar-s/CFMFN

Automatic pedestrian detection is a crucial task for intelligent vehicles as it helps avoid car-to-pedestrian accidents. Recently, several approaches have been proposed to improve detection accuracy, such as the channel-based family, which is based on the aggregation of channel features (e.g., HOG + LUV) to represent the shape of pedestrians. Previous handcrafted-based detectors significantly improved the detection performance by applying different filters to channel features. However, when there are similar-looking background objects, their detection performance suffers due to the lack of texture features. Another drawback of these approaches lies in their high computational cost, limiting their use for real-world applications, especially for resource constrained systems. To mitigate these drawbacks while maintaining a distinguishable detection accuracy, we propose a method that combines a simple but effective texture descriptor based on local binary patterns with HOG + LUV. The concatenated features are used to train a boosted decision trees classifier. To evaluate our approach, we carried out extensive experiments on four well-known datasets, including INRIA, ETH, KITTI, and Caltech. Experimental results show that the proposed method achieves a competitive performance to the channel-based detectors with significantly lower computational cost; it runs at 91.64 and 6.08 times faster than the closest competitors detectors on the INRIA and Caltech datasets, respectively. Furthermore, it outperforms these methods when evaluating under the corrected annotations of the Caltech dataset.

Although the autoregressive image captioning methods yield good-quality image descriptions, their sequential structures slow down the speed of sentence generation processes. With a view to overcome these shortcomings, some nonautoregressive models have been proposed, but the quality of sentences produced by them is lower than those obtained in autoregressive methods. We have designed a new structure based on nonautoregressive methods to not only find better relations between sentence words and image salient objects but also combine this information with some positional information, extracted from the sentence, to generate a more qualified target sentence. The experimental results on the standard benchmark show that our proposed model achieves performance better than general nonautoregressive captioning models.

The number of photosensitive elements in the image sensor determines the resolution of the original image. One promising method for increasing the resolution of an image without increasing hardware cost is image super-resolution. Many existing image super-resolution methods based on deep learning lack feedback mechanisms and deal with feature channels equally, which hinders the expressive ability of deep learning networks and makes it difficult to resolve the dependence between high- and low-resolution (LR) images. An image super-resolution method based on an attention mechanism feedback network was presented in this study. First, the proposed multiscale separable convolution layer is used to extract the characteristics of the input image. Second, a feedback module is designed to process the feedback connections between subnetworks and generate richer high-level representations through densely connected upper and lower projection units, enabling the recovery of high-resolution (HR) images from LR images. Subsequently, an attention mechanism is introduced in the feedback module to redistribute channel attention resources, enhance interdependence between channels, and focus the network on learning high-frequency information. Finally, the reconstructed image was gradually generated using the reconstruction layer through an iterative network. The experimental results for ×2 to ×8 image super-resolution on five standard datasets showed that the quality of the reconstructed image of the proposed method outperforms that of the comparative image super-resolution methods in terms of subjective perception and objective evaluation indices. It can retain more detail for super-resolution and obtain better HR images.

Video action recognition methods based on deep learning can be divided into two types: two-dimensional convolutional networks (2D-ConvNets) relied and three-dimensional convolutional networks (3D-ConvNets) relied. 2D-ConvNets are more efficient to learn spatial features, but cannot capture temporal relationships directly. 3D-ConvNets can jointly learn spatial–temporal features, but their learning is time-consuming because of a large number of networks’ parameters. We therefore propose an effective spatial–temporal interaction (STI) module. The 2D spatial convolution and the one-dimensional temporal convolution are combined through attention mechanism in STI to learn the spatial–temporal information effectively and efficiently. The computation cost of the proposed method is far less than 3D convolution. The proposed STI module can be combined with 2D-ConvNets to obtain the effect of 3D-ConvNets with far fewer parameters, and it can also be inserted into 3D-ConvNets to improve their ability to learn spatial–temporal features, so as to improve the recognition accuracy. Experimental results show that the proposed method outperforms the existing counterparts on benchmark datasets.

To improve the vehicle detection accuracy and solve the problem that small vehicles are difficult to detect, an adaptive multi-scale feature fusion network (AMFFN) is proposed to deal with the multi-scale problem, and better performance is achieved after applying it to you only look once (YOLO) v4. To improve the representation capability of features, spatial pyramid pooling modules were employed on each feature map. The proposed AMFFN fuses features of multiple scales across layers and assigns learnable weights to layers of different scales. To achieve detailed information better, we select dynamic rectified linear unit as the activation function, which can change dynamically with the input. AMFFN can be treated as a reusable module to obtain more refined features by repeatedly fusing features. To avoid the huge amount of parameters caused by the complex network, depthwise separable convolution is used to replace the normal convolution and increase the speed of detection. Experimental results show that the proposed method has higher detection accuracy and faster detection speed, and the performance is better than that of YOLO v5, which is the latest version of the YOLO series of algorithms.

Existing label assignment strategies have achieved promising performance for providing learning targets, typically rely on statistical information and location characteristics. However, classical label assignment strategies usually lack a comprehensive quality evaluation criterion. As a result, the quality of positive samples is not reliable, which weakens the performance of detectors. We propose a dynamic label assignment strategy to provide higher quality positive samples for detection models. Specifically, we propose a quality assessment criteria of candidate samples, which uses a joint representation guided by intersection over union (IoU). Consequently, the supervised information of both branches is included using only one metric. Beside, we propose a partitioning approach to eliminate local redundant sampling, allowing the selected positive sample points to focus more on the overall information of the target. Tests on the COCO dataset show that our work improves the baseline by 2.2% AP without additional modeling and supervisory information. In addition, extensive experiments on the MS COCO test-dev dataset using different backbones demonstrate that our best model outperforms most of the existing representative methods.

Biometrics is widely used in life, and finger-vein recognition also has preliminary applications. Finger-vein recognition methods based on deep learning have achieved state-of-the-art performance, but there are still some challenges, such as sensitivity to finger posture, low recognition efficiency, etc. In our study, a data augmentation strategy using the random sliding window is designed to alleviate the problem caused by changes in finger posture. In addition, to improve the recognition efficiency while ensuring the accuracy, a lightweight finger-vein feature extraction model is proposed based on vision transformer, in which the tokens to token structure is able to describe the global correlation of images at multiple scales. Experimental results on three public databases indicate that compared with other deep-learning methods, the proposed method has achieved higher recognition efficiency and accuracy.

Pavements need frequent maintenance and management, for which one needs detailed information related to the road condition. At present, much of the condition monitoring around the world is done manually in situ or using high-end equipment. This is quite impractical, especially on a national or global scale because it takes time and involves high cost and effort. Pothole recognition using deep-learning-inspired image classification is currently being researched. Convolutional neural networks (CNNs) have drastically enhanced the techniques of image classification. We discuss an improved model based on CNN for pothole recognition. It introduces a neural architecture, namely layer permutation scheme for CNN (LPS-CNN). There is always a trade-off between network complexity and processing time efficiency in neural network architectures. The specific importance of the proposed neural network is that it learns a sparse architecture of the receptive neurons, which turns out to be fruitful in attaining maximum efficiency (for pothole recognition) with a reasonable processing time. The practical applicability of the proposed model is tested on the publicly available DSH 2017 pothole dataset. The proposed LPS-CNN outperforms all other existing architectures for pothole recognition and achieves an exceptional average accuracy of 99.6% on the benchmark pothole dataset.

Restoring the original image from an observed noisy or blurred version is a very important issue in image processing. The variational regularization methods obtained by functional minimizing are popular and widely used for image restoration with promising accuracy, but they may suffer from limited performances due to the presence of the conflict between noise-and-blur-removing and detail-and-texture-preserving. To solve this problem, a hybrid variational regularization model is proposed to effectively reconstruct image textures and details while eliminating noise and blur. The proposed model uses an overlapping group of sparse second-order total variation (TV) as the regularizer to restore image textures and details, whereas the nonconvex TV serves as the other regularizer to preserve image edges and contours. An alternating directional multiplier method algorithm is developed to numerically solve the proposed model, where the majorization–minimization and the iterative reweighted l₁ (IRL1) algorithm are used to solve the overlapping group and the nonconvex minimization subproblem, respectively. The numerical experiments validate the proposed model and algorithm. In addition, compared with several state-of-the-art image restoration models, the proposed model shows the competitive performance in terms of PSNR, SSIM, and NIQE values.

Single-image super-resolution (SISR) studies have achieved superior improvement with the development of convolution neural networks. However, most methods sink into the high computation cost. To tackle this issue, we propose an involution-based lightweight method with contrastive learning for efficient SISR. Unlike the original involution, we set the group number of involution operations to the input feature channels. This setting guarantees the spatial- and channel-specific peculiarity. Moreover, our implemented involution not only learns the weight but also the bias for convolution. Simultaneously, we rethink the kernel generation functions of involution. Instead, we utilize Sigmoid with reparameterized convolution. We additionally apply residual path to involution operation. Furthermore, contrastive learning is adopted during training to learn universal features. Compared with state-of-the-art efficient SISR methods, our proposed methods achieve the best performance with similar or fewer parameters.

The autonomous perception of unmanned surface vessels in the natural sea environment has improved significantly; however, it is adversely affected by haze. Improving the U-Net network can retain and restore spatial information effectively and improve image dehazing. However, many dehazing algorithms that use the U-Net network as the main architecture face problems, such as distorted images and blurred contours of small distant objects. To overcome these limitations, we propose a generative adversarial network (GAN)-U-Net++ network that uses the U-Net++ network with a coordinate attention mechanism as the main part of a GAN generator. We used the coordinate attention mechanism to perform horizontal and vertical pixel level pooling to improve the attention on small targets in hazy images. The addition of the edge preservation loss function and image gradient loss function to small targets improved the sharpness of their edges. A Markov discriminator was used to evaluate the authenticity of the images. Several experiments were conducted to compare the proposed method with existing dehazing algorithms. The structural similarity value and peak signal-to-noise ratio increased by 15.74% and 29.14%, respectively.

With the great development of deep learning, the performance of single image super-resolution (SR) has achieved tremendous progress. As an emerging and promising branch of the SR task, the arbitrary-scale SR task is receiving increasing attention from researchers due to its efficiency and practicality. Although the recent work learning implicit image function opened a solution for arbitrary-scale image SR, its reconstructed images contained structural distortions caused by defective prediction of high-frequency textures. To overcome this problem and further improve the performance of arbitrary-scale image SR, we propose an effective arbitrary-scale SR network, namely, enhanced arbitrary-scale super-resolution, which achieves the arbitrary-scale SR task in a single model by introducing a local-global encoder and enhanced implicit image function. Unlike conventional SR methods, which only stack up convolutional blocks to extract the local feature, the local-global encoder has two branches in parallel, including the local feature branch and the global prior branch. The former effectively extracts the local feature from a low-resolution image and the latter extracts the global prior to assist in the high-resolution image reconstruction. Next, we redesigned an enhanced implicit image function in the form of a dual modulation multiplayer perceptron (MLP) by replacing the implicit image function with a vanilla MLP. Moreover, we introduce the spatial encoding to further reduce structural distortions of reconstructed images. Extensive experiments were conducted to evaluate the performance and demonstrate the superiority of our proposed model.

The multilabel classification method (MCM) is an active and effective research area in image analysis for the detection of various diseases. MCM provides insights and assistance to ophthalmologists in the detection of eye disease at an early stage. We propose comprehensive age-related eye disease detection at an early stage using retinal fundus images that were taken from the online public dataset. The flower pollination optimization algorithm was used for the optimization of the hyperparameters of the deep convolutional neural network (DCNN), which increased the speed and accuracy of the network. Initially, training was performed using the public datasets. The overall improvement in training accuracy (7.5%) was achieved using the optimized method compared with the nonoptimized method. Then, the output of the DCNN was applied to a multiclass support vector machine for the classification of eye diseases. The performance of the proposed method was compared with that of the other optimization techniques with the help of the standard performance measures, namely accuracy, specificity, sensitivity, precision, and F₁-score. Upon comparing the performance of the proposed method with that of the nonoptimized DCNN in terms of performance measures, it had an improvement in the detection accuracy of 9.62% for AMD, 3.57% for glaucoma, 6.59% for normal, 7.71% for DR, and 11.75% for cataract. The obtained results show the effectiveness of the proposed method.

Most existing trackers based on correlation filtering try to introduce different regularization items to improve the learning process of the tracking object. However, the parameters of the corresponding regularization items need to be adjusted dynamically to meet the corresponding tracking process. The regularization parameters are often fixed, so they cannot be updated for more accurate tracking. We propose a method to adaptively adjust the spatiotemporal regularization parameters during the tracking process. We make use of the important correlation between the response map and tracking quality in the tracking process of correlation filtering and introduce the confidence description index of the response map to optimize the spatiotemporal regularization parameters, so the training of correlation filtering can focus on the part with high confidence. In this way, the overfitting situation in training is reduced, and the accuracy and robustness of the tracker are enhanced. In performance evaluation experiments on the OTB2015 and DTB70 datasets, our algorithm outperforms the other comparative algorithms in terms of tracking success rate and distance accuracy, especially in complex scenarios, such as illumination variation, background clutter, out-of-view, and low resolution.

The research on the three-dimensional counting algorithm of suspension cells is very important for the online counting of cells. Suspension cells are usually observed and counted by taking out cell samples under the microscope. This method cannot realize the online counting of cells. At the same time, there will be weak movement of suspended cells in three-dimensional space, making space counting more difficult. Aiming at the above difficulties, a three-dimensional counting algorithm of suspended cells based on deep learning is proposed. The Faster regions with convolutional neural network (R-CNN) framework is used to identify the target cells, the feature pyramid structure is introduced to reduce the feature loss in the fusion process, and the soft non-maximum suppression algorithm is used to remove the bounding box to improve the detection effect. To reduce the manual labeling workload, an automatic labeling method is proposed. The improved Faster RCNN framework achieves the F1-score of more than 98% after identifying cells, and our method is more accurate than other advanced methods. In terms of three-dimensional counting algorithms, the cells are spatially tracked. Cell matching and correction are carried out by extracting multiple feature information of cells, and adhesion cells are extracted separately for priority processing. The algorithm carried out a tracking experiment on the image sequences of five groups of suspended cells, and the final tracking accuracy of the experiment reached more than 95%. The tracking accuracy is better than other advanced algorithms. Experimental results show that the algorithm achieves high-precision three-dimensional tracking and counting of suspended cells.

This article introduces the fastest line detector so far published in the literature. This algorithm is 14.88 times faster than the next fastest line detector, the EDLines, and 94.7 times faster than the line segment detector. It is hundreds of times faster and more reliable than the Standard Hough Transform. The article also introduces an ingeniously simple method for detecting and combining patterns. The algorithm takes advantage of several look-up tables to recognize and fit straight-line patterns. As the first step, it recognizes any possible 4 × 4 pixel line patterns among the binary edge pixels and then uses several small look-up tables to decide whether the connected patterns form a line or not. It is specially designed for real-time processing of the high-resolution images such as the ones used in computer vision applications.

There is a growing concern about public security, especially the discovery of explosives hidden in mobile phones during security checks. However, there is almost no public dataset of explosive mobile phones to study this topic. We contribute the first explosive mobile phone benchmark dataset for security screening, named explosive mobile phones x-ray image dataset, which will be publicly available. Note that explosive mobile phone classification is a typical class imbalance task. Although the number of explosive mobile phones is far smaller than the number of normal mobile phones, one missed phone can cause a huge loss of life and property. To accurately identify explosives hidden in mobile phones, we propose a module called position information attention module (PIAM). Benefiting from aggregating the position information encoded in networks along the channel and spatial domains, PIAM highlights informative features of explosives. In addition, PIAM combines with other networks effectively at a low cost, empowering them with the ability to identify important details. Furthermore, in the face of the class imbalance, we propose a sample-oriented coefficient called sample cost with an update rule. Extensive experimental results show that PIAM and sample cost significantly improve the performance of many excellent networks in explosive mobile phone classification.

Many real-world applications in image processing and computer vision require splitting an input image into a cartoon component and a texture component. We propose a nonconvex variational image decomposition model for simultaneously recovering cartoon and texture images. To induce the sparsity of gradient norms of the cartoon image more strongly than the classical total variation regularization, we applied the nonconvex firm penalty function as a regularizer for the cartoon image. The nonconvex firm penalty regularizer function has a better ability to separate the piecewise constant component with neat edges. The G-norm was used as an oscillating prior for the texture image. Converting the proposed optimization model to a constrained problem by variable splitting, we addressed it with the alternating direction method of multipliers. Experimental results and comparisons were given to verify the superiority of existing state-of-the-art methods in terms of correlation, peak signal-to-noise ratio, structural similarity, and visual quality. Finally, we demonstrated the effectiveness of the proposed model by several applications such as image abstraction and pencil sketching, artifact removal, image denoising, image composition, and detail enhancement.

We present a window-adaptive Gaussian guided filtering method to smooth images while preserving edge features. The key to our algorithm is designing a similarity-aware filtering window based on density clustering to protect the edge structure and introducing a small-scale Gaussian spatial kernel as the input of the Gaussian range kernel to construct a guided filter for image smoothing. Specifically, we first utilized the Gaussian spatial kernel filtering with a small spatial bandwidth to yield a guidance input. Then, the Mahalanobis metric was used to calculate the distance between the center point and the other pixels in the box filtering window, by which we employed the center density clustering algorithm to obtain a better nonbox region (i.e., similarity-aware window) in which each pixel was similar to the center point. Finally, based on the guidance input and the similarity-aware window, the guided Gaussian range filter performed better on image smoothing. Our proposed algorithm is simple and easy to implement. In particular, the window-aware technology effectively improved edge protection, and it can be widely used in image denoising, background smoothing, detexturing, detail enhancement, and edge extraction.

Infrared and visible image fusion is a popular research hotspot in the field of image processing. However, the existing fusion methods still have some limitations, such as insufficient use of intermediate information and inability to focus on features that are meaningful for fusion. To solve these problems, we propose an infrared and visible image fusion method based on generative adversarial networks with dense connection and attention mechanism (DAFuse). Since infrared and visible image are different modalities, we design two branches to extract the features in infrared and visible image, respectively. To make full use of the features extracted from the middle layer and make the model focus on useful information, we introduce the dense block, channel attention mechanism, and spatial attention mechanism into the generator. The self-attention model is incorporated into the discriminator. The proposed method not only retains rich texture detail features and sufficient contrast information but also conforms to human visual perception. Extensive qualitative and quantitative experimental results show that the proposed method has better performance in visual perception and quantitative evaluation than the existing state-of-the-art methods.

With the rapid development of information technology, one object can usually be described by multiple views. Intuitively, the diversity and complementarity of different views provide a more comprehensive data description, which can lead to better performance on multiview clustering tasks. Motivated by the fact that how to fully discover diversity and complementary information across views is the key point to deal with multiview clustering, we propose a multiview learning method, termed as graph-agreement nonnegative matrix factorization (GANMF). GANMF attempts to implement this goal based on matrix factorization technology while exploiting the rich multiview information based on the intra- and interview aspects. Specifically, the graph agreement between representation space and raw space is maximized to preserve the intrinsic geometric property in individual view for the intraview case. Similarly, the graph structures in different views are expected to keep consistent with each other by minimizing the divergence between pairwise views. To this end, the intrinsic information and geometric structure information in the intraview case and complementary and compatibility information in the interview case can be simultaneously formulated into one framework. To solve the proposed GANMF, we correspondingly develop an effective algorithm based on iterative alternating strategy. Extensive experimental results on seven multiview datasets demonstrate the superiority and effectiveness of our proposed method.

Cracks (craquelure) and paint losses are the main types of deterioration of master paintings as they are ageing. We explore the potential of deep-learning-based methods for virtual restoration of paintings focusing on crack detection and their digital inpainting. For the crack detection stage, we develop a model that combines the benefits of multimodal convolutional (MCN) and autoencoder neural networks based on U-Net. The proposed model, dubbed U-Net multimodal convolutional, proves to outperform both MCN and U-Net architectures as well as the benchmark machine learning models for multimodal crack detection, both visually and in terms of objective performance measures. The second stage in our virtual restoration framework, the digital crack inpainting, is an adaptive adversarial network. The obtained virtual restoration results show clear improvement in comparison with the reference methods in this domain. Also, we propose an original way of training an adversarial neural network, which allows us to apply it more successfully in practice. A series of experiments shows encouraging results compared to the current state-of-the-art and confirms the huge potential of deep learning in crack detection and virtual restoration of master paintings.

At present, most scene text recognition methods achieve good performance by training models on many synthetic data. However, many data lead to huge storage space and large amount of calculation. And there is a gap between synthetic and real data. To solve these problems, we use a few real data to train a novel proposed model named spatial attention contrastive network (SAC-Net). The SAC-Net consists of a background suppression network (BSNet), a feature encoder, an attention decoder (ADEer), and a feature contrastive network (FCNet). The BSNet based on U-Net is used to reduce the interference of background. For relatively low prediction accuracy brought by connectionist temporal classification, we design an ADEer to improve performance by using convolutional attention mechanism. Based on data augmentation, we design a FCNet which belongs to contrastive learning. Finally, our SAC-Net is almost equivalent to the state-of-the-art model trained on a few real data for word accuracy on six benchmark test datasets.

Few-shot class-incremental learning (FSCIL) is crucial and practical for artificial intelligence in the real world, which learns novel classes incrementally from few samples without forgetting the previously learned classes. However, FSCIL confronts two significant challenges: “catastrophic forgetting” and “overfitting new.” We focus on convolutional neural network (CNN)-based FSCIL and propose a human cognition-inspired FSCIL method, in which the knowledge of novel classes is learned under the guidance of the previously learned knowledge. Specifically, we learn a discriminative and generalized CNN feature extractor from the base classes in the first task. We generate the representations of base and novel classes in unified feature space without training on novel classes, thus avoiding “forgetting old.” For the novel classes in long sequential tasks, beyond the representation generation, we enhance the representation by exploring the correlations with the previously learned classes to alleviate overfitting new and ensure that the novel classes adapt to the feature space. Experimental results show that our proposed method achieves very competitive results on MiniImageNet and CIFAR-100 datasets.

In recent years, various convolutional neural networks have successfully applied to single-image super-resolution task. However, most existing models with deeper or wider networks require heavy computation and memory consumption that restrict them in practice. To solve the above questions, we propose a lightweight multiscale residual attention network, which not merely can extract more detail to improve the quality of the image but also decrease the usage of the parameters. More specifically, a multiscale residual attention block (MRAB) as the basic unit can fully exploit the image features with different sizes of convolutional kernels. Meanwhile, the attention mechanism can be adaptive to recalibrate channel and spatial information of feature mappings. Furthermore, a local information integration module (LFIM) is designed as the network architecture to maximize the use of local information. The LFIM consists of several MRAB and a local skip connection to complement information loss. Our experimental results show that our method is superior to the representative algorithms in performance with fewer parameters and computational overhead. Code is available at https://github.com/xiaotian3/EMRAB.

Visual ship detection is a crucial component in recognition systems for detecting objects at different scales. However, there remains a challenge in detecting multiscale ships with the influence of various sizes of ships and complex backgrounds. We propose an efficient multilayer attention receptive fusion network (MARN) that is based on the well-established YOLOv4. MARN is a one-stage algorithm that essentially extracts semantic information from feature maps at different scales to highlight salient features of ships, thereby improving the detection performance of multiscale ships. Moreover, a multilayer information interactive fusion module (MIFM) and attention receptive field block (ARFB) are applied and combined reasonably to build a bidirectional fine-grained feature pyramid. Specifically, MIFM is used to fuse features at different scales, which not only concatenates high-level semantic features from deep layers but also reshapes richer features from shallower layers. Meanwhile, ARFB consists of a receptive field block, spatial attention, and channel attention, effectively emphasizing the most important features and suppressing unnecessary ones. Finally, experiments on the Singapore maritime dataset demonstrate that MARN can effectively solve the detection problem of multiscale ships in the complex marine environment.

With the increasing coverage of video surveillance systems in modern society, demand for using artificial intelligence algorithm to replace humans in violent behavior recognition has also become stronger. By moving some channels in the temporal dimension, temporary shift module (TSM) can achieve the performance of three-dimensional convolution neural network (CNN) with the complexity of two-dimensional CNN, and extract the temporal and spatial information at the same time. Our intuition is that too many temporary shift modules may fuse too much action information in each frame, which weakens the capability of CNN on spatiotemporal information extraction. To verify the aforementioned conjecture, we adjusted the network structure based on TSM, proposed partial TSM, selected the optimal model through experiments, and verified the performance of the algorithm on multiple datasets and our expanded datasets. The proposed optimal model not only reduced the memory usage of hardware but also achieved higher accuracy on multiple datasets with 77.3% running time. Meanwhile, we achieved state-of-the-art performance of 91% on RWF-2000 dataset.

Discriminative dimensionality reduction (DDR) makes significant sense in high-dimensional data analysis. Over the decades, research has demonstrated that modeling the intrinsic local structure can well represent the data. Using pairwise neighboring data points to represent the between-class and within-class differences, local structure-based DDR methods are effective. However, how to set the parameters, e.g., the parameter of k-nearest neighbors, is still an open question, which makes the algorithms perform unstably. To address this issue, we propose a nonparametric local structure-based DDR method, named nonparametric locality reforming projection (NLRP). Also, we propose an effective analytical method for effectiveness analysis in both cases of sufficient and insufficient training samples for DDR methods. The experiment results on the AR, Carnegie Mellon University pose, illumination, and expression, and ORL databases are consistent with the theoretical analysis and verified the effectiveness of the proposed NLRP method. Especially, NLRP performs well in the case of very insufficient samples.

Classical UNet with an encoder and decoder structure and its variants perform very well in the field of medical image segmentation. They have a key similarity of a skip-connection, which combines deep, semantic, and coarse-grained feature maps from the decoder subnetwork with shallow, low-level, and fine-grained feature maps from the encoder subnetwork. We noted that, in many cases in medical image segmentation, the boundary of the segmentation target is fuzzy and complex. Traditional UNet cannot accurately segment these details. The main purpose is to solve the fuzzy boundary problem in medical image segmentation. To solve this problem, we combine the advantages of previous models and improve them and propose a new dense edge attention U-type network (DEA-UNet) for medical image segmentation. Starting from the traditional UNet, we modified the concat and skip-connection operations in the latter part. We designed an edge guidance module that fused the features of all layers. Starting from the upsample at the deepest layer, the reverse attention module was used step by step to extract features from high to low, and the edge guidance module was combined with it, so each layer could fully extract boundary details that were difficult to be noticed by previous models, thus solving the problem of the fuzzy boundary of the lesion region. We conducted experiments on two kinds of medical datasets (chest CT and colonoscopic polyp) and compared them with the traditional network. The experimental results showed that our DEA-UNet performed better in multiple indicators. In the segmentation of coronavirus disease-19 images, the results indicate that DEA-UNet has a Dice of 74.6%, sensitivity (Sen) of 70.8%, specificity (Spe) of 96.7%, structural measure (S_α) of 0.766%, enhanced-alignment measure (E_ϕ) of 0.910%, and mean absolute error (MAE) of 0.062%. Our DEA-UNET is 31%, 16%, 3%, and 0.7 and higher than the traditional medical segmentation model UNet, UNet++, the last model Few-shot UNet, and Inf-Net in Dice. In the segmentation of colonoscopic polyp dataset Kvasir, the results indicate that DEA-UNet has a Dice of 95%, structural measure (S_α) of 0.953%, enhanced-alignment measure (E_ϕ) of 0.974%, and MAE of 0.015%. Our DEA-UNet is 13%, 13%, 23%, and 5% higher than the traditional medical segmentation model UNet, UNet++, the last model SFA, and PraNet in Dice. In other evaluation metrics, our DEA-UNet also performed better. When designing DEA-UNet, we also consider the balance between model size and prediction accuracy. Experiments show that, by proper pruning, we can greatly reduce the number of model parameters while maintaining the accuracy of prediction results with little change. This proves that our DEA-UNET has great potential in the field of medical image segmentation.

Object detection based on computer vision is becoming popular in drone-captured images. However, real-time object detection in unmanned aerial vehicle (UAV) scenarios is a huge challenge for low-end devices. To deal with the problem, we have improved YOLOv3-tiny in the following aspects. First, the label rewriting problem, which is caused by network structure and dataset of YOLOv3-tiny in drone-captured images detection, is very serious. The method of increasing the size of the predicted feature map is used to reduce the ratio of label rewriting. Second, the features of small targets will be reduced in a small feature map, but the context information with large receptive fields in it can improve the performance of small target detection. So we use dilated convolution to expand the receptive field without reducing the size of the feature map. Third, multiscale feature fusion is very helpful for small target detection. The multidilated module is adopted to merge features in earlier layer and deeper layers. Finally, a pretraining strategy combining copy-paste data augmentation method is proposed to learn more features from categories with a small number of samples. We evaluated our model on the VisDrone2019-Det test set. It achieves compelling results compared to the counterparts of YOLOv3-tiny, including ∼86.1 % decline in model size, increasing ∼19.2 % AP50. Although our model is slower than YOLOv3-tiny, it is 2.96 times faster than YOLOv3. The results of experiments verify that our network is more effective than YOLOv3-tiny. It is more suitable for UAV object detection applications on low-end devices.

The huge scale variation of objects in scene images limits the performance of deep neural networks based on single-scale image input. To tackle the scale variation issue, we propose an improved multiscale attention network based on enhancing asymmetric convolution and strip pooling (EASNet), which uses three different inference scales. First, complementary weight masks are employed from branches of coarse scale to fine scale, to enhance predictions across scales. Second, the strip pooling module is adopted as strip pooling attention module (SPAM) in the coarser scales to produce the weight masks. It will make maximum use of the coarser scale, improving both the accuracy of boundary segmentation and the integrity of body segmentation for large targets or homogeneous objects but distributed discretely. Besides, enhancing asymmetric convolution block is designed to obtain richer and more robust feature information before SPAM. The effectiveness of EASNet experiments is validated using benchmarks (PASCAL VOC 2012 Aug and Cityscapes). The experimental results demonstrate that our method outperforms or is comparable to state-of-the-art methods by the widely used criteria mean intersection over union.

To achieve a high embedding capacity (EC) without any distortion in the directly decrypted result, a reversible data hiding in encrypted images (RDH-EI) scheme based on full bit-plane compression (FBPC) is proposed. FBPC is designed to vacate as much room as possible before image encryption. To enrich the adjacent redundancy within the most significant bit (MSB) planes, we use flip prediction on the first MSB and then replace the other MSB planes with the XOR result between that plane and its higher level one successively. Hilbert curve scanning is introduced to reduce the dimensionality of the planes to obtain eight highly redundant bit sequences. Huffman coding is then utilizing to compress the bit sequences. After FBPC, a compressed image is obtained. Stream cipher with self-feedback is adapted to encrypt the image. Additional data could be embedded into the reserved space of the encrypted domain. Compared with existing RDH-EI algorithms, the proposed scheme could have a higher EC with no distortion in the directly decrypted image due to the high compression performance of FBPC. Experimental results on the BOWS-2 and BOSSbass datasets demonstrate that the average EC can reach 3.56 and 3.71 bits per pixel, respectively.

Generative adversarial networks (GANs) are utilized for synthetic aperture radar (SAR) image super-resolution reconstruction, affording realistic texture details. However, existing GANs only discriminate the final generated high-resolution (HR) image after two consecutive upsampling processes, which ignore some high-frequency information of the reconstructed images. To resolve this issue, a multiresolution fusion discrimination (MRFD) algorithm is proposed to discriminate the reconstructed feature maps after each upsampling. First, a multiresolution discrimination process discriminates the authenticity of each upsampled feature map separately, which reduces the image distortion imposed during two consecutive upsampling processes. Besides, multiresolution feature fusion further preserves the consistent high-frequency texture structures. Finally, a multiscale dense network extracts image features in different scales, with multiscale dense block’s dense connections improving parameter utilization. The experimental results on a SAR dataset demonstrate that the proposed MRFD algorithm performs better in reconstructing the texture details of HR images.

To further improve the performance of image retrieval, we propose an image two-classification algorithm based on continuous-time quantum walk (CTQW) model. First, a weighted undirected complete graph is constructed based on a new weighted edge method, that is, the images in the image set are used to represent the nodes on the weighted complete graph, and every two images are connected to represent the edge on the weighted complete image, the weighted value of which represents the similarity between adjacent images. Then, the CTQW is executed to realize the image two-classification based on the quantum walk model of the weighted complete graph, that is, the CTQW is performed from two types of unrelated images, and the two-classification of images is realized that the relevant and unrelated labels are assigned to each image according to the obtained probability distribution two-classification. Finally, the experimental results show that the proposed two-classification algorithm based on CTQW has a good classification effect from the visual effect, and the evaluation indexes of image classification, such as the average precision rate, the recall rate, and the accuracy rate, are all higher than 0.85. In addition, the proposed image two-classification algorithm based on CTQW can be extended to image multiclassification if the quantum walk is performed using multiple irrelevant pictures as the starting point. Similarly, the extended image multiclassification method also has a better classification effect through the experiments and the evaluation indexes of image classification.

Video super-resolution (VSR) is a typical ill-posed problem aiming to increase spatial resolution of a video. To explore interframe information between adjacent frames, optical flow and deformable convolution are commonly used for temporal dependency modeling. However, they lack the ability to use nonlocal spatial information. We propose the dual-branch feature extraction network for VSR. Our method consists of feature extraction, feature fusion and refinement, and frame reconstruction. Specifically, the feature extraction is designed to explore both local and nonlocal information of input frames. For the local feature extraction branch, we use deformable convolutions to eliminate motion between frames. For the nonlocal feature extraction branch, we employ a nonlocal module to gather relevant features. We concatenate these extracted features on channel dimension for progressive feature fusion and refinement. The frame reconstruction part is responsible for increasing spatial resolution and reconstruction. Experimental results demonstrate the effectiveness of the proposed method by extensive quantitative and qualitative evaluations in public benchmark datasets.

Thangka Yidam classification is of great importance for understanding the image content and experiencing the Tibetan living culture. However, few samples, complex composition, and diverse Thangka images give the present methods a challenge for Thangka Yidam classification. To this end, we propose an improved method for Thangka Yidam classification based on DenseNet and SENet, which improves the transmission and reuse of feature information. First, a weighted pooling function is designed in the squeeze operation of SENet to alleviate the loss of feature information. Second, we further embed the improved SENet between dense blocks and transition layers of DenseNet to learn the weight correlation between feature channels. Finally, we adopt the Leaky ReLU function to optimize the network in DenseNet. The experimental results demonstrate that the accuracy of our method on the constructed Thangka dataset is increased by 5.54% to 95.89% compared with DenseNet model, and the accuracy on the Cifar-10 dataset is increased by 2.23% to 92.46%. Moreover, the experiments on Thangka dataset also show the significant improvements of our approach over the state-of-the-art methods, which proves the generalization ability of our method.

With the development of adversarial attacks, the performance of object detection based on deep learning is threatened. When adversarial examples are introduced into the detection task, the detector will suffer from poor detection performance, causing a large number of false detections. To handle this problem, we propose a defense method by combing bilateral filtering and the denoising autoencoder. Taking the you only look once (YOLO) v4 detection model as the research target, the proposed method proceeds as follows. First, it performs weighted average in the spatial domain and the pixel-range domain. The method retains important edge texture information when it reduces the perturbations in the image. Then, a three-layer denoising reduction autoencoder is designed, and a new optimization algorithm is proposed to minimize the distance between the input and output. Finally, experiments show that the method proposed has a better defense effect than the existing defense methods. When facing the projected gradient descent-based object detection bounding box disappearance adversarial attack, our defense method can improve the detection true-box rate indicator to 83.04% on the visual object classes challenge (VOC) dataset and 72.20% on the common objects in context (COCO) data. The number of bounding boxes correctly detected is 88.09% and 86.09% of the original one on the PASCAL VOC dataset and the Microsoft COCO dataset, respectively.

In robust video steganography, a message is embedded into a video such that video distortions are avoided while producing a stego video of imperceptible difference from the cover video. Traditional techniques achieved robustness against particular distortions but are complicated in computation and design, and rely on different compression standards. Nowadays, deep-learning-based methods can achieve impressive visual quality and robustness to attacks. We propose a framework with a channel-space attention mechanism for robust video steganography. The framework is composed of depthwise separable convolution layers that can learn channel-space segments for embedding and extraction. The secret messages are distributed across channel-space scales to increase imperceptibility and robustness to distortions. This end-to-end solution is trained with the 3-player game approach to conducting robust steganography, where three networks compete. Two of these handle embedding and extraction operations, while the third network simulates attacks and detection from a steganalyst as an adversarial network. Comparative results versus recent research show that our method is more robust against compression and video distortion attacks. Peak signal-to-noise ratio and structural similarity index were used for evaluating visual quality and demonstrate the imperceptibility of our method.

Even in modern imaging equipment, different types of noise make their way into the images especially under bad lighting conditions and rough communication channels. While applying denoising filters, the preservation of the edge in images has always been an essential albeit a difficult task. This is particularly important for accuracy of further image processing tasks, such as recognition. We described a denoising approach to process noisy images specific for the task of classification with minimal impact on image resolution using low-complexity operations. Specifically, support vector machine-based machine learning classifiers have been proposed to dynamically select an appropriate filtering kernel while processing individual pixels in the input image. The selection is based on the image features in a local window. The provided experimental results on standard dataset prove that the proposed scheme performs as well as contemporary complex convolutional neural network (CNN)-based approaches for the task of denoising images before classification in the presence of a variety of mixed-type noise. Moreover, the proposed filtering approach can run at much higher speeds than their CNN-based counterparts while delivering similar or better performance. The proposed framework has been made publicly available as open-source code to facilitate further work by researchers and practitioners interested in this area.

We propose a recaptured attack-resilient blind watermarking scheme. The work aims to recover the watermark information from the stego (watermarked image) reacquired through a digital liquid-crystal display monitor. In the age of digital globalization, problems like copyright and integrity infringement of digital content are growing exponentially. The content owner can prove their ownership by extracting the embedded watermark from the digitally breached data. The presented framework is robust to several geometrical and signal processing attacks, including lens distortions, intensity distortion, and moiré distortion caused by recapture attack. The singular values of a watermark image and a unique security key are embedded in the host image’s double-density dual-tree discrete wavelet transform (DDD-DWT) coefficients. However, most singular value decomposition (SVD) based watermarking algorithms are prone to false alarm rates and missed alarm rates, causing a security issue. On that account, the proposed watermarking scheme in this paper introduces a unique security key embedding for authentication at the decoder side. The experimental results show that the proposed framework provides high perceptibility with a WPSNR value >59 dB and robustness against various attacks and security.

Video denoising is an elementary but critical task in computer vision and has been widely studied in recent years. However, the existing denoising methods have inevitable drawbacks: some need to predefine rank, some ignore the local information, and most cannot deal with higher-order data. To overcome these shortcomings, we consider two high-order tensor low-rank approximation methods, aiming to achieve color video denoising in a mixed noise environment. First, we establish a high-order tensor framework. Based on this framework, high-order tensor robust principal component analysis (HRPCA) is proposed. Although HRPCA is capable of processing high-order data, there is still a loss of recovery details. Then, we develop another method called high-order tensor low-rank approximation with total variation regularization (HTV). In particular, the TV consists of frontal total variation (FTV) and global total variation (GTV), thus extending the HTV into HFTV and HGTV, respectively. Extensive experimental results of color videos show that the HRPCA and HTV are more efficient in dealing with denoising problems than other state-of-the-art methods.

Watermarking technology based on high-efficiency video coding (HEVC) is evolving such that it can be used in various scenarios, such as copyright protection, ownership certification, tamper detection, and authentication. However, video watermarking algorithms based on HEVC still have poor resistance to recompression attack, which seriously affects the practicability of watermarking algorithms. To solve this limitation, we propose a robust HEVC watermarking algorithm based on the analysis of the characteristics of HEVC recompression. The influencing factors of choosing embedding carrier are discussed, and the calculation formula of texture complexity is then deduced. According to the texture complexity, the watermark is embedded into the stable 4 × 4 luma blocks. Specifically, the watermark information is completely embedded after modulating the signs of the quantized discrete sinusoidal transform coefficients of the carrier. Experimental results show that the algorithm is robust to the recompression attack with constant quantization parameter. There is no obvious artifact after watermark embedding, which reflects the proposed algorithm has good imperceptibility. In addition, the embedding and extraction process of the algorithm is simple and fast, and the embedding of the watermark does not increase the video bit rate.

To understand the visual world, a device knows not only the instances but also how they interact. Humans are at the center of such interactions. Detection of human–object interaction (HOI) is one of the growing research fields in computer vision. However, identifying HOIs due to the large label space of verbs and their interaction with various object types still needs much research. We focus on HOIs in images, which is necessary for a deeper understanding of the scene. In addition to two-dimensional (2D) information, such as the appearance of humans and objects and their spatial location, three-dimensional (3D) status, especially in the configuration of the human body and object as well as their location and spatial, can play an important role in learning HOI. The mapping of 2D to 3D world adds depth information to the problem. These issues led us to collect 3D information along with the 2D features of the images to provide more accurate results. We show 3D attributes, such as face transformation, the viewing angle, the position of an object, and its related location to the human face, can improve HOI learning. The results of experiments on large-scale data show that our method has been able to improve the outcome of interactions.

In the digital age, network transmission of batch images is increasingly frequent. To prevent the information leak of image content and improve the security of encryption during the network transmission, we design an improved chaotic system, establish a dice-rotation model, and propose a multiple-image encryption algorithm. First, k plain images are combined into an image cube; second, the image cube is divided into dice blocks (i.e., small image cube); third, the dice blocks are rotated and scrambled according to the designed rules, which are controlled by chaotic sequences; finally, the forward addition and modular operations in the row and column directions are performed by the improved chaos. Experimental results and algorithm analyses indicate that the proposed algorithm can encrypt multiple interactive images at once with desirable encryption effect and have strong security.

The reconstruction-based coding methods utilize a dictionary to approximate an input feature as a sparse representation in image classification tasks by solving a least-squares problem with constraints. The input feature can be a handcrafted feature calculated over a small patch or a deep feature extracted by a pre-trained convolutional neural network model. As known, the contribution degrees of features extracted within different regions are distinct to the same classification task. Given this, we argue that for features with a large contribution degree, if their reconstruction errors induced by a reconstruction-based coding method can be decreased, more discriminative information will be retained to improve classification accuracy. Considering that different coding methods have the same lower bound of reconstruction error, we propose a visual dictionary refining method that adjusts only the lower bounds of the errors of features according to their contribution degrees. Concretely, given an initial dictionary used by a reconstruction-based coding method, the indexes (coding indexes) of the words from the dictionary for encoding each feature are obtained. Then, the initial dictionary is updated according to the contribution degrees and coding indexes of features. The highlight of our method is that it can be applied to many reconstruction-based coding methods without damaging their original constraints. Experimental results on 15-Scenes, Caltech-101, and UIUC-Sports show that the classification accuracies of four representative coding methods are improved by 0.17% to 1.5% after applying our method. These results exhibit the effectiveness and universality of our method.

To address the problems of error and omission detection in remote sensing image detection caused by the diverse scale changes of remote sensing object scales and the abundant proportion of small-scale objects, as well as the global and dense distribution of remote sensing objects, a remote sensing image detection improvement method based on YOLOv5-S is proposed. First, according to the characteristics of remote sensing objects, the data enhancement strategy is adopted to expand the dataset samples for the characteristics of remote sensing objects to improve the generalization ability of the model. Second, the contextual transformer module is introduced to the backbone feature extraction network and the feature fusion network to ensure the local feature extraction capability while improving the global information acquisition capability of the model, making full use of the input contextual information and guiding the dynamic attention matrix learning to improve the visual representation ability. Third, based on the original model, a shallow detection scale is added, and then a multiscale complex fusion structure is adopted. Meanwhile, the K-means++ algorithm replaces the original K-means algorithm and then clusters 12 anchor box sizes. Fourth, the efficient intersection over union loss is used to improve the accuracy of the remote sensing object recognition prediction. In the experiment on the on two optical remote sensing image datasets, a comparison with several object detection algorithms based on convolutional neural network is made, the results show that the mAP@0.5 tested on the remote sensing datasets is higher than the original YOLOv5-S. Compared with other models, the detection efficiency is higher, and the problems of small-scale object detection in remote sensing image have been significantly improved.

Low-light images suffer from many problems, including low contrast, low brightness, color distortion, blurred details, and noise, which adversely affect the performance of many advanced computer vision tasks. There have been a variety of deep-learning-based methods used to enhance low-light images in recent years. These methods, however, fail to calculate the illumination information and neglect the relationship between multi-scale features and contextual information, which lead to not only poor model generalization but also poor color and details enhancement. To address these concerns, we propose a two-stage low-light image enhancement network called the fractal pyramid network with illumination information (FPN-IL). On the one hand, we use a code network added spatial channel attention mechanism to extract the lighting information in case of uneven exposure and overexposure. On the other hand, we combine the fractal and pyramid networks to construct a new coding method. By having multiple processing paths for information, the FPN-IL is able to make full use of contextual information and interactions of features at different scales. Thus, the image’s details could be abundant. The results demonstrate the advantages of our method compared with other methods, from both qualitative and quantitative perspectives.

Arbitrary-shaped scene text detection is still a challenging task in computer vision, due to its high versatility and complexity. Most methods directly describe text instances by rectangles, quadrangles, or curves, probably leading to insufficient representations. We introduce wavelet descriptor (WD) for arbitrary-shaped text representation that decomposes text contours to wavelet coefficients and extracts feature sequences by discrete wavelet transform (DWT). Based on WD, we propose WD network, which predicts wavelet coefficients then reconstructs text instances accurately and smoothly via inverse DWT and nonmaximum suppression. We further study the restoration capability of various wavelet functions for text contour, including sym5, bior3.1, bior4.4, coif3, db5, dmey, and rbio2.8. To evaluate the proposed method, we conduct experiments on available public datasets. The F-measure of our method achieves 85.8% on challenging curved dataset CTW1500 and higher than 85% on Total-Text and ICDAR2015, demonstrating the excellent performance of our method.

In visual tracking, sometimes the target response value is high, but it is not the tracking result, which can result in the wrong judgment. Moreover, the threshold to decide the tracking result needs to be set artificially in the traditional discriminative methods. We propose a deep learning-based target drift discriminative network to judge whether the target is lost. We design a lightweight network without the threshold, using four convolutional layers, three full connection layers, and the Softmax function to judge the tracking results. When training the network, the established positive and negative samples are used, and we select difficult samples for further training to achieve a better target discriminative effect. Finally, a target drift discriminative network is introduced into the accurate tracking by overlap maximization. When it is judged that the target is lost, another search area is selected to quickly find the target. Numerous experiments show that our method achieves the best performance on datasets UAV123, UAV20L, and VOT2018-LT, especially on the UAV20L dataset, for which the tracking precision and tracking success rate are improved by 3.7% and 2.8%. Compared with several other classical threshold discriminative criteria, we do not need to set the threshold artificially and have better judgment performance.

The noncontact heart rate detection technology is very suitable for daily health monitoring due to its advantages of no contact with the human body, simple operation, and low cost. However, this technique is susceptible to ambient light variation interference and motion artifacts. In addition, most of the existing HR (real-time heart rate monitoring) detection studies based on remote photoplethysmography only focus on illumination variation interference or motion artifacts, and few studies consider both illumination variation interference and motion artifacts. First, a nonparametric skin color segmentation model and support vector machine (SVM) are used in the Cb and Cr components of the YCbCr color space to construct a joint skin color segmentation model specific to the current video sequence, to remove the skin-like background in the video frame in a coarse-to-fine manner. Subsequently, the CamShift algorithm color histogram is constructed within the Cr component and face tracking is performed. It should be noted that the use of SVM to build a skin color segmentation model is based on the following three considerations: first, the skin color segmentation algorithm based on SVM has a relatively fast running speed. Second, considering skin color and background sample information at the same time will help to improve the accuracy of skin color segmentation. Third, SVM can enhance the generalization ability of the learning machine according to the principle of structural risk minimization, which makes the skin color segmentation model based only on video standard frames can also play a better role in the segmentation of subsequent video frames. The experimental results show that the proposed algorithm has higher accuracy for noncontact heart rate detection than the original algorithm; compared with the tracking algorithm in recent years, it has good tracking accuracy and obvious speed advantage.

Social network users face a large number of images; therefore, it is very important that social media finds and recommends images that satisfy the user’s preferences properly. This article investigates the effect of using image style on the performance of a social image recommender system. The effect of distinctive visual styles is evident in art, photography, film, and advertising; however, to the best of our knowledge, it has not been used in social image recommendation systems. We propose image style recognition as a component of user preferences. To do this, we first propose a method based on deep features and compact convolutional transformer to recognize the style of images; then, an image recommender system based on an image style recognition is presented. We consider 15 distinct styles from the Flickr Style dataset, consisting of five categories: atmosphere, mood, genre, composition styles, and color. The experimental results show that style has a positive effect and can significantly improve Recall@k and Precision@k by ∼5 % − 10 % (for the 100 top images recommended) for personalized image recommendation compared with not using the.

With the increase in the crime rate, it becomes difficult for police officials to catch criminals. Face recognition is a technique that helps police officials to catch suspects easily with the help of CCTV footage. The constant checking of surveillance video is a tiresome approach that involves substantial visual consideration and is not mentally captivating, making it increasingly inclined to errors. Therefore, the authors proposed an automated criminal recognition system in the present paper. The present paper comprised three phases; the initial phase investigated the performance of various existing face detection algorithms. The second phase proposed a rectified fine-tuned DenseNet169 model for face recognition. In the third phase, the authors proposed an automated criminal recognition system in which the identification of crime-prone areas is made, based on the data collected where most of the criminals are identified. The proposed rectified model outperformed 1.02% to DeepID, 1% to deep convolutional neural network (CNN), and 1.05% to self-learning CNN on the LFW dataset, 1.09% to VGGFace2, 1.13% to SphereFace, and 1.03% to lightweight CNN on the CPLFW dataset, and achieved 88.7% recognition accuracy on a self-created dataset by meticulously selecting the hyperparameters and customizing the layers of the model.

Datasets play a crucial role in the development of facial expression recognition (FER), but most of the datasets suffer from obvious uncertainties and biases caused by different cultures and collection conditions. To look deeper into these issues, this paper first conducts two sets of experiments, face detection, and facial expression (FE) classification. They are based on three datasets (CK+, FER2013, and RAF–DB), which are collected from lab and wild environments. This paper proposes a network, depthwise separable convolutional neural network (CNN) with an embedded attention mechanism (DSA–CNN) for expression recognition. First, at the preprocessing stage, we obtain the maximum expression range clipping, which is calculated from 81 facial landmark points to filter nonface interferences. Then, we use DSA–CNN, which is based on a coordinate squeeze-and-excitation (CSE) attention for feature extraction. Finally, to further deal with imbalanced class biases and uncertainties issues, this paper proposes a class-weighted cross-entropy loss (CCE-loss) to alleviate the imbalance among seven emotional classes. Then, we combine CCE-loss with ranking regularization loss (RR-loss) and self-importance weighting cross-entropy loss (SCE-loss) at label amend stage, to jointly guide the training of the network. Extensive experiments on three FER datasets demonstrate that our proposed method outperforms most of the state-of-the-art methods eventually.