Artificial Intelligence for Remote Sensing
Theme Keywords： deep learningremote sensingobject detectionremote sensing imageattention mechanismU-Netzonal-stratified perceptionweakly supervisionurbantwo-branch
- The Paper
- Abstract：With the introduction of artificial-intelligence technologies such as deep learning into the field of optical remote-sensing detection, various algorithms have emerged. The use of these algorithms has gradually formed a new paradigm of data-driven optical remote-sensing image object detection. Consequently, high-quality remote-sensing data has become a prerequisite and a necessary resource for researching these paradigm algorithms. highlighting the increasing importance of remote-sensing data. To date, numerous optical remote-sensing image object detection datasets have been published by major research institutions domestically and internationally. These datasets have laid the foundation for the development of deep learning-based remote-sensing image detection tasks. However, no comprehensive summarization and analysis of the published optical remote-sensing image detection datasets have been conducted by scholars. Therefore, this paper aimed to provide a comprehensive review of the published datasets and an overview of algorithm applications. We also aimed to provide a reference for subsequent research in related fields.This paper presents an overview and synthesis of the optical remote-sensing image object detection datasets published between 2008 and 2023. The synthesis is based on an extensive and comprehensive survey of literature in the field. By reviewing and analyzing these datasets, we enable a comprehensive understanding of the progress and trends in optical remote-sensing image object detection dataset research.This paper categorizes the optical remote-sensing image object detection datasets published from 2008 to 2023 based on the annotation method. A comprehensive description of 11 representative datasets is provided, and all dataset information are summarized in tabular form. The analysis considers the information in the datasets themselves and also the spatial and spectral resolution of the images in the datasets. Other basic information including the number of categories, number of images, number of instances, and image-width information are also considered. This analysis effectively demonstrates the trend toward high quality, large scale, and multi-category development of object-detection datasets for optical remote-sensing images. Additionally, we provide an overview of the development and application of algorithms related to published datasets from different perspectives (e.g., horizontal bounding box object detection and rotated bounding box object detection), as well as a subdivision of detection directions (e.g., small object detection and fine-grained detection). Our findings confirm the influential role of remote-sensing data in driving algorithmic advances.In summary, we offer a comprehensive review of optical remote-sensing image object detection datasets from various perspectives. To our best knowledge, this comprehensive review is the first one on such datasets in the field. The work serves as a valuable reference for subsequent research on deep learning-based optical remote-sensing image object detection, providing insights into data availability and research directions. This study is expected to contribute to the advancement of this field by offering a solid foundation for further investigation and innovation.Keywords：deep learning;optical remote sensing imagery;data source;object detection;development of datasets7373|10895|5
<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22 - Abstract：Military aircraft recognition in remote sensing images locates military aircraft in remote sensing images and classify them at a fine-grained level. It plays a vital role in reconnaissance and early warning, intelligence analysis, and other fields. However, the development of military aircraft recognition in remote sensing images is relatively slow due to the lack of publicly available datasets. Therefore, constructing a high-quality and large-scale military aircraft recognition dataset is important.This study constructs a public remote sensing image military aircraft recognition dataset called MAR20 to promote the research progress in this field. The dataset has the following characteristics: (1) MAR20 is currently the largest remote sensing image military aircraft recognition dataset, which includes 3842 images, 20 types, and 22341 instances. Each instance has a horizontal bounding box and also an oriented bounding box. (2) Given that all fine-grained types belong to the aircraft category, different types of aircraft often have similar characteristics, which result in high similarity of different types of targets. (3) Large intra-class differences exist between targets of the same type due to the influence of climate, season, illumination, occlusion, and even the atmospheric scattering in the process of remote sensing imaging.To establish a benchmark for military aircraft recognition in remote sensing images, this paper study evaluates seven commonly used horizontal object recognition methods, namely, Faster R-CNN, RetinaNet, ATSS, FCOS, Cascade R-CNN, TSD, and Double-Head, as well as eight oriented object recognition methods, namely, Faster R-CNN-O, RetinaNet-O, RoI Transformer, Gliding Vertex, Double-Head-O, Oriented R-CNN, FCOS-O, and S2A-Net, on the MAR20 dataset. Through experimental comparisons in the tasks of horizontal object recognition and oriented object recognition, two-stage methods are proven to be more effective in target recognition than one-stage methods.In this study, 3842 high-resolution remote sensing images were collected from 60 military airports around the world through Google Earth, and a large-scale publicly available remote sensing image military aircraft recognition dataset, named MAR20, was established. In terms of data annotation, MAR20 provides two annotation methods, namely, horizontal bounding boxes and oriented bounding boxes, which correspond to the tasks of horizontal target recognition and oriented target recognition. We hope that the MAR20 dataset established in this study could promote the research progress in this field. MAR20 can be downloaded athttps://gcheng-nwpu.github.io/.Keywords：Military aircraft;object recognition;dataset;remote sensing images;Fine-grained recognition19528|13316|15<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：With the ongoing development of artificial intelligence technology, deep learning methods have become increasingly important role in the field of ship detection. However, the false alarms and missed detections that appear in deep learning algorithms hinder the application of technology in the field of ship detection. Although the classical deep learning methods can effectively deal with a single-background sea surface, the classical models can easily yield false alarms on shore when faced with data under complex backgrounds. In custom training, the model often tends to overly emphasize some salient features, which leads to feature overfitting. Detection can easily be missed when these salient features change. In the process of forward propagation of the model to the input, different network layers in the model generate corresponding mappings or feature maps from the input. Fully utilizing the semantic and spatial information of the feature maps is an effective way to reduce false alarms and missed detections. Compared with the traditional model, our proposed Feature Map Reinforcement Network (FMRNnet) can fully utilize feature maps to generate adaptive feature map masks and water–land segmentation masks. This method ultimately reduces false alarms and missed detections by avoiding feature overfitting of the model and weakening the effects caused by complex backgrounds. In FMRNnet, we design the Self Feature-map Mask Module (SFMM), which can selectively utilize the feature map through the attention mechanism for generating an adaptive mask. The mask prevents the model from focusing on a single feature point, which prevents feature overfitting. We also propose a Feature-map Sea-Land Segmentation Module (FSSM) that is parallel to SFMM. It reduces the false alarms of ship targets appearing in the land area by introducing the fusion between the water-land segmentation mask and the feature map. The experimental results, when compared with SOTA algorithms on publicly available datasets, show that the performance of the proposed method in this study is excellent and outperforms that of other SOTA algorithms. After FMRNet is added, the 10-fold average mAP value of the detection algorithm ROI trans largely improves. This enhancement increases the mean value of baseline mAP from 86.1% to 90.8%, which surpasses that of other SOTA algorithms. Benefiting from the adaptive mask, the mAP value of the model including the SFMM module is 90.4%, which achieves a 4.2% improvement over the baseline. Owing to the priori knowledge learned from the water-land distribution, FSSM improves the precision and recall of the model, which results in a MAP value of 86.4%. For the task of ship detection, we propose a novel backbone network, that is, FMRNet based on Resnet. Our proposed SFMM module enables the model to discern the target from multiple features for avoiding the overfitting of salient features. We design the FSSM module to reduce the false alarms caused by complex backgrounds. Suppressing the non-water surface area reduces the confidence level of targets appearing in non-water surface. FSSM achieves the purpose of removing unreasonable false alarms while improving the accuracy of the model.Keywords：remote sensing imaging;artificial intelligence;ship detection;neural networks;feature maps;network reinforcement;rotational object detection;overfitting suppression2518|1615|0<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Nowadays, object detection methods based on deep learning are widely used in the interpretation of remote sensing images. The anchor-based methods usually need to design the anchor boxes first, which requires more detection steps and time cost. This study proposed an object detection method of remote sensing images based on the improved CenterNet. The method can simplify the object detection process and improve efficiency.The CenterNet uses a fully convolutional network to directly predict the heat map of the center points, widths, and heights of the corresponding objects, and the position offsets of the center points. The heat maps are used to generate the rough positions of the objects, and the offsets can fine-tune the positions to make them more accurate. The widths and heights further constitute the shape of the object boxes. The different heat maps decide the object categories. On the basis of CenterNet, the proposed method first adopts the ResNet with transposed convolution as the backbone network. The transposed convolution can expand the output feature maps, and ResNet can reduce the number of parameters in the backbone network compared with the Hourglass network. Second, the proposed method defines the length of Gaussian kernel under three limit conditions between the predicted and real boxes in CenterNet. The Gaussian kernel is applied to generate the heat map label, which is used for network training. Finally, the multi-head attention mechanism is introduced into the backbone network to learn the importance of each element in the feature maps. The weights assigned to the elements reflect their effectiveness, which makes the effective features concentrate in the regions of the object key points as much as possible.The experiments use mean Average Precision (mAP) to evaluate the object detection results on the multiple categories. All the experiments are conducted at the DIOR dataset. The results show that the CenterNet using the ResNet with transposed convolution is 1.4% higher than that using the Hourglass. The proposed calculation of the length of the Gaussian kernel can increase the mAP by 1.1%. The addition of attention mechanism can further improve the mAP by 1.5%. At the same time, the proposed method reduces the time cost by 31.9% compared with the conventional method.The experimental results show that the proposed method can improve detection accuracy without sacrificing the detection speed. The ablation experiments of different parts also show that the ResNet with transposed convolution, the designed calculation method of the length of the Gaussian kernel, and the attention mechanism can effectively improve the mAP. The comparison with other methods also proves that the proposed method is practical.Keywords：remote sensing image;object detection;deep learning;CenterNet;attention mechanism1800|2319|6<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Oriented object detection in remote sensing images is an exceptionally challenging task that has elicited widespread attention. With the rapid advancement of deep learning, neural networks based on convolutional neural networks and self-attention networks (e.g., Transformers) have achieved remarkable progress in oriented object detection. However, the focus on boundary and salient feature information in oriented objects in remote sensing images is lacking. Specifically, extracting boundary information for objects with varying orientations is difficult, and the global dependency of salient features is sparse. To address these issues, we propose a method of small-object detection in remote sensing images on the basis of feature reassembly and self-attention. This method consists of a regression branch that incorporates spatial channel reassembly and a self-attention classification branch. The regression branch reassembles spatial information along the channel dimension and emphasizes boundary-sensitive information to achieve accurate localization of bounding boxes. The classification branch leverages self-attention with positional information to capture fundamentally discriminative object features, thus enhancing global feature dependencies for precise classification. Extensive experiments demonstrate the effectiveness and robustness of the proposed model and showcase its excellent performance on publicly available datasets, such as DOTA, HRSC2016, and SODA-A.Keywords：remote sensing image;small object detection;detection head;feature reorganization;transformer735|2715|5<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Oriented object detection is a basic task in the interpretation of high-resolution remote sensing images. Compared with general detectors, oriented detectors can locate instances with oriented bounding boxes, which are consistent with arbitrary-oriented ground truths in remote sensing images. Currently, oriented object detection has greatly progressed with the development of the convolutional neural network. However, this task is still challenging because of the extreme variation in object scales and arbitrary orientations. Most oriented detectors are evolved from horizontal detectors. They first generate horizontal proposals using the Region Proposal Network (RPN). Then, they classify these proposals into different categories and transform them into oriented bounding boxes. Despite their success, these detectors exploit only the annotations at the end of the network and do not fully utilize the angle and semantic information.This work proposes an Angle-based Region Proposal Network (ARPN), which learns the angle of objects and generates oriented proposals. The structure of ARPN is the same as that of RPN. However, for each proposal, instead of outputting four parameters for regression, ARPN generates five parameters, which are the center (x, y), shape (w, h), and angle (t). In the training, we first assign anchors with ground truths by the Intersection of Unions. Then, we directly supervise the ARPN with the shape and angle information of ground truths. We also propose a semantic branch to output image semantic results for utilizing the advantage of the semantic information. The semantic branch consists of two convolutional layers and is parallel with the detection head. We first assign objects to different scale levels according to their areas. Then, we create semantic labels in each scale and use them to supervise the semantic branch. With the semantic information supervision, the model will learn translation-variant features and improve accuracy. Moreover, the outputs of the semantic branch indicate the objectness in each place, which can filter out false positives of final predictions.We conduct comprehensive experiments on the DOTA dataset to validate the effectiveness of the proposed methods. In the data preparation, we first crop original images into 1024×1024 patches with the stride of 824. Compared with the baseline, the ARPN achieves a 2.2% increase in mAP, while the semantic branch contributes an additional 0.8% improvement in mAP. Finally, we combine both methods and achieve a 74.64% mAP, which is competitive with those obtained by other oriented object detectors. We visualize some results on the DOTA dataset. The results show that our method is highly effective for small objects and densely packed objects.We proposed ARPN and the semantic branch to utilize the multi-information in remote sensing images. The ARPN can directly generate oriented proposals, which can lead to better recall of oriented objects. The semantic branch increases the translation-variant property of the features. Experiments demonstrate the effectiveness of our method, which achieves a 74.64% mAP on the DOTA dataset. In the future works, we will focus on the model efficiency and the inference speed.Keywords：object detection;Oriented object detection;Region proposal generation;multi-information;remote sensing images1416|2555|3<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Sea fog is a common weather phenomenon at sea. It will reduce visibility at sea and greatly threaten maritime traffic and other operations. Traditional sea fog detection algorithms using satellite remote sensing have low accuracy, poor portability, and low automation. Although some existing deep learning-based sea fog monitoring algorithms have been improved, they do not consider the spectral characteristics of sea fog in different channels. The accuracy of sea fog monitoring is also low, especially in edge recognition.A daytime sea fog detection method, which is based on multi-scale feature fusion of generated adversarial network under attention mechanism, is proposed to improve the accuracy of sea fog detection. First, according to the spectral response of sea fog in different imaging channels of meteorological satellite, the satellite cloud images of different imaging channels that can reflect the characteristics of sea fog are selected as the input of the network. Meanwhile, a channel attention mechanism is introduced to calculate the weights of different input channels for prioritizing significant imaging channels within multichannel input. Then, a multi-scale feature fusion mechanism is adopted to fuse the feature maps of different levels of the network for obtaining the multi-scale features of the sea fog. In this way, the problem of losing detailed features in cloud images caused by the pooling operation of the traditional deep network can be solved. Finally, given the difficulty of traditional methods to accurately describe the edge of sea fog, a generation network for sea fog detection supervised by an adversarial network is used to accurately define the edge of sea fog and reduce the false alarm rate.This study takes the Yellow Sea and the Bohai Sea (116.5°—128.25°E，30°—42.5°N) as the research area. Given that March to June each year is the period of high incidence of sea fog in the Yellow Sea and the Bohai Sea, we produce a dataset based on the weather satellite monitoring report of the National Meteorological Center from March to June 2017—2020. After training the model, concerning the quantitative indicators of sea fog detection, our method achieves a probability of detection of 90.5%, a critical success index of 81.28%, and a false positive rate of 10.86%, which are better than those of other methods.The experimental results show that the proposed method can effectively improve the accuracy of sea fog identification, which is important for marine vessel navigation, fishery production, national defense, and military affairs.Keywords：Sea fog monitoring;satellite remote sensing;attention mechanism;Generate adversarial network;Multi-scale feature fusion1970|3564|1<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Hyperspectral anomaly detection is used to identify pixels with significant spectral contrast to their surrounding pixels. It plays a valuable role in military and civilian fields due to the characteristic that the priori spectral information is not required. The existing local contrast-based methods usually adopt dual rectangular window scheme for hyperspectral anomaly detection. However, they empirically set the size of dual window, which limits their generalization capability.A hyperspectral anomaly detection method via combining adaptive window saliency detection and improved superpixel segmentation is proposed in this study to address the abovementioned issue. An adversarial autoencoder is first introduced to reduce the dimension of the hyperspectral image for decreasing the computation complexity of the proposed method. Second, the dimension-reduced hyperspectral image is segmented by improved superpixel segmentation. The existing spectral distance measurements used in the superpixel segmentation are effective when the relationship between the spectral value and the intensity of each pixel is linear. However, this condition cannot be guaranteed in practical applications. The improved superpixel segmentation adopts the orthogonal projection divergence to measure the spectral distance for solving the aforementioned problem. Thereafter, an adaptive window-based saliency detection algorithm is proposed and used to obtain the initial detection results. Specifically, the size of the inner window is adaptively determined by the superpixels, which ensures that the pixels belonging to the same inner window are homogeneous. The outer window can be obtained by enlarging the inner window with fixed size. Finally, the domain transform recursive filter and thresholding operation are employed to optimize the initial detection results for reducing the false alarm rate.The comparisons between the orthogonal projection divergence and three common spectral distance measurements (Euclidean distance, spectral angular mapping, and spectral information divergence) in terms of AUC show that the orthogonal projection divergence-based method achieves the highest score on all five datasets. The comparisons between the adaptive window and traditional manual setting dual window in terms of AUC show that the adaptive window-based method achieves the highest score on all five datasets. Comprehensive comparisons between the proposed method and seven state-of-the-art methods on five public datasets are implemented to validate the overall performance of the proposed method. Specifically, the subjective comparisons show that the anomalous pixels detected by the proposed method are more precise and have stronger contrast to background regions. The objective comparisons demonstrate that the proposed method obtains the highest overall detection accuracy and offers the best separability between the anomalous and background pixels.Three conclusions can be derived from this study. First, the improved superpixel segmentation algorithm can enhance the segmentation results, and the proposed adaptive window scheme can increase the performance of saliency detection. Second, the proposed method has excellent detection accuracy, false alarm rate, and separability between the anomalous and background pixels. Finally, the overall performance of the proposed method is superior to that of state-of-the-art methods.Keywords：anomaly detection;hyperspectral image;orthogonal projection divergence;superpixel segmentation;adaptive window;saliency detection1605|1719|4<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Currently, combining remote sensing imagery with deep learning is a growing trend in individual tree crown detection. RGB image is the most commonly used data type in detection. However, given that the color and texture of the tree crowns are generally close, distinguishing the crowns of different individuals by using only the color and texture information of RGB image in areas with high density of crowns is difficult. In this study, the elevation information is superimposed to improve the accuracy of individual tree crown detection by using RGB images. In the experiment, RGB image (color image) and DSM (digital surface model) were used as data sources, and band combination and double-source detection network model were used to combine RGB and DSM for individual tree crown detection. In the former method, band combination of RGB and DSM was conducted to generate GBD, RGD, and RBD images, and the three kinds of images were used for network training and testing. In the latter method, RGB and DSM were input into the double-source detection network model, and the detection results were obtained. FPN-Faster-R-CNN and Yolov3 were used for experiments in this study. Compared with RGB scheme as the control scheme (which uses only the color and texture information of ground objects for individual tree crown detection), the average accuracy of FPN-Faster-R-CNN in the GBD scheme, RBD scheme, and double-source detection network scheme increased by 3.36%, 2.45%, and 7.77%, respectively; it decreased by 0.17% in the RGD scheme. The average accuracy of Yolov3 in the GBD scheme, RBD scheme, and double-source detection network scheme increased by 0.72%, 0.14%, and 5.71%, respectively; it decreased by 0.98% in the RGD scheme. Under the two networks, the double-source detection network scheme achieved the best detection result in each scheme. Compared with the RGB scheme, the improvement in average accuracy of double-source detection network scheme showed a rising trend with the increase in forest density. Comparative analysis of the experimental results shows that proper combination and utilization of the color, texture, and elevation information of the ground objects is beneficial to improve the performance in the urban individual tree crown detection task based on deep learning.Keywords：remote sensing;individual tree crown detection;deep learning;urban;elevation;color image;UAV2395|4314|1<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
- Abstract：Granular computing with data granulation as the basic is a frontier direction in the field of big data processing, which simulates human thinking and solves large-scale complex problems. It helps improve the accuracy and efficiency of pattern mining and knowledge discovery by means of structure and association. Therefore, incorporating this data analysis method into the process of mining information and discovering knowledge from remote sensing big data needs to be considered.In order to better implement intelligent processing and interpretation analysis of multi-source and multimodal remote sensing big data, and obtain spatiotemporal information that can serve precise applications, this study draws on the data processing thinking of granular computing, and builts a research path that follows the evolution route from visual understanding of external scene to relationship perspective of internal generation mechanism (spectrum analysis). The paper analyzes the granular structure of remote sensing big data and its multi-level and multi-granularity characteristics from three dimensions of space, time, and attribute. We further determine the corresponding granulation strategy based on the characteristics of remote sensing data. In addition, we build a methodology of remote sensing granular computing based on geo-parcels, which integrates the basic models of zonal-stratified perception, spatiotemporal collaborative inversion, and multi-granularity decision making. These models integrate geographical analysis methods, remote sensing mechanism models, and artificial intelligence algorithms. They also mine geographic information or knowledge including morphology, type, index, state, development trend, and mechanism of land geo-parcels.This study focuses on practical research guided by the application needs of precision agriculture. The case study shows that granular computing meets the requirements of intelligent computing of remote sensing big data from multiple perspectives. It is verified that the theory and method proposed in this study can systematically deconstruct and methodically address the multi-level complex problems of agricultural remote sensing. The case study also demonstrates its potential ability to support precise domain applications.This study develops a methodology of remote sensing intelligent computing under the guidance of granular computing. The corresponding problems and solutions in the aspects of space, time, and attribute are also analyzed. Based on the abovementioned work, we are confident that the proposed methodology of intelligent interpretation of remote sensing based on granular computing can effectively address and resolve complex surface cognitive problems in Earth observation through remote sensing.Keywords：remote sensing big data;granular structure/granular computing;geo-parcel;zonal-stratified perception;spatiotemporal collaborative inversion;multi-granularity decision making;precision agriculture application1790|7800|5<HTML><L-PDF><Enhanced-PDF><Meta-XML>Updated：2024-01-22
