Skip to content

satellite-image-deep-learning/techniques

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

Deep learning has revolutionized the analysis and interpretation of satellite and aerial imagery, addressing unique challenges such as vast image sizes and a wide array of object classes. This repository provides an exhaustive overview of deep learning techniques specifically tailored for satellite and aerial image processing. It covers a range of architectures, models, and algorithms suited for key tasks like classification, segmentation, and object detection.

How to use this repository: use Command + F (Mac) or CTRL + F (Windows) to search this page for e.g. 'SAM'

Techniques

Classification


The UC merced dataset is a well known classification dataset.

Classification is a fundamental task in remote sensing data analysis, where the goal is to assign a semantic label to each image, such as 'urban', 'forest', 'agricultural land', etc. The process of assigning labels to an image is known as image-level classification. However, in some cases, a single image might contain multiple different land cover types, such as a forest with a river running through it, or a city with both residential and commercial areas. In these cases, image-level classification becomes more complex and involves assigning multiple labels to a single image. This can be accomplished using a combination of feature extraction and machine learning algorithms to accurately identify the different land cover types. It is important to note that image-level classification should not be confused with pixel-level classification, also known as semantic segmentation. While image-level classification assigns a single label to an entire image, semantic segmentation assigns a label to each individual pixel in an image, resulting in a highly detailed and accurate representation of the land cover types in an image. Read A brief introduction to satellite image classification with neural networks

Segmentation


(left) a satellite image and (right) the semantic classes in the image.

Image segmentation is a crucial step in image analysis and computer vision, with the goal of dividing an image into semantically meaningful segments or regions. The process of image segmentation assigns a class label to each pixel in an image, effectively transforming an image from a 2D grid of pixels into a 2D grid of pixels with assigned class labels. One common application of image segmentation is road or building segmentation, where the goal is to identify and separate roads and buildings from other features within an image. To accomplish this task, single class models are often trained to differentiate between roads and background, or buildings and background. These models are designed to recognize specific features, such as color, texture, and shape, that are characteristic of roads or buildings, and use this information to assign class labels to the pixels in an image. Another common application of image segmentation is land use or crop type classification, where the goal is to identify and map different land cover types within an image. In this case, multi-class models are typically used to recognize and differentiate between multiple classes within an image, such as forests, urban areas, and agricultural land. These models are capable of recognizing complex relationships between different land cover types, allowing for a more comprehensive understanding of the image content. Read A brief introduction to satellite image segmentation with neural networks. Note that many articles which refer to 'hyperspectral land classification' are often actually describing semantic segmentation.

Segmentation - Land use & land cover

  • Automatic Detection of Landfill Using Deep Learning

  • laika -> The goal of this repo is to research potential sources of satellite image data and to implement various algorithms for satellite image segmentation

  • CDL-Segmentation -> Deep Learning Based Land Cover and Crop Type Classification: A Comparative Study. Compares UNet, SegNet & DeepLabv3+

  • LoveDA -> A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation

  • DeepGlobe Land Cover Classification Challenge solution

  • CNN_Enhanced_GCN -> CNN-Enhanced Graph Convolutional Network With Pixel- and Superpixel-Level Feature Fusion for Hyperspectral Image Classification

  • LULCMapping-WV3images-CORINE-DLMethods -> Land Use and Land Cover Mapping Using Deep Learning Based Segmentation Approaches and VHR Worldview-3 Images

  • MCANet -> A joint semantic segmentation framework of optical and SAR images for land use classification. Uses WHU-OPT-SAR-dataset

  • land-cover -> Model Generalization in Deep Learning Applications for Land Cover Mapping

  • generalizablersc -> Cross-dataset Learning for Generalizable Land Use Scene Classification

  • Large-scale-Automatic-Identification-of-Urban-Vacant-Land -> Large-scale automatic identification of urban vacant land using semantic segmentation of high-resolution remote sensing images

  • SSLTransformerRS -> Self-supervised Vision Transformers for Land-cover Segmentation and Classification

  • LULCMapping-WV3images-CORINE-DLMethods -> Land Use and Land Cover Mapping Using Deep Learning Based Segmentation Approaches and VHR Worldview-3 Images

  • DCSA-Net -> Dynamic Convolution Self-Attention Network for Land-Cover Classification in VHR Remote-Sensing Images

  • CHeGCN-CNN_enhanced_Heterogeneous_Graph -> CNN-Enhanced Heterogeneous Graph Convolutional Network: Inferring Land Use from Land Cover with a Case Study of Park Segmentation

  • TCSVT_2022_DGSSC -> DGSSC: A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspectral Imagery

  • DeepForest-Wetland-Paper -> Deep Forest classifier for wetland mapping using the combination of Sentinel-1 and Sentinel-2 data, GIScience & Remote Sensing

  • Wetland_UNet -> UNet models that can delineate wetlands using remote sensing data input including bands from Sentinel-2 LiDAR and geomorphons. By the Conservation Innovation Center of Chesapeake Conservancy and Defenders of Wildlife

  • DPA -> DPA is an unsupervised domain adaptation (UDA) method applied to different satellite images for larg-scale land cover mapping.

  • dynamicworld -> Dynamic World, Near real-time global 10 m land use land cover mapping

  • spada -> Land Cover Segmentation with Sparse Annotations from Sentinel-2 Imagery

  • M3SPADA -> Multi-Sensor Temporal Unsupervised Domain Adaptation for Land Cover Mapping with spatial pseudo labelling and adversarial learning

  • GLNet -> Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images

  • LoveNAS -> LoveNAS: Towards Multi-Scene Land-Cover Mapping via Hierarchical Searching Adaptive Network

  • FLAIR-2 challenge -> Semantic segmentation and domain adaptation challenge proposed by the French National Institute of Geographical and Forest Information (IGN)

  • flair-2 8th place solution

  • igarss-spada -> Dataset and code for the paper Land Cover Segmentation with Sparse Annotations from Sentinel-2 Imagery IGARSS 2023

  • cnn-land-cover-eco -> Multi-stage semantic segmentation of land cover in the Peak District using high-resolution RGB aerial imagery

Segmentation - Vegetation, deforestation, crops & crop boundaries

Note that deforestation detection may be treated as a segmentation task or a change detection task

Segmentation - Water, coastlines, rivers & floods

  • Houston_flooding -> labeling each pixel as either flooded or not using data from Hurricane Harvey. Dataset consisted of pre and post flood images, and a ground truth floodwater mask was created using unsupervised clustering (with DBScan) of image pixels with human cluster verification/adjustment

  • ml4floods -> An ecosystem of data, models and code pipelines to tackle flooding with ML

  • 1st place solution for STAC Overflow: Map Floodwater from Radar Imagery hosted by Microsoft AI for Earth -> combines Unet with Catboostclassifier, taking their maxima, not the average

  • hydra-floods -> an open source Python application for downloading, processing, and delivering surface water maps derived from remote sensing data

  • CoastSat -> tool for mapping coastlines which has an extension CoastSeg using segmentation models

  • deepwatermap -> a deep model that segments water on multispectral images

  • rivamap -> an automated river analysis and mapping engine

  • deep-water -> track changes in water level

  • WatNet -> A deep ConvNet for surface water mapping based on Sentinel-2 image, uses the Earth Surface Water Dataset

  • A-U-Net-for-Flood-Extent-Mapping

  • floatingobjects -> TOWARDS DETECTING FLOATING OBJECTS ON A GLOBAL SCALE WITHLEARNED SPATIAL FEATURES USING SENTINEL 2. Uses U-Net & pytorch

  • SpaceNet8 -> baseline Unet solution to detect flooded roads and buildings

  • dlsim -> Breaking the Limits of Remote Sensing by Simulation and Deep Learning for Flood and Debris Flow Mapping

  • Water-HRNet -> HRNet trained on Sentinel 2

  • semantic segmentation model to identify newly developed or flooded land using NAIP imagery provided by the Chesapeake Conservancy, training on MS Azure

  • BandNet -> Analysis and application of multispectral data for water segmentation using machine learning. Uses Sentinel-2 data

  • mmflood -> MMFlood: A Multimodal Dataset for Flood Delineation From Satellite Imagery (Sentinel 1 SAR)

  • Urban_flooding -> Towards transferable data-driven models to predict urban pluvial flood water depth in Berlin, Germany

  • MECNet -> Rich CNN features for water-body segmentation from very high resolution aerial and satellite imagery

  • SWRNET -> A Deep Learning Approach for Small Surface Water Area Recognition Onboard Satellite

  • elwha-segmentation -> fine-tuning Meta's Segment Anything (SAM) for bird's eye view river pixel segmentation

  • RiverSnap -> code for paper: A Comparative Performance Analysis of Popular Deep Learning Models and Segment Anything Model (SAM) for River Water Segmentation in Close-Range Remote Sensing Imagery

  • SAR-water-segmentation -> Deep Learning based Water Segmentation Using KOMPSAT-5 SAR Images

Segmentation - Fire, smoke & burn areas

Segmentation - Landslides

Segmentation - Glaciers

  • HED-UNet -> a model for simultaneous semantic segmentation and edge detection, examples provided are glacier fronts and building footprints using the Inria Aerial Image Labeling dataset

  • glacier_mapping -> Mapping glaciers in the Hindu Kush Himalaya, Landsat 7 images, Shapefile labels of the glaciers, Unet with dropout

  • GlacierSemanticSegmentation

  • Antarctic-fracture-detection -> uses UNet with the MODIS Mosaic of Antarctica to detect surface fractures

  • sentinel_lakeice -> Lake Ice Detection from Sentinel-1 SAR with Deep Learning

Segmentation - methane

Segmentation - Other environmental

  • Detection of Open Landfills -> uses Sentinel-2 to detect large changes in the Normalized Burn Ratio (NBR)

  • sea_ice_remote_sensing -> Sea Ice Concentration classification

  • EddyNet -> A Deep Neural Network For Pixel-Wise Classification of Oceanic Eddies

  • schisto-vegetation -> Deep Learning Segmentation of Satellite Imagery Identifies Aquatic Vegetation Associated with Snail Intermediate Hosts of Schistosomiasis in Senegal, Africa

  • Earthformer -> Exploring space-time transformers for earth system forecasting

  • weather4cast-2022 -> Unet-3D baseline model for Weather4cast Rain Movie Prediction competition

  • WeatherFusionNet -> Predicting Precipitation from Satellite Data. weather4cast-2022 1st place solution

  • marinedebrisdetector -> Large-scale Detection of Marine Debris in Coastal Areas with Sentinel-2

  • kaggle-identify-contrails-4th -> 4th place Solution, Google Research - Identify Contrails to Reduce Global Warming

  • MineSegSAT -> An automated system to evaluate mining disturbed area extents from Sentinel-2 imagery

  • asos -> Recognizing protected and anthropogenic patterns in landscapes using interpretable machine learning and satellite imagery

  • SinkSAM -> Knowledge-Driven Self-Supervised Sinkhole Segmentation Using Topographic Priors and Segment Anything Model

Segmentation - Roads & sidewalks

Extracting roads is challenging due to the occlusions caused by other objects and the complex traffic environment

Segmentation - Buildings & rooftops

Segmentation - Solar panels

Segmentation - Ships & vessels

Segmentation - Other manmade

  • Aarsh2001/ML_Challenge_NRSC -> Electrical Substation detection

  • electrical_substation_detection

  • MCAN-OilSpillDetection -> Oil Spill Detection with A Multiscale Conditional Adversarial Network under Small Data Training

  • mining-detector -> detection of artisanal gold mines in Sentinel-2 satellite imagery for Amazon Mining Watch. Also covers clandestine airstrips

  • EG-UNet Deep Feature Enhancement Method for Land Cover With Irregular and Sparse Spatial Distribution Features: A Case Study on Open-Pit Mining

  • plastics -> Detecting and Monitoring Plastic Waste Aggregations in Sentinel-2 Imagery

  • MADOS -> Detecting Marine Pollutants and Sea Surface Features with Deep Learning in Sentinel-2 Imagery on the MADOS dataset

  • SADMA -> Residual Attention UNet on MARIDA: Marine Debris Archive is a marine debris-oriented dataset on Sentinel-2 satellite images

  • MAP-Mapper -> Marine Plastic Mapper is a tool for assessing marine macro-plastic density to identify plastic hotspots, underpinned by the MARIDA dataset.

  • substation-seg -> segmenting substations in Sentinel 2 satellite imagery

  • SAMSelect -> An Automated Spectral Index Search for Marine Debris using Segment-Anything (SAM)

Panoptic segmentation

Segmentation - Miscellaneous

Instance segmentation

In instance segmentation, each individual 'instance' of a segmented area is given a unique lable. For detection of very small objects this may a good approach, but it can struggle seperating individual objects that are closely spaced.

  • Mask_RCNN generates bounding boxes and segmentation masks for each instance of an object in the image. It is very commonly used for instance segmentation & object detection

  • Building-Detection-MaskRCNN -> Building detection from the SpaceNet dataset by using Mask RCNN

  • Mask_RCNN-for-Caravans -> detect caravan footprints from OS imagery

  • parking_bays_detectron2 -> Detecting parking bays with satellite imagery. Used Detectron2 and synthetic data with Unreal, superior performance to using Mask RCNN

  • Circle_Finder -> Circular Shapes Detection in Satellite Imagery, 2nd place solution to the Circle Finder Challenge

  • Lawn_maskRCNN -> Detecting lawns from satellite images of properties in the Cedar Rapids area using Mask-R-CNN

  • CropMask_RCNN -> Segmenting center pivot agriculture to monitor crop water use in drylands with Mask R-CNN and Landsat satellite imagery

  • Mask RCNN for Spacenet Off Nadir Building Detection

  • CATNet -> Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images

  • Object-Detection-on-Satellite-Images-using-Mask-R-CNN -> detect ships

  • FactSeg -> Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS), also see FarSeg and FreeNet, implementations of research paper

  • aqua_python -> detecting aquaculture farms using Mask R-CNN

  • RSPrompter -> Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model

  • VMD-Mask-RCNN-pipeline -> Detecting and segmenting sand mining river vessels on the Vietnam Mekong Delta, using PlanetScope imagery and Mask R-CNN

Object detection


Image showing the suitability of rotated bounding boxes in remote sensing.

Object detection in remote sensing involves locating and surrounding objects of interest with bounding boxes. Due to the large size of remote sensing images and the fact that objects may only comprise a few pixels, object detection can be challenging in this context. The imbalance between the area of the objects to be detected and the background, combined with the potential for objects to be easily confused with random features in the background, further complicates the task. Object detection generally performs better on larger objects, but becomes increasingly difficult as the objects become smaller and more densely packed. The accuracy of object detection models can also degrade rapidly as image resolution decreases, which is why it is common to use high resolution imagery, such as 30cm RGB, for object detection in remote sensing. A unique characteristic of aerial images is that objects can be oriented in any direction. To effectively extract measurements of the length and width of an object, it can be crucial to use rotated bounding boxes that align with the orientation of the object. This approach enables more accurate and meaningful analysis of the objects within the image. Image source

Object tracking in videos

  • TCTrack -> Temporal Contexts for Aerial Tracking

  • CFME -> Object Tracking in Satellite Videos by Improved Correlation Filters With Motion Estimations

  • TGraM -> Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

  • satellite_video_mod_groundtruth -> groundtruth on satellite video for evaluating moving object detection algorithm

  • Moving-object-detection-DSFNet -> DSFNet: Dynamic and Static Fusion Network for Moving Object Detection in Satellite Videos

  • HiFT -> Hierarchical Feature Transformer for Aerial Tracking

Object detection with rotated bounding boxes

Orinted bounding boxes (OBB) are polygons representing rotated rectangles. For datasets checkout DOTA & HRSC2016. Start with Yolov8

  • mmrotate -> Rotated Object Detection Benchmark, with pretrained models and function for inferencing on very large images

  • OBBDetection -> an oriented object detection library, which is based on MMdetection

  • rotate-yolov3 -> Rotation object detection implemented with yolov3. Also see yolov3-polygon

  • DRBox -> for detection tasks where the objects are orientated arbitrarily, e.g. vehicles, ships and airplanes

  • s2anet -> Align Deep Features for Oriented Object Detection

  • CFC-Net -> A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote Sensing Images

  • ReDet -> A Rotation-equivariant Detector for Aerial Object Detection

  • BBAVectors-Oriented-Object-Detection -> Oriented Object Detection in Aerial Images with Box Boundary-Aware Vectors

  • CSL_RetinaNet_Tensorflow -> Arbitrary-Oriented Object Detection with Circular Smooth Label

  • r3det-on-mmdetection -> R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object

  • R-DFPN_FPN_Tensorflow -> Rotation Dense Feature Pyramid Networks (Tensorflow)

  • R2CNN_Faster-RCNN_Tensorflow -> Rotational region detection based on Faster-RCNN

  • Rotated-RetinaNet -> implemented in pytorch, it supports the following datasets: DOTA, HRSC2016, ICDAR2013, ICDAR2015, UCAS-AOD, NWPU VHR-10, VOC2007

  • OBBDet_Swin -> The sixth place winning solution in 2021 Gaofen Challenge

  • CG-Net -> Learning Calibrated-Guidance for Object Detection in Aerial Images

  • OrientedRepPoints_DOTA -> Oriented RepPoints + Swin Transformer/ReResNet

  • yolov5_obb -> yolov5 + Oriented Object Detection

  • How to Train YOLOv5 OBB -> YOLOv5 OBB tutorial and YOLOv5 OBB noteboook

  • OHDet_Tensorflow -> can be applied to rotation detection and object heading detection

  • Seodore -> framework maintaining recent updates of mmdetection

  • Rotation-RetinaNet-PyTorch -> oriented detector Rotation-RetinaNet implementation on Optical and SAR ship dataset

  • AIDet -> an open source object detection in aerial image toolbox based on MMDetection

  • rotation-yolov5 -> rotation detection based on yolov5

  • SLRDet -> project based on mmdetection to reimplement RRPN and use the model Faster R-CNN OBB

  • AxisLearning -> Axis Learning for Orientated Objects Detection in Aerial Images

  • Detection_and_Recognition_in_Remote_Sensing_Image -> This work uses PaNet to realize Detection and Recognition in Remote Sensing Image by MXNet

  • DrBox-v2-tensorflow -> tensorflow implementation of DrBox-v2 which is an improved detector with rotatable boxes for target detection in remote sensing images

  • Rotation-EfficientDet-D0 -> A PyTorch Implementation Rotation Detector based EfficientDet Detector, applied to custom rotation vehicle datasets

  • DODet -> Dual alignment for oriented object detection, uses DOTA dataset

  • GF-CSL -> Gaussian Focal Loss: Learning Distribution Polarized Angle Prediction for Rotated Object Detection in Aerial Images

  • Polar-Encodings -> Learning Polar Encodings for Arbitrary-Oriented Ship Detection in SAR Images

  • R-CenterNet -> detector for rotated-object based on CenterNet

  • piou -> Orientated Object Detection; IoU Loss, applied to DOTA dataset

  • DAFNe -> A One-Stage Anchor-Free Approach for Oriented Object Detection

  • AProNet -> Detecting objects with precise orientation from aerial images. Applied to datasets DOTA and HRSC2016

  • UCAS-AOD-benchmark -> A benchmark of UCAS-AOD dataset

  • RotateObjectDetection -> based on Ultralytics/yolov5, with adjustments to enable rotate prediction boxes. Also see PolygonObjectDetection

  • AD-Toolbox -> Aerial Detection Toolbox based on MMDetection and MMRotate, with support for more datasets

  • GGHL -> A General Gaussian Heatmap Label Assignment for Arbitrary-Oriented Object Detection

  • NPMMR-Det -> A Novel Nonlocal-Aware Pyramid and Multiscale Multitask Refinement Detector for Object Detection in Remote Sensing Images

  • AOPG -> Anchor-Free Oriented Proposal Generator for Object Detection

  • SE2-Det -> Semantic-Edge-Supervised Single-Stage Detector for Oriented Object Detection in Remote Sensing Imagery

  • OrientedRepPoints -> Oriented RepPoints for Aerial Object Detection

  • TS-Conv -> Task-wise Sampling Convolutions for Arbitrary-Oriented Object Detection in Aerial Images

  • FCOSR -> A Simple Anchor-free Rotated Detector for Aerial Object Detection. This implement is modified from mmdetection. See also TensorRT_Inference

  • OBB_Detection -> Finalist's solution in the track of Oriented Object Detection in Remote Sensing Images, 2022 Guangdong-Hong Kong-Macao Greater Bay Area International Algorithm Competition

  • sam-mmrotate -> SAM (Segment Anything Model) for generating rotated bounding boxes with MMRotate, which is a comparison method of H2RBox-v2

  • mmrotate-dcfl -> Dynamic Coarse-to-Fine Learning for Oriented Tiny Object Detection

  • h2rbox-mmrotate -> Horizontal Box Annotation is All You Need for Oriented Object Detection

  • Spatial-Transform-Decoupling -> Spatial Transform Decoupling for Oriented Object Detection

  • ARS-DETR -> Aspect Ratio Sensitive Oriented Object Detection with Transformer