CNN联合多尺度Transformer的高光谱与多光谱图像融合

Fusion of Hyperspectral and Multispectral Images Using a CNN Joint Multi-Scale Transformer

  • 摘要: 高光谱图像具有丰富的光谱信息,多光谱图像具有精妙的几何特征,融合高分辨率的多光谱图像和低分辨率的高光谱图像可以获取更为全面的遥感数据图像。然而现有的融合网络大多数基于卷积神经网络所设计,对于结构复杂的遥感类图像而言,依赖于核大小的卷积运算,容易导致特征融合阶段缺乏一些全局上下文信息。为保证图像融合的质量,本文提出了一种CNN(Convolutional Neural Network, CNN)联合多尺度transformer网络来实现多光谱和高光谱图像融合,结合了CNN的特征提取能力与transformer的全局建模优势。网络将融合任务分为了两个阶段,特征提取阶段和融合阶段。特征提取阶段,针对图像特性,基于卷积神经网络分别设计了不同模块用于特征提取。融合阶段,通过多尺度transformer模块从局部到全局建立信息间长距离关联,最后通过多层卷积层将特征映射为高分辨率的高光谱图像。经过在CAVE和Harvard数据集的实验结果表明,本文所提算法与其他经典算法相比,能更好地提升融合图像的质量。

     

    Abstract: Hyperspectral images contain rich spectral information, and multispectral images have exquisite geometric features. More comprehensive remote sensing images can be obtained by merging high-resolution multispectral and low-resolution hyperspectral images. However, most existing fusion networks are based on convolutional neural networks. For remote sensing images with complex structures, convolution operations dependent on the kernel size tend to lead to a lack of global context information in the feature fusion stage. To ensure the quality of image fusion, this study proposes a convolutional neural network (CNN) combined with a multi-scale transformer network to realize multispectral and hyperspectral image fusion, combining the feature extraction capability of the CNN and the global modeling advantage of the transformer. The network divides the fusion task into two stages: feature extraction and fusion. In the feature extraction stage, different modules are designed for feature extraction based on the CNN. In the fusion stage, a multi-scale transformer module is used to establish a long-distance correlation between local and global information, and the features are mapped into high-resolution hyperspectral images through multilayer convolution layers. Experimental results on the CAVE and Harvard datasets show that the proposed algorithm can improve the quality of fused images better than other classical algorithms.

     

/

返回文章
返回