Fusion of Hyperspectral and Multispectral Images Using a CNN Joint Multi-Scale Transformer

Hyperspectral images contain rich spectral information, and multispectral images have exquisite geometric features. More comprehensive remote sensing images can be obtained by merging high-resolution multispectral and low-resolution hyperspectral images. However, most existing fusion networks are based on convolutional neural networks. For remote sensing images with complex structures, convolution operations dependent on the kernel size tend to lead to a lack of global context information in the feature fusion stage. To ensure the quality of image fusion, this study proposes a convolutional neural network (CNN) combined with a multi-scale transformer network to realize multispectral and hyperspectral image fusion, combining the feature extraction capability of the CNN and the global modeling advantage of the transformer. The network divides the fusion task into two stages: feature extraction and fusion. In the feature extraction stage, different modules are designed for feature extraction based on the CNN. In the fusion stage, a multi-scale transformer module is used to establish a long-distance correlation between local and global information, and the features are mapped into high-resolution hyperspectral images through multilayer convolution layers. Experimental results on the CAVE and Harvard datasets show that the proposed algorithm can improve the quality of fused images better than other classical algorithms.