Compact filter arrays based effective multispectral image demosaicking

Abstract

In recent years, there has been an increase in the applications of multispectral images (MSIs) in different domains. However, the adoption of MSIs in the real world has been constrained primarily because current multispectral cameras are multi-sensor based and are costly and bulky. Similar to use of a color filter array (CFA) in the digital (RGB) camera, the multispectral filter array (MSFA) based single-sensor multispectral cameras can help address these concerns by using the multispectral image demosaicking (MSID) method to estimate the full MSI from the sparsely collected information in the raw image (MSFA image). MSID is challenging as the spatial and spectral correlation properties in the MSIs are quite different from color images. The MSFA image has a very sparse sampling of spectral bands, especially in the higher band MSFA image, weakening the spatial correlation. MSID also becomes challenging due to the weak spectral correlation of any spectral band with distant spectral bands. In this thesis, an attempt is made to investigate multiple aspects of MSID to make progress in this area of research. The role of the MSFAs in consideration to the performance of MSID is investigated, and a direct correlation with the compactness of MSFAs is identified. The compact filter arrays are found to be performing better for all traditional generic MSID methods. Out of different generic MSFA patterns, binary tree based multispectral filter array (BT MSFA) patterns are compact shaped, widely accepted, and extensively used in many single sensor based multispectral imaging systems. However, the maximum of existing MSID methods based on BT MSFA patterns, are restricted to specific band-size MSIs only. This thesis presents the different characteristics of the BT MSFAs and that helps to designs the new convolution filters based on the probability of appearance (PoA) of the spectral band in BT MSFAs and demonstrates that these filters allow us to generalize different MSID methods constrained to specific MSFA patterns to compact BT MSFA patterns. These generic MSID methods based on preferred BT MSFA patterns provide customizing options to the manufacturers to design single-sensor-based multispectral cameras of varying band sizes according to the need of the applications. Therefore, these new filters would be of great benefit to the industry working towards the manufacturing of multispectral cameras. In this thesis, different MSID methods are proposed that aim to better utilize spatial and spectral correlation in the MSFA image and present different effective and generic MSID methods for the preferred BT MSFAs. One of the proposed method uses the spectral correlation between bands and the pseudo panchromatic image estimated from the MSFA image using spatial filters. Another proposed method utilizes adaptive spectral correlation between novel spectrally localized images defined for each and the corresponding bands. Experimental results reveal that our proposed MSID methods outperform other state-of-the-art generic MSID methods in terms of objective and subjective evaluations. A new dataset of the different objects is created to examine the efficacy of the different MSID methods beyond the visible spectral range, and in different lighting conditions, and a new performance evaluation metric is proposed for a faster relative comparison of MSID methods. Lastly, an application related to the semantic segmentation of MSIs, is also used to evaluate the efficacy of the reconstructed MSIs using the different MSID methods compared with the original MSIs captured with the multi-sensor camera.