2.1.

Multispectral Snapshot Camera

Nowadays, the miniaturized multispectral snapshot cameras are mainly based on the principle of an on-chip multispectral filter array²¹ (MSFA). Generally, MSFAs are available with up to 25 spectral bands in the visible and near infrared (NIR) spectral range. They are composed of multiple mosaic filter elements, each of whose subelement corresponds to a special spectral band. The MSFA is pixel-synchronously mounted in front of a monochromatic image sensor, as demonstrated in Fig. 2(a). As usual, a demosaicing algorithm is necessary to recovery the missing spectral components at the individual pixels because each sensor pixel captures only one spectral component.

Fig. 2

Multispectral snapshot camera: (a) schematic of an MSFA-based multispectral camera and (b) spectral responses of the Silios multispectral NIR camera.²²

Figure 2(b) shows the spectral responses of the Silios multispectral cameras used in this work.²² This MSFA is composed of one panchromatic neutral band and eight spectral bands in the red to NIR spectral range. Their central wavelengths lie between 650 and 930 nm, and they have a full-width at half-maximum (FWHM) of about 40 nm. Hence, the spectral characteristics of the projector should be designed according to this MSFA.

Supposing a linear transfer function of the image sensor, the spectral response values $I$ of the ideal digital camera can be formulated by

2.2.

Multiwavelength Array Projector with Aperiodic Sinusoidal Fringe Patterns

For the realization of a simultaneous multiwavelength projection, the multiaperture principle¹⁴ is applied. In the implementation of this principle, it is unavoidable that some projection units are in the state of off-axis projection. Thus the use of pseudostatistical patterns is advantageous for the projection optics design and the alignment of single-projection units, because with these patterns the dispersion at the projector lens has no influence on 3-D results, and it is not necessary to control the patterns’ characteristics precisely, as well as their relation between each other. In this work, the aperiodic sinusoidal fringe patterns²³ are used by the reason of simpler pattern fabrication than the commonly used speckle patterns. Experimental investigations²⁴ have shown that reasonable 3-D measurement results can be obtained with the use of $N\ge 6$ vertical patterns. With the adaption that these patterns are spectrally coded instead of being temporally projected, an array projector is developed that consists of $N$ projection units to project $N$ different spectral-coded fringe patterns simultaneously

In the case of aperiodic patterns, the offset $a_{\lambda }(x)$ and amplitude $b_{\lambda }(x)$ are the properties of the spectral light sources in each single projection units, and these light sources should be adjusted to the same brightness level. The period length $2\pi /c_{\lambda }(x)$ and phase shift $d_{\lambda }(x)$ of each pattern are generated using the random method in Ref. 23. First, $N$ spatial-frequency spectra are generated with a pseudorandom number generator. Then a bandpass filter is applied to these spectra in order to control the maximal and minimal half period lengths of the corresponding intensity profiles. In the middle working distance, the maximal and minimal half period lengths of projected fringes observed by cameras should be 20 and 10 pixels, respectively. Finally, these filtered spectra are transformed backward into the spatial domain to generate aperiodic intensity profiles. Figure 3 shows the intensity profiles at different wavelengths within the marked horizontal line segment. Nevertheless, the fabrication of such patterns with continuously varying transmission is technically difficult. As a simplification, the intensity profiles in Fig. 3 are binarized so that the patterns can be fabricated as slides with gray codes, and the sinusoidal profile shapes are produced by a minor defocusing of projection.

Fig. 3

Spectral-coded aperiodic sinusoidal fringe patterns ($N=8$).

Figure 4 shows the schematic of the multiwavelength array projector. Figure 4(a) shows the optical setup of a single projection unit. The concentrator with a gold coating shapes the light emitted by a light-emitting diode (LED) into a homogeneous beam for a slide with an aperiodic fringe pattern. In Fig. 4(b), the arrangement of the projection units is shown. The projector consists of eight pipe projection units with different high-power LEDs (3 to 5 W). Figure 4(c) shows the central wavelengths of the used LEDs with an FWHM of ca. 50 nm. The cameras and single projection units are adjusted for a specified middle working distance with overlapping viewing and illumination fields, whereas the projection units are with some defocusing in order to generate the sinusoidal intensity profiles. The LEDs illuminate permanently, so that all patterns can be extracted from a single multispectral image.

Fig. 4

Optical setup of the multiwavelength array projector: (a) sketch of a single projection unit,¹⁴ (b) $N=8$ multiwavelength array projector, and (c) spectral bands of the multiwavelength array projector.

4.1.

Pattern Extraction and Image Data Denoising

At first, dark signal correction of the multispectral images is performed, and the image resolution is recovered to the original sensor resolution by demosaicing, for which we used a simple bilinear interpolation method. Because of the high spectral crosstalk at some bands due to the irregular shapes of their sensor spectral responses [see Fig. 6(a)], the fringes appear smeared on the images due to the cross-channel mixture of different spectral patterns, as shown in Fig. 7(a). In order to recovery the designed spatial modulation of the patterns, a computational crosstalk compensation is performed. As a result, a virtual multispectral image cube with lower crosstalk is reconstructed by a linear combination of the spectral bands, whose weighting coefficients are determined based on the real spectral sensitivity data of the multispectral image sensors.

Fig. 6

Crosstalk compensation: (a) original spectral response curves, (b) ideal Gaussian spectral response curves, and (c) corrected spectral response curves.

For the crosstalk compensation, a virtual spectral sensitivity matrix ${\mathbf{M}}_{\lambda }^v$ is defined, in which the spectral response curves at each band have narrow Gaussian shapes with the same FWHM values²⁶ [see Fig. 6(b)]. Subsequently, a correction matrix $\mathbf{T}$ containing the weighting coefficients of the original spectral bands is calculated from a linear mapping between the original spectral sensitivity matrix ${\mathbf{M}}_{\lambda }$ and the ideal virtual spectral sensitivity matrix ${\mathbf{M}}_{\lambda }^v$. This can be solved by minimizing the cost function $f$

As a verification of the obtained correction matrix $\mathbf{T}$, it is applied to the original spectral response curves ${\mathbf{M}}_{\lambda }$ to calculate the corrected spectral response curves that are shown in Fig. 6(c). In our experiment, the corrected spectral response curves exhibit a mean correlation coefficient of 0.989 to the ideal Gaussian spectral response curves ${\mathbf{M}}_{\lambda }^v$, verifying the used linear method. Using the correction matrix $\mathbf{T}$, the eight raw spectral values ${\mathbf{I}}_{\lambda }^r={[{\mathbf{I}}_1^r,\dots ,{\mathbf{I}}_8^r]}^{\mathbf{T}}$ at a pixel can be converted into virtual spectral data ${\mathbf{I}}_{\lambda }^v={[{\mathbf{I}}_1^v,\dots ,{\mathbf{I}}_8^v]}^{\mathbf{T}}$

By applying this crosstalk compensation at each pixel, the reconstructed virtual image appears as if the sensor had corrected spectral response curves in Fig. 6(c). Figure 7 shows the performance of the crosstalk compensation at the band 890 nm, which suffers from a high spectral crosstalk. It can be seen that the image after crosstalk compensation exhibits a higher fringe contrast and less smearing. Furthermore, the sensor sensitivity difference between the stereo multispectral cameras is reduced by performing crosstalk compensation with respect to the same ideal spectral response curves. The following step is to filter out the ambient light and the high-frequency sensor noise. Therefore, the bandpass filtering method by Guo and Huang²⁷ is implemented. Initially, the Fourier-transformed image is pixel-wise multiplied with a hamming window in order to mitigate the artifacts in the marginal areas of the image. Then, a horizontal bandpass filter can be designed based on the knowledge about the bandwidth range of the fringe patterns and applied to extract the projected patterns.

Fig. 7

Crosstalk compensation at the band 890 nm: (a) raw image and (b) after crosstalk compensation.

4.2.

Stereo Matching and 3-D Reconstruction

After the preprocessing, the projected fringe patterns are isolated from the raw image data. As preparation for the pixel matching, the rectification method by Hartley²⁸ is performed for the stereo cameras. In the stereo rectification, a pair of 2-D projective transformations are determined based on the fundamental matrix and a number of controlling point pairs and applied to the two images in order to match the epipolar lines, whereby the image distortion after the rectification should be minimal at both cameras. After the rectification, the stereo images are line-by-line registered, as shown in Fig. 8.

Fig. 8

Stereo rectification.

Furthermore, the depth range of the 3-D imaging system could provide another restriction for the search of corresponding points.²⁹ As shown in Fig. 9, the ray containing the 3-D object point $P$ in the first camera is restricted by $P_{\mathrm{near}}$ and $P_{\mathrm{far}}$ that correspond to the lower and upper boundaries of the depth range. Thus the search area for the corresponding point $P^{(2)}$ of image point $P^{(1)}$ is restricted within an epipolar segment. By applying the rectification transformations, this epipolar segment is transformed to a horizontal line segment in the rectified image of the second camera. In this way, an acceleration of stereo matching is achieved, and it reduces the possibility of global matching errors that could occur at a low number of statistical patterns.

Fig. 9

Illustration of depth range based search area restriction.

The stereo pixel matching is realized by calculating the normalized cross correlation between the sequence of eight spectral values $I_1^{(1)},\dots ,I_8^{(1)}$ at each pixel in the first camera and the spectral value sequence $I_1^{(2)},\dots ,I_8^{(2)}$ at all pixels of the corresponding line segment in the second camera. The matched points should have the highest correlation coefficient $\rho$

Moreover, a subpixel accuracy up to 1/10 pixel is realized in the pixel matching using a linear interpolation of the spectral values along the same row in all spectral subimages. The difference between the $x$-coordinates of the corresponding point pair is the so-called disparity value. The disparity map is then denoised by median filtering and completed by filling the small gaps with extrapolation. In the end, the disparity values are converted into homogeneous 3-D points via a mapping matrix²⁰ that can be calculated from the calibration data of the stereo vision system.