Although the applications described in Section 3.1 are interesting, they are limited to complex- valued images, whereas the majority of images of interest are real-valued. Beside these uses in complex-valued imaging, CVCNNs were rarely used for real-valued image recognition. As shown in Section 3.1, CVCNNs have better performance than their real-valued counterparts (RVCNNs) on real-valued image classification.
It is possible, however, to obtain a unique complex-valued representation of any real-valued image. This representation is given by the Discrete Fourier Transform (DFT) of an image.
Based on this bijective correspondence between real-valued images and their complex-valued DFTs, it can be deduced that the problem of classifying real-valued images is equivalent with the problem of classifying their complex-valued DFTs.
Taking all the above observations into account, this section introduces complex-valued Fourier transform-based image classification using CVCNNs. The presentation follows the author’s paper [175].
3.2.1 The Fourier transform
The French mathematician Jean-Baptiste Joseph Fourier proved that any periodic function can be expressed as a sum of sines and cosines of different frequencies, each multiplied by different coefficients, which is called the Fourier series. Based on it, the Fourier transform decomposes any function into its different frequencies. In the domain of image processing, the function represents a digital image, and thus we are concerned only with the Discrete Fourier Transform (DFT). The DFT has a wide range of applications in image processing, such as image analysis, image filtering, image denoising and reconstruction, and image compression. Our presentation of the main properties of the DFT mainly follows that of [63].
For anM ×N image denoted byf(x, y), with spatial coordinates x = 0,1,2, . . . , M −1 andy= 0,1,2, . . . , N −1, the Discrete Fourier Transform (DFT) is defined as
F(u, v) =
M−1
X
x=0 N−1
X
y=0
f(x, y)e−ı2π(ux/M+vy/N),
whereı2 =−1is the complex imaginary unit,u= 0,1,2, . . . , M−1andv = 0,1,2, . . . , N−1.
One of the most important properties of the Fourier transform is that it is a bijective function on the space of images, which means that we can define the Inverse Discrete Fourier Transform
(IDFT) ofF(u, v)by:
f(x, y) = 1 M N
M−1
X
u=0 N−1
X
v=0
F(u, v)eı2π(ux/M+vy/N),
forx= 0,1,2, . . . , M −1andy= 0,1,2, . . . , N −1. This means that each image has one and only one Fourier transform, and the Fourier transform of an image can be used to retrieve the original image. The two form a discrete Fourier transform pair, and we will write
f(x, y)↔F(u, v).
As it can be easily seen, in general, the Fourier transform of a real-valued image is complex- valued because of the Euler formula: eıx = cosx+ısinx. Thus, we can write
F(u, v) =R(u, v) +ıI(u, v) =|F(u, v)|eıφ(u,v), where
|F(u, v)|=p
R2(u, v) +ıI2(u, v), is the Fourier (or frequency) spectrum and
φ(u, v) = arctan
I(u, v) R(u, v)
,
represents the phase angle, foru= 0,1,2, . . . , M −1andv = 0,1,2, . . . , N −1.
The DFT of an image satisfies the following properties:
f(x, y)eı2π(u0x/M+v0y/N) ↔F(u−u0, v−v0), and
f(x−x0, y−y0)↔F(u, v)e−ı2π(ux0/M+vy0/N).
Based on these properties, the Fourier transform of an image can be shifted so thatF(0,0)is at the point(u0, v0) = (M/2, N/2), using the transformation
f(x, y)(−1)x+y ↔F(u−M/2, v−N/2).
Because of the periodicity of the DFT, which states that
F(u, v) =F(u+k1M, v+k2N),
for any integersk1,k2, the transformF(u−M/2, v−N/2)contains the exact information that the originalF(u, v)contains, but organized in a different way.
Taking all of the above properties of the Fourier transform into account, the problem of clas- sifying real-valued images is equivalent with the problem of classifying their complex-valued Fourier transforms. Because the Fourier transform offers a richer representation of an image, it can constitute the basis for classification algorithms that could potentially work better than their real-valued counterparts on the original images. This means that CVCNNs can be used to perform Fourier transform-based image classification of real-valued images.
3.2.2 Experimental results
All the experiments were done using the same network architectures for the real-valued convolu- tional neural networks (RVCNNs), complex-valued convolutional neural networks (CVCNNs) trained on the real-valued images (for which the imaginary part of the input was set to zero), and complex-valued convolutional networks trained on the complex-valued Discrete Fourier Transform (DFT) of the real-valued images.
The architecture consists of two (or three) convolutional layers, each followed by an aver- age pooling layer. The number of input channels of the first convolutional layer was 1or 3, depending on whether the input images were grayscale or RGB images. The number of input and output channels of the rest of the convolutional layers was 64 for the RVCNNs. For the CVCNNs trained on the real-valued images, this number was45, which is approximately1.41 times smaller than64, thus assuring the same number of real parameters for the two networks.
Because the CVCNNs trained on Fourier-transformed images have to deal with fully complex- valued inputs, the number of input and output channels for the convolutional layers was also 64, to ensure a fair comparison with the other networks. Thus, they have twice the number of real parameters, but also twice the number of real inputs. The average pooling was done over neighborhoods of2×2pixels.
The last average pooling layer was followed by two fully connected layers, which had 6 times, and3times, respectively, more units than the output channels of the convolutional layers (384, and192, respectively, for RVCNNs and CVCNNs trained on Fourier-transformed images, and 270, and 135, respectively, for CVCNNs trained on real-valued images). The rest of the details of the CVCNNs are the ones mentioned in Section 3.1, which were similarly applied to the RVCNNs.
Training was done using stochastic gradient descent for200epochs, with the learning rate starting from 0.01, which was then divided by 10 two times, first after 100 epochs, and then after150 epochs. One important aspect is that the DFTs of the input images were also shifted so that F(0,0)is at the center of the image, as mentioned in Section 3.2.1. This is usually a standard procedure in the image processing domain, and preliminary experiments have shown that training on the unshifted images has worse results than training on shifted ones.
3.2.2.1 MNIST
The full training set of 60,000 was used for training the three types of networks described above, with no augmentations.
The experimental results are given in Table 3.3. The first column represents the dimensions of the convolution kernels for the first two (or three) convolutional layers. The second column represents the errors measured on the full10,000image test set for the CVCNNs trained on the Discrete Fourier Transform (DFT) of the original images. Next, the errors on the test set for the CVCNNs trained on the original images are reported, for which the imaginary parts of the network inputs are considered to be zero. Lastly, the errors of the RVCNNs are given.
It can be seen that, although these last two types of networks have approximately the same number of parameters, CVCNNs perform better than RVCNNs. But even better performance than the CVCNNs trained on real-valued images was attained by the CVCNNs trained on the Fourier-transformed images. This shows that, in order to fully harness the power of complex- valued neural networks, they must be used with complex-valued inputs.
Table 3.3: Experimental results for MNIST Kernel sizes Complex DFT Complex Real
3,2 0.69% 0.89% 0.98%
3,4 0.63% 0.73% 0.79%
5,3 0.56% 0.78% 0.96%
5,5 0.55% 0.82% 0.88%
5,3,2 0.54% 0.65% 0.87%
3.2.2.2 SVHN
The Street View House Numbers (SVHN) dataset was created by [124], and consists of RGB images of digits cropped from house numbers in Google Street View images. The dataset con- tains73,257images for training,26,032images for testing, and531,131additional, somewhat less difficult samples, for use as extra training data. We only use the standard training and test- ing images, with no data augmentation. The size of each sample in the dataset is32×32pixels.
It is a more difficult task than MNIST, because the images are RGB, and may also contain distracting digits to the sides of the digit of interest.
The three types of networks described above were trained on this dataset, and the errors on the test set are reported in Table 3.4. The structure of the table is the same as the one in the previous experiment: first the kernel sizes of the convolutional layers are given, then the errors attained by the CVCNNs trained on the Fourier-transformed images, by the CVCNNs trained on the original images, and by the RVCNNs, respectively, are shown. The Fourier transform of an RGB image is computed separately for the three RGB channels, thus providing three complex-valued inputs for the CVCNNs.
The errors show that CVCNNs trained on the original images obtain better results than the RVCNNs trained on the same inputs. But the best performance was attained by the CVCNNs trained on the Fourier-transformed images.
Table 3.4: Experimental results for SVHN Kernel sizes Complex DFT Complex Real
3,2 8.66% 10.97% 12.17%
3,4 8.42% 10.61% 11.21%
5,3 7.94% 9.40% 10.45%
5,5 8.16% 9.10% 9.91%
5,3,3 8.69% 8.81% 9.27%
3.2.2.3 CIFAR-10
Training was done on the full 50,000 image training set, which were only randomly flipped before being given to the networks. No other types of data augmentation were used.
The results of training the three types of networks are given in the same format as above, in Table 3.5. The errors are reported for the full10,000image test set. The conclusion of the experiment is consistent with the ones of the above experiments: the CVCNNs perform better than their real-valued counterparts, but the CVCNNs trained on the Fourier-transformed images perform best.
Table 3.5: Experimental results for CIFAR-10 Kernel sizes Complex DFT Complex Real
3,2 22.26% 24.81% 27.13%
3,4 21.51% 24.86% 24.97%
5,3 21.77% 25.64% 25.99%
5,5 21.25% 25.24% 25.68%
5,3,3 21.61% 21.88% 22.32%