Local Feature Based on Moment Invariants for
Blurred Image Matching
Qiang Tong
1, Terumasa Aoki
1, 21 Graduate School of Information Sciences, Tohoku University，Japan 2 New Industry Creation Hatchery Center, Tohoku University，Japan
Abstract- This paper presents a new local feature scheme for image matching between a strongly blurred image and a non-blurred image. In recent years, a lot of local feature schemes have been proposed to improve the image matching performances. However, as far as the authors know, there are no local features which are robust to strong blur. In this paper, blur moment invariants are introduced into a local feature scheme. These blur moment invariants are robust to strong blur when they are used as global features. However, they cannot be used as a local feature. In this paper, we dig into this problem and clarify the reason why they cannot be used as a local feature. After that, we propose a new local feature scheme based on this study. Experimental results show that the proposed scheme is more effective and suitable for blurred image matching than any other existing local feature schemes.
Keywords –image matching, local feature, moment invariants, de-blurring
I.INTRODUCTION
Image matching is one of the most frequently used processes in computer vision because it can solve a lot of problems. However, it is not still mature and there is room for improving its performance. One of the biggest problems in existing image matching technologies is lack of robustness against strong blur. When a strong blur is included in an input image by wrong focus or camera motion etc., this input image cannot match the reference image (non-blurred image) stored in a database.
To solve this problem, we can take two approaches into account. The first approach is to introduce any de-blurring algorithms such as in [1-3]. In this approach, an input image (blurred image) is deblurred by these algorithms first. And the deblurred image generated by these algorithms is used to match with reference images. Although this is the most straightforward way to solve this problem, most of deblurring algorithms have the following two drawbacks. The first drawback is to require very high computation cost. In general, deblurring algorithms contain iterative process to estimate some parameters correctly. For this reason, they cannot be light-weight processes. The second drawback is to cause another artifact by these algorithms. This artifact may degrade the performance of image matching.
On the other hand, the second approach is to design a new image feature which is robust to a strong blur. In general, Image feature schemes can be classified into two groups – global feature schemes and local feature schemes. In most cases, a local feature scheme shows higher performance than a global feature scheme. It is because a local feature is robust to image changes (rotation, scale, illumination, etc.). However, as far as the authors know, there are no existing local feature schemes which are robust to a strong blur. To the contrary, moment invariants, which is one of the most famous global image features, has the characteristics to be robust to a strong blur. But, unfortunately this feature does not show good performances for image changes unlike local features.
The rest of the paper is organized as follows: In Section II we describe the related work. In Section III we discuss why moment invariants cannot be used as a local feature and what we should solve to overcome this limitation. After that, we propose a novel image feature scheme based on the above discussion. Experimental results and the evaluation of our scheme are described in section IV. Finally, we conclude this paper in section V.
II.RELATED WORK
The pioneer work in blur moment invariants was presented by Flusser and Suk [4], who firstly proposed a blur invariant feature based on moment invariants. After that, they proposed a lot of geometric moment invariants such as affine and blur invariants [5], linear motion blur invariants [6] etc. After their works, a lot of methods based on other moment invariants were proposed. For example, the method based on complex moments for blur, rotation and scale invariants was proposed by Liu and Zhang [7]. And the method based on Zernike moments for blur and rotation invariants was proposed by Zhu et al [8]. However all of the above moment invariants are global features as described in section I and global feature schemes are well known to show worse matching performances than local feature schemes in most cases. It is because a global feature scheme must treat an image as a whole, so that it is not robust to image changes in nature such as a part of image changes.
On the other hand, a local feature scheme is suited to many applications such as wide baseline matching [9], image retrieval [10], object recognition and image classification [11]. The main idea of a local feature scheme is to detect interest points in an image and then to describe the surrounding pixels' information around each key point as a local feature. The advantages of local feature schemes are that they are robust to image changes such as scales, rotation, illumination change, and occlusion, etc. One of the most popular local feature scheme is Lowe's SIFT [12], which is widely accepted as the highest-quality scheme currently available. Another popular method is Bay et al’s SURF [13], which maintains good robustness and at same time can be computed much faster. However, since the majority of existing local features focus on robustness of image changes, there are no local feature schemes to solve the robustness against a strong blur as far as the authors know.
III.LOCAL FEATURE BASED ON MOMENT INVARIANTS
In this section, we propose a new local feature scheme based on moment invariants. In IIIA, we discuss why blur moment invariants cannot be used as a local feature and what we should solve to overcome this limitation. After that, we propose a novel scheme based on the results of this discussion in IIIB and IIIC.
A. Limitations of Blur Moment Invariants –
It is widely acceptable that an image degradation is modeled by the following equation.
, = ∗ ℎ , (1)
Where , is a degraded image such as a blurred image, and , , ℎ , are its original image and the point-spread function (PSF) respectively. The original geometric moment and the degraded geometric moment
can be yielded as follows:
= ∬ , (2)
= ,
= ∬ [ , ∗ ℎ , ] (3)
According to [4], if we can assume that the PSF function ℎ , is an energy-preserving function, i.e.
∬ ℎ , = 1 (4) and a centrally symmetric image function, i.e.,
Unlike in the infinite-analog domain, a digital image must be processed as a finite-extent sequence of discrete signals in the finite-digital domain. In this domain, the relationship between the region of a blurred image and that of an original image are depicted in Figure 1. Figure 1 shows that an original image (whose size is ℎ × ) yields a smaller size of blurred image (whose size is ℎ − × − ). In other words, a blurred image generated from an original image has two regions: valid region (white pixels in Figure 1 (c)) and invalid region (dark pixels in Figure 1 (c)).
Figure 1. An Example of the relationship between a blurred image and its original image in finite-digital domain.
Based on this fact, we redefine the moment of a blurred image as follows:
= ∑ ,!∈#$ ,
= % % &, ' ℎ − &, − '
(,)∈#*
,!∈#$
= ∑ +,/0 ,- ∑ +./0 .- 1,, . ,,. (6)
Where 2 is the region of an original image (in Figure 1 (a)), 2 and 23 are the valid and invalid regions of a blurred image (in Figure 1 (c)) respectively. And the bad effect caused by this invalid region (boundary region) is called "boundary effect" in this paper.
Here, we think about a local patch in an image, which is a small region of an image. As shown in Figure 2, after two local patches in a blurred input image (right) and in its original reference image (left) are extracted with same
size 4 × 4, we denote blur moment invariants of these two local patches of the blurred input image and its original
Figure 1, the sizes (squares) of local patches in 2 and in the valid region of 2 (= 2 − 23) can be denoted as 4 × 4 and 47× 47 respectively.
Figure 2. The relationship of sizes of local patches extracted from between a blurred input image and its original image
Now, 5 46 can be approximately divided into two regions as:
5 46 ≈ 5 4
76 + ∆ 43 (7)
where 43(=46− 476 ) is the size (square) of a boundary region 23and it is associated with blur kernel size. ∆ 43 is a moment invariant calculated from this boundary region.
Furthermore, the distance between moment invariants in each local patches can be defined as follows:
;<= 5 , 5 = >5 46 − 5 46 > (8)
where ‖∙‖ is the Euclidean norm space.
Based on these preparation, the Euclidean distance between 5 46 and 5 46 can be calculated as:
;<= 5 , 5 ≈ >5 46 − [ 5 4
76 + ∆ 43 ]>
≈ ∆ 43 (9)
Note that 5 476 is equal to 5 46 by the equation (6) when two images are absolutely the same except for any blur.
The equation (9) makes it clear for the boundary effect to play a crucial role for the distance error. And this error is dependent on 43.
As a matter of fact, we have done a preliminary experiment to confirm whether the above hypothesis is correct or not. Figure 3 shows this result. In this experiment, we compared matching rate for different size of 4 × 4. And the same value of 4 is used in both an input image (blurred image) and a reference image (non-blurred image).
This experimental result accords with our hypothesis. Namely, as 4 becomes smaller or blur kernel size becomes larger, a relative value of boundary ACBD becomes larger and matching ratio is drastically decreasing.
As is clear in the above discussion, we can wrap up with the reasons why existing moment invariants cannot be used as a local feature as follows:
boundary effect problem: ∆ 43 should be smaller to minimize distance errors. Actually it is very difficult to reduce the absolute error of ∆ 43 , but we may reduce the relative error of ∆ 43 =∆ CB
In fact, ∆ 43 , 5 46 are monotonous increase functions of 43, 4. So, if 43 or AB
CD can be smaller, the
relative error is also smaller.
scaling problem: 5 46 must be equal to 5 476 to satisfy the equation (9). When two images are completely the same except for blur, this condition is automatically satisfied. However, when an input image is different from a reference image in terms of scales, this condition is difficult to satisfy. That is why scale adjustment between two images should be done as correctly as possible.
Figure 3. Experiment of image matching between blurred images and their original images.
B. System Overview –
We overview our proposed scheme which makes it possible to solve the boundary effect problem and the scaling problem as described above.
Figure 4. System Overview
Calculating blur moment invariants: local features are generated from each interest point. By employing SIFT detector, each interest point has an optimal scaling factor. Thus blur moment invariants are calculated in an optimal-scaled local patch. By selecting a suitable size of the local patch, the relative value AB
CD will get smaller.
Feature matching: the similarity between two interest points from each image pair can be calculated by comparing two vectors of invariants. If the similarity is minimum between two interest points, they are regarded as "matched". Finally image matching process is done. When two images have the maximum number of matching pairs of interest points, they are regarded as matched like existing other local feature schemes.
C. Blur Moment Invariants Descriptor –
A local patch around each interest point with scale σ is extracted from Gaussian blurred image, as a square window whose size is 10σ × 10σ. Then 13 blur moment invariants from 2nd order to 6th order are calculated according to [4, 6]. In most of the past work, moment invariants up to 3rd order are used because high order moments are sometimes too sensitive to small geometric distortions in an image. However, we extend blur moment invariants up to 6th order to obtain huge amount of moment invariants which are not variant to both Gaussian blur and linear motion blur.
In SIFT, Gaussian blurred images in each octave of DoG pyramid are sub-sampled by factor 2. By this reason, we need to calculate the normalized moment which is invariant to scale changing of a local patch as follows:
H = IJK
ILLM (10)
where γ = 6 6 and is the p-, q-th order geometric moment. Next, the following 13-dimensional vector is generated as a local descriptor by using H . O = PQ, P6, ⋯ , PQS T (11)
Here, each element of this vector is calculated as follows: PQ= HQQ (12)
P6= HS0 (13)
PS= H0S (14)
PU= H6Q (15)
PV= HQ6 (16)
PW= HS0+ HQ6 6+ H0S+ H6Q 6 (17)
PX= HS0− 3HQ6 6+ H0S− 3H6Q 6 (18)
PZ= HU0+ H0U− 6H66 6+ 4HSQ− 4HQS 6 (19)
P]= HSQ−S ^IDLLL ^__ (20)
PQ0= HV0−Q0 ^IDLLL ^`L (21)
PQQ= HUQ−W ^IDLLL ^D_ (22)
PQ6= HV0+ 5HQU− 10HS6 6+ H0V+ 5HUQ− 10H6S 6 (23)
PQS= HVQ−V ^bL ^__ILL6^DL cd (24)
where O7, ef are local descriptors of interest points in a input image and a reference image, and P,7, P,f are k-th elements of O7, ef respectively. If ;<= O7, ef is minimum and it is less than threshold, two interest points are regarded as "matched".
Figure 5. the preliminary evaluation regarding the effectiveness of high order moments
Figure 5 shows the preliminary evaluation regarding the effectiveness of high order moments. In this evaluation, three descriptors (7 dimensional vector up to 3rd order moments, 13 dimensional vector up to 4th order moments, and 13 dimensional vector up to 6th order moments) are compared by using 'Bikes' sequence in [19]. For example,"O3" and "D7" in "BMI-O3-D7" mean "up to 3rd order" and "7 dimensional vector". From Figure 5 we can see our descriptor, which is 13 dimensional vector up to 6th order moments, performs better than others.
IV.EXPERIMENTAL RESULTS
In this section, we evaluate our proposed scheme by using two different datasets, and prove the effectiveness of our proposed scheme. In the following experiments, the number of matched interest points between two images, the number of correct matches, and the number of total matches are used to evaluate our scheme.
A. Dataset –
We used two different datasets in our experiments: dataset-S and dataset-N. Dataset-S contains a variety of synthetic blurred images. To the contrary, dataset-N is a collection of natural images extracted from natural videos.
In dataset-S (see Figure 6), a variety of synthetically blurred images are included, which are generated by the convolution by two blur kernels (Gaussian blur and linear motion blur). Also, different-scale blurred images, partially blurred images, and non-blurred original images are included in this dataset. Parameter σ of Gaussian blur is 3 to 15, and parameter g of linear motion blur is 10 to 40. Original images which are used to generate synthetic images are randomly selected from some popular test images such as Lena, Baboon etc. and images of caltech-101 dataset.
On the other hand, dataset-N contains strongly blurred real images extracted from 'Bikes' and 'Trees' sequence in [19] and other video sequences captured by digital video camera(see Figure 7).
B. Choise of the Best Feature Detector –
Figure 7. Dataset -N
According to Figure 8, SIFT detector shows the highest performance of the three. Furthermore, when SIFT detector is employed, the matching rate of our scheme is the most stable for various kinds of conditions. From this experiment, we can conclude that SIFT is the most suitable detector for our scheme.
C. Evaluation for Dataset-S –
Next, we compare our proposed scheme with SIFT for dataset-S. As described in IIIB, the same detector (SIFT detector) is employed in both SIFT and our proposed scheme. Therefore, we can think the difference of the performance between them is caused by the descriptors (SIFT descriptor and blur moment invariants descriptor).
As shown in Figure 9, our proposed scheme shows better performance than SIFT in most cases. As a result, we can conclude our proposed scheme is better than SIFT scheme.
Figure 9. Performance comparisons between SIFT and our proposed scheme for dataset-S. Note that both two images of image pair 1/2 in (c) and (d) are not blurred images.
D. Evaluation for Dataset-N –
Finally, we compare our proposed scheme with SIFT for dataset-N. Since our work aims at providing a local descriptor which is robust to a strong blur, we compare our scheme and SIFT under the same condition, namely, the same detector and the same interest points are used in this experiment.
As shown in Figure 10, 1-precision-recall curves was used to evaluate performance. Due to the space limitations, we only show the result of image matching between the first image and the last image in each subsets. This means the strongest blurred image and its original image in dataset-N are used.
Figure 10. Performance comparisons between SIFT and our proposed approach for dataset-N by using curves of 1-precision and recall.
V.CONCLUSIONS AND FUTURE WORK
In this paper, we propose a novel local feature scheme based on blur moment invariants. By using a proper scale-invariant detector, this method can be robust to a strong blur and scale changes. First of all, we discussed why moment invariants cannot be used as a local feature and what we should solve to overcome this limitation. As a result, we clarify the two important limitations which need to be solved. After that, we propose a novel image feature scheme based on this study. Experimental results show our proposed scheme outperform other existing local feature schemes for strongly-blurred images.
In our proposed scheme, existing local point detector (SIFT detector) is employed, and this is the main limitation factor for robustness against a strong blur. Hence, creating a new local detector which is robust to a strong blur is our future work.
REFERENCE
[1] J.F. Cai, H. Ji, C. Liu, and Z. Shen, “Blind motion deblurring using multiple images,” J. Comput. Physics 228,Springer, pp. 5057-71, 2009. [2] M.Cannon, "Blind deconvolution of spatially invariant image blurs with phase," IEEE Transactions on Acoustics, Speech and Signal
Processing 24, vol. 1, pp. 58-63, 1976.
[3] P.J. Bones, C.R. Parker, B.L. Satherley,and R.W. Watson, " Deconvolution and phase retrieval with use of zero sheets," Journal of the Optical Society of AmericaA: Optics, ImageScience, and Vision 12, vol. 9, pp. 1842-57, 1995.
[4] J. Flusser, T. Suk, and S. Saic, " Recognition of blurred images by the method of moments ", IEEE Transactions on Image Processing, vol. 5, pp. 533-8, 1996.
[5] T. Suk and J. Flusser, " Combined blur and affine moment invariants and their use in pattern recognition", Pattern Recognition, vol. 36, pp. 2895-907, 2003.
[6] J. Flusser, T. Suk, and S.Saic," Recognition of images degraded by linear motion blur without restoration", Computing Supplement, vol. 11, pp. 37-51,1996.
[7] J. Liu, and T. Zhang, “Recognition of the blurred image by complex moment invariants,” Pattern Recognition Letters, vol. 26, pp. 1128-38, 2005.
[9] F. Schaffalitzky, and A. Zisserman, “Multi-view matching for unordered image sets,” In Proceedings of the 7th European Conference on Computer Vision, pp. 414-31, 2002.
[10] C. Schmid, and R. Mohr, “Local grayvalue invariants for image retrieval,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 5, pp. 530-4, 1997.
[11] A. Opelt, and M. Fussenegger, A. Pinz, and P. Auer, " Weak hypotheses and boosting for generic object detection and recognition", In Proceedings of the 8th European Conference on Computer Vision, pp. 71 – 84, 2004.
[12] D. Lowe, "Distinctive image features from scale-invariant keypoints", International Journal of Computer Vision, Vol. 60, pp. 91-110, 2004. [13] H. Bay, T. Tuytelaars, and L. VanGool, "Surf: Speeded up robust features", Computer Vision and Image Understanding, Vol. 10, pp.
346-59, 2008.
[14] J. Flusser, J. Boldyza, and B. Zitova, " Moment forms invariant to rotation and blur in arbitrary number of dimensions", IEEE Transactions Pattern Analysis and Machine Intelligence, Vol. 25, pp. 234-46, 2003.
[15] Z. Yang, B. Guo, " Image registration using feature points extraction and pseudo-zernike moments", International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 752-5, 2008.
[16] C. Harris, and M. Stephens, " A combined corner and edge detector", Proceedings of the Fourth Alvey Vision Conference, pp. 147-51, 1998.
[17] T. Lindeberg, "Feature detection with automatic scale selection", International Journal of Computer Vision, Vol. 2, pp. 79-116, 1998. [18] K. Mikolajczyk, and C. Schmid, " Indexing based on scale invariant interest points", In Proceedings of the 8th International Conference on
Computer Vision, pp. 525-31, 2001.
[19] K. Mikolajczyk, and C. Schmid, " A Performance Evaluation of Local Descriptors", IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1615-30, 2004.
[20] S. Leutenegger, M. Chli, and R. Siegwart, " BRISK: Binary Robust Invariant Scalable Keypoints", ICCV, 2011.
[21] Frank M. Candocia, " Moment relations and blur invariant conditions for finite-extent signals in one, two and N-dimensions", Pattern Recognition Letters, Vol. 25, pp. 437-47, 2004.
