On the Evaluation of Cost Functions for Parameter
Optimization of a Multiscale Shape Descriptor
Allan C. Carneiro
∗,1, Jos´e G. F. Lopes
1, Jeov´a F. S. Rocha Neto
2,
Marcelo M. S. Souza
1, F´atima N. S. Medeiros
1and Francisco N. Bezerra
31Universidade Federal do Cear´a, Cear´a, Brazil 2 Brown University, Providence, RI, USA
3Instituto Federal de Educac¸˜ao, Ciˆencia e Tecnologia do Cear´a, Cear´a, Brazil ∗ Corresponding author, Email: allanccarneiro@gmail.com
Abstract—Shape description often relies on parameter ad-justment in order to configure a meaningful scale that enables a computer vision task. Instead of manual interaction, which is prohibitive for large datasets, an alternative solution towards supporting multiscale methodology is to apply metaheuristic optimization. Nevertheless, the cost function assigned to the optimization process is an open question that we fully address in this paper. Our investigation describes the influence of the cost function on the performance of an optimized multiscale shape descriptor using three distinct clustering metrics: the Sil-houette, Davies-Bouldin and Calinski-Harabasz indices. Thus, we optimize the scale parameters of the Normalized Multiscale Bending Energy descriptor using the Simulated Annealing metaheuristic; both classification and retrieval experiments are conducted using a synthetic shape dataset (Kimia 99), two real plant leaf datasets (ShapeCN and Swedish) and the National Library of Medicine (NLM) pill image dataset (NLM Pills). Using the Bulls-eye ratio and the Accuracy measure, the performance evaluation showed that optimized descriptor with the Calinski-Harabasz cost function underperformed other functions for datasets where there is high level of dissimilarity between classes. Particularly for the NLM Pills, where each class has a well-defined pattern and differences within pill classes are quite small, the Normalized Multiscale Bending Energy descriptor did not benefit from the optimization methodology. We also present a qualitative assessment of the cluster arrangements produced by the Self-Organizing Map (SOM) which reinforced that the three cost functions performed differently within the optimized shape descriptor.
Index Terms—Shape description, Optimization, Clustering metrics, Shape retrieval, Shape classification.
I. INTRODUCTION
Several descriptors for shape representation are available in the literature and some of them are parametric descriptors [1]–[3]. In general, these representation approaches handle the setup of descriptor parameters manually using an ex-haustive search, generally leading to sub-optimal results. An example of this remark is the Inner Distance-Shape Context (IDSC) descriptor [1] which was used in leaf-based plant specimen identification [3] using parameter settings primarily calculated for the MPEG-7 [4] dataset. As a matter of fact, parameter setting is application-dependent and hence it requires a new parameter search for a particular application.
The optimization methodology introduced in [5] applies metaheuristic techniques and the Silhouette cluster metric [6] as a cost function for parameter optimization of a multiscale shape descriptor. However, it can be adapted to deal with different cost functions, data sets and applications. Inspired by [5], we investigate the role of three different cost functions for parameter optimization of the Normalized Multiscale Bending Energy (NMBE) [7] shape descriptor. Our approach for NMBE parameter optimization employed the Simulated Annealing (SA) metaheuristic [8] technique and three different cost functions: Silhoutte (SI), the Davies-Bouldin index (DB) [9] and Calinski-Harabasz index (CH) [10]. We carried out tests on images from two datasets of plant leaves (ShapeCN [11] and Swedish [12]), the Kimia 99 dataset [13] and a pill dataset (NLM Pills [14]). For performance evaluation of the optimized descriptor using three different cost functions, we carried out content-based image retrieval (CBIR) and supervised classification exper-iments. The quantitative evaluation of CBIR results and classification using the kNN algorithm [15] was performed with the Bulls-eye [4] and Accuracy measures. The U-matrices [16] were used for cluster quality analysis and visualization.
The remainder of this paper is organized as follows. In the next section, we discuss the materials and methods. Section III describes the experiments. The results and discussions are presented in Section IV, and finally, Section V summarizes our main conclusions.
II. MATERIALS ANDMETHODS
In order to search for the best sets of descriptor parame-ters using different cost functions, we employ the optimiza-tion methodology introduced in [5]. Our algorithm switches among three cost functions (SI, DB and CH) as shown in Fig. 1 and it starts by randomly selecting the descriptor parameters (scales). Then, it extracts shape features of the images from four dataset and computes the cost function. In fact, the cost function drives the optimization algorithm to search for the optimum set of shape descriptor parameters; that is, the one which maximizes the quality of clusters in
Stop Criterion Shape Descriptor Silhouette DB index CH index Cost Function Optimization Algorithm Optimization Shape Descriptor Retrieval Classification Image Dataset
Parameter Adjustment no yes Parameters
Fig. 1: Optimization methodology of a multiscale shape descriptor.
terms of the three different metrics. This process is repeated until a stop criterion is reached, here we adopt the number of rounds. Then, the optimization methodology provides a set of parameters which is input for the feature extraction step. Then, the resulting feature dataset is evaluated by CBIR and classification experiments.
For the optimization algorithm used to search for the best set of parameters of a multiscale shape descriptor, we use SA, according to the directions in [5]. According to the authors of [5], SA and the Differential Evolution (DE) algorithm were more efficient than the Particle Swarm Optimization (PSO) for this scenario. In addition, they also demonstrated that SA was less expensive than DE, since the former evaluated the cost function fewer times than the latter.
Cost functions: Table I describes the three cluster metrics considered in terms of the optimization cost functions (SI, DB and CH), their bounds, optimal values and computa-tional complexity.
• The Silhouette function measures the degree of affinity between a given object and its closest class. SI is considered a measure of cluster quality which assumes values between (−1, 1). A value close to 1 indicates a well classified object; a negative value, on the other hand, implies that the object apparently belongs to a neighbor class. When an object is close to a boundary, the SI value is close to zero, and hence there is uncertainty about which class the object belongs to [6].
• DB evaluates the compactness and separation between clusters. This metric is based on a ratio of the sum of within-cluster scatter to between-cluster separation, and it takes into account the cluster samples and their centroids [17]. DB is non-negative, and minimal values of this index indicate well-defined clusters.
• CH measures the cluster validity based on the average between-cluster and within-cluster sums of squares
[10]. Likewise, DB and CH use cluster centers to calculate the distances between samples. However, CH measures sample separation in terms of the center of the dataset, rather than particular clusters [18]. Multiscale descriptor: We employ NMBE in our experi-ments to follow up the assumption in [5], where the authors reported that the optimization methodology can be adapted to deal with different cost functions.
The bending energy relies on the curvature concept which arises from the differential geometry and involves interesting properties for contour-based shape representation [20]. However, it is well known that its calculation accen-tuates high frequency noise, which limits its application in representing discrete contours [21]. Hence, the concept of multiscale curvature [22], [23] introduced a more noise-robust method for shape invariant description than the curvature itself.
The multiscale curvature represents shapes at various levels of detail, employing a Gaussian kernel to filter out the parametric coordinates of the shape contour prior to its curvature calculation. Thus, the standard deviation of the Gaussian kernel represents the scale factor that controls the level of detail of the shape representation.
Let C = x(n), y(n) be the parametric coordinates of a discrete shape contour of N points, n ∈ [1, 2, · · · , N ]. Cσ = xσ(n), yσ(n) is the filtered version of C at scale σ. The multiscale curvature at scale σ is given by [22]:
κ(σ, n) = x˙σ(n)¨yσ(n) − ¨xσ(t) ˙yσ(n) [( ˙xσ(n))2+ ( ˙yσ(n))2]
3 2
, (1)
where x˙σ(n), ˙yσ(n) and ¨xσ(n), ¨yσ(n) are the first and second order derivatives of xσ(n), yσ(n), respectively.
Inspired by [23], we compute the first and second deriva-tives of the filtered contour coordinates using the derivative property of the Fourier transform as well as the multiscale curvature.
TABLE I: Clustering validation measures
Measure Definition Bounds Optimal value Complexity
SI 1 N C X i ( 1 ni X xCi b(x) − a(x) max (b(x), a(x)) ) (−1, 1) Maximum O(n2) [19] DB 1 N C X i max j,j6=i 1 ni X xCi d(x, ci) + 1 nj X xCj d(x, cj) /d(ci, cj) (0, ∞) Minimum O(n) [19] CH P i nid2(ci, c)/(N C − 1) P i P xCi d2(x, c i)/(n − N C) (0, ∞) Maximum O(n) [19]
D: dataset; n: number of objects in D; c: center of D; : attributes number of D; N C: number of clusters; Ci: the i–th cluster; ni: number of objects
in Ci; ci: center of Ci; d(x, y): distance between x and y; a(x) is the average distance from x to the other elements in the same cluster as x; b(x)
is the minimum average distance from x to elements in a different cluster, minimized over clusters.
The NMBE descriptor derived from the multiscale cur-vature is invariant to shape scale, translation and rotation [7]. Let (κ[σ, 1], κ[σ, 2], . . . , κ[σ, N ]) be the N -sampled values of the multiscale curvature at scale σ. The bending energy for a specific scale σ is given by [7]:
Eσ= L2 N N −1 X n=0 |κ(σ, n)|2, (2)
where L represents the contour perimeter and N is the number of points on the contour.
Thus, NMBE is computed from (1) (multiscale curvature) and (2) (bending energy) for a given number of discrete scales σ1, σ2, · · · , σM by:
NMBE = (log Eσ1, log Eσ2, . . . , log EσM). (3) Datasets: We conducted content-based image retrieval and classification experiments using four public datasets: ShapeCN, Swedish, NLM Pills and Kimia 99. We re-duced the images of the Swedish dataset to 20% of their original size in order to reduce the computation time of our algorithm. We used a subset of the NLM Pill dataset which excluded the class ”Others”, since this class contains shapes that did not match any other class. Furthermore, we randomly selected 166 shapes of each of the three remaining classes to balance the dataset.
The quantitative evaluation: Here, the Bulls-eye metric evaluates our retrieval experiments. For the classification tests, each set of shape descriptors was split into two random subsets (70% and 30%) for training and testing, respectively; this process was repeated for 100 rounds and the mean Accuracy of a kNN classifier (N = 3) was computed. The Accuracy measure is defined in terms of the ratio of correctly predicted classes to number of total predicted classes.
The qualitative evaluation: We apply the Kohonen clustering algorithm [24] represented by the U-matrix to
assess the quality of the clustering arrangement. The U-matrix shows the shapes in their corresponding topology using a particular color for each class. The darker the cluster borders in the U-matrix, the greater the neighbor distance in the topology. Moreover, a well-defined cluster presents all elements as close to each other and surrounded by a clear dark border.
III. EXPERIMENTS
We performed the experiments by applying the optimiza-tion methodology to the NMBE shape descriptor using each distinct cost function, i.e., SI, DB and CH. Our experiments on four datasets considered five descriptor scales, corresponding to the five σ parameter values to be optimized by SA in the range [0.4, 100]. We have tuned SA with the following parameters: the initial temperature [25] (T0 = 100), epoch length [25] (L = 10), the cooling ratio [8], [25] (α = 0.9) of the exponential cooling schedule, and number of rounds Nr = 100 that was the stop criterion. Then, the optimization algorithm was iterated 30 times for each cost function to optimize the NMBE descriptor; thus the mean and variance of the evaluation measures were calculated. Table II displays the experimental results which also considered tests using non-optimized NMBE parameters originally introduced in [7]. The non-optimized approach consists of a fixed set of scales; the Bulls-eye and Accuracy measures are then computed. However, this non-optimized approach does not address the problem of adjusting the descriptor to a particular application meaning that different problems or datasets require different scales for the NMBE descriptor. In contrast, the optimization methodology proposed in [5] is specific to the range of parameters, number of scales, shape descriptor and dataset.
IV. RESULTS ANDDISCUSSION
The optimization experiments employed three cost func-tions which measure the cluster quality of data and adjust
TABLE II: Retrieval and Classification.
SI DB CH Method proposed in [7]
Dataset Bulls-eye(%) Accuracy(%) Bulls-eye(%) Accuracy(%) Bulls-eye(%) Accuracy(%) Bulls-eye(%) Accuracy(%) ShapeCN 70.91 ± 1.81 71.00 ± 2.80 71.67 ± 1.61 75.16 ± 1.93 41.98 ± 5.86 39.28 ± 6.27 63.39 ± 0.00 67.67 ± 0.00 Swedish 86.53 ± 1.48 86.53 ± 1.97 76.70 ± 3.38 82.98 ± 1.94 63.90 ± 3.07 62.23 ± 8.38 76.08 ± 0.00 85.99 ± 0.00 Kimia 99 90.20 ± 1.30 84.53 ± 3.38 90.99 ± 0.92 85.83 ± 2.42 63.54 ± 0.00 39.24 ± 0.90 89.26 ± 0.00 76.67 ± 0.00 NLM Pills 99.95 ± 0.01 99.24 ± 0.13 99.95 ± 0.09 99.24 ± 0.09 99.96 ± 0.00 99.34 ± 0.04 97.57 ± 0.00 99.36 ± 0.00
the NMBE scales according to the shape details of each dataset. Here, the datasets posed several challenges which were revealed by the cost functions and thence the de-scriptor scales. The significant differences between the four datasets mean that different scales of the same descriptor are required to reveal the shape details of these image datasets. Related investigations in [19] and [26], where cluster metrics were tested to determine the correct number of dataset clusters, concluded that a metric which is suitable for a particular dataset may not be appropriate for others.
Table II shows that for shapes from the ShapeCN, Swedish and Kimia 99 datasets, the NMBE descriptor optimized with CH did not reveal remarkable shape details and therefore the Bulls-eye and Accuracy values were smaller than the non-optimized descriptor introduced in [7]. The optimized NMBE with SI and DB for the previously mentioned datasets outperformed the method proposed in [7] or achieved similar results. On the other hand, the optimization approach using SI, DB and CH performed relatively similar to the method proposed in [7], when applied to the NLM Pills, which comprises only three classes of shapes with a well-defined pattern and small differences within classes. Table II illustrates that the cost function CH was not suited to revealing shape details and it therefore did not succeed for the leaf datasets and Kimia 99. The CH optimized scales were not able to capture global and local shape details. It was concluded that the application of the methodology proposed in [5] to a specific dataset requires an appropriate cost function to achieve satisfactory results.
It is worth noting that there is a trade-off between the per-formance of the proposed method in retrieval and classifica-tion experiments and the DB cost funcclassifica-tion complexity (see Table I). The results in Table II show that the optimization methodology using DB and SI outperformed CH; however, the computational complexity of DB is significantly lower than SI. Although CH and DB are computationally equiv-alent, the former failed in the experiments involving the ShapeCN, Swedish and Kimia 99 datasets. These results led us to infer that a particular problem or dataset may require a specific cost function to optimize multiscale parameters of a shape descriptor, successfully.
Figs. 2a, 2b, 2c and 2d show the U-matrices with NMBE optimized by SA with the cost functions SI, DB, CH
and without optimization for the ShapeCN dataset [7], respectively. The cluster arrangements in Figs. 2a, 2b and 2d are more organized than in Fig. 2c. Likewise, Fig. 3 displays four U-matrices for the Kimia 99 dataset. The cluster arrangements in Fig. 3a, 3b and 3d are more organized than in Fig. 3c. Thus, the U-matrices using SI and DB gave similar results and the cluster arrangement of CH was worse than the method proposed in [7].
Figs. 2c and 3c show regions without borders and merged elements of different classes confirming that CH is an inappropriate cost function for the ShapeCN and Kimia 99 datasets. This leads us to conclude that the CH cost function cannot optimize the scale parameters of NMBE to pro-vide a more reliable shape representation of the ShapeCN and Kimia 99 datasets. This means that SA metaheuristic succeeds best in adjusting the descriptor scales with an appropriate cost function.
The similar U-matrices in Fig. 4 show that the optimized and non-optimized descriptors achieved similar results; this may be due to the high level of within class similarity and the geometry of the shapes in the NLM pill dataset. In this case, the optimization methodology achieved fairly similar results regardless the of cost function, and hence this dataset did not benefit from the optimization methodology. Note that the U-matrices in Figs. 2, 3 and 4 are in accordance with the results in Table II. Additionally, all U-matrices display the corresponding Bulls-eye values.
V. CONCLUSIONS
In this paper, we investigate a methodology for parameter optimization of a shape descriptor with three distinct cost functions applied to different datasets to search for the best set of parameters. Our goal is to show that a particular dataset requires an appropriate cost function to optimize the shape descriptor parameters and thus extract the best multiscale features. Our proposal achieves similar retrieval and classification results, for the considered cost functions, in comparison with a non-optimized method when the shapes within classes are remarkably similar and each class has a well-defined pattern, as in the NLM Pills. This suggests that, the parameter adjustment is not crucial for these shape patterns since the discrimination of different classes is not a difficult task for the multiscale descriptor. For the leaf datasets, SI and DB achieve better results than
(a) Optimized with SI cost function (Bulls-eye = 74.43%).
(b) Optimized with DB cost function (Bulls-eye = 72.06%).
(c) Optimized with CH cost function (Bulls-eye = 39.20%).
(d) Non-optimized method proposed in [7] (Bulls-eye = 63.39%).
Fig. 2: U-matrices for the ShapeCN dataset shapes using the optimization approach with three different cost functions and the non-optimized method proposed in [7].
(a) Optimized with SI cost function (Bulls-eye = 91.55%).
(b) Optimized with DB cost function (Bulls-eye = 92.56%).
(c) Optimized with CH cost function (Bulls-eye = 63.54%).
(d) Non-optimized method proposed in [7] (Bulls-eye = 89.26%).
Fig. 3: U-matrices for Kimia 99 dataset shapes using the optimization approach with three different cost functions and the non-optimized method proposed in [7].
(a) Optimized with SI cost func-tion (Bulls-eye = 99.96%).
(b) Optimized with DB cost func-tion (Bulls-eye = 99.96%).
(c) Optimized with CH cost func-tion (Bulls-eye = 99.96%).
(d) Non-optimized method proposed in [7] (Bulls-eye = 97.57%).
Fig. 4: U-matrices for NLM Pill dataset shapes using the optimization approach with three different cost functions and the non-optimized method proposed in [7].
the non-optimized method. However, SI involves a high order of complexity. When the optimization methodology employs the CH function, it performs less effective than the others indicating that this cost function is unsuitable for this problem. Thus, we conclude that the role of the cost function in the parameter optimization methodology is essential to achieve the best performance of a shape descriptor in retrieval and classification experiments.
ACKNOWLEDGEMENTS
We are grateful to CNPq and CAPES (#444784/2014 − 4,#401442/2014 − 4 and #306600/2016 − 1) for the financial support. We also thank Dr. C.J. Passarinho for insightful discussions, and improvements in the manuscript.
REFERENCES
[1] H. Ling and D. W. Jacobs, “Shape classification using the inner-distance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 286–299, 2007.
[2] J. Wang, X. Bai, X. You, W. Liu, and L. J. Latecki, “Shape match-ing and classification usmatch-ing height functions,” Pattern Recognition Letters, vol. 33, no. 2, pp. 134–143, 2012.
[3] B. Wang, D. Brown, Y. Gao, and J. La Salle, “March: Multiscale-arch-height description for mobile retrieval of leaf images,” Informa-tion Sciences, vol. 302, pp. 132–148, 2015.
[4] L. J. Latecki, R. Lakamper, and T. Eckhardt, “Shape descriptors for non-rigid shapes with a single closed contour,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1. IEEE, 2000, pp. 424–429.
[5] M. M. de Souza, F. N. Medeiros, G. L. Ramalho, I. C. de Paula Jr, and I. N. Oliveira, “Evolutionary optimization of a multiscale descriptor for leaf shape analysis,” Expert Systems with Applications, vol. 63, pp. 375–385, 2016.
[6] P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics, vol. 20, pp. 53–65, 1987.
[7] R. M. Cesar Jr and L. F. Costa, “Application and assessment of multiscale bending energy for morphometric characterization of neural cells,” Review Of Scientific Instruments, vol. 68, no. 5, pp. 2177–2186, 1997.
[8] A. P. Engelbrecht, Computational Intelligence: An Introduction, 2nd ed. John Wiley & Sons, 2007.
[9] D. L. Davies and D. W. Bouldin, “A cluster separation measure,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-1, no. 2, pp. 224–227, 1979.
[10] T. Cali´nski and J. Harabasz, “A dendrite method for cluster analysis,” Communications in Statistics-theory and Methods, vol. 3, no. 1, pp. 1–27, 1974.
[11] A. R. Backes, D. Casanova, and O. M. Bruno, “A complex network-based approach for boundary shape analysis,” Pattern Recognition, vol. 42, no. 1, pp. 54–67, 2009.
[12] O. S¨oderkvist, “Computer vision classification of leaves from swedish trees,” Master’s thesis, 2001.
[13] T. Sebastian, P. Klein, and B. Kimia, “Recognition of shapes by editing their shock graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 550–571, 2004. [14] D. Ushizima, A. Carneiro, M. Souza, and F. Medeiros, Investigating
Pill Recognition Methods for a New National Library of Medicine Image Dataset. Springer, 2015, vol. 9475, pp. 410–419.
[15] E. Fix and J. L. Hodges Jr, “Discriminatory analysis-nonparametric discrimination: consistency properties,” US Air Force School of Aviation Medicine, Tech. Rep. 4, 1951.
[16] A. Ultsch and H. P. Siemon, “Kohonen’s self organizing feature maps for exploratory data analysis,” in Proceedings of International Neural Networks Conference, vol. 1. Paris: Kluwer Academic Press, 1990, pp. 305–308.
[17] C.-H. Chou, M.-C. Su, and E. Lai, “A new cluster validity measure and its application to image compression,” Pattern Analysis and Applications, vol. 7, no. 2, pp. 205–220, 2004.
[18] J. Baarsch and M. E. Celebi, “Investigation of internal validity measures for k-means clustering,” in Proceedings of the International MultiConference of Engineers and Computer Scientists, vol. 1, 2012, pp. 14–16.
[19] S. Saitta, B. Raphael, and I. F. Smith, “A comprehensive validity index for clustering,” Intelligent Data Analysis, vol. 12, no. 6, pp. 529–548, 2008.
[20] V. V. Kindratenko, “On using functions to describe the shape,” Journal of Mathematical Imaging and Vision, vol. 18, no. 3, pp. 225–245, 2003.
[21] L. F. Costa and R. M. Cesar Jr, Shape Classification and Analysis: Theory and Practice, 2nd ed. Boca Raton, Florida, FL, USA: CRC Press Taylor & Francis Group, 2009.
[22] F. Mokhtarian and A. Mackworth, “A theory of multiscale, curvature-based shape representation for planar curves,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 8, pp. 789– 805, 1992.
[23] R. M. Cesar and L. F. Costa, “Towards effective planar shape representation with multiscale digital curvature analysis based on signal processing techniques,” Pattern Recognition, vol. 29, no. 9, pp. 1559–1569, 1996.
[24] T. Kohonen, M. R. Schroeder, and T. S. Huang, Eds., Self-Organizing Maps, 3rd ed. Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2001.
[25] M.-W. Park and Y.-D. Kim, “A systematic procedure for setting parameters in simulated annealing algorithms,” Computers & Op-erations Research, vol. 25, no. 3, pp. 207–217, 1998.
[26] G. W. Milligan and M. C. Cooper, “An examination of procedures for determining the number of clusters in a data set,” Psychometrika, vol. 50, no. 2, pp. 159–179, 1985.
