Pattern Recognition in Macroscopic and
Dermoscopic Images for Skin Lesion Diagnosis
Roberta B. Oliveira1, Aledir S. Pereira2, João Manuel R. S. Tavares1
1 Instituto de Ciência e Inovação em Engenharia Mecânica e Engenharia Industrial, Faculdade de
Engenharia, Universidade do Porto, Rua Dr. Roberto Frias, 4200-465 Porto, Portugal, Email: (roberta.oliveira, tavares)@fe.up.pt;
2 Departamento de Ciências de Computação e Estatística, Instituto de Biociências, Letras e
Ciências Exatas, Universidade Estadual Paulista, Rua Cristóvão Colombo, 2265, 15054-000 São José do Rio Preto, SP, Brazil, Email: aledir@ibilce.unesp.br.
Abstract Pattern recognition in macroscopic and dermoscopic images is a
chal-lenging task in skin lesion diagnosis. The search for better performing classification has been a relevant issue for pattern recognition in images. Hence, this work was particularly focused on skin lesion pattern recognition, especially in macroscopic and dermoscopic images. For the pattern recognition in macroscopic images, a computational approach was developed to detect skin lesion features according to the asymmetry, border, colour and texture properties, as well as to diagnose types of skin lesions, i.e., nevus, seborrheic keratosis and melanoma. In this approach, an anisotropic diffusion filter is applied to enhance the input image and an active con-tour model without edges is used in the segmentation of the enhanced image. Fi-nally, a support vector machine is used to classify each feature property according to their clinical principles, and also for the classification between different types of skin lesions. For the pattern recognition in dermoscopic images, classification mod-els based on ensemble methods and input feature manipulation are used. The fea-ture subsets was used to manipulate the input feafea-ture and to ensure the diversity of the ensemble models. Each ensemble classification model was generated by using an optimum-path forest classifier and integrated with a majority voting strategy. The performed experiments allowed to analyse the effectiveness of the developed approaches for pattern recognition in macroscopic and dermoscopic images, with the results obtained being very promising.
Keyword Image processing and analysis; Image segmentation; Feature extrac-tion and selecextrac-tion; Image classificaextrac-tion; Ensemble methods.
1. Introduction
Skin cancer is one of the most commonly diagnosed cancers worldwide and mela-noma is the most aggressive form of skin cancer, as well as the one with the highest mortality rate. For example, in the United States, 87,110 new cases of melanoma were estimated to be diagnosed in 2017. Furthermore, 9,730 deaths from melanoma
were estimated for the same year [1]. Computational systems have been proposed in order to assist dermatologists in skin cancer diagnosis, or even to monitor skin lesions [2,3]. Image acquisition, pre-processing, segmentation, feature extraction and selection, and classification are fundamental steps commonly found in compu-tational systems for diagnosing skin lesions. Macroscopic and dermoscopic images are examples of images acquired from non-invasive imaging techniques, which have been widely used in such systems. Macroscopic images are usually obtained by using common digital cameras, while dermoscopic images are acquired by a der-matoscope device that allows a more detailed visualization of the lesion patterns on the skin surface.
Segmentation is an important step that allows the extraction of the regions of interest (ROI) within an image. Previous studies have shown that computational methods for image segmentation can provide suitable results for the identification of skin lesions in images [4]. Usually, the images under analysis are pre-processed for image enhancement and artefact removal, so that more robust segmentations can be achieved [5]. The extraction of representative features of the ROI under analysis is essential for efficient classification of the skin lesions. The extracted features are usually based on clinical approaches used by dermatologists in diagnosing skin le-sions. The ABCD rule is a commonly used method to classify such lesions in mac-roscopic and dermoscopic images according to Asymmetry, Border, Colour and Di-ameter (or Differential structures in the case of dermoscopic images) criteria [6]. Additionally, texture analysis can be performed to assess the surface roughness of the lesions to assist in discriminating between benign and malignant lesions.
Skin lesion classification systems should demonstrate high performance and ef-ficacy, considering that they will be used to assist in dermatological diagnosis. The evaluation and improvement of the classification performance are essential require-ments of pattern recognition [7]. The defining what features are meaningful and essential to represent the skin lesion patterns is a relevant challenge that should be considered. The application of several descriptions may be needed, considering the enormous quantity of information present in images. Nevertheless, a larger feature space can include redundant and irrelevant data. One solution to this problem is the application of feature selection algorithms to define the most appropriate features from images, since such algorithms permit the removal of redundant and irrelevant features [8,9]. Additionally, these algorithms still can improve the classification ac-curacy rate.
Another solution to improve classification performance is based on ensemble methods [10]. Ensemble methods consist of integrating several classification mod-els in order to develop a more robust system that provides more accurate results than by using a single classifier. Such models can be composed of either only one
learning algorithm, classified as homogeneous or several learning algorithms, clas-sified as heterogeneous. Several algorithms for constructing homogeneous ensem-bles have been developed through data manipulation, such as manipulating the training samples or the input features [11,12]. Algorithms for manipulating the training samples allow the generation of multiple hypotheses, in which the learning algorithm is applied to different subsets of the training samples. Algorithms for ma-nipulating the input features allow the generation of ensembles based on different features available to the learning algorithm. This process can involve, for example, the splitting of a set of features into subsets.
The main objective of this work was to develop algorithms for pattern recogni-tion in both macroscopic and dermoscopic images in order to assist dermatologists in diagnosis of skin cancer. The achievements and contributions obtained from this work can be summarized as follows:
An effective computational approach developed for the segmentation and classification of pigmented skin lesions in macroscopic images. The ap-proach is based on asymmetry, border, colour and texture analysis for ex-tracting skin lesion features and it is able to identify different types of skin lesions;
An effective approach for the combination of skin lesion features by using different feature extraction algorithms. The main contribution of this ap-proach is the texture analysis based on several colour channels, as well as the application of different texture-based feature extraction algorithms; Effective classification models based on ensemble methods and input
fea-ture manipulation to improve the skin lesion computational diagnosis from dermoscopic images. The main contribution of these models is the feature subset selection based on specific feature groups and feature selection algo-rithms for the input feature manipulation, which allowed the finest genera-tion of diversity for the ensemble-based models and better classificagenera-tion ac-curacy.
2. Developed approaches
In order to fulfil the objective of this work, two approaches were defined: pattern recognition in macroscopic images, and pattern recognition in dermoscopic images. The first approach includes both the segmentation and the classification problems in macroscopic images. The second approach mainly outlines the feature extraction and classification problems in dermoscopic images. The overview of the developed work for the skin lesion pattern recognition is presented in Figure 1.
Figure 1: Overview of our research on skin lesion pattern recognition.
2.1. Image databases
The databases used to evaluate the developed approaches are composed of macro-scopic and dermomacro-scopic images of pigmented skin lesions. The macromacro-scopic image database have a total of 408 images, which were collected from several databases as described in Oliveira et al. [13]. A great deal of information concerning the diagnosis of the imaged lesions provided by an expert dermatologist was also available, includ-ing diagnostics on the lesions and their features (i.e., asymmetry, border, colour and texture). The images of the macroscopic database were resized to 200x200 pixels to simplify their processing.
The dermoscopic images of pigmented skin lesions used to evaluate the extraction of features were collected from the International Skin Imaging Collaboration (ISIC) dataset [14]. In addition, the images were paired with the expert manual annotation that contains the skin lesion diagnoses, as well as the ground truth lesion segmenta-tions in the form of binary masks. In this study, a feature extraction approach, based on shape properties, colour variation and texture analysis, was developed. Moreover, since the shape properties are obtained from the lesion borders, only the images where the lesion fitted completely within the image frame were selected so that the features could be extracted with greater accuracy. A total of 1,104 images were se-lected from the original dataset. The images of the dermoscopic dataset were resized to an average resolution of 400x299 pixels to simplify their processing.
2.2. Pattern recognition in macroscopic images
The difficulty in distinguishing some types of skin lesions has become a challenging research area. Hence, a computational approach was developed for the segmentation
and classification of pigmented skin lesions in macroscopic images. Briefly, the ap-proach involves the following steps: 1) image pre-processing, 2) image segmenta-tion and post-processing, 3) feature extracsegmenta-tion and classificasegmenta-tion, and 4) classifica-tion of type of skin lesions.
The first step intends to enhance input grey-level images corrupted by noise and it is based on an anisotropic diffusion filter [15]. The second step identifies the le-sion presented in the enhanced image by using an active contour model without edges (Chan-Vese model [16]), and for post-processing, the segmented region is identified based on morphological filtering (the opening operation followed by the closing operation) to improve the quality of the segmentation result. In the third step, features are extracted from the post-processed region, which include the asym-metry, border, colour, and texture properties. In addition, a support vector machine (SVM) [17] based on a histogram intersection kernel [18] is applied to classify each feature property into two categories according to their clinical principles. Finally, the last step concerns the binary classification between different types of skin le-sions, i.e., nevus, seborrheic keratosis, and melanoma, based on the SVM classifier.
The asymmetry properties are obtained from the ratios between each pair of the semi-lines that represents the perpendicular lines by overlapping the two sub-re-gions of the lesion along an axis. A number of peaks, valleys and straight lines of the border are computed to assess the border properties by using the vector product and inflexion point descriptors from a one-dimensional border [13]. Statistical measures, i.e., average, variance and standard deviation, are computed for each col-our channel of the RGB colcol-our space to extract the colcol-our properties. The fractal dimension method with a box-counting [19] is adopted based on grey-level images for the texture analysis; by using the methods mentioned earlier, 44 features were extracted to represent the lesion.
2.3. Pattern recognition in dermoscopic images
The ensemble methods are seen as promising models to be used for skin lesion clas-sification due their ability to integrate several clasclas-sification models to provide a more robust system. In addition, appropriate features and ensemble methods can be combined to achieve superior performance for skin lesion classification. Ensemble classification models based on input feature manipulation were proposed for the pattern recognition in dermoscopic images in order to improve the skin lesion com-putational diagnosis. This approach involves the following steps: 1) feature extrac-tion, 2) data pre-processing, 3) feature manipulaextrac-tion, and 4) ensemble model inte-gration and image classification.
Firstly, a combination of features was analysed based on shape properties, colour variation, and texture analysis by using different feature extraction methods. A total
of 510 features related to shape, colour and texture were extracted from the skin lesion images. The shape properties include the lesion area, border perimeter, equiv-alent diameter, compactness, circularity, solidity, rectangularity, aspect ratio, ec-centricity, lesion asymmetry, and border irregularity. The RGB, HSV, CIE Lab and CIE Luv colour spaces were used for extracting both colour and texture properties. The average, variance, standard deviation, minimum and maximum colours, and colour skewness were computed for each colour channel of the colour spaces. The texture features were also extracted for each colour channel of the colour spaces by using the fractal dimension analysis [19], discrete wavelet transform [20] and co-occurrence matrix [21] methods.
For the data pre-processing step, a normalization method and a resample proce-dure [22] are applied to the extracted features in order to scale all numeric values to within the same interval and distribute the samples evenly for each class. Different categories of classifiers [22] were compared to evaluate the effectiveness of the proposed combination of features in the benign or malignant lesion classification; namely, the k-nearest neighbours (KNN), Bayes networks, C4.5 decision tree, mul-tilayer perceptron (MLP), SVM and optimum-path forest (OPF) [23]. In addition, different feature selection algorithms were adopted, which include the Relief-F, in-formation gain-based feature selection, gain ratio-based feature selection (GRFS), Pearson’s correlation coefficient-based feature selection, correlation-based feature selection (CFS) and principal-component analysis (PCA) [22].
Feature subsets from the shape properties, colour variation and texture analysis were selected to manipulate the input features and to generate diversity for the en-semble models. Three enen-semble classification models based on input feature ma-nipulation are presented here: 1) feature subset ensemble (SE-OPF), 2) feature sub-set ensemble with feature selection (SEFS-OPF), and 3) feature sub-set ensemble with feature selection (FEFS-OPF). The feature subset selection for these models are based on specific feature groups, subset correlation, and different feature selection algorithms, respectively. Each ensemble classification model was built by using an OPF classifier [23] and integrated with a majority voting strategy [10]. For the OPF classifier, the Euclidean, Chebyshev and Manhattan distance functions [22] are used to measure the distances between the feature subsets. The effectiveness of the en-semble classification models was evaluated for the benign or malignant lesion clas-sification process.
3. Results and Discussion
In this section, the pattern recognition results in macroscopic and dermoscopic im-ages are described and discussed. First, the experiments for segmentation and clas-sification of the macroscopic images are presented. Afterwards, the experiments on
the feature extraction of skin lesions and classification of the dermoscopic images are introduced.
3.1. Segmentation and classification of the macroscopic images
A subjective evaluation was performed to analyse the obtained segmentation re-sults. Hence, the visual assessment by a specialist of the segmented regions classi-fied whether each lesion presented in the 408 images was correctly segmented or not. Figure 2 presents examples of the segmentation results obtained by applying the developed segmentation method to the original images. The segmentation re-sults obtained by our approach were compared against the threshold-based segmen-tation results achieved by using the well-known method proposed by Otsu [24]. Our approach obtained significantly superior results compared to the threshold-based method according to the visual assessment made by the specialist.
Figure 2: Example of segmentation results obtained by applying the developed com-putational approach: (a and c) original images and (b and d) original images with the detected borders (white contours) overlapped.
From the 385 correctly segmented images, 59 images were melanocytic nevi, 75 images were seborrheic keratosis, and 251 images were melanoma according to the expert dermatologist. In regard to the asymmetry criterion, the lesions were sym-metric in 124 images and in 261 images were asymsym-metric. In regard to the border criterion, the lesions have regular borders in 69 images and irregular borders in 316 images. As to the colour criterion, the lesions present uniform colours in 31 images and non-uniform colours in 354 images. In regard to the texture criterion, the lesions present regular texture in 219 images and in 166 images they present irregular tex-ture. Afterwards, the correctly segmented images were used for the classification process, in which the results obtained by the developed approach were compared with the results achieved by the SVM classifier based on a kernel commonly used, i.e., the radial basis function (RBF) kernel [17].
The results achieved for the feature classification are shown in Table 1, and the results obtained for the skin lesion classification are shown in Table 2. The applica-tion of a histogram intersecapplica-tion kernel showed superior performances for both im-age classifications. Although the border classification by using an RBF kernel had
better accuracy, the border classification by using a histogram intersection kernel achieved a more balanced classification result between the regular and irregular classes.
Table 1: Feature classification results of the developed approach.
Classification Radial basis function Histogram intersection
Accuracy Accuracy
Asymmetry 69.4% 96.5%
Border 81.3% 74.1%
Colour 73.1% 73.6%
Texture 62.7% 64.2%
Table 2: Skin lesion classification results of the developed approach.
Classification Radial basis function Histogram intersection
Accuracy Accuracy
Nevus – Keratosis 72.8% 79.0%
Nevus – Melanoma 69.9% 74.4%
Keratosis – Melanoma 69.7% 74.3%
3.2. Feature extraction and classification of the dermoscopic
images
Experiments to analyse the extracted features were performed for the image classi-fication in benign or malignant lesions by using different classifiers and feature se-lection algorithms that ensure the effectiveness of the developed approach. Our ap-proach was applied to a set of 1104 dermoscopic images by using a cross-validation procedure [22]. Of these, 552 images were benign lesions and 552 images were malignant lesions. Table 3 shows the best classification results using feature selec-tion algorithms. These results shown that the OPF classifier with the features se-lected by the CFS algorithm achieved superior results compared to the others. Table 3: The best classification results using feature selection algorithms.
Classifier Feature selection algorithm Features Accuracy
kNN CFS 50 75.8% Bayes Net CFS 37 74.4% C4.5 PCA 31 89.7% MLP GRFS 75 90.6% SVM Relief-F 75 80.1% OPF CFS 50 91.6%
The ensemble classification models were also applied on the set of 1104 dermo-scopic images and the best classification accuracy was obtained by the SE-OPF al-gorithm. The classification results are presented in more details in Figure 3, which
shows the variation of the accuracy (ACC) according to the ensemble size defined for each ensemble classification model. The classification results achieved by the best developed classification model were compared against the ones obtained using ensemble algorithms with and without feature selection that are commonly used in the literature; namely, bagging [10], AdaBoost [25] and random forest [26]. Fur-thermore, our classification model was also compared to the individual OPF classi-fier [23] to analyse the efficacy of the ensemble algorithms. Table 4 shows the re-sults using different classification algorithms, as well as the rere-sults of our model. The developed model achieved a better accuracy compared to other classification algorithms used in the literature.
Figure 3: Variation of the classification accuracy, according to the ensemble size established for each ensemble classification model.
Table 4: Comparative results between classification algorithms.
Classification algorithms Accuracy
OPF 92.3% OPF (CFS) 91.6% Bagging (OPF) 89.7% Bagging (CFS + OPF) 91.8% AdaBoostM1 (OPF) 92.3% AdaBoostM1 (CFS + OPF) 91.6% Random forest 93.9% Random forest (CFS) 93.7%
4. Conclusion and future works
Pattern recognition in macroscopic images are still little explored in research on pattern recognition field, and most studies do not deal with the classification of all features considered in present work. The approach using segmentation and classifi-cation of pigmented skin lesions in macroscopic images developed in this work al-lows the successful extraction of lesion features and the identification of different types of skin lesions, with the results obtained being very promising. The colour and texture analysis, based on several colour spaces combined with shape properties and by using different feature extraction methods, provided very promising results for skin lesion pattern recognition in dermoscopic images. In addition, the effec-tiveness of the combination of features was very important for the constructing en-semble classification models to improve the correctness of skin lesion computa-tional diagnosis.
In conclusion, a relevant issue to be approached is the application of the segmen-tation algorithm in colour macroscopic and dermoscopic images, and it still can be combined with the developed classification model based on ensemble methods. In order to approach other problems concerning the dermoscopic image diagnosis, the ensemble-based models may be applied; for example, to identify the presence of global and local patterns, since pattern analysis is a challenging task in discriminat-ing between benign and malignant skin lesions. In addition, deep learndiscriminat-ing architec-tures [27] may be taken into account in future studies related to the skin lesion clas-sification in both macroscopic and dermoscopic images, since these architectures have revealed one can stress the capacity of learning from enormous amounts of data.
Acknowledgments
The first author would like to thank CNPq (“Conselho Nacional de Desenvolvi-mento Científico e Tecnológico”), in Brazil, for her PhD grant. Authors gratefully acknowledge the funding of Project NORTE-01-0145-FEDER-000022 - SciTech - Science and Technology for Competitive and Sustainable Industries, co-financed by “Programa Operacional Regional do Norte” (NORTE2020), through “Fundo Eu-ropeu de Desenvolvimento Regional” (FEDER). Furthermore, the authors thank Dr. Ricardo Baccaro Rossetti, from Derm Clínica’s Dermatologist of São José do Rio Preto, in Brazil, for his suggestions and for evaluating the results obtained.
References
1. American Cancer Society (2017) Cancer Facts & Figures 2017. American Cancer Society, Atlanta
2. Oliveira RB, Papa JP, Pereira AS, Tavares JMRS (2016) Computational Methods for Pigmented Skin Lesion Classification in Images: Review and Future Trends. Neural Computing and Applications 27:1-24. doi:10.1007/s00521-016-2482-6
3. Oliveira RB, Filho ME, Ma Z, Papa JP, Pereira AS, Tavares JMRS (2016) Computational methods for the image segmentation of pigmented skin lesions: a review. Computer Methods and Programs in Biomedicine 131:127-141. doi:10.1016/j.cmpb.2016.03.027
4. Silveira M, Nascimento JC, Marques JS, Marcal ARS, Mendonca T, Yamauchi S, Maeda J, Rozeira J (2009) Comparison of Segmentation Methods for Melanoma Diagnosis in Dermoscopy Images. IEEE Journal of Selected Topics in Signal Processing 3 (1):35-45. doi:10.1109/jstsp.2008.2011119
5. Abbas Q, Fondón I, Rashid M (2011) Unsupervised skin lesions border detection via two-dimensional image analysis. Computer Methods and Programs in Biomedicine 104 (3):e1-e15. doi:10.1016/j.cmpb.2010.06.016
6. Abbasi NR, Shaw HM, Rigel DS, Friedman RJ, McCarthy WH, Osman I, Kopf AW, Polsky D (2004) Early diagnosis of cutaneous melanoma: revisiting the ABCD criteria. Jama 292 (22):2771-2776. doi:10.1001/jama.292.22.2771
7. Webb AR (2003) Statistical pattern recognition. 2 edn. John Wiley & Sons, England
8. Celebi ME, Kingravi HA, Uddin B, Iyatomi H, Aslandogan YA, Stoecker WV, Moss RH (2007) A methodological approach to the classification of dermoscopy images. Computerized Medical Imaging and Graphics 31 (6):362-373. doi:10.1016/j.compmedimag.2007.01.003
9. Garnavi R, Aldeen M, Bailey J (2012) Computer-Aided Diagnosis of Melanoma Using Border- and Wavelet-Based Texture Analysis. IEEE Transactions on Information Technology in Biomedicine 16 (6):1239-1252. doi:10.1109/titb.2012.2212282
10. Dietterich TG (2000) Ensemble methods in machine learning. In: Multiple Classifier Systems, vol 1857. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 1-15. doi:10.1007/3-540-45014-9_1
11. Schaefer G, Krawczyk B, Celebi ME, Iyatomi H (2014) An ensemble classification approach for melanoma diagnosis. Memetic Computing 6 (4):233-240. doi:10.1007/s12293-014-0144-8 12. Barata C, Emre Celebi M, Marques JS Melanoma detection algorithm based on feature fusion. In: 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society Milan, August 25-29 2015. IEEE, pp 2653-2656
13. Oliveira RB, Marranghello N, Pereira AS, Tavares JMRS (2016) A computational approach for detecting pigmented skin lesions in macroscopic images. Expert Systems with Applications 61:53-63. doi:10.1016/j.eswa.2016.05.017
14. Gutman D, Codella NCF, Celebi E, Helba B, Marchetti M, Mishra N, Halpern AC Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC), arXiv preprint arXiv:1605.01397.
15. Barcelos CAZ, Boaventura M, Silva Junior EC (2003) A well-balanced flow equation for noise removal and edge detection. IEEE Transactions on Image Processing 12 (7):751-763. doi:10.1109/tip.2003.814242
16. Chan TF, Vese LA (2001) Active contours without edges. IEEE Transactions on Image Processing 10 (2):266-277. doi:10.1109/83.902291
17. Burges CJC (1998) A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery 2 (2):121-167. doi:10.1023/A:1009715923555
18. Barla A, Odone F, Verri A Histogram intersection kernel for image classification. In: International Conference on Image Processing, Italy, September 14-17 2003. IEEE, pp 513-516. doi:10.1109/ICIP.2003.1247294
19. Al-Akaidi M (2004) Fractal speech processing. Cambridge university press, New York 20. Scheunders P, Livens S, Van de Wouwer G, Vautrot P, Van Dyck D (1998) Wavelet-based texture analysis. International Journal on Computer Science and Information Management 1 (2):22-34
21. Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification. IEEE Transactions on Systems, Man and Cybernetics SMC-3 (6):610-621. doi:10.1109/TSMC.1973.4309314
22. Witten IH, Frank E, Hall MA (2011) Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, San Francisco
23. Papa JP, Falcao AX, Suzuki CT (2009) Supervised pattern classification based on optimum‐ path forest. International Journal of Imaging Systems and Technology 19 (2):120-131. doi:10.1002/ima.20188
24. Otsu N (1979) A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man and Cybernetics 9 (1):62-66. doi:10.1109/tsmc.1979.4310076 25. Freund Y, Schapire RE (1997) A desicion-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1):119-139. doi:10.1006/jcss.1997.1504
26. Breiman L (2001) Random forests. Machine learning 45 (1):5-32. doi:10.1023/A:1010933404324
27. Bengio Y (2009) Learning deep architectures for AI. Foundations and trends® in Machine Learning 2 (1):1-127. doi:10.1561/2200000006
