Holdout Test Validation Results

In this subsection, we present the validation of the best trained classifiers discussed in the last sub- section, as well as a big picture on the classifiers’ validation using all of the proposed preprocessing techniques, similarly to the previous subsection. The best classifier, as referred before, is the Na¨ıve Bayes without preprocessing, for both GRACE+ and BL. The validation results of this classifier are repre- sented in Fig.4.5. Note that for validation, the results did not emerge from multiple repetitions, in contrary to training from which we had, for each experiment, five repetitions results. In this case, the results arrive from testing the classifiers in the holdout 25% validation dataset. The validation test is expected to result in overall worse scores than the training, since this dataset is much more unbiased and did not directly influence the training in any aspect. Thus, the classifiers are tested as whether they have small decreases and generalize well or great decreases and are poorer generalizers using new datasets.

Regarding the effects of GRACE+, we can observe that the validation results are relatively more positive than the training values, i.e. the overall results show that the expected decrease in validation is mitigated when using GRACE+. Relatively to BL, GRACE+ prediction accuracy undergoes a quite insignificant decrease (82.05% to 81.77%), while AUCW yields a relatively greater value (88.66% to 89.98%). Sensitivity, consistently with the training, decreases with GRACE+ (82.85% to 79.95%) but is compensated by a larger increase in specificity (80.85% to 85.82%).

Figure 4.5: Validation results using Baseline and GRACE+ datasets for the best trained classifier, Na¨ıve Bayes without preprocessing.

Validation results indicate that the NB classifier is a robust technique, as the values with this holdout dataset lie close to the values obtained from training (Fig. 4.3). Furthermore, comparing the two types of results, we observe that GRACE+ undergo a smaller decrease when generalizing. For instance, with GRACE+, specificity maintained the same for both train (85.8%) and validation (85.8%), sensitivity just slightly changed (from 79.2% to 79.04%), as well as prediction accuracy (from 81.89% to 81.77%) and

AUCW (from 91.1% to 89.98%). Regarding the BL dataset, such generalization power is not observed, as BL dataset yielded quite better results in the training relatively to the validation: prediction accuracy went from 83.28% in training to 82.05% in validation, AUCW from 91.1% to 88.7%, sensitivity from 86.2%

to 82.86%. Specificity, surprisingly, increases in the validation test, going from 79% to 80.9%, indicating that the BL models were already, to some extend, quite good generalizers.

Figure 4.6: Validation results using BL and GRACE+ datasets, across the top classifiers which yielded the highest AUCW in training.

Table 4.4: Differences between the training results and the validation results. 4represents the difference of those values from GRACE+ and BL. Positive 4 means GRACE+ generalized better than BL, i.e. relatively to BL, the classifier attained a lower reduction from training to validation. (*) classifier which yields better results in training using BL, and in validation using GRACE+. (^#) classifier which yields better results in training using GRACE+, and in validation using BL.

The validation results from the different classifiers are shown in Fig. 4.6. With this illustration, along with the comparisons between training and validation results presented in Table 4.4, we can observe quite good results with GRACE+ comparing to BL. A majority of classifiers yield better generalization power with GRACE+ than using BL. Small differences are observed between train and validation for most of the classifiers, except DT and SVM’s, which indicates that most classifier did not overfit to the dataset.

The different aspects can be summarized in the following:

• An overall improved result using GRACE+ is observed when comparing to BL. This is not the case for prediction accuracy of RF and sensitivity of NB and RF.

• Sensitivity, which was recurrently worse with GRACE+, demonstrates a different pattern for some classifiers, including kNN and Logistic Regression, in which Grace+ attains greater values than BL;

• Decision Tree and SVM’s perform a poor generalization, as we observe a significant decrease from training to validation results, in both BL and GRACE+ (as can be seen in Table4.4). For instance, DT, which yielded AUCW of 82.8% (GRACE) and 84.1% (BL) in training, culminates in 62.6% and 50.2%, respectively, with the validation dataset;

• kNN, Random Forest and Na¨ıve Bayes perform well in generalizing the classification and this ability is not significantly altered using GRACE+ instead of BL;

• Logistic regression proves to be an exceptionally good generalizer when using GRACE+. Logistic regression AUCW is actually greater in the validation test (91.47%) than in the training (90.57%);

• Specificity manifests the same pattern as the training set, being higher in all classifiers when using GRACE+ instead of BL.

To assess results variability from the different preprocessing techniques, we show the deviation of the classifiers’ performance including all the preprocessing combinations, and thus we are able to analyse the effect these techniques have in the variance of the validation results (Fig.4.7).

Figure 4.7:Validation results using Baseline and GRACE+ datasets, across all the classifiers, including all prepro- cessing techniques combination for each classifier. Some of the poorer results are not shown in order to adjust the scale of visualization.

Table 4.5: Validation results using Baseline and GRACE+ datasets, across all the classifiers, including all prepro- cessing techniques combination for each classifier.

As we can see, from Fig.4.7and complementary Table4.5, the validation values indicate great gen- eralization for the GRACE+ models, as well as an increased robustness from different preprocessing methodologies. This indicates that, for a general dataset, estimating each patient’s disease stage and in- corporate it in a MCI prognosis data mining approach is favourable for the purpose of attaining improved predictors. In fact, this greater likelihood of the model to attain good results in a general dataset is em- phasized by two aspects of this analysis: first, because GRACE+ attains recurrently better results in a validation dataset and second, because GRACE+ exhibits smaller variance around different preprocess- ing techniques, whose benefits are known to depend on the given dataset.