In our experiments, we evaluated both the detection rate and latency using equation (3.5) for models with different contributions of discrimination and reconstruction losses. Specifically, we consider
1. ADscore(·=1) =LD, which uses only the discrimination loss;
2. ADscore(·=0) =LR, which uses only the reconstruction loss;
3. ADScore(0<·<1) = ·LD + (1≠·)LR, which uses a combination of the discrimination and reconstruction losses.
The obtained results are compared to the results of the works in MAD-GAN [(LI et al., 2019)]
and ALAD [(ZENATI et al., 2018b)], which also compute discrimination and reconstruction losses to detect intrusions using a GAN.
3.5.1 Detection Rates
We use the AUC as the performance metric to evaluate the detection of intrusions and compare our results with [(LI et al., 2019)] and [(ZENATI et al., 2018b)]. Thus, we obtain the ROC curves for the detection results of the data samples in the testing sets of the three considered datasets. Different contributions for the discriminant and reconstruction losses are investigated, and the model that achieves the highest AUC is considered the best one. Figures
10a, 10b, and 10c show the ROC curves obtained by our IDS for the SWaT, WADI, and NSL-KDD datasets, respectively. In the same way, Figures 11a, 11b, and 11c depict the ROC curves of the IDS proposed by MAD-GAN, and Figures 12a, 12b, and 12c exhibit the ROC curves of the IDS proposed by ALAD. In contrast to our IDS and MAD-GAN’s IDS, ALAD’s IDS explores anomaly detection scores that considers only LD, onlyLR and a combination of LD and LR without relying on a parameter ·.
Figure 10 – ROC curves of the proposed FID-GAN (a) SWaT dataset
(b) WADI dataset (c) NSL-KDD dataset
Source: The author (2023).
These ROC plots demonstrate that the proposed FID-GAN achieves higher AUCs when combining both the discrimination and reconstruction losses. Moreover, the use of only the reconstruction loss achieves better detection results than the use of only the discrimination loss for the SWaT and WADI datasets. Therefore, the reconstruction loss computation is shown to enhance the detection performance of FID-GAN. In addition, the AUC results of FID-GAN are higher than the AUC results of MAD-GAN [(LI et al., 2019)] for all considered models
Figure 11 – ROC curves of the IDS in MAD-GAN [(LI et al., 2019)]
(a) SWaT dataset
(b) WADI dataset (c) NSL-KDD dataset
Source: The author (2023).
and datasets. Therefore, our IDS is shown to achieve better detection results than the IDS proposed by MAD-GAN. On the other hand, FID-GAN and ALAD essentially achieve the same AUCs for the SWaT dataset. However, FID-GAN achieves significantly better detection results than ALAD for the WADI and the NSL-KDD datasets, which are more complex and more challenging to detect intrusions from, since their AUCs are, in general, lower than the AUCs of the SWaT dataset. In addition, in contrast to our proposal, ALAD also does not support pre-trained GAN models, i.e., previously trained GANs. Precisely, the Encoder proposed by our architecture is trained independently from the GAN, and can thus be easily applied to enhance existing GAN based IDSs. On the other hand, ALAD requires their third network, which is responsible for reconstructing data samples, to be trained along with the GAN, such that previously trained GANs have to be re-trained. Since training GANs is not always an easy task due to mode collapse and stabilization issues [(ARJOVSKY; BOTTOU, 2017; SRIVASTAVA
Figure 12 – ROC curves of the IDS in ALAD [(ZENATI et al., 2018b)]
(a) SWaT dataset
(b) WADI dataset (c) NSL-KDD dataset
Source: The author (2023).
et al., 2017; SALIMANS et al., 2016)], this is a disadvantage in the use of ALAD for improving existing GAN based IDSs.
Furthermore, we also evaluate the Equal Error Rate (EER), a performance metric derived from the ROC that represents the point where the false positive rate and the false negative rate are equal. Table 1 exhibits the EER values for the model that combinesLDandLR. According to the AUCs, FID-GAN achieves lower EER than MAD-GAN for all considered datasets, and lower EER than ALAD for the two more complex datasets.
Table 1 – Equal error rate (EER) SWaT WADI NSL-KDD FID-GAN 0.1861 0.2049 0.2844 MAD-GAN 0.2416 0.2280 0.2921 ALAD 0.1768 0.2295 0.3485
Source: The author (2023).
3.5.2 Detection Latency
Since the detection of intrusions is a latency constrained application, the anomaly detection score needs to be computed in a short time. This time mainly depends on the computation time of the discrimination and reconstruction losses. Therefore, we compare the detection latency of our proposed IDS to that of the IDS in [(LI et al., 2019)] and in [(ZENATI et al., 2018b)]
when considering only the discrimination loss, only the reconstruction loss and a combination of both losses. Figures 13a, 13b, and 13c show the latency obtained for the SWaT, WADI, and NSL-KDD datasets, respectively.
Figure 13 – Mean detection latency (a) SWaT dataset
(b) WADI dataset (c) NSL-KDD dataset
Source: The author (2023).
For the three considered datasets, our IDS and the IDS in MAD-GAN achieved the same detection latency when considering only the discrimination loss. On the other hand, the latency increased when the reconstruction loss was also considered. This is because finding the latent
representation of a sample and computing its reconstruction loss demands time. Although the detection latency has increased for these two IDSs, our IDS shows a much lower latency compared to that of MAD-GAN. While other works solve optimization problems during the detection of intrusions, the Encoder in our architecture enables a major reduction in the time taken to detect intrusions because it obtains the latent representation of patterns through a direct mapping. Our IDS is shown to achieve a detection latency at least 5.5 times lower than MAD-GAN’s IDS when only the reconstruction loss is used. Therefore, it is much more suitable for latency constrained applications, such as the detection of intrusions in CPSs. On the other hand, the IDS proposed by ALAD achieves the shortest detection latency for the three considered datasets. In contrast to our proposed architecture, ALAD does not model data as time series or use RNN-LSTM networks to consider dependencies among data. In fact, ALAD uses neural networks with only fully-connected and convolutional layers, and therefore does not suffer from the limited parallelization allowed by RNN-LSTM networks. Thus, it requires a lower computing time, and consequently a shorter detection latency than our solution. However, as already presented, ALAD’s IDS is also the one that achieves the poorest AUCs for the WADI and NSL-KDD datasets, which indicates that it may not work well for more complex datasets and more sophisticated attacks. Thus, our IDS is more suitable than ALAD’s IDS to detect intrusions in cyber-physical systems.