Further improvements to the algorithm

1. Cryptograms are added to the list.

2. The simulated annealing application is run with the results from the previous step, the current weight tuple, and the remainder of its input.

3. The current weight tuple is updated.

4. The new weight tuple rating is checked to see if there was a substantial improvement. There is a substantial improvement if the following condition holds:

Rating(W^′,R,X)≥Rating(W,R,X) +α.

If there is substantial improvement the algorithm jumps back to step 1, adding more results of heuristics to R.

If there is not, and it is theNth consecutive time there is not, the algorithm stops, returning the current weight tuple and its rating. If it is not the Nth consecutive time, the weight increment is reduced to half, and the algorithm jumps back to step 2.

The reason the algorithm is allowed to fail up toN consecutive times is that it may be the case the rating is not improving because the weight increment is too big. As the weight tuple achieves a better rating and the number of cryptograms increases, finer adjustments are necessary to continue improving. On the other hand, since more results are added every time there is a significant improvement in rating, there is an assurance that the rating did not increase by chance.

Results

In this chapter, we present the following:

• The performance results obtained using the algorithm to compute the weight tuples. Given that the fine-tuning algorithm improves the weight tuple based on the performance of the classifier itself, just by analysing the results of the fine-tuning we can already have an idea of how both the algorithm and the classifier perform;

• The input that is given to our setup to get the results, as well as some details regarding the implementation of our classifier;

• An overview of how good the heuristics are at telling ciphers apart, focussing on their precision and splitting-ability. This is in contrast to what was done in Section 4.3, where we focussed on the consistency factor.

5.1 Setup for computing results

To put our classifier into action, we programmed the algorithm in Python along with the generation of cryptograms. The language was chosen for its simplicity.

In our program we avoided the use of floating-point variables in order to avoid precision loss in our calculations. Instead, we used rational numbers from the fractions Python module.

The downside of this is that operations with fractions take more time. Furthermore, after operating (adding, multiplying, etc.) a lot of fractions, there is a tendency for the numerator and denominator to become larger and larger, making the operations slower. This also makes it impossible for the user to understand the order of magnitude of these values. For these reasons, we did not fully remove floating-point usage from our code, but avoided it as much as possible where it made sense to do so.

In order to make the observations that follow, the simulated annealing algorithm was run 200 times, 100 times for each of the two identification measurement functions. Information about

69

the algorithm performance was collected on each run of the algorithm and compiled into a data set, which had 200 entries. Each entry had information of a different fine-tuning program run, namely:

• The identification function;

• The elapsed time;

• The number of iterations to converge;

• The number of cryptograms used;

• The rating of the final weight tuple;

• The number of correct guesses for the final weight tuple;

• The number of correct guesses plus the number of ties at first place for the final weight tuple;

• The weights of each heuristic in the final weight tuple.

The setup that follows is what we used to run the algorithm. The corpus used was the same as that we used for our observations in Chapter3. In Chapter4, two types of input are presented for the algorithm: the input necessary for the simulated annealing part of the algorithm and the input of the fine-tuning part of the algorithm, the “outer layer”. The input to the fine-tuning part of the algorithm was the following:

• Necessary improvement between iterations: ₁₀₀¹ ;

• Number of simulated annealing attempts: 3;

• Initial weight increment: ¹₆, (= _|H|¹ );

• Initial weights: _|C|¹ for c∈C;

• Number of cryptograms added per iteration: 3600. This number comes from the fact that for each iteration 300 texts are selected and each is ciphered with all ciphers inC, resulting in 300×12=3600 cryptograms. The cryptograms that were added in the last iteration of simulated annealing are not taken into account for our observations since they have no impact on the results.

For the cryptogram generation of periodic ciphers, we defined the minimum period to be 5 and the maximum 20. The size of the plaintexts used to create the cryptograms was 500. The input for the simulated annealing part of the algorithm was the following:

• Acceptable fallback ratio: ₂₀¹;

• Size of the sliding window of the previous tuples: 12;

• Minimum improvement value: ₂₀₀¹ ;

• Maximum number of attempts to improve the rating for simulated annealing: 3.

These inputs were chosen through trial and error, taking in consideration both the rating and the amount of correctly identified cryptograms. The program runtime was also taken into consideration.