Theoretical properties of the GDT etimators

5.4. Theoretical properties of the GDT etimators – with assumptions 2, the GDT solution is asymptotically optimal for a particular0< λ^∗n <1which decreases

to 0 as the training sample sizenincreases.

The first statement is very similar to a result given Chapter 2. We refer the reader to the Proposition 1 page 37, in which the discriminative estimator can be replaced by any GDT estimator withλ <1. This result is intuitive because the ML estimatorθˆ1estimates the true model parametersθ0which also minimizes the classification loss since the Bayes rule is optimal atθ0. The second case is more interesting, since it justifies the use of the GDT estimation when the conditional distribution is true, but the joint distribution is false. We give the main result of this chapter :

Theorem 1 If the following conditions are satisfied :

(i) For allε >0, the solution ofmaxθ∈Θlogp(Y|X;θ) +εlogp(X, Y;θ)is unique,

(ii) the conditional model belongs to the parametric family, i.e. there existθ0 ∈Θsuch thatlogp(Y|X;θ0) = logp(Y|X),

(iii) the asymptotic generative solutionθ^∗₁ = limn→ ∞θ⁽ⁿ⁾₁ has a higher classification loss than the asymptotic discriminative solutionθ₀^∗= limn→ ∞θ⁽ⁿ⁾₀ , i.e.E[logp(Y|X;θ^∗₁)]< E[logp(Y|X;θ₀^∗)].

(iv) the model pdfp(X, Y;θ)is twice differentiable with respect to the parameterθaroundθ₀^∗and the second derivative is bounded,

then the valueλ^∗_n = minλ∈[0,1]ηn(λ)tends to 0 asntends to∞. Moreover, fornsufficiently large, there exists a constantC >0such that

C

n ≤λ^∗_n<1. (5.14)

The proof of this theorem is given in appendix B. The main point is that the optimal value ofλis neither 0 nor1, there are intermediary estimates (λ ∈]0,1[) that have better generalisation performance than the standard generative and discriminative estimators. In addition, the theorem gives a lower bound on the convergence speed ofλ^∗_nto 0.

We apply this theorem to the LinearGDT estimator, i.e. when we use multivariate Gaussian distributions with a shared covariance matrix to model the class-conditional pdfs. This characterizes a class of distributions for which the GDT estimate has better generalization performance than LDA and linear logistic regression.

Corollary 1 If the sample distribution satisfies these conditions (i) and (ii) :

(i) There existβ∈R^dsuch thatp(Y = 1|X) = exp(βX)p(Y = 2|X), (ii) p(Y = 1|X)is not a Gaussian distribution,

then, for a sufficiently large training sizen, the expected loss of the LinearGDT estimator is strictly smaller than the expected loss of LDA and the linear logistic regression.

In other words, Corollary 1 states that the optimal tuning parameterλ^∗_n = minλ∈[0,1]ηn(λ)satisfies :

ηn(λ^∗n)< ηn(0) (better than the linear logistic regression) and ηn(λ^∗_n)< ηn(1) (better than the linear discriminant solution).

Hence, for any non-Gaussian distribution for which the linear logistic model is true, the GDT estimation with λ=λ^∗_nhave better prediction performances than LDA and the logistic regression.

5.4.2 Choice of λ .

In practice, the valueλ^∗_nis not directly available as it requires knowledge of the sample distributionp(X, Y).

The tuning parameterλfunctions like the smoothing parameter in regularization methods.λcannot be set on the basis of minimum classification loss on the training set, since by definition,λ = 0gives the optimalθ for training set classification. Instead,λis set to the valueλˆthat minimizes the cross-validated classification loss

λˆ= argmin

λ∈Λ

ν i=1

−logp(y⁽⁻ⁱ⁾|x⁽⁻ⁱ⁾; ˆθ⁽ⁱ⁾_λ ) (5.15)

whereΛis a set containing possible values ofλ(between 0 and 1),θˆ⁽ⁱ⁾_λ is the GDT parameter estimate based on the i^thtraining set and(x⁽⁻ⁱ⁾,y⁽⁻ⁱ⁾)is thei^thvalidation set. It might seen more natural to minimize 0-1 classification loss at this point, but experimentally, we find that it leads to less stable estimates ofλ.

If the optimal λˆ is close to one, the generative classifier is preferred. This suggests that the bias inpθ(x, y) (if any) does not affect the discrimination of the model too much. Similarly, if λˆis close to zero, it suggests that the modelpθ(x, y)does not fit the data well, and the bias of the generative classifier is too high to provide good classification results. In this case, a more complex model — i.e. with more parameters, or less constrained ones —

5.4. Theoretical properties of the GDT etimators may be needed to reduce the bias. For intermediateλ, there is an equilibrium between the bias and the variance,ˆ meaning that the model complexity is well adapted to the amount of training data.

It would be usefull to find computationally efficient alternatives to cross-validation for estimating the value of λthat give good classification performance on the test data.

5.4.3 Bias-variance decomposition

We now give some theoretical insights about the optimal value forλ. The classification error can be decompo- sed into bias and variance terms. The relative importance of these two terms is controlled by the GDT parameter λ. This explains why the generalization performance of the method is related to the choice of lambda.

θ^∗_λ= lim

n→ ∞θˆ⁽ⁿ⁾_λ

By the law of large numbers, the parametersθ^∗_λ are the expected values of their respective estimator θˆ_λ⁽ⁿ⁾. We denoteθ_λ^∗the values of the parameters minimizing the expected GDT loss function :

θ^∗_λ= argmin

θ∈Θ E[λlogp(X, Y;θ) + (1−λ) logp(Y|X;θ)]. (5.16) We assumed that the parameter estimates are consistent, i.e. thatθˆλ→θ_λ^∗almost surely asntends to infinity.¹⁹.

The “best” parameter in terms of classification loss isθ^∗_C = θ^∗₀, for which the discriminative estimator θˆ0is unbiased. However, this unbiasedness is associated with a high estimation variance. The GDT estimator, allows some bias in return for lower variance. Without loss of generality, we look for the estimator θˆλ minimizing the quantityη(λ)defined in (5.12). LetLC(θ) =E[logp(Y|X;θ)]. The overall classification lossη(λ)can be split into bias and variance terms :

η(λ) =E

LC(ˆθλ)−LC(θ_λ^∗)

variance(λ)

+LC(θ^∗_λ)−LC(θ^∗_C)

bias(λ)

+bias²₀ (5.17)

where bias²₀is the irreducible minimal model biasLC(θ^∗_C)−E[logp(Y|X)]. Due to the fact that the generative estimation is the minimum variance estimator ofθ, variance(λ)is minimal forλ= 1(the proof is found in the

19Sometimes the discriminative solution is not unique, but the GDT estimators forλ > 0are consistent. In this case we defineθ^∗₀ = lim_λ→0θ^∗_λ. This appears for example in the NB classifier and the LDA model.

appendix). Conversely, bias(0) = 0i.e. the bias term is minimum for the discriminative estimator. As for many other learning methods, the estimator should be chosen in order to balance these two quantities.