Nuisance parameters

with corresponding changes in the covariance matrix and thatZcan always be transformed todindependent standardized normal random variables. See also Note2.6.

It follows that the pivots

(θ− ˆθ)^Ti⁻¹(θ)(θ− ˆθ) (6.53) or

(θ− ˆθ)^Tˆ⁻¹(θ− ˆθ) (6.54) can be used to form approximate confidence regions forθ. In particular, the second, and more convenient, form produces a series of concentric similar ellipsoidal regions corresponding to different confidence levels.

The three quadratic statistics discussed in Section 6.3.4 take the forms, respectively

WE =(θˆ−θ0)^Ti(θ0)(θˆ−θ0), (6.55)

WL =2{l(θ)ˆ −l(θ0)}, (6.56)

WU =U(θ0;Y)^Ti⁻¹(θ0)U(θ0;Y). (6.57) Again we defer discussion of the relative merits of these until Sections6.6 and6.11.

6.4 Nuisance parameters

6.4.1 The information matrix

In the great majority of situations with a multidimensional parameterθ, we need to writeθ =(ψ,λ), whereψis the parameter of interest andλthe nuisance para-meter. Correspondingly we partitionU(θ;Y)into two componentsU_ψ(θ :Y) andU_λ(θ;Y). Similarly we partition the information matrix and its inverse in the form

i(θ)=

i_ψψ i_ψλ i_λψ i_λλ

, (6.58)

and

i⁻¹(θ)=

i^ψψ i^ψλ i^λψ i^λλ

. (6.59)

There are corresponding partitions for the observed informationˆ.

6.4.2 Main distributional results

A direct approach for inference about ψ is based on the maximum likeli-hood estimateψˆ which is asymptotically normal with meanψand covariance matrixi^ψψ or equivalentlyˆ^ψψ. In terms of a quadratic statistic, we have for testing whetherψ=ψ0the form

WE =(ψˆ −ψ0)^T(i^ψψ)⁻¹(ψˆ −ψ0) (6.60) with the possibility of using(ˆ^ψψ)⁻¹rather than(i^ψψ)⁻¹. Note also that even ifψ0were used in the calculation ofi^ψψit would still be necessary to estimateλ except for those special problems in which the information does not depend onλ. Now to studyψvia the gradient vector, as far as possible separated fromλ, it turns out to be helpful to write U_ψ as a linear combination of U_λ plus a term uncorrelated with U_λ, i.e., as a linear least squares regression plus an uncorrelated deviation. This representation is

U_ψ =i_ψλi⁻_λλ¹U_λ+U_ψ·λ, (6.61) say, whereU_ψ·λdenotes the deviation ofU_ψfrom its linear regression onU_λ. Then a direct calculation shows that

cov(U_ψ·λ)=i_ψψ·λ, (6.62)

where

i_ψψ·λ =i_ψψ −i_ψλi⁻_λλ¹i_λψ =(i^ψψ)⁻¹. (6.63) The second form follows from a general expression for the inverse of a partitioned matrix.

A further property of the adjusted gradient which follows by direct evaluation of the resulting matrix products by (6.63) is that

E(U_ψ·λ;θ+δ)=i_ψψ·λδ_ψ+O(δ²), (6.64) i.e., to the first order the adjusted gradient does not depend onλ. This has the important consequence that in using the gradient-based statistic to test a null hypothesisψ=ψ0, namely

WU =U_ψ·λ^T (ψ0,λ)i^ψψ(ψ0,λ)U_ψ·λ(ψ0,λ), (6.65) it is enough to replace λ by, for example, its maximum likelihood estimate givenψ0, or even by inefficient estimates.

The second version of the quadratic statistic (6.56), corresponding more directly to the likelihood function, requires the collapsing of the log likelihood into a function ofψalone, i.e., the elimination of dependence onλ.

6.4 Nuisance parameters 111

This might be achieved by a semi-Bayesian argument in whichλbut notψ is assigned a prior distribution but, in the spirit of the present discussion, it is done by maximization. For givenψwe defineλˆ_ψto be the maximum likelihood estimate ofλand then define the profile log likelihood ofψto be

lP(ψ)=l(ψ,λˆ_ψ), (6.66)

a function ofψalone and, of course, of the data. The analogue of the previous likelihood ratio statistic for testingψ=ψ0is now

WL =2{lP(ψ)ˆ −lP(ψ0)}. (6.67) Expansions of the log likelihood about the point (ψ0,λ) show that in the asymptotic expansion, we have to the first term thatWL = WU and there-fore thatWLhas a limiting chi-squared distribution withd_ψdegrees of freedom whenψ =ψ0. Further because of the relation between significance tests and confidence regions, the set of values ofψdefined as

{ψ: 2{lP(ψ)ˆ −lP(ψ)} ≤k_d^∗2

ψ;c} (6.68)

forms an approximate 1−clevel confidence set forψ.

6.4.3 More on profile likelihood

The possibility of obtaining tests and confidence sets from the profile log like-lihood lP(ψ) stems from the relation between the curvature oflP(ψ)at its maximum and the corresponding properties of the initial log likelihoodl(ψ,λ). To see this relation, let∇_ψand∇_λdenote thed_ψ×1 andd_λ×1 operations of partial differentiation with respect toψandλrespectively and letD_ψdenote total differentiation of any function ofψandλˆψ with respect toψ. Then, by the definition of total differentiation,

D^T_ψlP(ψ)= ∇_ψ^Tl(ψ,λˆψ)+ ∇_λ^Tl(ψ,λˆψ)D_ψ(λˆ^T_ψ}^T. (6.69) Now applyD_ψ again to get the Hessian matrix of the profile likelihood in the form

D_ψD^T_ψlP(ψ)= ∇_ψ∇_ψ^Tl(ψ,λˆ_ψ)+ {∇_ψ∇_λ^Tl(ψ,λˆ_ψ)(∇_ψλˆ^T_ψ)}^T

+ {∇_λ^Tl(ψ,λˆ_ψ)}{∇_ψ∇_ψ^Tλˆ^T_ψ} +(∇_ψλˆ^T)∇_λ∇_ψ^Tl(ψ,λˆ_ψ) +(∇ψλˆ^T_ψ){∇λ∇_λ^Tl(ψ,λˆψ)(∇ψλˆ^T_ψ)^T}. (6.70) The maximum likelihood estimate λˆψ satisfies for all ψ the equation

∇_λ^Tl(ψ,λˆ_ψ)=0. Differentiate totally with respect toψto give

∇ψ∇_λ^Tl(ψ,λˆψ)+(D_ψλˆ^T_ψ)∇λ∇_λ^Tl(ψ,λˆψ)=0. (6.71)

Thus three of the terms in (6.70) are equal except for sign and the third term is zero in the light of the definition ofλˆψ. Thus, eliminatingD_ψλˆψ, we have that the formal observed information matrix calculated as minus the Hessian matrix oflP(ψ)evaluated atψˆ is

ˆ

P,ψψ = ˆψψ− ˆψλˆ_λλ⁻¹ˆλψ = ˆψψ·λ, (6.72) where the two expressions on the right-hand side of (6.72) are calculated from l(ψ,λ). Thus the information matrix forψevaluated from the profile likelihood is the same as that evaluated via the full information matrix of all parameters.

This argument takes an especially simple form when bothψandλare scalar parameters.

6.4.4 Parameter orthogonality

An interesting special case arises wheni_λψ =0, so that approximatelyj_λψ =0.

The parameters are then said to beorthogonal. In particular, this implies that the corresponding maximum likelihood estimates are asymptotically independent and, by (6.71), thatD_ψλˆ_ψ =0 and, by symmetry, thatD_λψˆ_λ=0. In nonortho-gonal cases if ψ changes byO(1/√

n), thenλˆψ changes byOp(1/√ n); for orthogonal parameters, however, the change is Op(1/n). This property may be compared with that of orthogonality of factors in a balanced experimental design. There the point estimates of the main effect of one factor, being contrasts of marginal means, are not changed by assuming, say, that the main effects of the other factor are null. That is, theOp(1/n)term in the above discussion is in fact zero.

There are a number of advantages to having orthogonal or nearly orthogonal parameters, especially component parameters of interest. Independent errors of estimation may ease interpretation, stability of estimates of one parameter under changing assumptions about another can give added security to conclusions and convergence of numerical algorithms may be speeded. Nevertheless, so far as parameters of interest are concerned, subject-matter interpretability has primacy.

Example 6.4. Mixed parameterization of the exponential family.Consider a full exponential family problem in which the canonical parameterφ and the canonical statisticsare partitioned as(φ1,φ2)and(s1,s2)respectively, thought of as column vectors. Suppose that φ2is replaced by η2, the corresponding component of the mean parameter η = ∇k(φ), where k(φ)is the cumulant generating function occurring in the standard form for the family. Thenφ1and η2are orthogonal.

6.4 Nuisance parameters 113

To prove this, we find the Jacobian matrix of the transformation from(φ1,η2) to(φ1,φ2)in the form

∂(φ1,η2)/∂(φ1,φ2)=

I 0

∇1∇₂^Tk(φ) ∇2∇₂^Tk(φ)

. (6.73)

Here∇1denotes partial differentiation with respect toφlforl=1, 2.

Combination with (6.51) proves the required result. Thus, in the analysis of the 2×2 contingency table the difference of column means is orthogonal to the log odds ratio; in a normal distribution mean and variance are orthogonal.

Example 6.5. Proportional hazards Weibull model. For the Weibull distribution with density

γρ(ρy)^γ−1exp{−(ρy)^γ} (6.74) and survivor function, or one minus the cumulative distribution function,

exp{−(ρy)^γ}, (6.75)

the hazard function, being the ratio of the two, isγρ(ρy)^γ−1.

Suppose thatY1,. . .,Ynare independent random variables with Weibull dis-tributions all with the same γ. Suppose that there are explanatory variables z1,. . .,znsuch that the hazard is proportional toe^βz. This is achieved by writing the value of the parameterρcorresponding toYkin the form exp{(α+βzk)/γ}.

Here without loss of generality we takezk =0. In many applicationszandβ would be vectors but here, for simplicity, we take one-dimensional explanatory variables. The log likelihood is

(logγ+α+βzk)+(γ −1)logyk−exp(α+βzk)y_k^γ. (6.76) Direct evaluation now shows that, in particular,

E

− ∂²l

∂α∂β

=0, (6.77)

E

− ∂²l

∂β∂γ

= −βz_k²

γ , (6.78)

E

− ∂²l

∂γ ∂α

= −0.5771+α

γ , (6.79)

where Euler’s constant, 0.5771, arises from the integral _∞

0

vlogvdv. (6.80)

Now locally nearβ =0 the information elements involvingβ are zero or small implying local orthogonality ofβto the other parameters and in particular toγ. Thus not only are the errors of estimatingβalmost uncorrelated with those of the other parameters but, more importantly in some respects, the value of βˆ_γ will change only slowly withγ. In some applications this may mean that analysis based on the exponential distribution,γ =1, is relatively insensitive to that assumption, at least so far as the value of the maximum likelihood estimate ofβis concerned.