CONCLUDING REMARKS AND REFERENCES

CONCLUDING REMARKS AND REFERENCES 51

where

v ( n )

is a white noise process, and

a

¹

;::: ;a

M are constant coefficients (pa-rameters) of the AR model. The model order

M

gives the number of previous samples on which the current value

x ( n )

of the AR process depends. The noise term

v ( n )

introduces randomness into the model; without it the AR model would be completely deterministic. The coefficients

a

¹

;::: ;a

M of the AR model can be computed us-ing linear techniques from autocorrelation values estimated from the available data [419, 241, 169]. Since the AR models describe fairly well many natural stochastic processes, for example, speech signals, they are used in many applications. In ICA and BSS, they can be used to model the time correlations in each source process

s

i

( t )

. This sometimes improves greatly the performance of the algorithms.

Autoregressive processes are a special case of autoregressive moving average (ARMA) processes described by the difference equation

x ( n ) +

^XM

i⁼¹

a

i

x ( n

^?

i ) = v ( n ) +

^XN

i⁼¹

b

i

v ( n

^?

i )

^(2.127)

Clearly, the AR model (2.126) is obtained from the ARMA model (2.127) when the moving average (MA) coefficients

b

¹

;::: ;b

_N are all zero. On the other hand, if the AR coefficients

a

_i are all zero, the ARMA process (2.127) reduces to a MA process of order

N

. The ARMA and MA models can also be used to describe stochastic processes. However, they are applied less frequently, because estimation of their parameters requires nonlinear techniques [241, 419, 411]. See the Appendix of Chapter 19 for a discussion of the stability of the ARMA model and its utilization in digital filtering.

Problems

2.1 Derive a rule for computing the values of the cdf of the single variable gaussian (2.4) from the known tabulated values of the error function (2.5).

2.2 Let

x

¹

;x

²

;::: ;x

K be independent, identically distributed samples from a distribution having a cumulative density function

F

x

( x )

. Denote by

y

¹

;x

²

;::: ;y

K the sample set

x

¹

;x

²

;::: ;x

Kordered in increasing order.

2.2.1. Show that the cdf and pdf of

y

K ^{= maxf}

x

¹

;::: ;x

K^gare

F

yK

( y

K

) = [ F

x

( y

K

)]

^K

p

yK

( y

K

) = K [ F

x

( y

K

)]

^K?1

p

x

( y

K

)

2.2.2. Derive the respective expressions for the cdf and pdf of the random variable

y

^{1= minf}

x

¹

;::: ;x

K^g.

2.3 A two-dimensional random vector

x

⁼

( x

¹

;x

²

)

^T has the probability density function

p

^x

( x ^{) =}

(

1

2

( x

¹

+ 3 x

²

) x

¹

;x

²²

[0 ; 1]

0

^elsewhere

2.3.1. Show that this probability density is appropriately normalized.

2.3.2. Compute the cdf of the random vector

x

^.

2.3.3. Compute the marginal distributions

p

x¹

( x

¹

)

^and

p

x²

( x

²

)

^.

2.4 Computer the mean, second moment, and variance of a random variable dis-tributed uniformly in the interval

[ a;b ]

⁽

b > a

^).

2.5 Prove that expectations satisfy the linearity property (2.16).

2.6 Consider

n

scalar random variables

x

i^,

i = 1 ; 2 ;::: ;n

, having, respectively, the variances

²_xi. Show that if the random variables

x

iare mutually uncorrelated, the variance

_y²of their sum

y

^=Pn_i⁼¹

x

iequals the sum of the variances of the

x

i^:

_y²

=

^Xn

i⁼¹

_x²_i

2.7 Assume that

x

^{1 and}

x

² are zero-mean, correlated random variables. Any orthogonal transformation of

x

^1and

x

²can be represented in the form

y

¹

= cos( ) x

¹

+ sin( ) x

²

y

²

=

^?

sin( ) x

¹

+ cos( ) x

²

where the parameter

defines the rotation angle of coordinate axes. Let E^f

x

^21g=

^12,

E^f

x

^22g=

^{22, and Ef}

x

¹

x

^2g=

¹

². Find the angle

^{for which}

y

^1and

y

^2become

uncorrelated.

PROBLEMS 53

2.8 Consider the joint probability density of the random vectors

x ^{= ( x}

¹

^;x

²

⁾

^T

and

y ^{= y}

discussed in Example 2.6:

p

^x;^y

( x ^; y ^{) =}

(

( x

¹

+ 3 x

²

) y x

¹

;x

²²

[0 ; 1] ; y

²

[0 ; 1]

0

^elsewhere

2.8.1. Compute the marginal distributions

p

^x

( x ⁾

^,

^p

^y

⁽ y ⁾

^,

^p

x¹

( x

¹

)

^{, and}

p

x²

( x

²

)

^.

2.8.2. Verify that the claims made on the independence of

x

^1,

x

^{2, and}

y

ⁱⁿ

Example 2.6 hold.

2.9 Which conditions should the elements of the matrix

R ⁼

^{a b} _{c d}

satisfy so that

R

could be a valid autocorrelation matrix of 2.9.1. A two-dimensional random vector?

2.9.2. A stationary scalar-valued stochastic process?

2.10 Show that correlation and covariance matrices satisfy the relationships (2.26) and (2.32).

2.11 Work out Example 2.5 for the covariance matrix

C

^xof

x

, showing that similar results are obtained. Are the assumptions required the same?

2.12 Assume that the inverse

R

^?1x of the correlation matrix of the

n

-dimensional column random vector

x

exists. Show that

E^f

x

^T

R

^?1x

x

^g

^{= n}

2.13 Consider a two-dimensional gaussian random vector

x

with mean vector

m

^x

=

(2 ; 1)

^T and covariance matrix

C

^x

⁼

?

² 1 2

^?

¹

2.13.1. Find the eigenvalues and eigenvectors of

C

^x.

2.13.2. Draw a contour plot of the gaussian density similar to Figure 2.7.

2.14 Repeat the previous problem for a gaussian random vector

x

that has the mean vector

m

^x=

⁽

^?

^{2 ; 3)}

^T and covariance matrix

C

^x

⁼

?

² 2 5

^?

²

2.15 Assume that random variables

x

^and

y

are linear combinations of two uncor-related gaussian random variables

u

^and

v

, defined by

x = 3 u

^?

4 v

y = 2 u + v

Assume that the mean values and variances of both

u

^and

v

^{equal 1.}

2.15.1. Determine the mean values of

x

^and

y

^.

2.15.2. Find the variances of

x

^and

y

^.

2.15.3. Form the joint density function of

x

^and

y

^.

2.15.4. Find the conditional density of

y

^given

x

^.

2.16 Show that the skewness of a random variable having a symmetric pdf is zero.

2.17 Show that the kurtosis of a gaussian random variable is zero.

2.18 Show that random variables having

2.18.1. A uniform distribution in the interval

[

^?

a;a ] ;a > 0

, are subgaussian.

2.18.2. A Laplacian distribution are supergaussian.

2.19 The exponential density has the pdf

p

_x

( x ) =

(

exp(

^?

x ) x

0 0 x < 0

where

is a positive constant.

2.19.1. Compute the first characteristic function of the exponential distribution.

2.19.2. Using the characteristic function, determine the moments of the exponen-tial density.

2.20 A scalar random variable

x

has a gamma distribution if its pdf is given by

p

x

( x ) =

(

x

^b?1

exp(

^?

cx ) x

0

0 x < 0

where

b

^and

c

are positive numbers and the parameter

= c

^b

?( b )

is defined by the gamma function

?( b + 1) =

Z

1

0

y

^b

exp(

^?

y ) dy; b >

^?

1

The gamma function satisfies the generalized factorial condition

?( b + 1)

⁼

b ?( b )

^.

For integer values, this becomes

?( n + 1)

⁼

n !

^.

2.20.1. Show that if

b = 1

, the gamma distribution reduces to the standard exponential density.

2.20.2. Show that the first characteristic function of a gamma distributed random variable is

' ( ! ) = c

^b

( c

^?

|! )

^b

PROBLEMS 55

2.20.3. Using the previous result, determine the mean, second moment, and variance of the gamma distribution.

2.21 Let

k

( x )

^and

k

( y )

^{be the}

k

th-order cumulants of the scalar random variables

x

^and

y

, respectively.

2.21.1. Show that if

x

^and

y

are independent, then

k

( x + y ) =

k

( x ) +

k

( y )

2.21.2. Show that

_k

( x )

⁼

^k

_k

( x )

^{, where}

is a constant.

2.22 * Show that the power spectrum

S

_x

( ! )

is a real-valued, even, and periodic function of the angular frequency

!

^.

2.23 * Consider the stochastic process

y ( n ) = x ( n + k )

^?

x ( n

^?

k )

where

k

is a constant integer and

x ( n )

is a zero mean, wide-sense stationary stochastic process. Let the power spectrum of

x ( n )

^be

S

x

( ! )

and its autocorrelation sequence

r

x

(0) ;r

x

(1) ;:::

^.

2.23.1. Determine the autocorrelation sequence

r

y

( m )

of the process

y ( n )

^.

2.23.2. Show that the power spectrum of

y ( n )

^is

S

y

( ! ) = 4 S

x

( ! )sin

²

( k! )

2.24 * Consider the autoregressive process (2.126).

2.24.1. Show that the autocorrelation function of the AR process satisfies the difference equation

r

x

( l ) =

^?XM

i⁼¹

a

i

r

x

( l

^?

i ) ; l > 0

2.24.2. Using this result, show that the AR coefficients

a

ican be determined from the Yule-Walker equations

R

^x

a =

^?

r

^x

Here the autocorrelation matrix

R

^xdefined in (2.119) has the value

m

⁼

M

^?

1

^{, the}

vector

r

^x

^{= [ r}

x

(1) ;r

x

(2) ;::: ;r

x

( M )]

^T

and the coefficient vector

a = [ a

¹

;a

²

;::: ;a

_M

]

^T

2.24.3. Show that the variance of the white noise process

v ( n )

in (2.126) is related to the autocorrelation values by the formula

²_v

= r

x

(0) +

^XM

i⁼¹

a

i

r

x

( i )

Computer assignments

2.1 Generate samples of a two-dimensional gaussian random vector

x

having zero mean vector and the covariance matrix

C

^x

⁼

?

⁴ 1 2

^?

¹

Estimate the covariance matrix and compare it with the theoretical one for the fol-lowing numbers of samples, plotting the sample vectors in each case.

2.1.1.

K = 20

^.

2.1.2.

K = 200

^.

2.1.3.

K = 2000

^.

2.2 Consider generation of desired Laplacian random variables for simulation pur-poses.

2.2.1. Using the probability integral transformation, give a formula for generating samples of a scalar random variable with a desired Laplacian distribution from uniformly distributed samples.

2.2.2. Extend the preceding procedure so that you get samples of two Laplacian random variables with a desired mean vector and joint covariance matrix. (Hint: Use the eigenvector decomposition of the covariance matrix for generating the desired covariance matrix.)

2.2.3. Use your procedure for generating 200 samples of a two-dimensional Laplacian random variable

x

with a mean vector

m

^{x =}

(2 ;

^?

1)

^T and covariance matrix

C

^x

⁼

?

⁴ 1 2

^?

¹

Plot the generated samples.

2.3 * Consider the second-order autoregressive model described by the difference equation

x ( n ) + a

¹

x ( n

^?

1) + a

²

x ( n

^?

2) = v ( n )

Here

x ( n )

is the value of the process at time

n

^{, and}

v ( n )

is zero mean white gaussian noise with variance

²_v that “drives” the AR process. Generate 200 samples of the process using the initial values

x (0)

⁼

x (

^?

1)

= 0 and the following coefficient values.

Plot the resulting AR process in each case.

2.3.1.

a

¹

=

^?

0 : 1

^and

a

²

=

^?

0 : 8

^.

2.3.2.

a

¹

= 0 : 1

^and

a

²

=

^?

0 : 8

^.

2.3.3.

a

¹

=

^?

0 : 975

^and

a

²

= 0 : 95

^.

2.3.4.

a

¹

= 0 : 1

^and

a

²

=

^?

1 : 0

^.

3

Gradients and Optimization Methods

The main task in the independent component analysis (ICA) problem, formulated in Chapter 1, is to estimate a separating matrix

W

that will give us the independent components. It also became clear that

W

cannot generally be solved in closed form, that is, we cannot write it as some function of the sample or training set, whose value could be directly evaluated. Instead, the solution method is based on cost functions, also called objective functions or contrast functions. Solutions

W

to ICA are found at the minima or maxima of these functions. Several possible ICA cost functions will be given and discussed in detail in Parts II and III of this book. In general, statistical estimation is largely based on optimization of cost or objective functions, as will be seen in Chapter 4.

Minimization of multivariate functions, possibly under some constraints on the solutions, is the subject of optimization theory. In this chapter, we discuss some typical iterative optimization algorithms and their properties. Mostly, the algorithms are based on the gradients of the cost functions. Therefore, vector and matrix gradients are reviewed first, followed by the most typical ways to solve unconstrained and constrained optimization problems with gradient-type learning algorithms.

3.1 VECTOR AND MATRIX GRADIENTS

No documento Independent Component Analysis (páginas 73-79)