Suppose that we have access to n state variables, such as for instance numerical solutions of an analysis (degrees of freedom, data at Gauss points), or even experimental data (answers to a survey, strain probe, gray level of the pixels of a picture). Now suppose that we can get this information atmdifferent times, or formdifferent experimental conditions or random parameters values. We hence getmvectors of sizen, called snapshots and constituting the set{ui}mi=1. These vectors span a vector spaceRm. The objective now is to approximate it by another vector space of lower dimensionr. In other words, find a set of ordered orthonormal basis vectorsφi ri=1such that it approximates a random vectorxwhich takes its value inRnas:
x≈x˘r=
r
X
i=1
αi·φi (2.12)
Of course the representation (2.12) is not unique, and the number r of shape functionsφi required to obtain a correct approximation of x can be large. A very useful process to obtain an orthonormal basis of the subspace spanned by a vector set is the Gram-Schmidt algorithm,
detailed in the following section. However, it can be even more interesting to find a basis such that the approximation at rankp, hence usingpbasis vectors, is the best approximation at rankp, in the sense of a given norm. Of course, to be interesting, pshould be lower thann. To build a basis like this, the problem to solve is in fact to findφi ri=1that minimizes the mean square projection error of the snapshots:
Perr= 1 m·
m
X
i=1
ui−
r
X
j=1
αj·φj
2
(2.13)
Given the chosen accuracy, this ensures to use the minimal number of basis vectors for the approximation (2.12). Using such a basis, this approximation is then called theproper orthogonal decomposition(POD) ofxtruncated at the orderr.
The first works on the proper orthogonal decomposition date back to the very early 1900’s, with the research of Pearson [PEA 01]. It was originally developed in the field of statistics, aim- ing to express collections of multidimensional data using only some relevant, reduced dimensions to simplify the analysis. Since then, the POD concept has been applied to an impressive number of different fields such as for instance statistical analysis, data mining, image processing (and specifically image compression), mechanical engineering, and biomedical engineering. In fact, it is intimately connected to model reduction in the broad sense. Since almost all technological fields are dealing with an overwhelming quantity of data, in spite of the progress of the com- puter hardware, people were logically attracted by reducing their processing costs, explaining the widespread use of such techniques.
Consequently, three main methods emerged to build a POD in the discrete case, namely the principal component analysis (PCA), the Karhunen-Loève decomposition (KLD) and the singular value decomposition (SVD). They are in fact equivalent and yield similar results [LIA 02]. They are all described in the next sections.
3.2.1 Gram-Schmidt process
The Gram-Schmidt process is named for Jørgen Pedersen Gram and Erhard Schmidt [GRA 84].
It is a method to orthonormalize a set of vectors in an inner product space. Taking a set{ui}mi=1of vectors ofRn, it generates an orthogonal setφi mi=1that spans the same successive vector spaces:
Fj=Spann
φ1· · ·φjo
=Spann
u1· · ·ujo
∀j<m (2.14)
The main step of the algorithm consists in subtracting to the vectoruj+1its orthogonal pro- jection on the spaceFj. Let us first introduce the projection operator along the directionuby:
proju(u)=hu,ui
hu,ui·u (2.15)
Reduced basis approach: decreasing the computational cost during the propagation 67
Figure 2.1 Schematic principal component analysis of a dataset. The computed directionsηandζare those which maximize the variance of the data, originally expressed in terms ofxandy.
whereh ·,· idenotes the inner product inRn. The algorithm writes then:
u1=u1, φ1= u1 ku1k u2=u2−proju1(u2), φ2= u2
ku2k u3=u3−proju1(u3)−proju2(u3), φ3= u3
ku3k ... ... um=um−
m−1
X
j=1
projuj(um), φm= um kumk
(2.16)
In order to truncate the obtained basis at an orderr, a criterion on the norm of the obtained basis vectorφr is set up. Indeed, during the process the norm ofφris an indicator of how well the basis constituted by the set of orthonormal vectorsφi r−1i=1describes the initial set{ui}mi=1since it indicates how much “information” this extra vector adds to the basis. In short, the smaller the norm, the better the approximation.
It is highly emphasized that the Gram-Schmidt process does not necessarily yield the POD of a given set of vector, in the sense that the obtained orthonormal basis may not be optimal in the general case. However, it is a very practical tool since it can be applied in an incremental manner, allowing to enrich easily the basis with additional vectors. This is of a special interest in the context of solving evolution problems since the data coming from the solution vectors will be got, precisely, incrementally all along the computations.
Remark 19 In practice, for numerical stability reasons, thestabilizedGram-Schmidt algorithm is implemented, which is slightly different. Anyway, a Gram-Schmidt process should always be called twice on the set of vector that must be orthogonalized. Indeed, this is a specific requirement in order to obtain accurate numerical results.
3.2.2 PCA and KLD
The principal component analysis (PCA) [PEA 01,HOT 33] and the Karhunen-Loève decompo- sition (KLD) [KAR 43,LOÈ 55] are quite similar, both in their definition and in the way they
are computed. From a statistical viewpoint, they aim at building an orthonormal basis that maxi- mizes the variance of all the data along its directions. An artificial illustrating example is provided on Figure2.1. Qualitatively, the computed directionsηandζare those which contain the most information about the data, originally expressed in terms of x andy. Suppose that the set of snapshots{ui}mi=1is arranged in a matrixAsuch as:
A=
u1
1 u1
2 · · · u1m u21 u22 · · · u2m ... ... ... ...
un1 un2 · · · unm
(2.17)
Then thecovariancematrix is defined such as:
C=A AT (2.18)
The characteristics of this matrix are directly linked to the linear dependency of the snapshots among themselves. Indeed, a fully populated covariance matrix indicates very correlated snap- shots whereas a sparse one signalizes dependent entries. After that, computing the proper orthog- onal decomposition of the snapshots by whether the PCA or KLD method consist in solving the following eigenvalue problem:
Cφi=λiφi (2.19)
The orthonormal basis constituted by the set of vectorφi mi=1is the proper orthogonal decompo- sition.
Note that the matrixCis of dimensionn×n. Depending on the number of state variablesn compared to the number of instantsm, it can be interesting to solve the following modified prob- lem, which is ofsmallerdimensionm×m, using the so-called snapshot POD method [SIR 87]:
ATA
ui=ξiui φi= 1
√ξi Aui (2.20)
3.2.3 SVD
The singular value decomposition (SVD) is a powerful matrix factorization technique. It was introduced almost simultaneously by Beltrami and Jordan in the 1870’s and extended to complex rectangular matrices by Ekardt and Young in 1939. This method can be viewed as an extension of the eigenvalue decomposition for non-square matrices.
Computing the SVD of the matrix A (2.17) of dimensionsn×mconsists in computing its factorization under the form:
A=UΣVT (2.21)
where the superscriptTindicates matrix transpose, and
• U, orthogonal matrix of dimensionn×n, contains the left singular vectors ofA. The columns ofUconstitute in fact an orthonormal basis of the row space of A. In the present case, they are homogeneous to aspatialbasis of the data contained inA.
• V, orthogonal matrix of dimension m×m, contains the right singular vectors of A. Sym- metrically to the matrixU, The columns ofVconstitute an orthonormal basis of the column space ofA. In the present case, they are hence homogeneous to atemporalbasis of the data contained inA.
Reduced basis approach: decreasing the computational cost during the propagation 69
• Σ, diagonal matrix of dimensionn×m, containsp=min (n,m) nonnegative numbersσwhich are called the singular values ofAand are arranged, by convention, in decreasing order.
There is a strong link between the singular value decomposition and the principal component analysis of a matrix. Indeed, starting from the SVD of matrix A, it comes:
A AT =
UΣVT VΣTUT
=UΣVTVΣTUT
(2.22) The matrixVbeing an orthogonal matrixVTVis identity, andΣbeing a diagonal matrix:
A AT=UΣTΣUT
=UΣ2UT (2.23)
where we recognize the eigenvalue decomposition of A AT. Furthermore, the relation between the singular valuesσiand the PCA eigenvaluesλiof Acan then be deduced:
λi=σi2 (2.24)
A similar reasoning can be conducted withATA.
Now suppose that the truncated projection described in Equation (2.12) is performed using the rfirst columns ofU, or similarly thereigenvectorsφi of the PCA corresponding to ther greatest eigenvalues{λi}ri=1. A very interesting theorem [LIA 02] stands that these basis vectors constitute an optimal basis, which was the desired result, and also that the minimum error induced by the projection is equal to the square summation of them−rremaining singular values of the matrix A. This provides a truncation criterion to choose, with respect to a given desired error, how many basis vectors should be retained. It directly stems that observing the decreasing rate of the singular values gives a clear idea of how much the information contained in the matrix A will compress, in a general sense. This approach can be employed during a preliminary phase to estimate roughly the Kolmogorovn-width of a set of vector and thus the pertinence of applying a reduced basis method to it. Such kind of analysis is proposed several times in this typescript.
The computational cost of a singular value decomposition can be high, depending of course on the size of the matrix that must be decomposed. Therefore, in practice, variants of the SVD exist and among them, a very useful one is theThin SVD, in which only themfirst columns ofUare computed. Ifmn, it provides a significant reduction of the cost of the SVD computation. This situation arises quite often in numerical model reduction, since it corresponds to less snapshots vectors than degrees of freedom.
Remark 20 The singular value decomposition is a quite generic process and can be adapted to vectors or functions varying with respect to time, coordinates, and basically any kind of pa- rameters. It can also be used to compute low rank approximation of matrices. For instance in the context of image processing, it is often used to “compress” pictures, that is, from an initial picture, build one of a slightly lower quality but with a lower storage size. Such a process is pro- vided on Figure2.2, together with the normalized singular values of the initial image and various reconstructed images of different orders.