Multivariate Metric for Validation at Multiple Set Points:
Two Simple Forms
Table of Contents
1 Introduction ... 1
2 A Multivariate Metric for Results from Multiple Validation Set Points ... 3
2.1 Multivariate Metric definition ... 3
2.1.1 Evaluation of VD ... 3
2.1.2 Evaluation of Vnum ... 3
2.1.3 Evaluation of Vinput ... 4
2.2 Reference value of the Multivariate Metric ... 4
3 Proposed Exercise ... 5
3.1 Covariance Matrix without Input Uncertainty ... 5
3.2 Covariance Matrix with Input Uncertainty ... 5
4 Final Remarks ... 6
5 References ... 6
1 Introduction
This document presents two simple forms of the multivariate metric for Validation proposed by the ASME V&V 20 Standard Committee [1] as a supplement to the V&V 20-2009 standard [2].
The main goal of the multivariate metric is to assess the presence of significant modelling errors at multiple set points, whereas the V&V 20-2009 procedure provides an estimate of the interval that contains the modelling error at a single set point,
− ≤ ≤ + . 1
The comparison error and the validation uncertainty are the two quantities required to define this interval. is the difference between the simulation and the experimental measurement ,
= − , 2
of a quantity of interest and includes contributions from the experimental , numerical and parameter/input uncertainties. Independent , and
lead to
= + + . 3
The procedure proposed in [1] depends on the process for measuring the validation variables and it can be applied for the four situations described in [2]:
• Validation variables are directly measured (case 1);
• Validation variables are a result defined by a data reduction equation with or without shared error sources in the measured variables (case 2 and case3);
• Validation variables are evaluated from measured variables analyzed with a model (case 4).
Furthermore, the evaluation of the multivariate metric can also account for shared numerical, experimental and parameter/input errors at the multiple set points.
In the present document, we will restrict ourselves to two simple forms of the multivariate metric, which are based on the following assumptions:
a) The Validation variables are directly measured;
b) The numerical simulations do not share common errors at the multiple set points;
c) The experimental measurements do not share common errors at the multiple set points;
d) Errors in the input parameters are shared by all the set points.
The simplest form of the metric applies to the “strong” formulation of the model, i.e.
domain, equations and boundary conditions, which means that there is no parameter/input uncertainty = 0 [2]. On the other hand, the second alternative includes parameter/input uncertainties that can be evaluated using sensitivity coefficients determined by central-differences.
The multivariate metric provides a global assessment if the observed differences between measurements and simulations can be justified by the validation uncertainty. To this end, it defines a reference value for the metric that can be easily obtained if the uncertainty in all of the individual estimates of δmodel can each be represented by normal distributions. If the ratio between the multivariate metric and the reference value is much larger than one, there is an indication that the model is not able to reproduce the experimental data within the range of the validation uncertainty at each set point. Therefore, the point wise information is still essential to assess the level of the validation uncertainty. Furthermore, the multivariate metric can provide a global quantitative assessment of the comparison of
alternative mathematical models applied to the same problem. As for example, the simulation of a turbulent, viscous flow using different turbulence models [3].
2 A Multivariate Metric for Results from Multiple Validation Set Points
2.1 Multivariate Metric definition
The determination of the multivariate metric requires the following information:
• Selection of # Validation set points. These do not have to use the same type of variable;
• Comparison errors at each set point, = − ;
• The covariance matrix VVVVvalvalvalvalfor the comparison errors that depends on the experimental, numerical and input uncertainties.
The multivariate metric is defined by
= EEEET V V V V valvalvalval----1111 EEEE, 4 where EEEE is the array of rank #
EEEE= , -− -
−⋮ / 5
and VVVVvalvalvalval is the # × # covariance matrix, which for no shared error sources is given by
2345= 26+ 2789+ 2:7;8<. 6
2.1.1 Evaluation of VD
For experimental measurements with no shared error sources at the # set points, the matrix VD is a diagonal matrix given by
26=
>?
?@ .- 0 ⋯ 0 0 . 0 0 ⋮ 0 ⋱ 0 0 ⋯ 0 . CDDE
. 7
The experimental uncertainties , at the # set points can be determined with the techniques described in [2].
2.1.2 Evaluation of Vnum
For numerical simulations with no shared error sources at the # set points, the matrix Vnum is a diagonal matrix given by
2789=
>?
?@ .- 0 ⋯ 0 0 . 0 0 ⋮ 0 ⋱ 0 0 ⋯ 0 . CDDE
. 7
The numerical uncertainties , at the # set points can be determined with the techniques described in [2].
2.1.3 Evaluation of Vinput
The contribution of the input uncertainty to the covariance matrix in equation (6) for G uncertain input parameters that exhibit errors shared by all the set points is determined by
2:7;8<=HI2JHIK, 8
where HIis a # × G matrix containing the G sensitivity coefficients at the # set points and 2J is a # × # diagonal matrix including the uncertainties of the G input parameters MN
squared.
HI=
>?
??
@OOM-- ⋯ O -
⋮ ⋱ ⋮OM OOM- ⋯ O
OM CDDDE
9
and
2J=
>?
??
@ QR 0 ⋯ 0 0 QS 0 0
⋮ 0 ⋱ 0 0 ⋯ 0 QTCDDDE
10
As described in [2], a simple way to determine the sensitivity coefficients UVW
UQX is to apply finite-differences that require two extra simulations for each uncertain input parameter MN.
2.2 Reference value of the Multivariate Metric
If the uncertainty in all of the individual estimates of and can each be represented by normal distributions, then is distributed as Chi-squared, Y Z[ , with the degrees of freedom Z[ equal to the rank of Vval. If the measurements are independent, the rank of Vval
is equal to the number of set points. In these conditions, the expected value 〈 〉 and variance ^
T_S of the Y Z[ distribution are
〈 〉 = # 11
and
^T_S = var = 2#. 12 The sum of the expected value and the standard uncertainty (i.e., square root of the variance) of the Y Z[ distribution are used to define a reference value for the multivariate metric as
a b= # + √2#. 13
3 Proposed Exercise
The proposed assessment of the multivariate metric for Validation at multiple set points includes two different alternatives: (1) “strong formulation” of the model including computational domain, equations and boundary conditions, i.e. with and 2:7;8< equal to zero; input uncertainty included in the determination of the covariance matrix 2345. In both cases, the goal is to determine the ratio between the multivariate metric and the reference value a b given by
a b = dEEEET V V V V valvalvalval----1111 EEEE
# + √2#. 14
3.1 Covariance Matrix without Input Uncertainty
If 2:7;8< is set equal to zero and experiments and simulations do not share errors at the multiple set points, the covariance matrix VVVVvalvalvalval is a diagonal matrix of rank #, which is easily inverted leading to
= efg- ,
15
with
, = , + , . 16
This mean that the ratio ⁄ a b is just a weighted average of the comparison error at the different set points with the weights inversely proportional to each single set point validation uncertainty . In this case, the multivariate metric does not take into account any correlation between the different set points.
3.2 Covariance Matrix with Input Uncertainty
When the contribution of the parameter/input uncertainty is determined by equation (8), the covariance matrix VVVVvalvalvalval is a full matrix that will include the correlation between the
4 Final Remarks
The two simple forms of the multivariate metric presented in this document should be interpreted carefully. Obtaining ⁄ a b ≫ 1 gives a quantitative indication that mismatches between simulations and experiments cannot be globally justified by the validation uncertainties at the # set points. On the other hand, ⁄ a b< 1 is not a measure of success because the values of , may be larger than desirable. Therefore, the application of the multivariate metric is not a substitute of the point wise evaluation of and , which is still essential for the evaluation of the modelling error. However, the multivariate metric can take into account correlation between different validation set points and/or numerical and experimental data (not included in the two versions of this document).
5 References
[1] ASME (2019). ASME V&V 20.1 – Supplement to ASME V&V 20-2009 – Multivariate Metric for Validation, to appear in 2019.
[2] ASME (2009). ASME V&V 20-2009. Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer, 2009.
[3] Pereira F.S., Eça L. and Vaz G. - Verification and Validation exercises for the flow around the KVLCC2 tanker at model and full-scale Reynolds numbers - Ocean Engineering, Volume 129, 1 January 2017, Pages 133-148
