Sensitivity Analysis (SRC, SRRC, PCC, PRCC, SPEAR, PEAR, SMIRNOV indices)
The purpose of uncertainty analysis is to determine the uncertainty in estimates for dependent variables of interest, see references [2], [7], [13], [18] and [19]. The purpose of sensitivity analysis is to determine the relationships between the uncertainty in the independent variables used in an analysis and the uncertainty in the resultant dependent variables. Uncertainty analysis typically precedes sensitivity analysis since, if the uncertainty in a dependent variable is under an acceptable bound or within an acceptable range, then there is little reason to perform a sensitivity analysis. Further, when Monte Carlo analysis is being performed, the generation of summary measures such as means, variances and distribution functions used to represent uncertainty requires little effort once the necessary model evaluations have been performed.
When a Monte Carlo study is being performed, propagation of the sample through the model creates a mapping from analysis inputs to analysis results of the form
where n is
the number of independent variables and m is the sample size.
Once this mapping is generated and stored, it can be explored in many
ways to determine the sensitivity of model predictions to individual
input variables. This section considers sensitivity analysis techniques
based on regression analysis: PEAR and PCC, SRC, and their rank
transformation (SPEA, PRCC and SRRC).
Scatterplots, PEAR and SPEA
The generation of scatterplots is undoubtedly the simplest sensitivity
analysis technique. This approach consists of generating plots of the
points ( xij,
yi ) , i = 1, …, m, for
each independent variable xi.
Scatterplots may sometimes completely reveal the relationship between
model input and model predictions; this is often the case when only one
or two inputs dominate the outcome of the analysis. Further, they often
reveal non-linear relationships, thresholds and variable interactions,
so facilitating the understanding of the model behaviour. They can be
considered as a global measure of importance, and are model
independent, as the plots can reveal even strongly non-linear or
non-monotonic features.
When there is no relationship between the independent and the dependent
variable, the individual points will be randomly spread over the plot.
In contrast, the existence of a well-defined relationship between the
independent and the dependent variable is often revealed by the
distribution of the individual points.
One disadvantages of the method is that it needs generating and
inspecting a large amount of plots: at least one per input factor,
possibly multiplied by the number of time points if the output is time
dependent. Further, scatterplots offer a qualitative measure of
sensitivity: the relative importance of variables cannot be estimated
but not quantified.
Another simple measure of sensitivity is given by the Pearson product moment
correlation coefficient (PEAR) which is the usual linear
correlation coefficient computed on the xij,
yi (i = 1, …, m).
For non-linear models the Spearman
coefficient (SPEA) is preferred as a measure of
correlation, which is essentially the same as PEAR, but using the ranks
of both Y
and Xj
instead of the raw values i.e., SPEA( Y, Xj
) = PEAR( R(Y), R(Xj) ) where R (.) indicates the
transformation which substitutes the variable value with its rank.
The basic assumptions underlying the Spearman coefficient are:
(a) Both the xij and
yi are random samples
from their respective populations;
(b) the measurement scale of both
variables is at least ordinal.
The numerical value of SPEA, commonly known as the Spearman
‘rho’, can also be used for hypothesis testing, to
quantify the confidence in the correlation itself. The following
hypothesis is first formulated:
H0 : “no correlation exists between Y and Xj ”;
then the value of SPEA
(Y, Xj) is computed from a given
number N
of simulations and its value is compared with the quantiles of the
Spearman test distribution. A level of significance α is
selected which indicates the probability of erroneously rejecting the
hypothesis, i.e. in this example, the probability that the test
indicates correlation when Y
and Xj
are actually uncorrelated. Then, the hypothesis H0 is rejected if SPEA falls out of
the interval of extremes [W(α/2),
W(1-α/2)], where W’s are
the quantiles of the test distribution. The values of W can be read from
the table of the distribution according to the values of α and N.
Regression analysis
More quantitative measures of sensitivity are based on regression analysis. A multivariate sample of the input x is generated by some sampling strategy (dimension m x k), and the corresponding sequence of m output values is computed using the model under analysis. If a linear regression model is being sought, it takes the form:
yi = b0 + ∑i bj xij + εi
where yi = 1, …, m are the output values of the model, bj , j = 1, …,k ( k being the number of input variables) are coefficients that must be determined and εi is the error (residual) due to the approximation. One common way of determining the coefficients bj is using the least square method. In this least square approach, the bj’s are determined so that the function
F(b)= ∑i εi2
is a minimum. Once the bj’s are computed, they can be used to indicate the importance of individual input variables xj with respect to the uncertainty in the output y. In fact, assuming that b has been computed, the regression model can be rewritten as
where
The coefficients bjŝj/ŝ are called Standardised Regression Coefficients (SRC). These can be used for sensitivity analysis (when the xj are independent) as they can quantify the effect of varying each input variable away from its mean by a fixed fraction of its variance while maintaining all other variables at their expected values.
When using the SRC’s it is also important to consider the model coefficient of determination,
where ŷj denotes the estimate of yj obtained from the regression model.
Ry2 provides a measure of how well the linear regression model based on SRC’s can reproduce the actual output y. Ry2 represents the fraction of the variance of the output explained by the regression. The closer Ry2 is to unit, the better is the model performance. The validity of the SRC’s as a measure of sensitivity is conditional on the degree to which the regression model fits the data, i.e. to Ry2.
Correlation measures
Another interesting measure of variable importance is given by Partial Correlation Coefficients
(PCC). These coefficients are based on the concepts of
correlation and partial correlation. For a sequence of (xij , yj) observations, the
correlation rxjy between the
input variable Xj and the output Y is defined by
where
The correlation coefficient rxjy provides a measure of the linear relationship between Xj and Y. The partial correlation coefficient between the output variable Y and the input variable Xj is obtained from the use of a sequence of regression models. First the following two models are constructed
Then, the results of these two regressions are used to define
the new variables Y-Ŷ
and Xj-Ẋj . The partial
correlation coefficient between Y
and Xj is defined as the
correlation coefficient between Y-Ŷ
and Xj-Ẋj . Thus,
the partial correlation coefficients provide a measure of the strength
of the linear relationship between two variables after a correction has
been made for the linear effects of other variables in the analysis. In
other words, PCC gives the strength of the correlation between Y and a given
input Xj cleaned of any
effect due to any correlation between Xj and any
of the Xj , i ≠ j .
In particular PCC’s provide a measure of variable importance
that tends to exclude the effects of other variables.
However, in the particular case in which the input variables are
uncorrelated, the order of variable importance based either on
SRC’s or PCC’s (in their absolute values) is
exactly the same.
Rank transformation
Regression analysis often performs poorly when the relationships
between the input variables are non-linear. In this case, the value of
the Ry2 coefficient
computed on the raw values may be low. The problem associated with poor
linear fits to nonlinear data can often be avoided with the use of the
rank transformations.
The rank transform is a simple procedure which involves replacing the
data with their corresponding ranks, i.e. assign rank 1 to the smallest
observation and continue to rank N for the largest observation. The usual least square regression analysis is then performed entirely on these ranks
R(xk). The final regression equation expresses R(yi) in terms of R(xk).
The new value for Ry2 (on ranks) is then computed: if this new value is higher, then the new coefficients SRRC, Standardised Rank Regression Coefficients, can be used for sensitivity analysis instead of SRC’s. The ranked variables are more often used because the Ry2 associated with the SRC’s are generally lower than that associated with the SRRC’s, especially for non-linear models. The difference between the Ry2 , computed on the raw values and on the ranks, is a useful indicator of the non-linearity of the model.
The performance of the SRRC
is shown to be extremely satisfactory when the model output varies
linearly or at least monotonically with each independent variable.
However, in the presence of strong monotonicity, the accuracy of result
s may become dubious or completely misleading. Another limitation in
the use of ranks is that the transformation alters the model being
studied, so that the resulting sensitivity measures (e.g. SRRC)
give us information on a different model. The new model is not only
more linear but also more additive than the original one, i.e. more
variation can be explained in terms of summing elementary effect than
was the case with the original model. When this becomes, the SRRC become somehow a qualitative measure, as the relative importance of input cannot be assessed.
The PCC can be computed on the ranks (Partial Rank Correlation Coefficients). The performance of the PRCC shows the same features of performance as the SRCC: good for monotonic models, and not fully satisfactory in the presence of non-monotonicity. PCC’s provides related but not identical measures of variable importance.
Kolmogorov-Smirnov Test
Model simulations are classified as either behavioural (B) or non – behavioural (B). A set of binary elements are defined distinguishing between two sub-sets of each input factor Xi : (Xi|B) of m elements and (Xi|B) of n elements ( n + m = N, the total number of Monte Carlo runs performed). The Smirnov two-sample test is performed for each factor Xi independently. Under the null hypothesis that the two distributions fm(Xi|B) and fn(Xi|B) are identical:
H0 : fm(Xi|B) = fn(Xi|B)
H1 : fm(Xi|B) ≠ fn(Xi|B)
The test statistic if defined by:
dm,n(Xi) = supy|| Fm(Xi|B) - Fn(Xi|B)||
where F are marginal cumulative probability functions and f are probability density functions. At what significance level α does the computed value of dm,n determines the rejection of H0 ? A low level of α implies a significant difference between fm(Xi|B) and fn(Xi|B), suggesting that Xi is a key factor in producing the splitting between B and B.
To perform the Smirnov test, we choose the significance level α, which is the probability of rejecting H0 when it is true (i.e. to recognize a factor as important when it is not). From α we derive the critical level Dα at which the computed value dm,n determines the rejection of H0. If dm,n > Dα then H0 is rejected at significance level α and the factor Xi is considered as important. Simlab uses α=5%.