Sensitivity Analysis (SRC, SRRC, PCC, PRCC, SPEAR, PEAR, SMIRNOV  indices)

The purpose of uncertainty analysis is to determine the uncertainty in estimates for dependent variables of interest, see references [2], [7], [13], [18] and [19]. The purpose of sensitivity analysis is to determine the relationships between the uncertainty in the independent variables used in an analysis and the uncertainty in the resultant dependent variables. Uncertainty analysis typically precedes sensitivity analysis since, if the uncertainty in a dependent variable is under an acceptable bound or within an acceptable range, then there is little reason to perform a sensitivity analysis. Further, when Monte Carlo analysis is being performed, the generation of summary measures such as means, variances and distribution functions used to represent uncertainty requires little effort once the necessary model evaluations have been performed.

When a Monte Carlo study is being performed, propagation of the sample through the model creates a mapping from analysis inputs to analysis results of the form

[ yi,  xi1, xi2, ... , xin ],  i = 1,..,m


where n is the number of independent variables and m is the sample size.
Once this mapping is generated and stored, it can be explored in many ways to determine the sensitivity of model predictions to individual input variables. This section considers sensitivity analysis techniques based on regression analysis: PEAR and PCC, SRC, and their rank transformation (SPEA, PRCC and SRRC).


Scatterplots, PEAR and SPEA


The generation of scatterplots is undoubtedly the simplest sensitivity analysis technique. This approach consists of generating plots of the points ( xij, yi ) , i = 1, …, m, for each independent variable xi.
Scatterplots may sometimes completely reveal the relationship between model input and model predictions; this is often the case when only one or two inputs dominate the outcome of the analysis. Further, they often reveal non-linear relationships, thresholds and variable interactions, so facilitating the understanding of the model behaviour. They can be considered as a global measure of importance, and are model independent, as the plots can reveal even strongly non-linear or non-monotonic features.
When there is no relationship between the independent and the dependent variable, the individual points will be randomly spread over the plot. In contrast, the existence of a well-defined relationship between the independent and the dependent variable is often revealed by the distribution of the individual points.
One disadvantages of the method is that it needs generating and inspecting a large amount of plots: at least one per input factor, possibly multiplied by the number of time points if the output is time dependent. Further, scatterplots offer a qualitative measure of sensitivity: the relative importance of variables cannot be estimated but not quantified.
Another simple measure of sensitivity is given by the Pearson product moment correlation coefficient (PEAR) which is the usual linear correlation coefficient computed  on the xij, yi   (i = 1, …, m).
For non-linear models the Spearman coefficient (SPEA) is preferred as a measure of correlation, which is essentially the same as PEAR, but using the ranks of both Y and Xj  instead of the raw values i.e.,  SPEA( Y, Xj ) = PEAR( R(Y), R(Xj) ) where R (.) indicates the transformation which substitutes the variable value with its rank.

The basic assumptions underlying the Spearman coefficient are:

(a)    Both the xij and yi  are random samples from their respective populations;
(b)    the measurement scale of both variables is at least ordinal.

The numerical value of SPEA, commonly known as the Spearman ‘rho’, can also be used for hypothesis testing, to quantify the confidence in the correlation itself. The following hypothesis is first formulated:

H0 : “no correlation exists between Y and Xj ”;

then the value of SPEA (Y, Xj) is computed from a given number N of simulations and its value is compared with the quantiles of the Spearman test distribution. A level of significance α is selected which indicates the probability of erroneously rejecting the hypothesis, i.e. in this example, the probability that the test indicates correlation when Y and Xj  are actually uncorrelated. Then, the hypothesis H0 is rejected if SPEA falls out of the interval of extremes [W(α/2), W(1-α/2)], where W’s are the quantiles of the test distribution. The values of W can be read from the table of the distribution according to the values of α and N.

Regression analysis


More quantitative measures of sensitivity are based on regression analysis. A multivariate sample of the input x is generated by some sampling strategy (dimension m x k), and the corresponding sequence of m output values is computed using the model under analysis. If a linear regression model is being sought, it takes the form:

y= b+ ∑i bj xij + εi

where yi  = 1, …, m are the output values of the model, bj , j = 1, …,k ( k being the number of input variables) are coefficients that must be determined and εi is the error (residual) due to the approximation. One common way of determining the coefficients bj is using the least square method. In this least square approach, the bj’s are determined so that the  function

F(b)= ∑i εi2

is a minimum. Once the bj’s are computed, they can be used to indicate  the importance of individual input variables xj with respect to the uncertainty in the output y. In fact, assuming that b has been computed, the regression model can be rewritten as

where

The coefficients bjŝj/ŝ  are called Standardised Regression Coefficients (SRC). These can be used for sensitivity analysis (when the xj are independent) as they can quantify the effect of varying each input variable away from its mean by a fixed fraction of its variance while maintaining all other variables at their expected values.

When using the SRC’s it is also important to consider the model coefficient of determination,

where ŷj denotes the estimate of yj obtained from the regression model.

Ry2 provides a measure of how well the linear regression model based on SRC’s can reproduce the actual output yRy2 represents the fraction of the variance of the output explained by the regression. The closer Ry2 is to unit, the better is the model performance. The validity of the SRC’s as a measure of sensitivity is conditional on the degree to which the regression model fits the data, i.e. to Ry2.


Correlation measures


Another interesting measure of variable importance is given by Partial Correlation Coefficients (PCC). These coefficients are based on the concepts of correlation and partial correlation. For a sequence of (xij , yj) observations, the correlation rxjy between the input variable Xj and the output Y is defined by

where



The correlation coefficient rxjy provides a measure of the linear relationship between Xj and Y. The partial correlation coefficient between the output variable Y and the input variable Xj is obtained from the use of a sequence of regression models. First the following two models are constructed

Then, the results of these two regressions are used to define the new variables Y-Ŷ and Xj-Ẋj . The partial correlation coefficient between Y and Xj is defined as the correlation coefficient between Y-Ŷ and Xj-Ẋ. Thus, the partial correlation coefficients provide a measure of the strength of the linear relationship between two variables after a correction has been made for the linear effects of other variables in the analysis. In other words, PCC gives the strength of the correlation between Y and a given input Xj cleaned of any effect due to any correlation between Xand any of the X, i ≠ j .
In particular PCC’s provide a measure of variable importance that tends to exclude the effects of other variables.
However, in the particular case in which the input variables are uncorrelated, the order of variable importance based either on SRC’s or PCC’s (in their absolute values) is exactly the same.


Rank transformation


Regression analysis often performs poorly when the relationships between the input variables are non-linear. In this case, the value of the Ry2 coefficient computed on the raw values may be low. The problem associated with poor linear fits to nonlinear data can often be avoided with the use of the rank transformations.
The rank transform is a simple procedure which involves replacing the data with their corresponding ranks, i.e. assign rank 1 to the smallest observation and continue to rank N for the largest observation. The usual least square regression analysis is then performed entirely on these ranks
R(xk). The final regression equation expresses R(yi) in terms of R(xk).
The new value for Ry2 (on ranks) is then computed: if this new value is higher, then the new coefficients SRRC, Standardised Rank Regression Coefficients, can be used for sensitivity analysis instead of SRC’s. The ranked variables are more often used because the Ry2 associated with the SRC’s are generally lower than that associated with the SRRC’s, especially for non-linear models. The difference between the Ry2 , computed on the raw values and on the ranks, is a useful indicator of the non-linearity of the model.

The performance of the SRRC is shown to be extremely satisfactory when the model output varies linearly or at least monotonically with each independent variable. However, in the presence of strong monotonicity, the accuracy of result s may become dubious or completely misleading. Another limitation in the use of ranks is that the transformation alters the model being studied, so that the resulting sensitivity measures (e.g. SRRC) give us information on a different model. The new model is not only more linear but also more additive than the original one, i.e. more variation can be explained in terms of summing elementary effect than was the case with the original model. When this becomes, the SRRC become somehow a qualitative measure, as the relative importance of input cannot be assessed.
The PCC can be computed on the ranks (Partial Rank Correlation Coefficients). The performance of the PRCC shows the same features of performance as the SRCC: good for monotonic models, and not fully satisfactory in the presence of non-monotonicity. PCC’s provides related but not identical measures of variable importance.

Kolmogorov-Smirnov Test

Model simulations are classified as either behavioural (B) or non – behavioural (B). A set of binary elements are defined distinguishing between two sub-sets of each input factor Xi : (Xi|B) of m elements and (Xi|B) of n elements ( n + m = N, the total number of Monte Carlo runs performed). The Smirnov two-sample test is performed for each factor Xi independently. Under the null hypothesis that the two distributions fm(Xi|B) and fn(Xi|B) are identical:

H0 fm(Xi|B) = fn(Xi|B)

H1 fm(Xi|B)  fn(Xi|B)

The test statistic if defined by:

dm,n(Xi) = supy|| Fm(Xi|B) - Fn(Xi|B)||

where F are marginal cumulative probability functions and f are probability density functions. At what significance level α does the computed value of dm,n determines the rejection of H0 ? A low level of α implies a significant difference between fm(Xi|B) and fn(Xi|B), suggesting that Xi is a key factor in producing the splitting between B and B.
To perform the Smirnov test, we choose the significance level α, which is the probability of rejecting H0 when it is true (i.e. to recognize a factor as important when it is not). From α we derive the critical level Dα at which the computed value dm,n determines the rejection of H0. If dm,n > Dα then H0 is rejected at significance level α and the factor Xi is considered as important. Simlab uses α=5%.