Defining the Meaning of a Major Modeling and Simulation Change as Applied to Accreditation

Increasingly, decisions are being made based partially or entirely on models. These decisions may be quite important, with the potential of serious or unacceptable consequences if an incorrect decision is made. The models used to support the decisions may have been newly developed or modified versions of existing models. In the former case, it seems clear that a new model should undergo validation before it is used for any significant application. In the latter case, the matter may be less clear; must the modified model, which presumably was validated when it was initially developed, be subject to another round of validation due to the modifications?

Several existing methods address the re-validation question, including methods developed by the Johns Hopkins University Applied Physics Laboratory, the Institute of Electrical and Electronics Engineers, the Joint Accreditation Support Agency, and the project’s sponsoring agency. These existing methods vary widely in several respects including level of detail, specificity with respect to modeling and simulation, degree of quantitativeness, ease of use, and applicability to the sponsoring agency. All include some form of the notion of risk, which is conventionally defined as the product of the likelihood of an incorrect decision and the consequences of such a decision.

A new method, the Quantitative-to-Qualitative Risk-based (QQR) method, was developed to make a quantitative recommendation regarding the re-validation of a modified model. The QQR method was developed with these goals in mind: to be quantitative, repeatable, and transparent; to consider both model modifications and model use risk; to focus on model types and simulation applications of interest to the sponsoring agency; and to be simple and accessible so as to encourage its use in practical applications.

The QQR method estimates the probability that not re-validating a modified model will lead to unacceptable consequences, given the modifications made to it. That estimated probability is meant to be interpreted, in the context of a re-validation decision, as a quantitative recommendation to re-validate the model. The method is based on a conditional probability formula that separates the various parts of the estimated probability into distinct terms and factors, and it provides procedures for estimating each term and factor. A central feature of the QQR method is a missions-means decomposition that allows both the method’s user to precisely and effectively identify the nature and extent of the modifications that were made to the model and the method to consider those modifications in its estimate.

The QQR method was validated using a set of re-validation scenarios, each describing a model, the modifications made to it, and the decision to be based on the model. The QQR method’s re-validation recommendations for the scenarios were compared to those of a set of human experts who were selected based on their expertise and experience in model validation. A suitable statistical measure of correlation between the QQR method’s recommendations and the experts’ recommendations for the scenarios was calculated. It showed strong positive correlation between the method and the experts.