Home // International Journal On Advances in Systems and Measurements, volume 9, numbers 3 and 4, 2016 // View article
Authors:
Wim De Mulder
Bernhard Rengs
Geert Molenberghs
Thomas Fent
Geert Verbeke
Keywords: Gaussian process emulation; Agent-based models; Validation
Abstract:
A common way to evaluate surrogate models is by using validation measures. This amounts to applying a chosen validation measure to a test data set that was not used to train the surrogate model. The selection of a validation measure is typically motivated by diverse guidelines, such as simplicity of the measure, ease of implementation, popularity of the measure, etc., which are often not related to characteristics of the measure itself. However, it should be recognized that the validity of a model is not only dependent on the model, as desired, but also on the behavior of the chosen validation measure. Some, although very limited, research has been devoted to the evaluation of validation measures, by applying them to a given model that is trained on a data set with some known properties, and then evaluating whether the considered measures validate the model in an expected way. In this paper, we perform an evaluation of some statistical and non statistical validation measures from another point of view. We consider a test data set generated by an agent-based model and we successively remove those elements from it for which our previously developed Gaussian process emulator, a surrogate model, produces the worst approximation to the true output value, according to a selected validation measure. All considered validation measures are then applied to the sequence of increasingly smaller test data sets. It is desired that a validation measure shows improvement of a model when test data points on which the model poorly performs are removed, irrespective of the validation measure that is used to detect such data points. Our experiments show that only the considered statistical validation measures have this desired behavior.
Pages: 188 to 198
Copyright: Copyright (c) to authors, 2016. Used with permission.
Publication date: December 31, 2016
Published in: journal
ISSN: 1942-261x