Model repositories, as we collect them in the FMI Model Index, build an infrastructure for researchers to gain models for investigations and for evaluation of new model analysis or transformation techniques.

However, using model repositories substitutes the classical task of data collection and, thus, is a change in the research approach. This implies a big challenge for the research’s validity, especially the external and construction validity (as described by Wohlin et al. Experimentation in software engineering. Springer, 2012), which can be affected on two levels:

1. There is the correctness and the context of the captured and published models. Are these models correct and sufficiently complete to provide an appropriate picture of the reality? For example, is the published model from an early system design that was dismissed later on? Or was the model used for generation of productive code? Answers to these questions can only be provided by the persons who add the models to an repository.
2. Further, there is the question whether/how far the mix of models within a repository is representative and insights made on the basis of these models can be generalized. Selection biases within repositories seem to be not the exception but the rule, e.g. due to a specialization of the repository for certain domains, or due to the fact that most models are provided by a small group of researchers, only.

Surely the actual validity discussion always depends on the concrete research, studies or evaluations. Nonetheless, the information required for this discussion can only be provided by the publishers of the models or by operators of the model repositories. Moreover, for each repository it is clear that certain kinds of research cannot be done with the data. All these information might be provided by a repository in form of a “validity disclaimer”.

However, model repositories often do not include much metadata on the models. The better ones will provide information on the data source, e.g. the company the models stem from.

How can this shortcoming be resolved? Can operators of model repositories provide validity disclaimers? If yes how? Should they demand a basic set of information on validity threats from people who add models to these repositories? What happens when model repositories are filled by web crawlers instead of humans? To what extend can research results be reliable when a repository is used that does not provide some kind of a validity disclaimer?