There is the story of this colleague of mine, who is working in a big funded research project. Such projects bring together researchers and industrial partners, who provide use cases for research prototypes. Indeed my colleague had no problems to get access to a row of models that were used at the industrial partners’ sides. He used them successfully to evaluate his prototype and the cooperation was documented as official part of the overall project. However, all models remained confidential matter.

This is where the problem of my colleague began.

When publishing an evaluation or experiment it should be comprehensible and repeatable. However, this is not possible, when the used models cannot be shown or described. But: how to publish about his approach and prototype when the evaluation cannot be published, too? Was he going to produce yet another one of these papers that include the vague sentence: “We evaluated our prototype on the models of a big <<domain>> company from <<region>> and found that it can successfully be applied.”?

Most of us know this situation. While investing a lot of effort in industry cooperation to gain access to real problems, real use cases, and real models, researchers often end up not being able to publish evidence on their results – a bitter experience.

A class of pragmatic solutions: Ersatzmodels

During the discussion of the FMI’14 workshop in Vienna, it became clear that several researchers experiment with pragmatic solutions towards this problem. Although quite different in methodology and result, these solutions have a common target: the creation of Ersatzmodels, i.e. realistic models that can be used as substitute for real industrial models. During the workshop, we initially identified four types of Ersatzmodels, which are in use by researchers and differ in their costs and distance to real models.

• Generated Models: Models are automatically generated in order to gain different models for the evaluation of e.g. modeling tools. To make these models realistic, model generators might be build using individual real model.
• Pseudo Reference Models: When having exclusive access to a reasonable number of real but confidential models from one domain, scientists may manually create reference models, which shall reflect typical, reoccurring properties of the original models.
• Obfuscated Models: Single original models are systematically manipulated in order to reach confidentiality, while maintaining properties of interest. A typical goal here is to prevent that the company that provided the original models can be identified.
• Real Ersatzmodels: These models are created by researchers in close cooperation with a company, e.g. by simulating a project.

Hopes, Concerns, & To-Dos

However, most of these approaches are still in an experimental state. They are not widely accepted as standard research methods. To reach that the creation and usage of Ersatzmodels is accepted by research communities and reviewers, several so far open aspects need to be addressed.

Methods for creating the different forms of Ersatzmodels must move from “ideas with ad-hoc realization” towards well-documented standards. This is not only necessary to ensure comprehensibility, but also a precondition to discuss, design, and ensure quality of the resulting models and their applicability for further studies and experiments. Further, like for established research methods, it should be documented for what cases different Ersatzmodels can be used and for what cases not, i.e. what are typical threats to validity.

— Thus, Ersatzmodels are a promising, but still immature approach towards enabling research, while being exposed to pressure of both, quality needs, such as comprehensibility, repeatability, and reliability, on the one side and companies’ privacy needs on the other side.