All posts by alex

Model Review Criteria

Reviewing models, be it in academic exercises, of full-blown industrial designs and architectures, or as part of scientific research, is quite hard and it is non-obvious what good criteria for such an evaluation could be. Here is a story of a “model review”, told by Richard P. Feynman, in which he is considering a blueprint of a chemical plant (“Surely You’re Joking, Mr. Feynman!”, Vintage, 1992, pp. 123sqq.):

How do you look at a plant that isn’t built yet? I don’t know. Lieutenant Zumwalt […] takes me into this room where there are these two engineers and a loooooong table covered with a stack of blueprints representing the various floors of the proposed plant.

I took mechanical drawing when I was in school, but I am not good at reading blueprints. So they unroll the stack of blueprints and start to explain it to me, thinking I am a genius. Now, one of the things they had to avoid in the plant was accumulation. They had problems like when there’s an evaporator working, which is trying to accumulate the stuff, if the valve gets stuck or something like that and too much stuff accumulates, it’ll explode. So they explained to me that this plant is designed so that if any one valve gets stuck nothing will happen. It needs at least two valves everywhere.

Then they explain how it works. The carbon tetrachloride comes in here, the uranium nitrate from here comes in here, it goes up and down, it goes up through the floor, comes up through the pipes, coming up from the second floor, bluuuuurp—going through the stack of blueprints, down-up-down-up, talking very fast, explaining the very very complicated chemical plant.

I’m completely dazed. Worse, I don’t know what the symbols on the blueprint mean! There is some kind of a thing that at first I think is a window. It’s a square with a little cross in the middle, all over the damn place. I think it’s a window, but
no, it can’t be a window, because it isn’t always at the edge. I want to ask them what it is. […] I get an idea. Maybe it’s a valve. I take my finger and I put it down on one of the mysterious little crosses in the middle of one of the blueprints on page three, and I say “What happens if this valve gets stuck?” —figuring they’re going to say, “That’s not a valve, sir, that’s a window.”

So one looks at the other and says, “Well, if that valve gets stuck—” and he goes up and down on the blueprint, up and down, the other guy goes up and down, back and forth, back and forth, and they both look at each other. They turn around to me and they open their mouths like astonished fish and say “You’re absolutely right, sir.”

Looking at the story from a more general model review perspective, the fundamental model properties of mapping, reduction, and pragmatism coined by Herbert Stachowiak („Allgemeine Modelltheorie“, Springer, 1973) may be a first start. The blueprints seem to have worked rather well in this respect: The mapping in the blueprints made it possible to review a plant that had not been built yet, though, in fact, the mapping was not clear to all reviewers (and it is not told whether the plant could have been constructed successfully from these blueprints). The blueprints were pragmatically successful, i.e., the model was apt for the purpose of analysing whether only one valve getting stuck could lead to accumulation. However, heavy explanation was necessary, and still some mistakes could go undetected. It may be speculated that the reduction for the ultimate goal of fault analysis could have been more elaborate.

Are there other criteria that should be considered when reviewing a model? Maybe originality and novelty would be of some importance, in particular judging whether the right reduction has been chosen for some intended pragmatic purpose. But would for example also aesthetic criteria count? Or the (mis-)use of some arcane modelling language feature?