How to right-size your model

By Jack Homer, VP of Professional Practice

I once developed a relatively compact SD model to help a major chemical company restructure their plastics business.  The model was well received by senior management but was not considered granular enough for accurately projecting the financial consequences of a strategic shift.  The company had recently developed a rich dataset on the detailed costs of their operations, and they wanted these data incorporated into the model.

I spent several months expanding and deepening the model so that it could take these operational data into account—reflecting all the additional elements of structure the new data logically implied.  I presented the expanded model again to the management team, who seemed overwhelmed by all its detail and clearly did not trust it as much as they did the original smaller model.  This outcome surprised me: weren’t they the ones who had asked for the greater granularity in the first place?

In SD practice, we always face the question of how large and detailed a model should be. We cannot depend on clients to answer this question, because, as in the case above, they typically do not understand beforehand how excessive model detail can interfere with clarity and comprehensibility. 

Moreover, a bigger and more complete model is not necessarily a more reliable and trustworthy model, unless much more time and effort are invested to develop the necessary supporting evidence and do adequate sensitivity testing.  Doubling the number of variables in a model may lead, for example, to tripling or quadrupling the number of parameter values that need to be estimated and the amount of sensitivity testing that needs to be done. 

How, then, can we “right-size” our models?  After 40 years of doing SD, I still sometimes wrestle with this question, an indication that there is no simple or uniform answer.  But I’ll try to explain my usual approach.

Let’s first recognize that size has two components: model breadth or boundary, and level of aggregation.  I find that model boundary setting is usually not so hard.  When first developing a model, I set the initial model boundary based mostly on what the clients have to say about potential interventions as well as outcome measures of interest.  The boundary may change naturally over the course of a modeling project, growing larger as more issues are addressed, or sometimes shrinking as some variables are dropped for lack of relevancy or significance.  Such changes in boundary are made with little trouble in most cases. 

Level of aggregation, or the degree of “lumping”, on the other hand, is a more permanent decision, typically touching more parts of the model—and always causes me more concern.  How many demographic categories should I consider in modeling population health and disease?  How many sectors in modeling a national economy?  How many product or service lines in modeling a company? 

Clients are often used to seeing data broken out by multiple categories (as on a spreadsheet) and may push the modeler in the direction of greater disaggregation.  But SD modelers need to think for themselves.  They need to ask whether there is any strategic distinction among the multiple categories; that is, any way in which the interventions under consideration may cause the various categories to change in their relative proportions or move in different directions.  If the answer is yes, then disaggregation may be warranted; if no, then you can safely lump the categories together and deal in terms of weighted averages.

Data availability should also play a role in the aggregation decision.  I often find that data are available in disaggregated form for a few of the key variables but not most of them.  In this case, a decision to disaggregate will introduce more uncertainty about parameter values and could cast doubt on model results. 

In sum, you should disaggregate only when it serves the model’s strategic purpose and when the disaggregation is broadly supported by evidence.  If you get the right level of aggregation, the right size model will usually follow.

2 thoughts on “How to right-size your model

  1. I generally build for the same problem, different models of different boundaries, level of detail and even structure (CLD) and I verify that their results are compatible. I choose the model I am studying depending on the needs of the moment: if I need a high view I will work with a highly aggregated model…

    Like

  2. Being myself the modeler and the client, I find the right size of the model by starting first with a highly aggregated model as a modeler and verifying the level of my understanding and utility of the model as a client by using it with all possible ways (sensibility analysis, gaming, optimisation if possible and paying a particular attention to the built client interface and embedded reality checks). If I feel as a client that the model was useful but I am not satisfied with some points and need some more details, I build a slightly less aggregated model or even several models to avoid a too steep change in the model to make the change easier for me as a client. I repeat the process of the first stage (sensibility analysis included a modified client interface and reality checks) and I verify again the level of understanding and utility of that stage. If there is an added utility, but still a need of added details, I repeat the process, adding some more details, avoiding a too steep change in the model etc… I stop the process if at one stage the added utility is marginal compared to the preceding one. Of course, this process is very time consuming for the modeller and especially for the client, but I have not found, so far, another method.

    Like

Leave a Reply to Jean-Jacques Laublé Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.