First, look at the data

By Jack Homer, VP of Professional Practice

I recently received an e-mail from a college student, someone I’d never heard of before, wondering whether system dynamics was the right approach for studying a particular issue: how the government’s publication of secondary school rankings in his country might influence parents’ attitudes and behaviors.  The student wondered whether I could give him “some advice on creating equations in SD”.

I’m not sure how this student had heard of system dynamics, but it was clear he had never built a model and was thinking of SD as a piece of software, perhaps something like a simple spreadsheet program.  As much as I believe in mentoring (see my April 2019 blog post), I was not prepared to give this fellow much of my attention and wanted only to steer him in a useful direction.  I advised him to look into SD resources for beginners, including textbooks and the pre-conference Summer School.  I also advised him to gather data on his issue of interest and do some statistical analysis.

I want to talk about this second piece of advice and why I gave it.  For more than 20 years I’ve written about the importance of evidence—both structural and behavioral—as a foundation for reliable and scientific SD modeling.  I think that every SD model should start with a scan of the evidence surrounding the issue at hand, including gathering as much historical data as possible.  Any theorizing should emerge from this evidence, rather than from preconceptions and “common knowledge” that may have little or no basis in fact.  Statistical analysis can be useful in exploring the data, if done and interpreted carefully.  As statistician Edward Tufte said, “Correlation is not causation, but it sure is a hint.”   

As a master’s student many years ago, I took a class in exploratory data analysis.  The professor told us, “First, before you start theorizing, look at the data.  Plot it all, look at the relationships, and think carefully about what you see.  This is the best way to avoid going down the wrong road.”  I took his advice to heart, and I think it would be good for our field if we all did so.  Let’s look before we leap—it’s the scientific thing to do.

3 thoughts on “First, look at the data

  1. I think this is not all that different from the long-standing SD practice of creating reference modes as part of the model conceptualization process. For example:

    Click to access 00016.PDF

    Click to access building.pdf

    The difference is that, historically, reference modes were hand drawn in situations where there likely was little or no formal data. Now that we typically have quite a bit of data, there’s no reason to “wing it” it based on (possibly flawed) recollections when we have access to the real thing.

    Hand (or mouse) drawn reference modes are still useful for eliciting counterfactual trajectories, or hoped-for and feared future outcomes that go beyond the available data.


    1. I’m in violent agreement with the idea that AI/big data/machine learning can reinforce biases. But I don’t think the data itself is frequently biased; it’s the analysis. Even data that has sampling bias problems is likely to contain some information – 538 for example makes good use of this by debiasing bad pollsters

      I think AI goes wrong because people use it to model a single point in a system. They fit a model to the data, but they don’t bother to confront the data with a model that would reveal flaws, biases and side effects. An example: a couple years ago we worked on a project in which a household-name big data vendor happily used incarceration data to predict mental health status. Later, when we put the data into a model with a stock of prisoners, we discovered that the data didn’t conserve people, making the whole analysis rubbish in some unknown way.

      I think the winning attitude has to be “all models are wrong, all data are wrong, but some combinations are useful.”

      Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.