First, look at the data

By Jack Homer, VP of Professional Practice

I recently received an e-mail from a college student, someone I’d never heard of before, wondering whether system dynamics was the right approach for studying a particular issue: how the government’s publication of secondary school rankings in his country might influence parents’ attitudes and behaviors.  The student wondered whether I could give him “some advice on creating equations in SD”.

I’m not sure how this student had heard of system dynamics, but it was clear he had never built a model and was thinking of SD as a piece of software, perhaps something like a simple spreadsheet program.  As much as I believe in mentoring (see my April 2019 blog post), I was not prepared to give this fellow much of my attention and wanted only to steer him in a useful direction.  I advised him to look into SD resources for beginners, including textbooks and the pre-conference Summer School.  I also advised him to gather data on his issue of interest and do some statistical analysis.

I want to talk about this second piece of advice and why I gave it.  For more than 20 years I’ve written about the importance of evidence—both structural and behavioral—as a foundation for reliable and scientific SD modeling.  I think that every SD model should start with a scan of the evidence surrounding the issue at hand, including gathering as much historical data as possible.  Any theorizing should emerge from this evidence, rather than from preconceptions and “common knowledge” that may have little or no basis in fact.  Statistical analysis can be useful in exploring the data, if done and interpreted carefully.  As statistician Edward Tufte said, “Correlation is not causation, but it sure is a hint.”   

As a master’s student many years ago, I took a class in exploratory data analysis.  The professor told us, “First, before you start theorizing, look at the data.  Plot it all, look at the relationships, and think carefully about what you see.  This is the best way to avoid going down the wrong road.”  I took his advice to heart, and I think it would be good for our field if we all did so.  Let’s look before we leap—it’s the scientific thing to do.

One thought on “First, look at the data

  1. I think this is not all that different from the long-standing SD practice of creating reference modes as part of the model conceptualization process. For example:
    https://www.systemdynamics.org/assets/conferences/1998/PROCEED/00016.PDF
    https://ocw.mit.edu/courses/sloan-school-of-management/15-988-system-dynamics-self-study-fall-1998-spring-1999/readings/building.pdf

    The difference is that, historically, reference modes were hand drawn in situations where there likely was little or no formal data. Now that we typically have quite a bit of data, there’s no reason to “wing it” it based on (possibly flawed) recollections when we have access to the real thing.

    Hand (or mouse) drawn reference modes are still useful for eliciting counterfactual trajectories, or hoped-for and feared future outcomes that go beyond the available data.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.