Model Assumptions


Model Assumptions

20 May, 2021

Before the start of the 2022 6 Nations Rugby, a supercomputer predicted England would win the Grand Slam and Wales would have a dismal few weeks. As we shall see, the reality was somewhat different.

Whilst rugby is - for some - a trivial matter, statistical and computer models of everyday situations have a regular impact on decisions which affect our lives. 

During the past year we have seen models used across the globe to help us understand the possible impacts of Covid-19. 

So what exactly is a model and how good is it for predicting real life?

Take a pinch

A model is just like a recipe for a cake.

You mix all the ingredients together, put them in an appropriate container and bake in the oven at a certain temperature.

Change any one of those elements and the cake is likely to come out differently.

And even if you follow the recipe there is no way to account for the quality of the flour, the age of the baking powder or the temperamental thermostat on the oven.

What emerges from the recipe will not be the same every time because life cannot be controlled to that extent.

If this then that

Models are great for testing ideas and theories and to see how they might work out and what may happen in different scenarios.

Scenarios are based on different variables being changed in the model, assumptions about  things that might happen. 

But it is important to highlight that whilst these assumptions might be based on real world events and observations, the conclusions from the model remain theoretical. Though the scenarios they paint are a helpful insights they are only a possible guide to how changes in one or more variables may affect what happens overall.

If this then that. "If" being the important word.

Context is everything and realities of life that we cannot control matter significantly.  As the statistician Sir David Cox has said,

"How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis."

Question the assumptions

Conclusions are rarely definitive however the headline is written.

The insights models provide are neither right nor wrong because they are reliant on the assumptions that were made about the inputs to those models. Whilst headlines may interpret their conclusions as concrete fact (missing the subtlties every model contains), the reality is that these are insights and possibilities which raise further questions.  

For the model to be most effective we should ask those questions.

Take for example the recent rise of the Delta variant of Covid. Some scientists - and therefore media headlines - have, based on assumptions, drawn a conclusion that the variant is more transmissible and inevitably will become dominant as the data model suggests. 

But what if the assumptions are wrong or out-dated by the dynamics of the world. And what if we ask how the transmission is occurring and between whom (e.g. is it intra-household amongst largely unvaccinated individuals?), might that change the conclusions we draw?

Near misses, next questions

By the end of March, Wales had won the 6 Nations title and England finished fifth. The model didn’t give a very accurate prediction.

As for Covid, an increased understanding of the virus will allow us to update the assumptions in the models but they will never be able to predict a definitive conclusion to the pandemic.

Models are very helpful tools in helping us try and make sense of a very dynamic world. Yet just as with any good recipe there is always room for error because there are numerous influences that we cannot control.

Rather than a definitive predictor of what will happen next, most conclusions of models are a helpful guide to the further questions we need to ask if we want to understand how things might evolve.

It's good to read, even better to share