How to diagnose and treat machine learning models afflicted by COVID-19

COVID-19 has disrupted the lives of millions of people and affected businesses across the world. Its impact has been particularly significant on many machine learning (ML) models that companies use to predict human behavior.

Companies need to take steps to deeply examine ML models and acquire the insights needed to effectively update models and surrounding business rules.

The economic disruption of COVID-19 has been unprecedented in its swiftness, upsetting supply lines, temporarily closing retail stores and changing online customer behaviors. It has also dramatically increased unemployment overnight, increasing financial stress and systemic risks of both individuals and businesses. It is forecasted that global GDP could be affected by up to 0.9%, on a par with the 2008 financial crisis. While the nature of our recovery is unknown, if the 2008 crisis is any indicator, the impact of COVID-19 could be felt for years, through both short-term adjustments and long-term shifts in consumer and business behaviors and attitudes.

This disruption impacts machine learning models because the concepts and relationships the models learned when they were trained may no longer hold. This phenomenon is called “concept drift.” ML models may become unstable and underperform in the face of concept drift. That is precisely what is happening now with COVID-19. The effects of these drifts will be felt for quite some time, and models will need to be adjusted to keep up. The good news is that there have been significant developments in model intelligence technology, and through judicious use, models can nimbly adjust to those drifts.

As the effects of COVID-19 (and economic closure and reopening) play out, there will be distinct stages in the impact on social and economic behaviors. Updates to business rules and models will need to be done in sync with overall behavior shifts in each of these stages. Companies need to adopt an approach of measure-understand-act and to constantly examine, assess and adjust ML models in production or development and surrounding business rules.

Examining how ML models have been impacted means going through an exercise to both measure and understand how the models behaved prior to the coronavirus, how they are behaving now, why they are behaving differently (i.e., what inputs and relationships are the drivers of change), and then to determine if the new behavior is expected and accurate, or is no longer valid. Once this is determined, the next step is naturally to act: “So, what can we do about it?”