Decremental demand: how to correlate and adapt to drops in demand caused by severe weather, events and pandemics

Dr. Xuxu Wang
Chief Data Officer

The spread of Covid-19 is having major economic impact. In February, concerns about the coronavirus led to a 500% spike in cancellations and postponements of significant events when compared to February 2019.

Notable events such as Mobile World Congress in Barcelona and HSBC Women’s World Championship in Singapore are among those cancelled or postponed. More than 225 events ranked high-impact were cancelled in February when compared to the previous February’s total of 45.

alt

In March, this trend has risen to 875% by March 10, 2020 Pacific time.

alt

Our data team has quantified the economic impact of some major event cancellations for Recode – focusing on the attendee’s airline, accommodation, transport and daily dining decisions. Our calculations are conservative, as they don’t factor in the enormous cost borne by conference organizers, as well as the substantial lost employment and income of the people who would have been working at the conferences directly, or nearby to meet demand.

For example, the cancellation of San Francisco’s Game Developers Conference will cost at least $129 million. Barcelona’s Mobile World Tech conference will cost airlines, accommodation, transport and retail providers at least $325 million.

Three ways to track and adapt to periods of major event cancellations and postponements

To survive and thrive in these unprecedented times, businesses need to react as swiftly and strategically as possible. The PredictHQ API is helping our airline, accommodation, transport and retail customers to do this in three ways:

  1. Know when events are cancelled or postponed within minutes – all cancellations are verified and all events ranked so your team or machine learning models can respond immediately. How-to guide here.

  2. Find out when events are rescheduled immediately. How to use our event rankings to track your most impactful events.

  3. Understand the impact of decremental demand catalysts on your business by establishing correlation with your historical transactional data and our verified event data, which includes outbreaks of viruses such as coronavirus, as well as our live event coverage of unscheduled events such as severe weather, terrorism, airport delays and disasters.

We will unpack this third use case below as correlation is the fundamental building block for all demand forecasting strategies.

How to discover which events diminish demand for your business

Correlating your demand to events opens up new opportunities for better planning. The more you can explain variations in historical demand, the more accurate your forecast will be. PredictHQ usually focuses on positive anomalies – surges in demand – but grappling with decremental demand is necessary also.

Because many businesses use automated models to help them understand their demand and forecast it, identifying the events with negative impacts on demand is important to have accurate insights. If sales were notably down one quarter, being able to identify correlation between severe weather, or a disaster or similar enables you to include that information accurately in your future strategies.

  • For example the hurricane in Florida in March 2019 reduced demand for flights, hotels and retail notably. Without understanding why this drop-off occured, this could easily distort future planning. This drop off shouldn’t cause the 2020 forecasting to predict a drop in demand for March 2020.

  • Another example is the Australian bushfires will cause significant decremental demand for the first quarter of 2020. You need to ensure your demand forecasting model doesn’t use this data without qualification and shows lower demand in Q1 2021 (or for Q2 2020) because this is an anomalous or unusual demand that should be excluded when looking at future forecasting.

To understand how long an event’s negative impact will last, you’ll need to explore previous similar incidents. We provide you with the tools so that your forecasting can know when decremental demand is driven by these types of unpredictable events.

Unscheduled events in PredictHQ’s API

PredictHQ has 19 event categories. Our scheduled events are mostly attendance-based and tend to drive positive incremental demand. In our upcoming demand forecasting guide we’ll cover in detail how you can add features to your model for scheduled attendance-based events. We track three key types of unscheduled events that can have a major impact on decremental demand.

Severe Weather

Heavy rain, strong winds, tornado season all impact demand. Use this category of events to identify how to prepare for future severe weather incidents by pinpointing its previous impact on your demand. For example, if your orders drop by 30% during heavy rain, you can reduce staffing and stock for forecasted heavy rain periods.

Disasters

Disaster such as floods, droughts and tsunamis can have extensive impact on the communities they affect and the businesses that cater to them. Identifying how much impact these events have on your business will enable you to prepare and respond better as they occur.

Terror Attacks

We cover breaking terror attacks across 29,000 cities. Knowing these are occurring enables you to respond in a timely way, as well as gauge the impact of these events on your demand.

We also cover airport delays, but these only have high impact for short bursts of time.

Unscheduled events correlation improves demand forecasting models

Imagine you run a hotel chain in Seattle and use a machine learning model for your demand forecasting. You notice some decremental demand, and you want to make sure your forecasting models are handling this appropriately.

We will now explore how you can use unscheduled events in your demand forecasting model to improve the accuracy of your forecast and take into account the impact of severe weather, disasters and terror attacks. Pandemics are fortunately rare, and high-profile, making them easier to track – the same process works for them.

The way this works is that the features are used with a training set of historical data to train your model. The model learns the relationship between decremental demand and unscheduled events. Therefore, your model will give a more accurate and up-to-date prediction when unscheduled events are taken into account. Also, these features will allow you to explain the patterns in historic demand where there are negative impacts.

If you are a data scientist and new to working with our data, we recommend you read our getting started guide and correlation guide as background to this post.

Identifying correlation between pandemics and unscheduled events and your data

Below is an example of how you can call the PredictHQ API to get back unscheduled events for a specific time range. In this example we notice an increase in demand in May that we want to drill into.

alt

In this example, we are showing the URL for a call to the events API to find relevant unscheduled events. You can also use our data exporter to run the same type of query and export the events to a CSV or JSON file. In this example we use the events in our API as it shows how you could integrate calls to the API into your system. The same core data is returned by the data exporter so the examples can be used interchangeably.

If you were using Python (including our Python SDK) you would follow our API docs to make this call in Python (choose Python in the top right of the technical docs). Here is the example of calling the API:

In the examples below you need to use the appropriate access token for your account.

curl -X GET https://api.predicthq.com/v1/events/?place.scope=5809844&category=severe-weather,disasters,terror&active.gt=2019-05-01&active.lt=2019-05-28 \ -H "Accept: application/json" \ -H "Authorization: Bearer $ACCESS_TOKEN"

This example shows calling the events API and setting the category parameter to severe-weather, disasters, terror this limits the events returned to severe weather, disasters and terror attacks. We are using the place ID for Seattle to retrieve events in the Seattle area.

Below is the response to the API call. This has returned a HAZMAT event under the disaster category. On further investigation we find that this caused a decrease in demand in May for our hotel. Now let’s look at how we can use this data.

Now we’ve found how we can use the API to identify unscheduled events we can drill into other examples of decremental demand during the year. Here we identify periods in February and September.

alt

Calling the API for these dates we find the following:

  • In February we had a severe weather event which was a blizzard which impacted demand.

  • In September we had a severe weather event which was “a strong line of thunderstorms developed over Western Washington, bringing frequent lightning, heavy rains, flooding, and hail to the Puget Sound region Saturday evening and knocking out power to thousands across the region.” which also impacted demand

Now we need to look at how we can use this information to improve our demand forecasting. We don’t want our model to take these drops in demand and skew the results for the following year. While this is an extreme example the cumulative impact of unscheduled events can add up so ensuring you take them into account helps improve forecasting accuracy.

How to use unscheduled events in your demand forecasting

Now to use the information we have found we need to correlate decremental demand with events. We have identified dates with decremental demand. Let’s create a table showing the correlation of unscheduled events which have potential negative impacts on the business and decremental demand to clearly show the correlation. Next we will show how this data can be used.

DateRaw demandDecremental DemandEvent
9 Feb 2019500-120HAZMAT – North-America – USA
10 Feb 2019420-110
3 May 2019550-15severe weather event
4 May 2019509-21
8 Sep 2019480-45severe weather event
9 Sep 2019470-30

An example

Below is an example of a model trained using past unscheduled events and Aggregated Event Impact (AEI). There is a disaster on 4 June 2019 which causes demand to decrease in the following week (see the dip around the 9th of June). Using the approach outlined above we can show how the model predicts there is a negative impact on demand in the next 10 days. It does this by using the features described above and being trained on a historical data set for unscheduled events that lets it predict the impact of future unscheduled events once they occur.

alt

You can use PredictHQ’s data exporter to export event information to CSV or JSON format and use that to train your model.

Adding features to your model

In order to use events in a machine learning model you need to add features to your model that use our events information. For attendance based events we provided our Aggregate Event Impact API endpoint that can be used to model the impact of events. In this example we will show how to add features to your model to handle unscheduled events.

In the example code below we incorporate aggregate event impact and unscheduled events like in the disaster and terror categories into the forecasting model. This allows the model to learn the relationship between demand and event data resulting in a more accurate model for predicting the impact caused by unscheduled events.

# Create a DataFrame to hold disaster events information disaster = pd.DataFrame({ 'ds': pd.to_datetime(['2017-04-01', '2017-09-12', '2018-10-28', '2019-06-04']), 'lower_window': 0, 'upper_window': 14, }) # Create a DataFrame to hold terror events information terror = pd.DataFrame({ 'ds': pd.to_datetime(['2017-02-10', '2019-03-15', '2019-09-12']), 'lower_window': 0, 'upper_window': 14, }) # Put them together into a single DataFrame unscheduled_events = pd.concat([disaster, terror]) # Build a forecasting model using unscheduled events data model = ForecastModel(event=unscheduled_events, yearly_seasonality=True, weekly_seasonality=True, daily_seasonality=False, seasonality_mode='additive') # Add Aggregate Event Impact into the model as extra features model.add_regressor('aei', prior_scale=0.5, standardize=True, mode='additive') # Train the model model.fit(training_data) # Create a new DataFrame and make predictions future = model.make_future_dataframe(periods=30) forecast = model.predict(future)

Using unscheduled events

Once you have integrated unscheduled events into your model you can estimate the impact of future events like hurricanes (severe weather category) based on the impact of past events. Usually, these types of events have a time lag – they happen and then impact follows for a period of days or weeks. For example if after a hurricane occurred your demand went down by 50% for the following 2 weeks once your model is trained on the impact of historic hurricane events it can then predict the impact of the event after it occurs.

So, in this instance we don’t know when a hurricane will occur. But if you integrate calls to the PredictHQ API once it occurs you can pass this information into your model and it can model the effect on demand following the event.

Similarly, if there’s a terror event happening and you see demand drop for 2 weeks afterwards, your model can predict that in the 2 weeks following a similar event demand will drop again.

To use this model to predict the impact on a forecast you need to integrate calls to the PredictHQ demand intelligence API. In your demand forecasting engine integrate a call to the API as follows:

curl -X GET https://api.predicthq.com/v1/events/?place.scope=[location]&category=severe-weather,disasters,terror&active.gt=[start date]&active.lt=[end date] \ -H "Accept: application/json" \ -H "Authorization: Bearer $ACCESS_TOKEN"

Integrate this call so that when you call your demand forecasting system for a date range you call the API for that date range and pass the results to your model (in the format it expects) so it can take the appropriate action. You now have updated your forecast to use unscheduled events!

Need help?

Demand forecasting is complex and our expertise in event impacts only makes us more respectful of how challenging it can be. Events are complex and ever changing. Getting it right can be hard but the benefits are huge. We have an expert team of data scientists and engineers who are experts in the complexities of events data and how to use events for forecasting.