Finding the Right Tool

My desire to make popcorn for the evening of binge watching came to a sudden halt when the microwave door latch stuck.  Undeterred by this setback, I put on my repairman cap, opened my toolbox, prepared to open the appliance.  Within minutes, I ran into some odd looking screws – not the typical “slot” or “Phillips” head.  A quick internet search revealed that I had a “security hex socket” screw.  Without the proper screwdriver, I began to look for a different snack.

What do popcorn and microwaves have to do with forecasting?  Recently, I had a conversation with a utility budget forecaster tasked with developing the Integrated Resource Plan (IRP) forecast.  While the forecaster was familiar with budget forecasting models, he was concerned that the models could not be applied to the IRP forecast.

The budget models are simple econometric equations that capture customer and average use growth based on macroeconomic drivers, normal weather, and end-shift variables.  While economic and end-shift variables project growth and calibrate the model to the latest data, the weather variables capture heating and cooling response.  This structure is well suited to forecasts next year’s budget.  However, the forecaster’s concern is whether this type of model adequately captures the changes in the system over the next 30 years.

Applying a budget forecast models to a 30-year IRP forecast assumes that the relationship between the economic drivers and energy usages remain constant over the forecast horizon.  However, our intuition tells us that changes in energy efficiency standards and new technologies will change energy usage patterns creating deviations from this historic relationship.  A few potential changes are listed on the Appliance Standards Awareness Project website (http://www.appliance-standards.org/national).  The list includes energy efficiency changes for at least 14 residential end-uses such as air conditioners, clothes dryers, clothes washers, and furnaces.

In order to adequately capture energy consumption changes over the 30 year planning horizon, future changes should be included in the forecast models.  Including end-use energy efficiency changes is one modification needed to adapt the budget model to a 30-year model.  Additionally, projections of behind-the-meter generation (solar) and new technologies (electric vehicles) should also be included capture risk scenarios.

After much discussion, my budget forecaster agreed that applying a budget forecast model for an IRP is like trying to apply a slot or Phillips head screwdriver to the security hex socket.  While we could force the issue, applying the wrong tools in any situation only leads to frustration.


Are SAE Models Accurate?

If you follow Itron’s forecasting work, you know that we’ve employed Statistically Adjusted End Use (SAE) models since the early 2000’s.  To date, almost 60 companies across North America use the SAE models.  These models are applied to both internal company forecasts and external regulatory filings.

With the widespread adoption of the SAE models, comes the question “are they accurate?”

Considering that SAE models are applied to long-term forecasts (10+ years), accuracy is not the primary reason why these models are used.  The value of SAE models is developing an understanding of key energy drivers based on how customers use electricity.

Unlike traditional economic models which are based on macro-economic variables, the SAE model contains information about thermal shells, appliance saturations, and energy efficiency changes.  This information allows the forecast to bend with known changes in end-use codes and standards.  Explicit assumptions for energy efficiency trends create opportunities for developing scenarios and accounting for energy efficiency programs.  And, detailed end-use projections open the door for understanding which end-uses are responsible for forecast growth.

Ultimately, trust in a forecast lies in its ability to present a plausible picture of the future.  The greater the explanatory power of the forecast, the greater the trust.

But, is SAE accurate?  If there’s one thing every forecaster knows, it’s that every forecast is wrong.

In 2001, the SAE model predicted increasing average use due to increasing saturations of central air conditioners, larger homes, and greater penetration of consumer electronics such as home office equipment and security systems.  The figure below shows the 2001 indexed sales forecast (blue) compared against historical calculations from 2015 (red).  In this figure, the forecast aligns well with history through 2005.  Deviations begin to occur in 2006 with the first projected declines in average use due to higher energy prices and energy efficiency gains.  And of course, the 2001 forecast missed the Great Recession, 2005 EPACT legislation, and 2007 EISA impacts.

 

graph1

The SAE model allows us to examine the end-use components and assess where the errors occur.  In the figure below, the forecast of non-HVAC average usage shows consistency with history through 2006.  After 2006, the deviation begins.  Digging into the SAE data shows that improved lighting efficiency is the main factor driving non-HVAC usage down.  Obviously, the advances in lighting standards and technology were not known in 2001.

graph2

 

Once the error is understood, future forecasts in the SAE model are improved by updating information about lighting efficiency.
So, are they accurate? Remember, the value of SAE models is not in accuracy but in information.  Even so, Itron reviewed the accuracy of SAE models in its 2015 Benchmarking Survey.  When Itron asked forecasters how well their 2013 forecast performed in 2014, SAE modelers (35 responses) showed an average MAPE of 1.46% while non-SAE modelers (32 responses) showed an average MAPE of 1.53%.
While accuracy is important, remember it is not the only factor in selecting a forecast.

 


DON'T MISS A POST!
I agree to have my personal information transfered to AWeber ( more information )
Opt in to receive notifications when a blog post is published. Don't miss the thought leadership, insight and news from Itron.
We hate spam. Your email address will not be sold or shared with anyone else.