Using Key Metrics to Supercharge Your Demand Management and S&OP Process

1© 2014 Steelwedge Software, Inc. Confidential.
Single Line of Sight: Plan, Perform, Profit
Measuring Forecast Accuracy
Improved S&OP through Analysis of Forecast Accuracy

Today’s Presenter
Background
Rick Blair
VP, Solutions Design and Analytics
Steelwedge Software Inc.
3825 Hopyard Rd
Pleasanton, CA 94588
Tel : (925) 460-1700
rblair@steelwedge.com
• Over 25 years of experience in supply chain consulting and
operations management.
• Works directly with customers and team members to optimize solution
design, drive improvements and apply best practice methodologies.
• Leverages supply chain management practitioner experience in various
roles across S&OP, demand and supply planning

Why measure forecast accuracy?
Examples of forecast accuracy metrics
How is MAPE calculated?
Beyond a number: Key Considerations
Accuracy measures in action
Agenda

• What is Forecast Accuracy?
• A measure of deviation between plan and actual
• Less deviation = greater accuracy
• What motivates an organization to track accuracy?
• Hold individuals and groups accountable
• Benchmark against other companies
• Manage inventory levels
• Improve management of business

Quick Poll #1
Which reason is the most important?

• While each reason has its merit, I suggest that #3 and #4 are most
critical
1. Hold individuals and groups accountable
2. Benchmark against other companies
3. Manage inventory levels
4. Improve management of business
5. Something else

Manage Inventory Levels
• If demand fluctuates substantially from period-to-period, should
you carry more safety stock?
• Conventional supply chain planning says ‘Yes’
Carry buffer inventory to handle demand fluctuations
• But…what if demand variability is predictable?
– Buy or make to forecast
– Reduce safety stock to cover just unpredictable variability
High forecast accuracy translates into reduced inventory requirement

Improve Management of Business
• Questions worth asking
• What are we trying to accomplish?
• What are we trying to improve?
• If we measure forecast accuracy…
– At a high level such as Family or Business Unit
– Using most recent forecast values
• Our accuracy metric looks much better, so why not make ourselves
look good?
Are you trying to look good or improve the business?

More Reasons to Improve Forecast Accuracy
Benefits of Better Forecasting:
Improved customer service levels
Increased sales
Reduced inventory carrying costs
Improved cash flow projections
Production smoothing (level
loading)
Reduced employee costs
Increased ROI
Balancing Supply and Demand
0
10
20
30
40
50
60
70
80
90
100
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
Time
Quantity
Demand
Supply
Excess
Inventory
Lost
Opportunity

Agenda

Accuracy Metrics
• MAD = Mean Absolute Deviation
• RMSE = Root Mean Square Error
• MAPE = Mean Absolute Percent Error
• WMAPE = Weighted Mean Absolute Percent Error
• MAD/Mean = Mean Absolute Deviation/Average
• Yields same result as WMAPE
• Bias = Tendency to over or under forecast
• % of forecasts over actual
• % of forecasts under actual

Which accuracy metric does your company use?
Quick Poll #2

Accuracy Metrics
• MAD = Mean Absolute Deviation
• RMSE = Root Mean Square Error
• MAPE = Mean Absolute Percent Error
• WMAPE = Weighted Mean Absolute Percent Error
• MAD/Mean = Mean Absolute Deviation/Average
• Yields same result as WMAPE
• Bias = Tendency to over or under forecast
MAPE & WMAPE are the most widely
used forecast accuracy measures

Agenda

• Mean Absolute Percent Error = MAPE = ∑│PE│/N
• N = number of periods for which we have PE values
• │PE│ = absolute value of the PE (Percent Error)
• Weighted MAPE = average of individual MAPEs weighted by
actual shipments
= ∑item MAPE * │(item actual / total actual)│
Let’s consider an example…

MAPE Example
1. Error: The difference between Forecast and Actual
2. Absolute Error: Convert negatives to positives
3. Absolute Percent Error = Absolute Error / Actual
Error
SKU 1
Actual 100
Forecast 90
Error (10)
Absolute Error
SKU 1
Actual 100
Forecast 90
Error (10)
Abs Error 10
Absolute Percent Error
SKU 1
Actual 100
Forecast 90
Error (10)
Abs Error 10
APE 10.0%

MAPE Example
• Mean Absolute Percent Error (MAPE)
• Average of APEs for multiple items or multiple periods or both
MAPE Calculation
SKU 1 2 3 4 5 6 7 8 9 10 Family A
Actual 100 90 8 1,000 1 90 10 11 50 76 1,436
Forecast 90 100 10 970 3 80 13 10 50 80 1,406
Error (10) 10 2 (30) 2 (10) 3 (1) - 4 (30)
Abs Error 10 10 2 30 2 10 3 1 - 4 30
APE 10.0% 11.1% 25.0% 3.0% 200.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.1%
MAPE 30.5%
MAPE = 30.5% = Average
of 10 SKU APE values
MAPE for Family A:
At Family level = 2.1%
At SKU level = 30.5%

MAPE and WMAPE Calculations
SKU 1 2 3 4 5 6 7 8 9 10 Family A
Actual 100 90 8 1,000 1 90 10 11 50 76 1,436
Forecast 90 100 10 970 3 80 13 10 50 80 1,406
Error (10) 10 2 (30) 2 (10) 3 (1) - 4 (30)
Abs Error 10 10 2 30 2 10 3 1 - 4 30
APE 10.0% 11.1% 25.0% 3.0% 200.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.1%
Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.1% 0.7% 0.2% 0.1% 0.0% 0.3%
MAPE 30.5%
WMAPE 5.0%
WMAPE Example
• Weighted Mean Absolute Percent Error (WMAPE or WAPE)
• Average of individual MAPEs weighted by actual shipments
WMAPE = 5.0% = Weighted
Average of 10 SKU APE values
WMAPE for Family A:
At Family level = 2.1%
At SKU level = 5.0%

SKU 1 2 3 4 5 6 7 8 9 10 Family A
Actual 100 90 8 1,000 2 90 10 11 50 76 1,437
Forecast 90 100 10 970 3 80 13 10 50 80 1,406
Error (10) 10 2 (30) 1 (10) 3 (1) - 4 (31)
Abs Error 10 10 2 30 1 10 3 1 - 4 31
APE 10.0% 11.1% 25.0% 3.0% 50.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.2%
Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.1% 0.7% 0.2% 0.1% 0.0% 0.3%
MAPE 15.5%
WMAPE 4.9%
MAPE & WMAPE: Impact of Small Changes
• What if SKU5 Actual was 2 (not 1)?
MAPE changes from 30.5% to 15.5%
WMAPE changes from 5.0% to 4.9%
A one unit change can have a huge impact on MAPE.
WMAPE is less sensitive.

SKU 1 2 3 4 5 6 7 8 9 10 Family A
Actual 100 90 8 1,000 - 90 10 11 50 76 1,435
Forecast 90 100 10 970 3 80 13 10 50 80 1,406
Error (10) 10 2 (30) 3 (10) 3 (1) - 4 (29)
Abs Error 10 10 2 30 3 10 3 1 - 4 29
APE 10.0% 11.1% 25.0% 3.0% 0.0% 11.1% 30.0% 9.1% 0.0% 5.3% 2.0%
Wtd MAPE 0.7% 0.7% 0.1% 2.1% 0.0% 0.7% 0.2% 0.1% 0.0% 0.3%
MAPE 10.5%
WMAPE 4.9%
MAPE & WMAPE: Zero Actual Values
• What if SKU5 Actual was 0 (not 1)?
MAPE changes from 30.5% to 10.5%
WMAPE changes from 5.0% to 4.9%
Pay close attention to instances where Actual = 0. Dividing by
zero causes an error, so the MAPE formula may assign 0%.
This can be misleading since 0% should represent zero
deviation between forecast and actual.

Agenda

Key Considerations
• Error vs Accuracy
• Metrics measure degree of error
• Accuracy can be defined as (1 – MAPE) or (1 – WMAPE)
– If MAPE = 25%, then Forecast Accuracy = 75%
Notice that Forecast Accuracy is not negative
Loss of visibility to error magnitude if MAPE > 100%

Key Considerations
• Aggregation Level
• Higher aggregation levels usually yield lower MAPE and WMAPE
values
– Variation is dampened as peaks and valleys get smashed together
• Ask: What are we trying to improve?
– Measure accuracy at level where you can affect change
– You may decide to measure accuracy at multiple levels
– For example, Product Mix (SKU) & Product Line (Family)

Key Considerations
• Time Buckets
• Examples: Month, Quarter, Rolling 3 Months
• The bigger the time bucket, the lower the MAPE
MAPE Calculation
Month Jan Feb March Q1
Actual 105 92 75 272
Forecast 95 100 82 277
Error (10) 8 7 5
Abs Error 10 8 7 5
APE 9.5% 8.7% 9.3% 1.8%
MAPE 9.2%
Monthly deviations are
eliminated in Q1 MAPE value

Key Considerations
• “Lag” or “Offset”
• Use the forecast at time of decision
– May be production or raw material lead time
– Measure accuracy of forecast that manufacturing could actually use
• The most recent forecast provides little to no value in making business
decisions
– What can we do with a forecast for February provided in February?
In this example, the forecast from Month 2 of prior quarter -1 is compared to actual results
from prior quarter. For example, forecast created in May for Q3 (July, August and
September) is compared to actual results from Q3 (July, August and September).
Previous Quarter - 1 Previous Quarter Current Quarter
month 1 2 3 1 2 3 1 2 3
"as of date" Quarter to be measured

Key Considerations
Forecast Period:
As Of: Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13
Oct-12 124 140 90 120 120 85 120 120 100 120 100 120
Nov-12 110 130 100 100 100 100 100 100 90 110 100 100
Dec-12 120 125 100 100 115 90 85 90 95 100 110 110
Jan-13 90 95 100 130 120 125 100 120 115 90 95 80
Feb-13 100 105 110 120 100 100 110 100 90 80 80
Mar-13 120 95 100 120 120 100 120 100 120 115
Apr-13 100 95 115 123 85 90 95 100 110
May-13 85 103 113 102 85 90 95 100
Jun-13 100 96 88 118 120 100 120
Jul-13 111 112 115 112 85 90
Aug-13 120 102 115 120 85
Sep-13 100 102 115 97
Oct-13 95 106 135
Nov-13 95 100
Dec-13 115
Actual 95 105 120 105 90 95 101 130 95 95 100 120
MAPE
Offset Jan-10 Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10
0 5% 5% 0% 5% 6% 5% 10% 8% 5% 0% 5% 4%
1 26% 10% 13% 10% 6% 8% 5% 14% 7% 7% 6% 17%
2 16% 19% 17% 5% 11% 21% 12% 32% 21% 21% 15% 13%
3 31% 24% 17% 24% 33% 26% 22% 22% 24% 18% 20% 19%
In this example, monthly MAPE values are shown over time with
various offsets. Tracking MAPE trends is a great way to see
improvement. Multiple offsets allow insight into how accuracy may
improve as better information is available nearer to current period.

Key Considerations
• Units vs Revenue
• Accuracy measures need not be limited to Units
• $ may be more meaningful if Average Selling Prices vary considerably
– Use WMAPE and weight by $

Agenda

MAPE & Bias Analysis
Actual and Forecast values MAPE by Item
Bias
MAPE and WMAPE
Excel Slicers

Targeted MAPE Analysis
MAPE by Functional Group
WMAPE by Functional Group

Q&A

Using Key Metrics to Supercharge Your Demand Management and S&OP Process

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Using Key Metrics to Supercharge Your Demand Management and S&OP Process

Similar to Using Key Metrics to Supercharge Your Demand Management and S&OP Process (20)

More from Steelwedge

More from Steelwedge (20)

Recently uploaded

Recently uploaded (20)

Using Key Metrics to Supercharge Your Demand Management and S&OP Process