Presented to AITPM, August 2015

1. Toll Road Forecasting
Robert Bain
RBconsult | University of Leeds
PERTH Western Australia 10 August 2015

2. 2 Presentations for the Price of 1!
1. Recap from the National AITPM Conference
• Quick summary
2. Toll Road/Traffic Forecasting Error
• Quick summary
www.robbain.com 2

3. 2 Presentations for the Price of 1!
1. Recap from the National AITPM Conference
• Quick summary
2. Toll Road/Traffic Forecasting Error
• Quick summary
www.robbain.com 3

4. Recap from the National AITPM Conference

5. RBconsult | www.robbain.com 5

6. How To Do Better
Robert Bain
RBconsult | University of Leeds
PERTH Western Australia 10 August 2015

7. How To Do Better
• What does our product offering (our reports) tell us?
• What do our clients tell us?
• What do the lawyers tell us?
• What does the future hold for us?
www.robbain.com 7

8. How To Do Better
• What does our product offering (our reports) tell us?
• What do our clients tell us?
• What do the lawyers tell us?
• What does the future hold for us?
www.robbain.com 8

9. What Do Our Reports Tell Us?

10. Sample of Recent Reviews…
www.robbain.com 10

11. Sample of Recent Reviews…
www.robbain.com 11

12. Sample of Recent Reviews…
www.robbain.com 12

13. Sample of Recent Reviews…
www.robbain.com 13

14. Sample of Recent Reviews…
www.robbain.com 14

15. Sample of Recent Reviews…
www.robbain.com 15

16. Sample of Recent Reviews…
www.robbain.com 16
And others that the lawyers won’t let me show…

17. There are Some Standout Features
www.robbain.com 17

18. There are Some Standout Features
• 80-90% of the report is focussed on the base-year model
• 10-20% of the report is focussed on the future (ie. forecasts)
Time to re-think the 80:20 rule?
• 80-90% of the report focussed on the future?
• 80-90% of the report’s:
• insight
• intelligence, and
• value-add
• …focussed on the future
www.robbain.com 18

19. There are Some Standout Features
• 80-90% of the report is focussed on the base-year model
• 10-20% of the report is focussed on the future (ie. forecasts)
Time to re-think the 80:20 rule?
• 80-90% of the report focussed on the future?
• 80-90% of the report’s:
• insight
• intelligence, and
• value-add
• …focussed on the future
www.robbain.com 19

20. There are Some Standout Features
• 80-90% of the report is focussed on the base-year model
• 10-20% of the report is focussed on the future (ie. forecasts)
Time to re-think the 80:20 rule?
• 80-90% of the report focussed on the future?
• 80-90% of the report’s:
• insight
• intelligence, and
• value-add
• …focussed on the future
www.robbain.com 20

21. There are Some Standout Features
• 80-90% of the report is focussed on the base-year model
• 10-20% of the report is focussed on the future (ie. forecasts)
Time to re-think the 80:20 rule?
• 80-90% of the report focussed on the future?
• 80-90% of the report’s:
• insight
• intelligence, and
• value-add
• …focussed on the forecasts
www.robbain.com 21

22. What Do Our Clients Tell Us?

23. Selection of Bain’s Clients
www.robbain.com 23

24. Let’s Ask the Simple Question
www.robbain.com 24
How could
we do
better?

25. Top 10 Answers (not in priority order)
www.robbain.com 25

26. Top 10 Answers (not in priority order)
• Answers focussed on:
1. Improved transparency
2. Improved understanding (+ supporting evidence)
3. Keeping it real
4. Lessons from the past
5. Working with other experts
6. Embracing volatility & uncertainty
7. Focus on a range of outcomes
8. Think like a client
9. Push information - don’t wait to be asked
10. The ‘traffic story’
…and some other (miscellaneous) themes
www.robbain.com 26

27. Top 10 Answers (not in priority order)
• Answers focussed on:
1. Improved transparency
2. Improved understanding (+ supporting evidence)
3. Keeping it real
4. Lessons from the past
5. Working with other experts
6. Embracing volatility & uncertainty
7. Focus on a range of outcomes
8. Think like a client
9. Push information - don’t wait to be asked
10. The ‘traffic story’
…and some other (miscellaneous) themes
www.robbain.com 27

28. Client Feedback
• Improve understanding
• Take care with cause-and-effect
• Effects may be caused by factors other than the seemingly obvious
• Use comparisons, benchmarks and independent sense-checks
• “An imperfect benchmark is better than no benchmark at all”
• “Consultants overuse simulation and underuse empirical analysis”
• Bridge diagrams (‘waterfall charts’) are essential
• Provide logical, intuitive build-up (by driver) of traffic growth
• Break-out growth by contributions eg. population, employment, real income
growth, car ownership, network effects, impact of tolls etc.
• Excellent way to really understand a forecast
• Allows risk to be examined and understood on a tiered basis
• Rather than looking at risk on a bucketed, collective, traffic growth basis
www.robbain.com 28

29. Sample Bridge Diagram
www.robbain.com 29
47,921
- - - - - - - - - - -
78,397
-
20,000
40,000
60,000
80,000
100,000
120,000
Transactions(000s)

30. Client Feedback
• Embracing volatility and uncertainty in data inputs
• Don’t hide or mask it
• Don’t flatter it (eg. thru conveniently misspecified MC simulation)
• “Don’t show me unrealistically tight confidence intervals”
• Mine it - and explicitly track it through to forecasts
• What are the implications for forecasts and reliance?
• This is where a lot of time should be spent
• Show me the impact of alternative - yet still plausible - input values
www.robbain.com 30

31. • The traffic story
• Numbers are good (and important)
• …but I need the accompanying narrative (“a story that makes sense”)
• What are the defining characteristics of the area - and how do these translate into
trip-making and travel patterns?
• How will the study area develop? What really drives growth?
• What are the key movements?
• Where will people be travelling from/to: who, what, how, when and why?
• What are travellers’ preferences/sensitivities?
• What is their choice set?
• Using surveys & other data sources you can provide me with information
• They’re not just feedstock for your model
And the Big One!!!
www.robbain.com 31

32. The Traffic Story: Final Word
www.robbain.com 32
“Supporting stories convey
information very well.
It’s the story that clients
repeat and reflect on
once the traffic expert
leaves the room”

33. What Are The Lawyers Telling Us?

34. Toll Road-Related Litigation
• Lane Cove Tunnel, Sydney
• CLEM7, Brisbane (x2)
• Airport Link, Brisbane (x2)
• American Roads, US
www.robbain.com 34
Note: the following slides have been compiled from public information.

35. Lane Cove Tunnel, Sydney
• AMP v Parsons Brinckerhoff
• No. 2009/290489
• Claim: $144m
• $80m for initial investment
• $64m in interest
• Progress:
• Case settled in September 2014
• Settlement reported to be $50m-$100m
• Basis of Claim:
• Misleading and deceptive conduct (Trade Practices Act)
• [forecasts were ‘reverse engineered’ to win the bid]
www.robbain.com 35

36. CLEM7, Brisbane (1)
• Hopkins & Anor v AECOM Australia & Rivercity
• RiverCity class action
• NSD 757/2012
• Claim: $150m + interest
• Progress:
• Commenced May 2012
• Scheduled for trial 29 August 2016
• Basis of Claim:
• AECOM’s forecasts in PDS (relied-upon by investors) were defective,
misleading and made without reasonable grounds
• Issuing a defective PDS is a breach of the Corporations Act 2001 (s1022B)
www.robbain.com 36

37. CLEM7, Brisbane (2)
• RCM Finance v AECOM
• NSD 678/2012
• Portigon v AECOM
• NSD 697/2012
• Claim: $1.68bn
• Progress:
• SEC filings report that RCM Finance and Portigon v AECOM have settled
• Class action (previous slide) “remains pending”
• Basis of Claim:
• AECOM made representations regarding its forecasts that amounted to
• …false, misleading & deceptive conduct under the Trade Practices Act
www.robbain.com 37

38. Airport Link, Brisbane (1)
• Bulense Holdings v Arup
• Brisconnections class action
• NSD 770/2014
• Claim: $50m
• Progress:
• Class action commenced July 2014
• Settled for $13m in July 2015
• Basis of Claim:
• Arup’s forecasts in PDS were defective, misleading and made without
reasonable grounds
• Issuing a defective PDS is a breach of the Corporations Act
www.robbain.com 38

39. Airport Link, Brisbane (2)
• Brisconnections v Arup
• NSD 521/2014
• Claim: Over $1bn
• Progress:
• Ongoing
• Basis of Claim:
• Receivers allege misleading and deceptive
conduct and negligent misstatement
under the Trade Practices Act
www.robbain.com 39

40. American Roads, US
• Syncora v Alinda Capital, American Roads, Macquarie Securities
and John S Laxmi
• Index No. 651258/2012 (NY Sup.)
• Claim: Damages to be determined at trial
• Progress:
• Case has survived a motion to dismiss
• Currently in the middle of fact discovery
• Basis of Claim:
• Plaintiff alleges that Macquarie engaged in a fraudulent scheme to
manipulate the forecasts supporting a bond offering on a portfolio of
toll road assets
www.robbain.com 40

41. What Does It All Mean?
• People are not being sued for inaccurate forecasts per se
• Not, by itself, a cause of action to sue
• Emerging themes:
• Misleading and/or deceptive conduct
• Misleading and/or deceptive statements
• Omissions
• Negligence? Negligent misstatement? [requires duty-of-care]
• Content and form of required disclosure
• Key issues for practitioners:
• Did the forecaster act in accordance with competent professional practice?
• Did the forecaster have reasonable grounds for their forecasts (at the time)?
• Irrespective of whether they turn out to be right or hopelessly wrong
• Be aware of who your audience is (for forecasts) - not always obvious from the start
• Tread carefully if you are adjusting inputs/assumptions based on client directions
www.robbain.com 41

42. What Does It All Mean?
• People are not being sued for inaccurate forecasts per se
• Not, by itself, a cause of action to sue
• Emerging themes:
• Misleading and/or deceptive conduct
• Misleading and/or deceptive statements
• Omissions
• Negligence? Negligent misstatement? [requires duty-of-care]
• Content and form of required disclosure
• Key issues for practitioners:
• Did the forecaster act in accordance with competent professional practice?
• Did the forecaster have reasonable grounds for their forecasts (at the time)?
• Irrespective of whether they turn out to be right or hopelessly wrong
• Be aware of who your audience is (for forecasts) - not always obvious from the start
• Tread carefully if you are adjusting inputs/assumptions based on client directions
www.robbain.com 42

43. Where Do We Go From Here?
• Industry Response?
• Update our Terms & Conditions?
• …but unilateral action means that we
can be picked off one-by-one
• Lobby the professional bodies
(collective action)?
• Clarify, highlight (and strengthen) our
professional responsibilities
• to our peers
• to society
• the public…
• A new code-of-conduct emphasising
obligations, integrity etc.
• …and client responsibilities??
www.robbain.com 43

44. Where Do We Go From Here?
• Industry Response?
• Update our Terms & Conditions?
• …but unilateral action means that we
can be picked off one-by-one
• Lobby the professional bodies
(collective action)?
• Clarify, highlight (and strengthen) our
professional responsibilities
• to our peers
• to society
• the public…
• A new code-of-conduct emphasising
obligations, integrity etc.
• …and client responsibilities??
www.robbain.com 44

45. Toll Road/Traffic Forecasting Error

46. A Focus on Prediction Intervals
• Travel demand forecasts have prediction intervals
1. What do these prediction intervals look like?
2. What does empirical evidence tell us?
46
An estimate of an interval in which future observations will fall,
with a certain probability,
given what has already been observed.

47. A Focus on Prediction Intervals
• Travel demand forecasts have prediction intervals
1. What do these prediction intervals look like?
2. What does empirical evidence tell us?
47
An estimate of an interval in which future observations will fall,
with a certain probability,
given what has already been observed.

48. Research Methodology
Empirically
Derived
Prediction
Intervals
48

49. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
49

50. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
50

51. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
51

52. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
Examine
traffic model
flaws,
shortcomings
& limitations
52

53. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
Examine
traffic model
flaws,
shortcomings
& limitations
Investigate the
uncertainty
introduced
through model
inputs
53

54. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
Examine
traffic model
flaws,
shortcomings
& limitations
Investigate the
uncertainty
introduced
through model
inputs
Review of
existing
research into
prediction
intervals
54

55. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
Examine
traffic model
flaws,
shortcomings
& limitations
Investigate the
uncertainty
introduced
through model
inputs
Review of
existing
research into
prediction
intervals
Survey of
traffic
forecasting
practitioners
55

56. Research Methodology
Empirically
Derived
Prediction
Intervals
Measure traffic
forecasting
errors
(international)
Literature
review of
forecasting
errors
(international)
Review of
traffic
forecasting
errors
(UK)
Examine
traffic model
flaws,
shortcomings
& limitations
Investigate the
uncertainty
introduced
through model
inputs
Review of
existing
research into
prediction
intervals
Survey of
traffic
forecasting
practitioners
56

57. Empirically
Derived
Prediction
Intervals
Measure
traffic
forecasting
errors
(international)
Review of
traffic
forecasting
errors (UK)
Literature
review on
international
forecasting
errors
Review of
traffic model
short-comings
& limitations
Review of
errors
introduced
through
model inputs
Review of
existing
research into
predictive
ranges
Survey of
traffic
forecasting
practitioners
57

58. Research at Standard & Poor’s
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6
Actual/Forecast Traffic
Global Toll Road Sample (2005)
Normal (0.77, 0.26), n = 104
58

59. Empirically
Derived
Prediction
Intervals
Review of
traffic
forecasting
errors
(international)
Review of
traffic
forecasting
errors (UK)
Literature
review on
international
forecasting
errors
Review of
traffic model
short-comings
& limitations
Investigate the
uncertainty
introduced
through
model inputs
Review of
existing
research into
predictive
ranges
Survey of
traffic
forecasting
practitioners
59

60. Forecasting Inputs
• Forecasts of population are a key input for many (most?)
transport demand models
• Population forecasting should be relatively easy
– We know the population today
– There is a limited set of influences
• Births
• Deaths
• Migration
60

61. Small-Area Population Forecasts
61

62. Small-Area Population Forecasts
0%
10%
20%
30%
40%
50%
60%
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
MeanAbsolutePercentageError
Forecast Horizon (years)
Sources: Smith & Shahidullah (1995), Simpson et al (1997), Smith et al (2001), Shaw (2007) and Rayer et al (2009)62

63. Small-Area Population Forecasts
0%
10%
20%
30%
40%
50%
60%
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
MeanAbsolutePercentageError
Forecast Horizon (years)
Sources: Smith & Shahidullah (1995), Simpson et al (1997), Smith et al (2001), Shaw (2007) and Rayer et al (2009)63

64. Predictive Error v Sample Size
64

65. Predictive Error v Sample Size
0
10
20
30
40
50
60
70
80
500
2,000
3,500
5,000
6,500
8,000
9,500
11,000
12,500
14,000
15,500
17,000
18,500
20,000
21,500
23,000
24,500
26,000
27,500
29,000
30,500
32,000
33,500
35,000
36,500
38,000
39,500
41,000
42,500
44,000
45,500
47,000
48,500
50,000
MeanAbsolutePercentageError
Population Size
10-Year Small-Area Population Forecast Accuracy
Tayman et al (1998)
65

66. Our ‘Zone’ of Interest
0
10
20
30
40
50
60
70
80
100
200
300
400
500
600
700
800
900
1,000
1,100
1,200
1,300
1,400
1,500
1,600
1,700
1,800
1,900
2,000
2,100
2,200
2,300
2,400
2,500
2,600
2,700
2,800
2,900
3,000
3,100
3,200
3,300
3,400
3,500
10-YearPopulationForecastMAPE
Typical Traffic Model Zone Population Sizes
FHWA (recommended range) Ortúzar & Willumsen (reported average)
66

67. Our ‘Zone’ of Interest
0
10
20
30
40
50
60
70
80
100
200
300
400
500
600
700
800
900
1,000
1,100
1,200
1,300
1,400
1,500
1,600
1,700
1,800
1,900
2,000
2,100
2,200
2,300
2,400
2,500
2,600
2,700
2,800
2,900
3,000
3,100
3,200
3,300
3,400
3,500
10-YearPopulationForecastMAPE
Typical Traffic Model Zone Population Sizes
FHWA (recommended range) Ortúzar & Willumsen (reported average)
These are 10-year forecast MAPEs.
Deeper horizon MAPEs will be larger!
67

68. Conclusions 1
• Population forecasts have sizeable error ranges associated with them
• These error ranges increase as the forecasting horizon increases
• Linear relationship?
• These error ranges increase as the study area decreases
• Non-linear inverse relationship
68

69. Conclusions 2
• Population is one of the more predictable variables commonly
used to explain traffic growth
• Try forecasting employment
• ...and allocating it to the correct zones
• Evidence from the US suggests that employment projections can be
(nearly) twice as inaccurate as population forecasts
Transportation Research Board, 2009
• Try forecasting GDP, income or fuel price!
69

70. Empirically
Derived
Prediction
Intervals
Review of
traffic
forecasting
errors
(international)
Review of
traffic
forecasting
errors (UK)
Literature
review on
international
forecasting
errors
Review of
traffic model
short-comings
& limitations
Review of
errors
introduced
through
model inputs
Review of
existing
research into
prediction
intervals
Survey of
traffic
forecasting
practitioners
70

71. DfT Research & Guidance
• WebTAG Unit 3.15.5
• The Treatment of Uncertainty in Model Forecasting
• Use a range about the core scenario growth forecast of:
± 2.5% * √n (n = number of years ahead)
• Formula estimated from national traffic forecast performance
• Functional form is intuitively appealing
• If error variance increases linearly with time...
• ...standard deviation should vary with the square root of the forecast horizon
• Local forecasts will have a (much?) wider range
• NRTF97
• For total traffic at the level of a GOR...the uncertainty should widen to
about ± 25% at the 35th year
• “± 25% at GOR level feels narrow compared to ± 15% (Year 36) at the national level”
• “The range for individual area types/links will be greater than GOR level (>> ± 25%)”71

72. Empirically
Derived
Prediction
Intervals
Review of
traffic
forecasting
errors
(international)
Review of
traffic
forecasting
errors (UK)
Literature
review on
international
forecasting
errors
Review of
traffic model
short-comings
& limitations
Review of
errors
introduced
through
model inputs
Review of
existing
research into
predictive
ranges
Survey of
traffic
forecasting
practitioners
72

73. Traffic Forecasting Accuracy Survey
Survey Respondents:
• International responses (11 countries)
• Consultants/modelling practitioners
• President
• Managing director
• Director of transport planning
• Government officials
• Transport modelling manager
• Senior transport & economics advisor
• Traffic & toll modelling manager
• Academics/researchers
• 4 professors
• Including one of the authors of ‘Modelling Transport’
• Senior lecturers
• Deputy director, centre for transport studies
73

74. Forecasting Accuracy Survey Results
Forecast Horizon Traffic Forecasting Accuracy
Existing Road New Road
Next Day
1 Year
5 Years
20 Years
74

75. Forecasting Accuracy Survey Results
Forecast Horizon Traffic Forecasting Accuracy
Existing Road New Road
Next Day ± 7.5% n/a
1 Year ± 12.5% ± 17.5%
5 Years ± 20% ± 27.5%
20 Years ± 42.5% ± 47.5%
75

76. Putting It All Together…
76

77. Putting It All Together…
77
National Level
(country)
≈ ± 2.5% * √n

78. Putting It All Together…
78
National Level
(country)
Regional Level
(state)
≈ ± 2.5% * √n
≈ ± 5% * √n

79. Putting It All Together…
79
National Level
(country)
Regional Level
(state)
Local Level
(city)
≈ ± 2.5% * √n
≈ ± 5% * √n
≈ ± 7.5% * √n

80. So This is What Local Traffic
Forecasts Should Look Like...

81. Empirically-Derived Prediction Intervals
81
1990 2000 2010 2020 2030 2040 2050 2060
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
200,000
AADT
Project A
www.robbain.com

82. Empirically-Derived Prediction Intervals
82
2005 2010 2015 2020 2025 2030 2035 2040 2045
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
AnnualTransactions(000's)
Project B
www.robbain.com

83. Empirically-Derived Prediction Intervals
83
2010 2015 2020 2025 2030
10,000
20,000
30,000
40,000
50,000
60,000
70,000
AnnualRevenue($000's)
Project C
www.robbain.com

84. Further Information
84

85. 85

86. Further Information
86
All Research Papers & Published Reports
Available for Free Download From:
www.robbain.com