We generally assume the past is a good guide to the future, but well do we even know the past? What effect does this uncertainty when estimating inputs have on the notoriously unstable algorithms for portfolio optimization? I explore this issue, look at some commonly used solutions, and also introduce some alternative methods.

1. Portfolio optimisation
when you don’t know
the future
(or the past)
Robert Carver
QuantCon, Singapore
29th
September 2017

2. Legal boilerplate bit:
Nothing in this presentation constitutes investment
advice, or an offer or solicitation to conduct
investment business. The material here is solely for
educational purposes.
I am not currently regulated or authorised by the
FCA, SEC, CFTC, MAS, or any other regulatory
body to give investment advice, or indeed to do
anything else.
Futures trading carries significant risks and is not
suitable for all investors. Back tested and actual
historic results are no guarantee of future
performance. Use of the material in this
presentation is entirely at your own risk.

3. ● Two nightmares: Instability and Uncertainty
● The uncertainty of the past
● Common solutions
● Alternatives

4. ● Two nightmares: Instability and Uncertainty
● The uncertainty of the past
● Common solutions
● Alternatives

5. Instability

13. Instability...
● High correlation + small difference in  or  →
extreme portfolios
“In a mean-variance optimization
framework, accurate estimation of the
variance-covariance matrix is
paramount (… yeah and the mean
vector is pretty important too).”
Source: https://en.wikipedia.org/wiki/Portfolio_optimization
My comment in bold

14. Uncertainty
● Can we measure  or  accurately?
● NO
● We can’t predict the future – must assume the
past will repeat itself
● Need to fit a statistical model to historic returns
(e.g. estimate  and )

15. Need to fit a statistical model:
● Do we have the right model?
– Can use more complex models, with higher moments. Hard to
estimate. Markowitz optimisation built for Gaussian returns – hard
to change. Correct utility preferences?
● Will the model change?
– Could try Markov process. Many more parameters to estimate
(Given K states, KN+[K-1]2). Overlap with historical parameter
estimates.
● Do we have accurate historical parameter estimates?
– Easy to quantify. Can capture changes in model. Worth exploring…
“The Uncertainty of the Past”

16. ● Two nightmares: Instability and Uncertainty
● The uncertainty of the past
● Common solutions
● Alternatives

17. SP500 US10 US5
Return 0% 4.1% 3.1%
Std. dev 18.7% 7.1% 4.7%
Sharpe ratio 0.0 0.59 0.66
SP500/US5 SP500/US10 US5/US10
Correlation -0.28 -0.27 0.96
Based on weekly excess returns (futures, including rolldown) from
01/1998 to 11/2016
SP500 US10 US5
Weight 12.9% 0% 87.1%

18. Bootstrapped Correlations
10,000 monte carlo draws with replacement; same length as data history

19. Bootstrapped standard deviation
10,000 monte carlo draws with replacement; same length as data history

20. Bootstrapped means
10,000 monte carlo draws with replacement; same length as data history

21. Bootstrapped Sharpe Ratio
10,000 monte carlo draws with replacement; same length as data history

22. Estimates get better with more
data...
Upper 90%, Lower 10%, quantile of distribution of annualised standard deviation of US 5
year

23. … up to a point...
Upper 90%, Lower 10%, quantile of distribution of correlation of US 5 year and S&P 500

24. … which for Sharpe Ratios isn’t a
very good point
Upper 90%, Lower 10%, quantile of distribution of annualised Sharpe Ratio of US 5 year

25. Mean SP500 US10 US5
Lower -5.7% 2.0% 1.7%
Upper 5.6% 6.3% 4.6%
Correlation SP500/
US5
SP500/
US10
US10/
US5
Lower -0.33 -0.34 0.95
Upper -0.23 -0.21 0.97
Std dev SP500 US10 US5
Lower 17.3% 6.4% 4.2%
Upper 20.1% 7.8% 5.3%
Sharpe SP500 US10 US5
Lower -0.29 0.28 0.35
Upper 0.28 0.90 0.99
Lower: 10th
percentile of bootstrapped distribution, Upper: 90th
percentile

26. Correlation
SP500/US5
SP500 US10 US5
Lower 15.0% 0% 85.0%
Central 12.9% 0% 87.1%
Upper 10.8% 0% 89.2%
Std dev
SP500
SP500 US10 US5
Lower 13.0% 0% 87.0%
Central 12.9% 0% 87.1%
Upper 12.8% 0% 87.2%
Mean / SR
SP500
SP500 US10 US5
Lower 4.1% 0% 95.9%
Central 12.9% 0% 87.1%
Upper 17.2% 0% 82.8%
Mean / SR
US5
SP500 US10 US5
Lower 13.5% 86.5% 0%
Central 12.9% 0% 87.1%
Upper 12.9% 0% 87.1%
Portfolio weights when changing appropriate estimate to different quantile
of the distribution; keeping all other estimates constant

27. ● Land of low correlations:
– Eg Asset classes (also different vol)
– Relatively benign results
– Almost impossible to distinguish historically estimated Sharpe Ratios
● Land of high correlations:
– Eg Stocks in the same country (also similar vol)
– Trading rules with same algo, different parameters
– Extreme results
– Theoretically possible to distinguish estimated Sharpe Ratios: but
similar assets more likely to have similar performance.

28. Years before we get 95%
confidence in a mano et mano
comparision...

29. Costs….
● Uncertainty of Sharpe Ratio mostly comes from
returns…
● Uncertainty of post-cost returns mostly comes
from pre-cost returns…
● Costs can be estimated with relatively high
accuracy…
– Caveat: Large size – Illiquid markets - High frequency trading
● Differential in costs should be treated differently
to differential in pre-cost returns

30. ● Two nightmares: Instability and Uncertainty
● The uncertainty of the past
● Common solutions
● Alternatives

31. Common Solutions...
● Constraints eg minimum and maximum weights
● Ignore (some) estimated values eg assume
identical means
● Change the inputs (bayesian) to reflect
uncertainty
● Repeat the optimisation (bootstrapping)

32. Ignore some values
Correlation Sharpe SP500 US10 US5
Est Est
12.9% 0% 87.1%
Est Eq
20.2% 0% 79.8%
Eq Est
0% 37.0% 63.0%
Eq Eq
13.2% 34.8% 52.0%

33. Bayesian
Average between estimates and a prior

34. Bayesian
 = (1-w)p + wE
E estimated mean
w shrinkage factor
P prior
We can apply similar formula to 

35. Bayesian
Advantages
– Intuitive results (with no shrinkage will recover original
optimisation results; with full shrinkage will recover prior;
shrinkage related to uncertainty)
– Can be used with constraints
– Computationally fast (single optimisation)
Disadvantages
– What prior to use? (no cheating!)
– How much shrinkage?

36. Bayesian – Black Litterman style
Take estimated 
Take some prior weights wp
Infer p
 = (1-w)p + wE

37. Bayesian – Black litterman style
Advantages
– Intuitive results (with full shrinkage will recover prior weights, with
none will recover original optimisation results)
– Don’t have to come up with priors for  or .
– Focus Bayesian weapon on highest priority target: 
Disadvantages
– What prior weights to use? (market cap, heuristic cluster, equal
weights...)
– How much shrinkage on mean?
– No shrinkage on correlations or standard deviations – but still
uncertainty
– Constraints aren’t used on the first, inverse, optimisation – prior
weights have to be feasible or results will be distorted

38. Bootstrapping
● Non parametric: Repeat optimisation using random
subsets of data with replacement, take average of
weights
● Parameteric: Repeat optimisation using random
draws from fitted distribution, take average of
weights. Good with small samples. Bad for non
Gaussian (joint) returns.
● Michaud parametric: Repeat optimisation to create
efficient frontier, take average of efficient frontiers.

39. Bootstrapping
Advantages:
– Intuitive results depending on amount of uncertainty in data
(noise, horizon length)
– Can be combined with other methods
Disadvantages
– Final weights not intuitive in comparision to eg Bayesian
– Results can sometimes be unsatisfactory, especially with high
correlations
– Computationally intensive (especially rolling out of sample)
– Inefficient use of constraints

40. ● Two nightmares: Instability and Uncertainty
● The uncertainty of the past
● Boostrapping vs Shrinkage
● Alternatives

41. Alternatives:
● Risk weighting
● Clustering
● Heuristic methods
● Alternatives to portfolio construction:
– Fama-French factor sort (long / short baskets)
– Parametric portfolio policy [weights = f(factors)]
Can combine these, and ‘traditional’ methods.

42. Risk weighting
● Produce weights assume equal volatility, then scale weights
accordingly.
● Good for:
– Produces a more stable optimisation
– Volatility estimate is seperated (can account for differential noise, and use
other inputs eg implied vol, and account for vol tail risk)
● Bad:
– Investors without leverage, and with high risk appetite, and with low volatility
assets (asset allocation problems)
● Commonly used by CTAs
– Use to determine risk weighting of investment strategies, not optimise positions

43. Risk weighting + Optimisation
SP500 US10 US5
Risk weight 21.8% 0% 78.1%
Std. dev 18.7% 7.1% 4.7%
Cash weight
(estimated)
2.67 3.52 5.32
Final cash weight 6.6% 0.0% 93.4%
Standard deviation: 4.2%
Mean: 2.3% (assuming all assets have SR 0.4)
Assumed mean of all stock portfolio: 7.5%
→ avoid low volatility assets in portfolio if possible
→ add constraints on lower vol assets

44. Risk weighting + ignore estimate(s)
Equalise Sharpe SP500 US10 US5
Risk weight 49.7% 21.1% 29.2%
Std. dev 18.7% 7.1% 4.7%
Cash weight
(estimated)
2.65 2.97 6.21
Final cash weight 22.4% 25.1% 52.5%

45. Risk weighting + ignore estimate(s)
Equalise
Correlations
SP500 US10 US5
Risk weight 0% 46.8% 53.2%
Std. dev 18.7% 7.1% 4.7%
Cash weight
(estimated)
0 6.59 11.3
Final cash weight 0% 36.8% 63.2%

46. Risk weighting + ignore estimate(s)
Equalise
Correlations & SR
SP500 US10 US5
Risk weight 33.3% 33.3% 33.3%
Std. dev 18.7% 7.1% 4.7%
Cash weight
(estimated)
1.8 4.7 7.1
Final cash weight 13.1% 34.6% 52.3%

47. Bayesian with risk weighting
 = (1-w)p + wE
E estimated Sharpe ratio
w shrinkage factor
P prior Sharpe Ratio
We can apply similar formula to correlations, 

48. Risk weighting plus Bayesian
Corr->
SR
0.0 0.25 0.5 0.75 1.0
0.0 0.21, 0, 0.78 0.08, 0.11, 0.81 0, 0.3, 0.7 0, 0.37, 0.63 0, 0.4, 0.6
0.25 0.28, 0, 0.72 0.16, 0.14, 0.7 0, 0.32, 0.68 0, 0.38, 0.62 0, 0.42, 0.58
0.5 0.34, 0, 0.66 0.26, 0.17, 0.57 0.12, 0.31, 0.57 0, 0.4, 0.6 0, 0.43, 0.57
0.75 0.41, 0.0, 0.59 0.36, 0.21, 0.43 0.28, 0.29, 0.43 0.13, 0.38, 0.48 0, 0.47, 0.53
1.0 0.5, 0.25, 0.26 0.48, 0.26, 0.26 0.46, 0.27, 0.27 0.42, 0.29, 0.29 0.33, 0.33, 0.33

49. Risk weighting plus Bayesian
● Need high degree of shrinkage on SR to get non
zero weights → we don’t know much about SR
● Too much shrinkage on correlation destroys
information
● Precise optimal shrinkage depend on amount of
data available. Can use fake data + monte carlo /
closed form to derive optimal shrinkage.
● Rule of thumb: 0.95 on SR, 0.5 on correlations
● Optimal shrinkage is context specific.

50. Risk weighting plus Bayesian
SR shrinkage 0.95 (prior SR 0.25), correlation 0.5 (prior correlation matrix, off diagonal 0.5)

51. Bayesian with risk weighting – Black
Litterman style
Take estimated 
Take some prior weights wp
Infer Sharpe Ratio p
 = (1-w)p + wE

52. Bootstrapping with risk weights

53. Bootstrapping with risk weights &
identical SR

54. Clustering
● Group assets together into groups
● Create sets of portfolios that are more tractable
● Clustering can be formal (eg k-means) or heuristic
(‘handcrafting’)
● Clustering can work on multiple levels
● Optimisation within groups can be:
– Equal Weights (in risk weighting space)
– Fully optimised
– Heuristic optimisation

55. Clustering: equal risk weight
Group Intra group
weight
Group weight Risk Weight
S&P 500 Equity 100% 50% 50%
US 5 year Bonds 50%
50%
25%
US 10 year Bonds 50% 25%

56. SP500 US10 US5
Risk weight 50% 25% 25%
Std. dev 18.7% 7.1% 4.7%
Cash weight
(estimated)
2.67 3.52 5.32
Final cash weight 23.2% 30.6% 46.2%
Clustering: equal risk weight

57. Heuristic methods
● Miimicks what (experienced) people do when
hand optimising – high comfort level
● Should be informed by theory / experiement
● Sometimes hard to backtest - be careful of
implicit in sample fitting
– I use other methods to backtest my system, but then use
heuristic weights in live trading.
● Suitable for ‘one off’ exercises, eg strategy risk
allocation.

58. Heuristic methods
● Hueristic grouping when clustering
● Equal weight within groups
● Use ‘rule of thumb’ correlations, not estimates
● Heuristic optimisation using correlations
● Heruistic adjustment for different Sharpe Ratios
– Can apply different heuristic for pre-cost, and costs
● Use ‘rule of thumb’ risk levels, not estimates
● Heuristic risk levels (constant risk, or assume equal)

59. Heuristic method for correlations in
clustered groups (risk weightings)
Group of one asset
100% to that asset
Any group of two assets 50% to each asset
Any size group with identical correlations Equal weights
Four or more assets without identical correlations Split groups further or differently
until they match another row
Three assets with correlations AB, AC, BC: Weights for A, B, C
3 assets correlation 0.0, 0.5, 0.0 Weights: 30%, 40%, 30%
3 assets correlation 0.0, 0.9, 0.0 Weights: 27%, 46%, 27%
3 assets correlation 0.5, 0.0, 0.5 Weights: 37%, 26%, 37%
3 assets correlation 0.0, 0.5, 0.9 Weights: 45%, 45%, 10%
3 assets correlation 0.9, 0.0, 0.9 Weights: 39%, 22%, 39%
3 assets correlation 0.5, 0.9, 0.5 Weights: 29%, 42%, 29%
3 assets correlation 0.9, 0.5, 0.9 Weights: 42%, 16%, 42%

60. Correction for heterogenous groups
Given N assets with a correlation matrix of returns H and risk
weights W summing to 1, the diversification multiplier will be:
1 / [ ( W x H x WT )1/2 ]
1)Calculate diversification multiplier for each group
2)Multiply group weight by multiplier
3)Renormalise group weights to add up to 100%
4)Apply at all levels of a hierarchy

61. Correction for heterogenous groups
SP500 US10 US5
Start Risk weight 50% 25% 25%
Diversification
multiplier 1 1.01015 1.01015
Adjusted risk
weight 50% 25.3% 25.3%
Normalised risk
weight 49.7% 25.1% 25.1%

62. Heuristic correction for relative
Sharpe Ratios
-0.5 -0.4 -0.3 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.4 0.5
0
0.5
1
1.5
2
2.5
(A) With certainty eg costs
(B) Without certainty, more than ten years data
rule of thumb

63. Heuristic correction for relative
Sharpe Ratios
SP500 US10 US5
Start risk weight 50% 25% 25%
Sharpe Ratio 0.0 0.59 0.66
Adjustment 0.75 1.11 1.15
Multiplied risk
weight
37.5% 27.8% 28.8%
Normalised risk
weight
40% 29.5% 30.5

64. Conclusions...
● Be aware of uncertainty!
● Uncertainty in Sharpe Ratios is bad – very bad!
● Consider the application, especially natural differences
in correlation and vol.
● Use risk weighting; if no leverage & with mixture
including low vol assets apply constraints
● Use clustering (+ equal risk weights if it makes sense)
● Heuristics are good for real, one off, exercises.
● Shrinkage good for backtests.

65. And the final word goes to...
“I should have computed the historical co-variances of
the asset classes and drawn an efficient frontier.
Instead I visualised my grief if the stock market went
way up and I wasn't in it – or if it went way down and I
was completely in it.
My intention was to minimise my future regret.
So I split my contributions 50/50 between bonds and
equities”
Harry Markowitz, quoted in “The Intelligent Investor” by Jason
Zweig

66. My website:
systematicmoney.org
My blog:
qoppac.blogspot.com
Some python:
github.com/robcarver17/
Twittering:
@investingidiocy