Model risk and the importance of model risk management has gotten significant attention in the last few years. As financial companies increase their reliance on quants and quantitative models for decision making, they are increasingly exposed to model risk and are looking for ways to mitigate it. The financial crisis of 2008 and various high profile financial accidents due to model failures has brought model risk management to the forefront as an important topic to be addressed. Many regulatory efforts (Solvency II, Basel III, Dodd-Frank etc.) have been initiated obligating banks and financial institutions to incorporate formal model risk management programs to address model risk. In this talk, we will discuss the key aspects of model verification and validation and discuss best practices for stress and scenario testing for model verification and validation for MATLAB-based models. These best practices are meant to provide practical tips for companies embarking on a formal model risk management program or enhancing their model risk methodologies to address the new realities.

1. MODEL RISK MANAGEMENT: BEST PRACTICES FOR MATLAB MODELS
Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that
they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC.
Sri Krishnamurthy, CFA
Founder and CEO
www.QuantUniversity.com
August 14th 2016

2. AGENDA
About QuantUniversity
Stress Testing and Model Risk: A Brief Introduction
The Model Risk Analytics Solution
1
2
3
4
Agenda
Case Study5
Challenges to implementing Model Risk Management

3. ANALYTICS, RISK MANAGEMENT & STRESS TESTING FOR FINANCIAL INSTIT UTIONS
- ADVISORY SERVICES
- PLATFORM TO STRESS TESTING, ANALYTICS AND RISK ASSESSMENT
- ARCHITECTURE REVIEW, TRAINING AND AUDITS

4. INTRODUCTION
• Founder of QuantUniversity.
• Advisory and Consultancy for Financial Analytics
and Model Risk
• Prior Experience at MathWorks, Citigroup and
Endeca and 25+ financial services and energy
customers.
• Charted Financial Analyst and Certified Analytics
Professional
• Teaches Analytics in the Babson College MBA
program and at Northeastern University, Boston
Sri Krishnamurthy
Founder and CEO
4

5. STRESS TESTING AND ANALYTICS– A BRIEF INTRODUCTION

6. STRESSTESTSINTHENEWS
“Of the 33 banks the Fed tested—which included the largest U.S.
banks, like Bank of America, Citigroup, and Wells Fargo.. all were
deemed strong enough to weather a severe economic meltdown
without any help from the government.”

7. MODELVALIDATION
“Model risk is the potential for adverse consequences from
decisions based on incorrect or misused model outputs and
reports. “ [1]
“Model validation is the set of processes and activities
intended to verify that models are performing as expected, in
line with their design objectives and business uses. ” [1]
Ref:
[1] . Supervisory Letter SR 11-7 on guidance on Model Risk
Model Risk and Validation

8. STRESSTESTINGANDMODELRISKMANAGEMENT
Regulatory efforts
EIOPA: Europe-wide stress test for the insurance sector
What’s driving Stress Testing and Model Risk Management efforts?
https://eiopa.europa.eu/Publications/Surveys/eiopa-14-
215_stress_test_2014_specifications.pdf

9. STRESSTEST&SCENARIOTEST
Stress Tests and Scenario Tests
Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

10. DEFINITIONS
Definitions
1. Scenarios :
“A scenario is a possible future environment, either at a point in
time or over a period of time.”
“Considers the impact of a combination of events“
2. Sensitivity Analysis:
“A sensitivity is the effect of a set of alternative assumptions
regarding a future environment. “
3. Stress Testing:
• Analysis of the impact of single extreme events (or risk
factors)
• Extreme but plausible
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

11. STRESSTESTINGANDMODELRISKMANAGEMENT
Model verification:
• Migration of code from an older code-base/ from another
language
• Regression Testing
Model Deployment:
• Model tests to be done prior to deployment
Model Replication:
• Models that replicate alternate models or functionality/strategy
Model Back testing:
• Test how a model performs under different conditions with
historical data
What’s involved with Model Risk Management efforts?

12. MODELLIFECYCLEMANAGEMENTKEYTOMRM
How’s Model Risk Management implemented in the company?
Ref: The Decalogue
http://quantuniversity.com/w9.html

13. CHALLENGES TO IMPLEMENTING MODEL RISK MANAGEMENT

14. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING
Characteristics of Stress and Scenario testing
1. Difficult to build parametric models – Simulation driven
approach necessary
2. Parameter space can explode easily
3. Large number (10s of thousands) of tests needed
4. Human intervention required
5. Company needs are different : Customization required
6. Significant coordination and engagement with multiple
groups needed
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

15. IMPLEMENTINGSYSTEMATICSTRESSTESTINGISCHALLENGING
Challenges to systematic stress testing
1. Model Implementation
• Does it actually work for all intended use cases?
2. Model parameter testing
• How many parameters ?
• How many scenarios ?
3. Model Applicability
4. Model Benchmarking against Reference Implementation
• R vs Proprietary vs MATLAB
5. Model Migration (version)
• Regression Testing v1.0 to v2.0
6. Model use case validation
• Can we use the results to make decisions?

16. MODELLIFECYCLEMANAGEMENTISSTILLEVOLVING
Challenges in Model Life Cycle Management
1. Model risk management considered as a
regulatory/compliance task
2. No one-size-fits-all solution
3. Lack of transparency, standardization and centralized
model management
4. Disparate and disconnected systems (SVN, Sharepoint,
Excel sheets, custom systems)
5. Difficult to track and audit changes
6. Metrics to quantify model risk not established

17. MODELRISK ANALYTICS SOLUTION

18. LEVERAGETECHNOLOGYTOSCALEANALYTICS
Leverage technology to scale analytics
1.64 bit systems : Addressable space ~ 8TB
2.Multi-core processors : Explicit and Implicit Multi-threading
3.Parallel and Distributed Computing : Leverage
commodity/specialized hardware to scale problems
4.Micro services-based solution for fast deployment and
replication
5.Cloud Computing
Ref: Gaining the Technology Edge:
http://www.quantuniversity.com/w5.html

19. ADDRESSINGSTRESSTESTINGCHALLENGES
How can we address challenges in Stress testing?
1. Leverage hardware for scaling
1. Enable developers to plan for stress tests
2. Enable developers to scale without having to
rewrite code
2. Enable developers to run their code
1. Locally
2. On Enterprise Servers
3. Cloud

20. ADDRESSINGSTRESSTESTINGCHALLENGES
How can we address challenges in Stress testing?
3. Facilitate exhaustive and robust testing without having to
get lost in details
1. Enable testers to specify their tests and
configurations
2. Enable benchmarking and comparisons with other
models
4. Facilitate deployment of models built using different
frameworks including legacy code
1. Provide a choice of models (which could be built in
different languages)
2. Enable benchmarking and comparisons with models
built using other languages

21. ADDRESSINGMODELLIFECYCLEMANAGEMENTCHALLENGES
How can we address challenges of Model Lifecycle Management?
1. Provide a single comprehensive environment to manage
model validation tests
1. Enable integration with source control systems
2. Enable model validators to create, run and assess
tests
2. Facilitate all stages of model lifecycle management in a
single environment
1. Allow access to all model related artifacts
(documentation, tests and results)
2. Provide ability to collaborate with all stakeholders
responsible for model risk
3. Enable audit trails and analytics to manage model
risk

22. MODELRISKANALYTICSSYSTEM
ModelRisk Analytics System

23. MODELRISKANALYTICSSOLUTION
Model Risk Analytics Solution
1. Distributed System to support Model Verification, Validation,
Stress and Back Testing
2. Supports MATLAB and other open source packages like R,
Python, SVN
3. Architecture leverages state-of-the-art asynchronous
computing modes
4. Package management
1. Using Custom Client/ Images
2. Using Docker(Different containers for different apps)
5. Supports synchronous and asynchronous modes
6. Built for the cloud but can work on any cloud/internal
infrastructure (private or public)

24. CASE STUDY

25. Value-at-Risk & Conditional Value-at-Risk
• VaR : The predicted maximum loss of a portfolio
with a specified probability level (e.g., 95%) over a
certain period of time (e.g. one day)
• CVaR (Expected Shortfall) : The expected value of
the loss given that the loss exceeds VaR
Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press
Image courtesy: http://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )

26. How to Implement VaR and CVaR?
Methodology:
• Historical
• Variance-Covariance method
• Monte-Carlo simulations
Models are implemented in:
• MATLAB – Production Model
• Python – Alternate model developed by a 3rd-party
vendor
• R – Reference/Benchmark model from a paper

27. Methods to compute VaR and CVaR
• Historical method
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and sort them
in ascending order
2. For a given confidence level (e.g.
95%) , find VaRα (X) such that:
P(X<= VaR α(X)) = α
3. Compute CVaR by taking the average
loss of the tail

28. Methods to compute VaR and CVaR
• Variance-Covariance Method
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and fit a
Normal distribution to obtain mean
and Standard Deviation (µ & σ)
2. For a given confidence level (e.g.
95%) , find VaRα (X) such that:
P(X<= VaR α(X)) = α
Example 95% => -1.65* σ
3. Compute CVaR by taking the average
loss of the tail
(See Yamai and Yoshiba
http://www.imes.boj.or.jp/english/public
ation/mes/2002/me20-1-3.pdf )

29. Methods to compute VaR and CVaR
• Monte-Carlo Simulations
Image Courtesy: http://www.investopedia.com/articles/04/092904.asp
1. Compute Daily Returns and fit a
Normal distribution to obtain mean
and Standard Deviation (µ & σ)
2. Run n Monte-Carlo simulations with
random numbers drawn from a
normal distribution described by (µ &
σ)
3. For a given confidence level (e.g.
95%) , find VaRα (X) such that:
P(X<= VaR α(X)) = α
4. Compute CVaR by taking the average
loss of the tail

30. Model Summary
VaR Model
Historical
Method
Python
R
MATLAB
Variance-
Covariance
Method
Python
R
MATLAB
Monte-Carlo
Simulations
Python
R
MATLAB
Given:
1. Historical Daily price time series for a specified time period
2. Constituents of a 3-asset long-only portfolio
Compute:
VaR and CVaR

31. Model Verification criteria
1. Model Benchmarking
• MATLAB production model vs Alternate Python model vs
Reference R model
2. Parameter sweeps
• Different Confidence Intervals (90%, 95%, 99%)
3. Model Convergence
• How many simulations needed ? (100, 500, 1000)
4. How do different methods compare?
• Historical vs Variance-Covariance vs Monte-Carlo methods

32. VarModel.m

33. ModelRisk Studio facilitates easy structuring of tests

34. Input parameters can be specified for each model

35. ModelRisk Analytics allows you to run the model in multiple environments

37. The Model Testing results can be viewed in an interactive dashboard

42. Model Issues can be tracked, risks evaluated and addressed appropriately

43. FEATURES
Coming soon!
ModelRisk Model Definition Language
• To specify model parameters
ModelRisk Engine Optimization to leverage infrastructure in
the cloud
• Resource constraints:
• Budget & Time constraints
• Priority queues and jobs
• Dynamic scaling and load balancing
ModelRisk Audit to enable periodic testing and ongoing
monitoring of your models

44. MODELRISKANALYTICSBETAPROGRAM
If you are interested in being a part of the Beta program:
If you are interested in knowing more about the Model Risk Analytics system
Or
Want to be part of the Beta program,
email: sri@quantuniversity.com to be added to the interest list
Model Risk Analytics
Beta program

45. MODELRISKMANAGEMENTWORKSHOP
Also available:
A 2-day workshop in Model Risk Management with hands-on case studies
Email: sri@quantuniversity.com to be added to the interest list
Stress Testing and
Model Risk
Management for Quants

46. ANOMALYDETECTIONWORKSHOP
Coming soon:
A 2-day course in methods and best practices in Anomaly Detection
September 19th, 20th , New York
Visit www.analyticscertificate.com/Anomaly for more details
Anomaly Detection
Core Techniques and
Best Practices

47. WHITEPAPERS
To get copies of these articles, go to:
http://www.quantuniversity.com/articlerequest.html
OR
Email: sri@quantuniversity.com

48. Thank you!
Sri Krishnamurthy, CFA, CAP
Founder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be
distributed or used in any other publication without the prior written consent of QuantUniversity LLC.