Presentation about ICS Cybersecurity Effectiveness Measurement on Kaspersky ICS Cybersecurity Conference

1. Alexey Lukatsky
Security business development manager
ICS Cyber Security
Effectiveness Measurement

2. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Not Petya / Nyetya Tools
Tactics
• Supply chain and victim to victim pivoting
• Rapid Infection Spread
• Destroyed Countless Systems / Networks
Processes
• Designed to inflict damage as quickly and
effectively as possible.
• Appears to be Ransomware, but is purely
destructive
• Wormable Ransomware
• Designed to Spread Internally Not Externally
• Leveraged Eternal Blue / Eternal Romance and
Admin Tools (WMI/PSExec)
• Advanced Actor associated with a Nation State
• Destructive Attack Masquerading as Ransomware
• Most Expensive Incident in History
Description

3. ICS Kill Chain
1
2
3
4
5
6
7
8
Conficker
APT1
Иран vs
США
BE3
HAVEX
Stuxnet
Ukraine
2016
WannaCry
Neytya

4. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why we need to
measure our
effectiveness?
• Good security not visible
• We want to show that we
work well
• Top management often wants
to compare itself with others
• We want to see the dynamics

5. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Rare Remote Possible Likely Very likely
Catastrophic 6 7 8 9 10
Significant 5 6 7 8 9
Moderate 4 5 6 7 8
Minor 3 4 5 6 7
Insignificant 2 3 4 5 6
Accept
(score = 2,3)
Monitor
(score = 4,5)
Manage
(score = 6)
Avoid / Resolve
(score = 7)
Urgently avoid/
Resolve
(score = 8, 9, 10)
“Best practices” for security measurement
• Not specifically, not quantitatively, conditionally…
Impact
Probability

6. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

7. Cybersecurity is state of
protection of the interests of
enterprise stakeholders in the
information area, determined by
the totality of balanced
interests of the individual,
society, state, and business
Or process?
Not
important!

8. Efficiency/effectiveness
is the quantifiable
contribution to the
achievement of ultimate
goals

9. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What goals can we have?
• Fulfillment of NERC CIP or ISA/IEC 62443 requirements
• Categorization of all CI objects
• Certification of key processes for ISO/IEC 27019
• Reduce the number of ICS cybersecurity incidents to 3 per month
• Implementation of secure remote access to ICS for contractors
• Reduce downtime from ICS cybersecurity incidents to 2 hour on
average
• Cost reduction for ICS cybersecurity for 15%

10. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Operational (наиболее привычные)
• Realtime, day-to-day
• Logs, rules, signatures, etc.
• How effective is your security measures?
• Tactical
• Change control
• Scorecards and audits
• How effective is your security program?
• Strategic
• Corporate risk and business alignment
• How are we secure?
Strategic
Tactical
Operational
Measurements are different

11. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tactical metrics examples
• Incidents requiring manual cleanups
• Mean-Time-to-Fix
• Also TTR (Time-to-Recovery) or TTC (Time-to-Contain)
• Mean-Time-to-Detect
• Mean-Time-to-Patch
• Involvement of staff in cybersecurity activities
• Mean cost to mitigate vulnerabilities

12. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tactical metrics examples
• % of ICS without known severe vulnerabilities with CVSS >7.0
• % of changes with security review
• % of changes with security exceptions
• ICS cybersecurity budget allocation (% of total, IT, cybersecurity,
ICS)
• Compliance rate
• Cost of incidents

13. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Tactical metrics examples
• Time between creating and closing a ticket for an incident
• Ratio of open and "closed" incident reports
• Ratio of incidents and tickets
• Number of repeat incidents
• Ratio of communication methods (e-mail / calls / portal)
• Number of false positives (non-existent incidents)

14. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
SMART principle for metrics selection
• SMART – Specific, Measurable, Achievable, Relevant, Timely
• As concretely as possible, without double interpretations, for the right
target audience
• The result should be measurable, not ephemeral
• Why choose a goal that is unattainable?
• Relevance to goals
• Timeliness and relevance

15. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
SMART usage example for ICS Cybersecuirty
Characteristic Example of bad metric Example of good metric
Specific The number of failed login attempts to
the HMI
The number of failed login attempts
to the HMI for one week for one
employee
Measurable Income from the implementation of an
ICS cybersecurity
The employees loyalty level about
ICS Cybersecurity
Achivebale The absence of cyber security
incidents in ICS for the current
quarter
The number of ICS cybersecurity
incidents in the current quarter <5
Relevant The number of opened projects for
ICS cybersecurity
The number of completed on time
projects for ICS cybersecurity
Timely The number of patched ICS nodes
last year
The number of unpatched ICS
nodes current year

16. How to move from
hundreds of operational
metrics to one or two
strategic?

17. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
From individual metrics to measurement program
• EPRI (Electric Power Research
Institute) Research Program
• Creating Security Metrics for the
Electric Sector (Parts I, II, III, IV)
• Applicable to a wide range of
industrial enterprises outside the
electric power industry
3
strategic
metrics
10 tactical
metrics
45 operational
metrics

18. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
From individual metrics to measurement program
Strategic Metric Name Tactical Metric Name
Protection Score Network Perimeter Protection Score
Endpoint Protection Score
Physical Access Control Score
Human Security Score
Core Network Vulnerability Control Score
Core Network Access Control Score
Data Protection Score
Security Management Score - Protection
Detection Score Threat Awareness Score
Threat Detection Score
Security Management Score - Detection
Response Score Incident Response Score
Security Management Score - Response

19. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
From individual metrics to measurement program
Tactical Metric Name Operational Metric Name
Network Perimeter Protection Score Mean Access Point Protection Score
Mean Wireless Point Protection Score
Mean Internet Traffic Protection Score
Mean Count-M Malicious Email
Mean Count-M Malicious URL
Mean Count-M Network Penetration
Security Management Score - Protection Security Budget Ratio
Security Personnel Ratio
Cybersecurity Risk Tolerance Score

20. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
From individual metrics to measurement program
Operational Metric Data input to the Formula
Mean Access Point Protection Score Number of inbound connections per day
Number of dropped inbound connections per day
Number of all alerts per day
Number of security alerts per day
Number of probes per day
Number of confirmed DOS attempts per month
Чnumber of confirmed intrusion attempts per month
Number of confirmed incidents that required human
intervention per month

21. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Automation tool: EPRI MetCalc

22. What does the business
think of all these metrics?

23. Business thinks about cybersecurity, but in its
own way
Reservoir
Pump
Water intake
Water
treatment
plants
Underground
tank
Pump
Distribution
Cleaning with
reagents,
ozone and
coal
Sump
Flats /
Houses
Water
meter
Smooth operation
Correctand
uninterrupted
bills
Smooth operation
Continuous
diagnosis
Telemetry
control
Continuous
monitoring
Proper dosing
FZ-152
Order №31 CIP Law
Water supply process

24. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The difference in the perception of top
management and cybersecurity / IT / ICS
Cybersecurity / IT / ICS
• Deep dive to details
• Unwillingness to share
collected data
• Data for data, not for
decisions
• What? Where? When?
Top management
• Bird's-eye view
• Data for decision making
• What will happen? What
to do?

25. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Time
Productivity
0
20
40
60
80
100
А
В
С
ВТ
Т1 Т2 Т3
D = System failure / disaster
R = The possibility of attenuating or mitigating the
effect before or during a negative event
A = The ability to absorb and degrade
В = Lower limit; threshold value
ВТ = Lower limit duration
С = Ability to return to baseline
D → R
How does a business see security incidents?
Reduce А?
Reduce Вт?
Reduce С?
Reduce Т1, Т2 and Т3?

26. Let's try to reformulate our goals
Profit increase
Geo expansion
Sales increase
Production optimization
Reduction in logistics
costs
Loss reduction
X hours of downtime due
to ransomware
Y hours of process
downtime due to
DoS/DDoS-attack
Z hours of employee
downtime due to spam
N rubles fine from
supervisory authorities
Business
Cybersecurity

27. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
From the “for myself” measurement to the
measurement for business
75%
55%
Q2
Q1
The number if incidents
by sources
The number of ICS
incidents
Downtime
Incidents dynamics
Contracts loss
$35M127

28. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cybersecurity incidents loss types
Productivity
•Downtime
•Deterioration of the psychological climate
Response
•Incident forensics
•PR-activity
•Support Service
Replacement
•Equipment replacement
•Re-entry of information
Fines
•Legal costs, pre-settlement
•Suspension of deals
Competitors
•Know-how, commercial secrets
•Customer churn, overtaking by competitors
Reputation
•Goodwill
•Decrease in capitalization, stock price
Other
•Rate downgrade
•Decrease in profitability

29. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Impact categories Insignificant Minor Moderate Significant Catastrophic
Finance impact of
more than $Y
$1М $5М $10М $50М $100М
Let's be more specific and measure the money
• The cost of direct losses from disruption of business operations
• Business Transaction Recovery Cost
• Decrease in stock prices (dumb indicator, but sometimes also measurable)
• Fines
• Lost profit (if you can count it)
• Decrease in customer loyalty
• Replacing equipment or re-entering information
• Interaction with affected customers, etc.

30. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Questions for defining strategic business metrics
of cybersecurity
• What will stop or slow down operations in your organization?
• What will lead to a decrease in profits / revenue / margin / market
share of your company?
• What will lead to a decrease in the quality of the product / service?
• What will lead to a negative impact on the goal of the company /
business unit / business project / executive sponsor?

31. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Impact categories Insignificant Minor Moderate Significant Catastrophic
Outage of more
than X customers
10 customers 100 customers 500 customers 1000 customers 5000 customers
Business
operations
disruption of >= Z
min / hours / days
1 hour 4 hours 8 hours 2 days 5 days
Serious injury to
>= A people
0 people 0 people 1 person 10 people 50 people
Breach of data for
>= B customers
100 customers 1000 customers 5000 customers 10000 customers 100000 customers
Loss of >= C
customers
5 customers 10 customers 25 customers 50 customers 100 customers
Loss of market
share for D%
0% 0% 1% 3% 7%
Productivity loss
for E%
0% 1% 3% 5% 10%
If you can’t count in money?

32. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The duration of an cybersecurity incident in terms
of cybersecurity and business
§ The influence level and price components of an incident changes
over time
This illustration can be used to estimate recovery time after an attack
RPO – Recovery Point Objectives, RTO – Recovery Time Objectives, MAD – Maximum Allowable Downtime

33. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Impact categories Insignificant Minor Moderate Significant Catastrophic
Reduction of
power generation
by F megawatts
Power reduction is
acceptable
Power reduction is
acceptable
100 MW 1000 MW 10000 MW
Impact categories Insignificant Minor Moderate Significant Catastrophic
Publications in
mass media
Absent In local consumer
print media
On local TV or in
local industry
publications
On national TV or
in national
consumer print
media
Highlighted
broadcasts or
reporting on
national TV or in
national industry
print media
Industry specific metrics

34. How to measure
cybersecurity for a
business, but not with
money?

35. Can compare yourself with competitors?
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
План & бюджет
Организация
Защитные меры
Архитектура
Процессы и операции
Осведомленность
Реагирование
Управление уязвимостями
Оценка рисков
Корпоративное управление
В среднем по отрасли
У нас
Tricks: instead of comparing with competitors (if there is no data),
you can compare yourself in different states (there was - now - in a
year - ideal)

36. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
5 important metrics
• % of cybwersecurity activities unlinked to business goals
• Number of projects / activities linked to business goals
• % of projects / assets / services that are important for business that
do not meet cybersecurity requirements
• For example, uncontrolled remote access by contractors
• % of projects / assets / services that are important for business and
whose security measures are inadequate or ineffective
• Or for whom during the incident the response plan did not work
• The likelihood of providing services during an cybersecurity incident
You can still play with the risks ...

37. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Common errors in effectiveness measuring
• Choosing hundreds of metrics instead of focusing on strategic
• Measuring what is easier to measure instead of focusing on
measurement goals
• Lack of business focus
• Focus on operational result-oriented metrics instead of evaluating
process performance
• Lack of context
• Cybersecurity price reduction with incidents growth

38. Key Success Factors
• You must understand what you are doing
in the field of information security
• You must understand your business
• You must understand your target
audience
• You must be able to combine these three
elements together
• You need to know where the data is
• You must be able to code/program

39. ANewLookat
Cybersecurity
Measurement

40. Thank you!
security-request@cisco.com