A short presentation to my internal peer group on some of the potential shortcomings of current penetration testing practices and what might be done about it.

1. A customers perspective
1
Internal Practitioners Conference, May 2013
Phil Huggins

2. I have been
 Infrastructure penetration tester - late 90s
 Application penetration tester – early 00s
 Security Architect – till now
 Client-side advice
 LargeGovernment & Commercial Programmes of work
 Handling:
▪ System suppliers
▪ Pen test suppliers
▪ Client andThird Party security stakeholders
▪ ClientOperational teams
▪ Client Project teams
 I am an unusual customer of pen tests
 I understand what I’m buying and why.
2

3. 3
Gather
Information
Expert
Schema
Insight
Define
Action
Scan & Exploit
Characterise
Vulnerabilities
Understand
Causes &
Impacts
Recommend
Prioritised
Mitigations
SENSEMAKING
PENETRATION TESTING

4.  Team of technical guys with CREST,TIGER or
CHECK certifications
 A written methodology owned by the test
company
 A lot of pen testing tools
 A week or two of technical work
 A week of report writing
4

5.  Executive summary
 At least one graph
 Names of the pen testers involved
 Description of the commercial scope
 Extensive prose account of what was done
 Screen shots of tools / error messages
 A table of vulnerabilities
 Mapped to CVE numbers
 Some form of risk / RAG status
 A technical resolution
 A description of recommended further work
5

6.  High day rates for good
testers
 Poor margins as salaries are
high
 Quality can be very
variable
 Same testers over time
 Between testers
 Across companies
 Focus on fail results
 What tests were conducted
and passed?
 Focus on 0-day
 What threat model was used?
 Skipping the insight
 Little or no understanding of
causes and impacts
 Only two parts of the
report actually required
 Summary
 Vulnerability table
6

7.  Better customers
 Security requirements
 Better information
gathering:
 Automation of low hanging
fruit
 Recording of manual testing
 Supply of automation
scripts, raw results & manual
recordings to customer
 Better insight
 Explicit threat model
 Understanding of operational
processes
 Understanding of customer
business
 Better reporting
 Vulnerability tables in excel
 Record full scope
 Vulnerability Metrics:
▪ Ease of exploit
▪ Complexity of fix
▪ Extent of compromise
7

8. http://blog.blackswansecurity.com
8