Scott Crawford, Research Director of Information Security at 451 Research, shares: Why having a Vulnerability Disclosure Policy is now “table stakes” The what, how and why of Vulnerability Disclosure Policy documentation Tangible benefits and tradeoffs of incorporating bug bounties into software development How bug bounties make for a more secure software development lifecycle

1. Bug Bounties and
the Path to Secure Software
Scott Crawford – Research Director, Information Security

2. What’s a Bug Bounty? (And why should you care?)
• Non-software products must often
face rigorous testing against real-
world conditions to demonstrate their
safety and reliability
• But what about software?
4

3. “Hacker-powered security”
• Testing is only as good as the experts
applying their knowledge
• …and “users” are infinitely creative
• Bugs aren’t just about security
• …but security is a top concern
• …and success in finding & fixing is a race
against the clock
• Why not engage the same researchers
that find bugs, to help fix them?
5
An early (and
literal) “bug
bounty”: OS
company (and
aptly named)
Hunter &
Ready, 1983
Photo:
https://twitter.com/senorarroz/status/783
093421204393985

4. Bug Bounty Programs: From concept to maturity
• From (a sometimes contentious) opportunity to
formalized field – and for good reason
• The difference between discovering what others
know or could find out, and remaining in the
dark
• “Everyone gets a free penetration test –
whether or not they get a copy of the report is
up to them.”
6
At Black Hat US 2017, Facebook CSO Alex
Stamos highlighted a conference – and an
industry – that has grown from hacking to
an emphasis on mature and integrated
defense. BBPs align both.

5. Seeing results
• Facebook, Feb 2016: 38% YOY increase in high-
impact submissions1
• Google, June 2016: Up to 50% increase in
amounts paid for high-quality vulnerability
reports2
• Positive impact on safety and life-critical
issues, particularly with growth of IoT and
“smart” systems
7
1 https://www.facebook.com/notes/facebook-bug-
bounty/2015-highlights-less-low-hanging-
fruit/1225168744164016
2 https://security.googleblog.com/2016/06/one-year-of-
android-security-rewards.html

6. Is a BBP for you?
• Chief concern: From bug to bad outcome
• Not just security
• Safety, proper operation, (re)liability,
customer confidence… even cheating!
• 3 key considerations:
• Visibility
• Criticality
• Notoriety
• No longer just for tech companies
• HackerOne: 41% of bug bounties launched
in 2016 from non-tech industries3
8
3 https://www.hackerone.com/resources/hacker-powered-
security-report

7. Where to begin?
• If your digital assets have any exposure to inquisitive
minds…
• You may find that someone has discovered a bug or
vulnerability
• How will you handle it?
• 94% of the Forbes Global 2000 do not have known
vulnerability disclosure policies4
• Every organization with a pubic digital footprint
already has a stake in hacker-powered security
• Why not do it right from the outset?
9
4 https://www.hackerone.com/resources/hacker-powered-security-report

8. 7 steps toward
“hacker-powered” security

9. 1: Create a VDP (and make it easy to find!)
• A vulnerability disclosure policy needs to be
table stakes for any organization with any
public footprint
• Ensures a clear process for communicating
issues
• Enables the many who are well motivated to
help!
• Need not be limited to bugs
• Config errors or other detectable exposures
• Can be as simple as specifying an email
address
• But more detail would be ideal

10. Key elements of a VDP
1. Contact information
2. Clear description of reportable issue types
3. Rules for finding and reporting bugs
4. List of systems available on which to report bugs
5. Communication expectations: When to expect to hear back
after first contact
6. Rules of engagement: How much is OK, and how much is
going too far (i.e. potentially breaking the law)
7. Guidance on how to test may also be provided, such as providing a detailed
summary of the issue, including the
8. Target, steps, tools and artifacts used in discovery (helps the subject org reproduce
the issue)

11. An international standard
• ISO/IEC 29147: Guidelines for the
vulnerability disclosure process
• Freely available at
http://standards.iso.org/ittf/PubliclyAv
ailableStandards/c045170_ISO_IEC_291
47_2014.zip
• Related: ISO/IEC 30111: Guidelines for
vulnerability handling processes (more
on that shortly)
13

12. An NTIA template for VDP
• Brand promise ("The safety and security of
our customers is important to us…")
• Initial program and scope: Which systems and
capabilities are ‘fair game’ vs. ‘off limits’
• "We will not take legal action if…": Clear,
statements to guide good-faith efforts
• Communication mechanisms and process
• Non-binding submission preferences and
prioritizations
• Versioning of the policy
14
https://www.ntia.doc.gov/other-
publication/2016/multistakeholder-process-
cybersecurity-vulnerabilities

13. 2: Corporate comms must know how to handle
• Transparence and responsiveness can go a
long way toward making the best of an
incident or report
• Ensure that corporate communications
staff understand how to recognize and
handle a disclosure
• What not to do
• Automated emails with no follow up
• Cases of Win:
• Buffer breach
• CloudBleed
• GitLab DB incident
15

14. 3: Document and practice vulnerability handling
16
ISO/IEC 29147 – Vulnerability disclosure process
ISO/IEC 30111 – Vulnerability handling process

15. A vulnerability handling process overview
17
Critical:
• A clear,
common set of
rules and
expectations
• Easy to locate

16. Ready to take that next step?
18

17. 4: Select a Bug Bounty Platform Provider
A BBPP can help shoulder the burden – or completely offload – many processes critical
to BBP success:
• Help with design of BBPs
• Provide a software solution to manage submissions
• Expert guidance and implementation of processes vital to BBP success
• Response to reports
• Triage
• Disclosure assistance
• Community support
• Access to the talent pool
19
• Management platform features
• Workflow integration
• Automation and orchestration
• Flexible programs
• Metrics for success

18. BBPPs: Automation and orchestration
• So you’re going to accept incoming bug reports.
Maybe a lot of them
• Think fixing issues will be your biggest problem?
• How about sorting through the noise to triage
duplicates, false positives, or reports out of scope?
• Yelp: First 100 days of a public BBP:
• 564 reports
• 322 duplicates (57%)
• 525 not actionable - That’s 93% of reports that
people would have had to sort through without the
support of triage and workflow automation
20

19. Measuring success: BBP metrics
• What to measure? Bug severity or
quantity? Number fixed?
• How about reducing the number found in a
bounty in the first place?
• Some examples that might help measure
improvements in software quality:
• Number of issues per 1000 lines of code
(LOC)
• Number of critical flaws per development
cycle
• Time to resolve
21

20. 5: Start conservative, with a private BBP, then
6: Go public when comfortable
• Advantages of a private program
• Ability to control all constraints
• Choose testers, limit their number, improve
processes in private
• Finding and fixing flaws before production
release
• Quality and relevance of submissions
• Advantages of a public program
• Actionable results potentially more quickly
• Positive public image
22

21. 7: Refine and expand your program
23

22. Thank you!