You've got security issues to solve. Should you build a solution or buy something pre-built? If you choose to buy, what should your selection criteria be? What questions should you ask the vendor? How should you run a POC? How do you put a security product through it's paces? You can view a recording of this presentation here: https://www.youtube.com/watch?v=SPFam1FtPRY

1. 2019 Infosec Buyers Guide
Adrian Sanabria, VP of Strategy and
Product Marketing at NopSec
Paul Asadoorian, Founder & CTO at
Security Weekly

2. Who are these guys?
Paul Asadoorian
Practitioner
Instructor
Entrepreneur
Product Strategy
Owner, host of the Security Weekly Podcast
Cigar smoker, whiskey drinker
Adrian Sanabria
Practitioner
Consultant
Industry Analyst
Entrepreneur; Product Strategy
Has SEEN things, you know?
Cigar smoker, whiskey drinker

3. Agenda - Buckle up!
Part 0: Problems and Goals
Part 1: Shopping
Part 2: Evaluation
Part 3: Ownership

4. Quick Note about Handouts and Prizes
This ICON means there’s a related
handout in the SlideZip or at the end!
Scan the QR code to the right, or
email sawaba@zip.sh with
infosecworld2019 as the subject.

5. Questions To Ask Yourself
● Can I describe the problem I’m
having?
● What are my goals and
requirements related to this
problem?

6. Questions To Ask Yourself
Is there a chance I already own a workable solution?

7. Questions To Ask Yourself
● Build vs buy: could I roll with
FOSS or build it myself?
● Would it be more or less labor
and cost than going with a
COTS (Commercial
Off-The-Shelf) offering?

8. Build vs Buy: Prioritizing least expense/effort
1. Solve with existing resources
2. Build with existing resources
3. Buy solution; implement/use with existing resources
4. Acquire service (outsource solution)
5. Buy solution; add/train people with existing resources
6. Request more resources & do one of the previous five
Resources = People, Software, Assets and/or Budget

9. Build vs Buy in one simple flowchart*
*Sorry, maybe not that simple, but there’s a copy in the handouts for you to look at more closely later!
This came from: https://medium.com/@sawaba/when-to-purchase-a-solution-to-your-cybersecurity-problem-86de1fa203ba

10. Questions To Ask Yourself
Defining a few metrics:
1. Time-to-Value: The effort necessary to get a product
implemented and doing something useful.
2. Labor-to-Value Ratio: The effort necessary to keep a
product maintained and continuing to be useful
3. True Cost: CapEx + OpEx + Direct Labor costs + Indirect
Labor costs
Examples: Anti-Virus? SIEM? Others?

11. Who Are We Dealing With?
● Dealing with startups -
More like a partnership
● Dealing with established
companies who are
frequently acquiring or
being acquired

12. Agenda - Buckle up!
Part 0: Problems and Goals
Part 1: Shopping
Part 2: Evaluation
Part 3: Ownership

13. Shopping!
● Cutting through vendor marketing
● Understanding the pitch
● “It’s on the roadmap”
● Asking the right questions
● Asking the right people

14. Anatomy of a Pitch
The WebEx Tax (5 minutes sorry Cisco)
Introductions (5 minutes)
About the Company (5 minutes)
The Problem Statement (10 minutes or more)
The Product (10 minutes or more)
Demo (maybe?)
Roadmap, Competition, The Future (remainder)
Next Steps Discussion (Last 5 minutes)

15. Understand the sale by understanding the seller
● Pricing models
○ By endpoint
○ By device
○ Per employee
○ Base + modules
● Sales models
● Compensation
● Channel sales

16. Storytime!
“FireEye Buyer’s Remorse”
1. Aggressive sales/marketing
2. Poorly understood value prop
3. Customers bought for the wrong
reasons
4. Customer Regret
5. High churn, low renewals

17. Ten Eleven questions to ask and why you should ask them
1. What problems and/or challenges do you solve?
2. How is the solution implemented (architecture)?
3. How does your product work?
4. What is the value proposition?
5. Does the product have a ‘killer feature’?
6. On average, how long is the typical deployment?
7. How much effort does the product take to maintain?
8. Who is your competition?
9. What is the one feature that differentiates you?
10. How do you measure the success of the product?
11. What is your ideal customer? Actual software product
manager hard at work...

18. Who Should You Ask?
“It Depends”, however typically these are some
of the better roles to handle product questions
(in order):
1. Founder & CEO/CTO
2. Product Management
3. VP of Marketing or Product Marketing
For technical questions the VP/Head of
Research and Development is typically the best
source.

19. Agenda - Buckle up!
Part 0: Problems and Goals
Part 1: Shopping
Part 2: Evaluation
Part 3: Ownership

20. Evaluation Checklist
1. Define goals and objectives (success metrics)
2. Setup the correct test data and/or environment (its okay to cheat, ala
downloading bad domains vs. looking for them)
3. Testing in the lab, but perhaps with real data from the network or log sources
4. Continually testing various scenarios based on real-world experiences
5. Does it actually work?
6. How much effort will it take to make it work (value:labor ratio)?
7. How long will it take to implement (time-to-value)? 18 hours, days, weeks or
months?
8. How easy (or difficult) will it be to operationalize it?

21. Types Of Evaluations
● Hands-Off - You’ve spoken to the company at a high level
and saw a technical demo. You’ve talked to analysts and
companies using the product and received feedback.
● Open-Source / Free Trial - Typically limited in features,
but allows you to conduct a very scaled down test. May
only involve you, the security person, and if you find
something good, you tell others..
● Pre-Configured Testing - The vendor sets up a test,
using fake data (or data that does not come from your
environment). Allows you to explore all of the functionality
(typically only involves you).

22. Types Of Evaluations (2)
● Evaluating in your own lab - You’ve setup your own
virtual/cloud environment, that does not mirror your production
systems, but allows you to test solutions on your own. This may
involve other people in your organization.
● Evaluating in a mirror - Mirror your production/test/qa/other
environment, do the testing, likely with some others in your
organization.
● Evaluating in production, limited implementation - In
collaboration with other groups, implement the solution in a
small sample of your network/systems. Typically this will include
vendor support, at varying levels.

23. Agenda - Buckle up!
Part 0: Problems and Goals
Part 1: Shopping
Part 2: Evaluation
Part 3: Ownership

24. Long-term Ownership
Consider:
● The True Cost metric
● The post-purchase relationship
○ Second-class citizen?
○ Did you “plan to ditch before you hitched?”
● Is there still technical value?
○ Overlap with other products?
○ Shift in threats?
● Does it still make financial sense?
○ Can you do it cheaper?
○ Hand off to MSSP?
○ Run in the cloud?

25. Avoiding “Shelfware”
The solution has to be practical, meaning it is:
● Aligned with business goals and objectives
● Solves actual problem(s)
● Makes jobs easier, not more difficult Really cool != practical (but I
still want one)

26. Storytime!

27. Storytime!

28. 1. Staff was oblivious
2. Symantec
3. FireEye
4. Phone Calls
1. Staff was very aware
2. Web Scanner (DAST)
3. Network Scanner
4. SAST tool
5. SSLV Malfunction
6. Custom Snort rule
Storytime!

29. Storytime!
Every company breached had security products
What went wrong?
1. Understand the product’s coverage and limitations
2. Understand your staff’s coverage and limitations
3. Learn to use tools effectively
4. Test systems, tools and staff

30. Resources
Product Evaluation Form (Google Doc)
Incident Cost Calculator (Google Sheet)
When to purchase a ‘solution’ to your
cybersecurity problem (Blog Post)
What is your product and what does it do? (Blog
Post)
In the SlideZip:
● From the CISO’s Guide to Startups
○ Slides
○ Handout and Appendicies
○ Vendor Expo Challenge
● Some of the blogs from the left
● Product/Vendor Evaluation Form

31. For the handouts, email sawaba@zip.sh
with infosecworld2019 in the subject or
scan this QR code →
Twitter: @sawaba
Blog: https://medium.com/@sawaba
Twitter: @securityweekly
Email: paul@securityweekly.com
Podcasts:
https://securityweekly.com/subscribe
THANK YOU
Please Fill Out
Your Evaluations