You’ve heard about security startups on the bleeding edge and you’ve heard early adopters sharing success stories at conferences. Meanwhile, legacy security paradigms have been falling (and failing) around us. This session will discuss building a continuous program for evaluating startups and new technologies (on a budget) while avoiding unnecessary risk and instability to existing infrastructure.
2. Enjoy the presentation, but there’s more!
2
Three ways to get a copy of this session’s supplemental handout:
1. Send an email to sawaba@zip.sh with rsa2016 as the subject
2. Go to http://zip.sh/z/sawaba/rsa2016
3. Scan the QR code to the right
Note: I’ve been told QR scanning might not work
well in this environment, so YMMV.
3. Mature security products haven’t kept up
Products from startups are unproven - an unknown risk
Rock and a hard place?
Why are we here?
3
The process of buying security products for the
enterprise is broken
4. What are we up to?
4
Agenda
What you need to know about
startups before doing business with
them
This isn’t your CFO’s due diligence...
Due diligence in a 6-stage process
Advice and stories from the trenches
Goals
Learn tips and advice for fixing the
process of buying security products
Understand how doing business with
startups is different
Leave with a framework to put into
practice and the resources necessary
to be successful with it
6. The security industry moves fast
WE SEE… WE HAD…
6
9 new startups
every month
5
new categories
every six
months
1238 enterprise security
companies in our
database
134
security M&A
deals in 2015,
worth…
$9.98
billion, with
an average
of…
$192m paid by
acquirers
7. Greener grass
7
security start-up
noun si-ˈkyu̇r-ə-tē ˈstärt-ˌəp
A new company you will pay to do a better job at
something you already pay an older company for,
though the new company has less experience doing it,
there are no guarantees it will do a better job and you’re
going to keep paying the older company.
8. Why do security startups exist?
8
• Displace existing vendors
• Address (security) gaps
• Solve technical challenges
• Address new market
segments or environments
Security
startup
goals
aren’t that
different
9. Why do security startups exist?
9
Security is always a secondary or
enabling layer
10. Understanding the startup cycle
10
Idea
Founded
Seed
Funding
GA/MVP
Growth &
funding
Exit
Founders
leave
Acquisition?
Acquisition?
Acquisition?
Founders
leave?
12. How do I find a startup?
Security
startup
pool
InfoSec
Mgrs
Industry
Analysts
Cons
VCs
Forums
Email,
LinkedIn,
Cold Calls
Partners
Sales
Pres,
Demos
14. What does ‘due diligence’ mean to you?
14
That’s where I send the vendor a checklist with items like ISO
27000, SSAE 16, HIPAA and PCI on it, right?
List of references
Financial stability
Company history
Compliance
Customer Complaint history
Insurance
Audit results (SSAE 16, ISO
27001, PCI)
Contracts
Breach/IR plans
15. What does ‘due diligence’ mean to you?
15
Does the product work?
Can vendor claims be validated?
How could efficacy be measured and compared
to other options?
16. How do you validate a security product actually
works?
16
17. A startup-specific due diligence process
17
1 Get the big
picture
• Find gaps
• Determine
greatest needs
2 Create
requirements
• Based on needs
and resources
• Budget
• Staff
• Skills
3 Vendor
research
• Find targets
• Research
targets
4 Initiate
Relationship
• Start
conversation
• Test product
5 Make/Break
• Does it make
sense?
• Feedback loop
• Formal
relationship
6 Manage
relationship
• Product/vendor
monitoring
• Product
development
feedback loop
Search cycle Dating cycle
Not quite ready…
Try again!
19. The process
19
Research the startup (“Passive Recon”)
Engage the startup
Ensure a good product/environment fit (avoid Shelfware!)
This is a startup: the roadmap IS the product
Proper preparation makes the most of your PoC
Contracts, agreements, liability – rubber, meet road
Uh-oh, they got acquired!
20. When you engage…
20
Don’t shy from questions*: “We’re 62
minutes into this sales presentation and I
don’t know what your product is.”
“Plan to dump before you jump” (i.e.
Have an exit plan before you start)
You are a valuable asset to a
startup; this gives you leverage
Use this leverage!
* - real story
21. Ensure a good product/environment fit
What is shelfware?
Why does it occur?
What ends up on the shelf?*
21
* See handout
Top five reasons products become
shelfware according to buyers:
1. Compliance-driven purchase
2. Internal Politics (tied for #1)
3. Lack of staffing/headcount
4. Lack of time/expertise
5. Features overpromised or
missing
22. Roadmap fit
22
Be clear: what are you willing to wait on versus need now?
Integration path – just APIs or deeper partnership?
Platform-based architecture?
What are the long-term goals?
Are they feasible/reasonable?
Better Best Unicorn
Unimaginable wonders to
behold
The average roadmap
23. The value of security products
Can you calculate the value
you should get from it?
What’s the Time-to-
Implementation?
What’s the Time-to-Value?
What’s the True Cost?
23
Drawing and concept by Henrik Kniberg http://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp
24. Example: the value of threat intelligence
24
…box of rocks
threat
intelligence!
$10k
26. Advice from the trenches
26
Q: What are some challenges to watch out
for?
A: Overly vague descriptions of their IP. Not
being multi-platform ("oh, we'll support Macs
in our *next* major release!").
“…figure out how to short circuit the
purchasing system… the startup needs your
money more than you do...” –Richard Stiennen
30. Lessons learned
30
Why did this happen?
Small company
Three engineers
No Security expertise
No third-party security audit
Conclusions
Due diligence of technical products requires
technical assessments
Ask if a third-party audit has been performed
Consider impact and liability to other
customers before taking assessment too far
Keep pressure on the vendor to fix the issue,
even if you decide not to buy the product
31. Recommendations: brace for impact
31
Not comfortable? Don’t do it, or do it through a trusted partner
Don’t have the spare staff/skills/cycles? Don’t do it.
Plan to lose most of one FTE’s productivity to testing,
implementation and bug reporting activities, at least initially.
Look for products with a high potential reward/effort ratio -
threat prevention technologies, for example.
Check workflow integration before purchasing!
33. Apply what you have learned
33
Later today you should:
Check out Sounil Yu’s Cyber Defense Matrix Follow-On talk at 4:30pm in West 2016
This week you should:
Take the vendor marketing challenge in the expo: don’t be afraid to ask questions
Within three months of this conference:
Go through the first half (steps 1-3) of the due diligence cycle for at least one product
Have a few trusted sources for gathering information/recommendations on startups
Within six months:
Go through the second half of the due diligence cycle (steps 4-6)
Refine your due diligence process and share your results with others if comfortable
34. Thank you!
34
Please, continue the conversation, chat or ask questions:
Twitter: @sawaba
Adrian.Sanabria@451Research.com
Adrian.Sanabria@gmail.com
Spiceworks (sawaba)
Peerlyst
Hinweis der Redaktion
There was a lot more information I wanted get into this talk, but there just isn’t time to show it all to you. For that reason, I’ve created a handout and some other additional materials for you. Feel free to look through it during this presentation, or just go ahead and
The process of buying security products for the enterprise environment is broken.
Throw startups into the mix with that last statement, and you’re in for a ride if you aren’t prepared.
Traditional due diligence doesn’t work for security products, and often, it just doesn’t happen, period.
Mature security products haven’t kept up
Products from startups are an unknown risk
Rock and a hard place?
Two VERY important concepts with startups:
Startups are effectively a reverse lottery – we don’t actually know if their product has value until AFTER they’ve received money for it.
It is possible for a security startup to be a complete success in the BUSINESS sense, while being a complete FAILURE in terms of delivering value to the customer. It is inherently very, very difficult to prove or demonstrate efficacy or value with security products. THIS is why I say that the process of buying them is broken.
Emphasize the problem statement and goals of this talk:
Problem statement: Startups may get pitched as the solution to all our problems, but there’s a lot I’d like to share with you before you jump into these waters.
At the risk of spoiling my talk, I’m going to tell you up front that effective due diligence is going to be 90% of what you need to avoid getting burned when working with a startup. The other 10% is understanding startups and applying the former 90% to the latter 10.
Goals/Agenda
The title mentions CISOs, but this talk should be useful for anyone involved in the process of finding security products that are a good fit for their company.
Some tips on how to evaluate both startups and their products; I’ve also got some information from CISOs and startup founders to share.
Address the duality of cutting edge startup technology that may or may not be ready or suitable for your business’s production environment.
Is the grass always greener? Are we really getting any better with each startup iteration?
Or I could add a diagram showing how security is a layer on top of everything else
In order to have network security, we need to have a network.
In order to have endpoint security, endpoints first need to exist.
Ditto, cloud, mobile, etc.
It is important to understand this cycle prior to working with startups. We’ll reference this more later on.
Startups can stay in stealth mode for years! This is often when many startups begin to look for companies to be beta testers or early adopters/partners
The growth/funding cycles usually represent most of a startup’s life
Spend a little time talking about how startup relationships are different from normal vendor relationships:
though due diligence is pretty much the same, regardless of security product or vendor, the startup relationship is very different.
Point out that VCs/investors are often on the startup’s board, so they’re a good PoC to start the relationship from
You are not merely their customer - you are their beta tester, their partner, their guide, potentially their advocate. Don’t just ditch them if you don’t like things about the product, WORK with them. Be honest. Be prepared to spend a lot of time and effort giving feedback. Without brutally honest feedback and thorough reviews of issues and problems, the startup can’t get better – YOU have the perspective they lack – the perspectie they need.
If you can’t do these things, don’t go after startup relationships. By nature, they require more patience and support. Don’t expect mature version 8 products.
We’ve seen several acquisitions before some startups even come out of stealth. We don’t even know what we missed!
It is also important to understand the cycle represented here is a very loose representation of the path the average startup takes, and can be disrupted at any stage by a number of events.
We recently saw one startup get successfully sued for every dime of revenue it had ever made!
Can we tell what all thee products do? Not really – this is all hyperbole. The problem with a lot of this is that the marketing tends to intentionally divert your eyes away from the gaps each product has, making your job that much harder.
I will say, some of these vendors are very up front, and while their marketing might look like everyone else on the surface, when I do a briefing with them, they’ll even have a slide titled “what we don’t do” or “our challenges”. THAT’s a good sign. Any vendor that will point out their own products’ weak points is a vendor making your job easier. The less work you have to do to cut through the marketing, the more time you have to get the job done.
CHALLENGE: Visit the expo and find a company you aren’t familiar with. See how long it takes you to figure out what this company and their products do. If you have someone with you, make a game out of it, even – time it. I’d love to hear your experiences and results – this is what I do as part of my job almost every day! <-- Tweet this.
How do I find a good startup? They’re almost certainly already spamming you to death in your inbox, at conferences, on websites, in airports… How do I cut through marketing and hype? I can’t figure out what the hell they do or whom they’re actually competing with.
Put together a trustworthy network of other CISOs, VCs and industry analysts. You need informants!
Places to discover startups and discuss experiences: PeerLyst, Spiceworks forums, conferences
Put emphasis on sources that have experience with either the startup’s people or products. You need a good relationship with both.
VCs and Analysts are the most likely to know the founders, their vision and will likely have an opinion on whether they’d recommend doing business with them
Other InfoSec managers and forums are good places to find people familiar with using the product
Instead of considering it a bonus when new products can integrate with your other core security products, why not make integration a priority?
Open a dialogue with existing vendors you’re happy with. Ask them who their technology and integration partners are, and start your search there.
Most vendors will go through their own due diligence and vetting process before entering into a partnership agreement with a startup. It’s not a guarantee, but it is another layer of assurance that costs you nothing and you wouldn’t have otherwise.
Webinar-style sales presentations and Demos – forget it, at least as an introduction to the startup. If you do this, leave it for later, after you already have a good feeling about the company, a recommendation, etc. It is just too much time to spend on something for far too little value.
It generally takes me about 5 minutes on a company’s website to figure out:
what a product does
what it’s core features and capabilities are
Other products it can integrate with
Whether or not it roughly fits what I’m looking for.
Often, you can even find short videos or pre-recorded content to help explain how the product works
So… why the hell would I waste an hour watching a webinar or demo to get the same amount of information?
First, they’re going to spend at least 10-15 minutes explaining to you why you need the product.
We call this the “problem statement”.
You already understand the problem – that’s why you’re actively searching for a product in the first place.
Tell them you understand the problem already, and they can skip that part.
Better yet, if it is a scheduled meeting, set the agenda yourself and list your expectations for the meeting.
Ensure that there is someone technical on both sides
to understand any architectural or compatibility showstoppers
answer technical questions
spot (from your side) egregious BS
Personally, I’m thrilled when I can try out the product myself without having to talk to anyone in sales, or even a video of someone using the product can give you a good feel for what it can do without having to make a time commitment and blocking off time on your schedule.
Finally, there are just too many vendors out there to allow each of them to take an hour of your time individually. You also can’t PoC everything. It just isn’t feasible for anyone except dedicated labs.
Awww, patience? Really?
Ask the crowd how many already have some sort of formal due diligence process they use?
Chris Tarnovsky’s idea of due diligence might involve dipping a TPM chip into a hydrochloric acid bath and polishing away layers of ceramic and silicon so he can explore microprocessor secrets with an electron microscope.
That’s one extreme.
The other is almost entirely business-oriented.
Yeah, you should probably do all the business stuff. In fact, someone has probably already automated that process for you. HOWEVER...
THIS is what bothers me about due diligence. I’ve run into situations where security products actually lowered our security posture and had negative value for at least the first year.
The process of buying security products for the enterprise environment is broken.
Throw startups into the mix with that last statement, and you’re in for a ride if you aren’t prepared.
Traditional due diligence doesn’t work for security products, and often, it just doesn’t happen, period.
Mature security products haven’t kept up
Products from startups are an unknown risk
Rock and a hard place?
Due diligence is a process, rather than a cycle, but it contains several cycles.
The big picture – what are you missing?
Requirements – you know what you’re missing, but what do you have the staff, budget and skills to take on?
Vendor research – find startups that meet your needs and begin researching them.
Approach the vendor to begin the relationship – begin testing the product
Make/Break – does it work or not?
If yes, go to #6
If no, go back to #4
Manage relationship
maintain relevance
keep product up-to-date and running smoothly
Provide feedback for the startup on problems, successes and requests
What do we need most urgently?
What are our requirements?
What resources do we have to spare?
Architectural/deployment options?
Where do we need it to work?
How well does it need to scale?
What are the architectural/deployment options
Do I have a hard requirement for the technology to live on premises? Many do for a variety of reasons.
Do I prefer SaaS (not having to devote hardware, rack space or fire up a VM on-site)?
Do I prefer a managed offering (this is a labor-intensive product, and I know I can’t run it properly myself – we need help)?
Are there hybrid options? What would that look like?
What’s the environment coverage/efficacy? Where does it work?
Does the product work for my employees and assets that leave the corporate network? Examples:
There’s a few “agentless” endpoint security products out there – how does that protect my laptop while I’m here at RSAC?
Perimeter network security products (use my diagram?) usually does nothing for mobile devices, assets that are off-prem or assets that regularly travel on and off prem
The vast majority of security budgets are concentrated here!
Does the product cover all my endpoints?
Or just 0.1% of them (my crown jewels???)
AND only when they’re on the corporate network?
“How much of your security budget is still protecting endpoints when they are at employee’s homes? Coffee shops? Hotels? Airports?
Can I deploy it to servers?
What about auto-scale environments? What does creating and destroying 8000 servers every day do to the reporting and management console?
Can it scale?
What’s the asset coverage like?
Ask for a list of product features and capabilities
Ask to see the most recent roadmap
Who is going to be responsible for the product?
Who will work with support?
Who will patch it and upgrade it?
Who will go to training and become the SME?
Does it make sense to have an SME, or is a SPOF a bad idea?
You want to know as much as possible about them and their product before you ever talk to them. It will save you a TON of time, honestly.
Working with a startup is more like a partnership than typical customer/vendor relationship.
Don’t put all your eggs in this basket!
Always have a plan to replace this vendor, should something happen
The more critical this vendor is to your company’s security, the more important this plan becomes.
If you can’t easily replace this vendor, consider other options. Don’t get into a vendor relationship lightly if you know it will be tough to extract or replace their product.
You can potentially help them as much as they can help you!
You can spread recommendations through word of mouth
You can lend credibility to their product claims
You can help them land other customers, additional investors, potential acquirers...
You have leverage
offering to talk to other potential customers
allowing them to publicly state you as a customer (Be willing to trade public endorsement for promise to build for your needs. Startups need customer wins they can share publicly. Most companies don't like to publicly share what they're using internally, like that somehow makes them safer.)
allowing them to do a case study
Use this leverage!
Don’t settle for a product that still requires you to do significant amounts of custom development to get it working properly
Plan to spend considerable time reporting bugs, errors, issues - worst-case, be willing to devote at least one person full-time to this product.
That said, define your limits before you begin and be willing to cut your losses if you need to.
Do you really need it?
Platform or ‘feature as a product’?
Workflow fit
Staff
Managed options
Integration, APIs
Do you really need this product? How would it improve your company’s security posture? Be careful not to get sucked into “cool new toy” syndrome
Is it really a standalone product or something that will soon become a feature of a larger product?
Does it fit into your workflow? Does it fit with how you and your team run security in the organization, or does it require significant new training, new consoles, new operational responsibilities, a new source of logs...
Email is not an API! Does this product integrate with your existing IT and security products?
Are they planning on an MSSP offering of the product, or partnering with an MSSP?
Staff - Is your staff skilled enough to use the product effectively? Do you have enough staff to use the product effectively?
It doesn’t matter if their product is the greatest thing in the world if you can’t effectively use it. As boring as it sounds, things like active directory and ticketing system integration can be as critical as whether or not the product can actually make good on its claims
Ask to see their roadmap. What’s the long-term vision for the company? Does it jive with what your plans are?
Is the product architecture designed as an extensible/modular platform?
Are they integrating with other vendors now, or do they plan on formal product/technology integrations? Do they integrate with your current vendors and products you already use? Are they willing to?
Do they have an open API?
Fun Scenario/Analogy - the point here is to think about what the product does, and determine whether or not it even provides any value to you. It is shocking the number of companies that don’t consider this.
You own a shop, and someone recently threw a rock through the window.
To prevent this from happening again, you replaced the previous glass with shatterproof glass. (prevention technology)
A salesman comes into your shop the next day and says, “hey, I saw you had a rock thrown through your window the other day. I’ve got a great deal for you.”
He tries to sell you a “threat notification” service. “What’s that?” you say.
The salesman lays down a photo (box of rocks)
“That’s a box of rocks”, you say.
“No it isn’t!” the salesman protests - “this is threat intelligence! Someone could throw these at your nice, new windows! You need to know about them!”
“Okay”, you say, “how much are you charging for this… service?”
“I’ll send you these pictures every day, and you pay me $10k a year for this service.”
You point out that the shatterproof glass you installed isn’t perfect, but it should prevent against most rocks that the average person could even lift and throw.
Ditto for SIEM. Is it working? I don’t know, you have to do a bunch of custom development work to even figure out if the product is working.
Photo of construction supplies
“Here’s your house!"
$1.5m
Per year
And you’ll need to hire some people to build it
If you want a garage, that will be extra
No shingles, eh? The roof is extra.
Back to reality from analogy: SIEM is a development platform, not a product. Learn to recognize the difference!
You need skilled people to build it and put everything together
Support for logs from unsupported products or custom apps: build it yourself
Threat intel: “we support STIX and TAXII” <— in other words, go buy it separately yourself. Good luck integrating it.
Triggers/Automation: build it yourself
Also, you’ll randomly notice you’re not getting logs from 5% of your log sources, and you’ll have to chase down those issues. This will happen once a day.
Is due diligence different from validation?
I would argue NO. It isn’t. After all, if the product doesn’t do what it is supposed to, what good is the rest of your due diligence process?
This advice is specific to products with threat detection/prevention capabilities, but I felt it was important data to share, given that more than half the security industry is threat detection/prevention-focused.
Is due diligence different from validation?
I would argue NO. It isn’t. After all, if the product doesn’t do what it is supposed to, what good is the rest of your due diligence process?
This advice is specific to products with threat detection/prevention capabilities, but I felt it was important data to share, given that more than half the security industry is threat detection/prevention-focused.
So, a ways back, I joined an enterprise for a short time to help them get their security program going.
The CTO there brought me in, and I worked directly for him.
One day, he comes to me and says, ”We’re considering putting these cloud-managed wireless access points in all the manufacturing centers. Could you take a look at them and make sure they’re secure?”
There’s nothing more exciting as a security geek than when someone hands you something brand new and says, “I need you to try to break this”.
The hardware was pretty off the shelf, nothing remarkable. What was supposed to be really cool about these devices is that they were cloud-managed, like Cisco Meraki stuff, but at a fraction of the price.
I had experience with cloud-managed products before. In a previous life, we had some Qualys scanner appliances, which were the first cloud-managed devices I could remember. You just logged into Qualys.com, and controlled these scanners from there. You never had to worry about having poke holes in the firewall, set up VPNs or enable any sort of inbound connections. Qualys accomplished this by having the scanner reach out to Qualys.com every 60 seconds or so to check for new instructions. So, while it could take as long as a minute for your commands to make it to the scanner appliance, it was still a pretty slick and impressive setup back in 2004.
Well, these guys were trying to do the same thing. I must admit, I never dug into the details of how Qualys did it back in the day, but being 2012, I figured the process had been refined over the last 8 years. Meraki was well known for it, and I started this testing about a month before Cisco picked them up for $1.2bn. There was nothing refined about how these products were designed.
Due diligence is tough to define for any given scenario. Simply put, it is the result of the time, money and skill you feel like you have available to make sure you’re not making a big mistake. In this case, I decided if there was nothing else I had time for, I wanted to understand how these devices talked to the cloud management portal. I prioritized that task first.
While myself and another colleague ended up spending many hours validating this vulnerability, we knew within the first few hours that this product wasn’t secure, and wasn’t going to get our approval.
Due diligence time. Our first question: how does this vendor’s “cloud managed” architecture work?
Let’s see
Start a packet capture and turn it on
Check out the packet capture
Uh oh. Entire config pulled over HTTP. Not HTTPS. And then gets sent back via syslog
I wonder, how does the server know to give it this config? How does it authenticate?
All I see is a mac address. It couldn’t be that simple… Yep, it is.
If these were all manufactured in the same run... <wrote some python>
Ran the python – yes, we got the config for every other device by sequentially guessing the MAC.
Most configs included the name of the customer and the physical ADDRESS where these devices were installed.
A full year later, it was still serving up config files for whomever tossed a MAC address at it.
One year to get it fixed. We never bought it. I was even working for a different company when they finally fully put these vulnerabilities to bed.
Benefit: We didn’t spend money on OR deploy devices that had critically flawed security
Drawback: We got sucked down a year-long rabbit hole. Though we didn’t have to, we felt morally obligated and even wrote code to determine the full extent of the vulnerabilities and to ensure the vendor got them fixed. Also, it was really, really exciting.
Preventing shelfware! We hate shelfware, right? How many of you have shelfware? (ask the audience – explain shelfware if they appear confused)
Please, please, please don’t just go buy stuff!
One of my peer reviewers suggested I check out his slide deck.
His Cyber Defense Matrix concept is a great tool for implementing several of my suggestions in the due diligence process.
His 8th use case here is great for gap analysis exercises and visualizing the ‘big picture’ in your organization.