Introduction – BioHi, thank you, thanks Pike, my name is Doug and I’mHere to talk about how bronto uses splunk.A little bit about myself:Manager of Systems Engineering for 5 yearsMy team is responsible for all the technology for Bronto.So what does Bronto Software do?
Bronto’s a marketing platform which encompasses email, SMS, and social media , we have a lot of clients including PartyCity, Armani Exchange, and EtsyWe send 10’s of millions of time dependent emails per dayWhich leads us to process 100’s of millions of tracking events per day.We use splunk to index 100’s of millions of events across our infrastructure per day.
Our challenge was that we were trying to deal with a massive collection of disparate logs scattered throughout our infrastructure.We had rolled some in-house tools but could quickly tell this development effort would not be sustainable long term and wouldn’t scale. We recognized and agreed that something had to be done to manage all this dataWe discussed the trade offs of build vs buy?
Ok, so one option was to build:We looked at building our own data indexing platform using hadoop, hbase and flume to name a few. Also, we would be leaning heavily on engineering and development resources that were currently tasked with building new client facing featuresRetasking them to create internal tools We would have also incured significant cost with building our own. Build doesn’t mean cheapTriple annual license cost plus 6 months from day to hire start to implement less than 50% of functionality of SplunkClick SlideSo we looked into buying Splunk.Splunk would be more cost and time effective for engineering and development resources.Much better Speed to valueWith other products there was either signifigant challenges or was impossible to integrate into our opensource infrastructure.Splunk was very easy to setup and even dropped right into our existing puppet deploy framework.This is not a very common event for bronto, to find a 3rd party product that just drops right in.
The adoption of splunk was very easy within different groups at bronto, both inside and outside of engineering.Partly because of splunk’s ease of use, but also because the need to consume the information was already so great.The startup time with Splunk was very easy. The initial learning curve for new users was very short.
Splunk’s versatility lends itself well to solve multiple problems at Bronto.I’m sure a lot of you are using splunk to gain easy access to system level information across your environment both from logs and secondary events. And this has proven to be a huge win for us too.But specific to bronto, we use splunk to capture and correlate data for more bronto specific events like email metrics And helping us to understand and predict email deliverability trends, both good and bd.And one of the key features of splunk that’s been a huge win for us, is it’s enabled us to get people that are not necessarily technical or within the engineering organization empowered to help themselves.What I mean by that is that before splunk, if someone wanted to know what the reason code for a group of email bounces was, someone on my team had to dig through logs, or write a script to parse the logs for this specific query. This didn’t scale well at all.But by giving people access to splunk, it’s freed up a huge amount of time for my team, that we can then devote to other needs.And we use splunks alerting to let us know when things may not be going to well for deliverability, both customer caused an ESP caused. So it really helps us help our customers.
Dashboards have really proven to be helpful, specifically to give a quick and constant representation of how both deliverability successes and slowdowns are happening.This is one of the most important dashboards we use on a daily basis, that really wasn’t possible before splunk. It allows us to distill the deliverability data down into something quickly and easily digestible.Which helps us spot trends earlier.
Deliveries don’t always go according to the customer’s plan and this this graph helps us understand, very quickly what the reason is.It could be that a certain email recipient domain has decided today was the day to make a change to their acceptance rules.And again, being able to take multiple streams of analytical reporting and then concisely represent it is a great win for us and for our customers, to help us quickly respond to problems.And see how successful we are.
I’ve talked a lot so far about how Bronto uses splunk, and mostly about how people out side of Systems Engineering leverage splunk.But we use splunk within Engineering a lot too. And I’ve always been a sys-admin at heart.Splunk allows us to spend time doing the things we want, and need to be doing.Splunk looks at the logs for us and lets us know when there is a problem. We tried to do this with logster and nagios, but with the sheer volume of logs, it didn’t scale.And that’s on one server, let alone hundreds that need to correlate trends.Splunk is also a key component of our security infrastructure, by monitoring logins and security events and letting us know if someone trying to brute force a login for instance.And we can’t forget the Ad-hoc searching, which would have been almost impossible with the build option with hadoop. Being able to search across the infrastructure and see the logs in real time has been a huge hit.
I just want to get more people using splunk, there is still more that can be gleaned, more stones can be turned over.Overall we’re really happy with splunk, but we’re always looking to improve and grow with splunk.
So that’s the overview of how Bronto uses splunk.Thank you for having me, and if you have any questions I’ll be around for the rest of the day.
