SplunkLive! Utrecht 2016 - Exact
•Als PPTX, PDF herunterladen•
0 gefällt mir•840 views
DevOps - Lower meantime to resolution while facilitating growth.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie SplunkLive! Utrecht 2016 - Exact
Ähnlich wie SplunkLive! Utrecht 2016 - Exact (20)
Mehr von Splunk
Mehr von Splunk (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
SplunkLive! Utrecht 2016 - Exact
- 1. Copyright © 2015 Splunk Inc. DevOps - Lower meantime to resolution while facilitating growth André van de Graaf Principal, Quality Assurance Exact Software
- 2. © 2015 EXACT › 2 350,000 companies 7 countries LAUNCHED Exact Online, a SaaS-based version of the product We build business software aimed at SMBs 2005 Dutch based company Founded in 1984 by Dutch students 5 Datacenters
- 3. © 2015 EXACT › 3 Exact Infrastructure & Operations Team of 7 engineers running a platform supporting 350,000 COMPANIES Splunk team 0.5 FTE Setup and configuration 0.5 FTE Data import, dashboards, alerts, reports.
- 6. © 2015 EXACT6 Exact’s ambition: Exponential growth For exponential growth we need to automate. Splunk will facilitate this. Last quarter: 250 new companies added on a daily basis
- 7. © 2015 EXACT Situation before Splunk? 7 • Support department was our alert system • 2 datacenters • 4 countries • Weekly builds • Manually analyzing logs • At least 1 war room session per month
- 8. 8 How Did We Do? Splunk implementation step by step in 12 months
- 9. © 2015 EXACT9 Splunk Implementation • Operation Visibility • Business Insight • Pro Active Monitoring • Search and Investigation
- 10. © 2015 EXACT Operational Visibility: +/- 20 different datasources IIS log Eol log Perfmon counters
- 11. © 2015 EXACT Operational Visibility: Status Monitor dashboard
- 12. © 2015 EXACT Oeps 1 datacenter down!
- 13. © 2015 EXACT Drill down to Dashboard Current status
- 14. © 2015 EXACT Business insight: Trends and patterns
- 15. © 2015 EXACT Pro-active monitoring: Fair-use policy - Exceptional Uploads and Downloads
- 16. © 2015 EXACT Search and investigate: Detailed perfmon counters • We log performance counters every 5 seconds. So we are able to investigate issue to specific moments when did it start and when did it end. Logging per minute is not detailed enough for us. - TITLE OF PRESENTATION16 • For statistics, we aggregate performance counter per hour.
- 17. © 2015 EXACT Search + Investigation Bugreport: Splunk queries to see how many customers are affected by a bug accros all countries. This will help development teams to priorities the bug intake. After deployement we can use the same splunk query to see if it is really fixed.
- 18. © 2015 EXACT Alerts to Mail and VictorOps
- 19. © 2015 EXACT Where are we as of today? 19 • From 2 to 5 datacenters • From 4 to 7 countries • From Weekly to Daily builds • Adding 250 companies on a daily basis • Data size increased with 100 % in 1 year
- 20. © 2015 EXACT Where are we as of today? 20 • Fully operational in DevOps team • Lowered meantime to resolve with 75% • Inform support department pro-active • Scale the platform not the team • Splunk is part of the delivery process of new Exact Online functionality.
- 21. © 2015 EXACT21 What Did We Learn? • Start documentation from the beginning • Rubbish in = rubbish out. Fix the source! • Implement 1 naming convention which applies to all datacenters. • 1 naming convention within Splunk: Reports, lookups etc...
- 22. © 2015 EXACT22 What Did We Learn? • Do not hard code stuff. • End result of an incident is a new alert and or dashboard. Continuous improvements. Part of RCA process. • Analyze your imported sources. Is everything useful to import.
- 23. © 2015 EXACT23 Obstacles to overcome • No ADFS support. Supported as of Splunk 6.4 • Resources, small team
- 24. © 2015 EXACT24 Next Steps • Roll out Splunk to our Development and Support teams. • Upgrade to Splunk 6.5 • Implementation of Splunk IT Service Intelligence (ITSI) • POC Machine Learning
- 25. © 2015 EXACT Thank You ANDRE@EXACT.COM GRAA1005
Hinweis der Redaktion
- Max wants to get to the top. He has a great ambition to get on the top. This is also what Exact wants to achieve. Be part of the Top 3 Globally for SMB’s Performance and Reliability is what we do as Infrastructure and Operations team.
- Max wants to get to the top. He has a great ambition to get on the top. This is also what Exact wants to achieve. Be part of the Top 3 Globally for SMB’s Performance and Reliability is what we do as Infrastructure and Operations team.
- Started with 1 datacenter. Source by source (For all countries) Reimplementation because of DUAL DC switch. As of today adding datasource. For instance Akamai logs.
- More details about the current state of Exact Online. During an incidents this dashboard gives a better overview if the incident is software related or infra structure related. Is it 1 or multiple servers. Is it 1 database or multiple database. 1 country or multiple countries.
- In case customer is doing exceptional uploads and or downloads we send him a detailed reports of his up and download behavior. This will help the customers to find which app, company, IP address and username who is causing this up and or download behaviour. Most customers are not aware of what traffic they generate. With this report they are able to fix this.
- We log almost all perfmon counters e very 5 seconds. So we are able to investigate issue to specific moments when did it start and when did it end. Logging per minute is not detailed enough for us.
- In case we go GA for new functionality which requires changes in infrastructure, Splunk monitoring and alerting should be implemented
- Do not hard code stuff: Automatic detect when a web server is added. 1 naming convention: Analyze imported datasources. For instance Security log of AD in our case a lot of not useful entries.
- Do not hard code stuff: Automatic detect when a web server is added. 1 naming convention: Analyze imported datasources. For instance Security log of AD in our case a lot of not useful entries.