At Ustream the teams developing the streaming technology stack are also responsible for operating it. This means we have our monitoring and alerting in place (including those based on error logs I mentioned above, but many others too) which alert the engineers themselves. I would like to talk about how we made this transition from the traditional setup where the devs did the coding and the sysops did the operation - what lessons we learned, how we convinced the sysop guys to give us permissions and so on
2. About the
presenter Age 36, Married, father of
two
Geek since ~30 years
Ustreamer for 5 years
Dev, Ops, Management
3. at a
glance
Live streaming
Free broadcastingSoftware as a Service
Big events
• Sony Playstation
• Nintendo
• Lady Gaga
• Festivals
Citizen journalism
• Rescue of Chilean miners
• Earthquake in Japan
• Revolutions in Egypt, Syria
• Protests in Ukraine
• Obama campaign
• Climate Reality
• Football
• Justin Bieber
Concurrent viewership peaks at almost 2 million
4. Ustream timeline
2007: founded
2010-11: first big expansion
2013: shift focus to Pro Broadcasting
2014: SaaS product line launched
2015: current engineering setup: 12 teams, ~100
people
5. Agenda
What is DevOps?
Why DevOps?
Implementing
Organizational changes
Takeaways
6. Origins Agile 2008: “Agile
infrastructure”
DevOps Days, from 2009
Belgium
Cloud, scaling, automation
command line is not
enough
Netflix, Etsy, Github
O’Reilly Velocity
8. Devs Ops
They build it
Development is single
responsibility
No access to production
systems
“It worked on my machine”
They deploy and run it
Control and operate all
prod servers
Restrict access
On duty rotation
“You mess it up and I have
to wake up when it fails”
12. What is DevOps?
The DevOps movement emphasizes communication,
collaboration and integration between software
developers and IT operations (New Relic)
The set of cultural norms and technical practices that
enable organizations to have a fast flow of work from
development through test and deployment, while
preserving world-class reliability, availability, and
security. (Gene Kim)
13. What is DevOps? Agile Manifesto
Continuous Integration
Continuous Deployment
Cross functionality
Shared responsibilities,
trust
Everyone on Duty
Metrics and Monitoring
Toolchain
14. Implementing Redefine roles
“Old guard” problem
Knowledge sharing
Permissions and
responsibility
Audit logs
Ops reserved areas
15. Metrics Dashboards
Graphite
Dashing
Code metrics
Static analysis
Test coverage
Real user metrics
16.
17. Automate all the
things
Build
Jenkins
Gradle
Deployment
Deb packages
Immutable images
Automatic scaling
19. OpenDuty Open source alerting tool
Supports different
notification methods
Incident handling
Personal configuration
Escalation policies
PagerDuty compatible API
Mobile Web View
20. DevOps Culture Ownership
Autonomy
Alignment
Challenge
Culture as a Weapon
21. Takeaways We all row the same boat
Ownership leads to
responsibility
Automate all the things
Metrics, Monitoring,
Alerting
Tools
Where does devops come from:
Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure"
Typical org: many devs, few ops, wall and conflict
"you shouldn't mess with production systems because I am responsible for it”
Check it in, deployment is ops
Enterprise solution, formal
Great spof
4.1. sysop dev friendship
Redefine roles: mindenki azért felel amit épít, beleértve az üzemeltetést. Ops új szerepei: platform, tooling, tudásátadás
Régi emberek ellenállása
Sysop nem akar jogosultságot adni, hogyan lehet meggyőzni őket (tehermentesítés, átvesszük az ügyeletet, etc)
Ops reserved: user létrehozás, employee termination, firewall, network
Cyclomatic complexity
Mindenki ügyel, területek szerint külön rotációk, escalation policy