Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie How to run a global, cloud scale event for 10,000 people
Ähnlich wie How to run a global, cloud scale event for 10,000 people (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
How to run a global, cloud scale event for 10,000 people
- 1. How to run a global, cloud scale event for 10.000 people ROB BOS - @ROBBOS81 MICHIEL VAN OUDHEUSDEN - @MIVANO
- 3. 35 countries 10.000 participants 1340 web apps 4500 YouTube views 1530 resource groups 7 Azure DevOps environments 4 Azure subscriptions 3 outages in Azure DevOps Half of the budget Dedicated Microsoft SRE’s on call 36 hours 4 million impressions Free community event
- 4. WHAT’S IN IT FOR YOU? This does not only work for community events Know your end users Working together in a globally distributed team Technical design decisions Design with the end in mind
- 5. WHO ARE WE Rob Bos Michiel van Oudheusden @robbos81 @mivano
- 7. WHAT IS GDBC? • Global DevOps BootCamp • Community event • Day of learning • Free Event out of the box
- 8. EVENT OUT OF THE BOX FOR LOCAL ORGANIZERS • Provide location • WIFI • Host the local event • Local speaker • Provide enough proctors • Engage local community
- 9. EVENT OUT OF THE BOX FOR US • Global keynote • Local keynote • Exercises for attendees • Infrastructure • Azure • Azure DevOps • Communication • Marketing & Branding 2019: Niall Murphy Global Head of Azure SRE
- 10. GDBC CORE TEAM • Team to create + run the global event • Volunteers from the community • Planning / meetings • Sponsoring
- 12. SEARCH FOR VENUE ORGANIZERS Get local venues: • Google Forms • MVP Summit • Radio TFS Marketing: • Self-promotion • Local communities
- 13. COMMUNICATION IS KEY Build a community of local venues and GDBC Core Team • Slack • Community calls • Record and reshare • Centralized wiki
- 15. WHAT DO THE VENUES NEED? Attendees Content
- 16. ATTENDEES Platform to handle: • Attendee registration • Local landing page • Social marketing Eventbrite • Platform as a service • Venue organizers
- 17. ATTENDEES Eventbrite • API is not complete • Co-admins are hard • Not everything can be automated Automate as much as possible • ConsoleApp with Selenium to add co-admins • Auto-invite to Slack
- 18. SAAS OVER PAAS OVER IAAS
- 20. PREPARING THE CONTENT Keynotes Exercises Workspace Video content Styling Guide
- 21. EXERCISES
- 22. WORKSPACE
- 23. CHALLENGES! • SSL certificate expired • Flaky connection • Credential leak • Exception rate goes up • DDOS after CEO message • Supply chain attack
- 24. DDOS AFTER CEO MESSAGE
- 25. COMMUNITY • What is GDBC • Webshop • Challenges
- 26. SETTING UP AZURE
- 28. WHAT DO WE NEED Resource group App Service SQL Server Database Application Insights Azure Active Directory • Venue admin account • Per team: • User account + user group • Service principal for Azure DevOps
- 30. CONSIDERATIONS Cost • Azure Passes Sponsored subscriptions • Just in time Regionally divided • Spread the load • Latency for users
- 31. TOOLING Central data store for config Make it idempotent Started with Azure CLI Switched to ConsoleApp.exe Scalable, Repeatable process Azure DevOps
- 32. AZURE DEVOPS TO THE RESCUE
- 33. LEARNINGS SQL Server limits • 20 per region per subscription • 200 max. per subscription AAD role assignment limits • 2000 role assignments per subscription Cloud is not infinite • SQL Servers India: Max capacity reached • Application Insights Central US • Resource types off by default
- 34. COST MANAGEMENT
- 35. AZURE HAS BEEN TACKLED
- 36. AZURE DEVOPS
- 37. REQUIREMENTS FOR AZURE DEVOPS Organization Team project Git repository Build pipeline (CI) Deployment pipeline (CD) Service connection to Azure Artifact feed Azure Active Directory Link X 1200
- 38. AZURE DEVOPS PROVISIONING Multiple organizations to spread the load: Use a service account for setup • Australia • Brazil • Canada • East Asia • West Europe • India • United Kingdom • United States
- 39. BUILD AGENTS 7 organizations 1200 teams ±200 teams per org 1-10 concurrent pipelines 150 sponsored pipelines
- 42. Every day in preparation Azure Infrastructure Azure DevOps Certificates
- 45. LEARNINGS AZURE DEVOPS PROVISIONING YAML pipelines in repos
- 46. LEARNINGS AZURE DEVOPS PROVISIONING Hitting the service at scale can trigger some weird issues • Build pipeline outage • Two regions • Quick fix: 1 concurrent pipeline • Australia networking outage • Agent scale set network issue
- 47. AZURE DEVOPS – PRODUCT TEAM Firewall rules - Brownouts • June 12th IP-addresses will change • June 15th Event Day • Moved to the 17th
- 48. AZURE DEVOPS – PRODUCT TEAM Responsive team 24 hours 3 SRE’s assigned
- 49. SPONSOR BUY IN
- 50. AZURE DEVOPS
- 51. EVENT DAY
- 52. CHALLENGES WEBSITE Explanation Detect Respond: Quick fix Postmortem Recover
- 53. DISRUPTIONS Docker containers to disrupt the webshop Start, stop, validate and scoring Isolated Technology stack Parameters injected docker run -ti -e "settingsLocation=https://…/team01.json" gdbc/certificateexpired
- 54. CONTAINER REQUIREMENTS • Asynchronous • Fast • Scalable X 30.000
- 55. WHERE TO RUN Azure Container Instances No initial setup Slow start of containers Soft limits Azure Kubernetes Services Provision cluster Limited to available hardware Limited to nodes, scale up
- 57. MONITORING
- 61. YOU BUILD IT, YOU RUN IT 36 hours
- 62. RISK MANAGEMENT
- 63. MULTIPLE DEPLOYMENTS • Challenges website • Scoreboard • Extra teams
- 64. LEARNINGS 36 hours monitoring is hard! Find a SRE in a different timezone Preparation is key Caching and scaling really helps 4:00 AM
- 65. THE DAY AFTER
- 66. SCALING DOWN
- 68. TAKE AWAYS START WITH A VISION DESIGN WITH THE END IN MIND ENGAGE SPONSORS AND STAKEHOLDERS THINK BIG BUILD VS BUY RISK MANAGEMENT CUSTOMER HAPPINESS
- 70. https://xpir.it/LinksGDBC ROB BOS - @ROBBOS81 MICHIEL VAN OUDHEUSDEN - @MIVANO
Hinweis der Redaktion
- We started in New Zealand Stopped in Seattle
- You just saw the intro video for the global devops bootcamp 2019. A community event we organized for the last three years. We want to tell you this story on how we run this over 35 countries with 10000 participants, 7 azure devops environment and causing 3 outages in Azure DevOps.
- Who are you doing it for / with? Vision Working together in a globally distributed team Open source / daily job Clearly defined purpose Empowerement Isolated architecture as a starting point Communication Technical design decisions No big design up front Do think big Scalable Monetary restrictions Techical restrictions Design with the end in mind
- Who knows about Global DevOps Bootcamp?
- Team of volunteers to create content Volunteers Sponsoring in time/money/Azure Credits/Tweet wall/Snyk package scanner support Planning: Weekly meeting in MS Teams
- Community
- Radio TFS Podcast appearance MVP mailing list
- Build a community of local venues and GDBC Core Team Slack Community calls Twice in the preparation fase Explain ‘event out of the box’ Content planning Proctor preparation Different time zones Record and reshare Centralized wiki Stakeholder management
- Event out of the box: explain the issues with 2017 / 2018 handling
- 110 venues handled this way
- Starting to feel awesome, lets start promoting the event Microsoft DevOps Advocates: Damian Brady
- Designer Font and color selection Share the styling guide with vector graphics with the organizers Mugs, custom t-shirts, posters, etc. We don’t ship the swag
- Reference 2018 edition with the work-items in markdown YAML header Generates the challenges website Collaboration
- Focus on the run part, so easy application Reference “The Phoenix Project” Slimmed down version of the Webshop
- Message from the CEO of Parts Unlimited
- SWITCH SPEAKER What do we need to provision in Azure
- Goal is a working webshop for a team of attendees Webshop: App Service / Sql DB / Application Insights Deliver working webshop for attendees + CI/CD pipeline in Azure DevOps
- Webshop has been selected Team of 5 attendees
- Worldwide event spanning multiple time zones 80-100 venues
- Our own pipeline Azure Separated for fast iterations during preparation phase
- No interesting limits on e.g. App Service Plans (100 per resource group). SQL Server: 20 is a soft limit: resolved through support ticket (two days to late!) 1200 / 4 subs = 300 SQL Servers…. 200 is a hard limit
- Crucial factor, Credit Card
- Inception! What now? Rolling out webshops to production 1 Azure DevOps Project to provision Azure DevOps team projects (x1200)
- SWITCH SPEAKER
- Per team needed
- 7 Azure DevOps organizations: one for each supported region All attached to same AAD: so one account to rule them all Concurrent hosted pipelines: 100 or 150 per organization: 200 teams per organization
- 7 Azure DevOps organizations: one for each supported region All attached to same AAD: so one account to rule them all Concurrent hosted pipelines: 100 or 150 per organization: 200 teams per organization Peak usage: 700 concurrent pipelines
- Full pipeline overview
- Rows are the different regions / organizations Columns are the different steps
- Certificate separate : timing issue with App Service Ready
- Part 1: Export dataset = ConsoleApp.exe Provision Azure DevOps team projects + AD DNS here, because it takes a while to be ready Part 2: Init AzDo Team Project Part 3: Trigger all the builds (would incur cost)
- Tell the yaml issue story: New repo with yaml file for the build: no pipeline! Remove yaml file, re-add: pipeline! Extra step added to do this
- Build pipeline outage in 2 regions: starting 400 pipelines (with 400 CD releases after them!) causes some load on Azure DevOps Scaled down to 1 concurrent pipeline on all regions Austalia: Networking issue on Scaleset Npm restore failed Rene in call until 12 PM
- Supportive product team
- Supportive product team
- This is the break
- SWITCH SPEAKER
- SWITCH SPEAKER
- Switch to gdbc-challenge-com Secured by Azure AD integration Table storage for team state
- ACI: nice, no orchestration needed, might be cheaper ACI: to slow to start a container ACI Constraint: max 300 container create per hour
- 3 clusters in 3 regions 5 nodes Servicebus to queue and process Blob storage with JSON for team settings Start container in nearest AKS cluster (3 clusters, each 5 nodes)
- Google Analytics AppIication Insights
- Custom events with tracking ID
- Checking the load on the cluster – Region Europe We see that pods are starting, stopping isn’t that visible We use short lived pods, but the default garbage collection is on 12.000 pods, so that line doesn’t go down
- Slack channel + Bridge on Teams Slack channel #SRE Bridge on Microsoft Teams Command Center
- Monitoring, be in control Fallbacks, caching and backups
- No downtime Users did not notice During the event:
- Preparation is key Support tickets for raising SQL Servers limits took 2 weeks and where to late! Twitter preparation Event starts in New Zealand: that is 11 PM for us in NL New Zealand: 11:00 PM Started Europe: 09:00 AM West coast US ended at 02:00 AM
- Have content available for after event Thank you mails Summaries Behind the scenes
- Sunday cleanup Azure DevOps was easy, just delete the organizations In Azure we cleaned everything up. Took a whole day ConsoleApp!!