Installing one CI server or configuring a deployment pipeline for a specific application might be easy enough. However, as enterprises look to scale their DevOps adoption and optimize their software delivery practices across the organization (to support additional teams, product lines, application releases, processes and infrastructure) -- software delivery pipeline(s) need to scale to support enterprise workloads. For some enterprises, this means having a pipeline that can withstand the velocity and throughput of thousands of product releases, supporting tens of thousands of developers and distributed teams, hundreds of thousands of infrastructure nodes, multitudes of inter-dependent application components, or millions of builds and test-cases. This scale poses unique challenges and implications for your pipeline design. This talk covers best practices for analyzing and (re)designing your software delivery pipeline – regardless of your chosen tool-set or technologies. Obtain tips and tools for ensuring your pipelines and DevOps infrastructure have the right architecture and feature-set to support your software production as it scales, while also ensuring manageability, governance, security, and compliance. Learn best practices for how to: 1) Plan for scale: how to project for the types of performance indicators/vectors you’d need to scale across. 2) How to design of your pipeline and supporting infrastructure and operations (such as data retention, artifact retrieval, monitoring, etc.). 3) Design your pipeline workflows and processes to allow reusability and standardization across the organization, while also enabling flexibility to support the needs of specific teams/apps. 4) Design your pipeline in a way that enables fast rollout- easy onboarding thousands of applications, across hundreds of teams 5) Incorporate security access controls, approval gates and compliance checks as part of your pipeline and have them standard across all releases 6) Ensure your architecture support HA, DR and business continuity.

1. electric-cloud.com
#DOES16
Planning For Huge Scale:
Designing Your Pipelines for
Scalability and Resiliency
Avan Mathur | Product Manager
amathur@electric-cloud.com | @avantika_ec

2. electric-cloud.com
#DOES16
What do I Mean by Huge Scale?
• A Fortune 5 Bank
• Deploying to 140K endpoints
• 6000s of applications on-boarded at accelerated pace
• Chip Manufacturer
• 100K Jobs/Day
• Constant load on the system
• High job throughput

3. electric-cloud.com
#DOES16
Plan Ahead – Know Your End Goal
• Estimate and plan for long term scale
• Define your success criteria
• # of deployments per window
• Max duration of a deployment
• Number of concurrent deployments
• Benchmarking
• Validate numbers with benchmarks
• Testing real workloads in real environments

4. electric-cloud.com
#DOES16
Metrics to Measure
• CI Cycle Times
• Deployment Frequency and Duration
• Concurrent Deployments
• Release Frequency

5. electric-cloud.com
#DOES16
Focus Areas
• Infrastructure
• Data
• Pipeline Design
• Monitoring / feedback loop

6. electric-cloud.com
#DOES16
Infrastructure
• High Availability
• Horizontal Scalability
• Distribute load and
components
• Database sizing and reliability
• Understand and plan for
bottlenecks and hardware
limitations

7. electric-cloud.com
#DOES16
Recommended Architecture
Commander Server
Commander ServerClustered
Orchestration Servers
Web Server
Database
AgentAgentAgentAgentAgent
Workspace Repository 1
Browser
AgentAgentAgentAgentAgent
Web Server
Site 2
WAN
Workspace
Repository 2
Browser
Web Server
Site 3
Repository 3
Site 1
Browser
AgentAgentAgentAgentAgent
Workspace
HA/
HS

8. electric-cloud.com
#DOES16
Disaster Recovery Architecture
Commander Server
Commander ServerClustered
Orchestration Servers
Web Server
Database
AgentAgentAgentAgentAgent
Workspace Repository 1
DR Site
HA/
HS
Commander Server
Commander ServerClustered
Orchestration Servers
Web Server
Database
AgentAgentAgentAgentAgent
Workspace Repository 1
Primary
Site
HA/
HS
Disk Replication
WAN
DB Mirroring

9. electric-cloud.com
#DOES16
Managing Data
• Each process generates data
 Metadata, log files, build & test data,
monitors, artifacts …
• Understand growth rates and plan
accordingly
 Different areas to monitor
o Database
o Filesystem
o Artifact Repository
• Establish Data Retention & Archiving Policies early

10. electric-cloud.com
#DOES16
Pipeline Design: Scaling Across an Organization
1. Plan and monitor for performance at
scale at every step
2. Build a generic model to standardize
across apps
3. Reusable components that can be used
across projects
4. Automated onboarding of new projects
5. Plan and Define Access Controls and
Roles early in the process

11. electric-cloud.com
#DOES16
Monitoring
• Monitor key metrics
 Understand patterns
 Foresee upcoming peaks in usage
 Plan ahead for hardware expansion
or distribution
• Best Practices
 Unobtrusive
 Automated
 Not only in production
 One pane of glass

12. electric-cloud.com
#DOES16
Monitoring

13. electric-cloud.com
#DOES16
Questions?