Rakuten Developer Meetup vol.02: How to Manage Large-scale Infrastructure https://rakuten.connpass.com/event/243219/

1. Making Cloud Native CI/CD Services
May 19th, 2022
Takizawa Takeshi
Cloud Platform Department
Rakuten Group, Inc.

2. 2
About Me
Takizawa, Takeshi | Takky
Organization
Cloud Platform Department (CPD)
Application Platform Section: CI/CD Platform Group
Apr 2010 – Rakuten
Application Engineer
Automation Engineer
Application Architect
Manage
One Cloud Services:
- CI/CD (Jenkins)
- Registry-aaS (Harbor)
- 3rd Party Tools:
GitHub Enterprise
Artifactory
CircleCI

3. 3
CONTENTS
1. What is Cloud Native?
3. How large are our services?
2. Our System Architecture Diagrams4
4. Current issues
5. Future plans

4. 4
What is Cloud Native?
Cloud native technologies empower organizations to build and run scalable
applications in modern, dynamic environments such as public, private, and hybrid
clouds. Containers, service meshes, microservices, immutable infrastructure, and
declarative APIs exemplify this approach.
These techniques enable loosely coupled systems
that are resilient, manageable, and observable.
Combined with robust automation, they allow
engineers to make high-impact changes
frequently and predictably with minimal toil.
https://github.com/cncf/toc/blob/main/DEFINITION.md
CI/CD

5. 5
What is Cloud Native?
Cloud native technologies empower organizations to build and run scalable applications in modern,
dynamic environments such as public, private, and hybrid clouds. Containers, service meshes,
microservices, immutable infrastructure, and declarative APIs exemplify this approach.
These techniques enable loosely coupled systems that are resilient, manageable, and observable.
Combined with robust automation, they allow engineers to make high-impact changes frequently
and predictably with minimal toil.
https://github.com/cncf/toc/blob/main/DEFINITION.md
Private Cloud => One Cloud
Robust automation => Scope of Our Group
Run scalable applications => Container Platform Group

6. 6
System Architecture Diagrams
There are multiple Kubernetes clusters in a Region.
There is one Jenkins cluster in a Region.
There is one Harbor cluster in a Region.
JPE2 Region EUC1 Region USW1 Region

7. 7
System Architecture Diagrams
In order to use resource efficiently, each service uses multi-tenant approach.
JPE2 Region EUC1 Region USW1 Region

8. 8
Relationship Between Kubernetes Cluster and CI/CD Tools
Jenkins and Harbor are outside Kubernetes cluster.

9. 9
How Large Are Our Services? (The Largest Region JPE2)
DEV env: 350+ nodes, 7,500+ pods
PROD env: 900+ nodes, 15,000+ pods
8,400+ jobs
2,000+ builds / day
14,000+ projects
260,000+ images

10. 10
Current Issue
Compared to Kubernetes and Harbor, Jenkins is a legacy tool and is close to its
performance limits in its current architecture.
• Agent nodes are provisioned statically
• Controller node is single point of failure

11. 11
Solution
Dynamically Provision CI/CD tools for each Tenant on Kubernetes.

12. 12
There are more challenges
• Releases must be operated from the
corresponding Jenkins, which is inconvenient.
• Registry-aaS doesn't have geo-redundancy
yet.
• Etc.,
画像挿入予定

13. 13
The experience of rearchitecting
a CI/CD platform is very rewarding!

14. 14
Appendix: Flow of new namespace creation