This document summarizes a presentation given by Tim Bell on the CERN Infrastructure Evolution. It discusses how CERN has transformed its technology over time due to constraints on resources. CERN now uses open source tools like OpenStack and Puppet to manage over 50,000 computer cores across Geneva and Budapest. CERN contributes code back to open source communities and benefits from cultural changes around more agile development practices. The goal is to manage over 300,000 computer cores on the private cloud by 2015.

1. Tuesday, October 1, 13

2. CERN’S GRAND COLLISION EXPERIMENT: CLOUD
SPEAKER: Tim Bell
Infrastructure Manager
CERN
Tuesday, October 1, 13

3. Tuesday, October 1, 13

4. Tim Bell
tim.bell@cern.ch
@noggin143
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

5. 19/09/2013 CERN Infrastructure Evolution
CERN was founded 1954: 12 European States
“Science for Peace”
Today: 20 Member States
Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland,
France, Germany, Greece, Hungary, Italy, the Netherlands, Norway, Poland, Portugal,
Slovakia, Spain, Sweden, Switzerland and
the United Kingdom
Candidate for Accession: Romania
Associate Members in Pre-Stage to Membership: Israel, Serbia
Applicant States for Membership or Associate Membership:
Brazil, Cyprus (awaiting ratification), Pakistan, Russia, Slovenia, Turkey, Ukraine
Observers to Council: India, Japan, Russia, Turkey, United States of America;
European Commission and UNESCO
~ 2,300 staff
~ 1,000 other paid personnel
> 11,000 users
Budget (2013) ~1,000 MCHF
Tuesday, October 1, 13

6. What are the Origins of Mass ?
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

7. Matter/Anti Matter Symmetric?
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

8. Where is 95% of the Universe?
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

9. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

10. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

11. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

12. Collisions
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

13. A Big Data Challenge
19/09/2013 CERN Infrastructure Evolution 11
Tuesday, October 1, 13

14. The CERN Meyrin Data Centre
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

15. New Data Centre in Budapest
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

16. Good News, Bad News
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

17. We’re Not Special
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

18. 19/09/2013
Bamboo
Koji, Mock
AIMS/PXE
Foreman
Yum repo
Pulp
Puppet-DB
mcollective, yum
JIRA
Lemon /
Hadoop /
LogStash /
Kibana
OpenStack
Nova
Hardware database
Puppet
Active Directory /
LDAP
CERN Infrastructure Evolution
Tuesday, October 1, 13

19. The Agile Experience
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

20. Cultural Barriers
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

21. Cultural Barriers
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

22. Cultural Barriers
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

23. Cultural Barriers
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

24. Status
• Toolchain implemented in 18 months with
enhancements and bug fixes submitted back to the
community
• Now in production in 3 OpenStack clouds (over
50,000 cores in total) in Geneva and Budapest
managed by Puppet
• Target is more than 300,000 cores by 2015 and
90% compute resources in the private cloud
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

25. Summary
• Constraints on resources have led to major
technology transformations at CERN
• Open source community participation helps
drive cultural change and motivates staff
• CERN benefits and contributes back through
code and outreach
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

26. Questions ?
19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

27. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

28. Service Models
19/09/2013 CERN Infrastructure Evolution
• Pets are given names like pussinboots.cern.ch
• They are unique, lovingly hand raised and cared for
• When they get ill, you nurse them back to health
• Cattle are given numbers like vm0042.cern.ch
• They are almost identical to other cattle
• When they get ill, you get another one
Tuesday, October 1, 13

29. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

30. 19/09/2013 CERN Infrastructure Evolution
http://www.eucalyptus.com/blog/2013/04/02/cy13-q1-community-analysis-%E2%80%94-openstack-vs-opennebula-vs-eucalyptus-vs-
cloudstack
Tuesday, October 1, 13

31. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

32. 19/09/2013 CERN Infrastructure Evolution
Tier-­‐1
(11
centres):
•Permanent
storage
•Re-­‐processing
•Analysis
Tier-­‐0
(CERN):
•Data
recording
•Ini7al
data
reconstruc7on
•Data
distribu7on
Tier-­‐2
(~200
centres):
•
Simula7on
•
End-­‐user
analysis
• Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid
• In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs
Tuesday, October 1, 13

33. 19/09/2013 CERN Infrastructure Evolution
Tuesday, October 1, 13

34. 19/09/2013 CERN Infrastructure Evolution
Microsoft Active
Directory
CERN DB
on Demand
CERN Network
Database
Account mgmt
system
Horizon
Keystone
Network
Compute
Glance
Scheduler
Cinder
Nova
Block Storage
Provider
Tuesday, October 1, 13

35. Training for Newcomers
19/09/2013 CERN Infrastructure Evolution
Buy the book rather than guru mentoring
Tuesday, October 1, 13

36. Tuesday, October 1, 13