Resource: LCE13 Name: Linaro Infrastructure Update Date: 12-07-2013 Speaker: Video: https://www.youtube.com/watch?v=UeXIQIa-RTA

1. LAVA Infrastructure
Services Update
Paul Sokolovsky, Milo Casagrande,
James Tunnicliffe, Philip Colmer
LCE13 - Dublin, July 2013
Present & Future Infrastructure Review

2. ● There was was a separate Infrastructure
Team before 2013-03
● As Linaro infrastructure matures and
stabilizes, Infrastructure team was merged
with LAVA team, as account for "A" in its
name (Linaro Automation and Validation).
● As an automation team, LAVA owns few
codebases and tools (mostly related to CI)
and co-maintains services based on them
together with ITS team.
Intro and History

3. ● Non-exhaustive list of tools, projects and
services (co)maintained by LAVA team (not
counting LAVA server itself):
○ linaro-image-tools
○ Jenkins setup on ci.linaro.org & android-build.linaro.
org
○ Frontend app on android-build.linaro.org
○ AOSP/upstream mirror system
○ Gerrit on review.android.git.linaro.org
○ git.linaro.org & android.git.linaro.org
○ cbuild.validation.linaro.org
Intro

4. ● Servers migration from Canonical
Datacenter to Linaro Cloud (driven by ITS)
● Switching to Linaro Login as SSO for all
Linaro services (driven by ITS)
● Gerrit upgrade to 2.5
○ Contemporary Gerrit version with improved
permission granularity
○ Allowed to fully automate AOSP mirror without
compromising security
Recent Activity

5. ● Dynamic publishing
● Jenkins setup for complete public vs private
build separation
● Improved Git service with Rhodecode
frontend
● Patches.linaro.org
○ Patchwork codebase forked to add nice metrics.
Original project:
■ More or less popular among big FOSS projects
■ Undermaintained (Google for it)
○ plan to migrate to support Crowd auth
Upcoming Projects

6. System management software to
standardize on?
● We'd like to pass maintenance of existing infra
to ITS, but codebases are not in ideal shape for
production or devel deployment.
● Multitude of heterogeneous adhoc solutions.
● Existing system config management software:
○ Puppet, Chef - Legacy
○ SaltStack - Already used for LAVA lab deployment,
requires adhoc daemons
○ Ansible - Requires only ssh
Background tasks and questions

7. ● Scalability
○ git:// protocol is greedy - consumes RAM/resources
● Stability & Reliability
○ OOM errors on server
● Easy to use interface
○ No SSH to create a repository
● Integrated ACL system
○ With authentication
○ That can use Linaro authentication system
● Private repositories
○ Preferably on the same instance
We do not want to lock anybody out or blindly throw
hardware at the problem
Rhodecode - Problems it solves

8. ● Open source project
○ Written in Python
● Integrated LDAP support
○ Crowd support available in beta version of RhodeCode
● Easily scalable
○ We run 4 instances on the same machine already
● Web UI
○ Easy to create repositories
○ Customizable
● Admin interface, groups support, ...
Bonus:
● Has integrated code review (with inline comments)
Rhodecode - Why

9. ● Provides dumb HTTP/HTTPS clone support
○ git processes were spawned, could take up to 1.5GB RAM/process
○ Now done via Apache X-SendFile
■ Slightly slower than git protocol
○ Authorization always happens in RhodeCode
● Authorization caching for HTTP/HTTPS clone
operations
○ Files are served one at the time - auth happened on each file
○ Needed to speed up clones (from ~35s to ~5s for a small repo)
● User groups and users matched to system ones
○ Needed to provide git+ssh access to Linaro engineers
○ Needed for push operation to happen with SSH keys
○ Access is granted via file system level ACLs
● Small UI tweaks
○ Added git+ssh clone URLs on the repository page
Rhodecode - Linaro changes

10. ● Android
○ Git & Gerrit
● LAVA
○ BZR (planning to move to Git & Gerrit)
■ need to review Rhodecode/Gerrit Integration
● Git general
○ Rhodecode (previously gitweb)
■ Migration Timeline
Overall Git Architecture

11. ● System management software to
standardize on?
● Further optimizations to android-build
performance?
●
● LAVA
○ transitioning to Git / Gerrit
Future Plans