SlideShare ist ein Scribd-Unternehmen logo
1 von 16
Downloaden Sie, um offline zu lesen
CI/CD Pipelines and Cluster Management: Heterogeneous,
Lightweight, Safe
OCurrent, OBuilder, OCluster
Antonin Décimo _ @Rucikir a @MisterDA
July 19, 2022
Tarides https://tarides.com
1
Tarides & OCaml
The OCaml programming language: general-purpose, industrial-strength,
expressive, safe.
let rec total t =
match t with
| Leaf -> 0
| Node (l, x, r) -> total l + x + total r
(* val total : int tree -> int = <fun> *)
let rec flip t =
match t with
| Leaf -> Leaf
| Node (l, x, r) -> Node (flip r, x, flip l)
(* val flip : 'a tree -> 'a tree = <fun> *)
• type safe + memory safe = secure by construction!
• maintainable programs
2
opam: the OCaml Package Manager
A package manager and a package archive:
• ~3900 different packages, ~24000 versions
• explicit version constraints
Goals:
• test the ecosytem at scale
• empower the developers
3
CI Pipelines
opam-health-check weekly check of all ~24000 opam packages
http://check.ocamllabs.io/
opam-repo-ci check packages for inclusion in the archive
opam-docs-ci build and publish documentation for all packages
ocaml-ci CI for OCaml projects (~430 projects)
https://ci.ocamllabs.io/github/MisterDA/ocurrent
docker-base-images OCaml & opam Docker images (19 systems × 13 OCaml
versions)
https://images.ci.ocaml.org/
4
CI Pipelines in OCaml!
OCurrent: an OCaml embedded Domain Specific Language to describe pipelines.
• no more YAML or embedded JavaScript!
• expressiveness of a real programming language
• leverage the entire OCaml ecosystem
https://github.com/ocurrent/overview
5
OCurrent: build a Docker image from a git repo
head head commit build … run
let pull = false
let timeout = Duration.of_min 50
(* Run "docker build" on the latest commit in Git repository
* [repo]. *)
let pipeline ~repo () =
let src = Git.Local.head_commit repo in
let image = Docker.build ~pull ~timeout (`Git src) in
Docker.run image ~args:["./skynet.exe"; "--help"]
6
OCurrent libraries
current_incr • define changeable values and changeable computations
• keep tracks of changes and propagate them
• automatic caching
current_* plugins: a toolbox for CI/CD pipelines
7
OCurrent plugins: monitoring inputs
• git events
Current_git.clone : string -> Commit.t Current.t
• webhooks from GitHub and GitLab
Current_github.Api.ci_refs :
Current_github.Repo_id.t -> Commit.t list Current.t
• new Docker images
• web events
8
OCurrent plugins: outputs
• Docker images
• Notifications (Slack, Matrix, build statuses, ...)
Current_slack.post :
channel -> key:string -> string Current.t ->
unit Current.t
• Artifacts and logs
• Web UI
9
Inspectable
pull
debian-11 4.14 x86_32 opam-2.1 ✔
opam-vars ✔
pull
debian-11 4.14 x86_64 opam-2.1 ✔
Analyse ✔
pull
debian-11 4.03 x86_64 opam-2.1 ✔ opam-vars ✔
pull
debian-11 4.13 x86_64 opam-2.1 ✔ opam-vars ✔
pull
debian-11 4.14 x86_64 opam-2.1 ✔ opam-vars ✔
pull
debian-11 4.14 x86_32 opam-2.0 ✔ opam-vars ✔
pull
debian-11 4.14 x86_64 opam-2.0 ✔
pull
debian-11 4.03 x86_64 opam-2.0 ✔
opam-vars ✔
pull
debian-11 4.13 x86_64 opam-2.0 ✔
opam-vars ✔
pull
debian-11 4.14 x86_64 opam-2.0 ✔
opam-vars ✔
ocaml/opam-repository:refs/heads/master
head head commit
debian-11-4.13
debian-11-4.14
(lint-opam)
(lint-doc)
(lint-fmt)
build ✔
summarise
build ✔
build ✔
build ✔
build ✔
10
Reproductible
All jobs generate equivalent Dockerfiles
git clone --recursive "https://github.com/MisterDA/ocurrent.git" -b "master" && cd "ocurr
cat > Dockerfile <<'END-OF-DOCKERFILE'
FROM ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135
# debian-11-4.14
USER 1000:1000
RUN sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam
WORKDIR /src
RUN sudo chown opam /src
RUN cd ~/opam-repository && (git cat-file -e 8270cc5e433a2e14831089f1129059ef1077e5e5 ||
COPY --chown=1000:1000 current_web.opam current_slack.opam current_rpc.opam current_gitla
RUN opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.dev './' && o
ENV DEPS="alcotest.1.5.0 alcotest-lwt.1.5.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 asn1
RUN opam depext --update -y current_web.dev current_slack.dev current_rpc.dev current_git
RUN opam install $DEPS
COPY --chown=1000:1000 . /src/
RUN opam exec -- dune build @install @check @runtest && rm -rf _build
END-OF-DOCKERFILE
docker build .
11
Obuilder: execute jobs
A lightweight Docker build: takes a build script and performs build steps in a
sandboxed environment.
Linux Windows macOS
Cache BTRFS/ZSF Docker images Rsync
Sandbox runc Docker run User-level isolation
12
Obuilder jobs
Docker-like syntax or calls to obuilder-spec library
((from ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135
(comment debian-11-4.14)
(user (uid 1000) (gid 1000))
(run (shell "sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam"))
(workdir /src)
(run (shell "sudo chown opam /src"))
(run (cache (opam-archives (target /home/opam/.opam/download-cache)))
(network host)
(shell "cd ~/opam-repository && (git cat-file -e dcf32445c6c5322cbd5891bf0aa3cb6ba4
(copy (src current_web.opam current_slack.opam current_rpc.opam current_gitlab.opam curr
(dst ./))
(run (network host)
(shell "opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.de
(env DEPS "alcotest.1.6.0 alcotest-lwt.1.6.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 as
(run (cache (opam-archives (target /home/opam/.opam/download-cache)))
(network host)
(shell "opam depext --update -y current_web.dev current_slack.dev current_rpc.dev c
(run (cache (opam-archives (target /home/opam/.opam/download-cache)))
13
Pipelines with OCurrent!
• integrated with webhooks and services
• native and efficient build and sandboxing
• automagic caching
• reproducible
• inspectable
14
OCluster: the cluster management system
• manages pools of workers
• a scheduler accepts jobs from
clients and distributes then to
workers
• caching across machines
• logs forwarding
• global monitoring
• estimated time of execution
• communication using Cap'n
Proto (RPC protocol)
linux-x86_64
windows-x86_64
linux-arm64
worker 1
worker 2
worker 3
worker 4
worker 5
scheduler
ocaml-ci
15
Refs & Thanks!
All three independent components:
OCurrent the eDSL to describe pipelines
OBuilder the sandboxed build engine
OCluster the cluster management system
See our CI/CD pipelines at https://github.com/ocurrent/overview !
Started by Thomas Leonard and supported by contributors at Tarides.
Thanks! any questions?
16

Weitere ähnliche Inhalte

Ähnlich wie stackconf 2022: Cluster Management: Heterogeneous, Lightweight, Safe. Pick Three

もうひとつのコンテナ実行環境 runq のご紹介
もうひとつのコンテナ実行環境 runq のご紹介もうひとつのコンテナ実行環境 runq のご紹介
もうひとつのコンテナ実行環境 runq のご紹介Takehiko Amano
 
Cross-compilation native sous android
Cross-compilation native sous androidCross-compilation native sous android
Cross-compilation native sous androidThierry Gayet
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesPhil Estes
 
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned  Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned RightScale
 
Shifter: Containers in HPC Environments
Shifter: Containers in HPC EnvironmentsShifter: Containers in HPC Environments
Shifter: Containers in HPC Environmentsinside-BigData.com
 
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik DornJDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik DornPROIDEA
 
Introduction to Docker at the Azure Meet-up in New York
Introduction to Docker at the Azure Meet-up in New YorkIntroduction to Docker at the Azure Meet-up in New York
Introduction to Docker at the Azure Meet-up in New YorkJérôme Petazzoni
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetesLiran Cohen
 
Running Docker in Development & Production (DevSum 2015)
Running Docker in Development & Production (DevSum 2015)Running Docker in Development & Production (DevSum 2015)
Running Docker in Development & Production (DevSum 2015)Ben Hall
 
Powercoders · Docker · Fall 2021.pptx
Powercoders · Docker · Fall 2021.pptxPowercoders · Docker · Fall 2021.pptx
Powercoders · Docker · Fall 2021.pptxIgnacioTamayo2
 
Introduction to Docker - Learning containerization XP conference 2016
Introduction to Docker - Learning containerization  XP conference 2016Introduction to Docker - Learning containerization  XP conference 2016
Introduction to Docker - Learning containerization XP conference 2016XP Conference India
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis OverviewLeo Lorieri
 
Docker HK Meetup - 201707
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707Clarence Ho
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryImesh Gunaratne
 
時代在變 Docker 要會:台北 Docker 一日入門篇
時代在變 Docker 要會:台北 Docker 一日入門篇時代在變 Docker 要會:台北 Docker 一日入門篇
時代在變 Docker 要會:台北 Docker 一日入門篇Philip Zheng
 
Docker Clustering - Batteries Included
Docker Clustering - Batteries IncludedDocker Clustering - Batteries Included
Docker Clustering - Batteries IncludedC4Media
 
BDM32: AdamCloud Project - Part II
BDM32: AdamCloud Project - Part IIBDM32: AdamCloud Project - Part II
BDM32: AdamCloud Project - Part IIDavid Lauzon
 

Ähnlich wie stackconf 2022: Cluster Management: Heterogeneous, Lightweight, Safe. Pick Three (20)

もうひとつのコンテナ実行環境 runq のご紹介
もうひとつのコンテナ実行環境 runq のご紹介もうひとつのコンテナ実行環境 runq のご紹介
もうひとつのコンテナ実行環境 runq のご紹介
 
Cross-compilation native sous android
Cross-compilation native sous androidCross-compilation native sous android
Cross-compilation native sous android
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use Cases
 
Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned  Real-World Docker: 10 Things We've Learned
Real-World Docker: 10 Things We've Learned
 
Shifter: Containers in HPC Environments
Shifter: Containers in HPC EnvironmentsShifter: Containers in HPC Environments
Shifter: Containers in HPC Environments
 
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik DornJDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn
 
App container rkt
App container rktApp container rkt
App container rkt
 
Introduction to Docker at the Azure Meet-up in New York
Introduction to Docker at the Azure Meet-up in New YorkIntroduction to Docker at the Azure Meet-up in New York
Introduction to Docker at the Azure Meet-up in New York
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetes
 
Running Docker in Development & Production (DevSum 2015)
Running Docker in Development & Production (DevSum 2015)Running Docker in Development & Production (DevSum 2015)
Running Docker in Development & Production (DevSum 2015)
 
Powercoders · Docker · Fall 2021.pptx
Powercoders · Docker · Fall 2021.pptxPowercoders · Docker · Fall 2021.pptx
Powercoders · Docker · Fall 2021.pptx
 
Learning Docker with Thomas
Learning Docker with ThomasLearning Docker with Thomas
Learning Docker with Thomas
 
Introduction to Docker - Learning containerization XP conference 2016
Introduction to Docker - Learning containerization  XP conference 2016Introduction to Docker - Learning containerization  XP conference 2016
Introduction to Docker - Learning containerization XP conference 2016
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
 
Docker HK Meetup - 201707
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707
 
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App FactoryRevolutionizing WSO2 PaaS with Kubernetes & App Factory
Revolutionizing WSO2 PaaS with Kubernetes & App Factory
 
時代在變 Docker 要會:台北 Docker 一日入門篇
時代在變 Docker 要會:台北 Docker 一日入門篇時代在變 Docker 要會:台北 Docker 一日入門篇
時代在變 Docker 要會:台北 Docker 一日入門篇
 
Docker Clustering - Batteries Included
Docker Clustering - Batteries IncludedDocker Clustering - Batteries Included
Docker Clustering - Batteries Included
 
BDM32: AdamCloud Project - Part II
BDM32: AdamCloud Project - Part IIBDM32: AdamCloud Project - Part II
BDM32: AdamCloud Project - Part II
 
Docker.io
Docker.ioDocker.io
Docker.io
 

Kürzlich hochgeladen

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingScyllaDB
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jNeo4j
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 

Kürzlich hochgeladen (20)

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 

stackconf 2022: Cluster Management: Heterogeneous, Lightweight, Safe. Pick Three

  • 1. CI/CD Pipelines and Cluster Management: Heterogeneous, Lightweight, Safe OCurrent, OBuilder, OCluster Antonin Décimo _ @Rucikir a @MisterDA July 19, 2022 Tarides https://tarides.com 1
  • 2. Tarides & OCaml The OCaml programming language: general-purpose, industrial-strength, expressive, safe. let rec total t = match t with | Leaf -> 0 | Node (l, x, r) -> total l + x + total r (* val total : int tree -> int = <fun> *) let rec flip t = match t with | Leaf -> Leaf | Node (l, x, r) -> Node (flip r, x, flip l) (* val flip : 'a tree -> 'a tree = <fun> *) • type safe + memory safe = secure by construction! • maintainable programs 2
  • 3. opam: the OCaml Package Manager A package manager and a package archive: • ~3900 different packages, ~24000 versions • explicit version constraints Goals: • test the ecosytem at scale • empower the developers 3
  • 4. CI Pipelines opam-health-check weekly check of all ~24000 opam packages http://check.ocamllabs.io/ opam-repo-ci check packages for inclusion in the archive opam-docs-ci build and publish documentation for all packages ocaml-ci CI for OCaml projects (~430 projects) https://ci.ocamllabs.io/github/MisterDA/ocurrent docker-base-images OCaml & opam Docker images (19 systems × 13 OCaml versions) https://images.ci.ocaml.org/ 4
  • 5. CI Pipelines in OCaml! OCurrent: an OCaml embedded Domain Specific Language to describe pipelines. • no more YAML or embedded JavaScript! • expressiveness of a real programming language • leverage the entire OCaml ecosystem https://github.com/ocurrent/overview 5
  • 6. OCurrent: build a Docker image from a git repo head head commit build … run let pull = false let timeout = Duration.of_min 50 (* Run "docker build" on the latest commit in Git repository * [repo]. *) let pipeline ~repo () = let src = Git.Local.head_commit repo in let image = Docker.build ~pull ~timeout (`Git src) in Docker.run image ~args:["./skynet.exe"; "--help"] 6
  • 7. OCurrent libraries current_incr • define changeable values and changeable computations • keep tracks of changes and propagate them • automatic caching current_* plugins: a toolbox for CI/CD pipelines 7
  • 8. OCurrent plugins: monitoring inputs • git events Current_git.clone : string -> Commit.t Current.t • webhooks from GitHub and GitLab Current_github.Api.ci_refs : Current_github.Repo_id.t -> Commit.t list Current.t • new Docker images • web events 8
  • 9. OCurrent plugins: outputs • Docker images • Notifications (Slack, Matrix, build statuses, ...) Current_slack.post : channel -> key:string -> string Current.t -> unit Current.t • Artifacts and logs • Web UI 9
  • 10. Inspectable pull debian-11 4.14 x86_32 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.1 ✔ Analyse ✔ pull debian-11 4.03 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.13 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.1 ✔ opam-vars ✔ pull debian-11 4.14 x86_32 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.0 ✔ pull debian-11 4.03 x86_64 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.13 x86_64 opam-2.0 ✔ opam-vars ✔ pull debian-11 4.14 x86_64 opam-2.0 ✔ opam-vars ✔ ocaml/opam-repository:refs/heads/master head head commit debian-11-4.13 debian-11-4.14 (lint-opam) (lint-doc) (lint-fmt) build ✔ summarise build ✔ build ✔ build ✔ build ✔ 10
  • 11. Reproductible All jobs generate equivalent Dockerfiles git clone --recursive "https://github.com/MisterDA/ocurrent.git" -b "master" && cd "ocurr cat > Dockerfile <<'END-OF-DOCKERFILE' FROM ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135 # debian-11-4.14 USER 1000:1000 RUN sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam WORKDIR /src RUN sudo chown opam /src RUN cd ~/opam-repository && (git cat-file -e 8270cc5e433a2e14831089f1129059ef1077e5e5 || COPY --chown=1000:1000 current_web.opam current_slack.opam current_rpc.opam current_gitla RUN opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.dev './' && o ENV DEPS="alcotest.1.5.0 alcotest-lwt.1.5.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 asn1 RUN opam depext --update -y current_web.dev current_slack.dev current_rpc.dev current_git RUN opam install $DEPS COPY --chown=1000:1000 . /src/ RUN opam exec -- dune build @install @check @runtest && rm -rf _build END-OF-DOCKERFILE docker build . 11
  • 12. Obuilder: execute jobs A lightweight Docker build: takes a build script and performs build steps in a sandboxed environment. Linux Windows macOS Cache BTRFS/ZSF Docker images Rsync Sandbox runc Docker run User-level isolation 12
  • 13. Obuilder jobs Docker-like syntax or calls to obuilder-spec library ((from ocaml/opam@sha256:c9b4f14cd425a623c4ed33182b3a845175aa7494578d6997fbb9c71d4f2a8135 (comment debian-11-4.14) (user (uid 1000) (gid 1000)) (run (shell "sudo ln -f /usr/bin/opam-2.0 /usr/bin/opam")) (workdir /src) (run (shell "sudo chown opam /src")) (run (cache (opam-archives (target /home/opam/.opam/download-cache))) (network host) (shell "cd ~/opam-repository && (git cat-file -e dcf32445c6c5322cbd5891bf0aa3cb6ba4 (copy (src current_web.opam current_slack.opam current_rpc.opam current_gitlab.opam curr (dst ./)) (run (network host) (shell "opam pin add -yn current_web.dev './' && opam pin add -yn current_slack.de (env DEPS "alcotest.1.6.0 alcotest-lwt.1.6.0 angstrom.0.15.0 ansi.0.5.0 asetmap.0.8.1 as (run (cache (opam-archives (target /home/opam/.opam/download-cache))) (network host) (shell "opam depext --update -y current_web.dev current_slack.dev current_rpc.dev c (run (cache (opam-archives (target /home/opam/.opam/download-cache))) 13
  • 14. Pipelines with OCurrent! • integrated with webhooks and services • native and efficient build and sandboxing • automagic caching • reproducible • inspectable 14
  • 15. OCluster: the cluster management system • manages pools of workers • a scheduler accepts jobs from clients and distributes then to workers • caching across machines • logs forwarding • global monitoring • estimated time of execution • communication using Cap'n Proto (RPC protocol) linux-x86_64 windows-x86_64 linux-arm64 worker 1 worker 2 worker 3 worker 4 worker 5 scheduler ocaml-ci 15
  • 16. Refs & Thanks! All three independent components: OCurrent the eDSL to describe pipelines OBuilder the sandboxed build engine OCluster the cluster management system See our CI/CD pipelines at https://github.com/ocurrent/overview ! Started by Thomas Leonard and supported by contributors at Tarides. Thanks! any questions? 16